[12:05] <AnhVoMSFT> Hi folks, under what circumstance can we run into this error "IsADirectoryError: [Errno 21] Is a directory: '/var/lib/cloud/instance'"
[12:06] <AnhVoMSFT> full output: https://termbin.com/385u
[12:23] <smoser> AnhVoMSFT:i thought something went in very recently that intended to fix that.
[12:23] <smoser> but fwiw, you have errors before then
[12:25] <smoser> AnhVoMSFT: 0755cff078d5931e1d8e151bdcb84afb92bc0f02.
[12:31] <AnhVoMSFT> ah thanks - that indeed went in recently
[12:34] <AnhVoMSFT> yep, looking at the log I did see an error during writing ssh key
[12:34] <smoser> once htat gets created as a directory, then its done, it wont resolve itself.
[12:35] <smoser> cloud-init clean will probably fix it, or rm -Rf /var/lib/cloud/instace
[12:35] <AnhVoMSFT> how did it actually get created as a directory in the first place?
[12:36] <smoser> see the bug for one way
[12:37] <smoser> there may be other ways that it gets created, though. so it could be only same symptom
[12:39] <AnhVoMSFT> I have about 10 different cloud-init log with this issue, looking through some of them, the pattern seems to be an error during ssh-authkey-fingerprints
[12:39] <AnhVoMSFT> KeyError: "getpwnam(): name not found: 'ubuntu'"
[12:47] <AnhVoMSFT> not related to the last issue, there's another interesting one
[12:47] <AnhVoMSFT> File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 162, in maybe_perform_dhcp_discovery
[12:47] <AnhVoMSFT>     dhclient_path = subp.which('dhclient')
[12:47] <AnhVoMSFT> AttributeError: module 'cloudinit.subp' has no attribute 'which'
[12:59] <smoser> AnhVoMSFT: it feels like maybe you have a broken install (or maybe an upgrade or something?)...
[13:00] <smoser> cloudinit.subp definitely does have which in current
[13:00] <smoser> but it did move recently
[13:39] <AnhVoMSFT> these are just from random cloud-init.log that were collected as part of our telemetry for failed deployments so I don't have more context into that though. The error just seems odd because like you said, cloudinit.subp definitely has which
[14:32] <falcojr> blackboxsw_: my brain is in packaging hell right now
[14:32] <falcojr> https://github.com/canonical/cloud-init/commit/66e114a660c53400e389f119781f378311b65108#diff-4df3321932810ca4e004e72e5587ff28R95
[14:32] <falcojr> how are we getting the caplog error if we're including that?
[14:38] <falcojr> also, how did the upstream daily changelog get the latest SRU changelog when the non-daily branch doesn't have it yet?
[15:15] <blackboxsw_> minor branch for you falcojr https://github.com/canonical/cloud-init/pull/558
[15:15] <blackboxsw_> this should resolve the error because caplog is non-std on xenial's version of pytest
[15:16] <falcojr> blackboxsw_: how's this different from the other line I linked?
[15:20] <blackboxsw_> reading scrollback
[15:32] <blackboxsw_> ahh falcor, so that package dependency path differs from our daily build recipe debuild -S + sbuild-it path which I believe uses the hard/static file from ubuntu/xenial/debian/control during the build process
[16:41] <blackboxsw_> falcojr: thx for the review. https://github.com/canonical/cloud-init/pull/558 landed for ubuntu/xenial test pkg deps
[16:44] <blackboxsw_> falcojr: looks like master moved again with another commit today. So, we are ready to new-upstream-snapshot 371b392ced518e45be51089b6a67b362957b1dba to each ubuntu/X|B|F branch now
[16:44] <blackboxsw_> if you could re-fresh your existing ubuntu/* PRs then we should be good to go
[16:54] <blackboxsw_> falcojr: lucasmoura I just kicked off a daily build recipe for xenial to see that it 'works' with current ubuntu/xenial -> ubuntu/daily/xenial branches https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily-xenial
[16:54] <blackboxsw_> once we know the builder works, we can land falcor's new-upstream-snapshots
[16:54] <blackboxsw_> knowing that it didn't introduce regressions (which it shouldn't)
[17:33] <blackboxsw_> falcojr: case and point about revno in the daily build recipe not quite working as we hoped with occasional build collisions https://launchpadlibrarian.net/495249390/upload_2635183_log.txt
[17:34] <blackboxsw_> which seem to only happen when I manually click the build now button in launchpad
[18:00] <beantaxi> I notice that cloud-init 20.3 has been announced as Released, and that there is a package for Groovy, but I do not see a package for Bionic or anything else.
[18:01] <beantaxi> I am a novice at all this ... but is the idea that the cloud-init team is the cloud-init team, and it releases cloud-init code, and then different teams do the packaging for different distros? Even within Canonical? (that would make perfect sense, but that does not mean it's correct)
[18:01] <blackboxsw_> beantaxi: in order for cloud-init to publish into stable Ubuntu releases we have to follow this process for ubuntu https://wiki.ubuntu.com/CloudinitUpdates
[18:01] <blackboxsw_> we call this a Stable Release Update (SRU)
[18:02] <blackboxsw_> falcojr: and I are working today to resolve and queue a -proposed upload of 20.3 into those ubuntu releases
[18:03] <blackboxsw_> beantaxi: expectation is that once we queue uploads, the cloud-init updated packages will be available in the xenial-proposed bionic-proposed apt pockets and will be testable by anyone following these steps https://cloudinit.readthedocs.io/en/latest/topics/debugging.html#stable-release-updates-sru-testing-for-cloud-init
[18:03] <blackboxsw_> we'll send out an email to the list as well about this SRU.
[18:04] <blackboxsw_> generally when we do an upstream release to groovy 20.3 our team tries to queue an SRU into the stable releases within the next week.
[18:04] <beantaxi> Gotcha ... so it's still packaged by cloud-iniit, but there are extra steps for making a change to a stable release. That makes perfect sense.
[18:04] <blackboxsw_> that SRU testing minimally takes 7 days, but practically takes average of about 10
[18:04] <blackboxsw_> yep just so stable releases get more verification (and don't break existing users)
[18:04] <beantaxi> Is it generally safe, to grab a source release and install from source, over top of an apt package? (for cloud-init anyway; I imagine there's not a generally correct answer)
[18:05] <blackboxsw_> beantaxi: to watch progress on cloud-init SRUs you can subscribe to the SRU process bug we create
[18:06] <blackboxsw_> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1893064
[18:06] <ubot5> Ubuntu bug 1893064 in cloud-init (Ubuntu Focal) "sru cloud-init (20.2-45 to 20.3-0) Xenial, Bionic, and Focal" [Undecided,New]
[18:06] <blackboxsw_> we will update that bug attaching verification logs for every stable release we validate
[18:06] <blackboxsw_> and eventually the Xenial/Bionic/Focal items in that bug will go to Fix Released and comments added to that affect
[18:07] <blackboxsw_> details on this will come out in the SRU email that will be sent to the mailinglist
[18:08] <blackboxsw_> I need to update the IRC topic since we officially "released" 20.3 to groovy 2 days ago
[18:08] <beantaxi> Thanks! I just subscribed. Should be interesting to get a feel for the process.
=== blackboxsw_ changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting Aug 25 16:15 UTC | 20.3 (Aug 25) |  https://bugs.launchpad.net/cloud-init/+filebug
[18:08] <blackboxsw_> excellent
=== blackboxsw_ changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting Aug 25 16:15 UTC | 20.3 (Sep 8) |  https://bugs.launchpad.net/cloud-init/+filebug
[18:09] <blackboxsw_> also updated the topic for next cloud-init community status meeting: "Next status meeting Aug 25 16:15 UTC "
=== blackboxsw_ changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting Sept 816:15 UTC | 20.3 (Aug 25) |  https://bugs.launchpad.net/cloud-init/+filebug
[18:10] <blackboxsw_> let's try that again: Next status meeting Sept 816:15 UTC
=== blackboxsw_ changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting Sept 8 16:15 UTC | 20.3 (Aug 25) |  https://bugs.launchpad.net/cloud-init/+filebug
[18:10] <blackboxsw_> ... ok I really need to lay off the coffe
[18:10] <blackboxsw_> e
[18:38] <falcojr> blackboxsw_: is there something I can do to help figure out why that upload was rejected?
[18:54] <powersj> falcojr, where are you uploading to?
[18:55] <powersj> if it is a ppa you should get an email with a reason
[18:55] <falcojr> blackboxsw_ was doing it, so I'm not sure :)
[18:56] <blackboxsw_> falcojr: checking upload
[18:56] <blackboxsw_> falcojr: you mean groovy?
[18:56] <blackboxsw_> powersj: falcojr oooh you mean the daily recipe build for xenial that I linked
[18:56] <falcojr> I was asking about
[18:56] <falcojr> "falcojr: case and point about revno in the daily build recipe not quite working as we hoped with occasional build collisions https://launchpadlibrarian.net/495249390/upload_2635183_log.txt"
[18:56] <falcojr> yeah
[18:57] <blackboxsw_> right sorry. I'm context switching a bit today and not doing that very well.
[18:57] <blackboxsw_> so the daily build recipes show here https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily-xenial and https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily-bionic
[18:59] <blackboxsw_> you can dive into specific upload log failures clicking int each build
[18:59] <blackboxsw_> if needed.
[18:59] <rharper> falcojr: blackboxsw_:  no one uploads to daily pocket, for SRU we upload to the proposed pocket of the cloud-init-dev ppa ;
[18:59] <rharper> https://launchpad.net/~cloud-init-dev/+archive/ubuntu/proposed
[19:00] <blackboxsw_> right, the daily recipe just has occasional upload failures into the daily PPA because the specific version of the deb file already exists in the  daily ppa today (because I clicked the upload button multiple times)
[19:00] <rharper> yes
[19:00] <rharper> I think we have to wait for another commit, yeah ?
[19:00] <rharper> I merged a commit from paride today so tomorrow should get new hash
[19:00] <blackboxsw_> rharper: yes, it should just be a timing thing for our next commit
[19:01] <blackboxsw_> so I don't think we need to resolve this as falcojr is going to put up PRs for updated ubuntu/xenial|bionic|focal today for 20.3-X
[19:02] <rharper> yeah
[19:02] <blackboxsw_> once that lands the daily recipe will "just work" because commitish and commit count <revno> will have incremented
[19:03] <blackboxsw_> so ultimately there is nothing to do (and I think we may have already introduced the needed commit) because we are just waiting on
[19:03] <blackboxsw_> [  ]  cloud-init - 20.3-2566-g1f3a225a-0ubuntu1+1489~trunk~ubuntu16.04.1	 which is already queued in the recipe build
[19:03] <blackboxsw_> seen at https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily-xenial
[19:03] <blackboxsw_> so falcojr we should be able to proceed with a PR for your ubuntu/xenial with a  new-upstream-snapshot 371b392ced518e45be51089b6a67b362957b1dba
[19:03] <blackboxsw_> I think
[19:04] <blackboxsw_> that'd sync the latest released commit we published via ubuntu/devel into ubuntu/xenial, bionic and focal
[19:10] <falcojr> cool, sounds good
[19:47] <falcojr> ok, blackboxsw_ , maybe this one is right now??? https://github.com/canonical/cloud-init/pull/555
[19:47] <falcojr> I tried the merge and didn't get conflicts this time
[20:42] <blackboxsw_> woot falcojr xenial recipe build completed and uploaded https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily-xenial
[20:42] <blackboxsw_> falcojr: awesome getting this PR review now
[21:08] <blackboxsw_> falcojr: focal merged https://github.com/canonical/cloud-init/pull/553 :)
[21:09] <blackboxsw_> I'll queue an upload for focal -proposed of 20.3
[21:14] <blackboxsw_> falcojr: lucasmoura over in ubuntu-release channel we get a bot message confirming queued -proposed uploads Unapproved: cloud-init (focal-proposed/main) [20.2-45-g5f7825e2-0ubuntu1~20.04.1 => 20.3-2-g371b392c-0ubuntu1~20.04.1] (core, edubuntu, ubuntu-cloud)
[21:14] <blackboxsw_> hhhhhhhrm
[21:15] <blackboxsw_> right  20.3-2-g371b392c-0ubuntu1~20.04.1]
[21:15] <falcojr> what does that mean?
[21:17] <blackboxsw_> ok falcojr that means that I have uploaded (because I have "upload" rights to cloud-init)  into the focal-proposed pocket Unapproved queue... which can be found here https://launchpad.net/ubuntu/focal/+queue?queue_state=1&queue_text=cloud-init
[21:17] <falcojr> ahh, cool...just didn't know if the "unapproved" was significant
[21:18] <blackboxsw_> our next SRU step (once we actually have bionic and xenial uploads queued is to ping an SRU vangaurd in #ubuntu-release channel to say, please let our uploads into the official focal-proposed xenial-proposed bionic-proposed apt pockets so that public validation can take place
[21:18] <blackboxsw_> once sru vanguards "let the upload into -proposed" it goes from Unapproved -> Accepted I think
[21:18] <falcojr> gotcha
[21:19] <blackboxsw_> at which point public vms which have focal-proposed apt config enabled would be able to see, download and verify latest cloud-inbit
[21:20]  * blackboxsw_ looks over your bionic and xenial (I'm thinking we still might get into a bit of a pickle with the ubuntu_advantage quilt patches.....)
[22:05] <blackboxsw_> falcojr: so, something is wrong with ubuntu/bionic and the quilt refresh route because when we  `quilt push -a` on your branch, we want it to render an cc_ubuntu_advantage.py  that knows how to talk to the old ubuntu-advantage-tools back client, as in accept commands like "ubuntu-advantage disable-livepatch" instead of the new python client which only accepts commands like "ua enable livepatch".
[22:07] <falcojr> and quilt refresh doesn't do that after I check out the old version?
[22:07] <blackboxsw_> so, if we ran quilt push -f; quilt refresh, that basically drops the existing debian quilt patch, removing the config parsing and documentation that tells you things like https://cloudinit.readthedocs.io/en/18.5/topics/modules.html#ubuntu-advantage
[22:08] <blackboxsw_> notice we completely reworked the supported config keys for ubuntu_advantage in tip of cloud-init https://cloudinit.readthedocs.io/en/latest/topics/modules.html#ubuntu-advantage
[22:10] <blackboxsw_> falcojr: right, it would only do that if we want the final product of cc_ubuntu_advantage.py to be the value at the commitish git checkout f247dd20ea73f8e153936bee50c57dae9440ecf7^ cloudinit/config/cc_ubuntu_advantage.py
[22:11] <falcojr> ohhhhh, didn't realize that about the config keys
[22:11] <blackboxsw_> so I *think* what we need to do in both bionic and xenial cases is to start the quilt push -f (to apply all quilt patches that can apply) fix the end result to be the file fromgit checkout f247dd20ea73f8e153936bee50c57dae9440ecf7^ cloudinit/config/cc_ubuntu_advantage.py
[22:12] <blackboxsw_> we also (painfully) need to apply a minor subp patch (because we migrated subp utilities functions int tip to a new module and that needs to be applied to the result of   git checkout f247dd20ea73f8e153936bee50c57dae9440ecf7^ cloudinit/config/cc_ubuntu_advantage.py
[22:13] <blackboxsw_> soooo in short (not really short) to generate your ubuntu/xenial and ubuntu/bionic; you'll need to start over with a new-upstream-snapshot 371b392ced518e45be51089b6a67b362957b1dba
[22:13] <blackboxsw_> and then basically perform the following from this PR description
[22:13] <blackboxsw_> https://github.com/canonical/cloud-init/pull/435
[22:14] <blackboxsw_> it has an applicable ua-subp.patch that we'll need to use
[22:14] <blackboxsw_> and again, once we actually get the real python ubuntu-advantage-tools deb released to xenial and bionic , this whole mess goes away
[22:14] <blackboxsw_> as we can use tip of master for cc_ubuntu_advantage because the tooling supports it
[22:15] <blackboxsw_> "the tooling" == ubuntu-advantage-tools version 25.0 or later
[22:15] <blackboxsw_> which is our target shortly
[22:15] <blackboxsw_> falcojr: I'll push a PR up for ubuntu/xenial for you to compare against as you are doing it too
[22:18] <falcojr> blackboxsw_ is there an additional ubuntu advantage commit I need to cherry pick?
[22:18] <falcojr> after the initial checkout?
[22:18] <blackboxsw_> falcojr: the order of commands in the PR description...
[22:18] <blackboxsw_> quilt push -f
[22:18] <blackboxsw_> (refresh-fix) git checkout f247dd2^ cloudinit/config/cc_ubuntu_advantage.py
[22:18] <blackboxsw_> (refresh-fix) git checkout f247dd2^ cloudinit/config/tests/test_ubuntu_advantage.py
[22:18] <falcojr> so just the subp one?
[22:18] <blackboxsw_> then patch -p1 < ua-subp.patch
[22:19] <falcojr> gotcha, I'll follow the PR :)
[22:19] <blackboxsw_> falcojr: only diff in that description is the following:
[22:19] <blackboxsw_> you aren't working in ubuntu/daily/xenial but ubuntu/xenial
[22:20] <falcojr> right
[22:20] <blackboxsw_> and your starting with new-upstream-snapshot 371b392ced518e45be51089b6a67b362957b1db instead of new-upstream-snapshot --skip-release  # to update debian/changelog and bump daily version
[22:20] <blackboxsw_> the rest I think applies
[22:20] <blackboxsw_> or at least I'm going through that now too to confirm
[22:21] <blackboxsw_> at the end we want to quilt push -a in your local ubuntu/xenial and grep my-token in cloudinit/config/cc_ubuntu_advantage.py to make sure we have to old config docs
[22:21] <blackboxsw_> then quilt pop -a to make sure all patches are popped off the stack
[22:22] <blackboxsw_> falcojr: I pushed me ubuntu/xenial branch (but haven't created a PR)
[22:23] <blackboxsw_> so, appropriate for diffs
[22:23] <blackboxsw_> against yours
[22:23]  * blackboxsw_ is creating a PR documenting what I did for reference
[22:23] <blackboxsw_> 'cause this is a pain
[22:24] <blackboxsw_> ... not your fault, our fault for carrying around a quilt patch like this
[22:29] <falcojr> blackboxsw_: just pushed mine...I compared ours and the only difference was the names
[22:32] <blackboxsw_> sorry excellent falcojr lucasmoura I added a dummy WIP PR just with that context again https://github.com/canonical/cloud-init/pull/559
[22:33] <blackboxsw_> which we will reject and take falcor's I believe
[22:34] <blackboxsw_> cool, ok I'm trying to build yours and confirm
[22:34] <blackboxsw_> falcojr: if you could do the sameish for bionic then I think we are "good"
[22:36] <falcojr> yep
[22:37] <blackboxsw_> It's also EOD, so no worries if you want to grab that tomorrow. we'll probably have to release 2 mondays from now as it's late ubuntu-archive-wise anyway
[22:37] <falcojr> I'll finish up this branch real quick and then call it day
[22:37] <blackboxsw_> so trying to squeeze the -proposed upload in by this time may be too late to make the cut for "SRU aging" of 7 days
[22:37] <blackboxsw_> thanks sir
[22:39] <blackboxsw_> build-package/sbuild-it worked on your xenial branch I'm running build-and-push to upload it to -proposed queue now
[22:40] <falcojr> great, also I'm not able to push tags upstream, so if we want to tag these somebody else will have to do it
[22:40] <blackboxsw_> build-and-push does that
[22:40] <falcojr> ah, great
[22:41] <blackboxsw_> so it should work... and we need to sort your ability to push tags  in the future
[22:41] <blackboxsw_> ok should be a bot message coming up in #ubuntu-release channel shortly with the xenial upload
[22:42] <blackboxsw_> I'm generating my own local ubuntu/bionic now
[22:43] <falcojr> ahhh, was just gonna say mines different! :)
[22:43] <blackboxsw_> just pushed
[22:44] <blackboxsw_> oops need --force
[22:44] <blackboxsw_> ebbaff6ccf57777f729c386365f6a6969d1b6982 commitish on my branch is what I have
[22:44] <blackboxsw_> diffing yours now
[22:44] <blackboxsw_> +1 falcojr great work
[22:44] <blackboxsw_> ok I think we are good
[22:44] <blackboxsw_> I'll sort details if anything else is missing
[22:45] <blackboxsw_> I have a bit more before EOD
[22:45] <blackboxsw_> thanks again
[22:45] <falcojr> wahoo! Cool, thanks
[23:00] <blackboxsw_> ok uploads all queued in Unapproved state for xenial bionic and focal. just waiting on Ubuntu SRU vanguard to approve so we can begin testing