| AnhVoMSFT | Hi folks, under what circumstance can we run into this error "IsADirectoryError: [Errno 21] Is a directory: '/var/lib/cloud/instance'" | 12:05 |
|---|---|---|
| AnhVoMSFT | full output: https://termbin.com/385u | 12:06 |
| smoser | AnhVoMSFT:i thought something went in very recently that intended to fix that. | 12:23 |
| smoser | but fwiw, you have errors before then | 12:23 |
| smoser | AnhVoMSFT: 0755cff078d5931e1d8e151bdcb84afb92bc0f02. | 12:25 |
| AnhVoMSFT | ah thanks - that indeed went in recently | 12:31 |
| AnhVoMSFT | yep, looking at the log I did see an error during writing ssh key | 12:34 |
| smoser | once htat gets created as a directory, then its done, it wont resolve itself. | 12:34 |
| smoser | cloud-init clean will probably fix it, or rm -Rf /var/lib/cloud/instace | 12:35 |
| AnhVoMSFT | how did it actually get created as a directory in the first place? | 12:35 |
| smoser | see the bug for one way | 12:36 |
| smoser | there may be other ways that it gets created, though. so it could be only same symptom | 12:37 |
| AnhVoMSFT | I have about 10 different cloud-init log with this issue, looking through some of them, the pattern seems to be an error during ssh-authkey-fingerprints | 12:39 |
| AnhVoMSFT | KeyError: "getpwnam(): name not found: 'ubuntu'" | 12:39 |
| AnhVoMSFT | not related to the last issue, there's another interesting one | 12:47 |
| AnhVoMSFT | File "/usr/lib/python3/dist-packages/cloudinit/net/dhcp.py", line 162, in maybe_perform_dhcp_discovery | 12:47 |
| AnhVoMSFT | dhclient_path = subp.which('dhclient') | 12:47 |
| AnhVoMSFT | AttributeError: module 'cloudinit.subp' has no attribute 'which' | 12:47 |
| smoser | AnhVoMSFT: it feels like maybe you have a broken install (or maybe an upgrade or something?)... | 12:59 |
| smoser | cloudinit.subp definitely does have which in current | 13:00 |
| smoser | but it did move recently | 13:00 |
| AnhVoMSFT | these are just from random cloud-init.log that were collected as part of our telemetry for failed deployments so I don't have more context into that though. The error just seems odd because like you said, cloudinit.subp definitely has which | 13:39 |
| falcojr | blackboxsw_: my brain is in packaging hell right now | 14:32 |
| falcojr | https://github.com/canonical/cloud-init/commit/66e114a660c53400e389f119781f378311b65108#diff-4df3321932810ca4e004e72e5587ff28R95 | 14:32 |
| falcojr | how are we getting the caplog error if we're including that? | 14:32 |
| falcojr | also, how did the upstream daily changelog get the latest SRU changelog when the non-daily branch doesn't have it yet? | 14:38 |
| blackboxsw_ | minor branch for you falcojr https://github.com/canonical/cloud-init/pull/558 | 15:15 |
| blackboxsw_ | this should resolve the error because caplog is non-std on xenial's version of pytest | 15:15 |
| falcojr | blackboxsw_: how's this different from the other line I linked? | 15:16 |
| blackboxsw_ | reading scrollback | 15:20 |
| blackboxsw_ | ahh falcor, so that package dependency path differs from our daily build recipe debuild -S + sbuild-it path which I believe uses the hard/static file from ubuntu/xenial/debian/control during the build process | 15:32 |
| blackboxsw_ | falcojr: thx for the review. https://github.com/canonical/cloud-init/pull/558 landed for ubuntu/xenial test pkg deps | 16:41 |
| blackboxsw_ | falcojr: looks like master moved again with another commit today. So, we are ready to new-upstream-snapshot 371b392ced518e45be51089b6a67b362957b1dba to each ubuntu/X|B|F branch now | 16:44 |
| blackboxsw_ | if you could re-fresh your existing ubuntu/* PRs then we should be good to go | 16:44 |
| blackboxsw_ | falcojr: lucasmoura I just kicked off a daily build recipe for xenial to see that it 'works' with current ubuntu/xenial -> ubuntu/daily/xenial branches https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily-xenial | 16:54 |
| blackboxsw_ | once we know the builder works, we can land falcor's new-upstream-snapshots | 16:54 |
| blackboxsw_ | knowing that it didn't introduce regressions (which it shouldn't) | 16:54 |
| blackboxsw_ | falcojr: case and point about revno in the daily build recipe not quite working as we hoped with occasional build collisions https://launchpadlibrarian.net/495249390/upload_2635183_log.txt | 17:33 |
| blackboxsw_ | which seem to only happen when I manually click the build now button in launchpad | 17:34 |
| beantaxi | I notice that cloud-init 20.3 has been announced as Released, and that there is a package for Groovy, but I do not see a package for Bionic or anything else. | 18:00 |
| beantaxi | I am a novice at all this ... but is the idea that the cloud-init team is the cloud-init team, and it releases cloud-init code, and then different teams do the packaging for different distros? Even within Canonical? (that would make perfect sense, but that does not mean it's correct) | 18:01 |
| blackboxsw_ | beantaxi: in order for cloud-init to publish into stable Ubuntu releases we have to follow this process for ubuntu https://wiki.ubuntu.com/CloudinitUpdates | 18:01 |
| blackboxsw_ | we call this a Stable Release Update (SRU) | 18:01 |
| blackboxsw_ | falcojr: and I are working today to resolve and queue a -proposed upload of 20.3 into those ubuntu releases | 18:02 |
| blackboxsw_ | beantaxi: expectation is that once we queue uploads, the cloud-init updated packages will be available in the xenial-proposed bionic-proposed apt pockets and will be testable by anyone following these steps https://cloudinit.readthedocs.io/en/latest/topics/debugging.html#stable-release-updates-sru-testing-for-cloud-init | 18:03 |
| blackboxsw_ | we'll send out an email to the list as well about this SRU. | 18:03 |
| blackboxsw_ | generally when we do an upstream release to groovy 20.3 our team tries to queue an SRU into the stable releases within the next week. | 18:04 |
| beantaxi | Gotcha ... so it's still packaged by cloud-iniit, but there are extra steps for making a change to a stable release. That makes perfect sense. | 18:04 |
| blackboxsw_ | that SRU testing minimally takes 7 days, but practically takes average of about 10 | 18:04 |
| blackboxsw_ | yep just so stable releases get more verification (and don't break existing users) | 18:04 |
| beantaxi | Is it generally safe, to grab a source release and install from source, over top of an apt package? (for cloud-init anyway; I imagine there's not a generally correct answer) | 18:04 |
| blackboxsw_ | beantaxi: to watch progress on cloud-init SRUs you can subscribe to the SRU process bug we create | 18:05 |
| blackboxsw_ | https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1893064 | 18:06 |
| ubot5 | Ubuntu bug 1893064 in cloud-init (Ubuntu Focal) "sru cloud-init (20.2-45 to 20.3-0) Xenial, Bionic, and Focal" [Undecided,New] | 18:06 |
| blackboxsw_ | we will update that bug attaching verification logs for every stable release we validate | 18:06 |
| blackboxsw_ | and eventually the Xenial/Bionic/Focal items in that bug will go to Fix Released and comments added to that affect | 18:06 |
| blackboxsw_ | details on this will come out in the SRU email that will be sent to the mailinglist | 18:07 |
| blackboxsw_ | I need to update the IRC topic since we officially "released" 20.3 to groovy 2 days ago | 18:08 |
| beantaxi | Thanks! I just subscribed. Should be interesting to get a feel for the process. | 18:08 |
| === blackboxsw_ changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting Aug 25 16:15 UTC | 20.3 (Aug 25) | https://bugs.launchpad.net/cloud-init/+filebug | ||
| blackboxsw_ | excellent | 18:08 |
| === blackboxsw_ changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting Aug 25 16:15 UTC | 20.3 (Sep 8) | https://bugs.launchpad.net/cloud-init/+filebug | ||
| blackboxsw_ | also updated the topic for next cloud-init community status meeting: "Next status meeting Aug 25 16:15 UTC " | 18:09 |
| === blackboxsw_ changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting Sept 816:15 UTC | 20.3 (Aug 25) | https://bugs.launchpad.net/cloud-init/+filebug | ||
| blackboxsw_ | let's try that again: Next status meeting Sept 816:15 UTC | 18:10 |
| === blackboxsw_ changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting Sept 8 16:15 UTC | 20.3 (Aug 25) | https://bugs.launchpad.net/cloud-init/+filebug | ||
| blackboxsw_ | ... ok I really need to lay off the coffe | 18:10 |
| blackboxsw_ | e | 18:10 |
| falcojr | blackboxsw_: is there something I can do to help figure out why that upload was rejected? | 18:38 |
| powersj | falcojr, where are you uploading to? | 18:54 |
| powersj | if it is a ppa you should get an email with a reason | 18:55 |
| falcojr | blackboxsw_ was doing it, so I'm not sure :) | 18:55 |
| blackboxsw_ | falcojr: checking upload | 18:56 |
| blackboxsw_ | falcojr: you mean groovy? | 18:56 |
| blackboxsw_ | powersj: falcojr oooh you mean the daily recipe build for xenial that I linked | 18:56 |
| falcojr | I was asking about | 18:56 |
| falcojr | "falcojr: case and point about revno in the daily build recipe not quite working as we hoped with occasional build collisions https://launchpadlibrarian.net/495249390/upload_2635183_log.txt" | 18:56 |
| falcojr | yeah | 18:56 |
| blackboxsw_ | right sorry. I'm context switching a bit today and not doing that very well. | 18:57 |
| blackboxsw_ | so the daily build recipes show here https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily-xenial and https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily-bionic | 18:57 |
| blackboxsw_ | you can dive into specific upload log failures clicking int each build | 18:59 |
| blackboxsw_ | if needed. | 18:59 |
| rharper | falcojr: blackboxsw_: no one uploads to daily pocket, for SRU we upload to the proposed pocket of the cloud-init-dev ppa ; | 18:59 |
| rharper | https://launchpad.net/~cloud-init-dev/+archive/ubuntu/proposed | 18:59 |
| blackboxsw_ | right, the daily recipe just has occasional upload failures into the daily PPA because the specific version of the deb file already exists in the daily ppa today (because I clicked the upload button multiple times) | 19:00 |
| rharper | yes | 19:00 |
| rharper | I think we have to wait for another commit, yeah ? | 19:00 |
| rharper | I merged a commit from paride today so tomorrow should get new hash | 19:00 |
| blackboxsw_ | rharper: yes, it should just be a timing thing for our next commit | 19:00 |
| blackboxsw_ | so I don't think we need to resolve this as falcojr is going to put up PRs for updated ubuntu/xenial|bionic|focal today for 20.3-X | 19:01 |
| rharper | yeah | 19:02 |
| blackboxsw_ | once that lands the daily recipe will "just work" because commitish and commit count <revno> will have incremented | 19:02 |
| blackboxsw_ | so ultimately there is nothing to do (and I think we may have already introduced the needed commit) because we are just waiting on | 19:03 |
| blackboxsw_ | [ ]  cloud-init - 20.3-2566-g1f3a225a-0ubuntu1+1489~trunk~ubuntu16.04.1 which is already queued in the recipe build | 19:03 |
| blackboxsw_ | seen at https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily-xenial | 19:03 |
| blackboxsw_ | so falcojr we should be able to proceed with a PR for your ubuntu/xenial with a new-upstream-snapshot 371b392ced518e45be51089b6a67b362957b1dba | 19:03 |
| blackboxsw_ | I think | 19:03 |
| blackboxsw_ | that'd sync the latest released commit we published via ubuntu/devel into ubuntu/xenial, bionic and focal | 19:04 |
| falcojr | cool, sounds good | 19:10 |
| falcojr | ok, blackboxsw_ , maybe this one is right now??? https://github.com/canonical/cloud-init/pull/555 | 19:47 |
| falcojr | I tried the merge and didn't get conflicts this time | 19:47 |
| blackboxsw_ | woot falcojr xenial recipe build completed and uploaded https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily-xenial | 20:42 |
| blackboxsw_ | falcojr: awesome getting this PR review now | 20:42 |
| blackboxsw_ | falcojr: focal merged https://github.com/canonical/cloud-init/pull/553 :) | 21:08 |
| blackboxsw_ | I'll queue an upload for focal -proposed of 20.3 | 21:09 |
| blackboxsw_ | falcojr: lucasmoura over in ubuntu-release channel we get a bot message confirming queued -proposed uploads Unapproved: cloud-init (focal-proposed/main) [20.2-45-g5f7825e2-0ubuntu1~20.04.1 => 20.3-2-g371b392c-0ubuntu1~20.04.1] (core, edubuntu, ubuntu-cloud) | 21:14 |
| blackboxsw_ | hhhhhhhrm | 21:14 |
| blackboxsw_ | right 20.3-2-g371b392c-0ubuntu1~20.04.1] | 21:15 |
| falcojr | what does that mean? | 21:15 |
| blackboxsw_ | ok falcojr that means that I have uploaded (because I have "upload" rights to cloud-init) into the focal-proposed pocket Unapproved queue... which can be found here https://launchpad.net/ubuntu/focal/+queue?queue_state=1&queue_text=cloud-init | 21:17 |
| falcojr | ahh, cool...just didn't know if the "unapproved" was significant | 21:17 |
| blackboxsw_ | our next SRU step (once we actually have bionic and xenial uploads queued is to ping an SRU vangaurd in #ubuntu-release channel to say, please let our uploads into the official focal-proposed xenial-proposed bionic-proposed apt pockets so that public validation can take place | 21:18 |
| blackboxsw_ | once sru vanguards "let the upload into -proposed" it goes from Unapproved -> Accepted I think | 21:18 |
| falcojr | gotcha | 21:18 |
| blackboxsw_ | at which point public vms which have focal-proposed apt config enabled would be able to see, download and verify latest cloud-inbit | 21:19 |
| * blackboxsw_ looks over your bionic and xenial (I'm thinking we still might get into a bit of a pickle with the ubuntu_advantage quilt patches.....) | 21:20 | |
| blackboxsw_ | falcojr: so, something is wrong with ubuntu/bionic and the quilt refresh route because when we `quilt push -a` on your branch, we want it to render an cc_ubuntu_advantage.py that knows how to talk to the old ubuntu-advantage-tools back client, as in accept commands like "ubuntu-advantage disable-livepatch" instead of the new python client which only accepts commands like "ua enable livepatch". | 22:05 |
| falcojr | and quilt refresh doesn't do that after I check out the old version? | 22:07 |
| blackboxsw_ | so, if we ran quilt push -f; quilt refresh, that basically drops the existing debian quilt patch, removing the config parsing and documentation that tells you things like https://cloudinit.readthedocs.io/en/18.5/topics/modules.html#ubuntu-advantage | 22:07 |
| blackboxsw_ | notice we completely reworked the supported config keys for ubuntu_advantage in tip of cloud-init https://cloudinit.readthedocs.io/en/latest/topics/modules.html#ubuntu-advantage | 22:08 |
| blackboxsw_ | falcojr: right, it would only do that if we want the final product of cc_ubuntu_advantage.py to be the value at the commitish git checkout f247dd20ea73f8e153936bee50c57dae9440ecf7^ cloudinit/config/cc_ubuntu_advantage.py | 22:10 |
| falcojr | ohhhhh, didn't realize that about the config keys | 22:11 |
| blackboxsw_ | so I *think* what we need to do in both bionic and xenial cases is to start the quilt push -f (to apply all quilt patches that can apply) fix the end result to be the file fromgit checkout f247dd20ea73f8e153936bee50c57dae9440ecf7^ cloudinit/config/cc_ubuntu_advantage.py | 22:11 |
| blackboxsw_ | we also (painfully) need to apply a minor subp patch (because we migrated subp utilities functions int tip to a new module and that needs to be applied to the result of git checkout f247dd20ea73f8e153936bee50c57dae9440ecf7^ cloudinit/config/cc_ubuntu_advantage.py | 22:12 |
| blackboxsw_ | soooo in short (not really short) to generate your ubuntu/xenial and ubuntu/bionic; you'll need to start over with a new-upstream-snapshot 371b392ced518e45be51089b6a67b362957b1dba | 22:13 |
| blackboxsw_ | and then basically perform the following from this PR description | 22:13 |
| blackboxsw_ | https://github.com/canonical/cloud-init/pull/435 | 22:13 |
| blackboxsw_ | it has an applicable ua-subp.patch that we'll need to use | 22:14 |
| blackboxsw_ | and again, once we actually get the real python ubuntu-advantage-tools deb released to xenial and bionic , this whole mess goes away | 22:14 |
| blackboxsw_ | as we can use tip of master for cc_ubuntu_advantage because the tooling supports it | 22:14 |
| blackboxsw_ | "the tooling" == ubuntu-advantage-tools version 25.0 or later | 22:15 |
| blackboxsw_ | which is our target shortly | 22:15 |
| blackboxsw_ | falcojr: I'll push a PR up for ubuntu/xenial for you to compare against as you are doing it too | 22:15 |
| falcojr | blackboxsw_ is there an additional ubuntu advantage commit I need to cherry pick? | 22:18 |
| falcojr | after the initial checkout? | 22:18 |
| blackboxsw_ | falcojr: the order of commands in the PR description... | 22:18 |
| blackboxsw_ | quilt push -f | 22:18 |
| blackboxsw_ | (refresh-fix) git checkout f247dd2^ cloudinit/config/cc_ubuntu_advantage.py | 22:18 |
| blackboxsw_ | (refresh-fix) git checkout f247dd2^ cloudinit/config/tests/test_ubuntu_advantage.py | 22:18 |
| falcojr | so just the subp one? | 22:18 |
| blackboxsw_ | then patch -p1 < ua-subp.patch | 22:18 |
| falcojr | gotcha, I'll follow the PR :) | 22:19 |
| blackboxsw_ | falcojr: only diff in that description is the following: | 22:19 |
| blackboxsw_ | you aren't working in ubuntu/daily/xenial but ubuntu/xenial | 22:19 |
| falcojr | right | 22:20 |
| blackboxsw_ | and your starting with new-upstream-snapshot 371b392ced518e45be51089b6a67b362957b1db instead of new-upstream-snapshot --skip-release # to update debian/changelog and bump daily version | 22:20 |
| blackboxsw_ | the rest I think applies | 22:20 |
| blackboxsw_ | or at least I'm going through that now too to confirm | 22:20 |
| blackboxsw_ | at the end we want to quilt push -a in your local ubuntu/xenial and grep my-token in cloudinit/config/cc_ubuntu_advantage.py to make sure we have to old config docs | 22:21 |
| blackboxsw_ | then quilt pop -a to make sure all patches are popped off the stack | 22:21 |
| blackboxsw_ | falcojr: I pushed me ubuntu/xenial branch (but haven't created a PR) | 22:22 |
| blackboxsw_ | so, appropriate for diffs | 22:23 |
| blackboxsw_ | against yours | 22:23 |
| * blackboxsw_ is creating a PR documenting what I did for reference | 22:23 | |
| blackboxsw_ | 'cause this is a pain | 22:23 |
| blackboxsw_ | ... not your fault, our fault for carrying around a quilt patch like this | 22:24 |
| falcojr | blackboxsw_: just pushed mine...I compared ours and the only difference was the names | 22:29 |
| blackboxsw_ | sorry excellent falcojr lucasmoura I added a dummy WIP PR just with that context again https://github.com/canonical/cloud-init/pull/559 | 22:32 |
| blackboxsw_ | which we will reject and take falcor's I believe | 22:33 |
| blackboxsw_ | cool, ok I'm trying to build yours and confirm | 22:34 |
| blackboxsw_ | falcojr: if you could do the sameish for bionic then I think we are "good" | 22:34 |
| falcojr | yep | 22:36 |
| blackboxsw_ | It's also EOD, so no worries if you want to grab that tomorrow. we'll probably have to release 2 mondays from now as it's late ubuntu-archive-wise anyway | 22:37 |
| falcojr | I'll finish up this branch real quick and then call it day | 22:37 |
| blackboxsw_ | so trying to squeeze the -proposed upload in by this time may be too late to make the cut for "SRU aging" of 7 days | 22:37 |
| blackboxsw_ | thanks sir | 22:37 |
| blackboxsw_ | build-package/sbuild-it worked on your xenial branch I'm running build-and-push to upload it to -proposed queue now | 22:39 |
| falcojr | great, also I'm not able to push tags upstream, so if we want to tag these somebody else will have to do it | 22:40 |
| blackboxsw_ | build-and-push does that | 22:40 |
| falcojr | ah, great | 22:40 |
| blackboxsw_ | so it should work... and we need to sort your ability to push tags in the future | 22:41 |
| blackboxsw_ | ok should be a bot message coming up in #ubuntu-release channel shortly with the xenial upload | 22:41 |
| blackboxsw_ | I'm generating my own local ubuntu/bionic now | 22:42 |
| falcojr | ahhh, was just gonna say mines different! :) | 22:43 |
| blackboxsw_ | just pushed | 22:43 |
| blackboxsw_ | oops need --force | 22:44 |
| blackboxsw_ | ebbaff6ccf57777f729c386365f6a6969d1b6982 commitish on my branch is what I have | 22:44 |
| blackboxsw_ | diffing yours now | 22:44 |
| blackboxsw_ | +1 falcojr great work | 22:44 |
| blackboxsw_ | ok I think we are good | 22:44 |
| blackboxsw_ | I'll sort details if anything else is missing | 22:44 |
| blackboxsw_ | I have a bit more before EOD | 22:45 |
| blackboxsw_ | thanks again | 22:45 |
| falcojr | wahoo! Cool, thanks | 22:45 |
| blackboxsw_ | ok uploads all queued in Unapproved state for xenial bionic and focal. just waiting on Ubuntu SRU vanguard to approve so we can begin testing | 23:00 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!