| Carbrich_R | HI, I created https://bugs.launchpad.net/cloud-init/+bug/1892902 yesterday, is that the right place for this one? And if someone has an idea or time to help, that would be appreciated. | 06:42 |
|---|---|---|
| ubot5 | Ubuntu bug 1892902 in cloud-init "Cloud-init received SIGTERM and is thereby unable to install packages or running runcmd completely" [Undecided,New] | 06:42 |
| === Carbrich_r_ is now known as Carbrich_r | ||
| Carbric__ | stats p | 10:00 |
| falcojr | blackboxsw_: looks like next steps of the SRU requires a launchpad bug. Not sure how to fill that. Do we just copy the original template and fill it in later as we verify things? | 14:18 |
| rharper | falcojr: yeah; | 14:30 |
| rharper | falcojr: file new bug; paste in the template; and then update the description as you go; the first thing usually is the changelog generated from new-upstream-snapshot for the source package you SRU ; | 14:31 |
| falcojr | thanks! | 14:36 |
| rharper | falcojr: I almost always look at the previous SRU bug for reference | 14:36 |
| falcojr | I can't find anything with launchpad search :D | 14:38 |
| falcojr | ooooh, it's a different project, nm | 14:42 |
| blackboxsw_ | sorry missed that initial ping falcojr . and I think you gen'd the right bug template as you showed me https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1893064 | 16:24 |
| ubot5 | Ubuntu bug 1893064 in cloud-init (Ubuntu Focal) "sru cloud-init (20.2-45 to 20.3-0) Xenial, Bionic, and Focal" [Undecided,New] | 16:24 |
| blackboxsw_ | so basically now we just fill in the ubuntu-specific functional changes that affect this SRU in the section * <TODO: Create list with LP: # included> | 16:26 |
| blackboxsw_ | so grab those bug linked items from the Changelog file that got created due to the new-upstream-snapshot run on your ubuntu/xenial branch | 16:26 |
| blackboxsw_ | and manually redact any that list down to anything that actually affects ubuntu systems (so redhat or opensuse or freebsd stuff shouldn't be listed in the top section of the ubuntu process bug. | 16:29 |
| falcojr | sounds good, my 3 PRs are up too | 16:30 |
| blackboxsw_ | excellent reviewing now | 16:32 |
| blackboxsw_ | and falcojr here's more context for both of us on ubuntu/xenial branch creation https://github.com/canonical/cloud-init/pull/435 | 16:36 |
| beantaxi | Potentially dumb question ... on my Ubuntu/systemd VM, if the entry point to cloud-config.service and cloud-init.service is /usr/bin/cloud-init, then where is /usr/bin/cloud-init from in the distro? I can't seem to grep a matching source file. | 16:41 |
| beantaxi | I do see in setup.py that the entry point is cloudinit/cmd:main/main.py (and there's another one for cloud-id as well). But I don't see anywhere the package itself is being made -- no target in Makefile seem to match -- and I can't find the little cloud-init script either. My Python is weak so this could all be obvious. Thanks! | 16:44 |
| powersj | beantaxi, the packages directory is what is used to generate the deb and rpm packages | 16:44 |
| minimal | Hi folks, minor issue I've noticed with the 20.3 release, during boot I see on the console a "hi, now its" message which is due to a printf in cloudinit/util.py. Seems this is a left over from some testing/debugging | 16:44 |
| powersj | fix for ^ https://github.com/canonical/cloud-init/pull/556 | 16:49 |
| blackboxsw_ | haha | 16:50 |
| blackboxsw_ | thanks minimal and powersj ! | 16:50 |
| beantaxi | powersj: Thanks ... not seeing anything in there ... perhaps bits of the entry point stuff do not live in the canonical/cloud-init github repo? | 16:50 |
| powersj | beantaxi, if I recall correctly that file itself is not in the source, it is generated as part of the build | 16:51 |
| powersj | beantaxi, is there something you are trying to change with that file? | 16:51 |
| beantaxi | powersj: For my own edification, I'm trying to figure out what happens on startup. I'm ok with the file not being in the source; it means I can stop stomping around wondering why I'm so dumb and can't find it | 16:52 |
| blackboxsw_ | powersj: approved. waiting on ci | 16:52 |
| minimal | powersj: Thanks, could have raised a PR myself, just hadn't taken the time to dig through git history to see how/when that line got in the code | 16:53 |
| beantaxi | powersj: My specific underlying issue, is I'm writing a part handler and not getting what I expect (I'm getting None for filename, and can't figure out why)). So I figured I'd look into how exactly cloud-init calls part handlers. And I think ultimately I've found that in cmd/main.py; there was just a missing piece in getting there, which I think you've now provided (ie it ain't there) | 16:55 |
| meena | anyone had any chance to help out Carbrich_r ?with https://bugs.launchpad.net/cloud-init/+bug/1892902 | 17:07 |
| ubot5 | Ubuntu bug 1892902 in cloud-init "Cloud-init received SIGTERM and is thereby unable to install packages or running runcmd completely" [Undecided,Confirmed] | 17:07 |
| meena | oh, it's confirmed already | 17:08 |
| blackboxsw_ | falcojr: per your branches I gave you the wrong guidance on 'fixing' the quilt patches for these branches. In checking whether our daily build recipe would continue to work. we need to be able to git clone master; git merge your/ubuntu/xenial; git merge upstream/ubuntu/daily/xenial; quilt push -a # to apply all quilt patches without errors...... but we are getting quilt patch apply issues when merging daily | 17:16 |
| blackboxsw_ | so I need to investigate a little further taking into account how our ubuntu/daily/<release> needs to merge into ubuntu/<release> to see if we might need to refresh patches in the ubuntu/daily/xenial branch too because we had to "quilt refresh" in ubuntu/xenial | 17:17 |
| blackboxsw_ | this is all related to the behavior of ubuntu_advantage config module in Xenial/Bionic is patched specifically to "work" with the older/broken client.... once we have an ubuntu-advantage-tools pkg that is officially published to X, B and F ubuntu releases, then these patches should disappear. | 17:18 |
| blackboxsw_ | falcojr: also I'm wondering if we want to generate a new-upstream-snapshot against groovy now that https://github.com/canonical/cloud-init/pull/556 landed because it's kindof ugly to pull back that debug print statement in our SRU too | 17:21 |
| blackboxsw_ | so we cloud do a followup ubuntu/devel release as we did yesterday (easier this time) | 17:21 |
| blackboxsw_ | and then SRU what we have in tip of master (as we don't specifically have to SRU 20.3 we can SRU 20.3-2 | 17:21 |
| blackboxsw_ | then we don't have the blemish of debug statements dumped out in /var/log/cloud-init-output.log for our SRU content | 17:22 |
| blackboxsw_ | we can discuss and see if we want to do that today or not | 17:25 |
| * blackboxsw_ jumps into a couple back-to-back mtgs. | 17:25 | |
| Carbrich_r | @meena: thanks for memorising. | 17:59 |
| smoser | beantaxi: fyi, i dont know if you saw. but bug 1893064 is SRU cloud-init 20.3 | 18:09 |
| ubot5 | bug 1893064 in cloud-init (Ubuntu Focal) "sru cloud-init (20.2-45 to 20.3-0) Xenial, Bionic, and Focal" [Undecided,New] https://launchpad.net/bugs/1893064 | 18:09 |
| smoser | so that will get you an update you were after. | 18:09 |
| beantaxi | smoser: Thanks! That's thoughtful of you to give me the heads up. | 18:19 |
| johnsonshi | Hey upstream folks, is there any guidance on when LOG.warning or LOG.error should be used? | 19:22 |
| johnsonshi | For instance if cloud-init encounters an error when applying a config. Should that be treated as a LOG.warning or a LOG.error. | 19:25 |
| johnsonshi | blackboxsw_ falcojr: I've added necessary UT mocks in my PR (#549) in response to missing mocks in #539 (which caused the UTs to succeed and fail depending on the machine they are run on). Please see https://github.com/canonical/cloud-init/pull/549#discussion_r477546613 | 19:46 |
| blackboxsw_ | hi johnsonshi sorry will get to that review. thanks. | 20:35 |
| blackboxsw_ | falcojr: reviewed your upload PR to groovy with that debug message fix | 20:35 |
| blackboxsw_ | merged | 20:35 |
| blackboxsw_ | and upload landed. | 20:36 |
| blackboxsw_ | community-notice: new upload to ubuntu groovy with tip of master [ubuntu/groovy-proposed] cloud-init 20.3-2-g371b392c-0ubuntu1 (Accepted) | 20:37 |
| blackboxsw_ | thx minimal for the notice on that again & powers for the fix | 20:37 |
| blackboxsw_ | so falcojr we need to try to regenerate ubuntu/xenial|bionic|focal branches again; probably a git checkout ubuntu/xeniall git reset --hard upstream/ubuntu/xenial; new-upstream-snapshot; | 20:38 |
| beantaxi | smoser: I'm sure it's my limited Python skills, but I can't run make-mime.py locally. I cannot seem to get the interpreter to be happy with both `from cloudinit import log` and `from . import addLogHandlerCL`. Either it likes one or the other (or neither). If there is a better way to run on OSX, I'm happy to switch. | 20:39 |
| blackboxsw_ | one thing I think we might need to note is that (instead of the pulling in individual commits to fix the quilt patches) I *think* we might just need to run quilt push -f; quilt refresh (like the message tells us) in order to fix the quilt patches in xenial | 20:39 |
| blackboxsw_ | falcojr: ^ | 20:39 |
| blackboxsw_ | johnsonshi: I think LOG.warnings tend to get ignored. cloud-init historically tried it's best to bring up a server regardless of cloud-config shortcomings, typos or misconfigurations. But I think our general opinion on that has changed. | 20:41 |
| blackboxsw_ | If the user desired a certain config and cloud-init couldn't apply it, it might be best to LOG.error rather than expect the user to grep inadvertently discover a misconfiguration later when they try to use the desired service and have to manually fix it. | 20:41 |
| blackboxsw_ | johnsonshi: I'd be more inclined to LOG.warning if there are alternative solutions that could remedy the issue or something that is really trivial. That's pretty vague I'm sorry. We've talked in the past about trying to scrub our log levels and even introduce a trace level log due to how noisy cloud-init DEBUG logs are. We have also discussed the possibility of cloud-init status --long surfacing log warnings | 20:44 |
| blackboxsw_ | too instead of just errors. | 20:44 |
| blackboxsw_ | Neither of which have gotten enough priority in the past few years to make it a feature in upstream cloud-init :/ | 20:44 |
| blackboxsw_ | to be honest thogh, the log level scrubbing would be a fairly big undertaking and touch most of cloud-init. | 20:45 |
| blackboxsw_ | so, LOG.error is good, it stops cloud-init and alerts the user to a misconfiguration that they probably should corret before they launch and use the VM for extended periods of time. And I can attribute the quote "users don't look at logs" -- smoser (every couple weeks) | 20:46 |
| smoser | beantaxi: it should just run as './tools/make-mime.py' in a checked out cloud-init dir. (prior to it moving to cloudinit proper) | 20:47 |
| beantaxi | smoser: Yes, that worked fine. But the source is different now, & I actually wanted to confirm that an issue I had with tools/make_mime.py -- that it adds entries for a non-existent files, eg when I fat-finger a path name -- I wanted to confirm that was still an issue in the devel version. | 20:48 |
| beantaxi | I'm not 100% I correctly observed the issue, and there didn't seem much point in confirming with the older version since it's going away anyway | 20:50 |
| smoser | oh. from master, runt it as | 20:50 |
| smoser | $ python3 -m cloudinit.cmd.main devel make-mime | 20:50 |
| smoser | either installed for from . | 20:50 |
| smoser | (. == top level dir of cloud-init) | 20:50 |
| smoser | or | 20:51 |
| smoser | $ ./tools/tox-venv py3 python -m cloudinit.cmd.main devel make-mime | 20:51 |
| beantaxi | Thanks! I'll give that a try | 20:54 |
| johnsonshi | Thanks for the clarification blackboxsw_ :) | 21:08 |
| beantaxi | smoser: Ahhhh I get it, make-mime.py is not designed to be standalone any more. Instead, make-mime is a new command under `cloud-init devel` which simply happens to be defined in make-mime.py. Thanks for bearing with me! Incidentally it looks like it fails on missing files as expected, which is good :) | 21:13 |
| johnsonshi | blackboxsw_: Does LOG.error stop execution? Afaik LOG.critical stops cloud-init execution. | 21:24 |
| rharper | beantaxi: yeah, I put up a PR a few weeks back to make it available since it wasn't installed with the source; I believe you can use --force if you are really sure (it checks path and known supported mime-types, force will complain but generate things anyhow) | 21:24 |
| blackboxsw_ | johnsonshi: soo, what scenarios could lead to IMDS metadata being invalid/corrupted? This happens when IMDS metadata is invalid/corrupted (such as when it is missing network or interface metadata). This causes the rest of provisioning to fail. | 21:47 |
| johnsonshi | blackboxsw_: There have been cases where the Azure IMDS service fails to return network/interface metadata in the IMDS response. | 21:48 |
| blackboxsw_ | what I mean is, in what scenario would the IMDS network config by surfaced as corrupt/invalid? IS there something else we can look at as far as the platform telling us IMDS is "ready" for comsumers or "waiting" | 21:48 |
| blackboxsw_ | like a semaphore or long poll file cloud-init could "block" on until IMDS is ready ? | 21:49 |
| johnsonshi | This happens due to intermittent platform issues :/ There is no reliable "poll" mechanism for this specific case, as the root cause of IMDS failing to return networking metadata is only due to platform component issues. | 21:51 |
| blackboxsw_ | +1 ok | 21:53 |
| falcojr | blackboxsw_: "I *think* we might just need to run quilt push -f; quilt refresh (like the message tells us) in order to fix the quilt patches in xenial" | 22:04 |
| falcojr | that's what I did the first time around | 22:04 |
| blackboxsw_ | falcojr: ok, I think I did something bogus having forgotten the release steps :/. I'm trying to walk through ubuntu/xenial now given that we've uploaded 20.3-2 to groovy now to see if I can ensure I can git clone master; git merge ubuntu/xenial; git merge ubuntu/daily/xenial; quilt push -a | 22:05 |
| blackboxsw_ | which is what our daily build (xenial) recipe does @ https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily-xenial | 22:06 |
| blackboxsw_ | so if we can't merge ubuntu/daily/xenial into the new ubuntu/xenial branch, then daily builds fail | 22:06 |
| blackboxsw_ | last SRU we added the fix-daily-branch util to uss-tableflip repo to assist in these manners. and I'm re-reading to remind myself of the context there on what the intent is for fixing daily/xenial and the scenarios in which we expected to have to run it | 22:07 |
| blackboxsw_ | I expected we wouldn't need to run fix-daily-branch unless we ran cherry-pick on a given series. This issue might have come about as well for this SRU due to our modification of postinst scripts in xenial/bionic etc. | 22:09 |
| blackboxsw_ | falcojr: to support the grub changes | 22:09 |
| falcojr | hmmm, interesting | 22:09 |
| blackboxsw_ | so that *may* be why we are failing to merge ubuntu/daily/xenial.... but I'm just guessing at the moment | 22:09 |
| blackboxsw_ | want to confidently triage why we can't merge u/daily/x into u/x | 22:10 |
| blackboxsw_ | as it does seem to be that we have refreshed ubuntu_advantage quilt patches at the moment | 22:10 |
| * blackboxsw_ stops the <<hand wavvy>> guessing and gets to the bottom of it | 22:10 | |
| blackboxsw_ | falcojr: so at least part of the problem with merging ubuntu/daily/xenial into your ubuntu/xenial PR is that new grub debian/changelog entry https://paste.ubuntu.com/p/tgHs8YJZJF/ | 22:16 |
| blackboxsw_ | falcojr: which is also what's been killing our daily builds for xenial anyway for the last few days https://code.launchpad.net/~cloud-init-dev/+recipe/cloud-init-daily-xenial | 22:17 |
| blackboxsw_ | falcojr: https://launchpadlibrarian.net/495123633/buildlog.txt.gz | 22:18 |
| blackboxsw_ | shows the same traceback on the merge I'm seeing locally | 22:18 |
| blackboxsw_ | so I think we should try to fix daily builds first by fixing ubuntu/daily/(xenial|bionic|focal) branches first | 22:19 |
| blackboxsw_ | we control those branches (and there are no consumers of them beside our daily recipe tooling) | 22:19 |
| blackboxsw_ | so we can overwrite --force push those branches anytime we wish. though I'd prefer if our tooling took care of this | 22:20 |
| falcojr | hmmm, ok. Why would an updated changelog be causing a conflict? | 22:20 |
| falcojr | also, reaching EOD here for me, but I can dig in the morning as well | 22:21 |
| blackboxsw_ | falcojr: no worries I'll put something up, I think we might need to just refresh the ubuntu/daily/X|B|F branches due to that debian/changelog for the grub postinst changes. | 22:44 |
| blackboxsw_ | the reason the changelog conflict is because ubuntu/daily/xenial|bionic|focal branches do re-write the debian/changelog as it drops cherry-picked patches | 22:45 |
| blackboxsw_ | so we get a conflict there as your new-upstream-snapshot run also updates debian/changelog | 22:45 |
| blackboxsw_ | and that doesn't line up with the d/changelog in the ubuntu/daily/(x|b|f) branch | 22:45 |
| blackboxsw_ | have a good one either way falcojr we can sort this first thing tomorrow | 22:48 |
| falcojr | Thanks blackboxsw_, you too! | 22:49 |
| blackboxsw_ | johnsonshi: initial review on https://github.com/canonical/cloud-init/pull/549/files# we might be able to get two fixes with one PR (as fallback config needs to blacklist mlx5_core driver too I believe | 23:01 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!