[00:43] smoser: rharper : https://code.launchpad.net/~paul-meyer/cloud-init/+git/cloud-init/+merge/344538 is ready for review [01:36] blackboxsw: around ? [01:47] paulmey: just hit 'save comment' [01:47] feel free to ask questions here for th enext few minutes [02:05] thanks smoser [02:06] BTW, the error code from mount is always 32 - mount failure... so string match seems like the only reasonable option [02:08] also BTW, IIRC, ntfs is mounted via fuse now on ubuntu, so there is no ntfs module involved... [02:09] as an example where an assumption and a grep of /proc/filesystems might lead to a wrong conclusion... [02:09] fair point. [02:09] on the other hand, I fully support the "string match is brittle" argument [02:09] exit(32) [02:10] * smoser walks afk for the night [02:14] * paulmey is afk till tomorrow [09:31] i was able to provision a vm with libcloud to azure. i'm using cloud-init in libcloud's ex_customdata (azure customdata), but i think that's only for user-data and not meta-data [09:32] can i put the cloud-init meta-data somehow through the customdata mechanism? [13:14] rharper: Hello, remember the other day I told you about LP: #1531880 and that I was seeing a similar situation ? [13:14] Launchpad bug 1531880 in cloud-init "Failed to start Initial cloud-init job (pre-networking)" [Undecided,New] https://launchpad.net/bugs/1531880 [13:15] I'm able to reproduce it on our images. Seems to be caused by a race-condition between cloud-final.service that gets to run before cloud-init.service [13:16] it can be easily reproduced with : [13:16] cloud-init clean && cloud-init init --local && cloud-init modules --mode=final && cloud-init init [13:48] caribou: interesting; given the systemd unit requirements, how do you get final to run before cloud-init init ? [14:29] caribou: running as you are in that '&&' statement is not valid. [14:30] smoser: yeah, I know this was just to outline the fact that if cloud-final has a chance to run before cloud-init makes the link it will fail [14:30] and to the best of my knowledge, systemd units enforce order of cloud-init-local.service, cloud-init.service, cloud-config.service, cloud-final.service [14:31] but looking at my systemd-analyze plot output, the ordering seems correct [14:31] yet, all I need to do is cloud-init clean & reboot and I'll get the failure [14:32] FYI i'm not working off an Ubuntu Cloud Image but one of our own Scaleway images [14:32] caribou: on that system... /var/lib/cloud/instance is a link or a directoroy ? [14:32] can you let me in to look around ? [14:32] smoser: a directory. I straced the execution & I clearly see cloud-final doing a mkdir /var/lib/cloud/instance [14:33] sure let me pull in your ssh key & I'll PM the IP [14:34] smoser blackboxsw rharper: I'm in need of images with all of my cloud-init fixes by June 30. Can we wrap up my two merge proposals soon? https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/343712 https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/344168 [14:36] the networking reboot thing is harder. [14:37] what is needed to make it not so? [14:37] morning folks. [14:37] morning! [14:38] mgerdts: just work and thought oon how that will i nteract with other "dynamic" networking [14:38] on the other... i think you need updates to your sdc:routes one. the commit message i think is not correct now. [14:38] yeah. you removed the global routes. need to update the commit message. [14:39] If someone is doing dynamic networking that could interfere, setting maintain_networks to false (or leaving it unset, the default) will make it so that traditional behavior is maintained. [14:39] will update the commit message there. [14:47] I'm having troubles understanding the failure at https://jenkins.ubuntu.com/server/job/cloud-init-ci/1140/console [14:48] checking now mgerdts I just queued our CI to run against the branch so we'd have a point of reference [14:49] ahh pylint, should be easy getting you the error (it's buried above) [14:49] I see it nowtests/unittests/test_handler/test_handler_bootcmd.py:150: [W0612(unused-variable), TestSchema.test_duplicates_are_fine_array_array] Unused variable 'e' [14:49] this is in code I've not touched. [14:49] yep https://pastebin.ubuntu.com/p/mmrk3wWK4r/ checking [14:50] mgertds, could be that pylint upstream in ci has updated, you might need to 'git fetch master' 'get rebase -i master' [14:50] since the time that you originally posted this branch we've had a couple pycodestyle/pylint fixes land in master [14:51] I'll give that a whirl. Then git push -f? [14:51] correct mgerdts yeah then git push --force [14:51] after the rebase 'works' [14:51] works without conflicts (which I imagine it should) [14:53] we have removed pylint target in favor of pycodesyle/pyflakes. so master rebase should resolve the issue [14:53] done & pushed [14:53] cheers, I'll kick it again [14:55] mgerdts: I queued another build, but a banner on our jenkins says it's going down for service soon, so we may not get a response on that build until jenkins maintenance is done [14:55] ok, thanks [14:55] build 1141 here https://jenkins.ubuntu.com/server/job/cloud-init-ci/ [14:57] blackboxsw: we should do somethinng on those jobs... regex or somethign to grab errors at the end [14:57] it's a good idea smoser, same with the integration test job, we do dump a dict with results that's really hard to read as a human (and I think misread by jenkins as success too) [14:58] I think I saw an example of failures that jenkins thinks passed (but didn't) [14:58] well jenkins going for reboot now [14:59] but intregration-proposed-ec2-b I think was false success (because salt_minion int tests fail on bionic0 [15:00] stupid humans [15:00] pathetic [15:06] I should change the topic 'stupid pathetic humans' [15:18] rharper I was just looking through your latest feedback and am a little confused by why the code I inherited from a co-worker is doing things like: [15:18] rcfg = dict(...) [15:18] rcfg.update({...}) [15:19] Instead of using update when there's just one or two items to update, why not use [15:19] rcfg['network'] = route['dst'] [15:20] mgerdts: heh, so the general path is to copy keys out of the sdc: values into a config dict and the map values where the keys didn't match but are equivalent [15:20] same goes for subnet just below [15:22] I think that both accomplish the same thing, but one uses the dict.update method. Is there a reason that dict.update is preferred over just doing an assignment? [15:22] not for a single value no [15:24] I guess dict.update is used quite a bit for single values other places too. When in Rome... [15:26] mgerdts: heh; I'm not fussed either way; the comment was more focused on the valid_keys, include the values from sdc:routes that have keys that we want to pickup directly, and then update the dict for any key/value pairs that need a mapping instead [15:27] Yeah, I understand that. I started asking trying to head off the next review comment that says ".update() is pointless now" [15:27] But I see there's a lack of consistency in extant code. [15:27] https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/344168#diff-line-62 [15:27] pgpws[proto] and subnet each have one key modified, but differently. [15:50] robjo: https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/335406 [15:51] is that still relevant ? have we got what we needed for iproute2 support [15:59] smoser: was addressed by the other commit proposal deleted [15:59] thanks. [16:00] robjo: you have some others too... sorry to pester [16:00] https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/333904 [16:00] https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/334992 [16:01] I think 333904 is still valid [16:01] 334992 deleted [16:04] * powersj adds cosmic integration tests [16:12] powersj: cloud-init-ci job disappeared post reboot [16:12] https://jenkins.ubuntu.com/server/job/cloud-init-ci/1047/ etc missing now [16:13] LOL [16:13] same with git-ubuntu [16:13] shhhhooootttt [16:13] I see cloud-init-ci-lander and cloud-init-ci-trigger too, hrm [16:13] pipeline based jobs [16:14] yeah weren't in the redeploy logic/source? [16:15] jobs are deployed by us, why it got deleted is beyond me [16:16] I am deploying with ignored cache right now [16:17] blackboxsw: https://paste.ubuntu.com/p/spzBwbNy8w/ [16:17] barfed on pipeline jobs [16:17] others worked fine [16:18] ah shoot I can't even make it by han [16:18] hand [16:18] rt time [16:19] ahh man :/ [16:21] blackboxsw: I cc'ed you on it [16:27] powersj: yeah, I wonder if it's a missed plugin in the juju spec [16:28] possibly, however, he noticed we were using the pipeline plugin so he made sure the master wouldn't get turned off [16:28] like after the last #is update we didn't sync the new jenkins plugin in the spec [16:28] * blackboxsw tries to find that spec again [16:51] robjo: can you rebase 904 on master and push --force ? [17:11] smoser: I took my best guess at parsing 'just work and thought oon how that will i nteract with other "dynamic" networking' and replying. [17:12] I'm failing to see why preserving legacy behavior as the default and opting in is a problem. Then again, I'm not sure that you've acknowledged that this is the way that it works and it is a problem. [17:14] smoser: will do after my next call which starts in 10 minutes, got to stuff my face first [18:18] robjo: did you find out why PATH was not set in your environment ? [18:18] or in your users's enviroment at least ? [18:20] What I have learned is that on Debian/Ubuntu PID 1 is actually a script and and there is some re-direction happening and in that script the path gets set, we run systemd directly and thus no PATH [18:20] PID1 is most definitely not a script in ubuntu [18:21] Also people chimed in and stated that blkid should not run that early in the boot process [18:21] initramfs runs without /sbin/init == systemd, but pid1 there does exec /sbin/init (which is systemd) [18:21] not verified that's what I got as a comment in the bug report [18:21] and initramfs does not run everywhere, so thats not what is fixing it. [18:22] there's also supposedly etc/systemd.conf where the path maybe set [18:22] if you had actual evidence that blkid should not be run that early, then i'm very interested in it. [18:22] as i have been concerned that we might be missing devices that "show up" later. [18:23] I'll send you the suggestion one of our filesystem guys made via e-mail [18:23] but my experience in now almost 9 years of doing this is that i've never seen it. [18:24] if you can give me a reproduce that shows running blkid once that early in teh boot, and then running it later changed ds-identify behavior (outside of you attaching a disk late by design) then i'm willing to fix. [18:25] i'm also willing to fix PATH, even though i think that is really not the right fix, as cloud-init will not be the only thing that has to then decide on what is a sane path... better to do that once in the distro. [18:32] you should have an e-mail [18:32] the thing is it appears that things work in the AWS images, but I have had a customer report and then one in openSUSE that blkid is not being found [18:44] smoser: https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/333904 rebased and pushed [19:22] powersj: smoser: Did we reach a conclusion about how to enable SSHing to the instance from within it? [19:23] I've had a hectic couple of days, so I'm not sure if I'm just forgetting what we decided on. :p [19:27] Odd_Bloke: did smoser propose some option? Like pushing or writing the key on the instance and use it from there? [19:29] Yeah, that sounds familiar. [19:29] Shall I take a crack at that? [19:29] yeah. thats what i had suggested. [19:29] the only concern i have is that we would want to make sure we do not ever copy a user's private key. [19:29] only ones that we generated. [19:30] i guess maybe just raise an exception somewhere if the key didn't get generated with 'ci-key' or something... i'm not sure we're generating them like that now, but that way we don't end up exposing oddbloke@demigogue's private key to the world. [19:56] blackboxsw: what's that jq one-liner for dumping user-data out of the instance json ? [20:04] ahh got it one sec [20:04] hrm, need to somehow escape the dash ? [20:06] robjo: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/345786 [20:06] rharper: cat /run/cloud-init/instance-data.json | jq -r '.ds."user-data"' [20:07] ah, that's it [20:07] cat /run/cloud-init/instance-data.json | jq -r '.ds."user-data"' | base64 -d [20:07] the "user-data" [20:07] blackboxsw: thanks! [20:07] yeah I thought I had reference in ubuntu-sru, but turns out I didn't, had to back into it [20:07] np [20:07] hrm [20:07] rharper: or just "cloud-init query user-data" [20:07] ohh wait too soon [20:08] hehe [20:08] hrm, so I've got a collect logs tar [20:08] I ran that with the decode and I get garbage ... [20:08] what might be in the user-data raw for a maas deploy ? [20:08] # cat instance-data.json | jq -r '.ds."user-data"' | base64 -d [20:08] %^�(�yj�Ӻ� [20:08] looks like a qbert curse word [20:09] hmm trying to think there thought I had a pastebin of maas cloud-init user-data [20:09] as passed through curtin [20:10] ah [20:10] gziped [20:10] we do't tag that do we [20:10] cat instance-data.json | jq -r '.ds."user-data"' | base64 --decode | zcat [20:10] \o/ [20:11] nice, wow [20:11] we should probably tag that [20:11] #cloud-config [20:11] apt_mirror: "" [20:11] [20:11] that seems to make cloud-init apt unhappy [20:12] rharper: we only tag "base64-encoded-keys" in cloud-init instance-data.json [20:12] we did tag it [20:12] just not that it's gzipped [20:12] rharper: was ds/user-data in there [20:12] ok [20:12] https://paste.ubuntu.com/p/VP92hb9kvG/ [20:12] rharper: that's my bug against maas [20:12] oh ? [20:12] interesting [20:13] shouldn't we handle it more gracefully though ? [20:13] getting the # [20:13] well not my bug, but commented on a potential solution from maas https://bugs.launchpad.net/maas/+bug/1735950 [20:13] Ubuntu bug 1735950 in MAAS 2.3 "ValueError: Old and New apt format defined with unequal values True vs False @ apt_preserve_sources_list" [Low,Triaged] [20:14] rharper: I don't know if we can behave better [20:14] maas passes conflicting settings using the old key versus new key [20:14] I guess we could avoid a traceback and say, "hey buddy, you passed me a False and True for the same arguments, I'm trusting the new key" [20:14] hrm, in the user-data there's nothing apt_ there [20:14] instead of the deprecated key [20:15] except the apt_mirror: "" value [20:15] but that is the stack trace [20:15] rharper: right maas curtin->cloud-init config always passes both I think. [20:15] I'm asying in the cloudin config passed, there is *only* the key apt_mirror: "" [20:16] rharper: line 49 and 52 in https://pastebin.ubuntu.com/26355967/ [20:16] no other apt key at all [20:16] yes, I understand that, I'm seeing the same stacktrace but with no apt: key at all [20:16] the user-data only contains apt_mirror: "" [20:17] hrm. what I thought was that maas sets up apt_preserve user-data config regardless of user-data provided to maas. maybe I'm misreading [20:19] no, you're right, it does [20:19] but for whatever reason, the user-data from this maas deployed instance, does not contain that; now it has a ton of extra juju stuff too [20:22] hrm. [20:22] what maas version? [20:29] blackboxsw: not sure, I don't think it's a maas change; I don't really care about maas here, it's the apperance that user-data with apt_mirror: "" throws a stack ; maybe that's not what's triggering it and it's the bug you pointed to, but that would mean the instance json doesn't have the accurate user-data right ? [20:30] blackboxsw: secondary question, in our collect-logs, why don't we capture /var/lib/cloud ? [20:38] rharper: probably a good thing to catch, it just wasn't in the original suggestion/procedure for filing bugs. I thought we pulled /run/cloud-init which has a couple symlinks to result|status.json, but it'd probably be good to see what's in /var/lib/cloud/(instance|seed) too [20:39] right, the data dir [20:39] including the obj.pkl [20:42] smoser: re LANG=C when doing mount... [20:43] would it be acceptable to pull the subp update_env parameter up into the mount_cb def? [20:44] or is there a better way of doing that? [20:55] finally approved https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/343127 [20:57] landing it [21:13] smoser not around? [21:18] paulmey: should be back in a bit. though he may be eod [21:38] blackboxsw just did a resync to master and pushed it again. You can kick the ci tests again to resolve the failure. [21:38] Or maybe I can resubmit, but don't know how. [21:38] mgerdts: what is your merge request? I can check ci for you [21:39] https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/343712 [21:40] oh, I guess the message I received has a "click here to rebuild" link [21:41] blackboxsw: ok, thanks. I'll update the MP and hope it's the right thing... [21:42] running here: https://jenkins.ubuntu.com/server/job/cloud-init-ci/5/console [21:43] oops I had kicked again too , gonna get lots of ci votes [21:59] smoser: https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/333904 is ready for another look [21:59] blackboxsw: is still on the hook for noLnxDistro and emptyStageOK [22:00] and with that I have reached EOD for me, catch you tomorrow [22:00] <-- hooked. (cheers) [22:27] Hello. Stupid question. [22:27] Documentation says to use write_files like this [22:28] - encoding: gzip content: !!binary | H4sIAGUfoFQC/zMxAgCIsCQyAgAAAA== [22:28] how do I create the H4s... part [22:37] throwaway5234: typically you'll gzip and then base64 encode: cat /my/binary | gzip | base64 [22:39] % echo H4sIAGUfoFQC/zMxAgCIsCQyAgAAAA== | base64 --decode | zcat [22:39] 42 [22:39] I'm mainly trying to compress a config file that's too long. [22:40] you can use a URL for cloud config [22:40] so if I did your approach would I declare it gzip or gzip+b64 [22:40] if that's what you mead [22:40] s/mead/need [22:44] hrm, maybe the example needs updating, the gzip+b64 seems correct; I wonder what happens if you just say gzip though [22:45] yaml needs to have binary data b64 encoded anyhow , but I think best to use both, write_files will b64 decode then gunzip the payload before writing [22:46] I appreciate you help! I was looking online for a couple hours and everyone glosses over that part. [23:41] I'm still having trouble with it. Don't know what to say. [23:41] https://pastebin.com/kMXE17tk