[00:43] <paulmey> smoser:  rharper : https://code.launchpad.net/~paul-meyer/cloud-init/+git/cloud-init/+merge/344538 is ready for review
[01:36] <smoser> blackboxsw: around ?
[01:47] <smoser> paulmey: just hit 'save comment'
[01:47] <smoser> feel free to ask questions here for th enext few minutes
[02:05] <paulmey> thanks smoser
[02:06] <paulmey> BTW, the error code from mount is always 32 - mount failure... so string match seems like the only reasonable option
[02:08] <paulmey> also BTW, IIRC, ntfs is mounted via fuse now on ubuntu, so there is no ntfs module involved...
[02:09] <paulmey> as an example where an assumption and a grep of /proc/filesystems might lead to a wrong conclusion...
[02:09] <smoser> fair point.
[02:09] <paulmey> on the other hand, I fully support the "string match is brittle" argument
[02:09] <smoser> exit(32)
[02:10]  * smoser walks afk for the night
[02:14]  * paulmey is afk till tomorrow
[09:31] <akik> i was able to provision a vm with libcloud to azure. i'm using cloud-init in libcloud's ex_customdata (azure customdata), but i think that's only for user-data and not meta-data
[09:32] <akik> can i put the cloud-init meta-data somehow through the customdata mechanism?
[13:14] <caribou> rharper: Hello, remember the other day I told you about  LP: #1531880 and that I was seeing a similar situation ?
[13:15] <caribou> I'm able to reproduce it on our images. Seems to be caused by a race-condition between cloud-final.service that gets to run before cloud-init.service
[13:16] <caribou> it can be easily reproduced with :
[13:16] <caribou> cloud-init clean && cloud-init init --local && cloud-init modules --mode=final && cloud-init init
[13:48] <rharper> caribou: interesting; given the systemd unit requirements, how do you get final to run before cloud-init init ?
[14:29] <smoser> caribou: running as you are in that '&&' statement is not valid.
[14:30] <caribou> smoser: yeah, I know this was  just to outline the fact that if cloud-final has a chance to run before cloud-init makes the link it will fail
[14:30] <smoser> and to the best of my knowledge, systemd units enforce order of cloud-init-local.service, cloud-init.service, cloud-config.service, cloud-final.service
[14:31] <caribou> but looking at my systemd-analyze plot output, the ordering seems correct
[14:31] <caribou> yet, all I need to do is cloud-init clean & reboot and I'll get the failure
[14:32] <caribou> FYI i'm not working off an Ubuntu Cloud Image but one of our own Scaleway images
[14:32] <smoser> caribou: on that system... /var/lib/cloud/instance is a link or a directoroy ?
[14:32] <smoser> can you let me in to look around ?
[14:32] <caribou> smoser: a directory. I straced the execution & I clearly see cloud-final doing a mkdir /var/lib/cloud/instance
[14:33] <caribou> sure let me pull in your ssh key & I'll PM the IP
[14:34] <mgerdts> smoser blackboxsw rharper: I'm in need of images with all of my cloud-init fixes by June 30.  Can we wrap up my two merge proposals soon?  https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/343712 https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/344168
[14:36] <smoser> the networking reboot thing is harder.
[14:37] <mgerdts> what is needed to make it not so?
[14:37] <blackboxsw> morning folks.
[14:37] <mgerdts> morning!
[14:38] <smoser> mgerdts: just work and thought oon how that will i nteract with other "dynamic" networking
[14:38] <smoser> on the other... i think you need updates to your sdc:routes one. the commit message i think is not correct now.
[14:38] <smoser>  yeah. you removed the global routes. need to update the commit message.
[14:39] <mgerdts> If someone is doing dynamic networking that could interfere, setting maintain_networks to false (or leaving it unset, the default) will make it so that traditional behavior is maintained.
[14:39] <mgerdts> will update the commit message there.
[14:47] <mgerdts> I'm having troubles understanding the failure at https://jenkins.ubuntu.com/server/job/cloud-init-ci/1140/console
[14:48] <blackboxsw> checking now mgerdts I just queued our CI to run against the branch so we'd have a point of reference
[14:49] <blackboxsw> ahh pylint, should be easy getting you the error (it's buried above)
[14:49] <mgerdts> I see it nowtests/unittests/test_handler/test_handler_bootcmd.py:150: [W0612(unused-variable), TestSchema.test_duplicates_are_fine_array_array] Unused variable 'e'
[14:49] <mgerdts> this is in code I've not touched.
[14:49] <blackboxsw> yep https://pastebin.ubuntu.com/p/mmrk3wWK4r/ checking
[14:50] <blackboxsw> mgertds, could be that pylint upstream in ci has updated, you might need to 'git fetch master'  'get rebase -i master'
[14:50] <blackboxsw> since the time that you originally posted this branch we've had a couple pycodestyle/pylint fixes land in master
[14:51] <mgerdts> I'll give that a whirl.  Then git push -f?
[14:51] <blackboxsw> correct mgerdts yeah then git push --force
[14:51] <blackboxsw> after the rebase 'works'
[14:51] <blackboxsw> works without conflicts (which I imagine it should)
[14:53] <blackboxsw> we have removed pylint target in favor of pycodesyle/pyflakes. so master rebase should resolve the issue
[14:53] <mgerdts> done & pushed
[14:53] <blackboxsw> cheers, I'll kick it again
[14:55] <blackboxsw> mgerdts: I queued another build, but a banner on our jenkins says it's going down for service soon, so we may not get a response on that build until jenkins maintenance is done
[14:55] <mgerdts> ok, thanks
[14:55] <blackboxsw> build 1141 here https://jenkins.ubuntu.com/server/job/cloud-init-ci/
[14:57] <smoser> blackboxsw: we should do somethinng on those jobs... regex or somethign to grab errors at the end
[14:57] <blackboxsw> it's a good idea smoser, same with the integration test job, we do dump a dict with results that's really hard to read as a human (and I think misread by jenkins as success too)
[14:58] <blackboxsw> I think I saw an example of failures that jenkins thinks passed (but didn't)
[14:58] <blackboxsw> well jenkins going for reboot now
[14:59] <blackboxsw> but intregration-proposed-ec2-b  I think was false success (because salt_minion int tests fail on bionic0
[15:00] <smoser> stupid humans
[15:00] <smoser> pathetic
[15:06] <blackboxsw> I should change the topic 'stupid pathetic humans'
[15:18] <mgerdts> rharper I was just looking through your latest feedback and am a little confused by why the code I inherited from a co-worker is doing things like:
[15:18] <mgerdts> rcfg = dict(...)
[15:18] <mgerdts> rcfg.update({...})
[15:19] <mgerdts> Instead of using update when there's just one or two items to update, why not use
[15:19] <mgerdts> rcfg['network'] = route['dst']
[15:20] <rharper> mgerdts: heh, so the general path is to copy keys out of the sdc: values into a config dict and the map values where the keys didn't match but are equivalent
[15:20] <mgerdts> same goes for subnet just below
[15:22] <mgerdts> I think that both accomplish the same thing, but one uses the dict.update method.  Is there a reason that dict.update is preferred over just doing an assignment?
[15:22] <rharper> not for a single value  no
[15:24] <mgerdts> I guess dict.update is used quite a bit for single values other places too.  When in Rome...
[15:26] <rharper> mgerdts: heh;  I'm not fussed either way; the comment was more focused on the valid_keys, include the values from sdc:routes that have keys that we want to pickup directly, and then update the dict for any key/value pairs that need a mapping instead
[15:27] <mgerdts> Yeah, I understand that.  I started asking trying to head off the next review comment that says ".update() is pointless now"
[15:27] <mgerdts> But I see there's a lack of consistency in extant code.
[15:27] <mgerdts> https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/344168#diff-line-62
[15:27] <mgerdts> pgpws[proto] and subnet each have one key modified, but differently.
[15:50] <smoser> robjo: https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/335406
[15:51] <smoser> is that still relevant ? have we got what we needed for iproute2 support
[15:59] <robjo> smoser: was addressed by the other commit proposal deleted
[15:59] <smoser> thanks.
[16:00] <smoser> robjo: you have some others too... sorry to pester
[16:00] <smoser>  https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/333904
[16:00] <smoser>  https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/334992
[16:01] <robjo> I think 333904 is still valid
[16:01] <robjo> 334992 deleted
[16:04]  * powersj adds cosmic integration tests
[16:12] <blackboxsw> powersj: cloud-init-ci job disappeared post reboot
[16:12] <blackboxsw> https://jenkins.ubuntu.com/server/job/cloud-init-ci/1047/  etc missing now
[16:13] <powersj> LOL
[16:13] <powersj> same with git-ubuntu
[16:13] <powersj> shhhhooootttt
[16:13] <blackboxsw> I see cloud-init-ci-lander and cloud-init-ci-trigger too, hrm
[16:13] <powersj> pipeline based jobs
[16:14] <blackboxsw> yeah weren't in the redeploy logic/source?
[16:15] <powersj> jobs are deployed by us, why it got deleted is beyond me
[16:16] <powersj> I am deploying with ignored cache right now
[16:17] <powersj> blackboxsw: https://paste.ubuntu.com/p/spzBwbNy8w/
[16:17] <powersj> barfed on pipeline jobs
[16:17] <powersj> others worked fine
[16:18] <powersj> ah shoot I can't even make it by han
[16:18] <powersj> hand
[16:18] <powersj> rt time
[16:19] <blackboxsw> ahh man :/
[16:21] <powersj> blackboxsw: I cc'ed you on it
[16:27] <blackboxsw> powersj: yeah, I wonder if it's a missed plugin in the juju spec
[16:28] <powersj> possibly, however, he noticed we were using the pipeline plugin so he made sure the master wouldn't get turned off
[16:28] <blackboxsw> like after the last #is update we didn't sync the new jenkins plugin in the spec
[16:28]  * blackboxsw tries to find that spec again
[16:51] <smoser> robjo: can you rebase 904 on master and push --force ?
[17:11] <mgerdts> smoser: I took my best guess at parsing 'just work and thought oon how that will i nteract with other "dynamic" networking' and replying.
[17:12] <mgerdts> I'm failing to see why preserving legacy behavior as the default and opting in is a problem.  Then again, I'm not sure that you've acknowledged that this is the way that it works and it is a problem.
[17:14] <robjo> smoser: will do after my next call which starts in 10 minutes, got to stuff my face first
[18:18] <smoser> robjo: did you find out why PATH was not set in your environment ?
[18:18] <smoser> or in your users's enviroment at least ?
[18:20] <robjo> What I have learned is that on Debian/Ubuntu PID 1 is actually a script and and there is some re-direction happening and in that script the path gets set, we run systemd directly and thus no PATH
[18:20] <smoser> PID1 is most definitely not a script in ubuntu
[18:21] <robjo> Also people chimed in and stated that blkid should not run that early in the boot process
[18:21] <smoser> initramfs runs without /sbin/init == systemd, but pid1 there does exec /sbin/init (which is systemd)
[18:21] <robjo> not verified that's what I got as a comment in the bug report
[18:21] <smoser> and initramfs does not run everywhere, so thats not what is fixing it.
[18:22] <robjo> there's also supposedly etc/systemd.conf where the path maybe set
[18:22] <smoser> if you had actual evidence that blkid should not be run that early, then i'm very interested in it.
[18:22] <smoser> as i have been concerned that we might be missing devices that "show up" later.
[18:23] <robjo> I'll send you the suggestion one of our filesystem guys made via e-mail
[18:23] <smoser> but my experience in now almost 9 years of doing this is that i've never seen it.
[18:24] <smoser> if you can give me a reproduce that shows running blkid once that early in teh boot, and then running it later changed ds-identify behavior (outside of you attaching a disk late by design) then i'm willing to fix.
[18:25] <smoser> i'm also willing to fix PATH, even though i think that is really not the right fix, as cloud-init will not be the only thing that has to then decide on what is a sane path... better to do that once in the distro.
[18:32] <robjo> you should have an e-mail
[18:32] <robjo> the thing is it appears that things work in the AWS images, but I have had a customer report and then one in openSUSE that blkid is not being found
[18:44] <robjo> smoser: https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/333904 rebased and pushed
[19:22] <Odd_Bloke> powersj: smoser: Did we reach a conclusion about how to enable SSHing to the instance from within it?
[19:23] <Odd_Bloke> I've had a hectic couple of days, so I'm not sure if I'm just forgetting what we decided on. :p
[19:27] <powersj> Odd_Bloke: did smoser propose some option? Like pushing or writing the key on the instance and use it from there?
[19:29] <Odd_Bloke> Yeah, that sounds familiar.
[19:29] <Odd_Bloke> Shall I take a crack at that?
[19:29] <smoser> yeah. thats what i had suggested.
[19:29] <smoser> the only concern i have is that we would want to make sure we do not ever copy a user's private key.
[19:29] <smoser> only ones that we generated.
[19:30] <smoser> i guess maybe just raise an exception somewhere if the key didn't get generated with 'ci-key' or something... i'm not sure we're generating them like that now, but that way we don't end up exposing oddbloke@demigogue's private key to the world.
[19:56] <rharper> blackboxsw: what's that jq one-liner for dumping user-data out of the instance json ?
[20:04] <blackboxsw> ahh got it one sec
[20:04] <rharper> hrm, need to somehow escape the dash ?
[20:06] <smoser> robjo: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/345786
[20:06] <blackboxsw> rharper: cat /run/cloud-init/instance-data.json | jq -r '.ds."user-data"'
[20:07] <rharper> ah, that's it
[20:07] <blackboxsw> cat /run/cloud-init/instance-data.json | jq -r '.ds."user-data"' | base64 -d
[20:07] <rharper> the "user-data"
[20:07] <rharper> blackboxsw: thanks!
[20:07] <blackboxsw> yeah I thought I had reference in ubuntu-sru, but turns out I didn't, had to back into it
[20:07] <blackboxsw> np
[20:07] <rharper> hrm
[20:07] <blackboxsw> rharper: or just "cloud-init query user-data"
[20:07] <blackboxsw> ohh wait too soon
[20:08] <rharper> hehe
[20:08] <rharper> hrm, so I've got a collect logs tar
[20:08] <rharper> I ran that with the decode and I get garbage ...
[20:08] <rharper> what might be in the user-data raw for a maas deploy ?
[20:08] <rharper> # cat instance-data.json | jq -r '.ds."user-data"' | base64 -d
[20:08] <rharper> %^�(�yj�Ӻ�
[20:08] <blackboxsw> looks like a qbert curse word
[20:09] <blackboxsw> hmm trying to think there thought I had a pastebin of maas cloud-init user-data
[20:09] <blackboxsw> as passed through curtin
[20:10] <rharper> ah
[20:10] <rharper> gziped
[20:10] <rharper> we do't tag that do we
[20:10] <rharper>  cat instance-data.json | jq -r '.ds."user-data"' | base64 --decode | zcat
[20:10] <rharper> \o/
[20:11] <blackboxsw> nice, wow
[20:11] <blackboxsw> we should probably tag that
[20:11] <rharper> #cloud-config
[20:11] <rharper> apt_mirror: ""
[20:11] <rharper>  
[20:11] <rharper> that seems to make cloud-init apt unhappy
[20:12] <blackboxsw> rharper: we only tag "base64-encoded-keys" in cloud-init instance-data.json
[20:12] <rharper> we did tag it
[20:12] <rharper> just not that it's gzipped
[20:12] <blackboxsw> rharper: was ds/user-data  in there
[20:12] <blackboxsw> ok
[20:12] <rharper> https://paste.ubuntu.com/p/VP92hb9kvG/
[20:12] <blackboxsw> rharper: that's my bug against maas
[20:12] <rharper> oh ?
[20:12] <rharper> interesting
[20:13] <rharper> shouldn't we handle it more gracefully though ?
[20:13] <blackboxsw> getting the #
[20:13] <blackboxsw> well not my bug, but commented on a potential solution from maas https://bugs.launchpad.net/maas/+bug/1735950
[20:14] <blackboxsw> rharper: I don't know if we can behave better
[20:14] <blackboxsw> maas passes conflicting settings using the old key versus new key
[20:14] <blackboxsw> I guess we could avoid a traceback and say, "hey buddy, you passed me a False and True for the same arguments, I'm trusting the new key"
[20:14] <rharper> hrm, in the user-data there's nothing apt_ there
[20:14] <blackboxsw> instead of the deprecated key
[20:15] <rharper> except the apt_mirror: "" value
[20:15] <rharper> but that is the stack trace
[20:15] <blackboxsw> rharper: right maas curtin->cloud-init config always passes both I think.
[20:15] <rharper> I'm asying in the cloudin config passed, there is *only* the key apt_mirror: ""
[20:16] <blackboxsw> rharper: line 49 and 52 in https://pastebin.ubuntu.com/26355967/
[20:16] <rharper> no other apt key at all
[20:16] <rharper> yes, I understand that, I'm seeing the same stacktrace but with no apt: key at all
[20:16] <rharper> the user-data only contains apt_mirror: ""
[20:17] <blackboxsw> hrm. what I thought was that maas sets up apt_preserve user-data config regardless of user-data provided to maas. maybe I'm misreading
[20:19] <rharper> no, you're right, it does
[20:19] <rharper> but for whatever reason, the user-data from this maas deployed instance, does not contain that;  now it has a ton of extra juju stuff too
[20:22] <blackboxsw> hrm.
[20:22] <blackboxsw> what maas version?
[20:29] <rharper> blackboxsw: not sure, I don't think it's a maas change; I don't really care about maas here, it's the apperance that user-data with apt_mirror: "" throws a stack ;  maybe that's not what's triggering it and it's the bug you pointed to, but that would mean the instance json doesn't have the accurate user-data right ?
[20:30] <rharper> blackboxsw: secondary question, in our collect-logs, why don't we capture /var/lib/cloud  ?
[20:38] <blackboxsw> rharper: probably a good thing to catch, it just wasn't in the original suggestion/procedure for filing bugs. I thought we pulled /run/cloud-init which has a couple symlinks to result|status.json, but it'd probably be good to see what's in /var/lib/cloud/(instance|seed) too
[20:39] <rharper> right, the data dir
[20:39] <rharper> including the obj.pkl
[20:42] <paulmey> smoser: re LANG=C when doing mount...
[20:43] <paulmey> would it be acceptable to pull the subp update_env parameter up into the mount_cb def?
[20:44] <paulmey> or is there a better way of doing that?
[20:55] <blackboxsw> finally approved https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/343127
[20:57] <blackboxsw> landing it
[21:13] <paulmey> smoser not around?
[21:18] <blackboxsw> paulmey: should be back in a bit. though he may be eod
[21:38] <mgerdts> blackboxsw just did a resync to master and pushed it again.  You can kick the ci tests again to resolve the failure.
[21:38] <mgerdts> Or maybe I can resubmit, but don't know how.
[21:38] <powersj> mgerdts: what is your merge request? I can check ci for you
[21:39] <mgerdts> https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/343712
[21:40] <mgerdts> oh, I guess the message I received has a "click here to rebuild" link
[21:41] <paulmey> blackboxsw: ok, thanks. I'll update the MP and hope it's the right thing...
[21:42] <powersj> running here: https://jenkins.ubuntu.com/server/job/cloud-init-ci/5/console
[21:43] <blackboxsw> oops I had kicked again too , gonna get lots of ci votes
[21:59] <robjo> smoser: https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/333904  is ready for another look
[21:59] <robjo> blackboxsw: is still on the hook for noLnxDistro and emptyStageOK
[22:00] <robjo> and with that I have reached EOD for me, catch you tomorrow
[22:00] <blackboxsw> <-- hooked. (cheers)
[22:27] <throwaway5234> Hello. Stupid question.
[22:27] <throwaway5234> Documentation says to use write_files like this
[22:28] <throwaway5234>  -   encoding: gzip        content: !!binary |            H4sIAGUfoFQC/zMxAgCIsCQyAgAAAA==
[22:28] <throwaway5234> how do I create the H4s... part
[22:37] <rharper> throwaway5234: typically you'll gzip and then base64 encode:  cat /my/binary | gzip | base64
[22:39] <rharper> % echo H4sIAGUfoFQC/zMxAgCIsCQyAgAAAA== | base64 --decode | zcat
[22:39] <rharper> 42
[22:39] <throwaway5234> I'm mainly trying to compress a config file that's too long.
[22:40] <rharper> you can use a URL for cloud config
[22:40] <throwaway5234> so if I did your approach would I declare it gzip or gzip+b64
[22:40] <rharper> if that's what you mead
[22:40] <rharper> s/mead/need
[22:44] <rharper> hrm, maybe the example needs updating, the gzip+b64 seems correct;   I wonder what happens if you just say gzip though
[22:45] <rharper> yaml needs to have binary data b64 encoded anyhow , but I think best to use both, write_files will b64 decode then gunzip  the payload before writing
[22:46] <throwaway5234> I appreciate you help! I was looking online for a couple hours and everyone glosses over that part.
[23:41] <throwaway5234> I'm still having trouble with it. Don't know what to say.
[23:41] <throwaway5234> https://pastebin.com/kMXE17tk