/srv/irclogs.ubuntu.com/2018/05/17/#cloud-init.txt

paulmeysmoser:  rharper : https://code.launchpad.net/~paul-meyer/cloud-init/+git/cloud-init/+merge/344538 is ready for review00:43
smoserblackboxsw: around ?01:36
smoserpaulmey: just hit 'save comment'01:47
smoserfeel free to ask questions here for th enext few minutes01:47
paulmeythanks smoser02:05
paulmeyBTW, the error code from mount is always 32 - mount failure... so string match seems like the only reasonable option02:06
paulmeyalso BTW, IIRC, ntfs is mounted via fuse now on ubuntu, so there is no ntfs module involved...02:08
paulmeyas an example where an assumption and a grep of /proc/filesystems might lead to a wrong conclusion...02:09
smoserfair point.02:09
paulmeyon the other hand, I fully support the "string match is brittle" argument02:09
smoserexit(32)02:09
* smoser walks afk for the night02:10
* paulmey is afk till tomorrow02:14
akiki was able to provision a vm with libcloud to azure. i'm using cloud-init in libcloud's ex_customdata (azure customdata), but i think that's only for user-data and not meta-data09:31
akikcan i put the cloud-init meta-data somehow through the customdata mechanism?09:32
caribourharper: Hello, remember the other day I told you about  LP: #1531880 and that I was seeing a similar situation ?13:14
ubot5Launchpad bug 1531880 in cloud-init "Failed to start Initial cloud-init job (pre-networking)" [Undecided,New] https://launchpad.net/bugs/153188013:14
caribouI'm able to reproduce it on our images. Seems to be caused by a race-condition between cloud-final.service that gets to run before cloud-init.service13:15
caribouit can be easily reproduced with :13:16
cariboucloud-init clean && cloud-init init --local && cloud-init modules --mode=final && cloud-init init13:16
rharpercaribou: interesting; given the systemd unit requirements, how do you get final to run before cloud-init init ?13:48
smosercaribou: running as you are in that '&&' statement is not valid.14:29
caribousmoser: yeah, I know this was  just to outline the fact that if cloud-final has a chance to run before cloud-init makes the link it will fail14:30
smoserand to the best of my knowledge, systemd units enforce order of cloud-init-local.service, cloud-init.service, cloud-config.service, cloud-final.service14:30
cariboubut looking at my systemd-analyze plot output, the ordering seems correct14:31
caribouyet, all I need to do is cloud-init clean & reboot and I'll get the failure14:31
caribouFYI i'm not working off an Ubuntu Cloud Image but one of our own Scaleway images14:32
smosercaribou: on that system... /var/lib/cloud/instance is a link or a directoroy ?14:32
smosercan you let me in to look around ?14:32
caribousmoser: a directory. I straced the execution & I clearly see cloud-final doing a mkdir /var/lib/cloud/instance14:32
caribousure let me pull in your ssh key & I'll PM the IP14:33
mgerdtssmoser blackboxsw rharper: I'm in need of images with all of my cloud-init fixes by June 30.  Can we wrap up my two merge proposals soon?  https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/343712 https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/34416814:34
smoserthe networking reboot thing is harder.14:36
mgerdtswhat is needed to make it not so?14:37
blackboxswmorning folks.14:37
mgerdtsmorning!14:37
smosermgerdts: just work and thought oon how that will i nteract with other "dynamic" networking14:38
smoseron the other... i think you need updates to your sdc:routes one. the commit message i think is not correct now.14:38
smoser yeah. you removed the global routes. need to update the commit message.14:38
mgerdtsIf someone is doing dynamic networking that could interfere, setting maintain_networks to false (or leaving it unset, the default) will make it so that traditional behavior is maintained.14:39
mgerdtswill update the commit message there.14:39
mgerdtsI'm having troubles understanding the failure at https://jenkins.ubuntu.com/server/job/cloud-init-ci/1140/console14:47
blackboxswchecking now mgerdts I just queued our CI to run against the branch so we'd have a point of reference14:48
blackboxswahh pylint, should be easy getting you the error (it's buried above)14:49
mgerdtsI see it nowtests/unittests/test_handler/test_handler_bootcmd.py:150: [W0612(unused-variable), TestSchema.test_duplicates_are_fine_array_array] Unused variable 'e'14:49
mgerdtsthis is in code I've not touched.14:49
blackboxswyep https://pastebin.ubuntu.com/p/mmrk3wWK4r/ checking14:49
blackboxswmgertds, could be that pylint upstream in ci has updated, you might need to 'git fetch master'  'get rebase -i master'14:50
blackboxswsince the time that you originally posted this branch we've had a couple pycodestyle/pylint fixes land in master14:50
mgerdtsI'll give that a whirl.  Then git push -f?14:51
blackboxswcorrect mgerdts yeah then git push --force14:51
blackboxswafter the rebase 'works'14:51
blackboxswworks without conflicts (which I imagine it should)14:51
blackboxswwe have removed pylint target in favor of pycodesyle/pyflakes. so master rebase should resolve the issue14:53
mgerdtsdone & pushed14:53
blackboxswcheers, I'll kick it again14:53
blackboxswmgerdts: I queued another build, but a banner on our jenkins says it's going down for service soon, so we may not get a response on that build until jenkins maintenance is done14:55
mgerdtsok, thanks14:55
blackboxswbuild 1141 here https://jenkins.ubuntu.com/server/job/cloud-init-ci/14:55
smoserblackboxsw: we should do somethinng on those jobs... regex or somethign to grab errors at the end14:57
blackboxswit's a good idea smoser, same with the integration test job, we do dump a dict with results that's really hard to read as a human (and I think misread by jenkins as success too)14:57
blackboxswI think I saw an example of failures that jenkins thinks passed (but didn't)14:58
blackboxswwell jenkins going for reboot now14:58
blackboxswbut intregration-proposed-ec2-b  I think was false success (because salt_minion int tests fail on bionic014:59
smoserstupid humans15:00
smoserpathetic15:00
blackboxswI should change the topic 'stupid pathetic humans'15:06
mgerdtsrharper I was just looking through your latest feedback and am a little confused by why the code I inherited from a co-worker is doing things like:15:18
mgerdtsrcfg = dict(...)15:18
mgerdtsrcfg.update({...})15:18
mgerdtsInstead of using update when there's just one or two items to update, why not use15:19
mgerdtsrcfg['network'] = route['dst']15:19
rharpermgerdts: heh, so the general path is to copy keys out of the sdc: values into a config dict and the map values where the keys didn't match but are equivalent15:20
mgerdtssame goes for subnet just below15:20
mgerdtsI think that both accomplish the same thing, but one uses the dict.update method.  Is there a reason that dict.update is preferred over just doing an assignment?15:22
rharpernot for a single value  no15:22
mgerdtsI guess dict.update is used quite a bit for single values other places too.  When in Rome...15:24
rharpermgerdts: heh;  I'm not fussed either way; the comment was more focused on the valid_keys, include the values from sdc:routes that have keys that we want to pickup directly, and then update the dict for any key/value pairs that need a mapping instead15:26
mgerdtsYeah, I understand that.  I started asking trying to head off the next review comment that says ".update() is pointless now"15:27
mgerdtsBut I see there's a lack of consistency in extant code.15:27
mgerdtshttps://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/344168#diff-line-6215:27
mgerdtspgpws[proto] and subnet each have one key modified, but differently.15:27
smoserrobjo: https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/33540615:50
smoseris that still relevant ? have we got what we needed for iproute2 support15:51
robjosmoser: was addressed by the other commit proposal deleted15:59
smoserthanks.15:59
smoserrobjo: you have some others too... sorry to pester16:00
smoser https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/33390416:00
smoser https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/33499216:00
robjoI think 333904 is still valid16:01
robjo334992 deleted16:01
* powersj adds cosmic integration tests16:04
blackboxswpowersj: cloud-init-ci job disappeared post reboot16:12
blackboxswhttps://jenkins.ubuntu.com/server/job/cloud-init-ci/1047/  etc missing now16:12
powersjLOL16:13
powersjsame with git-ubuntu16:13
powersjshhhhooootttt16:13
blackboxswI see cloud-init-ci-lander and cloud-init-ci-trigger too, hrm16:13
powersjpipeline based jobs16:13
blackboxswyeah weren't in the redeploy logic/source?16:14
powersjjobs are deployed by us, why it got deleted is beyond me16:15
powersjI am deploying with ignored cache right now16:16
powersjblackboxsw: https://paste.ubuntu.com/p/spzBwbNy8w/16:17
powersjbarfed on pipeline jobs16:17
powersjothers worked fine16:17
powersjah shoot I can't even make it by han16:18
powersjhand16:18
powersjrt time16:18
blackboxswahh man :/16:19
powersjblackboxsw: I cc'ed you on it16:21
blackboxswpowersj: yeah, I wonder if it's a missed plugin in the juju spec16:27
powersjpossibly, however, he noticed we were using the pipeline plugin so he made sure the master wouldn't get turned off16:28
blackboxswlike after the last #is update we didn't sync the new jenkins plugin in the spec16:28
* blackboxsw tries to find that spec again16:28
smoserrobjo: can you rebase 904 on master and push --force ?16:51
mgerdtssmoser: I took my best guess at parsing 'just work and thought oon how that will i nteract with other "dynamic" networking' and replying.17:11
mgerdtsI'm failing to see why preserving legacy behavior as the default and opting in is a problem.  Then again, I'm not sure that you've acknowledged that this is the way that it works and it is a problem.17:12
robjosmoser: will do after my next call which starts in 10 minutes, got to stuff my face first17:14
smoserrobjo: did you find out why PATH was not set in your environment ?18:18
smoseror in your users's enviroment at least ?18:18
robjoWhat I have learned is that on Debian/Ubuntu PID 1 is actually a script and and there is some re-direction happening and in that script the path gets set, we run systemd directly and thus no PATH18:20
smoserPID1 is most definitely not a script in ubuntu18:20
robjoAlso people chimed in and stated that blkid should not run that early in the boot process18:21
smoserinitramfs runs without /sbin/init == systemd, but pid1 there does exec /sbin/init (which is systemd)18:21
robjonot verified that's what I got as a comment in the bug report18:21
smoserand initramfs does not run everywhere, so thats not what is fixing it.18:21
robjothere's also supposedly etc/systemd.conf where the path maybe set18:22
smoserif you had actual evidence that blkid should not be run that early, then i'm very interested in it.18:22
smoseras i have been concerned that we might be missing devices that "show up" later.18:22
robjoI'll send you the suggestion one of our filesystem guys made via e-mail18:23
smoserbut my experience in now almost 9 years of doing this is that i've never seen it.18:23
smoserif you can give me a reproduce that shows running blkid once that early in teh boot, and then running it later changed ds-identify behavior (outside of you attaching a disk late by design) then i'm willing to fix.18:24
smoseri'm also willing to fix PATH, even though i think that is really not the right fix, as cloud-init will not be the only thing that has to then decide on what is a sane path... better to do that once in the distro.18:25
robjoyou should have an e-mail18:32
robjothe thing is it appears that things work in the AWS images, but I have had a customer report and then one in openSUSE that blkid is not being found18:32
robjosmoser: https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/333904 rebased and pushed18:44
Odd_Blokepowersj: smoser: Did we reach a conclusion about how to enable SSHing to the instance from within it?19:22
Odd_BlokeI've had a hectic couple of days, so I'm not sure if I'm just forgetting what we decided on. :p19:23
powersjOdd_Bloke: did smoser propose some option? Like pushing or writing the key on the instance and use it from there?19:27
Odd_BlokeYeah, that sounds familiar.19:29
Odd_BlokeShall I take a crack at that?19:29
smoseryeah. thats what i had suggested.19:29
smoserthe only concern i have is that we would want to make sure we do not ever copy a user's private key.19:29
smoseronly ones that we generated.19:29
smoseri guess maybe just raise an exception somewhere if the key didn't get generated with 'ci-key' or something... i'm not sure we're generating them like that now, but that way we don't end up exposing oddbloke@demigogue's private key to the world.19:30
rharperblackboxsw: what's that jq one-liner for dumping user-data out of the instance json ?19:56
blackboxswahh got it one sec20:04
rharperhrm, need to somehow escape the dash ?20:04
smoserrobjo: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/34578620:06
blackboxswrharper: cat /run/cloud-init/instance-data.json | jq -r '.ds."user-data"'20:06
rharperah, that's it20:07
blackboxswcat /run/cloud-init/instance-data.json | jq -r '.ds."user-data"' | base64 -d20:07
rharperthe "user-data"20:07
rharperblackboxsw: thanks!20:07
blackboxswyeah I thought I had reference in ubuntu-sru, but turns out I didn't, had to back into it20:07
blackboxswnp20:07
rharperhrm20:07
blackboxswrharper: or just "cloud-init query user-data"20:07
blackboxswohh wait too soon20:07
rharperhehe20:08
rharperhrm, so I've got a collect logs tar20:08
rharperI ran that with the decode and I get garbage ...20:08
rharperwhat might be in the user-data raw for a maas deploy ?20:08
rharper# cat instance-data.json | jq -r '.ds."user-data"' | base64 -d20:08
rharper%^�(�yj�Ӻ�20:08
blackboxswlooks like a qbert curse word20:08
blackboxswhmm trying to think there thought I had a pastebin of maas cloud-init user-data20:09
blackboxswas passed through curtin20:09
rharperah20:10
rharpergziped20:10
rharperwe do't tag that do we20:10
rharper cat instance-data.json | jq -r '.ds."user-data"' | base64 --decode | zcat20:10
rharper\o/20:10
blackboxswnice, wow20:11
blackboxswwe should probably tag that20:11
rharper#cloud-config20:11
rharperapt_mirror: ""20:11
rharper 20:11
rharperthat seems to make cloud-init apt unhappy20:11
blackboxswrharper: we only tag "base64-encoded-keys" in cloud-init instance-data.json20:12
rharperwe did tag it20:12
rharperjust not that it's gzipped20:12
blackboxswrharper: was ds/user-data  in there20:12
blackboxswok20:12
rharperhttps://paste.ubuntu.com/p/VP92hb9kvG/20:12
blackboxswrharper: that's my bug against maas20:12
rharperoh ?20:12
rharperinteresting20:12
rharpershouldn't we handle it more gracefully though ?20:13
blackboxswgetting the #20:13
blackboxswwell not my bug, but commented on a potential solution from maas https://bugs.launchpad.net/maas/+bug/173595020:13
ubot5Ubuntu bug 1735950 in MAAS 2.3 "ValueError: Old and New apt format defined with unequal values True vs False @ apt_preserve_sources_list" [Low,Triaged]20:13
blackboxswrharper: I don't know if we can behave better20:14
blackboxswmaas passes conflicting settings using the old key versus new key20:14
blackboxswI guess we could avoid a traceback and say, "hey buddy, you passed me a False and True for the same arguments, I'm trusting the new key"20:14
rharperhrm, in the user-data there's nothing apt_ there20:14
blackboxswinstead of the deprecated key20:14
rharperexcept the apt_mirror: "" value20:15
rharperbut that is the stack trace20:15
blackboxswrharper: right maas curtin->cloud-init config always passes both I think.20:15
rharperI'm asying in the cloudin config passed, there is *only* the key apt_mirror: ""20:15
blackboxswrharper: line 49 and 52 in https://pastebin.ubuntu.com/26355967/20:16
rharperno other apt key at all20:16
rharperyes, I understand that, I'm seeing the same stacktrace but with no apt: key at all20:16
rharperthe user-data only contains apt_mirror: ""20:16
blackboxswhrm. what I thought was that maas sets up apt_preserve user-data config regardless of user-data provided to maas. maybe I'm misreading20:17
rharperno, you're right, it does20:19
rharperbut for whatever reason, the user-data from this maas deployed instance, does not contain that;  now it has a ton of extra juju stuff too20:19
blackboxswhrm.20:22
blackboxswwhat maas version?20:22
rharperblackboxsw: not sure, I don't think it's a maas change; I don't really care about maas here, it's the apperance that user-data with apt_mirror: "" throws a stack ;  maybe that's not what's triggering it and it's the bug you pointed to, but that would mean the instance json doesn't have the accurate user-data right ?20:29
rharperblackboxsw: secondary question, in our collect-logs, why don't we capture /var/lib/cloud  ?20:30
blackboxswrharper: probably a good thing to catch, it just wasn't in the original suggestion/procedure for filing bugs. I thought we pulled /run/cloud-init which has a couple symlinks to result|status.json, but it'd probably be good to see what's in /var/lib/cloud/(instance|seed) too20:38
rharperright, the data dir20:39
rharperincluding the obj.pkl20:39
paulmeysmoser: re LANG=C when doing mount...20:42
paulmeywould it be acceptable to pull the subp update_env parameter up into the mount_cb def?20:43
paulmeyor is there a better way of doing that?20:44
blackboxswfinally approved https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/34312720:55
blackboxswlanding it20:57
paulmeysmoser not around?21:13
blackboxswpaulmey: should be back in a bit. though he may be eod21:18
mgerdtsblackboxsw just did a resync to master and pushed it again.  You can kick the ci tests again to resolve the failure.21:38
mgerdtsOr maybe I can resubmit, but don't know how.21:38
powersjmgerdts: what is your merge request? I can check ci for you21:38
mgerdtshttps://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/34371221:39
mgerdtsoh, I guess the message I received has a "click here to rebuild" link21:40
paulmeyblackboxsw: ok, thanks. I'll update the MP and hope it's the right thing...21:41
powersjrunning here: https://jenkins.ubuntu.com/server/job/cloud-init-ci/5/console21:42
blackboxswoops I had kicked again too , gonna get lots of ci votes21:43
robjosmoser: https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/333904  is ready for another look21:59
robjoblackboxsw: is still on the hook for noLnxDistro and emptyStageOK21:59
robjoand with that I have reached EOD for me, catch you tomorrow22:00
blackboxsw<-- hooked. (cheers)22:00
throwaway5234Hello. Stupid question.22:27
throwaway5234Documentation says to use write_files like this22:27
throwaway5234 -   encoding: gzip        content: !!binary |            H4sIAGUfoFQC/zMxAgCIsCQyAgAAAA==22:28
throwaway5234how do I create the H4s... part22:28
rharperthrowaway5234: typically you'll gzip and then base64 encode:  cat /my/binary | gzip | base6422:37
rharper% echo H4sIAGUfoFQC/zMxAgCIsCQyAgAAAA== | base64 --decode | zcat22:39
rharper4222:39
throwaway5234I'm mainly trying to compress a config file that's too long.22:39
rharperyou can use a URL for cloud config22:40
throwaway5234so if I did your approach would I declare it gzip or gzip+b6422:40
rharperif that's what you mead22:40
rharpers/mead/need22:40
rharperhrm, maybe the example needs updating, the gzip+b64 seems correct;   I wonder what happens if you just say gzip though22:44
rharperyaml needs to have binary data b64 encoded anyhow , but I think best to use both, write_files will b64 decode then gunzip  the payload before writing22:45
throwaway5234I appreciate you help! I was looking online for a couple hours and everyone glosses over that part.22:46
throwaway5234I'm still having trouble with it. Don't know what to say.23:41
throwaway5234https://pastebin.com/kMXE17tk23:41

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!