paulmey | smoser: rharper : https://code.launchpad.net/~paul-meyer/cloud-init/+git/cloud-init/+merge/344538 is ready for review | 00:43 |
---|---|---|
smoser | blackboxsw: around ? | 01:36 |
smoser | paulmey: just hit 'save comment' | 01:47 |
smoser | feel free to ask questions here for th enext few minutes | 01:47 |
paulmey | thanks smoser | 02:05 |
paulmey | BTW, the error code from mount is always 32 - mount failure... so string match seems like the only reasonable option | 02:06 |
paulmey | also BTW, IIRC, ntfs is mounted via fuse now on ubuntu, so there is no ntfs module involved... | 02:08 |
paulmey | as an example where an assumption and a grep of /proc/filesystems might lead to a wrong conclusion... | 02:09 |
smoser | fair point. | 02:09 |
paulmey | on the other hand, I fully support the "string match is brittle" argument | 02:09 |
smoser | exit(32) | 02:09 |
* smoser walks afk for the night | 02:10 | |
* paulmey is afk till tomorrow | 02:14 | |
akik | i was able to provision a vm with libcloud to azure. i'm using cloud-init in libcloud's ex_customdata (azure customdata), but i think that's only for user-data and not meta-data | 09:31 |
akik | can i put the cloud-init meta-data somehow through the customdata mechanism? | 09:32 |
caribou | rharper: Hello, remember the other day I told you about LP: #1531880 and that I was seeing a similar situation ? | 13:14 |
ubot5 | Launchpad bug 1531880 in cloud-init "Failed to start Initial cloud-init job (pre-networking)" [Undecided,New] https://launchpad.net/bugs/1531880 | 13:14 |
caribou | I'm able to reproduce it on our images. Seems to be caused by a race-condition between cloud-final.service that gets to run before cloud-init.service | 13:15 |
caribou | it can be easily reproduced with : | 13:16 |
caribou | cloud-init clean && cloud-init init --local && cloud-init modules --mode=final && cloud-init init | 13:16 |
rharper | caribou: interesting; given the systemd unit requirements, how do you get final to run before cloud-init init ? | 13:48 |
smoser | caribou: running as you are in that '&&' statement is not valid. | 14:29 |
caribou | smoser: yeah, I know this was just to outline the fact that if cloud-final has a chance to run before cloud-init makes the link it will fail | 14:30 |
smoser | and to the best of my knowledge, systemd units enforce order of cloud-init-local.service, cloud-init.service, cloud-config.service, cloud-final.service | 14:30 |
caribou | but looking at my systemd-analyze plot output, the ordering seems correct | 14:31 |
caribou | yet, all I need to do is cloud-init clean & reboot and I'll get the failure | 14:31 |
caribou | FYI i'm not working off an Ubuntu Cloud Image but one of our own Scaleway images | 14:32 |
smoser | caribou: on that system... /var/lib/cloud/instance is a link or a directoroy ? | 14:32 |
smoser | can you let me in to look around ? | 14:32 |
caribou | smoser: a directory. I straced the execution & I clearly see cloud-final doing a mkdir /var/lib/cloud/instance | 14:32 |
caribou | sure let me pull in your ssh key & I'll PM the IP | 14:33 |
mgerdts | smoser blackboxsw rharper: I'm in need of images with all of my cloud-init fixes by June 30. Can we wrap up my two merge proposals soon? https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/343712 https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/344168 | 14:34 |
smoser | the networking reboot thing is harder. | 14:36 |
mgerdts | what is needed to make it not so? | 14:37 |
blackboxsw | morning folks. | 14:37 |
mgerdts | morning! | 14:37 |
smoser | mgerdts: just work and thought oon how that will i nteract with other "dynamic" networking | 14:38 |
smoser | on the other... i think you need updates to your sdc:routes one. the commit message i think is not correct now. | 14:38 |
smoser | yeah. you removed the global routes. need to update the commit message. | 14:38 |
mgerdts | If someone is doing dynamic networking that could interfere, setting maintain_networks to false (or leaving it unset, the default) will make it so that traditional behavior is maintained. | 14:39 |
mgerdts | will update the commit message there. | 14:39 |
mgerdts | I'm having troubles understanding the failure at https://jenkins.ubuntu.com/server/job/cloud-init-ci/1140/console | 14:47 |
blackboxsw | checking now mgerdts I just queued our CI to run against the branch so we'd have a point of reference | 14:48 |
blackboxsw | ahh pylint, should be easy getting you the error (it's buried above) | 14:49 |
mgerdts | I see it nowtests/unittests/test_handler/test_handler_bootcmd.py:150: [W0612(unused-variable), TestSchema.test_duplicates_are_fine_array_array] Unused variable 'e' | 14:49 |
mgerdts | this is in code I've not touched. | 14:49 |
blackboxsw | yep https://pastebin.ubuntu.com/p/mmrk3wWK4r/ checking | 14:49 |
blackboxsw | mgertds, could be that pylint upstream in ci has updated, you might need to 'git fetch master' 'get rebase -i master' | 14:50 |
blackboxsw | since the time that you originally posted this branch we've had a couple pycodestyle/pylint fixes land in master | 14:50 |
mgerdts | I'll give that a whirl. Then git push -f? | 14:51 |
blackboxsw | correct mgerdts yeah then git push --force | 14:51 |
blackboxsw | after the rebase 'works' | 14:51 |
blackboxsw | works without conflicts (which I imagine it should) | 14:51 |
blackboxsw | we have removed pylint target in favor of pycodesyle/pyflakes. so master rebase should resolve the issue | 14:53 |
mgerdts | done & pushed | 14:53 |
blackboxsw | cheers, I'll kick it again | 14:53 |
blackboxsw | mgerdts: I queued another build, but a banner on our jenkins says it's going down for service soon, so we may not get a response on that build until jenkins maintenance is done | 14:55 |
mgerdts | ok, thanks | 14:55 |
blackboxsw | build 1141 here https://jenkins.ubuntu.com/server/job/cloud-init-ci/ | 14:55 |
smoser | blackboxsw: we should do somethinng on those jobs... regex or somethign to grab errors at the end | 14:57 |
blackboxsw | it's a good idea smoser, same with the integration test job, we do dump a dict with results that's really hard to read as a human (and I think misread by jenkins as success too) | 14:57 |
blackboxsw | I think I saw an example of failures that jenkins thinks passed (but didn't) | 14:58 |
blackboxsw | well jenkins going for reboot now | 14:58 |
blackboxsw | but intregration-proposed-ec2-b I think was false success (because salt_minion int tests fail on bionic0 | 14:59 |
smoser | stupid humans | 15:00 |
smoser | pathetic | 15:00 |
blackboxsw | I should change the topic 'stupid pathetic humans' | 15:06 |
mgerdts | rharper I was just looking through your latest feedback and am a little confused by why the code I inherited from a co-worker is doing things like: | 15:18 |
mgerdts | rcfg = dict(...) | 15:18 |
mgerdts | rcfg.update({...}) | 15:18 |
mgerdts | Instead of using update when there's just one or two items to update, why not use | 15:19 |
mgerdts | rcfg['network'] = route['dst'] | 15:19 |
rharper | mgerdts: heh, so the general path is to copy keys out of the sdc: values into a config dict and the map values where the keys didn't match but are equivalent | 15:20 |
mgerdts | same goes for subnet just below | 15:20 |
mgerdts | I think that both accomplish the same thing, but one uses the dict.update method. Is there a reason that dict.update is preferred over just doing an assignment? | 15:22 |
rharper | not for a single value no | 15:22 |
mgerdts | I guess dict.update is used quite a bit for single values other places too. When in Rome... | 15:24 |
rharper | mgerdts: heh; I'm not fussed either way; the comment was more focused on the valid_keys, include the values from sdc:routes that have keys that we want to pickup directly, and then update the dict for any key/value pairs that need a mapping instead | 15:26 |
mgerdts | Yeah, I understand that. I started asking trying to head off the next review comment that says ".update() is pointless now" | 15:27 |
mgerdts | But I see there's a lack of consistency in extant code. | 15:27 |
mgerdts | https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/344168#diff-line-62 | 15:27 |
mgerdts | pgpws[proto] and subnet each have one key modified, but differently. | 15:27 |
smoser | robjo: https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/335406 | 15:50 |
smoser | is that still relevant ? have we got what we needed for iproute2 support | 15:51 |
robjo | smoser: was addressed by the other commit proposal deleted | 15:59 |
smoser | thanks. | 15:59 |
smoser | robjo: you have some others too... sorry to pester | 16:00 |
smoser | https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/333904 | 16:00 |
smoser | https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/334992 | 16:00 |
robjo | I think 333904 is still valid | 16:01 |
robjo | 334992 deleted | 16:01 |
* powersj adds cosmic integration tests | 16:04 | |
blackboxsw | powersj: cloud-init-ci job disappeared post reboot | 16:12 |
blackboxsw | https://jenkins.ubuntu.com/server/job/cloud-init-ci/1047/ etc missing now | 16:12 |
powersj | LOL | 16:13 |
powersj | same with git-ubuntu | 16:13 |
powersj | shhhhooootttt | 16:13 |
blackboxsw | I see cloud-init-ci-lander and cloud-init-ci-trigger too, hrm | 16:13 |
powersj | pipeline based jobs | 16:13 |
blackboxsw | yeah weren't in the redeploy logic/source? | 16:14 |
powersj | jobs are deployed by us, why it got deleted is beyond me | 16:15 |
powersj | I am deploying with ignored cache right now | 16:16 |
powersj | blackboxsw: https://paste.ubuntu.com/p/spzBwbNy8w/ | 16:17 |
powersj | barfed on pipeline jobs | 16:17 |
powersj | others worked fine | 16:17 |
powersj | ah shoot I can't even make it by han | 16:18 |
powersj | hand | 16:18 |
powersj | rt time | 16:18 |
blackboxsw | ahh man :/ | 16:19 |
powersj | blackboxsw: I cc'ed you on it | 16:21 |
blackboxsw | powersj: yeah, I wonder if it's a missed plugin in the juju spec | 16:27 |
powersj | possibly, however, he noticed we were using the pipeline plugin so he made sure the master wouldn't get turned off | 16:28 |
blackboxsw | like after the last #is update we didn't sync the new jenkins plugin in the spec | 16:28 |
* blackboxsw tries to find that spec again | 16:28 | |
smoser | robjo: can you rebase 904 on master and push --force ? | 16:51 |
mgerdts | smoser: I took my best guess at parsing 'just work and thought oon how that will i nteract with other "dynamic" networking' and replying. | 17:11 |
mgerdts | I'm failing to see why preserving legacy behavior as the default and opting in is a problem. Then again, I'm not sure that you've acknowledged that this is the way that it works and it is a problem. | 17:12 |
robjo | smoser: will do after my next call which starts in 10 minutes, got to stuff my face first | 17:14 |
smoser | robjo: did you find out why PATH was not set in your environment ? | 18:18 |
smoser | or in your users's enviroment at least ? | 18:18 |
robjo | What I have learned is that on Debian/Ubuntu PID 1 is actually a script and and there is some re-direction happening and in that script the path gets set, we run systemd directly and thus no PATH | 18:20 |
smoser | PID1 is most definitely not a script in ubuntu | 18:20 |
robjo | Also people chimed in and stated that blkid should not run that early in the boot process | 18:21 |
smoser | initramfs runs without /sbin/init == systemd, but pid1 there does exec /sbin/init (which is systemd) | 18:21 |
robjo | not verified that's what I got as a comment in the bug report | 18:21 |
smoser | and initramfs does not run everywhere, so thats not what is fixing it. | 18:21 |
robjo | there's also supposedly etc/systemd.conf where the path maybe set | 18:22 |
smoser | if you had actual evidence that blkid should not be run that early, then i'm very interested in it. | 18:22 |
smoser | as i have been concerned that we might be missing devices that "show up" later. | 18:22 |
robjo | I'll send you the suggestion one of our filesystem guys made via e-mail | 18:23 |
smoser | but my experience in now almost 9 years of doing this is that i've never seen it. | 18:23 |
smoser | if you can give me a reproduce that shows running blkid once that early in teh boot, and then running it later changed ds-identify behavior (outside of you attaching a disk late by design) then i'm willing to fix. | 18:24 |
smoser | i'm also willing to fix PATH, even though i think that is really not the right fix, as cloud-init will not be the only thing that has to then decide on what is a sane path... better to do that once in the distro. | 18:25 |
robjo | you should have an e-mail | 18:32 |
robjo | the thing is it appears that things work in the AWS images, but I have had a customer report and then one in openSUSE that blkid is not being found | 18:32 |
robjo | smoser: https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/333904 rebased and pushed | 18:44 |
Odd_Bloke | powersj: smoser: Did we reach a conclusion about how to enable SSHing to the instance from within it? | 19:22 |
Odd_Bloke | I've had a hectic couple of days, so I'm not sure if I'm just forgetting what we decided on. :p | 19:23 |
powersj | Odd_Bloke: did smoser propose some option? Like pushing or writing the key on the instance and use it from there? | 19:27 |
Odd_Bloke | Yeah, that sounds familiar. | 19:29 |
Odd_Bloke | Shall I take a crack at that? | 19:29 |
smoser | yeah. thats what i had suggested. | 19:29 |
smoser | the only concern i have is that we would want to make sure we do not ever copy a user's private key. | 19:29 |
smoser | only ones that we generated. | 19:29 |
smoser | i guess maybe just raise an exception somewhere if the key didn't get generated with 'ci-key' or something... i'm not sure we're generating them like that now, but that way we don't end up exposing oddbloke@demigogue's private key to the world. | 19:30 |
rharper | blackboxsw: what's that jq one-liner for dumping user-data out of the instance json ? | 19:56 |
blackboxsw | ahh got it one sec | 20:04 |
rharper | hrm, need to somehow escape the dash ? | 20:04 |
smoser | robjo: https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/345786 | 20:06 |
blackboxsw | rharper: cat /run/cloud-init/instance-data.json | jq -r '.ds."user-data"' | 20:06 |
rharper | ah, that's it | 20:07 |
blackboxsw | cat /run/cloud-init/instance-data.json | jq -r '.ds."user-data"' | base64 -d | 20:07 |
rharper | the "user-data" | 20:07 |
rharper | blackboxsw: thanks! | 20:07 |
blackboxsw | yeah I thought I had reference in ubuntu-sru, but turns out I didn't, had to back into it | 20:07 |
blackboxsw | np | 20:07 |
rharper | hrm | 20:07 |
blackboxsw | rharper: or just "cloud-init query user-data" | 20:07 |
blackboxsw | ohh wait too soon | 20:07 |
rharper | hehe | 20:08 |
rharper | hrm, so I've got a collect logs tar | 20:08 |
rharper | I ran that with the decode and I get garbage ... | 20:08 |
rharper | what might be in the user-data raw for a maas deploy ? | 20:08 |
rharper | # cat instance-data.json | jq -r '.ds."user-data"' | base64 -d | 20:08 |
rharper | %^�(�yj�Ӻ� | 20:08 |
blackboxsw | looks like a qbert curse word | 20:08 |
blackboxsw | hmm trying to think there thought I had a pastebin of maas cloud-init user-data | 20:09 |
blackboxsw | as passed through curtin | 20:09 |
rharper | ah | 20:10 |
rharper | gziped | 20:10 |
rharper | we do't tag that do we | 20:10 |
rharper | cat instance-data.json | jq -r '.ds."user-data"' | base64 --decode | zcat | 20:10 |
rharper | \o/ | 20:10 |
blackboxsw | nice, wow | 20:11 |
blackboxsw | we should probably tag that | 20:11 |
rharper | #cloud-config | 20:11 |
rharper | apt_mirror: "" | 20:11 |
rharper | 20:11 | |
rharper | that seems to make cloud-init apt unhappy | 20:11 |
blackboxsw | rharper: we only tag "base64-encoded-keys" in cloud-init instance-data.json | 20:12 |
rharper | we did tag it | 20:12 |
rharper | just not that it's gzipped | 20:12 |
blackboxsw | rharper: was ds/user-data in there | 20:12 |
blackboxsw | ok | 20:12 |
rharper | https://paste.ubuntu.com/p/VP92hb9kvG/ | 20:12 |
blackboxsw | rharper: that's my bug against maas | 20:12 |
rharper | oh ? | 20:12 |
rharper | interesting | 20:12 |
rharper | shouldn't we handle it more gracefully though ? | 20:13 |
blackboxsw | getting the # | 20:13 |
blackboxsw | well not my bug, but commented on a potential solution from maas https://bugs.launchpad.net/maas/+bug/1735950 | 20:13 |
ubot5 | Ubuntu bug 1735950 in MAAS 2.3 "ValueError: Old and New apt format defined with unequal values True vs False @ apt_preserve_sources_list" [Low,Triaged] | 20:13 |
blackboxsw | rharper: I don't know if we can behave better | 20:14 |
blackboxsw | maas passes conflicting settings using the old key versus new key | 20:14 |
blackboxsw | I guess we could avoid a traceback and say, "hey buddy, you passed me a False and True for the same arguments, I'm trusting the new key" | 20:14 |
rharper | hrm, in the user-data there's nothing apt_ there | 20:14 |
blackboxsw | instead of the deprecated key | 20:14 |
rharper | except the apt_mirror: "" value | 20:15 |
rharper | but that is the stack trace | 20:15 |
blackboxsw | rharper: right maas curtin->cloud-init config always passes both I think. | 20:15 |
rharper | I'm asying in the cloudin config passed, there is *only* the key apt_mirror: "" | 20:15 |
blackboxsw | rharper: line 49 and 52 in https://pastebin.ubuntu.com/26355967/ | 20:16 |
rharper | no other apt key at all | 20:16 |
rharper | yes, I understand that, I'm seeing the same stacktrace but with no apt: key at all | 20:16 |
rharper | the user-data only contains apt_mirror: "" | 20:16 |
blackboxsw | hrm. what I thought was that maas sets up apt_preserve user-data config regardless of user-data provided to maas. maybe I'm misreading | 20:17 |
rharper | no, you're right, it does | 20:19 |
rharper | but for whatever reason, the user-data from this maas deployed instance, does not contain that; now it has a ton of extra juju stuff too | 20:19 |
blackboxsw | hrm. | 20:22 |
blackboxsw | what maas version? | 20:22 |
rharper | blackboxsw: not sure, I don't think it's a maas change; I don't really care about maas here, it's the apperance that user-data with apt_mirror: "" throws a stack ; maybe that's not what's triggering it and it's the bug you pointed to, but that would mean the instance json doesn't have the accurate user-data right ? | 20:29 |
rharper | blackboxsw: secondary question, in our collect-logs, why don't we capture /var/lib/cloud ? | 20:30 |
blackboxsw | rharper: probably a good thing to catch, it just wasn't in the original suggestion/procedure for filing bugs. I thought we pulled /run/cloud-init which has a couple symlinks to result|status.json, but it'd probably be good to see what's in /var/lib/cloud/(instance|seed) too | 20:38 |
rharper | right, the data dir | 20:39 |
rharper | including the obj.pkl | 20:39 |
paulmey | smoser: re LANG=C when doing mount... | 20:42 |
paulmey | would it be acceptable to pull the subp update_env parameter up into the mount_cb def? | 20:43 |
paulmey | or is there a better way of doing that? | 20:44 |
blackboxsw | finally approved https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/343127 | 20:55 |
blackboxsw | landing it | 20:57 |
paulmey | smoser not around? | 21:13 |
blackboxsw | paulmey: should be back in a bit. though he may be eod | 21:18 |
mgerdts | blackboxsw just did a resync to master and pushed it again. You can kick the ci tests again to resolve the failure. | 21:38 |
mgerdts | Or maybe I can resubmit, but don't know how. | 21:38 |
powersj | mgerdts: what is your merge request? I can check ci for you | 21:38 |
mgerdts | https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/343712 | 21:39 |
mgerdts | oh, I guess the message I received has a "click here to rebuild" link | 21:40 |
paulmey | blackboxsw: ok, thanks. I'll update the MP and hope it's the right thing... | 21:41 |
powersj | running here: https://jenkins.ubuntu.com/server/job/cloud-init-ci/5/console | 21:42 |
blackboxsw | oops I had kicked again too , gonna get lots of ci votes | 21:43 |
robjo | smoser: https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/333904 is ready for another look | 21:59 |
robjo | blackboxsw: is still on the hook for noLnxDistro and emptyStageOK | 21:59 |
robjo | and with that I have reached EOD for me, catch you tomorrow | 22:00 |
blackboxsw | <-- hooked. (cheers) | 22:00 |
throwaway5234 | Hello. Stupid question. | 22:27 |
throwaway5234 | Documentation says to use write_files like this | 22:27 |
throwaway5234 | - encoding: gzip content: !!binary | H4sIAGUfoFQC/zMxAgCIsCQyAgAAAA== | 22:28 |
throwaway5234 | how do I create the H4s... part | 22:28 |
rharper | throwaway5234: typically you'll gzip and then base64 encode: cat /my/binary | gzip | base64 | 22:37 |
rharper | % echo H4sIAGUfoFQC/zMxAgCIsCQyAgAAAA== | base64 --decode | zcat | 22:39 |
rharper | 42 | 22:39 |
throwaway5234 | I'm mainly trying to compress a config file that's too long. | 22:39 |
rharper | you can use a URL for cloud config | 22:40 |
throwaway5234 | so if I did your approach would I declare it gzip or gzip+b64 | 22:40 |
rharper | if that's what you mead | 22:40 |
rharper | s/mead/need | 22:40 |
rharper | hrm, maybe the example needs updating, the gzip+b64 seems correct; I wonder what happens if you just say gzip though | 22:44 |
rharper | yaml needs to have binary data b64 encoded anyhow , but I think best to use both, write_files will b64 decode then gunzip the payload before writing | 22:45 |
throwaway5234 | I appreciate you help! I was looking online for a couple hours and everyone glosses over that part. | 22:46 |
throwaway5234 | I'm still having trouble with it. Don't know what to say. | 23:41 |
throwaway5234 | https://pastebin.com/kMXE17tk | 23:41 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!