| redguy | hi, I am restarting systemd-journald as a part of my cloud-init final scripts and this seems to break the execution of cloud-final.service on Debian jessie (exit status 1). Is this known and is there some workaround I could use? | 13:29 |
|---|---|---|
| redguy | or rephrased: what is the cannonical way of managing journald config with cloud-init ? | 14:57 |
| redguy | s/cannonical/recommended/ | 14:58 |
| smoser | redguy: hm.. | 15:06 |
| smoser | thats not somethign i've done or considered. | 15:06 |
| smoser | how is 'break execution' ? | 15:07 |
| smoser | as in it doesnt run ? | 15:07 |
| redguy | as in the execution stops | 15:11 |
| redguy | trying to do it via runcmd now, but I think this will also break | 15:11 |
| smoser | that seems bad. | 15:12 |
| redguy | https://pastebin.com/0m8QVEZq | 15:12 |
| smoser | bad that restarting systemd-journald would stop a service | 15:12 |
| redguy | yeah, possibly this is a bug in systemd :-) | 15:12 |
| smoser | can you get /var/log/cloud-init-output.log and /var/log/cloud-init.log ? | 15:13 |
| smoser | and fyi, https://hastebin.com/ doesn't hurt your eyes to look at :) | 15:13 |
| redguy | oh, nice | 15:14 |
| redguy | will get logs in couple of secs, but this time with runcmd approach | 15:14 |
| redguy | hmm, runcmd didn't fail, but it seems that it it wasn't executed at all (as in the mkdir in there didn't create the directory I wanted it to create). This most likely this is a config problem on my part. Going back to the pervious setup with restarting systemd-journald in a shell script | 15:22 |
| smoser | systemd doesn't do as well as upstart did with restarting things otherwise being "dynamic" during boot | 15:24 |
| smoser | we had to move the installation of packages to final rather than cloud-config stage because of that | 15:25 |
| redguy | https://hastebin.com/fanahigiwo.php | 15:33 |
| redguy | so systemd says status 1, but the cloud-init.log says 141 | 15:33 |
| redguy | which is EPIPE afair | 15:33 |
| redguy | no, a result of SIGPIPE | 15:34 |
| redguy | so my current theory is that this is caused by syslog log handler: cloud-init has /dev/log socket open, journald is stopped and closes the socket, cloud-init breaks | 15:49 |
| redguy | can I override log_cfgs with user_data ? | 16:21 |
| smoser | redguy: i think that paste didnt work ? | 16:47 |
| smoser | redguy: i think you should be able to write to log configs. | 16:48 |
| smoser | but your disgnosis could be wrong i think | 16:48 |
| smoser | no.. system can't be that broken | 16:48 |
| smoser | i was thinking that it might be just SIGPIPE issue because journal is catching cloud-init's output | 16:49 |
| smoser | hm. | 16:49 |
| redguy | somehow hastebin doesn't work for me :/ | 16:58 |
| redguy | https://pastebin.com/5ev0Z7La | 16:58 |
| redguy | https://pastebin.com/raw/5ev0Z7La ? :-) | 16:59 |
| powersj | blackboxsw: smoser: https://git.launchpad.net/~powersj/cloud-init/commit/?id=3793d3d24500d5f9a6ee03cd28bd9f3e6182554c | 17:23 |
| smoser | powersj: just loking at the diff i tlooks like ou changed the param name in the doc string | 17:27 |
| smoser | but not in the usage | 17:27 |
| powersj | oh shoot | 17:28 |
| smoser | i dont think it needs _ base on it. | 17:28 |
| smoser | its the only 'image' thing in that class. | 17:28 |
| powersj | until the snapshot function | 17:28 |
| powersj | but yes only class var | 17:28 |
| powersj | I can revert it back to just image_ami | 17:28 |
| smoser | powersj: hwere does the snapshot get removed ? | 17:31 |
| smoser | the "Amazon EC2 Snapshot" | 17:31 |
| smoser | that backs the ami | 17:31 |
| powersj | the backing instance is removed still in image.py (that was already there from last night) | 17:32 |
| smoser | no.. | 17:32 |
| smoser | hm,.. | 17:32 |
| powersj | lol | 17:33 |
| smoser | ok. so 'create_image' does | 17:35 |
| smoser | a.) stop instance | 17:35 |
| smoser | b.) snapshot volume | 17:35 |
| smoser | (which creates a snap-XXXXXX thing) | 17:35 |
| smoser | c.) register an ami with snap-XXXXX as its root volume. (creating a ami-XXXXX thing) | 17:36 |
| smoser | then in CII-EC2Snapshot.destroy, you deregister the image | 17:37 |
| smoser | whicih deletes form AWS the ami-XXXX thing | 17:37 |
| powersj | yes | 17:37 |
| smoser | but does the snap-XXXXX thing get removed ? | 17:37 |
| powersj | to my knowledge of how the deregistering of an AMI works yes, looking at the console there are no longer anything listed under images or snapshots | 17:40 |
| smoser | interesting. | 17:42 |
| smoser | that does make snse, and much nicer than aking the user clean up the snapshot themselve. | 17:42 |
| smoser | which i thought you'd have to do. | 17:42 |
| smoser | oh... ik now why. | 17:42 |
| smoser | in ubuntu publishing, we would register the snapshot and then create an ami from that snapshot separately | 17:43 |
| smoser | so that we could register multiple ami from the same snapshot | 17:43 |
| smoser | (one as 'released' and one as 'daily') | 17:43 |
| smoser | so... good. thank you. | 17:43 |
| powersj | smoser: np, I am double checking now though :) | 17:47 |
| === shardy is now known as shardy_afk | ||
| powersj | smoser: ugh I'm wrong, there are ec2 snapshots listed. I was looking at wrong region (didn't change my web browser to match my config) | 17:55 |
| powersj | there is a snap-03ac... left over | 17:56 |
| powersj | smoser: thanks for comments I have a clean up for the snap done and will make your last change | 18:26 |
| powersj | blackboxsw: smoser: final commit with Scott's fixes to removing the snapshot + variable naming: https://git.launchpad.net/~powersj/cloud-init/commit/?id=14beefdcc7d848f0b5852a797753f4758ff647e1 | 18:41 |
| blackboxsw | sorry powersj was in a hole on lanscape stuff for a bit. looking now | 18:43 |
| blackboxsw | and reading backlog | 18:43 |
| powersj | blackboxsw: no worries smoser was fixing my invalid ways | 18:44 |
| smoser | i think seems good. | 18:45 |
| blackboxsw | spare the rod, spoil the child | 18:45 |
| smoser | at least good enough. | 18:45 |
| smoser | we do not support instance store right now | 18:45 |
| smoser | should probably doc that | 18:45 |
| smoser | as ' | 18:45 |
| smoser | 'create_image' doesnt work on instance store isntance | 18:45 |
| smoser | and our Destroy of the ami will fail for such a thing too | 18:46 |
| smoser | powersj: do we shut down the booted system that we collect ? | 18:46 |
| smoser | i was thinking that we should do that. | 18:46 |
| smoser | i had this in my buffer | 18:47 |
| smoser | http://paste.ubuntu.com/26327283/ | 18:47 |
| powersj | smoser: we terminate during destroy | 18:47 |
| smoser | right | 18:47 |
| smoser | but we dont shut the instance down | 18:47 |
| smoser | ie, '/sbin/poweroff' | 18:47 |
| blackboxsw | powersj: do we need to loop through snapshotids per + snapshot_id = image.block_device_mappings[0]['Ebs']['SnapshotId'] | 18:47 |
| smoser | if we did that, then we could collect the log *after* doing so | 18:47 |
| smoser | and we'd have a mnore complete log on every platform i think | 18:48 |
| blackboxsw | or are we saying that we only really support one ebs snapshot which we will ultimately destroy during deregister | 18:48 |
| * blackboxsw adds a pdb in there and tries to look at what we get from ec2 | 18:49 | |
| smoser | blackboxsw: well, the block device mapping is something like: | 18:50 |
| powersj | blackboxsw: for now we could make that assumption, if we starting adding disks then no | 18:50 |
| smoser | {'root-disk': 'snap-XXXXXXX', 'disk1': 'snap-XXXXXXX', 'disk2'....} | 18:51 |
| smoser | the snapshots we create with create_ami will most probaly have the snap-XXX at as the '0' eleement | 18:51 |
| smoser | in the block device mapping | 18:51 |
| smoser | ie, the root device | 18:51 |
| blackboxsw | yeah I was just wondering about whether it was worth a snapshot_ids = set([dev_map.get('Ebs', {}).get('SnapshotId') for dev_map in image.block_device_mappings]) | 18:54 |
| blackboxsw | but doesn't matter for this use case | 18:54 |
| powersj | smoser: so attempt the console log capture twice, and console_log_final would return the existing console data on platforms like lxd and nocloud-kvm? | 18:54 |
| blackboxsw | and maybe my approach is wrong anyway | 18:55 |
| smoser | powersj: well that was what my paste did, yeah. but it is silly. | 18:59 |
| smoser | blackboxsw: i think you probalby would not want to do that. | 18:59 |
| smoser | hm.. maybe. i dont know. | 18:59 |
| blackboxsw | no worries, we can dig into it if we actually get a use case | 19:00 |
| powersj | blackboxsw: this I got everything from yesterday https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/335774 | 19:51 |
| powersj | smoser: any further thoughts? or are you trying to get the console log going? | 19:51 |
| smoser | i was atrying to get console log going | 19:51 |
| powersj | :) | 19:51 |
| smoser | and now noticedc that with recent (bionic) lxc it is broken | 19:51 |
| smoser | :-( | 19:52 |
| smoser | so ... can't test that way :) | 19:52 |
| blackboxsw | powersj: looks good, I just noticed that epel repo on centos doesn't contain awscli package "yet/ever?" so we'll have an issue there as far as dependencies, but generally the renames are good for the rest | 19:59 |
| blackboxsw | s/centos/centos 6/ | 19:59 |
| blackboxsw | centos7 has awscli | 19:59 |
| powersj | blackboxsw: see my comment about bzr+lp? | 19:59 |
| blackboxsw | yeah I liked it. the simplestreams comment? | 20:00 |
| powersj | well if you run read-dependencies on that it tries to install bzr+lp:simplestreams | 20:00 |
| powersj | so should I do a) a rename to specify python[3]-simplestreams or b) add something to read-dependencies to filter out the 'bzr+lp:' prefix? | 20:01 |
| blackboxsw | ohh you mean per read-dependencies behavior. no I didn't see that comment. hrm | 20:02 |
| blackboxsw | hrm, might have a new rename rule. lemme think during lunch | 20:03 |
| blackboxsw | we have a couple other lp cases right? | 20:03 |
| powersj | hmm I thought this was the only one | 20:03 |
| blackboxsw | hmm maybe not | 20:03 |
| blackboxsw | yeah I thought we were doing something ahh, it was just simplestreams in tox.ini | 20:04 |
| blackboxsw | ok that was the only case, and you moved it. | 20:04 |
| blackboxsw | hmm ok will toy with a rename rule for bzr+lp | 20:05 |
| blackboxsw | powersj: WDYT? I like calling out the needs | 20:21 |
| blackboxsw | oops | 20:21 |
| blackboxsw | http://paste.ubuntu.com/26327747/ rather | 20:21 |
| blackboxsw | add the ability for pip dependency exclusions if system packages aren't available | 20:22 |
| blackboxsw | at least we'd be noisy on unsupported platforms | 20:22 |
| blackboxsw | ok really grabbing lunch | 20:22 |
| powersj | blackboxsw: well if simplestreams isn't even available then I question why we should even add integration requirements to the read-depens file | 20:25 |
| powersj | if you don't have simplestreams forget running integration tests, yes you could run lxd backend, but that would be it | 20:25 |
| powersj | from a rhel user who is just hacking on cloud-init and wants to run tests, this would also install aws-cli which is pointless | 20:28 |
| blackboxsw | true powersj, so simplestreams system package is avail on ubuntu... but nothing else. so we could just do something like the following: http://paste.ubuntu.com/26327790/ | 20:30 |
| blackboxsw | then no need to worry about integration deps for non ubuntu | 20:30 |
| powersj | agreed | 20:31 |
| powersj | so that last pastebin + the rename for simplestreams for "debian" should do it | 20:31 |
| blackboxsw | we'd still need the "renames" : { | 20:31 |
| blackboxsw | + "bzr+lp:simplestreams": { | 20:31 |
| blackboxsw | + "2": "python-simplestreams", | 20:31 |
| blackboxsw | + "3": "python3-simplestreams" | 20:31 |
| blackboxsw | + }, | 20:31 |
| powersj | yeah | 20:31 |
| blackboxsw | just for 'debian' in pkg-deps.json | 20:31 |
| powersj | ok I'll get that worked up in a bit | 20:31 |
| blackboxsw | thanks sir | 20:31 |
| powersj | thank you | 20:31 |
| smoser | blackboxsw: shoudl the package deps have been even referening simplestreams ? | 20:36 |
| blackboxsw | hrm .... package itself shouldn't... ohh hold the phone... right, forgot we generate all test deps into package control file deps. | 20:37 |
| blackboxsw | right will work up a tiny tweak ... make ci-deps-ubuntu should install those system packages for integration dependencies, but it shouldn't be included when generating the debian/control file. one min | 20:38 |
| blackboxsw | smoser I *think* that is safe... packages/bddeb calls specifically 'read-dependencies', | 20:39 |
| blackboxsw | ['--requirements-file', 'test-requirements.txt', | 20:39 |
| blackboxsw | so it doesn't call see integration-requirements.txt | 20:39 |
| * blackboxsw runs bddep and checks the pkg created | 20:40 | |
| smoser | so what path was exposing the need for a change ? | 20:42 |
| blackboxsw | http://paste.ubuntu.com/26327885/ yeah build-deps don't get integration test dependencies | 20:52 |
| blackboxsw | smoser: if we want to continue to support installing system package installation setup for ci via 'make ci-deps-ubuntu' I'd like to have us also install the integration test dependencies via this mechanism | 20:53 |
| smoser | well, that makes sense. but some things i think will not be resolveable that way. | 20:53 |
| blackboxsw | make ci-deps-ubuntu calls ./tools/read-dependencies -d ubuntu -v 3 --test-distro to install system packages instead of python packages | 20:54 |
| smoser | such as softlayer python library | 20:54 |
| smoser | or azure library | 20:54 |
| blackboxsw | right, there are integration python package requirements that won't fit this mold.... and cases like powersj mentioned for awscli | 20:56 |
| smoser | powersj: http://paste.ubuntu.com/26327905/ | 20:56 |
| blackboxsw | where it's snapped, so we have a different vehicle for the installation of those deps that would need to be sorted | 20:56 |
| smoser | that works for me in nocloud | 20:56 |
| smoser | i'd be interested in if it works on ec2 | 20:56 |
| smoser | and i guess i could find a xenial system to test lxd | 20:57 |
| blackboxsw | but I feel like that step will need to be grown when we actually include azure or softlayer integration support | 20:57 |
| smoser | hm. | 20:57 |
| smoser | maybe | 20:57 |
| blackboxsw | as it is currently, the python pip vs. system-package logic 'works' for our current integration needs and doesn't preclude the option of growing an option to install a non-system-package dependency like azure's or softlayer's python packages | 20:58 |
| blackboxsw | it just doesn't support that level of granularity yet | 20:58 |
| powersj | smoser: I'll try ec2 shortly, be aware our test systems use lxd from ppa so it will be on par with bionic | 20:59 |
| blackboxsw | it does beg something slightly smarter than a simple translation from pip-pkg-name -> system-package-name which is what we currently do wholesale | 20:59 |
| powersj | Which reminds me we need to move to the snap | 20:59 |
| smoser | powersj: well, it works in that it doesn tfail | 21:00 |
| smoser | it just doenst get a console log | 21:00 |
| smoser | stgraber proposed fix for lxd at https://github.com/lxc/lxd/pull/4140 | 21:01 |
| powersj | smoser: do you want this change as a part of ec2 or a follow on? | 21:01 |
| smoser | i was just going to ask. | 21:02 |
| smoser | i can either propose it as stand alone | 21:02 |
| smoser | or as part of yours. | 21:02 |
| smoser | powersj: i think pull it into yours | 21:03 |
| powersj | smoser: ok I'll pull and test a bit, that means hopefully merge monday? ;) | 21:04 |
| smoser | powersj: yeah. | 21:08 |
| powersj | \o/ an ec2 console log | 21:45 |
| smoser | woot! | 21:47 |
| powersj | now I just need my coffee shop connection to get a boost to download an image for nocloud | 21:50 |
| powersj | pylxd doesn't like the snap lxd | 22:12 |
| powersj | smoser: do you want me to squash all my commits? | 22:21 |
| powersj | otherwise once this lxd test is done I think we are good to go | 22:21 |
| powersj | smoser: pushed a version I confirmed working console log with lxd, nocloud-kvm, and ec2 | 22:30 |
| powersj | I'm calling it done :D | 22:30 |
| powersj | thanks for the help | 22:30 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!