/srv/irclogs.ubuntu.com/2018/01/05/#cloud-init.txt

redguyhi, I am restarting systemd-journald as a part of my cloud-init final scripts and this seems to break the execution of cloud-final.service on Debian jessie (exit status 1). Is this known and is there some workaround I could use?13:29
redguyor rephrased: what is the cannonical way of managing journald config with cloud-init ?14:57
redguys/cannonical/recommended/14:58
smoserredguy: hm..15:06
smoserthats not somethign i've done or considered.15:06
smoserhow is 'break execution' ?15:07
smoseras in it doesnt run ?15:07
redguyas in the execution stops15:11
redguytrying to do it via runcmd now, but I think this will also break15:11
smoserthat seems bad.15:12
redguyhttps://pastebin.com/0m8QVEZq15:12
smoserbad that restarting systemd-journald would stop a service15:12
redguyyeah, possibly this is a bug in systemd :-)15:12
smoser can you get /var/log/cloud-init-output.log and /var/log/cloud-init.log ?15:13
smoserand fyi, https://hastebin.com/ doesn't hurt your eyes to look at :)15:13
redguyoh, nice15:14
redguywill get logs in couple of secs, but this time with runcmd approach15:14
redguyhmm, runcmd didn't fail, but it seems that it it wasn't executed at all (as in the mkdir in there didn't create the directory I wanted it to create). This most likely this is a config problem on my part. Going back to the pervious setup with restarting systemd-journald in a shell script15:22
smosersystemd doesn't do as well as upstart did with restarting things otherwise being "dynamic" during boot15:24
smoserwe had to move the installation of packages to final rather than cloud-config stage because of that15:25
redguyhttps://hastebin.com/fanahigiwo.php15:33
redguyso systemd says status 1, but the cloud-init.log says 14115:33
redguywhich is EPIPE afair15:33
redguyno, a result of SIGPIPE15:34
redguyso my current theory is that this is caused by syslog log handler: cloud-init has /dev/log socket open, journald is stopped and closes the socket, cloud-init breaks15:49
redguycan I override log_cfgs with user_data ?16:21
smoserredguy: i think that paste didnt work ?16:47
smoserredguy: i think you should be able to write to log configs.16:48
smoserbut your disgnosis could be wrong i think16:48
smoserno.. system can't be that broken16:48
smoseri was thinking that it might be just SIGPIPE issue because journal is catching cloud-init's output16:49
smoserhm.16:49
redguysomehow hastebin doesn't work for me :/16:58
redguyhttps://pastebin.com/5ev0Z7La16:58
redguyhttps://pastebin.com/raw/5ev0Z7La ? :-)16:59
powersjblackboxsw: smoser: https://git.launchpad.net/~powersj/cloud-init/commit/?id=3793d3d24500d5f9a6ee03cd28bd9f3e6182554c17:23
smoserpowersj: just loking at the diff i tlooks like ou changed the param name in the doc string17:27
smoserbut not in the usage17:27
powersjoh shoot17:28
smoseri dont think it needs _ base on it.17:28
smoserits the only 'image' thing in that class.17:28
powersjuntil the snapshot function17:28
powersjbut yes only class var17:28
powersjI can revert it back to just image_ami17:28
smoserpowersj: hwere does the snapshot get removed ?17:31
smoserthe "Amazon EC2 Snapshot"17:31
smoserthat backs the ami17:31
powersjthe backing instance is removed still in image.py (that was already there from last night)17:32
smoserno..17:32
smoserhm,..17:32
powersjlol17:33
smoserok. so 'create_image' does17:35
smoser a.) stop instance17:35
smoser b.) snapshot volume17:35
smoser   (which creates a snap-XXXXXX thing)17:35
smoser c.) register an ami with snap-XXXXX as its root volume. (creating a ami-XXXXX thing)17:36
smoserthen in CII-EC2Snapshot.destroy, you deregister the image17:37
smoserwhicih deletes form AWS the ami-XXXX thing17:37
powersjyes17:37
smoserbut does the snap-XXXXX thing get removed ?17:37
powersjto my knowledge of how the deregistering of an AMI works yes, looking at the console there are no longer anything listed under images or snapshots17:40
smoserinteresting.17:42
smoserthat does make snse, and much nicer than aking the user clean up the snapshot themselve.17:42
smoserwhich i thought you'd have to do.17:42
smoseroh... ik now why.17:42
smoserin ubuntu publishing, we would register the snapshot and then create an ami from that snapshot separately17:43
smoserso that we could register multiple ami from the same snapshot17:43
smoser(one as 'released' and one as 'daily')17:43
smoserso... good. thank you.17:43
powersjsmoser: np, I am double checking now though :)17:47
=== shardy is now known as shardy_afk
powersjsmoser: ugh I'm wrong, there are ec2 snapshots listed. I was looking at wrong region (didn't change my web browser to match my config)17:55
powersjthere is a snap-03ac... left over17:56
powersjsmoser: thanks for comments I have a clean up for the snap done and will make your last change18:26
powersjblackboxsw: smoser: final commit with Scott's fixes to removing the snapshot + variable naming: https://git.launchpad.net/~powersj/cloud-init/commit/?id=14beefdcc7d848f0b5852a797753f4758ff647e118:41
blackboxswsorry powersj was in a hole on lanscape stuff for a bit. looking now18:43
blackboxswand reading backlog18:43
powersjblackboxsw: no worries smoser was fixing my invalid ways18:44
smoseri think seems good.18:45
blackboxswspare the rod, spoil the child18:45
smoserat least good enough.18:45
smoserwe do not support instance store right now18:45
smosershould probably doc that18:45
smoseras '18:45
smoser'create_image' doesnt work on instance store isntance18:45
smoserand our Destroy of the ami will fail for such a thing too18:46
smoserpowersj: do we shut down the booted system that we collect ?18:46
smoseri was thinking that we should do that.18:46
smoseri had this in my buffer18:47
smoser http://paste.ubuntu.com/26327283/18:47
powersjsmoser: we terminate during destroy18:47
smoserright18:47
smoserbut we dont shut the instance down18:47
smoserie, '/sbin/poweroff'18:47
blackboxswpowersj: do we need to loop through snapshotids per +            snapshot_id = image.block_device_mappings[0]['Ebs']['SnapshotId']18:47
smoserif we did that, then we could collect the log *after* doing so18:47
smoserand we'd have a mnore complete log on every platform i think18:48
blackboxswor are we saying that we only really support one ebs snapshot which we will ultimately destroy during deregister18:48
* blackboxsw adds a pdb in there and tries to look at what we get from ec218:49
smoserblackboxsw: well, the block device mapping is something like:18:50
powersjblackboxsw: for now we could make that assumption, if we starting adding disks then no18:50
smoser {'root-disk': 'snap-XXXXXXX', 'disk1': 'snap-XXXXXXX', 'disk2'....}18:51
smoserthe snapshots we create with create_ami will most probaly have the snap-XXX at as the '0' eleement18:51
smoserin the block device mapping18:51
smoserie, the root device18:51
blackboxswyeah I was just wondering about whether it was worth a snapshot_ids = set([dev_map.get('Ebs', {}).get('SnapshotId') for dev_map in  image.block_device_mappings])18:54
blackboxswbut doesn't matter for this use case18:54
powersjsmoser: so attempt the console log capture twice, and console_log_final would return the existing console data on platforms like lxd and nocloud-kvm?18:54
blackboxswand maybe my approach is wrong anyway18:55
smoserpowersj: well that was what my paste did, yeah. but it is silly.18:59
smoserblackboxsw: i think you probalby would not want to do that.18:59
smoserhm.. maybe. i dont know.18:59
blackboxswno worries, we can dig into it if we actually get a use case19:00
powersjblackboxsw: this I got everything from yesterday https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/33577419:51
powersjsmoser: any further thoughts? or are you trying to get the console log going?19:51
smoseri  was atrying to get console log going19:51
powersj:)19:51
smoserand now noticedc that with recent (bionic) lxc it is broken19:51
smoser:-(19:52
smoserso ... can't test that way :)19:52
blackboxswpowersj: looks good, I just noticed that epel repo on centos doesn't contain awscli package "yet/ever?" so we'll have an issue there as far as dependencies, but generally the renames are good for the rest19:59
blackboxsws/centos/centos 6/19:59
blackboxswcentos7 has awscli19:59
powersjblackboxsw: see my comment about bzr+lp?19:59
blackboxswyeah I liked it.  the simplestreams comment?20:00
powersjwell if you run read-dependencies on that it tries to install bzr+lp:simplestreams20:00
powersjso should I do a) a rename to specify python[3]-simplestreams or b) add something to read-dependencies to filter out the 'bzr+lp:' prefix?20:01
blackboxswohh you mean per read-dependencies behavior. no I didn't see that comment. hrm20:02
blackboxswhrm, might have a new rename rule. lemme think during lunch20:03
blackboxswwe have a couple other lp cases right?20:03
powersjhmm I thought this was the only one20:03
blackboxswhmm maybe not20:03
blackboxswyeah I thought we were doing something ahh, it was just simplestreams in tox.ini20:04
blackboxswok that was the only case, and you moved it.20:04
blackboxswhmm ok will toy with a rename rule for bzr+lp20:05
blackboxswpowersj: WDYT? I like calling out the needs20:21
blackboxswoops20:21
blackboxswhttp://paste.ubuntu.com/26327747/   rather20:21
blackboxswadd the ability for pip dependency exclusions if system packages aren't available20:22
blackboxswat least we'd be noisy on unsupported platforms20:22
blackboxswok really grabbing lunch20:22
powersjblackboxsw: well if simplestreams isn't even available then I question why we should even add integration requirements to the read-depens file20:25
powersjif you don't have simplestreams forget running integration tests, yes you could run lxd backend, but that would be it20:25
powersjfrom a rhel user who is just hacking on cloud-init and wants to run tests, this would also install aws-cli which is pointless20:28
blackboxswtrue powersj, so simplestreams system package is avail on ubuntu... but nothing else. so we could just do something like the following:  http://paste.ubuntu.com/26327790/20:30
blackboxswthen no need to worry about integration deps for non ubuntu20:30
powersjagreed20:31
powersjso that last pastebin + the rename for simplestreams for "debian" should do it20:31
blackboxswwe'd still need the        "renames" : {20:31
blackboxsw+         "bzr+lp:simplestreams": {20:31
blackboxsw+             "2": "python-simplestreams",20:31
blackboxsw+             "3": "python3-simplestreams"20:31
blackboxsw+        },20:31
powersjyeah20:31
blackboxswjust for 'debian' in pkg-deps.json20:31
powersjok I'll get that worked up in a bit20:31
blackboxswthanks sir20:31
powersjthank you20:31
smoserblackboxsw: shoudl the package deps have been even referening simplestreams ?20:36
blackboxswhrm .... package itself shouldn't... ohh hold the phone... right, forgot we generate all test deps into package control file deps.20:37
blackboxswright will work up a tiny tweak ... make ci-deps-ubuntu should install those system packages for integration dependencies, but it shouldn't be included when generating the debian/control file. one min20:38
blackboxswsmoser I *think* that is safe... packages/bddeb calls specifically       'read-dependencies',20:39
blackboxsw        ['--requirements-file', 'test-requirements.txt',20:39
blackboxswso it doesn't call see integration-requirements.txt20:39
* blackboxsw runs bddep and checks the pkg created20:40
smoserso what path was exposing the need for a change ?20:42
blackboxswhttp://paste.ubuntu.com/26327885/ yeah build-deps don't get integration test dependencies20:52
blackboxswsmoser: if we want to continue to support installing system package installation setup for ci via 'make ci-deps-ubuntu' I'd like to have us also install the integration test dependencies via this mechanism20:53
smoserwell, that makes sense. but some things i think will not be resolveable that way.20:53
blackboxswmake ci-deps-ubuntu calls  ./tools/read-dependencies -d ubuntu -v 3  --test-distro to install system packages instead of python packages20:54
smosersuch as softlayer python library20:54
smoseror azure library20:54
blackboxswright, there are integration python package requirements  that won't fit this mold.... and cases like powersj mentioned for awscli20:56
smoserpowersj: http://paste.ubuntu.com/26327905/20:56
blackboxswwhere it's snapped, so we have a different vehicle for the installation of those deps that would need to be sorted20:56
smoserthat works for me in nocloud20:56
smoseri'd be interested in if it works on ec220:56
smoserand i guess i could find a xenial system to test lxd20:57
blackboxswbut I feel like that step will need to be grown when we actually include azure or softlayer integration support20:57
smoserhm.20:57
smosermaybe20:57
blackboxswas it is currently, the python pip vs. system-package logic 'works' for our current integration needs and doesn't preclude the option of growing an option to install a non-system-package dependency like azure's or softlayer's  python packages20:58
blackboxswit just doesn't support that level of granularity yet20:58
powersjsmoser: I'll try ec2 shortly, be aware our test systems use lxd from ppa so it will be on par with bionic20:59
blackboxswit does beg something slightly smarter than a simple translation from pip-pkg-name -> system-package-name which is what we currently do wholesale20:59
powersjWhich reminds me we need to move to the snap20:59
smoserpowersj: well, it works in that it doesn tfail21:00
smoserit just doenst get a console log21:00
smoserstgraber proposed fix for lxd at https://github.com/lxc/lxd/pull/414021:01
powersjsmoser: do you want this change as a part of ec2 or a follow on?21:01
smoseri was just going to ask.21:02
smoseri can either propose it as stand alone21:02
smoseror as part of yours.21:02
smoserpowersj: i think pull it into yours21:03
powersjsmoser: ok I'll pull and test a bit, that means hopefully merge monday? ;)21:04
smoserpowersj: yeah.21:08
powersj\o/ an ec2 console log21:45
smoserwoot!21:47
powersjnow I just need my coffee shop connection to get a boost to download an image for nocloud21:50
powersjpylxd doesn't like the snap lxd22:12
powersjsmoser: do you want me to squash all my commits?22:21
powersjotherwise once this lxd test is done I think we are good to go22:21
powersjsmoser: pushed a version I confirmed working console log with lxd, nocloud-kvm, and ec222:30
powersjI'm calling it done :D22:30
powersjthanks for the help22:30

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!