[13:29] <redguy> hi, I am restarting systemd-journald as a part of my cloud-init final scripts and this seems to break the execution of cloud-final.service on Debian jessie (exit status 1). Is this known and is there some workaround I could use?
[14:57] <redguy> or rephrased: what is the cannonical way of managing journald config with cloud-init ?
[14:58] <redguy> s/cannonical/recommended/
[15:06] <smoser> redguy: hm..
[15:06] <smoser> thats not somethign i've done or considered.
[15:07] <smoser> how is 'break execution' ?
[15:07] <smoser> as in it doesnt run ?
[15:11] <redguy> as in the execution stops
[15:11] <redguy> trying to do it via runcmd now, but I think this will also break
[15:12] <smoser> that seems bad.
[15:12] <redguy> https://pastebin.com/0m8QVEZq
[15:12] <smoser> bad that restarting systemd-journald would stop a service
[15:12] <redguy> yeah, possibly this is a bug in systemd :-)
[15:13] <smoser>  can you get /var/log/cloud-init-output.log and /var/log/cloud-init.log ?
[15:13] <smoser> and fyi, https://hastebin.com/ doesn't hurt your eyes to look at :)
[15:14] <redguy> oh, nice
[15:14] <redguy> will get logs in couple of secs, but this time with runcmd approach
[15:22] <redguy> hmm, runcmd didn't fail, but it seems that it it wasn't executed at all (as in the mkdir in there didn't create the directory I wanted it to create). This most likely this is a config problem on my part. Going back to the pervious setup with restarting systemd-journald in a shell script
[15:24] <smoser> systemd doesn't do as well as upstart did with restarting things otherwise being "dynamic" during boot
[15:25] <smoser> we had to move the installation of packages to final rather than cloud-config stage because of that
[15:33] <redguy> https://hastebin.com/fanahigiwo.php
[15:33] <redguy> so systemd says status 1, but the cloud-init.log says 141
[15:33] <redguy> which is EPIPE afair
[15:34] <redguy> no, a result of SIGPIPE
[15:49] <redguy> so my current theory is that this is caused by syslog log handler: cloud-init has /dev/log socket open, journald is stopped and closes the socket, cloud-init breaks
[16:21] <redguy> can I override log_cfgs with user_data ?
[16:47] <smoser> redguy: i think that paste didnt work ?
[16:48] <smoser> redguy: i think you should be able to write to log configs.
[16:48] <smoser> but your disgnosis could be wrong i think
[16:48] <smoser> no.. system can't be that broken
[16:49] <smoser> i was thinking that it might be just SIGPIPE issue because journal is catching cloud-init's output
[16:49] <smoser> hm.
[16:58] <redguy> somehow hastebin doesn't work for me :/
[16:58] <redguy> https://pastebin.com/5ev0Z7La
[16:59] <redguy> https://pastebin.com/raw/5ev0Z7La ? :-)
[17:23] <powersj> blackboxsw: smoser: https://git.launchpad.net/~powersj/cloud-init/commit/?id=3793d3d24500d5f9a6ee03cd28bd9f3e6182554c
[17:27] <smoser> powersj: just loking at the diff i tlooks like ou changed the param name in the doc string
[17:27] <smoser> but not in the usage
[17:28] <powersj> oh shoot
[17:28] <smoser> i dont think it needs _ base on it.
[17:28] <smoser> its the only 'image' thing in that class.
[17:28] <powersj> until the snapshot function
[17:28] <powersj> but yes only class var
[17:28] <powersj> I can revert it back to just image_ami
[17:31] <smoser> powersj: hwere does the snapshot get removed ?
[17:31] <smoser> the "Amazon EC2 Snapshot"
[17:31] <smoser> that backs the ami
[17:32] <powersj> the backing instance is removed still in image.py (that was already there from last night)
[17:32] <smoser> no..
[17:32] <smoser> hm,..
[17:33] <powersj> lol
[17:35] <smoser> ok. so 'create_image' does
[17:35] <smoser>  a.) stop instance
[17:35] <smoser>  b.) snapshot volume
[17:35] <smoser>    (which creates a snap-XXXXXX thing)
[17:36] <smoser>  c.) register an ami with snap-XXXXX as its root volume. (creating a ami-XXXXX thing)
[17:37] <smoser> then in CII-EC2Snapshot.destroy, you deregister the image
[17:37] <smoser> whicih deletes form AWS the ami-XXXX thing
[17:37] <powersj> yes
[17:37] <smoser> but does the snap-XXXXX thing get removed ?
[17:40] <powersj> to my knowledge of how the deregistering of an AMI works yes, looking at the console there are no longer anything listed under images or snapshots
[17:42] <smoser> interesting.
[17:42] <smoser> that does make snse, and much nicer than aking the user clean up the snapshot themselve.
[17:42] <smoser> which i thought you'd have to do.
[17:42] <smoser> oh... ik now why.
[17:43] <smoser> in ubuntu publishing, we would register the snapshot and then create an ami from that snapshot separately
[17:43] <smoser> so that we could register multiple ami from the same snapshot
[17:43] <smoser> (one as 'released' and one as 'daily')
[17:43] <smoser> so... good. thank you.
[17:47] <powersj> smoser: np, I am double checking now though :)
=== shardy is now known as shardy_afk
[17:55] <powersj> smoser: ugh I'm wrong, there are ec2 snapshots listed. I was looking at wrong region (didn't change my web browser to match my config)
[17:56] <powersj> there is a snap-03ac... left over
[18:26] <powersj> smoser: thanks for comments I have a clean up for the snap done and will make your last change
[18:41] <powersj> blackboxsw: smoser: final commit with Scott's fixes to removing the snapshot + variable naming: https://git.launchpad.net/~powersj/cloud-init/commit/?id=14beefdcc7d848f0b5852a797753f4758ff647e1
[18:43] <blackboxsw> sorry powersj was in a hole on lanscape stuff for a bit. looking now
[18:43] <blackboxsw> and reading backlog
[18:44] <powersj> blackboxsw: no worries smoser was fixing my invalid ways
[18:45] <smoser> i think seems good.
[18:45] <blackboxsw> spare the rod, spoil the child
[18:45] <smoser> at least good enough.
[18:45] <smoser> we do not support instance store right now
[18:45] <smoser> should probably doc that
[18:45] <smoser> as '
[18:45] <smoser> 'create_image' doesnt work on instance store isntance
[18:46] <smoser> and our Destroy of the ami will fail for such a thing too
[18:46] <smoser> powersj: do we shut down the booted system that we collect ?
[18:46] <smoser> i was thinking that we should do that.
[18:47] <smoser> i had this in my buffer
[18:47] <smoser>  http://paste.ubuntu.com/26327283/
[18:47] <powersj> smoser: we terminate during destroy
[18:47] <smoser> right
[18:47] <smoser> but we dont shut the instance down
[18:47] <smoser> ie, '/sbin/poweroff'
[18:47] <blackboxsw> powersj: do we need to loop through snapshotids per +            snapshot_id = image.block_device_mappings[0]['Ebs']['SnapshotId']
[18:47] <smoser> if we did that, then we could collect the log *after* doing so
[18:48] <smoser> and we'd have a mnore complete log on every platform i think
[18:48] <blackboxsw> or are we saying that we only really support one ebs snapshot which we will ultimately destroy during deregister
[18:49]  * blackboxsw adds a pdb in there and tries to look at what we get from ec2
[18:50] <smoser> blackboxsw: well, the block device mapping is something like:
[18:50] <powersj> blackboxsw: for now we could make that assumption, if we starting adding disks then no
[18:51] <smoser>  {'root-disk': 'snap-XXXXXXX', 'disk1': 'snap-XXXXXXX', 'disk2'....}
[18:51] <smoser> the snapshots we create with create_ami will most probaly have the snap-XXX at as the '0' eleement
[18:51] <smoser> in the block device mapping
[18:51] <smoser> ie, the root device
[18:54] <blackboxsw> yeah I was just wondering about whether it was worth a snapshot_ids = set([dev_map.get('Ebs', {}).get('SnapshotId') for dev_map in  image.block_device_mappings])
[18:54] <blackboxsw> but doesn't matter for this use case
[18:54] <powersj> smoser: so attempt the console log capture twice, and console_log_final would return the existing console data on platforms like lxd and nocloud-kvm?
[18:55] <blackboxsw> and maybe my approach is wrong anyway
[18:59] <smoser> powersj: well that was what my paste did, yeah. but it is silly.
[18:59] <smoser> blackboxsw: i think you probalby would not want to do that.
[18:59] <smoser> hm.. maybe. i dont know.
[19:00] <blackboxsw> no worries, we can dig into it if we actually get a use case
[19:51] <powersj> blackboxsw: this I got everything from yesterday https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/335774
[19:51] <powersj> smoser: any further thoughts? or are you trying to get the console log going?
[19:51] <smoser> i  was atrying to get console log going
[19:51] <powersj> :)
[19:51] <smoser> and now noticedc that with recent (bionic) lxc it is broken
[19:52] <smoser> :-(
[19:52] <smoser> so ... can't test that way :)
[19:59] <blackboxsw> powersj: looks good, I just noticed that epel repo on centos doesn't contain awscli package "yet/ever?" so we'll have an issue there as far as dependencies, but generally the renames are good for the rest
[19:59] <blackboxsw> s/centos/centos 6/
[19:59] <blackboxsw> centos7 has awscli
[19:59] <powersj> blackboxsw: see my comment about bzr+lp?
[20:00] <blackboxsw> yeah I liked it.  the simplestreams comment?
[20:00] <powersj> well if you run read-dependencies on that it tries to install bzr+lp:simplestreams
[20:01] <powersj> so should I do a) a rename to specify python[3]-simplestreams or b) add something to read-dependencies to filter out the 'bzr+lp:' prefix?
[20:02] <blackboxsw> ohh you mean per read-dependencies behavior. no I didn't see that comment. hrm
[20:03] <blackboxsw> hrm, might have a new rename rule. lemme think during lunch
[20:03] <blackboxsw> we have a couple other lp cases right?
[20:03] <powersj> hmm I thought this was the only one
[20:03] <blackboxsw> hmm maybe not
[20:04] <blackboxsw> yeah I thought we were doing something ahh, it was just simplestreams in tox.ini
[20:04] <blackboxsw> ok that was the only case, and you moved it.
[20:05] <blackboxsw> hmm ok will toy with a rename rule for bzr+lp
[20:21] <blackboxsw> powersj: WDYT? I like calling out the needs
[20:21] <blackboxsw> oops
[20:21] <blackboxsw> http://paste.ubuntu.com/26327747/   rather
[20:22] <blackboxsw> add the ability for pip dependency exclusions if system packages aren't available
[20:22] <blackboxsw> at least we'd be noisy on unsupported platforms
[20:22] <blackboxsw> ok really grabbing lunch
[20:25] <powersj> blackboxsw: well if simplestreams isn't even available then I question why we should even add integration requirements to the read-depens file
[20:25] <powersj> if you don't have simplestreams forget running integration tests, yes you could run lxd backend, but that would be it
[20:28] <powersj> from a rhel user who is just hacking on cloud-init and wants to run tests, this would also install aws-cli which is pointless
[20:30] <blackboxsw> true powersj, so simplestreams system package is avail on ubuntu... but nothing else. so we could just do something like the following:  http://paste.ubuntu.com/26327790/
[20:30] <blackboxsw> then no need to worry about integration deps for non ubuntu
[20:31] <powersj> agreed
[20:31] <powersj> so that last pastebin + the rename for simplestreams for "debian" should do it
[20:31] <blackboxsw> we'd still need the        "renames" : {
[20:31] <blackboxsw> +         "bzr+lp:simplestreams": {
[20:31] <blackboxsw> +             "2": "python-simplestreams",
[20:31] <blackboxsw> +             "3": "python3-simplestreams"
[20:31] <blackboxsw> +        },
[20:31] <powersj> yeah
[20:31] <blackboxsw> just for 'debian' in pkg-deps.json
[20:31] <powersj> ok I'll get that worked up in a bit
[20:31] <blackboxsw> thanks sir
[20:31] <powersj> thank you
[20:36] <smoser> blackboxsw: shoudl the package deps have been even referening simplestreams ?
[20:37] <blackboxsw> hrm .... package itself shouldn't... ohh hold the phone... right, forgot we generate all test deps into package control file deps.
[20:38] <blackboxsw> right will work up a tiny tweak ... make ci-deps-ubuntu should install those system packages for integration dependencies, but it shouldn't be included when generating the debian/control file. one min
[20:39] <blackboxsw> smoser I *think* that is safe... packages/bddeb calls specifically       'read-dependencies',
[20:39] <blackboxsw>         ['--requirements-file', 'test-requirements.txt',
[20:39] <blackboxsw> so it doesn't call see integration-requirements.txt
[20:40]  * blackboxsw runs bddep and checks the pkg created
[20:42] <smoser> so what path was exposing the need for a change ?
[20:52] <blackboxsw> http://paste.ubuntu.com/26327885/ yeah build-deps don't get integration test dependencies
[20:53] <blackboxsw> smoser: if we want to continue to support installing system package installation setup for ci via 'make ci-deps-ubuntu' I'd like to have us also install the integration test dependencies via this mechanism
[20:53] <smoser> well, that makes sense. but some things i think will not be resolveable that way.
[20:54] <blackboxsw> make ci-deps-ubuntu calls  ./tools/read-dependencies -d ubuntu -v 3  --test-distro to install system packages instead of python packages
[20:54] <smoser> such as softlayer python library
[20:54] <smoser> or azure library
[20:56] <blackboxsw> right, there are integration python package requirements  that won't fit this mold.... and cases like powersj mentioned for awscli
[20:56] <smoser> powersj: http://paste.ubuntu.com/26327905/
[20:56] <blackboxsw> where it's snapped, so we have a different vehicle for the installation of those deps that would need to be sorted
[20:56] <smoser> that works for me in nocloud
[20:56] <smoser> i'd be interested in if it works on ec2
[20:57] <smoser> and i guess i could find a xenial system to test lxd
[20:57] <blackboxsw> but I feel like that step will need to be grown when we actually include azure or softlayer integration support
[20:57] <smoser> hm.
[20:57] <smoser> maybe
[20:58] <blackboxsw> as it is currently, the python pip vs. system-package logic 'works' for our current integration needs and doesn't preclude the option of growing an option to install a non-system-package dependency like azure's or softlayer's  python packages
[20:58] <blackboxsw> it just doesn't support that level of granularity yet
[20:59] <powersj> smoser: I'll try ec2 shortly, be aware our test systems use lxd from ppa so it will be on par with bionic
[20:59] <blackboxsw> it does beg something slightly smarter than a simple translation from pip-pkg-name -> system-package-name which is what we currently do wholesale
[20:59] <powersj> Which reminds me we need to move to the snap
[21:00] <smoser> powersj: well, it works in that it doesn tfail
[21:00] <smoser> it just doenst get a console log
[21:01] <smoser> stgraber proposed fix for lxd at https://github.com/lxc/lxd/pull/4140
[21:01] <powersj> smoser: do you want this change as a part of ec2 or a follow on?
[21:02] <smoser> i was just going to ask.
[21:02] <smoser> i can either propose it as stand alone
[21:02] <smoser> or as part of yours.
[21:03] <smoser> powersj: i think pull it into yours
[21:04] <powersj> smoser: ok I'll pull and test a bit, that means hopefully merge monday? ;)
[21:08] <smoser> powersj: yeah.
[21:45] <powersj> \o/ an ec2 console log
[21:47] <smoser> woot!
[21:50] <powersj> now I just need my coffee shop connection to get a boost to download an image for nocloud
[22:12] <powersj> pylxd doesn't like the snap lxd
[22:21] <powersj> smoser: do you want me to squash all my commits?
[22:21] <powersj> otherwise once this lxd test is done I think we are good to go
[22:30] <powersj> smoser: pushed a version I confirmed working console log with lxd, nocloud-kvm, and ec2
[22:30] <powersj> I'm calling it done :D
[22:30] <powersj> thanks for the help