[15:49] <mgerdts> @smoser - I pushed some updates for my merge proposal.  Not sure if I should have clicked "resubmit proposal".  Still learning the ropes with LP.  https://code.launchpad.net/~mgerdts/cloud-init/+git/cloud-init/+merge/343118
=== mgagne_ is now known as mgagne
[15:51] <smoser> mgerdts: no. dont push 'resubmit' just git push over the top
[15:53] <mgerdts> I did a git push.  Should I have squashed, then push -f?
[15:56] <smoser> mgerdts: you dont have to squash. we squash when we merge.
[15:56] <smoser> and generally, it is nice to  see commits that say "addressed feedback"
[15:56] <smoser> ie, as you did it is really nice. thank you.
[15:56] <mgerdts> :)
[15:57] <mgerdts> The one thing I've not figured out is a portable way to deal with the byte comparisons, other than ord()
[15:58] <mgerdts> I dislike having special code to help tests pass - there's probably a better way for me to do side_effects but I don't know what that is.
[16:00] <smoser> mgerdts: ok. i can poke a bit anad see if i can improve that.
[16:01] <mgerdts> thanks
[16:01] <blackboxsw> looks like it's cloud-init status meeting time again
[16:01] <smoser> o/
[16:01] <blackboxsw> #startmeeting Cloud-init bi-weekly status meeting
[16:01] <meetingology> Meeting started Mon Apr 16 16:01:49 2018 UTC.  The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology.
[16:01] <meetingology> Available commands: action commands idea info link nick
[16:02] <rharper> o/
[16:03] <blackboxsw> Hi folks, welcome to cloud-init's bi-weekly status meeting. Feel free to interject at any time or bring up branches/bugs/questions over the next 30-60 mins. We'll have a number of folks around to get eyes and/or keyboards onto any problems.
[16:03] <dpb1> o/
[16:04] <blackboxsw> As always we'll go through the following topics (feel free to suggest others): Recent-changes, In-progress Development, and ~30 mins Office Hours)
[16:04] <blackboxsw> #topic Recent-changes
[16:06] <blackboxsw> a quick rundown of the hi level changes landed:
[16:06] <blackboxsw> Prune integration test artifacts
[16:06] <blackboxsw> Add support for LXD 3.0, fix pylxd integration test dependency
[16:06] <blackboxsw> Fix Ubuntu proposed integration test CI job
[16:06] <blackboxsw> Fix ec2 validation of instance-data.json network info
[16:06] <blackboxsw> Do not retry optional userdata on 404 (LP: #1702160)
[16:06] <blackboxsw> Add explicit cloud-init package dependency on isc-dhcp-client (LP: #1759307)
[16:06] <ubot5`> Launchpad bug 1702160 in cloud-init "OpenStack datasource should not retry user-data on 404" [Medium,Fix released] https://launchpad.net/bugs/1702160
[16:06] <ubot5`> Launchpad bug 1759307 in cloud-init (Ubuntu) "missing dependency on isc-dhcp-client (dhclient)" [Medium,Fix released] https://launchpad.net/bugs/1759307
[16:08] <blackboxsw> additionally from most recent commits we have:
[16:08] <blackboxsw> tools: Fix make-tarball cli tool usage for development
[16:08] <blackboxsw> renderer: support unicode in render_from_file.
[16:08] <blackboxsw> Implement ntp client spec with auto support for distro selection
[16:08] <blackboxsw> Apport: add Brightbox, IBM, LXD, and OpenTelekomCloud to list of clouds.
[16:08] <blackboxsw> tests: fix ec2 integration network metadata validation
[16:08] <blackboxsw> tests: fix integration tests to support lxd 3.0 release
[16:08] <blackboxsw> correct documentation to match correct attribute name usage.
[16:08] <blackboxsw> cc_resizefs, util: handle no /dev/zfs
[16:09] <blackboxsw> Last week rharper found and fixed a regression in zfs resize behavior that was blocking our ubuntu SRU
[16:09] <blackboxsw> We have uploaded those fixes, as well as rharper's ntp spec changes (which should incorporate a number of robjo's opensuse/sles needs too)
[16:10] <blackboxsw> anything else notable that I'm missing gentlemen?
[16:10] <blackboxsw> If not, I'll jump to in-progress development
[16:11] <blackboxsw> #topic In-progresss Development
[16:13] <blackboxsw> So, on the ubuntu side of the house we are about to approve the cloud-init 18.2 SRU (Stable release update) into xenial and artful. Just one more validation run and we should be good to see 18.2.4 on xenial, artful. Ubuntu Bionic is already a few commits beyond that.
[16:13] <blackboxsw> On Ubuntu as well we are beating the drop to the Bionic LTS (Long term release) feature/bug freeze.
[16:14] <blackboxsw> This week marks the last week for use to get fixes into Bioinic images before that release is cut.
[16:14] <blackboxsw> so we'll be heads down on any Bionic-specific changes that need to get in.
[16:14] <blackboxsw> Feel free to checkout our trello board @
[16:14] <blackboxsw> #link https://trello.com/b/hFtWKUn3/daily-cloud-init-curtin
[16:15] <blackboxsw> we track all tasks we are working on in public view there so if there are any questions you can ping one of us here about our development efforts
[16:17] <blackboxsw> additional tasks that are in flight:  bash-autocompletion for cloud-init CLI (rhaper).  dropping ifconfig and route in favor of 'ip' (bbsw), and moving openstack datasource to cloud-init's local stage
[16:17] <blackboxsw> (smoser)
[16:17] <blackboxsw> also a couple of bugs to fix such as #1570997
[16:17] <blackboxsw> #link https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1759406
[16:17] <ubot5`> Ubuntu bug 1759406 in cloud-init (Ubuntu) "sru cloud-init (17.2-35-gf576b2a2-0ubuntu1~16.04.1 update to 18.2-4-g05926e48-0ubuntu1)" [Medium,Confirmed]
[16:17] <blackboxsw> oops paste fail:
[16:17] <blackboxsw> #link https://bugs.launchpad.net/bugs/1570997
[16:17] <ubot5`> Ubuntu bug 1570997 in ssh-import-id (Ubuntu Xenial) "fail if HOME environment variable is not set" [Low,Fix committed]
[16:18] <blackboxsw> I think that about wraps in-progress development, anything else that should be noted smoser rharper ?
[16:18] <rharper> I think you covered it
[16:18] <smoser> ssh-import-id is not relally at all related to cloud-init
[16:19] <smoser> thanks blackboxsw
[16:19] <blackboxsw> oops grabbed the wrong one, was thinking about this card
[16:19] <blackboxsw> #link https://trello.com/c/JVaXSfpo/749-eol-fix-for-ssh-file
[16:19] <smoser> right.
[16:21] <blackboxsw> also of note, in some of our SRU testing we found time-tracking gaps in cloud-init analyze tracking on Azure. rharper put of a logging tracker fix to avoid those tracking gaps
[16:21] <blackboxsw> #link https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/343123
[16:21] <smoser> i'll point out one thing i just finished up with...
[16:22] <smoser> for testing ubuntu, the https://github.com/cloud-init/ubuntu-sru/ has 'get-proposed-cloudimg' and 'lxc-proposed-snapshot'
[16:22] <smoser> which now work more like each other.l and can do more than just upgrade cloud-init.
[16:23] <blackboxsw> thanks smoser. Great tools to quicken dev-test cycles and make cloud-init development easier. That wraps up what we've been up to. We can probably move to the open forum for any discussions folks want to have
[16:23] <blackboxsw> #topic Office Hours (next ~30 mins)
[16:24] <blackboxsw> We'll be hanging out here for anyone who wants more eyes on a review, feature discussions or bug triage....
[16:28] <smoser> mgerdts: i'm poking at the branch i think i shoudl be able to get something.
[16:29] <mgerdts> I'm working on a bunch of fixes for things that have turned up on bhyve with SmartOS.  Since we are looking to transition from KVM to bhyve, we will need to provide updates at least as far back as xenial and probably trusty.  Is the process for this any more complicated than get the fixes in master, then cherry-pick the fixes into branches?
[16:29] <mgerdts> thanks @smoser
[17:09] <rharper> mgerdts: we preferrer not to cherry; rather we release master back to xenial via our SRU (Stable Release Update) process; however, we spend a lot of effort to not modify existing behavior on prevlous SRU releases;  so if the changes to support bhyve can be done in a compatible way (working with either) that'd be best;  worst-case, we patch in release specific bahvior into the release branch.
[17:13] <mgerdts> Pretty much everything that I've got queued up is fully compatible.
[17:14] <mgerdts> https://bugs.launchpad.net/cloud-init/+bug/1667735 implements proper protocol negotiation over the serial port.   The lack of this has caused problems with KVM at times too.
[17:14] <ubot5`> Ubuntu bug 1667735 in cloud-init (Ubuntu Trusty) "cloud-init doesn't retry metadata lookups and hangs forever if metadata is down" [Medium,Confirmed]
[17:15] <mgerdts> https://bugs.launchpad.net/cloud-init/+bug/1746605 adressess times when cloud-init and other software may be trying to use the metadata serial port at the same time.  This is purely a bug fix.
[17:15] <ubot5`> Ubuntu bug 1746605 in cloud-init "DataSourceSmartOS needs locking" [Medium,Confirmed]
[17:15] <mgerdts> I hit it when rc.local and cloud-init were both trying to get metadata.
[17:17] <mgerdts> https://bugs.launchpad.net/cloud-init/+bug/1763480 makes it so that cloud-init doesn't stack trace and exit when there is no customer_metadata.  This is an unlikely case, but something that is hit when you are testing things that don't need ssh keys, etc.
[17:17] <ubot5`> Ubuntu bug 1763480 in cloud-init "DataSourceSmartOS list() should always return a list" [Medium,Confirmed]
[17:18] <mgerdts> https://bugs.launchpad.net/cloud-init/+bug/1763512 finishes off the partial implementation of sdc:routes support.  Previously, we didn't publish the required information to VMs, so it is fair to consider this a new feature.
[17:18] <ubot5`> Ubuntu bug 1763512 in cloud-init "DataSourceSmartOS ignores sdc:routes" [Medium,Confirmed]
[17:19] <mgerdts> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1763511 is probably the most incompatible change.  New ephemeral disks will get ext4 instead of ext3, which is needed for larger disks that seem to be getting more common.
[17:19] <ubot5`> Ubuntu bug 1763511 in cloud-init (Ubuntu) "DataSourceSmartOS should default to ext4" [Undecided,New]
[17:21] <mgerdts> I think that's all that I have in the works right now.  Do any of these sound like they would be problematic as far back as Xenial?
[17:29] <rharper> new features are OK ,bug fixes are fine as well; I think that different filesystem is somewhat tricky
[17:30] <smoser> i think i'm generally ok with the different filesystem.
[17:31] <smoser> if the reason is that simply ext3 can't handle super-big
[17:31] <rharper> yeah, that doesn't seem so user-visible w.r.t configuration;
[17:32] <smoser> we could ensure being more backward compat if we checked the size of the disk and made an ext4 if > that-size
[17:32] <smoser> then there'd less issue
[17:32] <smoser> but more complication and future we'd be stuck with that
[17:32] <rharper> yeah, it wouldn't have worked on ext3 then it would be fine to use ext4
[17:32] <smoser> so i'd rather really just bite the bulleet
[17:32] <rharper> it could be a metadata flag that the Datasource looks for
[17:32] <smoser> rather than describing to pepole forever "well, if your  disk is < X you'll get ext3 otherwise ext4"
[17:49] <blackboxsw> rharper: got time for a netplan global dns hangout?
[17:49] <rharper> y
[17:49] <blackboxsw> I want to make sure I'm reading the tea leaves right
[17:52] <smoser> mgerdts: http://paste.ubuntu.com/p/5qdtFzY8w7/
[17:52] <blackboxsw> ok rharper I'm in cloud-init hangout
[17:52] <smoser> that makes tests pass. and i think the changes to the code path are right
[17:52] <rharper> ok
[17:52] <rharper> brt
[17:54] <smoser> it is still hacked in a sense that the response only deals with fp.read(1) rather than possibly anything that read more than 1.
[17:56] <cyphermox> blackboxsw: rharper: what's this about netplan global dns?
[17:57] <rharper> cyphermox: converting network v1 syntax from maas into something that works with netplan which doesn't have "dns" unbound to any interfaces
[17:57] <cyphermox> ok
[17:57] <rharper> maas I believe has fixed this for 2.4.x
[17:58] <rharper> they no longer will emit the type: nameserver  but legacy maas would have that, so we've a branch that stuffs them in reasonable places under defined interfaces which don't already have DNS values
[17:58] <blackboxsw> cyphermox: just SRU validation w.r.t. https://bugs.launchpad.net/cloud-init/+bug/1750884
[17:59] <ubot5`> Ubuntu bug 1750884 in cloud-init "[2.4, bionic] /etc/resolv.conf not configured correctly in Bionic, leads to no DNS resolution" [Medium,Fix released]
[17:59] <cyphermox> all good
[18:11] <blackboxsw> ooops, and /me forgot the end the epic meeting
[18:11] <blackboxsw> thx rharper for the chat
[18:11] <blackboxsw> #endmeetiung
[18:11] <blackboxsw> #endmeeting
[18:11] <meetingology> Meeting ended Mon Apr 16 18:11:33 2018 UTC.
[18:11] <meetingology> Minutes:        http://ubottu.com/meetingology/logs/cloud-init/2018/cloud-init.2018-04-16-16.01.moin.txt
[18:11] <blackboxsw> e
[18:11] <rharper> y
[18:34] <mgerdts> smoser: everything checks out on my end with your change.  I've pushed an update to LP.  Thanks!
[20:32] <blackboxsw> ok just finished SRU verification
[20:32] <blackboxsw> officially.
[20:32] <blackboxsw> I'm adding the verification-done tags now
[20:34] <rharper> \o/
[20:48] <smoser> mgerdts: still there?
[20:48]  * mgerdts here
[20:48] <smoser> ok. i'm on 62cafc5080ce3a21180770b65c43bb0964cdd70d of your (i think out of date branch)
[20:48] <smoser> i paste-failed earlier
[20:49] <smoser> http://paste.ubuntu.com/p/MtHPY4PKRb/
[20:49] <smoser> thats what i have now. i have been working on it since.
[20:51] <smoser> http://paste.ubuntu.com/p/Qq9YZkfJQY/
[20:52] <smoser> that one with flake8 fixes
[20:53] <smoser> http://paste.ubuntu.com/p/kbhVHvJHCs/ <-- and that with one fix
[20:53] <mgerdts> ok, is it this last one I should be grabbing?
[20:53] <smoser> yeah.
[20:53] <smoser> i can paste files too if thats easier
[20:54] <mgerdts> Is there a trick to make curl work with paste.ubuntu.com?
[20:54] <smoser> not really
[20:54] <smoser> here https://hastebin.com/raw/exukejojec
[20:54] <smoser> https://hastebin.com/raw/fagotonixu
[20:55] <mgerdts> I'm fine with (and prefer) patches.  It's just the download to my mac, scp where I need it to be, then run patch workflow that is tedious.
[20:55] <mgerdts> curl | patch is much eaiser
[21:21] <smoser> mgerdts: patch raw https://hastebin.com/raw/damujotohe
[21:21] <smoser> sorry. went afk for a bit
[21:22] <mgerdts> no worries.. I had it worked out already.  Was just reading through trying to make sense of it all.
[21:22] <smoser> i think its fairly straight forward. the reader just returns the bytes it has. like BytesIO
[21:23] <smoser> but allows the 'end_byte' to indicate an EOF. so as to get your short readds in there.
[21:23] <mgerdts> Yeah, mostly so.  It looks as though it has more features than are strictly needed and I was just looking to see if there was something I was missing.
[21:23] <smoser> i did the ShortReader rather than what i initially had because previously only 'read(1)' would work. if we changed the implementation it'd be busted.
[21:23] <smoser> yeah... i knwo.
[21:24] <mgerdts> I think we pretty much have to stick with read(1), as we don't have a way to put bytes that shouldn't have been read back.  But maybe that'
[21:25] <mgerdts> s not a big deal
[21:25] <smoser> yeah.
[21:26] <mgerdts> tox is happy, and it still seems to work in a VM.
[21:27] <mgerdts> I've pushed this change up to LP.
[21:27] <mgerdts> thanks, again, for your help with this.
[21:35] <mgerdts> from my read of the docs, I should be able to put cloud-config into cloud-init:user-data and that should override defaults.   But that seems not to work, at least for disk_setup and fs_setup.  Any idea where I might be going wrong?  https://hastebin.com/ovopajituh.php
[21:45] <rharper> mgerdts: do you have your cloud-init.log ?
[21:47] <mgerdts> https://hastebin.com/oxekuluhum.sql
[21:48] <mgerdts> I guess it is sql because it found 'blob' in it somewhere.
[21:48] <rharper> looks like yourr layout isn't correct
[21:48] <rharper> you have a boolean and it should be a list of partition sizes
[21:49] <rharper> oh, I see
[21:49] <rharper> the single partition
[21:50] <rharper>  Failed to find device during available device search.
[21:50] <rharper> 2018-04-16 21:13:42,054 - cc_disk_setup.py[DEBUG]: Automatic device for /dev/vdb identified as None
[21:50] <rharper> 2018-04-16 21:13:42,054 - cc_disk_setup.py[DEBUG]: No device aviable that matches request.
[21:50] <rharper> it didn't find your /dev/vdb
[21:53] <mgerdts> Doing a fresh run... one minute
[21:56] <mgerdts> https://hastebin.com/wexarotilu.go
[21:56] <rharper> looking
[21:56] <mgerdts> changed layout and overwrite to false.  cloud-init clean, rm /var/log/cloud-init*
[21:57] <mgerdts> then halted the vm, rolled-back the zvol under vdb to its initial state (all zeroes), then booted it.
[21:57] <rharper> that looks good
[21:57] <rharper> cc_disk_setup.py[DEBUG]: Creating file system ephemeral0 on /dev/vdb
[21:57] <rharper> 2018-04-16 21:53:25,025 - cc_disk_setup.py[DEBUG]:      Using cmd: ['/sbin/mkfs.ext3', '/dev/vdb', '-L', 'ephemeral0', '-F']
[21:57] <rharper> 2018-04-16 21:53:25,025 - util.py[DEBUG]: Running command ['/sbin/mkfs.ext3', '/dev/vdb', '-L', 'ephemeral0', '-F'] with allowed return codes [0] (shell=False, capture=True)
[21:57] <rharper> 2018-04-16 21:53:25,402 - util.py[DEBUG]: Creating fs for /dev/vdb took 0.386 seconds
[21:58] <mgerdts> but I specified xfs
[21:58] <rharper> oh, fs isn't right though, you wanted xfs
[21:58] <rharper> hrm
[21:59] <mgerdts> It looks like I'm specifying it correctly but it's just being mishandled, right?
[22:00] <rharper> I don't know what's getting passed to mkfs, but it appears that it's not the fs_setup entry
[22:00] <mgerdts> 2018-04-16 21:53:25,011 - cc_disk_setup.py[DEBUG]: Partitioning disks: {'/dev/vdb': {'_origname': 'ephemeral0', 'overwrite': False, 'table_type': 'mbr', 'layout': False}}
[22:01] <mgerdts> that comes some time after it shows that it picked up the config that I set in user-data
[22:01] <rharper> hrm, lemme check something
[22:02] <mgerdts> notice that it says mbr rather than gpt
[22:02] <rharper> y
[22:02] <rharper> this smells like user-data from DataSource isn't getting updated
[22:10] <rharper> hrm, let's see if we can find it in object; on the instance: # python3 -c "from cloudinit import stages; import cloudinit; cloudobj = cloudinit.stages._pkl_load('/var/lib/cloud/instance/obj.pkl'); print(cloudobj.userdata_raw)"
[22:10] <mgerdts> device_aliases: {'ephemeral0': '/dev/vdb'}
[22:10] <mgerdts> disk_setup:
[22:10] <mgerdts>     ephemeral0:
[22:10] <mgerdts>          table_type: gpt
[22:11] <mgerdts>          layout: False
[22:11] <mgerdts>          overwrite: False
[22:11] <mgerdts> fs_setup:
[22:11] <mgerdts>     - label: ephemeral0
[22:11] <mgerdts>       filesystem: xfs
[22:11] <mgerdts>       device: ephemeral0.0
[22:11] <rharper> I see your debug message
[22:11] <rharper> but I want to see if it's recorded into the object; the 'ud' param in the datasource is kepted as self.userdata_raw
[22:11] <rharper> that's referenced in the cloudify() stage which merges config before calling modules
[22:27]  * rharper steps away for a bit 
[22:37] <mgerdts> yay, figured it out
[22:37] <mgerdts> needs to start with '#cloud-config`
[22:38] <mgerdts> and with that, I'm stepping away too.
[22:38] <mgerdts> thanks for your help rharper