[13:42] <randomhack> Is it possible to override the default cloud-init values inside the OS image?  I'm trying to have it always run the disable-ec2-metadata module. TIA!
[14:13] <larsks> randomhack: yes, you can pass in a cloud configuration that will override what is provided in the image.
[14:13] <smoser> randomhack, or just put that setting into /etc/cloud/cloud.cfg.d/
[14:13] <smoser> i think thats what you're asking.
[14:14] <smoser> echo "disable_ec2_metadata" >> /etc/cloud/cloud.cfg.d/99-disable-ec2.cfg
[14:14] <smoser> er... with the value too
[15:08] <randomhack> Thanks!
[15:13] <smoser> rharper, around ?
[15:13] <rharper> here
[15:13] <smoser> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1675576
[15:13] <rharper> y
[15:13] <smoser> shall i attempt to know understand the 00-snapd-config.yaml
[15:14] <smoser> and delete the files it writes undconditionally
[15:14] <smoser> or just unconditionally delete those ?
[15:14] <rharper> I think we do it like the eni one;  it has known content
[15:14] <rharper> it's baked into the core snap image;
[15:14] <rharper> so, just like the eth0.cfg in the cloud-images
[15:15] <smoser> ok. so if the 00-snapd-config.yaml exists with known content
[15:15] <smoser> then i unconditionally delete the 3 other files
[15:18] <rharper> I think so; let me walk through the paths
[15:19] <rharper> 1) if cloud-init is disabled, then the image will use the baked in config; no issue  2) if cloud-init is enabled and has no network-config; it should still generate a fallback config, like we do today; uc16 will enable the netplan renderer; the baked in config should be removed  3) cloud-init is enabled, and we have a network config;  uc16 is configured to use netplan renderer, so we should delete
[15:20] <rharper> (2) and (3) I think cover the cases for if content is known, delete it as cloud-init will render a fallback (or specified) network config
[15:20] <rharper> and yes to wiping the files that netplan would have generated as the netplan renderer will call 'netplan generate' which will create those files as needed (from the cloud-init config it rendered)
[16:01] <jgrimm> powersj and magicalChicken, you are probably interested in reviewing this too -> https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/321397
[16:05] <jgrimm> rharper, thanks for writing ^^ fantastic
[16:22] <powersj> rharper: that doc is incredibly helpful. thank you!
[16:26] <rharper> cool
[16:28] <smoser> rharper, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/321465
[16:28] <rharper> k
[16:29] <rharper> smoser: I proposed that the other night
[16:29] <smoser> oh. link ?
[16:29] <smoser> i thought i'd seen it
[16:30] <rharper> https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/321222
[16:30] <rharper> mines better because you need to import logging and create  LOG object
[16:31] <smoser> :)
[16:31] <smoser> yeah
[16:31] <smoser> ok
[16:59] <smoser> rharper, http://paste.ubuntu.com/24282363/
[16:59] <smoser> are you opposed to that on top of yours ?
[17:05] <rharper> looking
[17:05] <rharper> smoser: that looks fine
[17:09] <smoser> rharper, thats merged. if you want to read... https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/321471
[17:09] <smoser> i need to add tests
[17:10] <smoser> its very simplistic and targetted. i had more generic looking things, but the scaffolding is unnecessary at this time. if we need somethign smarter later, add it then.
[17:56] <rharper> smoser: ok, looking
[17:57] <smoser> *just* p ushed with teests
[17:57] <rharper> smoser: also, is it reasonable for ds-identify to also look in /etc/cloud/cloud.cfg.d/ ? for maas datasource ?
[17:57] <rharper> the debconf_setselections hook for maas IIRC, writes the maas datasource cfg to /etc/cloud/cloud.cfg.d
[17:58] <rharper> but ds-identify detect_maas only searchs PATH_CLOUD_CONFD/*maas*cfg ;  which I believe will miss the .cfg.d dir
[18:00] <smoser> rharper, it does look for maas datasource there, no?
[18:00] <smoser> it should. let me check.
[18:01] <rharper> would be nice if check_config mentioned the files it checked in the debug log so we could be certain
[18:01] <smoser> bah. i thjin its busted.
[18:01] <rharper> debugging a uc16 deploy with plars and it's not finding the datasource
[18:15] <rharper> smoser:  it does check files in .cfg.d/*maas*;  it;'s looking for a top-level key 'MAAS' but I believe we specify it via datasource:\n\tMAAS: {}
[18:15] <rharper> so it doesn't match
[18:16] <smoser> no. its just not looking right. the file names input are wrong.
[18:16] <smoser> i dont think it checks in .cfg.d
[18:16] <smoser> at all
[18:16] <rharper> hrm
[18:16] <rharper> I added a debug log to check_config to print the fnames that match
[18:17] <rharper> [up 659.59s] ds-identify
[18:17] <rharper> policy loaded: mode=search report=false found=all maybe=all notfound=disabled
[18:17] <rharper> checking config file: /etc/cloud/cloud.cfg
[18:17] <rharper> checking config file: /etc/cloud/cloud.cfg.d/05_logging.cfg
[18:17] <rharper> checking config file: /etc/cloud/cloud.cfg.d/50-cloudconfig-maas-cloud-config.cfg
[18:17] <rharper> checking config file: /etc/cloud/cloud.cfg.d/90_dpkg.cfg
[18:17] <rharper> shows it reading it
[18:17] <rharper> now checking why we don't match
[18:19] <smoser>  http://paste.ubuntu.com/24282702/
[18:19] <smoser> hm..
[18:19] <smoser> oh. those are just other things reading it.
[18:19] <smoser> it does read those for some sources
[18:19] <smoser> for some calls to check_config
[18:20] <smoser> but not for maas's
[18:20] <smoser> try that paste ?
[18:20] <rharper> ok;  currently it looks like key was getting clobbered from a value in the file
[18:21] <rharper> ahh, I see
[18:21] <rharper> other methods were looking there but the dscheck_MAAS wasn't
[18:21] <rharper> lemme try yours
[18:23] <rharper> smoser: ok, that works!
[18:23] <rharper> checking config file: /etc/cloud/cloud.cfg for key: MAAS
[18:23] <rharper> checking config file: /etc/cloud/cloud.cfg.d/05_logging.cfg for key: MAAS
[18:23] <rharper> checking config file: /etc/cloud/cloud.cfg.d/50-cloudconfig-maas-cloud-config.cfg for key: MAAS
[18:23] <rharper> checking config file: /etc/cloud/cloud.cfg.d/90_dpkg.cfg for key: MAAS
[18:23] <rharper> check for 'MAAS' returned found
[18:23] <smoser> that PATH_CLOUD_CONFD
[18:23] <smoser> was very confusing
[18:23] <rharper> smoser: that said, I think we want to check for datasource: MAAS  though
[18:24] <smoser> i need to get my unit tests for ds-identify
[18:24] <rharper> not just 'MAAS' does it walk indented colons ?
[18:24] <smoser> thers a comment about that
[18:24] <smoser> "currently does not respect any hierarchy in searching for key."
[18:24] <rharper> urg
[18:24] <rharper> so, how do specify the datasource: MAAS  then ?
[18:24] <rharper> we need to emit MAAS: foo and the datasource: MAAS ?
[18:25] <smoser> i'm confused.
[18:25] <smoser> datasource:
[18:25] <smoser>  MAAS:
[18:25] <smoser>    stuf here
[18:25] <smoser> that will work
[18:25] <smoser> the parser doesn't know that 'MAAS' there is under datasource.
[18:25] <smoser> iti just finds a key named MAAS
[18:25] <rharper> ok; the comment about hierarchy led me to believe we'd not look at anything but the top level keys
[18:26] <smoser> its just broken in a different way then you imagined :)
[18:26] <rharper> if it was written datasource: { MAAS: { } }
[18:26] <smoser> basically it just flattens everything and looks at only key names.
[18:26] <rharper> it'd break as well
[18:26] <smoser> maybe, yes.
[18:26] <rharper> ok, well, that's good enough until we do unittests and sort out edge cases
[18:26] <smoser> yeah. it doesnt support reading that.
[18:26] <smoser> wait. maybe it does.
[18:26]  * rharper tries
[18:27] <rharper> nope
[18:28] <smoser> right. it doesnt. we need a tool that can merge cloud config in the same way cloud-init does and run fast.
[18:29] <rharper> y
[19:26] <smoser> rharper, mp https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/321482
[19:26] <rharper> k
[19:39] <smoser> rharper, fwiw, stumbled upon free speedup in cloud-init at
[19:39] <smoser>  http://rmcgibbo.github.io/blog/2013/05/23/faster-yaml-parsing-with-libyaml/
[19:39] <smoser> seemingly free
[19:40] <jgrimm> smoser, have you benchmarked?
[19:41] <smoser> it probably wont be huge for cloud-init, but it is faster per
[19:41] <smoser>  http://stackoverflow.com/questions/16864132/is-there-an-up-to-date-fast-yaml-parser-with-python-bindings
[19:41] <smoser>  http://stackoverflow.com/questions/27743711/can-i-speedup-yaml
[19:42] <jgrimm> ack, yeah that's what i was curious about, whether it actually makes a difference, tho  small, but free is something
[19:45] <smoser> rharper, cloudinit/net/netplan.py
[19:45] <smoser> bah
[19:45] <smoser> https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/321471
[19:45] <smoser> prints removed
[19:56] <rharper> smoser: whoa , we have libyaml ?
[20:00] <smoser> well, in a new container this works:
[20:00] <smoser>  python3 -c 'import yaml; from yaml import CLoader'
[20:00] <smoser> so i'd think that'd trow exception otherwisd
[20:00] <jgrimm> i had a trusty container handy, seemed to work their too (and i can see libyaml in dpkg -l too)
[20:00] <jgrimm> s/their/there
[20:01] <smoser> # apt-cache show python3-yaml | grep Depe
[20:01] <smoser> Depends: python3 (<< 3.6), python3 (>= 3.5~), python3:any (>= 3.3.2-2~), libc6 (>= 2.14), libyaml-0-2
[20:01] <rharper> do we run py3 on trusty c-i ?
[20:01] <smoser> probably python2 there.
[20:02] <jgrimm> i ran python2 there
[20:02] <smoser> and we should do a try: except ImportError anyway
[20:02] <jgrimm> https://github.com/msmbuilder/msmbuilder-legacy/pull/199/commits/beb21bf10892dd566fccf567f907404957db1aed
[20:02] <rharper> oh I suppose it should still work for py2 to call to libyaml though right ?
[20:02] <jgrimm> looks like easy pattern to adopt
[20:02] <smoser> yeah
[20:48] <smoser> rharper, intentionally or not, your bond rename devices test case shows that vlans have the same mac as the nic they use also
[20:48] <smoser> so have to ignore those
[20:49] <smoser> so, good that you did that .
[20:49] <rharper> huh
[20:49] <rharper> yeah, I guess that makes  sense
[20:49] <rharper> they just need to check for the vlan header but the mac is going to be the same
[20:49] <rharper> w.r.t "is this packet for me"
[20:49] <rharper> yay unittests!
[20:59] <jgrimm> powersj, any strong reason to require --data-dir to the integration test 'collect'?  That is, why not allow for tmpname as it gets output to stdout with 'run' (so could be inspected if left alone and/or verify'd later)
[21:01] <powersj> jgrimm: I don't think it is a very strong feeling, but it was to prevent writing it to /tmp so you didn't loose your collect after each reboot :)
[21:01] <powersj> strong reason* ugh
[21:02] <jgrimm> cool, i assumed so. just didn't work as i'd guessed it would (since i didn't read the docs, doh)
[21:02] <powersj> :)
[21:02] <powersj> yeah if I recall when I was writing all the initial tests it would go to tmp and then the next day I would spend the first 20mins running collect again :\
[21:02] <jgrimm> fwiw, the docs could be a bit more explicit on 'use collect' if you want to actually inspect a run
[21:02] <jgrimm> i can certain submit a pull request if agreed
[21:03] <powersj> magicalChicken has a MR to keep the records around when doing a run
[21:03] <jgrimm> powersj, cool, that's what i was wanting to do and figured collect was the intended way of doing that
[21:03] <powersj> https://code.launchpad.net/~wesley-wiedenmeier/cloud-init/+git/cloud-init/+merge/320401
[21:04] <jgrimm> powersj, do we run integration tests on trusty as part of automation?
[21:04] <powersj> that however does still need a doc update :)
[21:04] <powersj> not yet
[21:05] <powersj> A few too many of the tests initially did not work with trusty versus xenial and newer due to changes in behavior
[21:05] <jgrimm> heh, ok i want to play with the tests and better understand.. randomly chose a module.. blows up on trusty, not elsewhere. :)
[21:05] <jgrimm> cool. known issue(s). :)
[21:05] <powersj> haha yeah magicalChicken has yet another MR for distro flags, to be able to specify what tests can run on what distros
[21:06] <powersj> https://code.launchpad.net/~wesley-wiedenmeier/cloud-init/+git/cloud-init/+merge/321029
[21:06] <jgrimm> indeed, looking forward to that landing, really nice addition
[21:07]  * jgrimm checks whether this one is fixed already. 
[21:08] <jgrimm> powersj, but overall I think the integration test infrastructure is really nice.. easy to add to it
[21:08] <powersj> jgrimm: thanks! magicalChicken: ^^^ :)
[21:09] <jgrimm> \o/
[21:11] <magicalChicken> powersj, jgrimm: nice thanks :)
[21:12] <magicalChicken> jgrimm: you mentioned getting everything going on trusty, i still have a couple more feature flags to mess with there i think
[21:13] <magicalChicken> jgrimm: part of the issue is out of date cloud-init, so i need to build a py2 deb with current master for trusty and see how many tests fail just because of the old version
[21:18] <jgrimm> magicalChicken, very cool
[21:19] <jgrimm> indeed, i'm guessing this is an issue with being out of date