=== shardy is now known as shardy_mtg === shardy_mtg is now known as shardy [13:42] Is it possible to override the default cloud-init values inside the OS image? I'm trying to have it always run the disable-ec2-metadata module. TIA! [14:13] randomhack: yes, you can pass in a cloud configuration that will override what is provided in the image. [14:13] randomhack, or just put that setting into /etc/cloud/cloud.cfg.d/ [14:13] i think thats what you're asking. [14:14] echo "disable_ec2_metadata" >> /etc/cloud/cloud.cfg.d/99-disable-ec2.cfg [14:14] er... with the value too [15:08] Thanks! [15:13] rharper, around ? [15:13] here [15:13] https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1675576 [15:13] y [15:13] shall i attempt to know understand the 00-snapd-config.yaml [15:14] and delete the files it writes undconditionally [15:14] or just unconditionally delete those ? [15:14] I think we do it like the eni one; it has known content [15:14] it's baked into the core snap image; [15:14] so, just like the eth0.cfg in the cloud-images [15:15] ok. so if the 00-snapd-config.yaml exists with known content [15:15] then i unconditionally delete the 3 other files [15:18] I think so; let me walk through the paths [15:19] 1) if cloud-init is disabled, then the image will use the baked in config; no issue 2) if cloud-init is enabled and has no network-config; it should still generate a fallback config, like we do today; uc16 will enable the netplan renderer; the baked in config should be removed 3) cloud-init is enabled, and we have a network config; uc16 is configured to use netplan renderer, so we should delete [15:20] (2) and (3) I think cover the cases for if content is known, delete it as cloud-init will render a fallback (or specified) network config [15:20] and yes to wiping the files that netplan would have generated as the netplan renderer will call 'netplan generate' which will create those files as needed (from the cloud-init config it rendered) [16:01] powersj and magicalChicken, you are probably interested in reviewing this too -> https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/321397 [16:05] rharper, thanks for writing ^^ fantastic [16:22] rharper: that doc is incredibly helpful. thank you! [16:26] cool [16:28] rharper, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/321465 [16:28] k [16:29] smoser: I proposed that the other night [16:29] oh. link ? [16:29] i thought i'd seen it [16:30] https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/321222 [16:30] mines better because you need to import logging and create LOG object [16:31] :) [16:31] yeah [16:31] ok [16:59] rharper, http://paste.ubuntu.com/24282363/ [16:59] are you opposed to that on top of yours ? [17:05] looking [17:05] smoser: that looks fine [17:09] rharper, thats merged. if you want to read... https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/321471 [17:09] i need to add tests [17:10] its very simplistic and targetted. i had more generic looking things, but the scaffolding is unnecessary at this time. if we need somethign smarter later, add it then. [17:56] smoser: ok, looking [17:57] *just* p ushed with teests [17:57] smoser: also, is it reasonable for ds-identify to also look in /etc/cloud/cloud.cfg.d/ ? for maas datasource ? [17:57] the debconf_setselections hook for maas IIRC, writes the maas datasource cfg to /etc/cloud/cloud.cfg.d [17:58] but ds-identify detect_maas only searchs PATH_CLOUD_CONFD/*maas*cfg ; which I believe will miss the .cfg.d dir [18:00] rharper, it does look for maas datasource there, no? [18:00] it should. let me check. [18:01] would be nice if check_config mentioned the files it checked in the debug log so we could be certain [18:01] bah. i thjin its busted. [18:01] debugging a uc16 deploy with plars and it's not finding the datasource [18:15] smoser: it does check files in .cfg.d/*maas*; it;'s looking for a top-level key 'MAAS' but I believe we specify it via datasource:\n\tMAAS: {} [18:15] so it doesn't match [18:16] no. its just not looking right. the file names input are wrong. [18:16] i dont think it checks in .cfg.d [18:16] at all [18:16] hrm [18:16] I added a debug log to check_config to print the fnames that match [18:17] [up 659.59s] ds-identify [18:17] policy loaded: mode=search report=false found=all maybe=all notfound=disabled [18:17] checking config file: /etc/cloud/cloud.cfg [18:17] checking config file: /etc/cloud/cloud.cfg.d/05_logging.cfg [18:17] checking config file: /etc/cloud/cloud.cfg.d/50-cloudconfig-maas-cloud-config.cfg [18:17] checking config file: /etc/cloud/cloud.cfg.d/90_dpkg.cfg [18:17] shows it reading it [18:17] now checking why we don't match [18:19] http://paste.ubuntu.com/24282702/ [18:19] hm.. [18:19] oh. those are just other things reading it. [18:19] it does read those for some sources [18:19] for some calls to check_config [18:20] but not for maas's [18:20] try that paste ? [18:20] ok; currently it looks like key was getting clobbered from a value in the file [18:21] ahh, I see [18:21] other methods were looking there but the dscheck_MAAS wasn't [18:21] lemme try yours [18:23] smoser: ok, that works! [18:23] checking config file: /etc/cloud/cloud.cfg for key: MAAS [18:23] checking config file: /etc/cloud/cloud.cfg.d/05_logging.cfg for key: MAAS [18:23] checking config file: /etc/cloud/cloud.cfg.d/50-cloudconfig-maas-cloud-config.cfg for key: MAAS [18:23] checking config file: /etc/cloud/cloud.cfg.d/90_dpkg.cfg for key: MAAS [18:23] check for 'MAAS' returned found [18:23] that PATH_CLOUD_CONFD [18:23] was very confusing [18:23] smoser: that said, I think we want to check for datasource: MAAS though [18:24] i need to get my unit tests for ds-identify [18:24] not just 'MAAS' does it walk indented colons ? [18:24] thers a comment about that [18:24] "currently does not respect any hierarchy in searching for key." [18:24] urg [18:24] so, how do specify the datasource: MAAS then ? [18:24] we need to emit MAAS: foo and the datasource: MAAS ? [18:25] i'm confused. [18:25] datasource: [18:25] MAAS: [18:25] stuf here [18:25] that will work [18:25] the parser doesn't know that 'MAAS' there is under datasource. [18:25] iti just finds a key named MAAS [18:25] ok; the comment about hierarchy led me to believe we'd not look at anything but the top level keys [18:26] its just broken in a different way then you imagined :) [18:26] if it was written datasource: { MAAS: { } } [18:26] basically it just flattens everything and looks at only key names. [18:26] it'd break as well [18:26] maybe, yes. [18:26] ok, well, that's good enough until we do unittests and sort out edge cases [18:26] yeah. it doesnt support reading that. [18:26] wait. maybe it does. [18:26] * rharper tries [18:27] nope [18:28] right. it doesnt. we need a tool that can merge cloud config in the same way cloud-init does and run fast. [18:29] y [19:26] rharper, mp https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/321482 [19:26] k [19:39] rharper, fwiw, stumbled upon free speedup in cloud-init at [19:39] http://rmcgibbo.github.io/blog/2013/05/23/faster-yaml-parsing-with-libyaml/ [19:39] seemingly free [19:40] smoser, have you benchmarked? [19:41] it probably wont be huge for cloud-init, but it is faster per [19:41] http://stackoverflow.com/questions/16864132/is-there-an-up-to-date-fast-yaml-parser-with-python-bindings [19:41] http://stackoverflow.com/questions/27743711/can-i-speedup-yaml [19:42] ack, yeah that's what i was curious about, whether it actually makes a difference, tho small, but free is something [19:45] rharper, cloudinit/net/netplan.py [19:45] bah [19:45] https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/321471 [19:45] prints removed [19:56] smoser: whoa , we have libyaml ? [20:00] well, in a new container this works: [20:00] python3 -c 'import yaml; from yaml import CLoader' [20:00] so i'd think that'd trow exception otherwisd [20:00] i had a trusty container handy, seemed to work their too (and i can see libyaml in dpkg -l too) [20:00] s/their/there [20:01] # apt-cache show python3-yaml | grep Depe [20:01] Depends: python3 (<< 3.6), python3 (>= 3.5~), python3:any (>= 3.3.2-2~), libc6 (>= 2.14), libyaml-0-2 [20:01] do we run py3 on trusty c-i ? [20:01] probably python2 there. [20:02] i ran python2 there [20:02] and we should do a try: except ImportError anyway [20:02] https://github.com/msmbuilder/msmbuilder-legacy/pull/199/commits/beb21bf10892dd566fccf567f907404957db1aed [20:02] oh I suppose it should still work for py2 to call to libyaml though right ? [20:02] looks like easy pattern to adopt [20:02] yeah [20:48] rharper, intentionally or not, your bond rename devices test case shows that vlans have the same mac as the nic they use also [20:48] so have to ignore those [20:49] so, good that you did that . [20:49] huh [20:49] yeah, I guess that makes sense [20:49] they just need to check for the vlan header but the mac is going to be the same [20:49] w.r.t "is this packet for me" [20:49] yay unittests! [20:59] powersj, any strong reason to require --data-dir to the integration test 'collect'? That is, why not allow for tmpname as it gets output to stdout with 'run' (so could be inspected if left alone and/or verify'd later) [21:01] jgrimm: I don't think it is a very strong feeling, but it was to prevent writing it to /tmp so you didn't loose your collect after each reboot :) [21:01] strong reason* ugh [21:02] cool, i assumed so. just didn't work as i'd guessed it would (since i didn't read the docs, doh) [21:02] :) [21:02] yeah if I recall when I was writing all the initial tests it would go to tmp and then the next day I would spend the first 20mins running collect again :\ [21:02] fwiw, the docs could be a bit more explicit on 'use collect' if you want to actually inspect a run [21:02] i can certain submit a pull request if agreed [21:03] magicalChicken has a MR to keep the records around when doing a run [21:03] powersj, cool, that's what i was wanting to do and figured collect was the intended way of doing that [21:03] https://code.launchpad.net/~wesley-wiedenmeier/cloud-init/+git/cloud-init/+merge/320401 [21:04] powersj, do we run integration tests on trusty as part of automation? [21:04] that however does still need a doc update :) [21:04] not yet [21:05] A few too many of the tests initially did not work with trusty versus xenial and newer due to changes in behavior [21:05] heh, ok i want to play with the tests and better understand.. randomly chose a module.. blows up on trusty, not elsewhere. :) [21:05] cool. known issue(s). :) [21:05] haha yeah magicalChicken has yet another MR for distro flags, to be able to specify what tests can run on what distros [21:06] https://code.launchpad.net/~wesley-wiedenmeier/cloud-init/+git/cloud-init/+merge/321029 [21:06] indeed, looking forward to that landing, really nice addition [21:07] * jgrimm checks whether this one is fixed already. [21:08] powersj, but overall I think the integration test infrastructure is really nice.. easy to add to it [21:08] jgrimm: thanks! magicalChicken: ^^^ :) [21:09] \o/ [21:11] powersj, jgrimm: nice thanks :) [21:12] jgrimm: you mentioned getting everything going on trusty, i still have a couple more feature flags to mess with there i think [21:13] jgrimm: part of the issue is out of date cloud-init, so i need to build a py2 deb with current master for trusty and see how many tests fail just because of the old version [21:18] magicalChicken, very cool [21:19] indeed, i'm guessing this is an issue with being out of date