[00:21] <smoser> rharper, blackboxsw well, it will have *some* per message hit on ubuntu
[00:22] <smoser> but on the order of 5/3 of what it was.  thats roughly the ratio of difference for a 'import' of an already imported library compared to bool(variable is None)
[00:22] <smoser> now in the missing library case, python will probably go statting a bunch of possible locations again.
[00:22] <smoser> and there could be a larger hit
[00:22] <smoser> but in reality, if you hit that code it wont go doing it again (because it raises a nonimplementederror)
[00:59] <smoser> rharper, or blackboxsw around ?
[00:59] <smoser> i have a fix for trunk's current busted tests
[00:59] <smoser> or powersj
[00:59] <smoser> http://paste.ubuntu.com/25435503/
[01:00] <smoser> failure is seen at https://jenkins.ubuntu.com/server/job/cloud-init-ci/236/console
[01:12] <powersj> smoser: still there?
[01:15] <powersj> seems you already fixed it
[01:15]  * powersj disappears again
[01:21] <smoser> yeah. shoudl be good now. and uploading to artful
[01:22] <smoser>    e5bbf884..f713c6e9  HEAD -> ubuntu/devel
[01:22] <smoser>  * [new tag]           ubuntu/0.7.9-259-g7e76c57b-0ubuntu1 -> ubuntu/0.7.9-259-g7e76c57b-0ubuntu1
[01:22] <smoser> and uploaded.  cloud-init_0.7.9-259-g7e76c57b-0ubuntu1_source.changes
[10:41] <Odd_Bloke> It would be nice if ds-identify/cloud-id emitted something that would end up in the nova console log; it would make working out why an instance hasn't come up a little easier.
[14:37] <blackboxsw> smoser: if you run "make deb" on your drop-ubuntu-init-switch branch does it also generate a deb for you that doesn't contain etc/init files
[14:39] <smoser> doesnt ?
[14:39] <smoser> i'm confused
[16:39] <smoser> blackboxsw, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/329946 is updated now.
[16:40] <smoser> changed the default 'init-system' for bddeb.
[16:49] <powersj> Long running tests on CI system: https://paste.ubuntu.com/25439503/
[16:49] <blackboxsw> ahh sure enough that's the issue I was seeing, or rather, what was still producing upstart scripts in etc/init  in my debs
[16:49] <blackboxsw> +1
[16:50] <blackboxsw> powersj: ahh that's in progress
[16:50] <blackboxsw> good find
[16:50] <powersj> blackboxsw: as in there an MP to review yet?
[16:50] <blackboxsw> it's the fact that httppretty in those tests isn't mocking the min datasource version
[16:50] <powersj> ah ok I do recall you talking about this
[16:50] <blackboxsw> so it's reaching out to the internet and timing out
[16:50] <powersj> I didn't know that it resulted in long running tests as well
[16:51] <powersj> yeah
[16:51] <blackboxsw> ok, larsks branch is up which would fix that. but it has a needs fixing comment
[16:55] <blackboxsw> smoser: approved https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/329946
[16:55] <blackboxsw> thanks for the fix
[16:55] <blackboxsw> powersj: https://code.launchpad.net/~larsks/cloud-init/+git/cloud-init/+merge/329660 is the 'fix' but needs a tweak before it lands.
[16:56] <powersj> yeah was looking at it - his CI run resulted in much more normal times too
[16:57] <blackboxsw> larsks: do you know if you'll have cycles on the branch  https://code.launchpad.net/~larsks/cloud-init/+git/cloud-init/+merge/329660 ?
[16:59] <blackboxsw> if not, I think we might patch that branch and try to get that landed as the breakage (the unit tests timing out when hitting the unmocked metadata min_version) is now something we are seeing in our CI too
[17:04] <blackboxsw> thanks for this powersj  https://bugs.launchpad.net/cloud-init/+bug/1714117
[17:06] <blackboxsw> looking over the larsks branch, I think I'd like to limit a bit of the mocking that is being done as well as the tests in his branch are also mocking out all of the min_metadata_version urls for all metadata when we only really need to mock instance-id.
[17:06] <blackboxsw> so I'll probably tweak the branch a bit and try to only mock the necessary minimum content w/ httpretty
[17:10] <blackboxsw> powersj: I like your nosetests params, I'd like to capture that in tox.ini too (maybe just a comment about the additional params so we can check for costly unit tests easily in the future
[17:12] <powersj> blackboxsw: nosetests --with-timer --timer-ok 1 --timer-warning 5
[17:13] <powersj> that uses the nose-timer plugin
[17:13] <smoser> but you'd have to pip install nose-timeer
[17:13] <smoser> i think
[17:13] <powersj> yes
[17:53] <stanguturi> I just updated my 'merge request'. CI build failed https://jenkins.ubuntu.com/server/job/cloud-init-ci/241/ The failure is something related to the infrastructure and not at all related to my change.
[17:53] <stanguturi> How to rebuild the failed jenkins build?
[17:54] <powersj> stanguturi: yep, looks like we had issue getting to a mirror. I can re-kick shortly
[17:55] <blackboxsw> hi sankar. good to see you again
[18:00] <stanguturi> Thanks Josh, Chad
[18:00] <powersj> stanguturi: CI is running again, if it fails again let me know.
[18:01] <blackboxsw> as long as unit tests succeed there, I'll land that branch today stanguturi
[18:02] <stanguturi> Great. Thanks Chad
[19:28] <blackboxsw> powersj: larsks I've got it on the CI unit test fixes/proper mocks
[19:28] <powersj> sweet
[19:29] <blackboxsw> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+ref/bug/ec2-tests-unmocked-metadata for the win. I'll put it up for review
[19:29] <blackboxsw> it also uses httpretty.HTTPretty.allow_net_connect = False   which raises a MockError if we try to reach out to anything on the internet not mocked
[19:33] <rharper> nice
[19:44] <blackboxsw> https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/330040 is done powersj. it should help
[19:51] <powersj> blackboxsw: style will fail, but I +1 with comment
[19:52] <blackboxsw> powersj: my wife tells me that a lot
[19:52] <powersj> also there are still two long-ish running tests that we should fix in another bug
[19:52]  * blackboxsw fails style
[19:52] <powersj> https://paste.ubuntu.com/25440455/
[19:52] <powersj> LOL
[19:53] <blackboxsw> powersj: actually it looks like a httpretty problem too.
[19:53] <rharper> powersj: is that faster?  what was the the previous time, I thought you said 20 minutes or something crazy
[19:53] <blackboxsw> I'll try to grab those too in this branch
[19:53] <blackboxsw> s/httpretty/unmocked httpretty urls
[19:53] <powersj> these ran 20 seconds for each enviornment so ~1min and 40 seconds total versus 20 minutes ;)
[19:54] <rharper> heh
[19:55] <powersj> if we fix those other two tests we can speed things up enough people might run tox before pushing ;)
[19:55] <rharper> wha? I always run tox; doesn't take that long
[19:55]  * powersj points at blackboxsw 
[19:56] <rharper> heh
[19:56] <rharper> tox -e flake8
[19:56] <rharper> yeah
[19:58] <blackboxsw> rharper: if I  time tox -e py27 tests/unittests/test_datasource/test_ec2.py -- -s  it takes 0m6.011s now, without the changeset 0m20.199s
[19:58] <rharper> blackboxsw: that's a nice win
[19:58] <blackboxsw> also certain environments seem to timeout faster.. as larsks saw the tests take ~20 mins when he runs in centos env I think
[19:59] <blackboxsw> I'm making a change for the two openstack tests now
[20:05] <powersj> sweet
[20:16] <blackboxsw> hmm so, powersj I think we need to worry about those two in a separate bug, it's spending most of it's time I believe in a unittest callback  which does pattern matching and processing on every metadata file request. instead of completely mocking each url under the metadata service, so I think it's just a costly way to setup the mock (with a callback each time)
[20:16] <powersj> yeah no need to pile on
[20:16] <powersj> separate is preferred anyway
[21:24] <sather> is there a way to enforce exeuction of only one module from the user-data.txt file?
[21:25] <sather> like, if a specific module is present, only execute that module and do not execute any others?
[21:41] <blackboxsw> bah powersj https://jenkins.ubuntu.com/server/job/cloud-init-ci/242/console we hit issues with scratch on the builder I think.
[21:41] <blackboxsw> E: 10mount: mount: special device /home/jenkins/ubuntu/scratch does not exist
[21:41] <blackboxsw> E: xenial-amd64-c24fe5b1-313c-4f25-b2e1-58283e767c03: Chroot setup failed: stage=setup-start
[21:41] <blackboxsw> Chroot setup failed
[21:41] <blackboxsw> E: Error creating chroot session: skipping cloud-init
[21:42] <powersj> Shoot
[21:43] <powersj> I know what the issue is I can fix shortly
[21:52] <rharper> sather: in general no; however if you want to customize which modules get run; you can modify /etc/cloud/cloud.cfg which has a list of which modules will run at the different stages.  It's likely if you only run one stage that the instance generally won't be configured properly but maybe in a custom image that's not an issue
[21:52] <rharper> sather: also, one can run single modules via cloud-init single --file=user-data.txt --force always --name cc_modulename; in case you just need to re-run one module
[21:54] <blackboxsw> thx rharper I had a doorbell, I think we need the "--frequency always" as in:    sudo cloud-init single --file=user-data.txt --force ---frequency always --name cc_modulename
[21:54] <rharper> blackboxsw: yeah
[21:54] <rharper> honestly, I think the frequency should be defaulted to always if we run in single mode
[21:54] <rharper> that's not a typical path, why make users do that
[21:56] <sather> rharper: thanks for the info. I'm just trying to settle a security concern where we have a custom module, and only want that to run via cloud-init with a specific datasource...
[21:57] <rharper> sather: interesting;  modules run independent of datasources in general
[21:58] <rharper> but the module is called with the cloud object which has a datasource in it; the module could check the datasource type (or attribute) and exit out if not the datasource you want
[21:58] <blackboxsw> rharper: any way for us to close https://code.launchpad.net/~wesley-wiedenmeier/cloud-init/+git/cloud-init/+merge/317589. I don't seem to have perms to mark that branch as merged
[21:58] <rharper> blackboxsw: is it merged? if it's merged or superceeded, we usually "reject" and add comment about what we really did
[21:59] <rharper> I'll update that
[21:59] <rharper> comment that we're merged (via different branch) so marking it rejected (sorry for the harsh language)
[21:59] <sather> I was actually hoping for a logic like this
[21:59] <blackboxsw> I think the docstr are Reject, the actual feature name has already been mergedd
[21:59] <sather> if specific-datasource; then run only this custom_module
[21:59] <blackboxsw> thanks rharper
[21:59] <sather> the specific-datasource is something someone did in-house
[22:00] <sather> to allow for usb to be read by cloud-init
[22:00] <sather> but we now just opened up our box to run any code if someone plugs a usb into it
[22:00] <sather> :\
[22:01] <rharper> heh
[22:01] <rharper> sounds like the custom module could use some tightening up
[22:02] <sather> I guess I was wondering if it's possible to do
[22:02] <blackboxsw> hrm, so in-house modules should be pluggable (as cloudinit/stages calls find_module on anything installed in cloudinit/config/cc_*py. Right it seems that module could be coded to reject other datasources from inspecting cloud.datasource
[22:02] <rharper> in general the cloud.cfg controls the list of modules that run by default;  It sounds like you could use a custom datasource which could be derived from say config-drive, which checked various usb devices/features before running
[22:02] <sather> if custom_module present in cloud-config; don't execute any other modules ;p
[22:02] <rharper> and the module itself can check that your specific derivative datasource was run before attempting to handle things.
[22:03] <sather> cloud-config passed from userdata
[22:03] <rharper> well, generally there doesn't seem to be a way to disable the other modules dynamically that I can think of;  but maybe smoser would know if I'm forgetting a way to do that
[22:05] <sather> rharper: thakns for the help
[22:05] <blackboxsw> rharper:  so to be certain, even if a custom module is loaded on an instance w/ cloud-init installed, that custom module doesn't get run even though it's discovered if it isn't listed in /etc/cloud/cloud.cfg right.
[22:05] <blackboxsw> so the goal is to prevent direct injection of custom modules in cloud-init without express intent as defined in /etc/cloud/cloud.cfg right
[22:06] <blackboxsw> though I *think* that custom module could be run via "cloud-init single" even if it's not listed
[22:07] <blackboxsw> just validated w/ cc_ntp
[22:08] <rharper> sather: sure, let us know if you get stuck or something else isn't working and we'll do our best to help you along the way
[22:09] <blackboxsw> rharper: ok validated that a module name that isn't present in /etc/cloud/cloud.cfg can *not* be run even on commandline w/ --force
[22:09] <rharper> blackboxsw: hrm;  I know the design allowed for user-date to write out modules to be run via user-data; so I may be missing something
[22:09] <blackboxsw> 2017-08-31 22:08:27,056 - cc_ntp.py[DEBUG]: Skipping module named cc_ntp, not present or disabled by cfg
[22:09] <blackboxsw> rharper: yeah I thought so too. poking around
[22:11] <blackboxsw> "honestly, I think the frequency should be defaulted to always if we run in single mode"  :  +1 on that
[22:34] <powersj> new cloud-id bug: LP: #1714358
[23:23] <powersj> blackboxsw: here is a bug for the OpenStack tests that are lengthy https://bugs.launchpad.net/cloud-init/+bug/1714376