[00:21] rharper, blackboxsw well, it will have *some* per message hit on ubuntu [00:22] but on the order of 5/3 of what it was. thats roughly the ratio of difference for a 'import' of an already imported library compared to bool(variable is None) [00:22] now in the missing library case, python will probably go statting a bunch of possible locations again. [00:22] and there could be a larger hit [00:22] but in reality, if you hit that code it wont go doing it again (because it raises a nonimplementederror) [00:59] rharper, or blackboxsw around ? [00:59] i have a fix for trunk's current busted tests [00:59] or powersj [00:59] http://paste.ubuntu.com/25435503/ [01:00] failure is seen at https://jenkins.ubuntu.com/server/job/cloud-init-ci/236/console [01:12] smoser: still there? [01:15] seems you already fixed it [01:15] * powersj disappears again [01:21] yeah. shoudl be good now. and uploading to artful [01:22] e5bbf884..f713c6e9 HEAD -> ubuntu/devel [01:22] * [new tag] ubuntu/0.7.9-259-g7e76c57b-0ubuntu1 -> ubuntu/0.7.9-259-g7e76c57b-0ubuntu1 [01:22] and uploaded. cloud-init_0.7.9-259-g7e76c57b-0ubuntu1_source.changes [10:41] It would be nice if ds-identify/cloud-id emitted something that would end up in the nova console log; it would make working out why an instance hasn't come up a little easier. [14:37] smoser: if you run "make deb" on your drop-ubuntu-init-switch branch does it also generate a deb for you that doesn't contain etc/init files [14:39] doesnt ? [14:39] i'm confused [16:39] blackboxsw, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/329946 is updated now. [16:40] changed the default 'init-system' for bddeb. [16:49] Long running tests on CI system: https://paste.ubuntu.com/25439503/ [16:49] ahh sure enough that's the issue I was seeing, or rather, what was still producing upstart scripts in etc/init in my debs [16:49] +1 [16:50] powersj: ahh that's in progress [16:50] good find [16:50] blackboxsw: as in there an MP to review yet? [16:50] it's the fact that httppretty in those tests isn't mocking the min datasource version [16:50] ah ok I do recall you talking about this [16:50] so it's reaching out to the internet and timing out [16:50] I didn't know that it resulted in long running tests as well [16:51] yeah [16:51] ok, larsks branch is up which would fix that. but it has a needs fixing comment [16:55] smoser: approved https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/329946 [16:55] thanks for the fix [16:55] powersj: https://code.launchpad.net/~larsks/cloud-init/+git/cloud-init/+merge/329660 is the 'fix' but needs a tweak before it lands. [16:56] yeah was looking at it - his CI run resulted in much more normal times too [16:57] larsks: do you know if you'll have cycles on the branch https://code.launchpad.net/~larsks/cloud-init/+git/cloud-init/+merge/329660 ? [16:59] if not, I think we might patch that branch and try to get that landed as the breakage (the unit tests timing out when hitting the unmocked metadata min_version) is now something we are seeing in our CI too [17:04] thanks for this powersj https://bugs.launchpad.net/cloud-init/+bug/1714117 [17:04] Ubuntu bug 1714117 in cloud-init "tox: long CI times" [Undecided,Confirmed] [17:06] looking over the larsks branch, I think I'd like to limit a bit of the mocking that is being done as well as the tests in his branch are also mocking out all of the min_metadata_version urls for all metadata when we only really need to mock instance-id. [17:06] so I'll probably tweak the branch a bit and try to only mock the necessary minimum content w/ httpretty [17:10] powersj: I like your nosetests params, I'd like to capture that in tox.ini too (maybe just a comment about the additional params so we can check for costly unit tests easily in the future [17:12] blackboxsw: nosetests --with-timer --timer-ok 1 --timer-warning 5 [17:13] that uses the nose-timer plugin [17:13] but you'd have to pip install nose-timeer [17:13] i think [17:13] yes [17:53] I just updated my 'merge request'. CI build failed https://jenkins.ubuntu.com/server/job/cloud-init-ci/241/ The failure is something related to the infrastructure and not at all related to my change. [17:53] How to rebuild the failed jenkins build? [17:54] stanguturi: yep, looks like we had issue getting to a mirror. I can re-kick shortly [17:55] hi sankar. good to see you again [18:00] Thanks Josh, Chad [18:00] stanguturi: CI is running again, if it fails again let me know. [18:01] as long as unit tests succeed there, I'll land that branch today stanguturi [18:02] Great. Thanks Chad [19:28] powersj: larsks I've got it on the CI unit test fixes/proper mocks [19:28] sweet [19:29] https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+ref/bug/ec2-tests-unmocked-metadata for the win. I'll put it up for review [19:29] it also uses httpretty.HTTPretty.allow_net_connect = False which raises a MockError if we try to reach out to anything on the internet not mocked [19:33] nice [19:44] https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/330040 is done powersj. it should help [19:51] blackboxsw: style will fail, but I +1 with comment [19:52] powersj: my wife tells me that a lot [19:52] also there are still two long-ish running tests that we should fix in another bug [19:52] * blackboxsw fails style [19:52] https://paste.ubuntu.com/25440455/ [19:52] LOL [19:53] powersj: actually it looks like a httpretty problem too. [19:53] powersj: is that faster? what was the the previous time, I thought you said 20 minutes or something crazy [19:53] I'll try to grab those too in this branch [19:53] s/httpretty/unmocked httpretty urls [19:53] these ran 20 seconds for each enviornment so ~1min and 40 seconds total versus 20 minutes ;) [19:54] heh [19:55] if we fix those other two tests we can speed things up enough people might run tox before pushing ;) [19:55] wha? I always run tox; doesn't take that long [19:55] * powersj points at blackboxsw [19:56] heh [19:56] tox -e flake8 [19:56] yeah [19:58] rharper: if I time tox -e py27 tests/unittests/test_datasource/test_ec2.py -- -s it takes 0m6.011s now, without the changeset 0m20.199s [19:58] blackboxsw: that's a nice win [19:58] also certain environments seem to timeout faster.. as larsks saw the tests take ~20 mins when he runs in centos env I think [19:59] I'm making a change for the two openstack tests now [20:05] sweet [20:16] hmm so, powersj I think we need to worry about those two in a separate bug, it's spending most of it's time I believe in a unittest callback which does pattern matching and processing on every metadata file request. instead of completely mocking each url under the metadata service, so I think it's just a costly way to setup the mock (with a callback each time) [20:16] yeah no need to pile on [20:16] separate is preferred anyway [21:24] is there a way to enforce exeuction of only one module from the user-data.txt file? [21:25] like, if a specific module is present, only execute that module and do not execute any others? [21:41] bah powersj https://jenkins.ubuntu.com/server/job/cloud-init-ci/242/console we hit issues with scratch on the builder I think. [21:41] E: 10mount: mount: special device /home/jenkins/ubuntu/scratch does not exist [21:41] E: xenial-amd64-c24fe5b1-313c-4f25-b2e1-58283e767c03: Chroot setup failed: stage=setup-start [21:41] Chroot setup failed [21:41] E: Error creating chroot session: skipping cloud-init [21:42] Shoot [21:43] I know what the issue is I can fix shortly [21:52] sather: in general no; however if you want to customize which modules get run; you can modify /etc/cloud/cloud.cfg which has a list of which modules will run at the different stages. It's likely if you only run one stage that the instance generally won't be configured properly but maybe in a custom image that's not an issue [21:52] sather: also, one can run single modules via cloud-init single --file=user-data.txt --force always --name cc_modulename; in case you just need to re-run one module [21:54] thx rharper I had a doorbell, I think we need the "--frequency always" as in: sudo cloud-init single --file=user-data.txt --force ---frequency always --name cc_modulename [21:54] blackboxsw: yeah [21:54] honestly, I think the frequency should be defaulted to always if we run in single mode [21:54] that's not a typical path, why make users do that [21:56] rharper: thanks for the info. I'm just trying to settle a security concern where we have a custom module, and only want that to run via cloud-init with a specific datasource... [21:57] sather: interesting; modules run independent of datasources in general [21:58] but the module is called with the cloud object which has a datasource in it; the module could check the datasource type (or attribute) and exit out if not the datasource you want [21:58] rharper: any way for us to close https://code.launchpad.net/~wesley-wiedenmeier/cloud-init/+git/cloud-init/+merge/317589. I don't seem to have perms to mark that branch as merged [21:58] blackboxsw: is it merged? if it's merged or superceeded, we usually "reject" and add comment about what we really did [21:59] I'll update that [21:59] comment that we're merged (via different branch) so marking it rejected (sorry for the harsh language) [21:59] I was actually hoping for a logic like this [21:59] I think the docstr are Reject, the actual feature name has already been mergedd [21:59] if specific-datasource; then run only this custom_module [21:59] thanks rharper [21:59] the specific-datasource is something someone did in-house [22:00] to allow for usb to be read by cloud-init [22:00] but we now just opened up our box to run any code if someone plugs a usb into it [22:00] :\ [22:01] heh [22:01] sounds like the custom module could use some tightening up [22:02] I guess I was wondering if it's possible to do [22:02] hrm, so in-house modules should be pluggable (as cloudinit/stages calls find_module on anything installed in cloudinit/config/cc_*py. Right it seems that module could be coded to reject other datasources from inspecting cloud.datasource [22:02] in general the cloud.cfg controls the list of modules that run by default; It sounds like you could use a custom datasource which could be derived from say config-drive, which checked various usb devices/features before running [22:02] if custom_module present in cloud-config; don't execute any other modules ;p [22:02] and the module itself can check that your specific derivative datasource was run before attempting to handle things. [22:03] cloud-config passed from userdata [22:03] well, generally there doesn't seem to be a way to disable the other modules dynamically that I can think of; but maybe smoser would know if I'm forgetting a way to do that [22:05] rharper: thakns for the help [22:05] rharper: so to be certain, even if a custom module is loaded on an instance w/ cloud-init installed, that custom module doesn't get run even though it's discovered if it isn't listed in /etc/cloud/cloud.cfg right. [22:05] so the goal is to prevent direct injection of custom modules in cloud-init without express intent as defined in /etc/cloud/cloud.cfg right [22:06] though I *think* that custom module could be run via "cloud-init single" even if it's not listed [22:07] just validated w/ cc_ntp [22:08] sather: sure, let us know if you get stuck or something else isn't working and we'll do our best to help you along the way [22:09] rharper: ok validated that a module name that isn't present in /etc/cloud/cloud.cfg can *not* be run even on commandline w/ --force [22:09] blackboxsw: hrm; I know the design allowed for user-date to write out modules to be run via user-data; so I may be missing something [22:09] 2017-08-31 22:08:27,056 - cc_ntp.py[DEBUG]: Skipping module named cc_ntp, not present or disabled by cfg [22:09] rharper: yeah I thought so too. poking around [22:11] "honestly, I think the frequency should be defaulted to always if we run in single mode" : +1 on that [22:34] new cloud-id bug: LP: #1714358 [22:34] Launchpad bug 1714358 in cloud-init "ds-identify does not find CloudStack datasource" [Undecided,New] https://launchpad.net/bugs/1714358 [23:23] blackboxsw: here is a bug for the OpenStack tests that are lengthy https://bugs.launchpad.net/cloud-init/+bug/1714376 [23:23] Ubuntu bug 1714376 in cloud-init "unittests: OpenStack DS escape or timeout" [Undecided,New]