| smoser | rharper, blackboxsw well, it will have *some* per message hit on ubuntu | 00:21 |
|---|---|---|
| smoser | but on the order of 5/3 of what it was. thats roughly the ratio of difference for a 'import' of an already imported library compared to bool(variable is None) | 00:22 |
| smoser | now in the missing library case, python will probably go statting a bunch of possible locations again. | 00:22 |
| smoser | and there could be a larger hit | 00:22 |
| smoser | but in reality, if you hit that code it wont go doing it again (because it raises a nonimplementederror) | 00:22 |
| smoser | rharper, or blackboxsw around ? | 00:59 |
| smoser | i have a fix for trunk's current busted tests | 00:59 |
| smoser | or powersj | 00:59 |
| smoser | http://paste.ubuntu.com/25435503/ | 00:59 |
| smoser | failure is seen at https://jenkins.ubuntu.com/server/job/cloud-init-ci/236/console | 01:00 |
| powersj | smoser: still there? | 01:12 |
| powersj | seems you already fixed it | 01:15 |
| * powersj disappears again | 01:15 | |
| smoser | yeah. shoudl be good now. and uploading to artful | 01:21 |
| smoser | e5bbf884..f713c6e9 HEAD -> ubuntu/devel | 01:22 |
| smoser | * [new tag] ubuntu/0.7.9-259-g7e76c57b-0ubuntu1 -> ubuntu/0.7.9-259-g7e76c57b-0ubuntu1 | 01:22 |
| smoser | and uploaded. cloud-init_0.7.9-259-g7e76c57b-0ubuntu1_source.changes | 01:22 |
| Odd_Bloke | It would be nice if ds-identify/cloud-id emitted something that would end up in the nova console log; it would make working out why an instance hasn't come up a little easier. | 10:41 |
| blackboxsw | smoser: if you run "make deb" on your drop-ubuntu-init-switch branch does it also generate a deb for you that doesn't contain etc/init files | 14:37 |
| smoser | doesnt ? | 14:39 |
| smoser | i'm confused | 14:39 |
| smoser | blackboxsw, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/329946 is updated now. | 16:39 |
| smoser | changed the default 'init-system' for bddeb. | 16:40 |
| powersj | Long running tests on CI system: https://paste.ubuntu.com/25439503/ | 16:49 |
| blackboxsw | ahh sure enough that's the issue I was seeing, or rather, what was still producing upstart scripts in etc/init in my debs | 16:49 |
| blackboxsw | +1 | 16:49 |
| blackboxsw | powersj: ahh that's in progress | 16:50 |
| blackboxsw | good find | 16:50 |
| powersj | blackboxsw: as in there an MP to review yet? | 16:50 |
| blackboxsw | it's the fact that httppretty in those tests isn't mocking the min datasource version | 16:50 |
| powersj | ah ok I do recall you talking about this | 16:50 |
| blackboxsw | so it's reaching out to the internet and timing out | 16:50 |
| powersj | I didn't know that it resulted in long running tests as well | 16:50 |
| powersj | yeah | 16:51 |
| blackboxsw | ok, larsks branch is up which would fix that. but it has a needs fixing comment | 16:51 |
| blackboxsw | smoser: approved https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/329946 | 16:55 |
| blackboxsw | thanks for the fix | 16:55 |
| blackboxsw | powersj: https://code.launchpad.net/~larsks/cloud-init/+git/cloud-init/+merge/329660 is the 'fix' but needs a tweak before it lands. | 16:55 |
| powersj | yeah was looking at it - his CI run resulted in much more normal times too | 16:56 |
| blackboxsw | larsks: do you know if you'll have cycles on the branch https://code.launchpad.net/~larsks/cloud-init/+git/cloud-init/+merge/329660 ? | 16:57 |
| blackboxsw | if not, I think we might patch that branch and try to get that landed as the breakage (the unit tests timing out when hitting the unmocked metadata min_version) is now something we are seeing in our CI too | 16:59 |
| blackboxsw | thanks for this powersj https://bugs.launchpad.net/cloud-init/+bug/1714117 | 17:04 |
| ubot5 | Ubuntu bug 1714117 in cloud-init "tox: long CI times" [Undecided,Confirmed] | 17:04 |
| blackboxsw | looking over the larsks branch, I think I'd like to limit a bit of the mocking that is being done as well as the tests in his branch are also mocking out all of the min_metadata_version urls for all metadata when we only really need to mock instance-id. | 17:06 |
| blackboxsw | so I'll probably tweak the branch a bit and try to only mock the necessary minimum content w/ httpretty | 17:06 |
| blackboxsw | powersj: I like your nosetests params, I'd like to capture that in tox.ini too (maybe just a comment about the additional params so we can check for costly unit tests easily in the future | 17:10 |
| powersj | blackboxsw: nosetests --with-timer --timer-ok 1 --timer-warning 5 | 17:12 |
| powersj | that uses the nose-timer plugin | 17:13 |
| smoser | but you'd have to pip install nose-timeer | 17:13 |
| smoser | i think | 17:13 |
| powersj | yes | 17:13 |
| stanguturi | I just updated my 'merge request'. CI build failed https://jenkins.ubuntu.com/server/job/cloud-init-ci/241/ The failure is something related to the infrastructure and not at all related to my change. | 17:53 |
| stanguturi | How to rebuild the failed jenkins build? | 17:53 |
| powersj | stanguturi: yep, looks like we had issue getting to a mirror. I can re-kick shortly | 17:54 |
| blackboxsw | hi sankar. good to see you again | 17:55 |
| stanguturi | Thanks Josh, Chad | 18:00 |
| powersj | stanguturi: CI is running again, if it fails again let me know. | 18:00 |
| blackboxsw | as long as unit tests succeed there, I'll land that branch today stanguturi | 18:01 |
| stanguturi | Great. Thanks Chad | 18:02 |
| blackboxsw | powersj: larsks I've got it on the CI unit test fixes/proper mocks | 19:28 |
| powersj | sweet | 19:28 |
| blackboxsw | https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+ref/bug/ec2-tests-unmocked-metadata for the win. I'll put it up for review | 19:29 |
| blackboxsw | it also uses httpretty.HTTPretty.allow_net_connect = False which raises a MockError if we try to reach out to anything on the internet not mocked | 19:29 |
| rharper | nice | 19:33 |
| blackboxsw | https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/330040 is done powersj. it should help | 19:44 |
| powersj | blackboxsw: style will fail, but I +1 with comment | 19:51 |
| blackboxsw | powersj: my wife tells me that a lot | 19:52 |
| powersj | also there are still two long-ish running tests that we should fix in another bug | 19:52 |
| * blackboxsw fails style | 19:52 | |
| powersj | https://paste.ubuntu.com/25440455/ | 19:52 |
| powersj | LOL | 19:52 |
| blackboxsw | powersj: actually it looks like a httpretty problem too. | 19:53 |
| rharper | powersj: is that faster? what was the the previous time, I thought you said 20 minutes or something crazy | 19:53 |
| blackboxsw | I'll try to grab those too in this branch | 19:53 |
| blackboxsw | s/httpretty/unmocked httpretty urls | 19:53 |
| powersj | these ran 20 seconds for each enviornment so ~1min and 40 seconds total versus 20 minutes ;) | 19:53 |
| rharper | heh | 19:54 |
| powersj | if we fix those other two tests we can speed things up enough people might run tox before pushing ;) | 19:55 |
| rharper | wha? I always run tox; doesn't take that long | 19:55 |
| * powersj points at blackboxsw | 19:55 | |
| rharper | heh | 19:56 |
| rharper | tox -e flake8 | 19:56 |
| rharper | yeah | 19:56 |
| blackboxsw | rharper: if I time tox -e py27 tests/unittests/test_datasource/test_ec2.py -- -s it takes 0m6.011s now, without the changeset 0m20.199s | 19:58 |
| rharper | blackboxsw: that's a nice win | 19:58 |
| blackboxsw | also certain environments seem to timeout faster.. as larsks saw the tests take ~20 mins when he runs in centos env I think | 19:58 |
| blackboxsw | I'm making a change for the two openstack tests now | 19:59 |
| powersj | sweet | 20:05 |
| blackboxsw | hmm so, powersj I think we need to worry about those two in a separate bug, it's spending most of it's time I believe in a unittest callback which does pattern matching and processing on every metadata file request. instead of completely mocking each url under the metadata service, so I think it's just a costly way to setup the mock (with a callback each time) | 20:16 |
| powersj | yeah no need to pile on | 20:16 |
| powersj | separate is preferred anyway | 20:16 |
| sather | is there a way to enforce exeuction of only one module from the user-data.txt file? | 21:24 |
| sather | like, if a specific module is present, only execute that module and do not execute any others? | 21:25 |
| blackboxsw | bah powersj https://jenkins.ubuntu.com/server/job/cloud-init-ci/242/console we hit issues with scratch on the builder I think. | 21:41 |
| blackboxsw | E: 10mount: mount: special device /home/jenkins/ubuntu/scratch does not exist | 21:41 |
| blackboxsw | E: xenial-amd64-c24fe5b1-313c-4f25-b2e1-58283e767c03: Chroot setup failed: stage=setup-start | 21:41 |
| blackboxsw | Chroot setup failed | 21:41 |
| blackboxsw | E: Error creating chroot session: skipping cloud-init | 21:41 |
| powersj | Shoot | 21:42 |
| powersj | I know what the issue is I can fix shortly | 21:43 |
| rharper | sather: in general no; however if you want to customize which modules get run; you can modify /etc/cloud/cloud.cfg which has a list of which modules will run at the different stages. It's likely if you only run one stage that the instance generally won't be configured properly but maybe in a custom image that's not an issue | 21:52 |
| rharper | sather: also, one can run single modules via cloud-init single --file=user-data.txt --force always --name cc_modulename; in case you just need to re-run one module | 21:52 |
| blackboxsw | thx rharper I had a doorbell, I think we need the "--frequency always" as in: sudo cloud-init single --file=user-data.txt --force ---frequency always --name cc_modulename | 21:54 |
| rharper | blackboxsw: yeah | 21:54 |
| rharper | honestly, I think the frequency should be defaulted to always if we run in single mode | 21:54 |
| rharper | that's not a typical path, why make users do that | 21:54 |
| sather | rharper: thanks for the info. I'm just trying to settle a security concern where we have a custom module, and only want that to run via cloud-init with a specific datasource... | 21:56 |
| rharper | sather: interesting; modules run independent of datasources in general | 21:57 |
| rharper | but the module is called with the cloud object which has a datasource in it; the module could check the datasource type (or attribute) and exit out if not the datasource you want | 21:58 |
| blackboxsw | rharper: any way for us to close https://code.launchpad.net/~wesley-wiedenmeier/cloud-init/+git/cloud-init/+merge/317589. I don't seem to have perms to mark that branch as merged | 21:58 |
| rharper | blackboxsw: is it merged? if it's merged or superceeded, we usually "reject" and add comment about what we really did | 21:58 |
| rharper | I'll update that | 21:59 |
| rharper | comment that we're merged (via different branch) so marking it rejected (sorry for the harsh language) | 21:59 |
| sather | I was actually hoping for a logic like this | 21:59 |
| blackboxsw | I think the docstr are Reject, the actual feature name has already been mergedd | 21:59 |
| sather | if specific-datasource; then run only this custom_module | 21:59 |
| blackboxsw | thanks rharper | 21:59 |
| sather | the specific-datasource is something someone did in-house | 21:59 |
| sather | to allow for usb to be read by cloud-init | 22:00 |
| sather | but we now just opened up our box to run any code if someone plugs a usb into it | 22:00 |
| sather | :\ | 22:00 |
| rharper | heh | 22:01 |
| rharper | sounds like the custom module could use some tightening up | 22:01 |
| sather | I guess I was wondering if it's possible to do | 22:02 |
| blackboxsw | hrm, so in-house modules should be pluggable (as cloudinit/stages calls find_module on anything installed in cloudinit/config/cc_*py. Right it seems that module could be coded to reject other datasources from inspecting cloud.datasource | 22:02 |
| rharper | in general the cloud.cfg controls the list of modules that run by default; It sounds like you could use a custom datasource which could be derived from say config-drive, which checked various usb devices/features before running | 22:02 |
| sather | if custom_module present in cloud-config; don't execute any other modules ;p | 22:02 |
| rharper | and the module itself can check that your specific derivative datasource was run before attempting to handle things. | 22:02 |
| sather | cloud-config passed from userdata | 22:03 |
| rharper | well, generally there doesn't seem to be a way to disable the other modules dynamically that I can think of; but maybe smoser would know if I'm forgetting a way to do that | 22:03 |
| sather | rharper: thakns for the help | 22:05 |
| blackboxsw | rharper: so to be certain, even if a custom module is loaded on an instance w/ cloud-init installed, that custom module doesn't get run even though it's discovered if it isn't listed in /etc/cloud/cloud.cfg right. | 22:05 |
| blackboxsw | so the goal is to prevent direct injection of custom modules in cloud-init without express intent as defined in /etc/cloud/cloud.cfg right | 22:05 |
| blackboxsw | though I *think* that custom module could be run via "cloud-init single" even if it's not listed | 22:06 |
| blackboxsw | just validated w/ cc_ntp | 22:07 |
| rharper | sather: sure, let us know if you get stuck or something else isn't working and we'll do our best to help you along the way | 22:08 |
| blackboxsw | rharper: ok validated that a module name that isn't present in /etc/cloud/cloud.cfg can *not* be run even on commandline w/ --force | 22:09 |
| rharper | blackboxsw: hrm; I know the design allowed for user-date to write out modules to be run via user-data; so I may be missing something | 22:09 |
| blackboxsw | 2017-08-31 22:08:27,056 - cc_ntp.py[DEBUG]: Skipping module named cc_ntp, not present or disabled by cfg | 22:09 |
| blackboxsw | rharper: yeah I thought so too. poking around | 22:09 |
| blackboxsw | "honestly, I think the frequency should be defaulted to always if we run in single mode" : +1 on that | 22:11 |
| powersj | new cloud-id bug: LP: #1714358 | 22:34 |
| ubot5 | Launchpad bug 1714358 in cloud-init "ds-identify does not find CloudStack datasource" [Undecided,New] https://launchpad.net/bugs/1714358 | 22:34 |
| powersj | blackboxsw: here is a bug for the OpenStack tests that are lengthy https://bugs.launchpad.net/cloud-init/+bug/1714376 | 23:23 |
| ubot5 | Ubuntu bug 1714376 in cloud-init "unittests: OpenStack DS escape or timeout" [Undecided,New] | 23:23 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!