/srv/irclogs.ubuntu.com/2017/01/09/#cloud-init.txt

=== StoneTable is now known as aisrael
=== natorious_ is now known as natorious
=== Guest96631 is now known as mgagne
powersjmagicalChicken: still getting too many open files https://paste.ubuntu.com/23771548/18:06
powersjis this more in line with what you were seeing?18:06
magicalChickenpowersj: yeah, especially first ones there where the stacktrace is from in pylxd18:15
powersjmagicalChicken: ok, how can I help here?18:16
magicalChickeni guess switching back to polling inside the instance lets the test suite get further before it hits it18:16
magicalChickenpowersj: pylxd is broken, there isn;t really anything that can be done here18:16
magicalChickenI'm going to look into the pylxd bug sometime today or tomorrow I guess and see if i might be able to patch it18:16
powersjok - should I fall back to pylxd 2.1?18:16
magicalChickenI'll have to modify the test suite again to get it working with 2.1, the response from execute() is different18:17
powersjok then let's leave it at 2.2 and get pylxd fixed18:17
magicalChickenWe'd also lose all of the error handling for setup image18:17
magicalChickenYeah, fixing pylxd shouldn't be too difficult. I've read through the code a couple times before when I was starting on the test suite18:18
magicalChickenThe pylxd bug will probably cause issues for other projects as well at some point, so good to fix it18:18
powersjmagicalChicken: ok sounds like a plan. I am going to send you a list of scenarios that I am using today with no issues and what I am using to test your merge. Hopefully that helps18:19
magicalChickenSure18:20
magicalChickenRaising the open file limit might buy the test suite enough space, but it probably isn't needed18:20
magicalChickenpowersj: I've also traced down the last centos issue to a systemd bug in the centos lxd images, so the test suite is doing the right thing there18:21
powersjoh nice!18:21
rharperpowersj: magicalChicken: question on using integration tests... in the conf yaml which supplies the cloud-init, I've got a specific IP and hostname that I'd like to reference ... in the test datacollections, there isn't any variable references; I just need to make sure I copy the value into the collect script ?18:48
magicalChickenrharper: you can access the cloud-config provided to the test from the test script with self.cloud_config18:49
magicalChickenoh, wait you mean the collect scripts themselves18:50
magicalChickenI haven't added in any variables there, its definitely doable though18:50
rharperso I have user-data with server: 192.168.12.23;   was thinking I'd set a variable, but that's probably too much trouble;18:51
rharperif I didn't use shell to run the collect program, I could use python instead and import the user-data configuration to fetch the values18:51
magicalChickenMost of the current collect scripts are like that18:51
magicalChickenThe script itself just grabs a file or the output of a command, then the python verifier script checks if its correct18:52
magicalChickenIn the python verify scripts there's some helper utilities as well to read through the cloud-config the test ran with18:53
rharperah, yeah, that's fair18:54
rharperI forgot things were spread apart18:54
magicalChickenI figured this way its easier to debug the verification, because downloading the images and all is slow18:54
rharperhrm, most of the collect scripts don't redirect output to a file ... I suppose that's what I was confused; most of them run a test (grep $this $that) and return zero or non-zero18:56
rharperah, i see, output is stored via the script name18:57
magicalChickenI don't actually have the test suite looking at the return code from a colelct script18:57
rharpersorry18:57
magicalChickenoh, yeah stdout is saved18:57
* rharper is coming up to speed on adding one18:57
rharperor rather modifying18:57
magicalChickenThere's also a 'create' command that can give you a template18:57
rharpermagicalChicken: powersj:  I added a test_ method to one of the ntp_server ones ... what's the top-level command I'd use to kick off say just the 'ntp_servers' integration test ?21:08
powersjpython3 -m tests.cloud_tests run -v -n xenial -t tests/cloud_tests/configs/modules/user_groups.yaml21:09
powersjis an example of running on a xenial based one with the user_groups test21:09
rharpersweet21:10
* rharper fires away and watches the world burn 21:10
powersj:)21:10
rharperhttp://paste.ubuntu.com/23772570/21:15
rharperpowersj: magicalChicken ^^21:15
powersjrharper: what version of pylxd? `pip show pylxd21:16
rharperpip?21:18
rharper% apt-cache policy python3-pylxd21:18
rharperpython3-pylxd:21:18
rharper  Installed: 2.0.5-0ubuntu1.121:18
magicalChickenrharper: also, which version the test suite21:18
rharperI'm on xenial21:18
rharpermagicalChicken: how do I know ?21:18
rharperI branched cloud-init from trunk last week21:18
rharperfor this bug fix21:18
magicalChickenI haven't tested the version in trunk21:19
rharperwhat's running on jenkins ?21:19
magicalChickenalso, what's in trunk will only work with pylxd 2.121:19
magicalChickenthe version in trunk21:19
powersjIn my experience only 2.1.x of pylxd has worked sufficiently well.21:19
rharperboo21:19
rharpersmoser:  ^^21:19
magicalChickenthere's a bug in pylxd 2.221:19
magicalChickenthe version based on 2.2 is much better, but can't be used until that bug is fixed21:19
rharperdo you have a cloud-init-testing ppa with 2.1.x for xenial ?21:20
* rharper isn't going to do the pip thing here21:20
magicalChickenrharper: that's probably a good call, pip has messed up my system pretty bad21:20
rharperhehe21:20
magicalChickenthere isn't a ppa afaik21:20
powersjThe jenkins slaves have to use pip to grab a few things, so grabbing pylxd wasn't to big of a deal, but I agree a ppa would be nice...21:20
magicalChickenthe version in repos is 2.0 which is also incompatible21:20
rharperpowersj: cheater =P21:20
magicalChickena ppa would be good21:21
rharperyeah21:21
powersjrharper: I know :P21:21
rharperis 2.1 in yak or zesty ?21:21
rharperwe can just copy archive into any old ppa21:21
magicalChickenrharper: its in yakkety21:21
* rharper rmadisons21:21
rharpercool21:21
rharperyeah, y and z are 2.121:21
magicalChickenthe problem with 2.1 is there's no way for me to do error checking at all during setup, thats why the switch to 2.2 was needed21:22
rharpersure21:24
smoserwait...21:30
smoserwe should run the tests in a tox virtual env21:30
smoserpowersj, and i talked about that.21:30
smoserthen we can get the pylxd that we want/need21:30
magicalChickensmoser: no longer possible with the newer version of the test suite as it requires shell access21:31
smoser>21:31
smoser?21:31
magicalChickensmoser: i couldn't get support for other distros going without it21:31
smoseri dont follow that.21:31
magicalChickenutil.subp() doesn21:31
magicalChickenutil.subp() doesn't work inside tox env right?21:31
smoserwhy wouldnt it?21:32
magicalChickenisn't tox inside a chroot?21:32
rharpersmoser: I'm fine with tox; but I'd like a jenkins-runner like top-level entry point so I don't have to figure out how to call it21:32
smosertox is not a chroot, no.21:34
magicalChickensmoser: oh, nvm, sorry21:34
magicalChickensmoser: tests should run fine inside tox then21:34
magicalChickeni can write a wrapper script21:34
smosertox kiind of complains in some cases if you run a command (through it directly) that is not in the virtual env21:34
smoserbut, you'd run 'tox -e integration-tests'21:34
smoseror something.21:34
smoserthe issue with this...21:35
powersjwell we have a whole host of arguments we need to pass21:35
smoseris that pylxd requires c interfaces21:35
smoserwhich sucks21:35
magicalChickenoh, yeah that does suck21:35
smosermeaning 'pip install pylxd' compiles code with gcc (and requires python-dev and the like)21:35
magicalChickeni remember when we were trying to test curtin with python parted21:35
magicalChickenI'm almost considering just writing a wrapper around lxc command line then21:36
magicalChickenSince the current hold up on the test suite is a pylxd bug21:36
magicalChickenAnd the previous one was also a pylxd bug21:36
smoseri have one question...21:37
smoserwhen i saw powersj running this... i have to run this more locally...21:37
smosereach of the shells into a container took like 1 second to run21:37
smoseror more21:37
magicalChickenblame pylxd :)21:37
smoserso collecting things was painful to watch21:37
magicalChickenit creates a websocket interface then shuts it down each execute() call21:37
smoserif i blame pylxd, then i'm quite open to pitch pylxd21:37
magicalChickenyeah, there's no reason that should take that long21:37
smoserbut that doesntn seem like it relaly should be that slow.21:38
powersjsmoser: after upgrading to a newer version of pylxd that fixed auth issue it is far faster21:38
rharperpowersj: newer version being ?21:38
powersjrecall my updates about 50 mins -> 12 mins (or something of that magnitude)21:38
magicalChickenI'm not sure why it is, but compare instance.execute() to 'lxc execute cloud_test_... "cat /var/log/cloud-init.log"21:38
powersj2.1.321:38
powersjor newer21:38
magicalChickenpowersj: that's interesting, maybe that's part of why tests were taking so much longer on jenkins than my laptop for a while21:39
smoserhm..21:39
smoser$ time sh -c 'for i in "$@"; do lxc exec x1 /bin/true; done' -- $(seq 1 10)21:39
smoserreal0m1.705s21:39
smoseruser0m0.240s21:39
smosersys0m0.116s21:39
powersjthat plus zfs backend :)21:39
magicalChickenpowersj: haha yeah that definitely helped21:39
powersjyeah - everything is running as fast as you were seeing now, so speed isn't an issue imo21:39
magicalChickenthe current devel version should be even faster, image download times are less than half what they were before21:39
smosermagicalChicken, yeah, your instance.execute()  is what i'mo asking about.  above, there i did that in ~ 0.17 seconds per each21:40
smoser(which is still slow)21:40
smoserbut i recal that being like 3 seconds when watching it on powersj21:40
rharperpowersj: magicalChicken:  if I want to pip install python3-pylxd which version string should I use ?21:40
smoserif those are much more similar with the lxc cmdline, then i'd say we keep pylxd21:40
smoserand maybe we should anyway21:40
smoserrharper, really... dont do that :)21:40
smoserjust use tox or virtual env to get you waht you need21:40
magicalChickenrharper: for the version of the test suite in master, 2.1.321:41
magicalChickenbut yeah pip is terrible21:41
magicalChickeni have 4 or 5 different versions of every python library installed on my system21:41
magicalChickensmoser: all my instance.execute() method does is call pylxd's execute()21:41
rharpermaybe a diff to tox.ini with a cloud-test group with the right defs ?21:41
magicalChickenI am thinking that dropping pylxd may be good just so it doesn't break the test suite again21:42
smoseri dont kmpw/21:44
smoserknow21:44
magicalChickenpylxd does keep the code clean though21:44
smoseryeah... and honestly it shouldnt be that bad to bring up a web socket21:45
smoserso it *shouldnt* be that slow21:46
smoserand we happen to work with the people who write it :)21:46
smoseri dont like that i have to have gcc for it in a virtual env...21:46
magicalChickeni think the speed issue is mostly fixed21:46
magicalChickenthe current issue is https://github.com/lxc/pylxd/issues/20921:46
magicalChickenits leaking file descriptors, so the test suite can't get through a full run without running out21:47
smoserwell, we can ulimit -a for now21:47
smoserand the speed thing shoudl be fixable for sure...21:47
magicalChickensmoser: there's a half finished fix for that bug, I just need to get it to pass tests and ping someone to pull it21:47
smoserand we can/should  improve our collection and such to do fewer execs21:47
smoseras over other arches, those will be  more expensive21:48
smoserie, ssh as tghe transport21:48
magicalChickeni think 1 exec per collect script isn't too unreasonable21:49
magicalChickenit may be possible to reduce the number of default collect scripts though21:50
smosersure its possible.21:50
smoserwe wioll just need to work things differently.21:50
smoserbut thats fine21:50
smosernot a big deal now21:50
magicalChickenYeah, on kvm execute() should be pretty fast as well21:51
magicalChickenits only when we get to remote instances that it'll be slow21:51
rharperhttp://paste.ubuntu.com/23772854/22:09
rharpersmoser: powersj: magicalChicken: that let's me run it under tox on my xenial host22:09
* rharper will figure out how to pass the specific test as args and just include the main run command eventually 22:10
magicalChickenrharper: nice22:10
magicalChickenrharper: default behavior if -t is not specified is to just run everything22:10
rharperright22:11
powersjrharper: sweet... btw is that list of dependencies just a cut and paste of everything you had? Can we just specify pylxd and still run?22:11
rharperthat was the list of deps from apt-cache show python3-pylxd22:11
rharperit's from the xenial testenv22:11
powersjok!22:11
rharperand then the last 5 or below pylxd are package deps22:12
rharper017-01-09 16:08:06,209 - tests.cloud_tests - DEBUG - running collect script: ntp_conf_servers22:13
rharper2017-01-09 16:08:06,373 - tests.cloud_tests - DEBUG - running collect script: cloud-init-output.log22:13
rharper2017-01-09 16:08:06,534 - tests.cloud_tests - DEBUG - running collect script: cloud-init-version22:13
rharpersmoser, I'm seeing sub-second collects on 2.1.3 pylxd22:13
rharperalso on zfs backend, but I think that looks reasonable;  but will wait until we see the whole thing run22:13
magicalChickenrharper: on my system its usually ~8-10 seconds per test case, of which 6 of that is booting the system22:14
rharperyeah22:17
rharperwhy do we delete the base image each time ?22:18
rharperthat's somewhat annoying since I already use ubuntu-daily:xenial22:18
magicalChickenrharper: save disk space22:18
rharperhad it on my system22:18
magicalChickenthere's a modified version of the image that the tests actually run from, which should be deleted22:19
rharperright22:19
magicalChickeni could set it to leave the base image behind22:19
rharperbut the base images could stay22:19
rharpersorta like sync-images in curtin22:19
magicalChickenthe issue i run into is i only have 2G given to zfs on my system22:19
rharperideally we'd have a sync stage, and run stage which uses what's present22:19
magicalChickenso I can only have 1 image at a time22:19
magicalChickenbut yeah, the test suite doesn't need to do that22:19
* rharper hands magicalChicken external SSD disk 22:20
magicalChickenlol22:20
magicalChickeni have an external 1T disk, but no power cable :(22:20
rharperbut ssd ?22:20
magicalChickenNo, not ssd22:20
rharperthis is usb3 128G ssd, should be helpful22:20
rharperpowered via usb bus which is nice22:20
magicalChickennice22:21
rharperI'll pass it along at the next meet up if you like22:21
magicalChickenserious? thanks :)22:21
magicalChickenthat'll actually help a lot22:21
rharperyeah22:21
magicalChickenI can add in a image-sync command that'll download every image available22:22
magicalChickens/available/needed22:23
rharperyeah22:28
rharperso when using tox -e foo ;;; it injects a cloud-init-0.7.9.zip ;;; how and when is that made?  ie, how do I know that it includes what's committed or what's uncommitted ?22:28
rharperpowersj: ^^22:29
rharperwondering if the cloud-init that's running during the cloud_tests is my modified version or something else?22:29
magicalChickenrharper: the cloud-init in the cloud-tests is either from a repo, a ppa or a .deb22:32
powersjrharper: unless you specify a specific deb to inject... what he said :)22:32
magicalChickenthe test suite doesn't modify the cloud-init itself22:32
rharperbut how does it get injected into tox22:32
powersjit's not tox... it's the lxc image22:33
rharperhrm22:33
rharperdon't we have a curtin-like pack where we package up what's in-tree and inject it ?22:33
magicalChickenrharper: Oh, the version that gets copied into tox is probably just a clone of the current version22:33
magicalChickenrharper: but i don't think its used22:33
magicalChickenThere isn't a pack equivalent just because the way cloud-init gets built and what version of python it uses all depends on the release22:34
rharperso, I know at least when we run the tox command, we're using the right bits since I see a change in the server values22:34
rharperbut when the cloud_tests runner launches an lxc container, it starts with the base image, and then doesn't it inject the current git tree ?22:34
magicalChickenIt can install the current git tree if you want it to with a setup script22:35
rharperor are we not yet to to using mount-image-callback lxd:container-name22:35
rharperwell, if I'm developing a new test, how do I validate it ?22:35
magicalChickenrharper: the way setup works right now is yuo can execute a script or use one of the setup args22:35
powersjrharper: I typically build the deb and specify it22:35
rharperpowersj: urg, really ?22:36
magicalChickenrharper: I normally use --ppa "ppa:cloud-init-dev/daily"22:36
rharperthat's a long round-trip22:36
rharpermagicalChicken: well, you were creating tests for existing code22:36
magicalChickenrharper: you don't have to build a new deb just to test a new test case22:36
rharperI'm fixing a bug22:36
rharperso I need to run my fixed code; would prefer not to have a ppa build in the iteration loop, no ?22:36
magicalChickenI'd just push to a ppa22:36
magicalChickenOh22:36
rharper-1 =P22:37
magicalChickenI can set a script up to do a build automatically22:37
powersjI get the need for fast turn around, but these aren't unit tests so I guess I can accept some time required. Maybe a full ppa is too much then?22:37
rharperso, we're doing this via tox; so let me figure out what cloud-init-0.7.9.zip contains22:37
magicalChickenI think with tox, that zip will be the current contents of cloudinit/22:37
rharperif that's the current tree bits, then it's a matter of having the runner pick that up and inject that into the snapshot22:37
magicalChickenSince that's used for unittests as well22:37
rharpermagicalChicken: I hope so22:38
rharperyeah22:38
rharperthat makes sense22:38
* rharper checks .tox 22:38
magicalChickenrharper: the tricky part is automatically building though22:38
magicalChickenI've never managed to build cloud-init on my system22:38
rharperlooks like the zip is installed but not kept22:38
magicalChickenAlthough I guess using setup.py rather than building a deb would work22:39
magicalChickenrharper: If you modify a file then run tox -e flake8, it sees the change immediately22:39
magicalChickenSo I think its just the current tree22:39
rharperyeah22:39
rharperok, yeah, so pip has tree installed;  but we don't yet have a way of putting tree into the container22:41
magicalChickenrharper: I can add that to setup_image22:41
rharpermagicalChicken: so with proper deps on the host, package/bddeb should create a deb from current tree ... you're saying that doesn't work for you ?22:41
magicalChickenI've not managed to get bddeb working yet22:41
rharperpossibly due to your pip cruft ;  but on a clean setup (like in a container ) it should work22:41
magicalChickenYeah, it works fine in a ppa, so that should work22:42
rharperso, we could 1) copy in source into the container and then build cloud-init via tools/bddeb22:42
rharperthen inject that deb for the snapshot22:42
magicalChickenThat should work fine assuming that bddeb works22:42
rharperit should, and if we run bddeb in a container, that should make it repeatable22:42
magicalChickenThe other option is setup.py install, but that could be messy22:43
rharperit's probably pretty close though it will miss any pre-post stuff that dpkg did; (which I don't think is that much )22:43
magicalChickenYeah, I think dpkg mainly just does systemctl enable cloud-init22:44
magicalChickenI guess if I add the image sync feature then firing up containers is fast22:44
magicalChickenSo there could be a 'build container' that runs before the rest of the tests start22:44
magicalChickenIf the test suite is told to use current tree22:44
rharperright, I'd definitely sync the images needed;22:45
magicalChickenThat should be pretty straight forward, I guess general policy could be always leave around unmodified versions of images22:46
rharperthen you could copy in the src, into a build container;  once it's built and installed, snapshot that;  use that snapshot as the base for each test22:46
rharperwe might need to purge the build-deps and other things (that's going to add some startup cost)22:47
rharperbut I think for the developer path; that's not bad22:47
magicalChickenThe build container could even be separate from the test containers22:47
rharperfor the normal ci path, pointing at PPA or repo is fine22:47
rharperright22:47
magicalChickenJust build, copy the deb back into the host, then nuke it22:47
rharperthat's also possible22:47
rharperbuild once, copy out, then run your normal 'use this cloud-init.deb'22:48
magicalChickenYeah, for ci it should work well to just use a ppa22:48
rharpercool22:48
magicalChickenThat would also make it possible to just build 1 deb for all ubuntu images too22:48
rharperyeah22:48
rharperit's python22:48
magicalChickenOnly issue is trusty would need py222:48
rharperbddeb takes flags22:49
magicalChickenbut trusty can just be run separately22:49
rharperfor python2 or python322:49
rharperyou could run the build twice22:49
rharperand generate the 2 and 3 versions, copyout etc22:49
magicalChickenSimplest way might be just to add a new command to test suite to build a deb in a clean container22:49
magicalChickenand then a script that does that, then runs the test suite with the given args22:50
rharpersure;  do we have a way of chaining these commands?  right now, I just call the 'run' command22:50
magicalChickenYou can tell 'run' to use multiple distros in 1 go22:51
magicalChickenlike 'run -n xenial -n yakkety -n zesty -n stretch'22:52
magicalChickenalso multiple tests and multiple platforms22:52
powersjrharper: thanks for the tox starter :) here is a simpler version that works for me and an example of passing in arguments https://paste.ubuntu.com/23773398/23:34
powersjnow as you suggested if we combine that with building the local checkout in a 'build' container and you should have fast tests then for easier development ;)23:36
magicalChickenpowersj: nice, I'll pull that + rharper's tox config into the devel branch23:37
powersjmagicalChicken: cool, really there are only 2 differences 1) is you only need pylxd specified to run, if we want to lock down other requirements we can and 2) the default arguments23:38
magicalChickenI'm going to try getting the build container + 'run from current tree' script put together tonight23:38
powersjin mine {posargs:-n xenial} basically means run with -n xenial unless you get something else and in that case run that23:38
powersjok23:38
magicalChickenRight yeah, passing in args is definitely nice there23:39
powersjI'm actually surprised tox worked so well... I could swear you and I tried it and it failed terribly23:39
magicalChickenI'll try to base that off of the pylxd 2.1.3 version so that it doesn't have to wait on pylxd being fixed23:39
magicalChickenpowersj: I'm pretty sure I had tried it before too23:39
magicalChickenI may have just done something wrong with the environment though23:39
powersjmagicalChicken: any concerns about doing this for when we add KVM backend?23:40
powersjstarting to look at that is what I hope to do tomorrow23:40
magicalChickenhmm23:40
powersjmake sure we don't back ourselves into a corner now23:41
magicalChickenwell, its not going to affect the actual cli23:41
powersj(you can think about it) ;)23:41
magicalChickenso we can always just switch bach23:41
magicalChickenWe're going to have to have root permission to modify the images for kvm23:41
magicalChickenAlthough I may be able to rig something up where that happens inside of lxd :)23:42
powersjlol23:42
magicalChickenI have wanted to do that for a while, so vmtests can stop needing root23:42
magicalChickenSomething like a fuse mount of a directory inside lxd where the image has been mount-image-callbacked23:43

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!