[00:04] bigjools: ti is a recommends of python-bson [00:04] which would get installed as part of bson's installation i think [00:06] ok === freeflying is now known as freeflying_away [01:55] smoser: Do you know why cloud-tools wants to remove ubuntu-minimal? https://pastebin.canonical.com/98848/ [02:00] jpds, don't know. but i'd like to knwo. same with dhcp-client doesn't make any senes. [02:00] will investigate manana if you dont want to now [02:01] I can now. [02:01] If you want some apt-cache outputs ror something. [02:22] smoser: current fpi image is 12.04.1, not 12.04.3? [02:27] danwest, we can/should update that. [02:27] we really need to get that on a 3 week like cadence. [02:28] smoser: k, just noticed it [02:28] .X is really garbage [02:28] it does show its old [02:28] but .X versus .Y doesn't mean anything. [02:35] jpds, http://paste.ubuntu.com/6216432/ [02:35] thats the reason [02:35] smoser: Nice. [02:39] jtv, or bigjools, would you please take a cursory look at http://paste.ubuntu.com/6216361/ [02:39] Looking... [02:39] it still has printfs in it. but some of tha tis important [02:39] i think right now when we ryn 'sync' it will end up re-extrracting the tarball pointlessly [02:39] and has no status on download at all. [02:39] What am I looking at? [02:40] patch to mass-import-ephemerals that adds [02:40] * some bad status on stderr [02:40] * not needlessly extracting a 250M tarball multiple times [02:40] * basically says "if stuff is in place, its good" [02:40] and dont put files in place until they're good. [02:41] Great. [02:42] I think there's still some debug output in there... [02:42] Of course tests will need updating. [02:42] yeah i said that :) [02:42] printf debugging still there [02:43] Not what you said. [02:43] But it falls into place when you word it that way. [02:43] yeah. [02:44] jpds, would you mind opening a bug ? [02:44] with our discussion above ? [02:44] i'll now upload iproute2 to staging [02:44] or... to -next [02:44] and i think that will actually solve our problem [02:44] smoser: In a sec, doing a live MAAS demo. [02:45] It's a lot of changes for one branch. If feasible, I'd advise breaking it down into smaller branches. [02:45] jtv, yeah. [02:45] smoser: Bug where? [02:45] you can open against cloud-archive project [02:45] Will do. [02:48] jtv, the other hting that we're doing now that we really dont need to do is actually caching the .tar.gz files that are downloaded. === freeflying_away is now known as freeflying [02:48] that increases the size requirements for /var/lib/maas [02:48] we dont need them really. its a side affect of using the mirror that we're using. [02:51] I was wondering about that. Those files are huge. [02:51] It's been complicating testing, even. [02:52] Will the mirror writer still function well when we delete those files? [02:52] I mean, it won't re-download images we already have, that sort of thing? [02:55] The docstring for extract_image_tarball needs updating. It's hard to figure out these changes from code without any explanation, but it's especially hard for the next person maintaining the code because they'll get documentation that is basically a lie! [02:58] smoser: https://bugs.launchpad.net/cloud-archive/+bug/1237751 [02:58] Ubuntu bug 1237751 in ubuntu-cloud-archive "cloud-tools removes ubuntu-minimal" [Undecided,New] [03:03] smoser: are your fixes for import-ephemerals hitting trunk? [03:15] smoser: another note about the diff, besides "document!": we normally require boolean tests to be explicit about what they test, e.g. "if name is not None:" or "if len(items) == 0:" instead of just "if name" or "if not items" respectively. [03:20] jtv: is anything else landing nowish? [03:31] roaksoax: not that I know. [03:31] jtv: ok, just uploaded maas anyway [03:32] There are two approved branches, from Gavin & Raphaël respectively. I could see if they need immediate landing. [03:34] Gavin's is irrelevant for production, and Raphaël may still want to consider options for his branch. [03:36] smoser: don't forget to document dlstatus()... it also needs a clearer name, preferably a verby one! [03:41] smoser: By the way, can you explain what "flat" is, in insert_item? It gets returned from from util.products_exdata. [03:42] Oh, and don't forget to run "make lint" — it catches some of the most obvious violations of coding standards. === freeflying is now known as freeflying_away === freeflying_away is now known as freeflying [04:19] jtv: any outstandng problems that you know of? [04:22] bigjools: no, I was waiting to ask you the same thing. Scott has a patch he's working on. [04:22] there's allenap's bson problem [04:22] that's all I know [04:27] jtv: if you're up for a technical challenge, see bug 1237615 [04:27] bug 1237615 in MAAS "python-bson-ext does not encode binary in Apache with mod_wsgi" [Critical,Triaged] https://launchpad.net/bugs/1237615 [04:32] Slightly under the weather, probably post-deadline letdown, but I'll give it a look. [04:47] jtv: it's possible the garage maas is buggered - first st [04:47] jtv: it's possible the garage maas is buggered - first step is to recreate === freeflying is now known as freeflying_away [05:45] bigjools: afaict we're not actually implementing the workaround from that blog article — we have WSGIApplicationGroup %{GLOBAL} but not WSGIProcessGroup %{GLOBAL}. [05:46] I still feel I'm missing something though... If we're running only one application, what other interpreter in the same process is using the bson extension? === CyberJacob|Away is now known as CyberJacob === freeflying_away is now known as freeflying [06:29] jtv: threads [06:29] it broke in other ways without the WSGIApplicationGroup too [07:20] bigjools: it says threads=1, so that's part of the picture I'm missing. But I'm not saying WSGIApplicationGroup %{GLOBAL} looks wrong, just that we don't have the workaround the blog article suggested. [07:56] jtv: hmmm [07:56] it also says processes=2 [08:00] roaksoax: so is the bson issue resolved? [08:00] and rvba ^ ? [08:00] is bson biting? [08:01] is it from oops, or something else that you're using bson for? [08:01] lifeless: python-bson and wsgi don't get along [08:01] https://launchpad.net/bugs/1237615 [08:01] Ubuntu bug 1237615 in MAAS "python-bson-ext does not encode binary in Apache with mod_wsgi" [Critical,Triaged] [08:01] bigjools: wsgi in general, or mod_wsgi.. ah [08:02] bigjools: I don't know, allenap was looking into it… Gavin ^? [08:03] rvba: the reason I ask is because of the last comment you made on the email thread [08:03] "Andres figured it out… I tested his fix in the lab and it worked! " [08:05] bigjools: right, but the title of the email says "Commissioning with a Saucy image sets node status to "Failed tests"" and the bug referenced in there is bug 1237364 . [08:05] bug 1237364 in maas (Ubuntu) "Commissioning with a Saucy image sets node status to "Failed tests"" [Critical,Confirmed] https://launchpad.net/bugs/1237364 [08:06] rvba: urgh right [08:23] see this? https://bugs.launchpad.net/bugs/1237615 [08:23] Ubuntu bug 1237615 in MAAS "python-bson-ext does not encode binary in Apache with mod_wsgi" [Critical,Triaged] [08:23] argh [08:23] https://bugs.launchpad.net/bugs/1237615 [08:23] no [08:23] paste fail [08:24] https://bugs.launchpad.net/bugs/1237159 [08:24] that [08:24] Ubuntu bug 1237159 in MAAS "MAAS environment can not be bootstrapped or destroyed with Declared or Commissioning node" [Undecided,New] [08:24] rvba: ^ ? [08:25] Looking… [08:26] looks like the "ownership" bug combined with the "cruft left in the filestore" bug [08:27] "cruft left in the filestore"? [08:28] I just think juju is trying to release all the nodes it can see (the juju-core bug) and some of the nodes' status don't allow that transition. [08:28] yeah [08:45] bigjools: Are you stlil around? [08:47] MAAS is doing something quite special. [08:50] Actually, no, that's just very confusing. === CyberJacob is now known as CyberJacob|Away === CyberJacob|Away is now known as CyberJacob [12:19] jpds, did you open me a bug ? [12:24] smoser: Yes. [12:24] oh? [12:24] i didn't see it. and just opened one myself. [12:24] smoser: https://bugs.launchpad.net/cloud-archive/+bug/1237751 [12:24] Ubuntu bug 1237751 in ubuntu-cloud-archive "cloud-tools removes ubuntu-minimal" [Undecided,Confirmed] [12:31] jpds, just copied iproute2 from -staging to -proposed. [12:31] it has to re-build there, but then wlil land. [12:32] smoser: Cool. [13:26] allenap: howdy [13:26] allenap: did you test the wsgi issue in a clean environment ? [13:27] allenap: and unfortunately we cant do anything about pyhon-bson-ext being installed [13:27] it will be [13:29] roaksoax: I tried it in the garage maas. Moving _cbson.so out of the way (and restarting apache) fixed it. fixed it [13:30] allenap: but what was the issue in the first placr.. the bug doesnt really say [13:31] allenap: i would tests in a clean environment [13:31] roaksoax: mod_wsgi uses a simplified threading model that not all python c extentions are compatible with [13:31] symptom is generally segfaults [13:31] mgz: right [13:32] but my question is... do maas webui dies? does celery die? [13:34] roaksoax: Nothing so obvious. It encodes binary data as UTF-8 strings, which causes an exception in the celery worker on the cluster, thus preventing tag expressions from being recalculated en masse. [13:35] allenap: can you provide an exact way to reproduce and see how it fails to donso(we will need it eventuslly fornsru if we find a solution) but would like to test in a clean environment [13:36] it would not be the first time i see issues doe tue a broken environment [13:36] issues due to* [13:39] thanks ;) [13:49] roaksoax: I'll try to reproduce in a dev environment. [14:27] allenap: thank you! [14:33] re: bug https://bugs.launchpad.net/maas/+bug/1237723, going to start work on this today, would like some feedback on comment #3 [14:33] Ubuntu bug 1237723 in MAAS "preseed documentation is outdated and doesn't apply to latest curt installer bits" [High,Triaged] === freeflying is now known as freeflying_away === freeflying_away is now known as freeflying === freeflying is now known as freeflying_away [15:33] so should the maas api be exposing things like network information (vlans) ? [15:34] i would think we'd need some sort of agent on the machines to respond to those types of requests [15:35] i did something like this before using qpid [15:35] amqp [15:36] stokachu: Currently it exposes the routers to which a machine is connected, as discovered by LLDP. It also records the LLDP XML as produced from lldpctl in the lldpd package which can be used for tag expressions. [15:36] allenap: ok im not entirely familiar with lldp so ill need to research that [15:37] stokachu: When I say "currently", I mean as of this week :) [15:37] ahh [15:38] allenap: im going to go digging into the code today to make sense of it [15:41] stokachu: We are behind with documenting it, so please ask if you have questions. To the maas-devel mailing list is probably best, so that we have a record and others see it. [15:46] allenap: ok will do thanks! [16:11] allenap: awesome lldp gathers the vlan name [16:11] so i can work to expose that information b/c so far i think maas only exposes the mac address [16:25] allenap: so have you guys tested DHCP with portfast + LLDP? [16:26] allenap: in real world escenarios we've seen that that portfast prevents nodes from obtaining DHCP [16:27] roaksoax: I haven't, and I don't think rvba or anyone else has. [16:28] that's probably gonna conflict wouldn't it? since I read somewhere that the lldp needs portfast enabled [16:47] do you guys have switches to test against [16:47] i dont have any of that hardware and im not sure if openvswitch would emulate vlans in the same manner as cisco [16:59] * roaksoax bbl [17:12] matsubara, [17:12] when the QA lab tests, does it test with released images or dailies ? [17:15] smoser, it uses the image from: CLOUDIMGURL=http://cloud-images.ubuntu.com/$RELEASE/$BUILDID as the testbed [17:15] matsubara, i'm sorry. i meant the ephemeral images. [17:16] maas.ubuntu.com/images [17:22] smoser, hmm we have a local mirror of maas.ubuntu.com/images here: http://10.98.0.13/mirrors/maas-ephemeral/, then the tests configure /etc/maas/import_ephemerals to use that mirror. (See http://bazaar.launchpad.net/~maas-maintainers/maas/qa-lab-tests/view/head:/utils.py lines 70 to 92). The test used to set up STREAM=daily but that's been disabled a long time ago as you can see by the XXX there [17:33] matsubara, could i get a run done with daily ? [17:34] smoser, sure. let me trigger that for you. give me a few [17:34] k. thanks. [17:35] roaksoax, how does maas-import-ephemerals get run ? [17:35] smoser, with the saucy maas package, I assume? [17:35] matsubara, sure [17:38] maas-imoort-pxe-files runs it [17:39] smoser ^^ [17:40] roaksoax, when does that get run [17:40] smoser, Currently rvba left a job running (#90) but I queued a test run with the STREAM=daily enabled. It's going to be build #91 on jenkins: http://10.189.74.2:8080/job/saucy-adt-maas-manual/ [17:41] smoser, I'll keep an eye on it and let you know how it goes [17:41] matsubara, thanks. [17:41] np [17:43] smoser: i font think we have a cronjob anymore [17:44] Hooray for us. [18:27] smoser, the test run you asked is running: http://10.189.74.2:8080/job/saucy-adt-maas-manual/91/ARCH=amd64,label=lenovo-RD230-01/console, looks like the STREAM=daily didn't cause any problems and the tests are running the juju tests now. You can inspect the test instance running MAAS here: http://10.98.0.90/MAAS/nodes/ [18:29] I don't remember exactly the reason why STREAM=daily was commented out from import_ephemerals but I think there's a small window between when a release is made and the new devel images are avaialble in maas.ubuntu.com/images [18:29] need to ask rvba about that as it was his XXX :-) [18:32] matsubara, there were bugs in the images. [18:33] mainly i think before https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1050487 was backported [18:33] Ubuntu bug 1050487 in open-iscsi (Ubuntu Precise) "resolvconf not updated on iscsi root" [Medium,Fix released] [18:58] smoser, tests passed. should I MP the change to re-enable the STREAM=daily on import_ephemerals? I kicked off test runs for precise, quantal and raring too just to be sure. [18:59] matsubara, i think i'd prefer that, yeah. [18:59] it probably only acutally tests precise enlistment though, right? [19:00] smoser, yes [19:00] ewll, for this purpose thats pretty good. [19:01] matsubara, would it be easy to tell it to use a different release ? [19:01] i dont recall how you tell it to use a different release for commissioning than precise [19:04] smoser, we have a NODE_SERIES variable but that's used to tell juju which release to use for deployment. If the maas cli allows us to change the main maas setting for commissioning, then it should be easy to do. [19:05] roaksoax, do you know ? [19:06] smoser, apparently it's possible, yes: http://maas.ubuntu.com/docs/configure.html#choosing-a-series-to-install [19:06] thats just the defaul treleease. [19:06] i dont know if it that is used for commissioning [19:07] hm.. [19:09] matsubara, well, its release=Config.objects.get_config('commissioning_distro_series')) [19:19] well, looks like that's setable using the maas-cli: $ maas-cli maas maas get-config name="commissioning_distro_series" [19:19] "precise" [19:23] nice. [19:29] cool. I added a new COMMISSIONING_SERIES var to the test (Default to precise) which allows us to run the tests with MAAS commissioning with different releases. I'll MP and ask for a review from the maas guys [19:33] matsubara: commissioning has it owns drfault variable so does deployment [19:33] juju passes fhe release for deployment [19:33] yeah. we saw. and go tthat right. [19:34] if juju does pass it then it uses default or the one set [19:34] juju [19:34] thats why we need default-series in the environment for juju [19:34] there are other things than juju ;) [19:34] in environments.yaml [19:34] we were talking about commissioning though [19:35] which has nothing to do with juju [19:35] (i know, roaksoax, saying something has nothing to do with juju is shameful to you) [19:35] commissioning you only need to set the variable [19:36] smoser: haha it is not was just trying to give background info [19:47] roaksoax: I've got a simple fix for the bson bug. [19:48] roaksoax: I've confirmed it works on the Garage MAAS, and I have a reproducible example (see https://bugs.launchpad.net/ubuntu/+source/pymongo/+bug/1237615). [19:48] Ubuntu bug 1237615 in MAAS "python-bson-ext does not encode binary in Apache with mod_wsgi" [Critical,Triaged] [19:48] roaksoax: I'll put up a branch for review in a few seconds. [19:50] roaksoax: https://code.launchpad.net/~allenap/maas/fix-wsgi-application-group/+merge/190458 [19:51] allenap: cool thanks [20:06] roaksoax: If you review it, I land it, can we rebuild and upload in time? [20:07] allenap: so what's the regression potential of doing this "application-group=%{GLOBAL}" [20:07] allenap: i mean, could that affect how other things such as logs? or anything else that requires everything under wsgi to run under the maas user/grpup [20:08] roaksoax: I honestly don't know. The change ensures that the wsgi.py script is imported in the first Python interpreter to be created instead of a later one. My guess is that this is what we wanted all along anyway. [20:09] roaksoax: Right now tag recalculation isn't working at all :-/ [20:12] allenap: right, so we are between the sword and the wall... [20:13] roaksoax: Indeed! [20:13] allenap: let's get that tested in many ways possible and if there's no apparent issue, let's merge it [20:13] rvba: Are you able to help test a new package in the lab? [20:14] allenap: we will need to test both upgrades and fresh installs [20:14] roaksoax: I don't have the means to test those. matsubara, do you? [20:15] * matsubara reads backlog [20:15] I can test on the qa lab [20:15] would that help? [20:17] we need to check that everything starts, that all the logs are written [20:17] we need to make sure no errors are present in apache logs [20:17] nor maas logs [20:17] power commands [20:17] allenap, roaksoax: what's the package that needs fresh install testing? And what upgrade path do you want me to test? raring to saucy and or precise to saucy (not even sure if that's possible)? [20:17] everything that involes wsgi [20:17] matsubara: saucy archives, to saucy fix [20:19] roaksoax: Can you +1 https://code.launchpad.net/~allenap/maas/fix-wsgi-application-group/+merge/190458 and I'll start the ball rolling. [20:19] the lab is currently tied up running some tests for smoser, once that's done (should be done in a couple of hours) then I can do the testing [20:19] which will give you guys time to land and get the recipe going to build a new package, right? [20:25] matsubara: Should be. Thanks Diogo, that's great. [20:32] matsubara, roaksoax: I've requested a built at https://code.launchpad.net/~julian-edwards/+recipe/maas-daily-saucy [20:34] allenap, there's a message there saying it couldn't build because sp was superseded. is this related to build you requested? [20:34] matsubara: I think that's an old message. [20:35] allenap, ok [20:50] roaksoax: That built fine, but https://code.launchpad.net/~maas-maintainers/+archive/dailybuilds says a newer version is in the distro. How can I fix this? [21:07] allenap: oh hold on [21:09] allenap: ok changes to packaging should land in a few minutes [21:09] allenap: https://code.launchpad.net/~andreserl/maas/packaging_bzr1693/+merge/190488 [21:11] roaksoax: When they're in I'll kick off a new build. [21:11] allenap: merged, go for it [21:11] allenap: i need to leave now, will be back tonight. Just email me an update and I'll do my share of testing when I get back [21:13] roaksoax: That's brilliant. It's bedtime for me, so I'll speak to you tomorrow. [21:14] allenap: alright! thanks for thr hard work [21:14] roaksoax: You too :) [21:15] allenap, could you cc me too? would be helpful to have some pointers on what I need to test for [21:17] allenap, roaksoax: from the conversation above, I'm thinking of doing: start a new saucy testbed, install the saucy package from the archive, upgrade it to the PPA version and then check logs, web ui. Anything else I should check for? [21:17] matsubara: The thing that has to work is tag expression recalculation. When you define a new tag, or update and existing tag, containing an XPath expression, jobs get sent to the clusters asking them to do the work. The jobs on the clusters were crashing with ValueError (https://bugs.launchpad.net/maas/+bug/1237463). [21:17] Ubuntu bug 1237463 in MAAS "ValueError using lxml when evaluating tags in worker" [Critical,In progress] [21:19] allenap, cool. thanks for the bug. [21:19] matsubara: What you need is a node that has completed commissioning (and so has lshw and/or lldp XML recorded against it) and then add a new tag with an expression (the expression "true()" - without quotes - is enough to test this). [21:21] allenap, perfect. thank you [21:36] matsubara: 1.4+bzr1693+dfsg-0+1695+210~ppa0~ubuntu13.10.1 is now available in ppa:maas-maintainers/dailybuilds. [21:37] allenap, the lab should be available soon and will get the testing started === CyberJacob is now known as CyberJacob|Away [21:37] matsubara: Tip top, thanks again. I'm going to get some sleep now. Cheerio until tomorrow. [21:38] allenap, you're welcome. good night! [22:43] morning === freeflying_away is now known as freeflying [23:34] roaksoax: why add power/moonshot.template which is an almost carbon copy of ipmi? [23:35] it has an old bug in it (bug 1171418) [23:35] bug 1171418 in maas (Ubuntu Quantal) "MAAS fails to power up machines when trying to install nodes" [High,New] https://launchpad.net/bugs/1171418 [23:37] bigjools: ipmitool? [23:37] moonshot uses ipmitool [23:38] bigjools: besides we need a template named moonshot eithwrway [23:38] really? [23:38] to use thst power method [23:38] yep really [23:38] why not make it generic "ipmitool" [23:38] either way, it has a bug [23:39] and I am sure we can refactor it with ipmipower [23:39] * bigjools files bug [23:39] bigjools: it does not worl exactly as the one using ipmipower [23:39] roaksoax: what are the main differences? [23:39] the behavior is different [23:40] bigjools: t [23:40] wwhat moonshot returns on ipmi commands [23:40] on ipmi power we use cycle [23:40] ipmitool doesnt [23:41] :/ [23:42] yeah it sucks [23:42] but btw when the power typenis set to "moonshot" the code looks for a template cslled moonshot.template [23:43] anyway im off [23:44] roaksoax: ok fair enough... it's a shame. Thanks for getting loads of stuff fixed, tremendous job!