/srv/irclogs.ubuntu.com/2013/10/10/#maas.txt

roaksoaxbigjools:  ti is a recommends of python-bson00:04
roaksoaxwhich would get installed as part of bson's installation i think00:04
bigjoolsok00:06
=== freeflying is now known as freeflying_away
jpdssmoser: Do you know why cloud-tools wants to remove ubuntu-minimal? https://pastebin.canonical.com/98848/01:55
smoserjpds, don't know. but i'd like to knwo. same with dhcp-client doesn't make any senes.02:00
smoserwill investigate manana if you dont want to now02:00
jpdsI can now.02:01
jpdsIf you want some apt-cache outputs ror something.02:01
danwestsmoser: current fpi image is 12.04.1, not 12.04.3?02:22
smoserdanwest, we can/should update that.02:27
smoserwe really need to get that on a 3 week like cadence.02:27
danwestsmoser: k, just noticed it02:28
smoser.X is really garbage02:28
smoserit does show its old02:28
smoserbut .X versus .Y doesn't mean anything.02:28
smoserjpds, http://paste.ubuntu.com/6216432/02:35
smoserthats the reason02:35
jpdssmoser: Nice.02:35
smoserjtv, or bigjools, would you please take a cursory look at http://paste.ubuntu.com/6216361/02:39
jtvLooking...02:39
smoserit still has printfs in it. but some of tha tis important02:39
smoseri think right now when we ryn 'sync' it will end up re-extrracting the tarball pointlessly02:39
smoserand has no status on download at all.02:39
jtvWhat am I looking at?02:39
smoserpatch to mass-import-ephemerals that adds02:40
smoser * some bad status on stderr02:40
smoser * not needlessly extracting a 250M tarball multiple times02:40
smoser * basically says "if stuff is in place, its good"02:40
smoserand dont put files in place until they're good.02:40
jtvGreat.02:41
jtvI think there's still some debug output in there...02:42
jtvOf course tests will need updating.02:42
smoseryeah i said that :)02:42
smoserprintf debugging still there02:42
jtvNot what you said.02:43
jtvBut it falls into place when you word it that way.02:43
smoseryeah.02:43
smoserjpds, would you mind opening a bug ?02:44
smoserwith our discussion above ?02:44
smoseri'll now upload iproute2 to staging02:44
smoseror... to -next02:44
smoserand i think that will actually solve our problem02:44
jpdssmoser: In a sec, doing a live MAAS demo.02:44
jtvIt's a lot of changes for one branch.  If feasible, I'd advise breaking it down into smaller branches.02:45
smoserjtv, yeah.02:45
jpdssmoser: Bug where?02:45
smoseryou can open against cloud-archive project02:45
jpdsWill do.02:45
smoserjtv, the other hting that we're doing now that we really dont need to do is actually caching the .tar.gz files that are downloaded.02:48
=== freeflying_away is now known as freeflying
smoserthat increases the size requirements for /var/lib/maas02:48
smoserwe dont need them really. its a side affect of using the mirror that we're using.02:48
jtvI was wondering about that.  Those files are huge.02:51
jtvIt's been complicating testing, even.02:51
jtvWill the mirror writer still function well when we delete those files?02:52
jtvI mean, it won't re-download images we already have, that sort of thing?02:52
jtvThe docstring for extract_image_tarball needs updating.  It's hard to figure out these changes from code without any explanation, but it's especially hard for the next person maintaining the code because they'll get documentation that is basically a lie!02:55
jpdssmoser: https://bugs.launchpad.net/cloud-archive/+bug/123775102:58
ubot5Ubuntu bug 1237751 in ubuntu-cloud-archive "cloud-tools removes ubuntu-minimal" [Undecided,New]02:58
roaksoaxsmoser: are your fixes for import-ephemerals hitting trunk?03:03
jtvsmoser: another note about the diff, besides "document!": we normally require boolean tests to be explicit about what they test, e.g. "if name is not None:" or "if len(items) == 0:" instead of just "if name" or "if not items" respectively.03:15
roaksoaxjtv: is anything else landing nowish?03:20
jtvroaksoax: not that I know.03:31
roaksoaxjtv: ok, just uploaded maas anyway03:31
jtvThere are two approved branches, from Gavin & Raphaël respectively.  I could see if they need immediate landing.03:32
jtvGavin's is irrelevant for production, and Raphaël may still want to consider options for his branch.03:34
jtvsmoser: don't forget to document dlstatus()... it also needs a clearer name, preferably a verby one!03:36
jtvsmoser: By the way, can you explain what "flat" is, in insert_item?  It gets returned from from util.products_exdata.03:41
jtvOh, and don't forget to run "make lint" — it catches some of the most obvious violations of coding standards.03:42
=== freeflying is now known as freeflying_away
=== freeflying_away is now known as freeflying
bigjoolsjtv: any outstandng problems that you know of?04:19
jtvbigjools: no, I was waiting to ask you the same thing.  Scott has a patch he's working on.04:22
bigjoolsthere's allenap's bson problem04:22
bigjoolsthat's all I know04:22
bigjoolsjtv: if you're up for a technical challenge, see bug 123761504:27
ubot5bug 1237615 in MAAS "python-bson-ext does not encode binary in Apache with mod_wsgi" [Critical,Triaged] https://launchpad.net/bugs/123761504:27
jtvSlightly under the weather, probably post-deadline letdown, but I'll give it a look.04:32
bigjoolsjtv: it's possible the garage maas is buggered - first st04:47
bigjoolsjtv: it's possible the garage maas is buggered - first step is to recreate04:47
=== freeflying is now known as freeflying_away
jtvbigjools: afaict we're not actually implementing the workaround from that blog article — we have WSGIApplicationGroup %{GLOBAL} but not WSGIProcessGroup %{GLOBAL}.05:45
jtvI still feel I'm missing something though...  If we're running only one application, what other interpreter in the same process is using the bson extension?05:46
=== CyberJacob|Away is now known as CyberJacob
=== freeflying_away is now known as freeflying
bigjoolsjtv: threads06:29
bigjoolsit broke in other ways without the WSGIApplicationGroup too06:29
jtvbigjools: it says threads=1, so that's part of the picture I'm missing.  But I'm not saying WSGIApplicationGroup %{GLOBAL} looks wrong, just that we don't have the workaround the blog article suggested.07:20
bigjoolsjtv: hmmm07:56
bigjoolsit also says processes=207:56
bigjoolsroaksoax: so is the bson issue resolved?08:00
bigjoolsand rvba ^ ?08:00
lifelessis bson biting?08:00
lifelessis it from oops, or something else that you're using bson for? </idlecuriosity>08:01
bigjoolslifeless: python-bson and wsgi don't get along08:01
bigjoolshttps://launchpad.net/bugs/123761508:01
ubot5Ubuntu bug 1237615 in MAAS "python-bson-ext does not encode binary in Apache with mod_wsgi" [Critical,Triaged]08:01
lifelessbigjools: wsgi in general, or mod_wsgi.. ah08:01
rvbabigjools: I don't know, allenap was looking into it… Gavin ^?08:02
bigjoolsrvba: the reason I ask is because of the last comment you made on the email thread08:03
bigjools"Andres figured it out… I tested his fix in the lab and it worked! "08:03
rvbabigjools: right, but the title of the email says "Commissioning with a Saucy image sets node status to "Failed tests"" and the bug referenced in there is bug 1237364 .08:05
ubot5bug 1237364 in maas (Ubuntu) "Commissioning with a Saucy image sets node status to "Failed tests"" [Critical,Confirmed] https://launchpad.net/bugs/123736408:05
bigjoolsrvba: urgh right08:06
bigjoolssee this? https://bugs.launchpad.net/bugs/123761508:23
ubot5Ubuntu bug 1237615 in MAAS "python-bson-ext does not encode binary in Apache with mod_wsgi" [Critical,Triaged]08:23
bigjoolsargh08:23
bigjoolshttps://bugs.launchpad.net/bugs/123761508:23
bigjoolsno08:23
bigjoolspaste fail08:23
bigjoolshttps://bugs.launchpad.net/bugs/123715908:24
bigjoolsthat08:24
ubot5Ubuntu bug 1237159 in MAAS "MAAS environment can not be bootstrapped or destroyed with Declared or Commissioning node" [Undecided,New]08:24
bigjoolsrvba: ^ ?08:24
rvbaLooking…08:25
bigjoolslooks like the "ownership" bug combined with the "cruft left in the filestore" bug08:26
rvba"cruft left in the filestore"?08:27
rvbaI just think juju is trying to release all the nodes it can see (the juju-core bug) and some of the nodes' status don't allow that transition.08:28
bigjoolsyeah08:28
jpdsbigjools: Are you stlil around?08:45
jpdsMAAS is doing something quite special.08:47
jpdsActually, no, that's just very confusing.08:50
=== CyberJacob is now known as CyberJacob|Away
=== CyberJacob|Away is now known as CyberJacob
smoserjpds, did you open me a bug ?12:19
jpdssmoser: Yes.12:24
smoseroh?12:24
smoseri didn't see it. and just opened one myself.12:24
jpdssmoser: https://bugs.launchpad.net/cloud-archive/+bug/123775112:24
ubot5Ubuntu bug 1237751 in ubuntu-cloud-archive "cloud-tools removes ubuntu-minimal" [Undecided,Confirmed]12:24
smoserjpds, just copied iproute2 from -staging to -proposed.12:31
smoserit has to re-build there, but then wlil land.12:31
jpdssmoser: Cool.12:32
roaksoaxallenap: howdy13:26
roaksoaxallenap: did you test the wsgi issue in a clean environment ?13:26
roaksoaxallenap: and unfortunately we cant do anything about pyhon-bson-ext being installed13:27
roaksoaxit will be13:27
allenaproaksoax: I tried it in the garage maas. Moving _cbson.so out of the way (and restarting apache) fixed it. fixed it13:29
roaksoaxallenap: but what was the issue in the first placr.. the bug doesnt really say13:30
roaksoaxallenap: i would tests in a clean environment13:31
mgzroaksoax: mod_wsgi uses a simplified threading model that not all python c extentions are compatible with13:31
mgzsymptom is generally segfaults13:31
roaksoaxmgz: right13:31
roaksoaxbut my question is... do maas webui dies? does celery die?13:32
allenaproaksoax: Nothing so obvious. It encodes binary data as UTF-8 strings, which causes an exception in the celery worker on the cluster, thus preventing tag expressions from being recalculated en masse.13:34
roaksoaxallenap: can you provide an exact way to reproduce and see how it fails to donso(we will need it eventuslly fornsru if we find a solution) but would like to test in a clean environment13:35
roaksoaxit would not be the first time i see issues doe tue a broken environment13:36
roaksoaxissues due to*13:36
roaksoaxthanks ;)13:39
allenaproaksoax: I'll try to reproduce in a dev environment.13:49
roaksoaxallenap: thank you!14:27
stokachure: bug https://bugs.launchpad.net/maas/+bug/1237723, going to start work on this today, would like some feedback on comment #314:33
ubot5Ubuntu bug 1237723 in MAAS "preseed documentation is outdated and doesn't apply to latest curt installer bits" [High,Triaged]14:33
=== freeflying is now known as freeflying_away
=== freeflying_away is now known as freeflying
=== freeflying is now known as freeflying_away
stokachuso should the maas api be exposing things like network information (vlans) ?15:33
stokachui would think we'd need some sort of agent on the machines to respond to those types of requests15:34
stokachui did something like this before using qpid15:35
stokachuamqp15:35
allenapstokachu: Currently it exposes the routers to which a machine is connected, as discovered by LLDP. It also records the LLDP XML as produced from lldpctl in the lldpd package which can be used for tag expressions.15:36
stokachuallenap: ok im not entirely familiar with lldp so ill need to research that15:36
allenapstokachu: When I say "currently", I mean as of this week :)15:37
stokachuahh15:37
stokachuallenap: im going to go digging into the code today to make sense of it15:38
allenapstokachu: We are behind with documenting it, so please ask if you have questions. To the maas-devel mailing list is probably best, so that we have a record and others see it.15:41
stokachuallenap: ok will do thanks!15:46
stokachuallenap: awesome lldp gathers the vlan name16:11
stokachuso i can work to expose that information b/c so far i think maas only exposes the mac address16:11
roaksoaxallenap: so have you guys tested DHCP with portfast + LLDP?16:25
roaksoaxallenap: in real world escenarios we've seen that that portfast prevents nodes from obtaining DHCP16:26
allenaproaksoax: I haven't, and I don't think rvba or anyone else has.16:27
roaksoaxthat's probably gonna conflict wouldn't it? since I read somewhere that the lldp needs portfast enabled16:28
stokachudo you guys have switches to test against16:47
stokachui dont have any of that hardware and im not sure if openvswitch would emulate vlans in the same manner as cisco16:47
* roaksoax bbl16:59
smosermatsubara,17:12
smoserwhen the QA lab tests, does it test with released images or dailies ?17:12
matsubarasmoser, it uses the image from: CLOUDIMGURL=http://cloud-images.ubuntu.com/$RELEASE/$BUILDID as the testbed17:15
smosermatsubara, i'm sorry. i meant the ephemeral images.17:15
smosermaas.ubuntu.com/images17:16
matsubarasmoser, hmm we have a local mirror of maas.ubuntu.com/images here: http://10.98.0.13/mirrors/maas-ephemeral/, then the tests configure /etc/maas/import_ephemerals to use that mirror. (See http://bazaar.launchpad.net/~maas-maintainers/maas/qa-lab-tests/view/head:/utils.py lines 70  to 92). The test used to set up STREAM=daily but that's been disabled a long time ago as you can see by the XXX there17:22
smosermatsubara, could i get a run done with daily ?17:33
matsubarasmoser, sure. let me trigger that for you. give me a few17:34
smoserk. thanks.17:34
smoserroaksoax, how does maas-import-ephemerals get run ?17:35
matsubarasmoser, with the saucy maas package, I assume?17:35
smosermatsubara, sure17:35
roaksoaxmaas-imoort-pxe-files runs it17:38
roaksoaxsmoser ^^17:39
smoserroaksoax, when does that get run17:40
matsubarasmoser, Currently rvba left a job running (#90) but I queued a test run with the STREAM=daily enabled. It's going to be build #91 on jenkins: http://10.189.74.2:8080/job/saucy-adt-maas-manual/17:40
matsubarasmoser, I'll keep an eye on it and let you know how it goes17:41
smosermatsubara, thanks.17:41
matsubaranp17:41
roaksoaxsmoser: i font think we have a cronjob anymore17:43
smoserHooray for us.17:44
matsubarasmoser, the test run you asked is running: http://10.189.74.2:8080/job/saucy-adt-maas-manual/91/ARCH=amd64,label=lenovo-RD230-01/console, looks like the STREAM=daily didn't cause any problems and the tests are running the juju tests now. You can inspect the test instance running MAAS here: http://10.98.0.90/MAAS/nodes/18:27
matsubaraI don't remember exactly the reason why STREAM=daily was commented out from import_ephemerals but I think there's a small window between when a release is made and the new devel images are avaialble in maas.ubuntu.com/images18:29
matsubaraneed to ask rvba about that as it was his XXX :-)18:29
smosermatsubara, there were bugs in the images.18:32
smosermainly i think before https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1050487 was backported18:33
ubot5Ubuntu bug 1050487 in open-iscsi (Ubuntu Precise) "resolvconf not updated on iscsi root" [Medium,Fix released]18:33
matsubarasmoser, tests passed. should I MP the change to re-enable the STREAM=daily on import_ephemerals? I kicked off test runs for precise, quantal and raring too just to be sure.18:58
smosermatsubara, i think i'd prefer that, yeah.18:59
smoserit probably only acutally tests precise enlistment though, right?18:59
matsubarasmoser, yes19:00
smoserewll, for this purpose thats pretty good.19:00
smosermatsubara, would it be easy to tell it to use a different release ?19:01
smoseri dont recall how you tell it to use a different release for commissioning than precise19:01
matsubarasmoser, we have a NODE_SERIES variable but that's used to tell juju which release to use for deployment. If the maas cli allows us to change the main maas setting for commissioning, then it should be easy to do.19:04
smoserroaksoax, do you know ?19:05
matsubarasmoser, apparently it's possible, yes: http://maas.ubuntu.com/docs/configure.html#choosing-a-series-to-install19:06
smoserthats just the defaul treleease.19:06
smoseri dont know if it that is used for commissioning19:06
smoserhm..19:07
smosermatsubara, well, its             release=Config.objects.get_config('commissioning_distro_series'))19:09
matsubarawell, looks like that's setable using the maas-cli: $ maas-cli maas maas get-config name="commissioning_distro_series"19:19
matsubara"precise"19:19
smosernice.19:23
matsubaracool. I added a new COMMISSIONING_SERIES var to the test (Default to precise) which allows us to run the tests with MAAS commissioning with different releases. I'll MP and ask for a review from the maas guys19:29
roaksoaxmatsubara: commissioning has it owns drfault variable so does deployment19:33
roaksoaxjuju passes fhe release for deployment19:33
smoseryeah. we saw. and go tthat right.19:33
roaksoaxif juju does pass it then it uses default or the one set19:34
smoserjuju19:34
roaksoaxthats why we need default-series in the environment for juju19:34
smoserthere are other things than juju ;)19:34
roaksoaxin environments.yaml19:34
smoserwe were talking about commissioning though19:34
smoserwhich has nothing to do with juju19:35
smoser(i know, roaksoax, saying something has nothing to do with juju is shameful to you)19:35
roaksoaxcommissioning you only need to set the variable19:35
roaksoaxsmoser: haha it is not was just trying to give background info19:36
allenaproaksoax: I've got a simple fix for the bson bug.19:47
allenaproaksoax: I've confirmed it works on the Garage MAAS, and I have a reproducible example (see https://bugs.launchpad.net/ubuntu/+source/pymongo/+bug/1237615).19:48
ubot5Ubuntu bug 1237615 in MAAS "python-bson-ext does not encode binary in Apache with mod_wsgi" [Critical,Triaged]19:48
allenaproaksoax: I'll put up a branch for review in a few seconds.19:48
allenaproaksoax: https://code.launchpad.net/~allenap/maas/fix-wsgi-application-group/+merge/19045819:50
roaksoaxallenap: cool thanks19:51
allenaproaksoax: If you review it, I land it, can we rebuild and upload in time?20:06
roaksoaxallenap: so what's the regression potential of doing this "application-group=%{GLOBAL}"20:07
roaksoaxallenap: i mean, could that affect how other things such as logs? or anything else that requires everything under wsgi to run under the maas user/grpup20:07
allenaproaksoax: I honestly don't know. The change ensures that the wsgi.py script is imported in the first Python interpreter to be created instead of a later one. My guess is that this is what we wanted all along anyway.20:08
allenaproaksoax: Right now tag recalculation isn't working at all :-/20:09
roaksoaxallenap: right, so we are between the sword and the wall...20:12
allenaproaksoax: Indeed!20:13
roaksoaxallenap: let's get that tested in many ways possible and if there's no apparent issue, let's merge it20:13
allenaprvba: Are you able to help test a new package in the lab?20:13
roaksoaxallenap: we will need to test both upgrades and fresh installs20:14
allenaproaksoax: I don't have the means to test those. matsubara, do you?20:14
* matsubara reads backlog20:15
matsubaraI can test on the qa lab20:15
matsubarawould that help?20:15
roaksoaxwe need to check that everything starts, that all the logs are written20:17
roaksoaxwe need to make sure no errors are present in apache logs20:17
roaksoaxnor maas logs20:17
roaksoaxpower commands20:17
matsubaraallenap, roaksoax: what's the package that needs fresh install testing? And what upgrade path do you want me to test? raring to saucy and or precise to saucy (not even sure if that's possible)?20:17
roaksoaxeverything that involes wsgi20:17
roaksoaxmatsubara: saucy archives, to saucy fix20:17
allenaproaksoax: Can you +1 https://code.launchpad.net/~allenap/maas/fix-wsgi-application-group/+merge/190458 and I'll start the ball rolling.20:19
matsubarathe lab is currently tied up running some tests for smoser, once that's done (should be done in a couple of hours) then I can do the testing20:19
matsubarawhich will give you guys time to land and get the recipe going to build a new package, right?20:19
allenapmatsubara: Should be. Thanks Diogo, that's great.20:25
allenapmatsubara, roaksoax: I've requested a built at https://code.launchpad.net/~julian-edwards/+recipe/maas-daily-saucy20:32
matsubaraallenap, there's a message there saying it couldn't build because sp was superseded. is this related to build you requested?20:34
allenapmatsubara: I think that's an old message.20:34
matsubaraallenap, ok20:35
allenaproaksoax: That built fine, but https://code.launchpad.net/~maas-maintainers/+archive/dailybuilds says a newer version is in the distro. How can I fix this?20:50
roaksoaxallenap: oh hold on21:07
roaksoaxallenap: ok changes to packaging should land in a few minutes21:09
roaksoaxallenap: https://code.launchpad.net/~andreserl/maas/packaging_bzr1693/+merge/19048821:09
allenaproaksoax: When they're in I'll kick off a new build.21:11
roaksoaxallenap: merged, go for it21:11
roaksoaxallenap: i need to leave now, will be back tonight. Just email me an update and I'll do my share of testing when I get back21:11
allenaproaksoax: That's brilliant. It's bedtime for me, so I'll speak to you tomorrow.21:13
roaksoaxallenap: alright! thanks for thr hard work21:14
allenaproaksoax: You too :)21:14
matsubaraallenap, could you cc me too? would be helpful to have some pointers on what I need to test for21:15
matsubaraallenap, roaksoax: from the conversation above, I'm thinking of doing: start a new saucy testbed, install the saucy package from the archive, upgrade it to the PPA version and then check logs, web ui. Anything else I should check for?21:17
allenapmatsubara: The thing that has to work is tag expression recalculation. When you define a new tag, or update and existing tag, containing an XPath expression, jobs get sent to the clusters asking them to do the work. The jobs on the clusters were crashing with ValueError (https://bugs.launchpad.net/maas/+bug/1237463).21:17
ubot5Ubuntu bug 1237463 in MAAS "ValueError using lxml when evaluating tags in worker" [Critical,In progress]21:17
matsubaraallenap, cool. thanks for the bug.21:19
allenapmatsubara: What you need is a node that has completed commissioning (and so has lshw and/or lldp XML recorded against it) and then add a new tag with an expression (the expression "true()" - without quotes - is enough to test this).21:19
matsubaraallenap, perfect. thank you21:21
allenapmatsubara: 1.4+bzr1693+dfsg-0+1695+210~ppa0~ubuntu13.10.1 is now available in ppa:maas-maintainers/dailybuilds.21:36
matsubaraallenap, the lab should be available soon and will get the testing started21:37
=== CyberJacob is now known as CyberJacob|Away
allenapmatsubara: Tip top, thanks again. I'm going to get some sleep now. Cheerio until tomorrow.21:37
matsubaraallenap, you're welcome. good night!21:38
bigjoolsmorning22:43
=== freeflying_away is now known as freeflying
bigjoolsroaksoax: why add power/moonshot.template which is an almost carbon copy of ipmi?23:34
bigjoolsit has an old bug in it (bug 1171418)23:35
ubot5bug 1171418 in maas (Ubuntu Quantal) "MAAS fails to power up machines when trying to install nodes" [High,New] https://launchpad.net/bugs/117141823:35
roaksoaxbigjools: ipmitool?23:37
roaksoaxmoonshot uses ipmitool23:37
roaksoaxbigjools: besides we need a template named moonshot eithwrway23:38
bigjoolsreally?23:38
roaksoaxto use thst power method23:38
roaksoaxyep really23:38
bigjoolswhy not make it generic "ipmitool"23:38
bigjoolseither way, it has a bug23:38
bigjoolsand I am sure we can refactor it with ipmipower23:39
* bigjools files bug23:39
roaksoaxbigjools: it does not worl exactly as the one using ipmipower23:39
bigjoolsroaksoax: what are the main differences?23:39
roaksoaxthe behavior is different23:39
roaksoaxbigjools: t23:40
roaksoaxwwhat moonshot returns on ipmi commands23:40
roaksoaxon ipmi power we use cycle23:40
roaksoaxipmitool doesnt23:40
bigjools:/23:41
roaksoaxyeah it sucks23:42
roaksoaxbut btw when the power typenis set to "moonshot" the code looks for a template cslled moonshot.template23:42
roaksoaxanyway im off23:43
bigjoolsroaksoax: ok fair enough... it's a shame.  Thanks for getting loads of stuff fixed, tremendous job!23:44

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!