/srv/irclogs.ubuntu.com/2012/11/07/#maas.txt

jamso mgz, what pieces can I pick up so we can finish up the kernel params stuff, since we got Jenkins running again09:15
mgzso, what I still don't know how we're going to do is get the kernel options to the provisioningserver09:17
mgzalso, did you see the notes from scott in last nights log?09:18
jammgz: I did. I'm pretty confident about the specific setup, the most interesting part was the '--' issue.09:19
jamI think for a first pass, we can pick a spot to put it09:20
jamand if it is important, we can detect '--' in the parameters being set, and split it to that side of a '--' in the commandline.09:20
mgzthat makes sense09:20
jammgz: though maas itself doesn't seem to be setting '--', wh09:21
jamwhich means that in the immediate term, we don't have to do anything.09:21
jamif they need to pass "opts more opts -- extra_opts after doubledash"09:21
mgzit perhaps should, but we can probably punt on it then.09:21
jamit would just work09:21
dimiternjam, mgz: do we have the call this morning?09:28
jamdimitern: not explicitly, though I'm happy to chat with you guys. Shall we just meet up on mumble?09:29
dimiternsure, I have a headset now :) noise cancelling09:29
jammgz: ^^09:30
mgzer, let me install that, I did the g+ stuff09:30
mgzer... that won't be happening any time soon09:32
mgzcan we do hangout instead?09:32
jammgz: sure, though what happened to you being able to use mumble? (out of curiosity)09:39
mgznot using the special laptop, due to the screen borkage09:40
mgzand this borrowed one lacks mumble... or the qt libraries09:41
jammgz: sure, I was wondering more about "that won't be happenign any time soon"09:41
mgzall will become clear :)09:41
jamdimitern: I seem to have 3 google identities for you, should I use the one with you picture associated with it?09:42
jammgz: dimitern: https://plus.google.com/hangouts/_/216f679bd782c69f54ec4960241fd8cedef4340f?authuser=2&hl=en#09:42
dimiternjam: well, I have the @canonical one - that's it09:42
jamdimitern: when Martin joined, you suddenly disconnected09:45
mgzsorry, mistouchpadded the close button10:22
=== jtv1 is now known as jtv
=== mcclurmc is now known as mcclurmc_away
=== nveitch is now known as evilnickveitch
=== dimitern is now known as dimitern_lunch
jamallenap: poke about the tftp server, I'm trying to figure out if it ever talks to the maas server12:40
jamSpecifically, I realize now why mgz was worried about getting the provisioning server code to get information from the maas server.12:40
jamBecause tftp is where the kernel commandline is getting set12:40
jamand AFAICT it never talks to the MAAS_URL12:40
jamso I can import 'auth' and use that stuff, but I want to make sure that is a sane thing to do.12:40
jam(I don't quite understand the layout given that tftp sits right next to the other worker tasks, maybe they run in the same process?)12:41
jamrvba: maybe you have an idea about how the tftp service is set up?12:45
rbasakAre we still supposed to be using quantal for development, or have we switched to raring?12:48
rbasakI ask because I just tried on quantal and maas-import-squashfs appears broken (in a way that the daily ppa probably fixes)12:48
jamrbasak: the lp:maas/1.2 branch is meant to be the stable release series for Quantal, I'm not sure about the ppas12:50
rbasakjam: what about trunk?12:50
jamrbasak: I don't really know. I would think that it isn't worth breaking Q compatibility yet, but I don't know if there is a reason to diverge.12:51
rbasakOK12:52
rbasakI just had to add multiverse for python-selenium, which wasn't a requirement previously, but fair enough12:52
rbasakNext import-squashfs failed :-(12:52
=== dimitern_lunch is now known as dimitern
dimiternjam, mgz: how is it going guys?13:01
=== mcclurmc_away is now known as mcclurmc
jamd13:02
jamdimitern: I belive mgz is finishing up his lunch. I've gotten the apis exposed, which should unblock your work.13:02
jamThough I have now hit the tftp wall, and while I can force my way through, I was hoping for feedback from red squad to make sure that was ok to do.13:02
dimiternjam: I see also your initial MP I depended on is approved now, how about landing it?13:03
jamdimitern: I marked it as such, jenkins should pick it up and land it shortly13:07
dimiternjam: ok, sweet - i'll get it from there so I can finish my tests13:08
rvbajam: I'm not really familiar with the tftp stuff.  But AFAIK, it's a separated twisted process.  It creates 'readers', one reader reads from the local filesystem to serve the images, the other uses the API to generated PXE files on the fly (the url for this is read from pserv.yaml).13:17
jamrvba: right, I roughly see the code that does it. The problem is it doesn't seem to use any MAAS logic to actually generate the pxe files.13:22
jam(it doesn't talk to the maasserver for anything, it just grabs the request parameters and uses it to fill out a template file)13:22
jamat least that I can see.13:22
rvbajam: it is supposed to talk to the API to generated the PXE file.13:22
jamrvba: k, it might be, I see a 'generator_url' in there.13:24
jamok, found it, I think we can just poke data onto the KernelParameters, which should make things reasonable for us. Yay!13:26
jamrvba: thanks!13:27
rvbanp13:27
jamuntangling how it was all talking to eachother was a bit tricky13:27
rvbaIndeed, that twisted code is… twisted.13:27
jamrvba: stuff like having a 'context' object that you can 'get()' data out of13:27
jamrather than passing parameters13:27
jamand just sort of magical 'get()' calls.13:28
rvbaYeah, that makes following what's going on harder.13:28
jamsomething seems fishy about the 1.2 series config. 'make run' is failing to startup again, which seems to be an '/etc/bin/rndc.key' failure-to-read13:33
mgzhm, have not tried make run in 1.2 yet14:05
jammgz: I get the feeling people generally don't :).14:10
jammgz: https://code.launchpad.net/~jameinel/maas/1.2-pxeconfig-includes-kernel-opts/+merge/133255 is closing the loop to the tftp server14:10
jammgz: and now I'm 1 hr past EOD, and my son is right in getting me to stop.14:12
mgzthe son is right14:13
mgzplay time! see you tomorrow :)14:13
* mgz gets on with some branches14:13
roaksoaxrobbiew: hwody! is trunk fully operational?14:37
roaksoaxerr14:37
roaksoaxsorry :)14:37
roaksoaxrvba: ^^14:37
robbiewheh14:37
rvbarobbiew: hi, we have to fix bug 1075597 but that's underway.14:39
ubot5Launchpad bug 1075597 in MAAS "Duplicated prefix in the url used by the CLI" [Critical,Triaged] https://launchpad.net/bugs/107559714:39
rvbaerr14:39
rvbaroaksoax: ^14:39
rvbaSorry again Robbie :)14:39
roaksoaxrvba: alright, I'd like to upload to raring so I can start preparing the SRU for quantal14:39
roaksoaxrvba: but now I'll look into the precise -> quantal upgrade to see whether i can reproduce the issue or not14:40
rvbaAll right.14:40
rvbajtv is working on a fix for bug 1075597.14:40
ubot5Launchpad bug 1075597 in MAAS "Duplicated prefix in the url used by the CLI" [Critical,Triaged] https://launchpad.net/bugs/107559714:40
roaksoaxok cool14:41
=== matsubara is now known as matsubara-lunch
roaksoaxrvba: what were the consequences of upgrade failure?16:18
roaksoaxrvba: not being able to deploy?16:18
rvbaroaksoax: if you're talking about 1072772, the main symptom was that the DNS config was not being written.  So yes, unable to deploy anything.16:20
roaksoaxrvba: ok cool i'm upgrading now so we'll see what's the outcome16:20
roaksoaxrvba: what else does that affect (without having to enable DNS/DHCP?)16:23
rvbaroaksoax: no task can be fired so: no power_on/power_off task can be fired, the reporting of the boot images cannot happen.16:26
roaksoaxrvba: ack!16:26
roaksoaxrvba: alright, so I just upgraded and the issue didn't appear16:35
roaksoaxrvba: message about images dissappear correctly16:36
roaksoaxrvba: testing power management now16:36
roaksoaxif it works, then DNS should also16:36
roaksoaxrvba: alright, what seems to be the problem now is that tftpd-hpa does not stop16:41
roaksoaxrvba: hence maas-pserv doesn't start16:42
rvbaroaksoax: that looks like another problem.16:42
roaksoaxrvba: python-maas-provisioningserver should probably conflict with tftpd-hpa... but that's weird because maas already does16:43
roaksoaxerr maas-cluster-controller16:44
roaksoaxrvba: ok so I upgraded again17:19
roaksoaxrvba: seems to be working just fine17:19
roaksoaxrvba: but the logfile for cluster-celery wasn't created17:19
rvbaroaksoax: and the celery process for the cluster is running?17:20
roaksoaxrvba: nope17:20
=== matsubara-lunch is now known as matsubara
rvbaroaksoax: any error in the upstart log?17:21
roaksoaxrvba: there's no log17:22
rvbaWeird.17:22
roaksoaxrvba: you can access the instance17:22
rvbaYep, I'm in.17:23
roaksoaxrvba: so nodes enlist and commission17:24
rvbaNov  7 17:17:47 server-0fd520a0-454d-43b7-8e86-a7d5a4661e8e kernel: [ 1507.296247] init: maas-cluster-celery main process (1229) terminated with status 117:25
roaksoaxyeah there's no error log though17:25
roaksoaxrvba: what's the logfile that it should have? celery.log?17:25
roaksoaxor celery-cluster.log17:25
rvba/var/log/maas/celery.log17:26
rvbaroaksoax: I wonder how nodes could be enlisted if pserv is not running.17:26
roaksoaxrvba: i mnanually restarted pserv17:27
rvbaok17:27
roaksoaxrvba: so I stopped tftpd-hpa and started pserv17:27
roaksoaxrvba: ok so the maas-pserv issue is definitely becuase tftpd-hpa is running17:30
roaksoaxrvba: but maas-cluster-celery is the issue17:30
rvbaroaksoax: I tried to restart maas-cluster-celery and it crashed. maas-cluster-celery main process (8707) terminated with status 117:32
roaksoaxrvba: ok i think i got it17:36
roaksoaxrvba: i think it is the way how it is sourcing MAAS_URL17:36
roaksoaxor the file that containst it17:36
roaksoaxrvba: ah no17:37
rvbaroaksoax: I ran what the upstart job runs and I got this17:37
rvbahttps://pastebin.canonical.com/77847/17:37
roaksoaxrvba: yep, exaclt what i get17:38
rvba/usr/share/maas/celeryconfig_cluster.py is not on the path.17:38
roaksoaxrvba: yeah I think i saw a bug fixing that17:38
roaksoaxas it was sourcing something else instead17:38
rvbaThis must have been caused by a very recent change because it was working fine last week.17:38
roaksoaxindeed17:39
roaksoaxrvba: IIRC, we looked at it last week and it worked17:40
roaksoaxrvba: i mean, we found the root cause17:40
roaksoaxrvba: could this be it? "Another thing that came up is that maas-provision complains about not finding celeryconfig.py, a configuration module that we only use on the region controller.  In a separate branch I'll make the import script select cluster_celeryconfig.py instead."17:54
rvbaNo, I don't think this is related.17:54
=== mcclurmc is now known as mcclurmc_away
roaksoaxrvba: i'm pretty sure you debug this couple weeks ago, I came to you and you found the problem17:56
rvbaroaksoax: yeah, this rings a bell indeed… but I can't find what the problem is this time :(17:56
roaksoaxrvba: wasn't it something in /usr/sbin/maas-provision?18:06
rvbaroaksoax: I don't really remember tbh.  Did you manually changed that file?  Looks like it's changed since I checked.18:08
roaksoaxrvba: yeah but set it back to default18:08
roaksoaxrvba: it is not that file, i can't remember what was it18:09
rvbaroaksoax: I think I found the pb.18:14
roaksoaxrvba: what is it?18:14
rvbaroaksoax: /etc/maas/maas_cluster.conf contains the wrong URL.18:14
rvbaMAAS_URL=http://10.55.60.86/18:14
rvbaNo, I don't think this is related.I changed it to:18:14
rvbaMAAS_URL=http://10.55.60.86/MAAS/18:14
rvbaThen I restarted the service.18:14
roaksoaxhmmm18:15
rvbaIf you look in /var/log/apache2/access.log you will see that it was desperately trying to register.18:15
roaksoaxyeah18:15
roaksoaxlet me test this locally real quick18:15
roaksoaxrvba: yeah that seems to be the solution18:17
rvbaroaksoax: We really need to improve the error logging here :)18:20
roaksoaxindeed :)18:20
rvbaroaksoax: filed bug 1076080.18:25
ubot5Launchpad bug 1076080 in MAAS "No error message is printed anywhere when the cluster controller fails to register itself." [Low,Triaged] https://launchpad.net/bugs/107608018:25
roaksoaxcool, I filed bug 107607518:25
ubot5Launchpad bug 1076075 in maas (Ubuntu) "maas_cluster.conf ends up with the wrong URL after upgrade." [Critical,Confirmed] https://launchpad.net/bugs/107607518:25
rvbaCool.18:26
=== mcclurmc_away is now known as mcclurmc
Lele_hi guys, im running maas from the ppa dailybuiilds on precise18:48
Lele_when im trying to install maas-dhcp18:48
Lele_i get18:48
Lele_Setting up maas-dhcp (0.1+bzr971+dfsg-0+998+75~ppa0~precise1) ...18:49
Lele_Usage: /usr/bin/django-admin config_master_dhcp [options]18:49
Lele_Initialize master DHCP settings.18:49
Lele_a18:49
Lele_usr/bin/django-admin: error: no such option: --interface18:49
Lele_dpkg: error processing maas-dhcp (--configure):18:50
Lele_subprocess installed post-installation script returned error exit status 218:50
Lele_Errors were encountered while processing:18:50
Lele_maas-dhcp18:50
Lele_and indeed --interface its not a valid django-admin option18:50
Lele_is this a BUG ?18:50
Lele_a18:52
Lele_anyone guys ?19:01
Lele__i submited a bug for the dailybuild https://bugs.launchpad.net/maas/+bug/107609219:30
ubot5Ubuntu bug 1076092 in MAAS "maas-dhcp fails to install on Precise - dailybuild ppa" [Undecided,New]19:30
Lele__hope someone can take a look at it19:30
=== Guest15779 is now known as dpb___
=== mcclurmc is now known as mcclurmc_away
bigjoolsmorning20:54
bigjoolsroaksoax: hi, want to talk about SRU?20:55
roaksoaxbigjools: yes20:56
roaksoaxbut im coming back from lunch20:56
bigjoolsroaksoax: np, I am free for an hour20:56
roaksoaxsonif you can give a few minuted will be grear20:56
bigjoolsare you drunk? :)20:57
roaksoaxlol i wish it was friday night for me to do so20:57
roaksoaxbut im from the ohone20:58
roaksoaxphone20:58
melmothon 12.04, is there an easy way to tell maas to use a specific preseed file for _some_ nodes ?20:58
melmoththe idea would be to use lvm for computes nodes that have 2 drives.20:58
melmothright now, maas install computes nodes without touching the 2 drive. So i ended up with 14 drive not being used20:59
melmoth(actually, 7 drive not being used)20:59
bigjoolsmelmoth: no, that feature will be in 13.04 though21:00
melmoththanks.21:00
roaksoaxmelmoth: u would have to ply witth cobbler21:06
roaksoaxbigjools: ready when you are21:08
bigjoolsroaksoax: don't encourage people to play with cobbler21:08
bigjoolssince we're getting rid of it :)21:09
bigjoolsroaksoax: ok so is there anything else special needed for the SRU?21:09
roaksoaxbigjools: I'm not encouraging :) i'm just saying... maybe costumer needs it21:09
roaksoaxbigjools: ok so, yes.21:09
bigjoolsso far we have: Backport the Django 1.4 feature we're using and any other new dependencies (need to check those)21:09
bigjoolsand test like hell :)21:10
roaksoaxyes21:10
roaksoaxbigjools: so, the plan is basically, finish with those 12.10 bugs21:10
bigjoolsand the SRU team agree to this already?21:10
roaksoaxbigjools: and implement the features the qa requrested, plus those missing such as kernel parameters21:10
bigjoolsI will make a 12.04 PPA build toda to get the ball rolling21:10
roaksoaxand SRU that to both, quantal, and 12.0421:10
roaksoaxbigjools: i already have precise in PPA21:11
roaksoaxits in experimental, same version we have in quantal21:11
bigjoolsroaksoax: the 1.2 branch?21:11
bigjoolsah21:11
bigjoolsnice21:11
bigjoolswhat did you backport?21:11
roaksoaxbigjools: nothing yet, I simply needed that to test upgrade failures21:11
roaksoaxbut so far is the django thing21:12
roaksoaxthe problem21:12
bigjoolsok, it won't work then :)21:12
roaksoaxand dependencies in universe21:12
roaksoaxbigjools: now, I did want to discuss this21:12
bigjoolsuniverse in 12.04 and main in 13.10 I presume21:12
roaksoaxbigjools: when are you guys looking to finish the SRU-able features?21:12
bigjoolsso the "features" were 13.04 only really21:13
roaksoaxbigjools: right, but kernel parameters are needed to be SRUable21:13
bigjoolshowever given the QA team wanted it in 12.04 I think we should backport those separately21:13
roaksoaxbigjools: uhmmmm maybe21:13
bigjoolsso the initial backport should be the bugs in the stabilization milestone21:13
roaksoaxbigjools: well the thing is that we discussed this with smoser and jamespage21:13
bigjoolsyeah21:14
bigjoolsso I want to minimise risk21:14
roaksoaxbigjools: and we agreed that the best approach was to SRU the QA team features as well21:14
bigjoolsif we change too much at once it's not minimised21:14
roaksoaxbigjools: our mplan is to have precise SRU'd by 12.04.221:14
bigjoolsok21:14
smoserbigjools, its not minimised :)21:14
bigjoolsthat's January, plenty of time21:14
smoserthat ship has sailed21:14
roaksoaxyeah21:14
bigjoolssmoser: there's relative levels of minimising :)21:15
roaksoaxbigjools: however, note that we will have to make incremental SRU's, so I'm gonna start with quantal21:15
bigjoolsso my suggestion would be to SRU the quantal stabilization stuff first, then backport that to precise, then SRU the new features to both21:15
roaksoaxbigjools: my plan is that, sru stabilization, and then sru to precise21:16
roaksoaxbigjools: not backport, SRU21:16
bigjoolsroaksoax: perfect21:16
bigjoolsyes, I meant SRU21:16
bigjoolsbut it involves backporting dependencies of course21:17
roaksoaxbigjools: we can't introduce new dependencies21:17
roaksoaxbigjools: so they will have to get shipped with the maas source21:17
bigjoolsgaaahhhhhh21:17
bigjoolsreally?21:17
smoserwhat dependencies are we talking about here?21:17
roaksoaxbigjools: i have that covered since yui3, and python-tx-tfpt are21:17
roaksoaxsmoser: yui3, python-tx-tftp21:17
smoseras in those are not available in 12.04 ?21:18
bigjoolscorrect21:18
smoseris it just a matter of pulling versions in quantal to 12.04 ?21:18
roaksoaxsmoser: both yui3 and python-tx-tftp are two new sources not in 12.0421:19
bigjoolsthis is also a matter for the security team21:20
bigjoolsand who will win? :)21:20
roaksoaxbigjools: us... we have no other option than doing it :)21:21
smoserroaksoax, given the source of this request, i suggest that new packages to 12.04 is at least possibly negotiable.21:21
roaksoaxsmoser: i wasn't aware, nor never heard of being able to introduce new sources into the archive for an older release21:22
smoserand from security team perspective, if you put yourself in their shoes, and you were going to have to suppor this one way or another...21:22
bigjoolssmoser: +121:22
smoserwhich would you rather hvae?  "real packages" or "all shoved inside maas"21:22
bigjoolsexactly - security team will need to track stuff shoved into maas21:23
roaksoaxsmoser: that's out of the question, definitely real packages, but in terms of policies, I don't think that's covered, is it?21:23
smoserhttps://launchpad.net/ubuntu/+source/walinuxagent21:23
smoserroaksoax, note, that package is not present in 12.04 "release" pocket.21:23
roaksoaxsmoser: right, alright, if it is possible21:24
smoseralso note, that i'm not terribly happy with using it as an example, but i'm not aware of others.21:24
roaksoaxthen we should do that21:24
smoserie, i'm not happy with that crappy package . but it is an example of something that went from "not packaged" to "main" in a SRU21:24
roaksoaxsmoser: right,21:24
roaksoaxsmoser: i don't really have issues in doing so, i just wasn't aware it was even possible21:25
roaksoaxbigjools: ok so, the plan would be to finish 12.10 stabilization, SRU to quantal, then start looking at precise, sru that21:26
roaksoaxand then SRU raring21:26
roaksoaxas a second mayor SRU21:26
roaksoaxsmoser: agreed?21:26
smoserroaksoax, its not *really* possible21:26
smoser:)21:26
roaksoaxsmoser: right :)21:27
bigjoolsroaksoax: yes21:27
smoserroaksoax, yeah, i think that is a reasonable path. the only change i woudl suggest is21:27
smoserthat you do not wait until "finish 12.10 stabilization" before "start looking at precise"21:27
roaksoaxsmoser: oh definitely not, I already have precise packaged21:27
smoserie, lets get a ppa functional with what is in quantal now.21:27
smoserright.21:27
roaksoaxsmoser: just need to pull in django fix21:28
smoserbigjools, are you just awake because you're stuck on europe time?21:28
roaksoaxand now split the other dependencies into their own packages21:28
bigjoolssmoser: pretty much21:28
bigjoolssmoser: body clock is screwed to hell21:28
bigjoolssmoser: was up at 4am21:28
bigjoolssun comes up at 521:28
smoserroaksoax, we should consult security team to see what they suggest on the new dependencies.21:28
roaksoaxsmoser: will do21:29
bigjoolsthen my twins wake up ... not much hope of sleep after that :)21:29
roaksoaxsmoser: the only real problem would be yui3, which is a new source that really replaces yui21:29
smoserand point out to them that a.) we really, really want SRU of 12.10 maas to 12.04, and b.) walinux did it, so maybe we can too21:29
roaksoaxso they won't be co-installable IIRC21:29
smoserroaksoax, hm.. yeah, i dont know. probably can be sorted out.21:30
roaksoaxalright, i'll talk to them and then sort things out21:31
roaksoaxthem = security team21:31
bigjoolsroaksoax: great, I'll catch up with you next week about that then (I am off work Friday)21:33
roaksoaxbigjools: same here21:36
=== mcclurmc_away is now known as mcclurmc
=== mcclurmc is now known as mcclurmc_away

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!