[09:15] so mgz, what pieces can I pick up so we can finish up the kernel params stuff, since we got Jenkins running again [09:17] so, what I still don't know how we're going to do is get the kernel options to the provisioningserver [09:18] also, did you see the notes from scott in last nights log? [09:19] mgz: I did. I'm pretty confident about the specific setup, the most interesting part was the '--' issue. [09:20] I think for a first pass, we can pick a spot to put it [09:20] and if it is important, we can detect '--' in the parameters being set, and split it to that side of a '--' in the commandline. [09:20] that makes sense [09:21] mgz: though maas itself doesn't seem to be setting '--', wh [09:21] which means that in the immediate term, we don't have to do anything. [09:21] if they need to pass "opts more opts -- extra_opts after doubledash" [09:21] it perhaps should, but we can probably punt on it then. [09:21] it would just work [09:28] jam, mgz: do we have the call this morning? [09:29] dimitern: not explicitly, though I'm happy to chat with you guys. Shall we just meet up on mumble? [09:29] sure, I have a headset now :) noise cancelling [09:30] mgz: ^^ [09:30] er, let me install that, I did the g+ stuff [09:32] er... that won't be happening any time soon [09:32] can we do hangout instead? [09:39] mgz: sure, though what happened to you being able to use mumble? (out of curiosity) [09:40] not using the special laptop, due to the screen borkage [09:41] and this borrowed one lacks mumble... or the qt libraries [09:41] mgz: sure, I was wondering more about "that won't be happenign any time soon" [09:41] all will become clear :) [09:42] dimitern: I seem to have 3 google identities for you, should I use the one with you picture associated with it? [09:42] mgz: dimitern: https://plus.google.com/hangouts/_/216f679bd782c69f54ec4960241fd8cedef4340f?authuser=2&hl=en# [09:42] jam: well, I have the @canonical one - that's it [09:45] dimitern: when Martin joined, you suddenly disconnected [10:22] sorry, mistouchpadded the close button === jtv1 is now known as jtv === mcclurmc is now known as mcclurmc_away === nveitch is now known as evilnickveitch === dimitern is now known as dimitern_lunch [12:40] allenap: poke about the tftp server, I'm trying to figure out if it ever talks to the maas server [12:40] Specifically, I realize now why mgz was worried about getting the provisioning server code to get information from the maas server. [12:40] Because tftp is where the kernel commandline is getting set [12:40] and AFAICT it never talks to the MAAS_URL [12:40] so I can import 'auth' and use that stuff, but I want to make sure that is a sane thing to do. [12:41] (I don't quite understand the layout given that tftp sits right next to the other worker tasks, maybe they run in the same process?) [12:45] rvba: maybe you have an idea about how the tftp service is set up? [12:48] Are we still supposed to be using quantal for development, or have we switched to raring? [12:48] I ask because I just tried on quantal and maas-import-squashfs appears broken (in a way that the daily ppa probably fixes) [12:50] rbasak: the lp:maas/1.2 branch is meant to be the stable release series for Quantal, I'm not sure about the ppas [12:50] jam: what about trunk? [12:51] rbasak: I don't really know. I would think that it isn't worth breaking Q compatibility yet, but I don't know if there is a reason to diverge. [12:52] OK [12:52] I just had to add multiverse for python-selenium, which wasn't a requirement previously, but fair enough [12:52] Next import-squashfs failed :-( === dimitern_lunch is now known as dimitern [13:01] jam, mgz: how is it going guys? === mcclurmc_away is now known as mcclurmc [13:02] d [13:02] dimitern: I belive mgz is finishing up his lunch. I've gotten the apis exposed, which should unblock your work. [13:02] Though I have now hit the tftp wall, and while I can force my way through, I was hoping for feedback from red squad to make sure that was ok to do. [13:03] jam: I see also your initial MP I depended on is approved now, how about landing it? [13:07] dimitern: I marked it as such, jenkins should pick it up and land it shortly [13:08] jam: ok, sweet - i'll get it from there so I can finish my tests [13:17] jam: I'm not really familiar with the tftp stuff. But AFAIK, it's a separated twisted process. It creates 'readers', one reader reads from the local filesystem to serve the images, the other uses the API to generated PXE files on the fly (the url for this is read from pserv.yaml). [13:22] rvba: right, I roughly see the code that does it. The problem is it doesn't seem to use any MAAS logic to actually generate the pxe files. [13:22] (it doesn't talk to the maasserver for anything, it just grabs the request parameters and uses it to fill out a template file) [13:22] at least that I can see. [13:22] jam: it is supposed to talk to the API to generated the PXE file. [13:24] rvba: k, it might be, I see a 'generator_url' in there. [13:26] ok, found it, I think we can just poke data onto the KernelParameters, which should make things reasonable for us. Yay! [13:27] rvba: thanks! [13:27] np [13:27] untangling how it was all talking to eachother was a bit tricky [13:27] Indeed, that twisted code is… twisted. [13:27] rvba: stuff like having a 'context' object that you can 'get()' data out of [13:27] rather than passing parameters [13:28] and just sort of magical 'get()' calls. [13:28] Yeah, that makes following what's going on harder. [13:33] something seems fishy about the 1.2 series config. 'make run' is failing to startup again, which seems to be an '/etc/bin/rndc.key' failure-to-read [14:05] hm, have not tried make run in 1.2 yet [14:10] mgz: I get the feeling people generally don't :). [14:10] mgz: https://code.launchpad.net/~jameinel/maas/1.2-pxeconfig-includes-kernel-opts/+merge/133255 is closing the loop to the tftp server [14:12] mgz: and now I'm 1 hr past EOD, and my son is right in getting me to stop. [14:13] the son is right [14:13] play time! see you tomorrow :) [14:13] * mgz gets on with some branches [14:37] robbiew: hwody! is trunk fully operational? [14:37] err [14:37] sorry :) [14:37] rvba: ^^ [14:37] heh [14:39] robbiew: hi, we have to fix bug 1075597 but that's underway. [14:39] Launchpad bug 1075597 in MAAS "Duplicated prefix in the url used by the CLI" [Critical,Triaged] https://launchpad.net/bugs/1075597 [14:39] err [14:39] roaksoax: ^ [14:39] Sorry again Robbie :) [14:39] rvba: alright, I'd like to upload to raring so I can start preparing the SRU for quantal [14:40] rvba: but now I'll look into the precise -> quantal upgrade to see whether i can reproduce the issue or not [14:40] All right. [14:40] jtv is working on a fix for bug 1075597. [14:40] Launchpad bug 1075597 in MAAS "Duplicated prefix in the url used by the CLI" [Critical,Triaged] https://launchpad.net/bugs/1075597 [14:41] ok cool === matsubara is now known as matsubara-lunch [16:18] rvba: what were the consequences of upgrade failure? [16:18] rvba: not being able to deploy? [16:20] roaksoax: if you're talking about 1072772, the main symptom was that the DNS config was not being written. So yes, unable to deploy anything. [16:20] rvba: ok cool i'm upgrading now so we'll see what's the outcome [16:23] rvba: what else does that affect (without having to enable DNS/DHCP?) [16:26] roaksoax: no task can be fired so: no power_on/power_off task can be fired, the reporting of the boot images cannot happen. [16:26] rvba: ack! [16:35] rvba: alright, so I just upgraded and the issue didn't appear [16:36] rvba: message about images dissappear correctly [16:36] rvba: testing power management now [16:36] if it works, then DNS should also [16:41] rvba: alright, what seems to be the problem now is that tftpd-hpa does not stop [16:42] rvba: hence maas-pserv doesn't start [16:42] roaksoax: that looks like another problem. [16:43] rvba: python-maas-provisioningserver should probably conflict with tftpd-hpa... but that's weird because maas already does [16:44] err maas-cluster-controller [17:19] rvba: ok so I upgraded again [17:19] rvba: seems to be working just fine [17:19] rvba: but the logfile for cluster-celery wasn't created [17:20] roaksoax: and the celery process for the cluster is running? [17:20] rvba: nope === matsubara-lunch is now known as matsubara [17:21] roaksoax: any error in the upstart log? [17:22] rvba: there's no log [17:22] Weird. [17:22] rvba: you can access the instance [17:23] Yep, I'm in. [17:24] rvba: so nodes enlist and commission [17:25] Nov 7 17:17:47 server-0fd520a0-454d-43b7-8e86-a7d5a4661e8e kernel: [ 1507.296247] init: maas-cluster-celery main process (1229) terminated with status 1 [17:25] yeah there's no error log though [17:25] rvba: what's the logfile that it should have? celery.log? [17:25] or celery-cluster.log [17:26] /var/log/maas/celery.log [17:26] roaksoax: I wonder how nodes could be enlisted if pserv is not running. [17:27] rvba: i mnanually restarted pserv [17:27] ok [17:27] rvba: so I stopped tftpd-hpa and started pserv [17:30] rvba: ok so the maas-pserv issue is definitely becuase tftpd-hpa is running [17:30] rvba: but maas-cluster-celery is the issue [17:32] roaksoax: I tried to restart maas-cluster-celery and it crashed. maas-cluster-celery main process (8707) terminated with status 1 [17:36] rvba: ok i think i got it [17:36] rvba: i think it is the way how it is sourcing MAAS_URL [17:36] or the file that containst it [17:37] rvba: ah no [17:37] roaksoax: I ran what the upstart job runs and I got this [17:37] https://pastebin.canonical.com/77847/ [17:38] rvba: yep, exaclt what i get [17:38] /usr/share/maas/celeryconfig_cluster.py is not on the path. [17:38] rvba: yeah I think i saw a bug fixing that [17:38] as it was sourcing something else instead [17:38] This must have been caused by a very recent change because it was working fine last week. [17:39] indeed [17:40] rvba: IIRC, we looked at it last week and it worked [17:40] rvba: i mean, we found the root cause [17:54] rvba: could this be it? "Another thing that came up is that maas-provision complains about not finding celeryconfig.py, a configuration module that we only use on the region controller. In a separate branch I'll make the import script select cluster_celeryconfig.py instead." [17:54] No, I don't think this is related. === mcclurmc is now known as mcclurmc_away [17:56] rvba: i'm pretty sure you debug this couple weeks ago, I came to you and you found the problem [17:56] roaksoax: yeah, this rings a bell indeed… but I can't find what the problem is this time :( [18:06] rvba: wasn't it something in /usr/sbin/maas-provision? [18:08] roaksoax: I don't really remember tbh. Did you manually changed that file? Looks like it's changed since I checked. [18:08] rvba: yeah but set it back to default [18:09] rvba: it is not that file, i can't remember what was it [18:14] roaksoax: I think I found the pb. [18:14] rvba: what is it? [18:14] roaksoax: /etc/maas/maas_cluster.conf contains the wrong URL. [18:14] MAAS_URL=http://10.55.60.86/ [18:14] No, I don't think this is related.I changed it to: [18:14] MAAS_URL=http://10.55.60.86/MAAS/ [18:14] Then I restarted the service. [18:15] hmmm [18:15] If you look in /var/log/apache2/access.log you will see that it was desperately trying to register. [18:15] yeah [18:15] let me test this locally real quick [18:17] rvba: yeah that seems to be the solution [18:20] roaksoax: We really need to improve the error logging here :) [18:20] indeed :) [18:25] roaksoax: filed bug 1076080. [18:25] Launchpad bug 1076080 in MAAS "No error message is printed anywhere when the cluster controller fails to register itself." [Low,Triaged] https://launchpad.net/bugs/1076080 [18:25] cool, I filed bug 1076075 [18:25] Launchpad bug 1076075 in maas (Ubuntu) "maas_cluster.conf ends up with the wrong URL after upgrade." [Critical,Confirmed] https://launchpad.net/bugs/1076075 [18:26] Cool. === mcclurmc_away is now known as mcclurmc [18:48] hi guys, im running maas from the ppa dailybuiilds on precise [18:48] when im trying to install maas-dhcp [18:48] i get [18:49] Setting up maas-dhcp (0.1+bzr971+dfsg-0+998+75~ppa0~precise1) ... [18:49] Usage: /usr/bin/django-admin config_master_dhcp [options] [18:49] Initialize master DHCP settings. [18:49] a [18:49] usr/bin/django-admin: error: no such option: --interface [18:50] dpkg: error processing maas-dhcp (--configure): [18:50] subprocess installed post-installation script returned error exit status 2 [18:50] Errors were encountered while processing: [18:50] maas-dhcp [18:50] and indeed --interface its not a valid django-admin option [18:50] is this a BUG ? [18:52] a [19:01] anyone guys ? [19:30] i submited a bug for the dailybuild https://bugs.launchpad.net/maas/+bug/1076092 [19:30] Ubuntu bug 1076092 in MAAS "maas-dhcp fails to install on Precise - dailybuild ppa" [Undecided,New] [19:30] hope someone can take a look at it === Guest15779 is now known as dpb___ === mcclurmc is now known as mcclurmc_away [20:54] morning [20:55] roaksoax: hi, want to talk about SRU? [20:56] bigjools: yes [20:56] but im coming back from lunch [20:56] roaksoax: np, I am free for an hour [20:56] sonif you can give a few minuted will be grear [20:57] are you drunk? :) [20:57] lol i wish it was friday night for me to do so [20:58] but im from the ohone [20:58] phone [20:58] on 12.04, is there an easy way to tell maas to use a specific preseed file for _some_ nodes ? [20:58] the idea would be to use lvm for computes nodes that have 2 drives. [20:59] right now, maas install computes nodes without touching the 2 drive. So i ended up with 14 drive not being used [20:59] (actually, 7 drive not being used) [21:00] melmoth: no, that feature will be in 13.04 though [21:00] thanks. [21:06] melmoth: u would have to ply witth cobbler [21:08] bigjools: ready when you are [21:08] roaksoax: don't encourage people to play with cobbler [21:09] since we're getting rid of it :) [21:09] roaksoax: ok so is there anything else special needed for the SRU? [21:09] bigjools: I'm not encouraging :) i'm just saying... maybe costumer needs it [21:09] bigjools: ok so, yes. [21:09] so far we have: Backport the Django 1.4 feature we're using and any other new dependencies (need to check those) [21:10] and test like hell :) [21:10] yes [21:10] bigjools: so, the plan is basically, finish with those 12.10 bugs [21:10] and the SRU team agree to this already? [21:10] bigjools: and implement the features the qa requrested, plus those missing such as kernel parameters [21:10] I will make a 12.04 PPA build toda to get the ball rolling [21:10] and SRU that to both, quantal, and 12.04 [21:11] bigjools: i already have precise in PPA [21:11] its in experimental, same version we have in quantal [21:11] roaksoax: the 1.2 branch? [21:11] ah [21:11] nice [21:11] what did you backport? [21:11] bigjools: nothing yet, I simply needed that to test upgrade failures [21:12] but so far is the django thing [21:12] the problem [21:12] ok, it won't work then :) [21:12] and dependencies in universe [21:12] bigjools: now, I did want to discuss this [21:12] universe in 12.04 and main in 13.10 I presume [21:12] bigjools: when are you guys looking to finish the SRU-able features? [21:13] so the "features" were 13.04 only really [21:13] bigjools: right, but kernel parameters are needed to be SRUable [21:13] however given the QA team wanted it in 12.04 I think we should backport those separately [21:13] bigjools: uhmmmm maybe [21:13] so the initial backport should be the bugs in the stabilization milestone [21:13] bigjools: well the thing is that we discussed this with smoser and jamespage [21:14] yeah [21:14] so I want to minimise risk [21:14] bigjools: and we agreed that the best approach was to SRU the QA team features as well [21:14] if we change too much at once it's not minimised [21:14] bigjools: our mplan is to have precise SRU'd by 12.04.2 [21:14] ok [21:14] bigjools, its not minimised :) [21:14] that's January, plenty of time [21:14] that ship has sailed [21:14] yeah [21:15] smoser: there's relative levels of minimising :) [21:15] bigjools: however, note that we will have to make incremental SRU's, so I'm gonna start with quantal [21:15] so my suggestion would be to SRU the quantal stabilization stuff first, then backport that to precise, then SRU the new features to both [21:16] bigjools: my plan is that, sru stabilization, and then sru to precise [21:16] bigjools: not backport, SRU [21:16] roaksoax: perfect [21:16] yes, I meant SRU [21:17] but it involves backporting dependencies of course [21:17] bigjools: we can't introduce new dependencies [21:17] bigjools: so they will have to get shipped with the maas source [21:17] gaaahhhhhh [21:17] really? [21:17] what dependencies are we talking about here? [21:17] bigjools: i have that covered since yui3, and python-tx-tfpt are [21:17] smoser: yui3, python-tx-tftp [21:18] as in those are not available in 12.04 ? [21:18] correct [21:18] is it just a matter of pulling versions in quantal to 12.04 ? [21:19] smoser: both yui3 and python-tx-tftp are two new sources not in 12.04 [21:20] this is also a matter for the security team [21:20] and who will win? :) [21:21] bigjools: us... we have no other option than doing it :) [21:21] roaksoax, given the source of this request, i suggest that new packages to 12.04 is at least possibly negotiable. [21:22] smoser: i wasn't aware, nor never heard of being able to introduce new sources into the archive for an older release [21:22] and from security team perspective, if you put yourself in their shoes, and you were going to have to suppor this one way or another... [21:22] smoser: +1 [21:22] which would you rather hvae? "real packages" or "all shoved inside maas" [21:23] exactly - security team will need to track stuff shoved into maas [21:23] smoser: that's out of the question, definitely real packages, but in terms of policies, I don't think that's covered, is it? [21:23] https://launchpad.net/ubuntu/+source/walinuxagent [21:23] roaksoax, note, that package is not present in 12.04 "release" pocket. [21:24] smoser: right, alright, if it is possible [21:24] also note, that i'm not terribly happy with using it as an example, but i'm not aware of others. [21:24] then we should do that [21:24] ie, i'm not happy with that crappy package . but it is an example of something that went from "not packaged" to "main" in a SRU [21:24] smoser: right, [21:25] smoser: i don't really have issues in doing so, i just wasn't aware it was even possible [21:26] bigjools: ok so, the plan would be to finish 12.10 stabilization, SRU to quantal, then start looking at precise, sru that [21:26] and then SRU raring [21:26] as a second mayor SRU [21:26] smoser: agreed? [21:26] roaksoax, its not *really* possible [21:26] :) [21:27] smoser: right :) [21:27] roaksoax: yes [21:27] roaksoax, yeah, i think that is a reasonable path. the only change i woudl suggest is [21:27] that you do not wait until "finish 12.10 stabilization" before "start looking at precise" [21:27] smoser: oh definitely not, I already have precise packaged [21:27] ie, lets get a ppa functional with what is in quantal now. [21:27] right. [21:28] smoser: just need to pull in django fix [21:28] bigjools, are you just awake because you're stuck on europe time? [21:28] and now split the other dependencies into their own packages [21:28] smoser: pretty much [21:28] smoser: body clock is screwed to hell [21:28] smoser: was up at 4am [21:28] sun comes up at 5 [21:28] roaksoax, we should consult security team to see what they suggest on the new dependencies. [21:29] smoser: will do [21:29] then my twins wake up ... not much hope of sleep after that :) [21:29] smoser: the only real problem would be yui3, which is a new source that really replaces yui [21:29] and point out to them that a.) we really, really want SRU of 12.10 maas to 12.04, and b.) walinux did it, so maybe we can too [21:29] so they won't be co-installable IIRC [21:30] roaksoax, hm.. yeah, i dont know. probably can be sorted out. [21:31] alright, i'll talk to them and then sort things out [21:31] them = security team [21:33] roaksoax: great, I'll catch up with you next week about that then (I am off work Friday) [21:36] bigjools: same here === mcclurmc_away is now known as mcclurmc === mcclurmc is now known as mcclurmc_away