[00:00] bigjools: i've seen similar issues but seemed to be DHCP issues [00:01] bigjools: for example, once it DHCP's once, it seems to lock in the IP address until you set static and then DHCP again [00:01] or if you reset the power (like disconnected the servers from power source) [00:01] I did that [00:04] weird [00:05] very [00:55] roaksoax: ok bmc-config just returns "Unable to get Number of Users" [00:55] presumably I need some modules/options? [00:55] bigjools: yes [00:56] * bigjools looks in commissioning_user_data [00:56] bigjools: during commissioning, enable the options in the commission-user-data in /etc/maas [00:56] well I am in deployment mode [00:56] can't ssh to a commissioning box :) [00:57] bigjools: modprobe ipmi_msghandler modprobe ipmi_devintf [00:57] modprobe ipmi_si type=kcs ports=0xca2 [00:57] sorted, ta [00:58] roaksoax: didn't help :( [00:58] bigjools: rmmod ipmi_si [00:58] roaksoax: as in, the bmc-config worked [00:58] then modprobe ipmi_si type=kcs ports=0xca2 [00:58] but the dhcp is still borked [00:59] uhmm that's pretty werird though [00:59] DHCPDISCOVER followed by DHCPOFFER [00:59] over and over [00:59] it's simply not accepting the offered address [01:00] bigjools: that's pretty weird though [01:00] very [02:55] bigjools: when you install maas from a package, do you get a /home/maas? [02:55] jtv: no [02:55] deliberately so [04:01] smoser: around? [04:39] bigjools: do you happen to know if anny changes were made to the power handling? i'm getting an error when trying to execute the ipmi template, which wasn't there before [04:45] bigjools: nevermind found the issue [04:53] bigjools: https://code.launchpad.net/~andreserl/maas/maas_enlist_ipmi_username/+merge/128857 please :) [05:00] roaksoax: looking [05:03] bigjools: thanks [05:04] de nada [05:04] dpu:) [05:04] :) [05:04] aren't you up late? [05:06] bigjools: yeah :) [08:51] fix for bug 1064734 [08:51] Launchpad bug 1064734 in MAAS "ERROR Invalid 'tags' constraint 'set(['test-tag'])': No such tag using maas-tags to deploy with juju" [High,Triaged] https://launchpad.net/bugs/1064734 [09:31] mgz: I think you meant the juju bug. [09:32] rvba: the WSGIApplicationGroup setting from the article I found works, but what I wanted to ask you is: can you think of any problems it might cause us? [09:33] Meanwhile, I'm getting a JS test failure in trunk. :( [09:33] jtv: that is a juju bug [09:34] jtv: we probably want to use the daemon configuration for mod_wsgi longer term [09:34] mgz: argh, misread the number! [09:34] blame matsubara! :) [09:35] jtv: I can't think of anything really. [09:36] mgz: you mean WSGIDaemonProcess? If so, we already use that. [09:36] rvba: thanks, that gives me a bit more confidence than me bunging it in and saying "hey look, it seems to work now!" :) [09:37] jtv: from what I can see, the combination WSGIDaemonProcess/WSGIApplicationGroup seems to be wildly used to deploy Django applications. [09:37] Wildly? That sounds exciting. [09:38] :) [09:38] Sorry, widely :) [09:38] haha :) [09:38] :( [09:39] Does anybody want to review the fix? https://code.launchpad.net/~jtv/maas/bug-1064737/+merge/128891 [09:45] hm, had misremembered, thought daemon/appgroup was either/or, but the internet does claim using both as in your branch is the right thing [09:48] jtv: approved. [09:50] Thanks mgz [09:56] Is longpoll broken in the packaging? [09:56] I see "GET /MAAS/longpoll/?uuid=amq.gen-wyS-xRaw0j0Xo1_9NW_leV&sequence=4 HTTP/1.1" 404 1885 [09:56] Come to think of it I don't think I've seen it working in a while [10:00] rbasak: I don't think this has been reported. I'll test it. [10:00] thanks [10:00] And I just saw a JS test failure in trunk... could it be related? [10:01] I also filed bug 1064922 btw [10:01] Launchpad bug 1064922 in maas (Ubuntu) "Enlistment fails" [Undecided,New] https://launchpad.net/bugs/1064922 [10:01] The failure rbasak is seeing looks like server-side stuff. [10:01] Enlistment is completely broken in quantal until the maas-enlist precise SRU lands, AFAICT [10:02] (since by default we use precise ephemerals which use precise maas-enlist) [10:21] rbasak: I'm seeing the problem with txlongpoll too. [10:21] rbasak: I'll file a bug and investigate. [11:09] rbasak: just in case you'd like txlongpoll to work, all you need to do is: sudo a2enmod proxy_http && sudo service apache2 restart. The fix is up for review. [11:10] thanks rvba! [11:10] roaksoax: hi, could you please have a look at https://code.launchpad.net/~rvb/maas/packaging.fix-longpoll/+merge/128904 ? [11:24] I'll be off. Good night everyone. [13:17] okay, folded down from a new 20 line test case, to a rewrite of a test class... to a single added assertion [13:17] really want this to be a minimal patch [13:37] anyone know what the point of the maas-cli files command is? I.e., when would you use it? [13:52] is simplejson already a dependency? [13:52] or will that require a packaging change? [14:19] <_rvba> Daviey: can you please have a look at bug 1065062? I think we want that fixed in 12.10. [14:19] Launchpad bug 1065062 in maas (Ubuntu) "/var/lib/maas/celerybeat-cluster-schedule cannot be created by the cluster controller." [Critical,Triaged] https://launchpad.net/bugs/1065062 [14:19] <_rvba> flacoste: can you please have a look at bug 1065055? I think we want that fixed in 12.10. [14:20] Launchpad bug 1065055 in MAAS "celeryconfig_cluster.py imports utility method from maas (import_settings)" [Critical,Triaged] https://launchpad.net/bugs/1065055 === _rvba is now known as rvba [14:20] evilnick_: if you didn't get an answer, it's probably not used passed debugging [14:21] maas needs a files api because juju expects to be able to poke some strings into file storage, and the cli just exposes all the apis [14:21] evilnick: ^ in case your underscore breaks hilight [14:22] mgz: yeah, thx. I think there may be a few other commands that fall into the same category, so it is a bit of a waste me writing docs for them :) [14:22] some basic "internal thingy used for X" is fine, rather than nice user focuses stuff [14:22] that way a curious kitten doing --help knows to move on [14:24] mgz: my understanding is that simplejson is part of python 2.7 now [14:25] mgz: in the json module [14:26] mgz: but maybe not the fast C implementation... [14:26] rvba: indeed we do :-) [14:26] flacoste: mine too, but under the name 'json'... and the reason for that change is that bob ippolito continued hacking upstream and did various perf improvements... that didn't then get back into the json module [14:28] so, just doing s/import json/import simplejson/ will break unless the python-simplejson is also intalled [14:30] rvba: Oh golly [14:32] ...something is pulling it in currently, but I'm not sure how concrete that dep is [14:46] Daviey: care to have a look at bug 1065080 ? [14:46] Launchpad bug 1065080 in maas (Ubuntu) "The host in BROKER_URL is hardcoded to 'localhost'." [Critical,Triaged] https://launchpad.net/bugs/1065080 [14:46] anything for a giggle [14:47] done [14:47] Daviey: medium really? [14:48] Looks pretty critical to me. [14:51] rvba: https://wiki.ubuntu.com/Bugs/Importance [14:51] TBH, we rarely use critical [14:52] Daviey: ah ok. "Renders essential features or functionality of the application or dependencies broken or ineffective"… High then ;) [14:53] rvba: Right, but it's a config option? [14:53] it's REALLY a low :) [14:54] Daviey: the hostname of the broker_url passed around is hardcoded to 'localhost'. None of the cluster controllers apart from the one installed next to the region controller will be able to connect to the region controller. [14:54] rvba: I hate to go all technical on you.. but swirly alert [14:55] rvba: Do you want us to go to Blue alert ? [14:55] Daviey: ok. We have to fix that anyway :) [14:55] http://youtu.be/81W8tG3wH_4 [14:56] hehe :) [15:03] rvba: Oh, i thought it was coded into a config file [15:03] ? [15:04] Daviey: it is in a config file on the region controller side, then send to the cluster controller over the API when the cluster controller 'registers'. [15:05] s/send/sent/ [15:07] rvba: so can a human fix this? [15:09] Daviey: yes, sed -i s/localhost/real.ip.of.the.region/g /etc/maas/maas_local_celeryconfig.py will fix it (I've tested it). [15:10] well then.. it's a Medium! [15:10] :) [15:10] All right :) [15:10] If it was in .py files.. then sure.. I could see the urgency [15:11] rvba: the above bug, that's why you do sudo dpkg-reconfigure maas-cluster-controller [15:12] roaksoax: I don't think so. [15:12] roaksoax: 'sudo dpkg-reconfigure maas-cluster-controller' lets you specific the url of the API. [15:12] Then the cluster controller connects to the region (using that URL) and gets back the BROKER_URL. [15:13] Then it tries to get celeryd to connect to RabbiMQ using that BROKER_URL. [15:13] rvba: ah I see, so that should be in the region side [15:13] roaksoax: it is on the region side indeed. [15:13] In the config of the region. [15:14] That broker url is also used by the region worker (in this case it's fine, because the region worker is on the same machine as the region controller) [15:15] rvba: right, so a function needs to be added to update that when updating maas_local_settings to [15:16] roaksoax: I'm a bit unsure what to do here. Because the host of the region controller is already in many places. [15:17] rvba: I;ll take care of it [15:18] roaksoax: it's in DEFAULT_MAAS_URL in /etc/maas/maas_local_settings.py (region). [15:18] roaksoax: we could also use the 'host' part of the url given when running 'sudo dpkg-reconfigure maas-cluster-controller' [15:18] roaksoax: what do you think? [15:19] rvba: well I think that whenever we update DEFAULT_MAAS_URL, the BROKER_URL should be updated too [15:19] roaksoax: but what if the user later changes DEFAULT_MAAS_URL manuall? [15:19] manually* [15:21] rvba: that's their problem :). We don't care about that... I mean... the packaging should have taken care of setting that up on initial install [15:21] rvba: if the user chances it manually, the packaging cannot do anything about it [15:21] roaksoax: maybe we should extract the 'hostname' part out of DEFAULT_MAAS_URL. That would solve that problem. [15:22] I mean dynamically, in the upstream code. [15:22] rvba: yes, that's what I was gonna say, if that sits in the region controller and will sit there all the time, why not infer the broker url from maas_url if at the end of the day, they are gonna be the same [15:22] (as in the same IP/hostname) [15:22] roaksoax: I think it's the best thing to do. [15:23] roaksoax: I'll take care of it then :) [15:23] roaksoax: on update, when i get asked to keep orig config or maintainers version.. what should i do? [15:23] Daviey: Yes to overwrite [15:24] rvba: awesome thanks!! [15:24] Daviey: are we going to be releasing a new upstream release by the end of the week? [15:24] roaksoax: good, i did that [15:24] roaksoax: Yes, a new upstream.. latest.. before i go to bed on Thursday [15:24] Daviey: ack [15:25] That is the last point i can guarantee it to be on the cd [15:25] After that, it might be a 0-day SRU [15:26] rvba: could you please update your packaging MP, I had to merge some other stuff first because that had hit archive [15:27] Daviey: ok cool [15:27] roaksoax: I accepted that upload, as i assumed it would be overridden on your next trunk snapshot? [15:28] roaksoax: sure [15:28] Daviey: yes thank you! patches recently uploaded will be dropped with new upstream snapshot. I'm just keeping packaging branch in sync just in case [15:28] rvba: thank you!! [15:28] ok [15:35] roaksoax: changes pushed. [15:39] rvba: thank you [15:42] rvba: that proxy_http module comes with apache2 itself right? [15:42] roaksoax: yes [15:42] That module is shipped with apache: [15:42] $ dpkg -S /etc/apache2/mods-available/proxy_http.load [15:42] apache2.2-common: /etc/apache2/mods-available/proxy_http.load [15:43] rvba: ok fcool,t hat's weird though when loading wsgi (i think) the proxy_http should have been loaded too [15:43] roaksoax: apparently not. [15:44] yeah that's very rare cases maybe [15:44] i have never seen that issue :) [15:44] i always saw it being loaded [15:46] Yeah, I'm surprise too. But rbasak found that problem and I was able to reproduce on a canonistack instance so… [15:46] surprised* [15:53] rvba: are we wnating to eliminate maas-common some day? [15:54] roaksoax: I haven't thought much about it TBH. It's probably possible and not really complicated but definitely not something we want to do now. [15:55] rvba: ok cool then, because I think i'll move the creation of /var/lib/maas /var/log/maas etc etc to it [15:55] rvba: instead of doing it in cluster and region [15:55] roaksoax: sounds good. We're sure it will be installed before the cluster controller or the region controller right? [15:55] rvba: the only thing that cluster installs in /var/lib/maas is the scheduler for celery right? [15:56] rvba: maas-common is a dependency of both so it will [15:56] roaksoax: cool. Yes, that's the only thing that the cluster puts in there. [16:08] rvba: so cluster-controller only has 1 log too? [16:09] ls /var/log/maas [16:09] celery.log oops pserv.log [16:09] 2 actually [16:09] rvba: oops are also needed? [16:09] Well, it has been created by the oops package. [16:09] I /think/. [16:10] nope it is ceated by cluster [16:13] rvba: https://code.launchpad.net/~andreserl/maas/packaging_update_lp1065062/+merge/128988 [16:13] It's referenced in /etc/maas/pserv.yaml (oops:directory: "/var/log/maas/oops") [16:14] roaksoax: I thought you wanted to do that in maas-common…? [16:15] rvba: yes, but I have to do it, test it, etc etc, so I first prefer fixing that [16:15] rvba: and then evaluate the other [16:15] roaksoax: all right. [16:18] rvba: could you approve though please? [16:18] roaksoax: approved. [16:18] rvba: thankyou [16:18] :) === matsubara is now known as matsubara-lunch === matsubara-lunch is now known as matsubara [18:21] rbasak: still around? [18:22] Daviey: yes [18:22] smoser & rbasak: So how are we fixing the power problem? [18:22] enlistment + power setting [18:22] I thought that would be fixed with a precise sru? Or is the problem that this will take too long? [18:22] rbasak: take too long [18:23] smoser: How do you feel about including it in the ephemeral images? [18:23] (before it hits -updates) [18:24] i'm not opposed to doing that. [18:25] 2 comments [18:25] a.) its not a big deal to me, since we're already ppa'ing 4 packages in the ephemeral images for precise [18:25] Next smoser Okay JFDI, and create a new ephemeral image pronto? [18:25] what's the ubuntu equiv. of ftpmasters and how do I see what packages are queued to land in quantal... [18:25] b.) i'd like to hold off on spinning a "release" image until after (or on) release. [18:25] Daviey: enlist with quantal [18:25] mgz: Archive Admins, https://launchpad.net/ubuntu/quantal/+queue [18:26] Daviey, so i'd do it to 'daily' build. [18:26] which is easily configurable [18:26] Daviey: you ca enlist with quantal in the meantime too [18:26] Daviey: thanks, you beat google. [18:26] smoser / roaksoax: Okay. How do i fix this /right now/? [18:26] mgz: Wassup? [18:26] Daviey: go to the Settings and select quantal as the commssiioning release [18:26] Daviey: and make sure you have quantal, and that's it [18:26] mgz: I am an AA.. is there something you are expecting ? [18:27] Daviey: quantal ephemerals by enabling it /etc/maas/import_pxe_files [18:27] roaksoax: how do i make sure? [18:27] ok [18:28] Daviey: pinging SpamapS about it now. [18:29] need my brown paper bag juju fix. [18:29] mgz: sorry I got de-railed by wifi problems [18:29] mgz: will hopefully finish shortly [18:30] How do i cleanly delete all nodes now? [18:30] SpamapS: no problem! [18:30] Daviey: is the problme commissioning on enlistment? [18:30] mgz: can you describe the issue ? [18:31] roaksoax: i have 16 nodes with no power paramas.. would make sense to delete and recycle them.. no? [18:31] Daviey: right, so you can delete them thru maas shell [18:31] Daviey: the issue is I'm a moron. see bug 1064734 [18:31] or maybe the CLI if it is supported [18:31] Launchpad bug 1064734 in juju (Ubuntu Quantal) "ERROR Invalid 'tags' constraint 'set(['test-tag'])': No such tag using maas-tags to deploy with juju" [High,In progress] https://launchpad.net/bugs/1064734 [18:32] roaksoax: we still don't have multi-slect in GUI, ffs [18:33] SpamapS: Can you get that patch in asap? [18:34] Daviey: i have enlisted/commissioned with quantal without issues [18:34] so you should be good using that [18:34] i'll agree, enlistment should work fine from quantal. [18:34] Daviey, do you want me to create a new daily of quantal? [18:35] okay, that is fine.. i want to delete all nodes [18:35] Please advice how :) [18:35] smoser: is there a need? [18:35] i'm trying to check [18:35] well, you want that maas-enlist in it. [18:35] right? [18:35] SpamapS: Are you handling that upload? [18:36] Daviey: yes its almost done, just want to finish the test build [18:36] Daviey: these are not HP MicroServers either right [18:36] ? [18:36] SpamapS: did you see Kapil's comment? <-- mgz ? [18:36] (it still strips operators [18:38] Daviey: i guess from maas shell would be [18:38] from maasserver.models import Node [18:38] nodes = Node.objects.get() [18:38] for node in nodes: [18:38] node.delete() [18:38] Yeah.. but earlier n the cycle doing that caused an explosion [18:39] Daviey: i have done that verious times without issue TBH [18:39] ok [18:39] ITYM, node = Node.objects.all() [18:40] Daviey: yup, and updated in response, will note that in the mp [18:40] Daviey: yeah [18:40] Daviey: i just deleted nodes without issue [18:41] roaksoax: seems to have worked [18:41] thanks [18:42] Daviey: so now, are these HP MicroServers, no right? [18:42] roaksoax: no, these are proper rack servers [18:42] Daviey: ok, cool. so the ponly thing would be to enable quantal ephemeral images [18:42] maas-import-pxe-files to get them [18:42] and that should be all [18:43] adn of course changing the default release for commissioning (which will be used for enlistment too) [18:43] and that's done on the WebUI settings [18:47] http://pb.daviey.com/in2m/ :-) [18:47] ahh, i imported ephermals [18:47] yeah [18:49] roaksoax: Using maas-cli, how do i accept them all? [18:52] Daviey: that i don't know i haven't played much with it [18:52] matsubara: ^^ [18:54] smoser: Did we remove serial console then? [18:54] (for intel) [18:54] maas-cli maas nodes accept-all [18:54] Daviey, :-(. yes. [18:54] Daviey, ^ [18:54] https://code.launchpad.net/~smoser/maas/drop-ttyS0-kernel-opt/+merge/128747 :( [18:55] read the bug referenced. [18:55] there wasn't a lot of option. [18:56] if /dev/console gets assigned badly (which is possible if you specify console=/dev/ttyS0 and there is no hardware), then basically a lot of stuff ceases to work as writes to stdout start failing. [18:56] * roaksoax lunch [18:56] well ok.. But the sitation now is i have 16 nodes in epidermal with NFI why they haven't enlisted [18:56] you can "configure" console=ttyS0 on by editing the config file. [18:57] which just happens to be python source code in /usr/share/pyshared [18:57] :-( [18:57] and lost on upgrade, right? [18:57] smoser: i marked that bug as high [18:57] Daviey: have you figured out the way to delete systems? [18:57] Daviey: you can use the cli now to do that [18:57] Daviey, yes. [18:58] Daviey: list-systems to get the id and then a for loop calling delete-node [18:58] i might get the name of the commands wrong though [18:58] flacoste: ok, and to accept? [18:58] y [18:58] err [18:58] flacoste: Are the doc's being updated for this? [18:58] Daviey: there is an accept-all command [18:58] Daviey: yes, evilnick has all the details [18:58] Daviey: uploaded [18:58] mgz: ^^ [18:59] Daviey: i have re-added the console stuff tothe code [18:59] so you should be able to see it now [19:00] SpamapS: What was mgz's response to it still missing something, from Kapil? [19:00] roaksoax: You've added it to Mark's garage? [19:00] Daviey: yes [19:00] roaksoax: I really think we need to get this fixed properly.. as an /etc/ config option. [19:01] flacoste: ^ What do you think? [19:01] Daviey: i think that this should be kernel_opts node attribute [19:01] Daviey: allowing to set it independently to some value [19:01] w.t.f.. http://pb.daviey.com/Nr2C/ [19:01] however, it should default [19:01] I have an idea of how to do it [19:02] Daviey: yes, I,ve targetted that bug for 12.10 [19:02] flacoste: bug number? [19:04] roaksoax: Did you spot that /etc/maas/import_pxe_files has oneiric as a comment? shouldn't that be quantal as a comment? [19:04] Daviey: 2nd patch, all included [19:04] Daviey:yes i saw [19:05] SpamapS: cool [19:06] Daviey: bug 1044503 [19:06] Launchpad bug 1044503 in MAAS "kernel command line is not easily customizable" [High,Triaged] https://launchpad.net/bugs/1044503 [19:07] SpamapS / mgz: Accepted [19:07] Daviey: ty [19:08] matsubara: you can dailybuild now [19:08] flacoste: thanks [19:08] roaksoax, already did. [19:08] but thanks anyway :-) [19:08] cool :) [19:09] * roaksoax finally off to lunch [19:09] roaksoax: o/ [19:12] hmm, still no serial console [19:13] smoser: do you know why we lost rsyslog? [19:14] during enlistment? [19:14] yah [19:14] because we used to get it via the installer [19:14] ah [19:14] i suspec thtat we can probably get it back with some craftyness. [19:15] would be good, as currently i a SOB [19:16] enable ttyS0 [19:17] ok. i'm copying maas-enlist and building a daily with it inside. [19:18] smoser, roaksoax: who is taking care of bug 1064922? [19:18] Launchpad bug 1064922 in MAAS "Enlistment fails with 12.04 ephemeral images" [Critical,In progress] https://launchpad.net/bugs/1064922 [19:19] flacoste, that is fixed [19:19] i thought. [19:20] were you referring to bug 1063946 ? [19:20] Launchpad bug 1063946 in maas-enlist (Ubuntu Precise) "maas-enlist does not take power-type and power-params" [Undecided,New] https://launchpad.net/bugs/1063946 [19:20] smoser: ttyS1 is enabled.. i wonder if this is ttyS0 [19:20] smoser: right, it's a dupe [19:21] flacoste, not really a dupe [19:21] one is fails [19:21] one is "doesnt do ipmi enlistment" [19:21] indeed [19:21] we are going to get ipmi enlistment done in 2 ways [19:21] do we support enlistment now? [19:22] a.) SRU the maas-enlist package to 12.04 [19:22] or we reverted that? [19:22] smoser: I am massively concerned it is reporting an invalid mac address [19:22] b.) i'm going to build a daily image with quantal's maas-enlist inside. [19:22] Daviey, do you want some help? [19:22] smoser: please [19:22] http://pb.daviey.com/ssaw/ [20:14] smoser: SpampS already accepted maas-enlist in precise-proposed [20:15] hm. [20:17] smoser: were you able to help Daviey with the mac error? [20:17] getting there. [20:18] smoser: i'd say just ssh into the enlistment image and see why it is doing that [20:18] and that's it [20:18] done [20:18] Daviey: fixed? What was it? [20:18] roaksoax: no.. [20:19] :( [20:29] smoser: i figured out the issue [20:30] join to explain [20:43] https://code.launchpad.net/~smoser/maas/fix-login-block-reboot/+merge/129035 [20:43] anyone want that? [21:38] smoser: status? [21:38] roaksoax: ^? [21:39] smoser / roaksoax: How far off having it functioning are wek? [21:39]