[00:19] smoser: around already?> [00:29] here, roaksoax [00:30] smoser: so we are gonna need to pass the "config" to my script, instead of harcoding it [00:31] smoser: so I'm guessing I should pass the config to maas-signal, which will pass it to maas-autodetect-ipmi? [00:31] what do you mean pass [00:31] smoser: as it tell it where's the template / config [00:31] as in* [00:31] smoser: https://pastebin.canonical.com/75758/ [00:32] smoser: that one, we need to tell the location of that template [00:32] template/config [00:32] i'm sorry. i'm still being dense. [00:33] smoser: bmc-config --commit --filename /location/of/cofig/file.ipmi [00:34] hello guys [00:34] bigjools: good morning [00:35] I might have done something bad to my debconf database, I am getting this trying to install the maas package after previously purging it: http://paste.ubuntu.com/1255593/ [00:35] (I need to sh -x the postinst to get that output, otherwise it's fails silently) [00:35] bigjools: what new package are you trying to install? the latest I made available in experimental PPA? [00:35] s/it's/it/ [00:36] roaksoax: one I made myself yesterday [00:36] bigjools: that's fixed then :) [00:36] it's not the same problem as the one you fixed [00:36] since I'd already found that and tried to fix it and then got held up by this :/ [00:37] bigjools: are you referring that this doesn't fix it? http://bazaar.launchpad.net/~maas-maintainers/maas/packaging/revision/113 [00:38] see the postinst [00:39] daily ppa has maas-dhcp installable again [00:39] roaksoax: it's hard to see what the effect of the changes are - I was assuming you were referring to the fix for "service isc_dhcp_server stop" [00:39] bigjools, just curious, what does "fix committed" / "fix released" mean in maas ? [00:40] bigjools, he was referring to the celery postinst [00:40] i think [00:40] smoser: not a a lot. Tarmac changes bugs to fix-committed but we don't distinguish that with fix-released on trunk [00:40] https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1050523 [00:40] Ubuntu bug 1050523 in maas (Ubuntu) "maas kernel cmdline must include iscsi_initiator" [High,Triaged] [00:40] so you marked that "fix released" [00:41] i'm just curios, so i can at least attempt to be consistent. [00:41] smoser: yes, they are all fix released if they are in trunk, I am not using fix-committed [00:41] then they disappear off bug listings [00:42] bigjools: check the changes made in maas-region-controller.postinst, I removed and if statement [00:42] an* [00:42] roaksoax: aha! [00:43] nice one [00:43] bigjools: there's a related issue though that I can't find a solution for [00:43] bigjools: if you remove (not purge) and install again, it will show the same error [00:44] bigjools: and I already know how to fix the maas-celery thing, i'll work on it in a bit though [00:44] roaksoax: ok - jtv landed a change so we can pass uids I think [00:45] bug 1060114 [00:45] Launchpad bug 1060114 in MAAS "DELETE operations are not idempotent" [Medium,Invalid] https://launchpad.net/bugs/1060114 [00:45] heh [00:45] bigjools, https://bugs.launchpad.net/maas/+bug/1058137 [00:45] Ubuntu bug 1058137 in MAAS "no way to get first user's first set of api creds" [High,Triaged] [00:45] if i just put that in comment 4 into a python script and run it [00:45] i stack trace [00:46] matsubara: that python script works for you? --^ [00:46] http://paste.ubuntu.com/1257137/ [00:47] smoser: export DJANGO_SETTINGS_MODULE=path/to/settings.py [00:47] /etc/maas/whatever [00:48] bigjools, yes, I think I did some path hackery to get it to work [00:53] grumble grumble, roaksoax removed my changelog attribution :) [00:54] bigjools: no i didn't :) [00:54] -- Andres Rodriguez Tue, 02 Oct 2012 13:39:47 -0400 [00:54] you did :) [00:54] bigjools: yeah I made changes to the changelog [00:54] dch -i changes that :) [00:55] * bigjools shakes fist at dch [00:55] -u surely? [00:55] or -e? [00:55] bigjools: btw... an assumption I am working with if the fact that it doens't matter how many releases we release in PPA [00:55] bigjools: the packaging should remain UNRELEASED [00:55] ok [00:55] and only change when we release to archives [00:56] bigjools: but since the changelog is being pilling up I think is ok [00:56] roaksoax: well dch -i doesn't change the release it adds a new one, so something changed the attribution I added [00:56] bigjools: because entries of releases that haven't been uploaded to the archives will appear in the changelog even though we didn't [00:57] yup [00:57] soyuz is great :) [00:57] bigjools: dch -e -DUNRELEASED works? [00:57] errerr dch -e -Dquantal [00:59] roaksoax: in r113 the change line was updated, what dch option does that? [00:59] i tihnk in thie change log there we need to ditch the double bzr branch [00:59] we get one "for free" from the daily ppa [00:59] biuld [01:00] so we should either be pedantic about getting that updated in the code on each commit (or tooling) or just remove the bzr revno from there. [01:00] bigjools: dch -i does it for me [01:00] true, but it's needed to build locally from recipe isn't it? [01:00] roaksoax: -i adds a new one for me [01:01] bigjools, well we can just add a script that does the build locally and inserts it. [01:01] smoser: just need to fix the debian/rules target I expect [01:01] roaksoax: bzr diff -c 113 debian/changelog [01:05] bigjools: dch -e --nomainttrailer [01:06] roaksoax: aha [01:06] roaksoax: useful, it saves me having to use -k when recipe building :) [01:06] indeed :) [01:07] bigjools: btw.. i already told allenap... no scaping from Peruvian Pisco is allowed at UDS [01:07] smoser: ^^ [01:07] ?! [01:07] escaping* [01:07] :) [01:07] does that involve naked bodies and mud? [01:07] otherwise I am totally not up for it [01:08] lol [01:09] roaksoax: getting this on the experimental package: http://paste.ubuntu.com/1257167 [01:13] bigjools: as an error? [01:13] ok. this is silly. [01:13] bigjools: can you pastebin a full log? [01:13] http://paste.ubuntu.com/1257171/ [01:14] matsubara: can you paste a full hackery for smoser please? [01:14] smoser: do you think i should allow sending several config files rather than just one? [01:15] roaksoax: http://paste.ubuntu.com/1257172 [01:15] roaksoax, do you have a reason for doing so [01:15] ? [01:15] smoser: so if someone sets custom ipmi configs they can be sourced automatically? [01:15] bigjools: yeah that's dbconfig being an asshole i think [01:15] it doesn't sound unreasonable. [01:15] roaksoax: yay [01:16] roaksoax: I'll re-purge [01:16] bigjools: yes, just to be sure, remove dbconfig-common too and make sure /etc/dbconfig-common is empty [01:16] bigjools: i discovered that maas.conf under dbconfig-common is preserverd, and we also need to remove that [01:16] bigjools: i'll take care of that along the celery stuff [01:16] along with* [01:16] bigjools, smoser: something like this should work: https://pastebin.canonical.com/75761/ [01:18] matsubara, well, thats basically what i did [01:18] roaksoax: /etc/dbconfig-common/maas-region-controller.conf exists [01:18] weird I'll remove it [01:19] roaksoax: on re-install (after removing it) I now get a debconf page telling me it was currently iunstalled and locally mnodified [01:19] installed* [01:20] * bigjools installs package version [01:21] ok, guys, gotta go. talk to you tomorrow. [01:21] have a good night/day === matsubara is now known as matsubara-afk [01:22] bigjools: say yes [01:22] :) [01:23] smoser, roaksoax: how long are you in CPH? [01:23] bigjools: 2 weeks [01:23] just UDS? [01:23] ah [01:23] same as roaksoax [01:23] too long [01:23] smoser: yay!! [01:23] lol [01:24] ok plenty of time to meet up for an evening food/drink [01:24] yeah too long [01:24] my wife is going to self destruct looking after our twins on her own [01:25] Hi folks [01:25] hi jtv [01:26] i'm really sorry for being a dolt. but some help would be greatly appreciated. [01:26] http://paste.ubuntu.com/1257179/ [01:29] User isn't a MAAS thing. It comes with Django. [01:30] django.contrib.auth.models.User [01:30] How it got to be in contrib I may learn someday. How it got to stay there I just won't believe no matter what you tell me. [01:30] jtv: should still work [01:31] however I get this: [01:31] from django.db import utils [01:31] ImportError: cannot import name utils [01:31] at the bottom of the trace [01:31] https://bugs.launchpad.net/maas/+bug/1058137/comments/5 [01:31] Ubuntu bug 1058137 in MAAS "no way to get first user's first set of api creds" [High,Triaged] [01:31] smoser: http://paste.ubuntu.com/1257181/ with http://paste.ubuntu.com/1257183/ --> what do you think? [01:31] got it [01:32] smoser: phew :) [01:32] thank you bigjools jtv [01:32] np [01:33] bigjools: btw... I think we need to be able to pass kernel arguments [01:33] bigjools: and be able to edit that [01:33] pxe template? [01:34] bigjools: console=ttyS0 [01:34] bigjools we need to be able to edit things like that [01:34] bigjools: so yes, it should be added to the pxe template [01:34] bigjools: but should probably be per node [01:34] if one wishes to change that from a node, or set of nodes [01:35] roaksoax: how important is this? [01:36] should be easy enough to do, but I need to prioritise against the thousand other bugs [01:36] bigjools: ver important i'd say, even sabdlf mention we should be able to edit it. For example, console configuration needs to be done manually directly in the BIOS for ipmi cards [01:36] eugh [01:36] bigjools: so they configure serial to be ttyS20 [01:37] so maas, as it stands right now, sends console=ttyS1 [01:37] ok, Node needs a custom "kernel_params" field then, and we can add that to the pxe template [01:38] bigjools: indeed, so we don't really need it to be displayed in the WebUI [01:38] roaksoax, bigjools well, per-node kernel settings are not all that important really. [01:38] i opened that bug [01:38] and i do think they need to be editable [01:38] smoser: right, so we can set global settings, but if the user wants to change them, per node, he needs to [01:38] but realistically, in a large scale, you're going to have a couple groups of systems [01:38] my question is, important for *this release* [01:38] so per-node would suck anyway. [01:38] i cant think of them being a hard requirement for this release. [01:39] smoser: right but for example, in sabdlf's cluster I had to manually edit what console to output in maas code, because it was configured differently on the BIOS [01:39] but the ability to append cmdline arguments without editing python source would be nice. [01:39] smoser: I am thinking that the cluster controller would want to set the parameters when it uploads node definitions (in the future) [01:39] bigjools, ok. heres the whole sucky problem. [01:39] all this stuff will/should be automated [01:39] a.) before a node is enlisted, you know nothing about it. so you can't really customize stuff per-node at that point [01:40] b.) after that point, you could have some custom settings [01:40] roaksoax: https://code.launchpad.net/~julian-edwards/maas/broken-dns-install-bug-1060549/+merge/127621 [01:40] but *really* [01:40] c.) in a perfect world, give some knowledge of the setup and the node (hw and the like) you could perfectly generate all that anyway. [01:40] I disagree with (a) since the cluster controller may know all about its hardware already [01:40] in which case you're to 'c' [01:40] so you dont need them [01:40] :) [01:42] roaksoax, i think you'd have been fine with the ailbiyt to append cmdline settings. [01:42] right? [01:42] and i would have in all my testing been ok to do the same. [01:42] that would releieve a lot of pain [01:42] bigjools: looks good, make sure + add_user_group [01:42] 49 + [01:42] is ident'd correctly [01:42] smoser: huh? [01:43] roaksoax: GRARGH. The rest of the file has got TABS in it [01:43] I used 4 spaces [01:43] TABS ARE EVIL [01:43] smoser: btw... i'm just gonna pass maas-signal with an argument with the json stuff, such as maas-signal --config --power-setings [01:43] * roaksoax uses tabas for shell scripts and spaces for python code [01:44] TABS ARE EVIL [01:44] :) [01:44] xD [01:44] because you end up with what you see on the MP [01:45] do you care if I s/\t/ /g ? [01:45] bigjools: different schooling I guess.. I was always a TAB user [01:45] bigjools: go for it [01:45] we're very particular about our code formatting in the Launchpa^WCloud Engineering team [01:45] smoser: yeah I think we need to be able to append that [01:46] smoser: adding the per-node isn't really that big of a deal, we don't even need to display it in the UI, just be able to modify it per node from maascli [01:47] roaksoax: can you approve the MP please [01:48] bigjools: done! [01:48] smoser: ok I think it is done i just need to test it [01:49] * roaksoax heads home [01:49] smoser: i'll go home and test it on the HP mini servers [01:49] cheers [01:56] jtv: it looks like you didn't fix the upstart conf to expect a fork, or am I missing something? [01:57] bigjools: I didn't get around to looking into that; I assumed it already expected a fork but got the wrong one. [01:58] jtv: ok I 'll sort it out [01:58] Based on what you said, I thought the packaging branch already expected a fork. [01:58] jtv: I tried it but it hung, I'll try it again [01:59] :( [01:59] now there *is* a fork :) [01:59] it should work [01:59] There was a fork before as well. [01:59] That hasn't changed. [02:00] But I removed the spurious fork that preceded it. [02:00] hmm [02:02] jtv: something else is forking [02:02] jtv: http://paste.ubuntu.com/1257202 [02:03] Well it can fork right off. [02:03] * bigjools straces it [02:04] ARGH [02:04] it's the sodding wrapper script [02:11] jtv: when starting a python script, straces shows that it forks off this lot: "id, sh, ldconfig, sh, ldconfig, sh, uname" [02:12] wtf [02:12] wtf indeed [02:13] So celeryd is a wrapper script? [02:13] no [02:13] maas-provision [02:13] the one installed by the package that checks for uid==0 [02:14] Grrr [02:14] so basically python is forking a load of stuff off before it starts your actual code [02:14] unless it's hiding in maas-provision? [02:15] I bet it's import side effects. :( [02:15] I tried this and found no forks: $ strace python -c 'print' 2>&1 | grep fork [02:16] I am doing this: [02:16] sudo PYTHONPATH="/usr/share/maas${PYTHONPATH:+:}${PYTHONPATH}" CELERY_CONFIG_MODULE="celeryconfig_cluster" strace -o /tmp/strace.log -fFv /usr/bin/python -m provisioningserver start-cluster-controller http://maas.internal.example.com/ [02:16] (ignoring the wrapper for now) [02:17] oh god [02:17] even if we get this right it won't work jtv [02:17] Yes, my son? [02:17] because the fork can happen ages later [02:17] so the upstart script will always hang until the admin accepts the cluster [02:18] Yes [02:18] * bigjools gives up [02:18] Then maybe we should kill two birds with one stone. [02:18] Move the polling into the child process, and instead of exec'ing, just have it hang around as basically a wrapper to celeryd. [02:19] not sure it'll work ATM anyway [02:19] because of these 6 spurious forks [02:19] That way, start-cluster-controller will hang around for as long as celeryd is running. [02:19] I'd rather not do that [02:20] but let;s see what roaksoax thinks [02:21] * bigjools lols at jtv's continued wrong merge targets :D [02:21] Yes, it's become part of the process. Why change it now? [02:30] bigjools, roaksoax: I think upstart can manage a process that just never exits. If so, I think it would simplify our problem a lot if we could just keep the maas-provision process alive, instead of forking & exiting. [02:30] http://paste.ubuntu.com/1257227/ [02:30] thoughts? [02:31] is node-group-interfaces valid? [02:32] different than http://paste.ubuntu.com/1257229/ [02:32] ok [02:33] * bigjools looks at code [02:33] smoser: nodegroup called "master" does not exist [02:33] Wait... did I seriously just catch a snippet of advertising for the miraculous innovation of... coffee without sugar!? [02:34] well, uuid of master I mean [02:34] so what is 'master' there? [02:35] must be the uuid of an existing nodegroup [02:35] maas-cli api localmaas node-groups list [02:36] it would seem like no [02:36] http://paste.ubuntu.com/1257235/ [02:37] got 'master' as you said [02:37] hm.. [02:39] if you list existing NGIs does it show? [02:39] hey, this is not a maas bug, where should be be retargeted? https://bugs.launchpad.net/maas/+bug/1060194 [02:39] Ubuntu bug 1060194 in MAAS "cannot find "bnx2-mips-09-6.2.1a.fw" when booting from maas" [Undecided,New] [02:40] woohoo. [02:42] bigjools, initramfs-tools maybe [02:42] ok [02:42] i actually think i saw this. [02:42] thanks [02:42] err... jsut reading code thought "how strange, it brings up networking without looking waiting for udev" [02:42] smoser: or cloud-initramfs-tools? [02:42] no [02:42] initramfs-tools [02:42] ok done [02:42] incidently, it is probably fixed now [02:43] because cloud-initramfs-tools code goes looking for NICs and waits for udev [02:43] :) [02:43] smoser: i'm gonna have to test the cloud-init stuff tomorrow... the mini servers are gonna take forever to install [02:44] smoser: err upgrade [02:44] upgrade? [02:44] smoser: my maas mini server needs to be upgraded :) [02:46] bigjools, ok. so my goal in the last escapade is to configure dns [02:46] http://paste.ubuntu.com/1257248/ [02:47] that gets me a valid response [02:47] but my goal is to get dhcpd functioning [02:47] cool [02:47] and i still have no /etc/maas/dhcpd.conf [02:47] Hi roaksoax! Did you see our problem with the fork-tracking for the cluster controller? [02:47] smoser: check the celery log [02:48] jtv: howdy. no I didn't [02:48] jtv: any lp [02:48] jtv: any lp bug? [02:48] smoser: 1. has it received secrets from the region, 2. did it get any DNS tasks? [02:48] roaksoax: bug 1059453. Turns out maas-provision does a lot of forks before it gets to the one we'd want upstart to track. [02:49] Launchpad bug 1059453 in maas (Ubuntu) "The celery cluster worker is not properly stopped" [Critical,Triaged] https://launchpad.net/bugs/1059453 [02:49] smoser: is https://bugs.launchpad.net/bugs/1060331 fixed now? [02:49] Ubuntu bug 1060331 in MAAS "daily ppa install fails fully automated" [Undecided,New] [02:50] bigjools, sure. [02:50] i tihnk it was the isc_service_stop thing [02:50] yeah [02:50] jtv: i think that's a question for james hunt [02:50] jtv: if not, we are probably have to run it as we used to [02:50] roaksoax: oh FWIW there's a trick to conf file removal [02:50] I asked colin about it [02:51] bigjools: what did he say? [02:51] roaksoax: "man dh_installdeb, man dpkg-maintscript-helper" :) [02:51] and search for "conffile removal" [02:51] bigjools: ah yeah, i know that :) [02:52] can i turn of header output from the cli ? [02:52] so basically upstart confs are not treated specially [02:52] smoser: not yet :( [02:52] bigjools: nope, so we can use rm_conffile [02:52] Thanks roaksoax -- I'll see if I can reach JH later. [02:52] roaksoax: exactly [02:52] bigjools: we need to do the same for maas.conf under /etc/dbconfig-common [02:52] bigjools: i'll take care of it :) [02:52] roaksoax: ok thanks! [02:55] bigjools, http://paste.ubuntu.com/1257255/ [02:55] that would seem the relevant bit of celery log [02:57] yay [02:58] smoser: anything for dhcp? [02:58] http://paste.ubuntu.com/1257259/ [02:58] full log [03:00] is maas-dns installed? [03:02] no [03:02] that might help :)_ [03:02] also maas-dhcp? [03:02] well the former will ensure the latter [03:03] maas-dhcp, yes. [03:04] you're saying dns ensures dhcp? [03:04] yes [03:04] k. [03:04] we can't do dns without dhcp [03:04] I suspect a signal is missing to push out the dhcp config when NGI changes [03:06] hmm no that's wired [03:11] bigjools, so am i at least going down the correct path here? [03:11] smoser: yes [03:11] is there some expected way that someone would set this up? [03:11] this is fine so far, if it's not working it's a bug [03:14] smoser: ok so the script itself seems to work [03:15] smoser: just need to test in a deployment [03:16] roaksoax, nice work. [03:19] i'll try to get that tonight [03:19] i just realized that i need to reconfigure the network :( [03:20] my home network :( [03:20] Setting up maas-dns (0.1+bzr1134+dfsg-0+1136+113~ppa0~quantal1) ... [03:20] chown: invalid user: `maas:root' [03:20] dpkg: error processing maas-dns (--configure): [03:20] subprocess installed post-installation script returned error exit status 1 [03:20] smoser: that's already been fixed by bigjools [03:20] smoser: there's a fix just landed [03:20] needs rebuilding [03:20] * bigjools hits the daily rebuild button [03:21] should be ready (published) in 10-15 mins [03:23] I <3 recipes [03:29] * bigjools grabs food [03:33] bigjools, ok [03:33] http://paste.ubuntu.com/1257282/ [03:34] ^ results in http://paste.ubuntu.com/1257283/ [03:35] and no /etc/maas/*dhcp* [03:44] well, i have to go to bed. [03:48] tell me what stupid thing i've done there. [03:48] http://paste.ubuntu.com/1257288/ is some node group info. [03:48] it seems my operation to update my node worked. [03:50] ok. bed. [03:51] good night. [03:51] smoser: hmmm looks like a bug. I'll get rvb to look, unless jtv wants to [03:51] nn smoser [03:51] nn smoser [03:51] I'm looking up the pastes [03:52] oh. one thing to note. there are things in /etc/bind/maas [03:52] $ ls /etc/bind/maas/ [03:52] named.conf.maas rndc.conf.maas zone.master [03:52] named.conf.rndc.maas zone.77.168.192.in-addr.arpa [03:52] that looks expected [03:52] right [03:52] but no dhcp [03:52]