[01:17] I've uninstalled maas and now I can't re-install it. I get an error message about password authentication failed for "maas" when I try. [03:10] smoser: would you be around for some help with omshell by any chance? [03:37] roaksoax maybe? [03:38] bigjools: hi [03:38] bigjools: shall we dance ? [03:38] Oh hi lifeless [03:38] He may still be lunching. [03:45] kk [04:21] lifeless: I iz back [04:21] cool [04:21] video ? [04:21] yarp, hang on [04:22] jtv, want to join? [04:23] ? [04:23] the call [04:23] about architecture [04:23] uh sure [04:23] plus? [05:54] bigjools: stuck again with dhcpd. Error reporting is minimal. I get an I/O error whenever I try a host with its IP address as the name. [05:54] Which could mean just about anything. There's certainly no attempt at I/O. [05:56] At least, not at the level that strace would see. [05:57] The server receives my omshell request, and replies with an I/O error. That's pretty much all that strace sees. [06:02] (Surprisingly, dhcpd checks for errors from fprintf/fputs/fputc by seeing if they set errno — but ignoring return value. I thought those functions were allowed to set a nonzero errno on successful runs, in which case the application should ignore it.) [06:04] jtv: paste session [06:05] bigjools: http://paste.ubuntu.com/1188377/ [06:05] Problem is, dhcpd can decide that the leases file isn't sane. The logging in the source doesn't look very helpful, but I might try getting at it. [06:08] jtv: you need quotes around the name value [06:08] [06:08] > set [06:08] argh [06:08] set name = "192.168.35.70" [06:08] Grrr. It parses an IP address quite happily. [06:08] yes [06:09] Didn't see the quotes in the template, either. [06:09] then it's buggy :/ [06:09] So not clear where they come from in our Omshell class. [06:09] But then how could it possibly work? [06:09] Oh well, at least I managed to reproduce the problem with a quoted name. Just not with the way we seem to do it in the Omshell code. [06:11] Aren't you getting Omshell errors in your Q/A setup, with manage_dhcp enabled? [06:11] omshell wasn;t getting called the last time I QAed [06:11] you fixed it recently [06:12] That'd do it. [06:12] Shall I just file a bug then, so that we at least check this? [06:12] we also need [06:12] set hardware-type = 1 [06:12] which is missing ATM [06:13] * jtv files one bug, 2 cards [06:27] jtv: "it would be good to see confirmation in Q/A" What? [06:28] The "I/O error" I'm getting might still be something weird in my setup. For whatever reason the omshell just accepts an IP address there, and parses & represents it in its own format. [06:29] Maybe it's just too generic and accepts any kind of argument though. [06:29] Anyway, I suspect we'd be wanting the generated hostname there, right? [06:31] jtv: I recreated locally [06:31] quoting fixes [06:31] jtv: hostname is irrelevant actually [06:31] Yes, I just don't know what the reason was why we had the ip address there. [06:32] jtv: it just needs to be a fixed identifier [06:32] If hostname is irrelevant, do we need it here at all!? [06:32] Ah [06:32] IP address works fine [06:32] Puzzling stuff! [06:32] omapi is shite [06:32] I'll note it on the bug. [06:32] It's from an age when getting a string all the way to the user was a job in itself. [06:32] mmyes [06:33] But I don't see how the errno handling could have been correct or portable. [06:33] errno = 0; [06:33] fprintf(...); [06:33] if (errno) handle_error(); [06:33] I've been bitten by that. AFAIK standard-library functions can set a nonzero errno even on success. [06:37] Well, since we've discussed it now, I'll just write up that fix. [11:56] roaksoax, are you around? [11:56] Too early, probably -- sorry for waking you, better go back to sleep. :) [11:57] Daviey, maybe you can help? I'm creating a custom upstart job to run dhcpd with our own defaults & config, and updating HACKING.txt. I have a few questions about it. [11:58] jtv: not yet oficially but what's up :) [11:58] Good morning :0 [11:58] :) [11:59] morning :) [11:59] jtv: what are your questions about it? [11:59] I'm creating a custom upstart job, so that we can use custom config files that we're free to modify. [11:59] Questions: [11:59] - Where do I put the upstart script? In the maas package, or in the packaging branch, or elsewhere? [12:00] - Where do I put the new /etc/default/maas-dhcp-server config/script file? [12:00] jtv: in the packaging branch [12:00] Ah OK [12:00] jtv: look at debian/maas.maas-celery.upstart [12:00] jtv: look at debian/maas.maas-*.upstart [12:00] - Are we keeping the maas-dhcp package? I'm updating HACKING.txt but just installing isc-dhcp-server won't do it any more. You'll need the custom stuff installed. [12:00] jtv: what's custom stuff? [12:00] Ahh and I just add my own maas.maas-dhcp-server.upstart ? [12:01] The custom upstart script and the custom /etc/default/... file. [12:01] jtv: yes, well kind of. It would be: maas-dhcp.maas-dhcp-server.upstart [12:01] Wow :) [12:01] So that answers my question about the maas-dhcp package. [12:01] jtv: and about the default, i think you can also do maas-dhcp.default [12:01] and the package system will take care of installing them in the appropriate places [12:01] Will it all automatically be owned by maas:maas? [12:02] jtv: nope [12:02] For files we want to write to, I thought it'd make sense to skip the sudo dance while we're at it. [12:02] Say, maybe it would be good to have a quick face-to-face chat about it? [12:02] jtv: so the idea is to write files in a maas owned location [12:02] That way I can say I had a proper pre-imp call. :) [12:02] and dhcp would just source from there [12:03] jtv: sure, give me 5 [12:03] * jtv gives roaksoax 5 [12:06] jtv: ready [12:06] See if this works... https://plus.google.com/hangouts/_/51b1cfe53b7b186ae3519d28fac1eff5b7ce2bdb?authuser=0&hl=en-GB [13:14] rbasak, https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1031065 [13:14] Ubuntu bug 1031065 in cloud-init (Ubuntu) "cloud-init-nonet runs 'start networking' explicitly" [Medium,Triaged] [13:15] right now i'm suspecting that you're hitting the other race condition that slangasek mentioned. [13:15] that you have /run but not "virtual-filesystems". [13:15] hm.. [13:16] yeah. thats the issue. [13:16] i wonder which if th evirtual fileysstems you're not getting. [13:16] rbasak, can you get the same log, but with '--verbose' added to the mountall command line in /etc/init/mountall.conf ? [13:17] smoser: sure. Though I'm running some tests right now - I'll need to do it a bit later [13:17] smoser: you still want the upstart --verbose too? [13:17] yeah. [13:17] OK [14:55] roaksoax: turns out the only real sticking point with the existing apparmor profile for dhcpd is the pidfile. And I can't find any documentation to support having multiple profiles for the same executable... what if we just hardlinked /usr/sbin/maas-dhcpd to /usr/sbin/dhcpd and created a new profile? [14:56] Oh hi there smoser — thanks for spotting the fatal flaw in our grand config-writing scheme. Currently working around it. [14:57] jtv, i talked with jdstrand about the apport changes. [14:57] and he's fine with them. [14:57] s/apport/apparmor/ ? [14:57] ie, adding a sectoin to the apport stanza to allow maas to have well defined config. [14:57] yes [14:57] sorry [14:57] i also use the words 'landscape' and 'launchpad' interchangeably [14:57] Burn in hell! [14:58] You unox geeks are all the same. [14:58] lol [14:58] Rolix, Relix — what's the difference? [14:58] jtv maybde jdstrand knows better [14:58] smoser but that doesnt solve precise [14:59] we can solve that. [14:59] but i'm hesitent to tell you how [14:59] because i think you'll like the idea for quantal also [15:00] And that's bad? Now I'm curious. :) [15:00] hold on, collecting data [15:01] k [15:02] ok . so /etc/apparmor.d/usr.sbin.dhcpd is pastebinned at http://paste.ubuntu.com/1189094/ [15:02] note lines 57 and 58 [15:03] i believe that renders to "/etc/apparmor.d/local/usr.sbin.dhcpd" (i'm not sure of the path, but obvioysly it does *something*) [15:03] that file is *not* a config file, so its more ok for maas to poke at it [15:03] Why isn't it a config file? [15:03] (It exists — I looked at it earlier) [15:04] hm.. strange that it exists. [15:04] Placeholder comment, basically. [15:04] I guess mostly to stop the #include from breaking. [15:04] But yes, I had that A-HAH moment too. :( [15:04] hm.. [15:04] well jdstrand suggested it. [15:05] if it is a conf file, but you are explicitly expecting someone to update it, then thats broken by design [15:05] Or BSD, depending on your nomenclature. [15:05] well, maybe its not so bad. [15:05] the conf file prompt comes when you have local changes to a file and hte package has changes to the file [15:06] it would seem *very* unlikely for the package to ship a change to that header stub [15:06] and if it did, the package maintainer would surely be aware that they were going to force config file prompts for all users. [15:06] so they'd want to avoid if at all possible. [15:06] anyway... i'd much rather see us change that file, and would seem ok for us to do that for precise. [15:07] reasonable? [15:07] Wouldn't the hardlink solution make it all a bit easier? [15:07] Assuming it would even work. [15:07] well. [15:07] a.) we shouldn't just look for the easiest solution [15:08] b.) how would you package the hardlink? [15:08] Create it in postinst? [15:08] then you have an unpackaged file in /usr/sbin [15:08] which screams WHAT_THE_%*#*_IS_THIS! [15:08] Ship a dummy, replace with hardlink? [15:09] then you have a file in /usr/bin that looks like it has been modified if you check checksums of your package [15:09] which, again screams *#*$** [15:09] True, true — but it's about the most policy-compliant trick we have so far, no? [15:10] i think editing the local file is ok. imo. [15:10] with hardlink, what happens on upgrade of package? [15:10] Heh. [15:10] with upgrade of isc-dchp-server? [15:10] You're right. [15:11] so my suggestion is update apparmor profile for quantal. [15:11] and we can even ask security team if they're ok with that change SRU to precise [15:11] Just add the maas directories & pidfile? [15:11] yeah. basically like eucalyptus [15:12] at least that is seemingly a designed solution to start off of [15:12] * jtv was tempted, so tempted, to jam our own pidfile into /run/eucalyptus [15:13] Actually the pidfile looks like the only one that we really can't circumvent decently. [15:13] There is some wiggle room for our own leases and config files. [15:13] jtv, well we were still running into permission issues on those files [15:13] but we could work around that. [15:14] We could own them. [15:14] well, there were issues. [15:14] at least when i actually tried this. [15:14] The upstart script ensures ownership/permissions of the leases file (and backup). [15:15] I imagine maas installation would take care of ownership/permissions of the config file. [15:15] you're saying running as the isc-dhcp user? [15:15] I was thinking of continuing to run dhcpd as dhcp, yes. [15:15] Make the config file owned by maas, but group dhcp. [15:16] if we can separate and run our own, i really think thats the best. [15:16] Or maas:maas for all I care, as long as dhcpd can read it. [15:16] then you can know that you can just kill it [15:16] when the person pusshes the button "do not run dhcpd" [15:16] its is cleaner separation. and well known path. [15:16] Wait... are you saying running as dhcp is best? Or that running as maas is best? [15:17] i'm saying maas running/maintaining its own dhcpd is best. [15:17] *not* using the isc-dhcpd [15:17] Oh, yes, that's what I'm coding up. [15:17] Well, it uses the isc-dhcpd *binary* of course. But not its config etc. [15:18] right. [15:18] But we need at least a new pidfile in the apparmor profile, I think. Everything else we _could_ keep as-is afaics. [15:19] you could use /ltsp :) [15:20] * jtv snarls at smoser [15:24] I guess I'll just add rules for /etc/maas/dhcpd.conf, /var/lib/maas/dhcpd.leases, and _something_ in /run. === matsubara is now known as matsubara-lunch [16:37] rvba: around? [16:39] jtv: still around? === matsubara-lunch is now known as matsubara [16:50] allenap: are you? [17:25] roaksoax: I'm just passing (it's dinner time here). Email :-/ [17:27] allenap hehe just left for lunch myself im from the phone [17:28] allenap just wantwd to know how to create a db migration to be able to add distro realease support [17:29] if theres template or what [17:29] roaksoax: have a look at HACKING.txt, it's a one line command once you've made the change to the model code. [17:31] rvba ok cool thanks [19:08] roaksoax, ok. you around [19:08] smoser: nope, this is an automated message [19:09] smoser: hehe, what's u? [19:09] up* [19:09] ok. so where can i get your dhcp fixed maas? [19:10] smoser: it is in the quantal archives, just not on the cd [19:10] ah. ok. [19:10] so we have generally functional dhcpd? [19:11] how? [19:11] smoser: maas-provision binary is the one generating the config [19:11] so we just needed sudoers access to it [19:12] not the best approach obviously, but enough for now until we have the other fixes in place [19:12] roaksoax, ok. i'm gonna try to walk through this, so i'll probably bother you again [19:14] alrighty === bjf is now known as bjf[afk] [19:53] roaksoax, ok. [19:53] so bug 1044229 is marked as fixed [19:53] Launchpad bug 1044229 in MAAS "DHCP config doesn't get written unless an inhuman combination of scripts is run" [Critical,Fix released] https://launchpad.net/bugs/1044229 [19:53] and that was a dupe of bug 1044061 [19:53] Launchpad bug 1044229 in MAAS "duplicate for #1044061 DHCP config doesn't get written unless an inhuman combination of scripts is run" [Critical,Fix released] https://launchpad.net/bugs/1044229 [19:53] so, how do i enable maas dhcp ? [19:53] oh. fun. from the api [19:54] any other way than api and ui ? [19:54] hm.. [19:54] and i go to my maas ui and it has 'manage dhcp' but ihave no dhcp process running [20:03] roaksoax, ? [20:04] smoser give me a sec plz [20:04] deal [20:04] issue is that dhcpd is not configured to listen on any interfaces [20:09] smoser you enable it by installing maad-dhcp [20:11] $ dpkg-query --show maas-dhcp [20:11] maas-dhcp 0.1+bzr971+dfsg-0ubuntu1 [20:11] button is highlighed in maas web ui [20:11] maas-dhcp installed [20:12] no isc-dhcp process [20:12] http://paste.ubuntu.com/1189590/ [20:12] that is everythign i've done so far [20:12] (except not added daily builds) [20:12] as you told me archive updates was good [20:14] roaksoax, what am i doing wrong? [20:14] smoser did you accept thr cresyion of the dhxp betwork [20:14] so, no [20:15] smoser so when you install maas-dhcp and say yes and configure the subnet then it gets enabled [20:16] i just dpkg-reconifgure maas-dhcp [20:16] and isc-dhcp-server still wont run === bjf[afk] is now known as bjf [20:17] what am i doing wrong? [20:18] smoser, roaksoax: I just reproduced the same thing in the lenovo lab [20:19] installed the latest package, answered the dhcp config questions but isc-dhcp-server keeps respawning saying there's not subnet declaration for eth0, eth1 or eth2 [20:19] and /etc/dhcp/ doesn't have any maas config [20:20] roaksoax, is the version i have listed above sufficient? [20:20] yeah. i have latest package per https://launchpad.net/ubuntu/+source/maas (44 minutes old) [20:21] oh wait. i have 0ubuntu1 and 0ubuntu2 is 44 minutes old [20:23] I'm using 0.1+bzr971+dfsg-0+972+74~ppa0~precise1 from https://code.launchpad.net/~maas-maintainers/+archive/dailybuilds [20:23] actually: 0.1+bzr971+dfsg-0+975+74~ppa0~precise1 [20:23] which is the latest latest crack [20:23] :-) [20:24] yeah, an di just by hand applied https://launchpadlibrarian.net/114878785/maas_0.1%2Bbzr971%2Bdfsg-0ubuntu1_0.1%2Bbzr971%2Bdfsg-0ubuntu2.diff.gz [20:24] to no avail [20:32] smoser: ok i'm back [20:36] k [20:36] i can let you in if you want to poke around, but i [20:36] i've basically done nothing other than that pb [20:40] smoser: ok so, the first time you install maas-dhcp, if you configure it, and dhcp is disabled, then it will be enabled [20:40] so what are you saying ? [20:40] its enabled. [20:40] thats fine. [20:40] it doesnt run [20:40] or do anything [20:40] thats not. [20:41] smoser: right, but is it enabled on the WebUI? [20:41] yes [20:41] smoser: and no config file in /etc/dhcp/dhcpd.conf? [20:41] smoser: have you rebooted the machine? [20:41] err [20:41] i mean, restart apache2? [20:41] or pserv or celery? [20:42] well, /etc/dhcp/dhcpd.conf exists. as ait always did. [20:42] smoser: right, but maas should have written a config to it [20:42] i did for p in celery pserv txlongpoll; do sudo restart maas-$p; done [20:42] and have actually rebooted since installing maas. [20:42] let me see [20:43] ubuntu@10.55.60.167 [20:44] [2012-09-06 20:44:27,803: INFO/PoolWorker-3] Not sending DHCP leases to server: not all required knowledge received from server yet. Missing: api_credentials, maas_url, nodegroup_name [20:45] smoser: Version: 0.1+bzr971+dfsg-0ubuntu1 [20:45] that's the reason [20:45] it is not ubuntu2 [20:46] roaksoax, I get the same behaviour (not the same log entry though) on 0.1+bzr971+dfsg-0+975+74~ppa0~precise1 [20:46] matsubara: the fis is on packaging, not on upstream [20:47] s/fis/fix [20:47] ah ok [20:48] roaksoax, i manually added the sudoers [20:48] what's the package revno I should be using to build with? [20:48] and restarted since then [20:48] smoser: that should work then [20:48] you can look. [20:49] $ sudo cat /etc/sudoers.d/99-maas-sudoers [20:49] maas ALL= NOPASSWD: /usr/sbin/service isc-dhcp-server restart [20:49] maas ALL= NOPASSWD: /usr/sbin/maas-provision [20:49] matsubara: the release ubuntu2 [20:49] matsubara: 1 before last one i think [20:50] smoser: disable/enable from the web ui [20:51] $ ls -altr /etc/dhcp/dhcpd.conf [20:51] -rw-r--r-- 1 root root 3602 Jul 10 21:29 /etc/dhcp/dhcpd.conf [20:51] smoser: lket me check locally first as network is crappy [20:51] again [20:51] (Jul 10). no dice [20:54] smoser: is DNS enabled? [20:55] i have 2 nice check boxes checked [20:55] "Enable Dns", "Manage DHCP". They're really pretty, and orange. [21:01] smoser: return self.run(*args, **kwargs) [21:01] File "/usr/lib/python2.7/dist-packages/provisioningserver/tasks.py", line 322, in restart_dhcp_server [21:01] check_call(['sudo', 'service', 'isc-dhcp-server', 'restart']) [21:01] File "/usr/lib/python2.7/subprocess.py", line 511, in check_call [21:01] raise CalledProcessError(retcode, cmd) [21:01] CalledProcessError: CalledProcessError [21:01] weird [21:02] where do you see that? [21:02] duh. [21:02] not wierd [21:02] "sudo service" [21:02] maas can't sudo [21:02] yeah [21:02] which is weird [21:03] smoser: and magically, it now works [21:03] it can though [21:04] it is in /etc/sudoers.d/ [21:04] for maas [21:04] what magically works ? [21:04] smoser: [21:04] smoser: DHCP config was written now [21:05] /etc/dhcp/dhcpd.conf ? [21:05] on my system? [21:05] smoser: http://paste.ubuntu.com/1189688/ [21:05] smoser: on my test env [21:05] let me access yours [21:05] ubuntu@10.55.60.167 [21:07] smoser: can you disable/enable dhcp config again please [21:07] from maas webui [21:08] sure [21:08] disabled [21:09] enabled [21:09] smoser: so just did a dpkg-reconfigure maas-dhcp and it worked [21:10] hm.. [21:11] smoser: seems to be (wild guess) that when maas gets installed (maas-dhcp really) it tries to create the DHCP database, and enables it (that's why it is enabled on webui). However, since the creation of the dhcpd config file fails (because maas-provision not in sudoers) the dhcp client network doesn't become permanent [21:11] so when we add maas-provision to sudoers [21:11] and dpkg-reocnfigure maas-dhcp [21:11] it success on creating [21:11] hence the network is created [21:11] i will check your theory [21:12] cool [21:12] * roaksoax will try to finish the quantla support [21:15] hm.. [21:15] it seems to have worked. [21:15] although i think i'd done that before [21:15] where did you see the stack trace you showed above? [21:16] smoser: celery log [21:16] /var/log/maas/celery.log [21:17] hm.. i never saw that htere. [21:19] matsubara, could you open a bug based on the above? [21:20] smoser: there's no bug really, it is fixed [21:20] smoser: the problem ois that it has not been released [21:20] in the archives [21:21] smoser: https://launchpad.net/ubuntu/+source/maas/0.1+bzr971+dfsg-0ubuntu2 [21:22] http://paste.ubuntu.com/1189725/ [21:22] roaksoax, ^ [21:22] file gets written now, but it is bad [21:23] smoser: isn't that wrong configuration [21:24] smoser: ah yes, it is badupstream side [21:24] http://paste.ubuntu.com/1189729/ [21:24] smoser, roaksoax I think there's another bug. the dhcp config is written but next-server is set to 127.0.0.1, is that correct? [21:24] right. [21:24] i was going to point that out [21:24] ok [21:24] * smoser has to go now [21:24] matsubara: *and* I think there's another issue [21:24] subnet 192.168.1.5 netmask 255.255.255.0 [21:25] that should be the network address not the starting range [21:25] maas-provision asks for starting range [21:25] and should probably calculate the network address [21:25] roaksoax, this is the config the package wrote for me: https://pastebin.canonical.com/73987/ [21:27] roaksoax, this https://code.launchpad.net/~maas-maintainers/maas/packaging.precise has the latest fix you mention that's included in ubuntu2 right? My package was built from that source [21:27] matsubara: try dpkg-reconfigure maas [21:28] roaksoax, ok. it offered to change the MAAS pxe address. [21:29] matsubara: err i meant maas-dhcp [21:29] matsubara: htat's a bug for sure when it tells next-server [21:29] or mabe not as it bings to localhost [21:29] we;ll have to do some real pxde boot testing on that [21:30] roaksoax, ok, reconfigured the maas-dhcp package, it asked me the same questions asked when I installed and the dhcpd.conf still have next-server as 127.0.0.1 [21:31]