=== lifeless_ is now known as lifeless [12:06] bigjools: fwiw I see the app server reporting upload_leases requests on the API — so at least the initial loop of the beat jobs is closed! [12:07] Hmm… looks like the request is being redirected though. [12:07] For authentication. [12:08] oops [12:08] It's getting reported as success, but that may just be because the client is successfully told to bugger off — and I guess it just gives up. [12:09] It follows the redirect, gets the login page, and… done. [12:12] Hmm. Let me check the database. [12:19] Nope. Definitely not coming through. [12:30] allenap: I just rebased my armhf patch series to today's trunk and have a conflict due to https://code.launchpad.net/~allenap/maas/new-tftp-layout/+merge/120182 [12:30] bigjools: group write means changing the permissions for path's where other package has been installed [12:30] bigjools: meaning we would be messing with the permissions of other applications [12:31] allenap: not sure what the implications are for me right now. It feels like this hinders my progress since there's no replacement right now. Any ideas? [12:31] rbasak: ok [12:31] oops [12:31] roaksoax: ok [12:32] rbasak: There are a few more branches on the way. See https://code.launchpad.net/maas/+activereviews for the ones I need to land. [12:32] bigjools: which I don't think/know is the right thing to do [12:32] bigjools: for that matter, sudoers might be a better approach [12:33] roaksoax: ok, we need to make a code change for that [12:33] bigjools: but AFAIK sudoers only takes care of binaries, not files [12:33] unless we can make celeryd run as root? [12:33] too dangerous? [12:33] bigjools: I don't think the security team would want that [12:33] ok [12:33] bigjools: another option could be to add the maas user to the daemon group [12:34] bigjools: i.e. add maas user to the bind group [12:34] that should cover it right? [12:35] roaksoax: I don;t know how that affects things, can you explain? [12:36] bigjools: so if the maas user is in bind group, it would supposedly be able to access files that are for the bind group, right? [12:37] oh what about dhcpd.conf? [12:37] bigjools: we should be able to do the same [12:37] bigjools: ig what I'm saying is right, then it should work [12:37] we'll have to test it out [12:38] roaksoax: will it allow maas to restart the daemons? [12:38] well we don't need to restart DNS other than at installation time [12:38] since it uses an included config + rndc [12:39] bigjools: that it wont do. That would supposedly allow us to write in /etc/bind and /etc/dhcp/ [12:39] bigjools: for the daemons, we would need sudoers [12:39] ok [12:40] well it would still make my life easier to do the daemon group thing, it means a lot less code needs to change :) [12:40] indeed [12:40] so let me package tthis stuff first and then we can play with the group adding thing and sudoers [12:40] check_call(['sudo', 'service', 'isc-dhcp-server', 'restart']) [12:40] is what it does to restart [12:41] awesome [12:41] bigjools: yeah, so we won't need to change anything [12:41] ok, I need to go to bed RSN [12:41] roaksoax: you rock :) [12:48] allenap: there seem to be merge conflict markers in the diff in https://code.launchpad.net/~allenap/maas/pxe-intel-arch-config/+merge/121039. Is this intentional? [12:49] rbasak: Oh, I'll check that out. Thanks. [12:50] rbasak: They're real; I need to merge trunk. [12:51] Oh, so this is how merge requests are supposed to work? I've not seen it before. [12:51] allenap: can you please check that http://paste.ubuntu.com/1164368/ has all your patches to python-tx-tftp [12:51] Ah. "493 lines (+68/-315) 4 files modified (has conflicts)" [12:51] I get it now :) [12:51] allenap: there might be one missin [12:54] jtv: Any chance of a +1 for https://code.launchpad.net/~allenap/maas/pxe-intel-arch-config/+merge/121039 [12:54] Please :) [12:54] coming [12:54] roaksoax: I'll take a look. [12:55] * allenap can't remember how to use git [12:58] allenap: conflict. ( [12:59] Conflicts, even. :( [13:00] jtv: It's fixed and pushed, but Launchpad is being slow to catch up. [13:00] Yes, it's slow at the moment. [13:00] allenap: bzr-git? :) [13:01] bigjools: I did try that, and I just didn't get on with it. [13:01] allenap: done [13:01] yeah I think it's incomplete [13:01] jtv: Ta! [13:02] Not seeing why the API is redirecting the worker to the login page. If only piston didn't confuse Forbidden with Unauthorized... [13:03] authenticated url type? [13:04] lawnmower fiscal epidermis? [13:08] bigjools: no idea what your question meant, sorry. [13:08] is it unauth or auth api [13:08] Authenticated. [13:08] * jtv goes to kitchen to make some hot toddy [13:09] is the auth header getting added? [13:13] roaksoax: Full diff, I think: http://paste.ubuntu.com/1164399/ [13:13] allenap: ah, so there was things missing then [13:13] I have to go out for a bit, back in ~1h. [13:18] bigjools: it should be, since the request goes through the maas client… any tips on how to verify? [13:19] my brain shut down an hour ago [13:19] Eww toddy is vile without the honey [13:20] Oh seriously? My logs dir is empty? [13:21] bigjools: have you gusy hit an issue on PXE booting that causes an error with "Unable to locate configuration file" [13:21] bigjools: or is this due to the missing patch to python-tx-tftp [13:21] roaksoax: could be to do with allenap's latest changes? [13:21] otherwise did you run maas-import-pxe-files? [13:22] bigjools: i see this error since yesterday, so I'll wait for his changes to hit trunk [13:22] bigjools: yes I did run it [13:22] ok I've not done a full boot for a few days [13:28] * bigjools eod [13:28] nn [13:29] allenap: have you seen this? https://bugs.launchpad.net/maas/+bug/1041158 [13:29] Ubuntu bug 1041158 in MAAS "Worker not authenticating to API in upload_leases" [Critical,Triaged] [13:44] -D debian/maas-dhcp.config [13:44] -D debian/maas-dhcp.lintian-overrides [13:44] -D debian/maas-dhcp.postinst [13:44] -D debian/maas-dhcp.postrm [14:29] jtv: I saw that someone (you?) marked it critical, that's it. What's up? [14:32] allenap: it means that the worker can't send its leases back to the server. [14:34] jtv: Gah. I hope that's a simple fix :) I guess you're EOD nowish, and I'm away until Thursday after today. I'll try and look at it later this afternoon. [14:34] allenap: what's really fun is that as far as I can make out, the request isn't touching any code in piston's authentication.py. [14:35] Thanks. Technically I'm about EOD and a half. [14:36] jtv: It's POSTing to /nodegroups/master/ - no /api/... in front. [14:37] jtv: Indeed, you should be tucked up in bed with some cocoa. [14:37] Well I didn't have honey for the toddy, so I found some chocolate that's supposed to have honey in it. Is that good? [14:37] (It was needed. The concoction was horrible.) [14:38] jtv: Chocolate is almost always good for the mental state. [14:39] Mental yes. Physical… Haven't been out at all today, so the sitting flesh isn't really supporting me any more. [14:39] Anyway, let me see about that URL [14:39] Yup, that's it! Missing prefix. [14:42] Just got a notice that the worker hadn't had a refresh from the server yet, so it wasn't sending DHCP. [14:43] Maybe that's something to do with the restart logic. [14:47] allenap: yay! Just a POST now. Not a lot of detail in the logs, but... [15:02] allenap: do you know off the top of your head how the API test can get at the netloc it's running its API at? [15:02] At least, I _think_ it runs real http on a real port… I hope it does. [15:07] jtv: settings.DEFAULT_MAAS_URL... and get_maas_facing_server_host() returns the hostname part of that. [15:08] Of course, that doesn't work outside of Django. [15:08] Yeah but does that apply to tests? [15:08] That's why I'm doing this in the API test, not in the worker test. [15:09] jtv: self.client.get(reverse(...), ...) or absolute_reverse(). [15:09] Hmm… it uses http://localhost/ [15:10] Wonder why it 404s when I feed that to the worker [15:10] jtv: I think self.client cheats. [15:11] Yes, I would expect it to. [15:11] Yay Django. [15:11] But I also thought there was a real service waiting for real http requests during the tests. [15:47] allenap: if you have time… https://code.launchpad.net/~jtv/maas/bug-1041158/+merge/121207 [15:47] Ubuntu bug 121207 in gcc-3.4 (Ubuntu) "[Feisty] Dangling symlink in amd64 libg2c0-dev package" [Undecided,Won't fix] [15:49] jtv: Sure :) [15:49] Thanks [15:49] ubot5 has quite an imagination. [15:49] jtv: I am only a bot, please don't think I'm intelligent :) [15:50] Shut up. [15:54] allenap: oh dear... lots and lots of really strange-looking failures on a fresh branch. [15:55] Gah. [15:55] Mostly DNS stuff I think. [15:58] jtv: +1 for that mp. [15:58] Thanks. [15:59] And it looks like either start_up is now being called 3× where previously it was called once, or tests have been converted from FakeMethod to Mock and become accidentally stricter in the process. [16:00] jtv: Got time for a short review? https://code.launchpad.net/~allenap/maas/patch-mock-thing/+merge/121210 [16:00] Yes [16:00] Thanks. [16:01] Weird. [16:01] Done. [16:04] Thanks. [16:05] Well, trunk looks thoroughly broken & I don't see a clear single reason [16:06] Or does it? Maybe it's just my fresh branch that's broken. [16:07] Yup. It's my branch. Maybe I shouldn't have named it all-you-zombies. [16:07] Heh [16:08] jtv: Do you ever use `lsof +D` btw? It is remarkably useful for finding still running stuff that's causing weird failures. [16:08] Well, `lsof +D $directory`. [16:08] Oh, no, I don't use the whole program much actually. [16:09] And usually I just grep my way around the output of such tools. [16:10] jtv: I guess I'm interested to know how other people do this stuff. lsof is the best thing I've found for this, but fuser can do similar things. [16:11] There's so much of this out there that I prefer to remember just a few simple things. [16:19] lsof has always confused me. I use it as a last resort, usually combined with grep [16:20] Tools like “cut” can also help query and manipulate structured output uniformly. [16:38] roaksoax: I've just landed the last of my branches for this week. Any chance of a new package? [16:40] * allenap has to go now, have a good weekend everyone. [16:42] nn allenap! [16:42] allenap: there's a newer precise version [17:00] jtv: you know anything about the tftp server? [17:00] roaksoax: a bit [17:01] jtv: when trying to PXE boot, what happens if no PXE IP/MAC is found [17:02] That's not really a question of the tftp server — by convention, the pxe client loads a config named “default” in that situation. [17:02] jtv: wheres is that default ? where is it being handled? [17:03] jtv: i keep getting an "unable to locate configuration file" [17:03] It's dynamically generated by the tftp server. [17:03] But that's probably not the issue here. [17:03] The problem would seem to be in matching the path. [17:04] jtv: right, (i get this on pserv.log: 2012-08-24 13:03:32-0400 [TFTP (UDP)] Datagram received from ('192.168.123.101', 49162): [17:04] ) [17:04] That means your DHCP server isn't configured right. [17:05] jtv: my DHCP is right [17:05] jtv: i have been working with it [17:05] jtv: pre cobblerless MAAS [17:05] But everything was very different back then. [17:05] jtv: right, but DHCP only needs to tell where the bootp is for it to PXE boot [17:06] jtv: it is an external DHCP [17:06] jtv: this is on a virtual environment though [17:06] Is this i386? [17:07] jtv: its a VM [17:07] An i386 VM? [17:07] can't recall TBH [17:08] it is my local dev environemtn [17:08] So either i386 or amd64 then? [17:08] that's been working since orchestra [17:08] yep [17:08] OK [17:08] Do you see any sign in the TFTP logs of pxelinux.0 being requested? [17:09] jtv: where is that particular log stored? the only that has anything releated is pserv.log [17:09] Because from what I heard, there's a weirdness where that file's directory effectively becomes the root for the session. [17:09] That should be the one, yes [17:09] jtv: http://paste.ubuntu.com/1164794/ [17:10] That's taking a while to load. [17:10] May be beacuse I'm doing a big fsck. [17:11] Was there no mention of pxelinux.0 in that log? It's possible because plain files are served in a different way from config files. [17:13] jtv: the only mention is: [TFTP (UDP)] Datagram received from ('192.168.123.101', 1024): [17:13] That makes it harder to find the information we need. [17:13] Is there any helpful logging on the client side? [17:13] Oh wait,I see now [17:14] Sorry; it's late here and I'm not well. [17:14] Having a bit of trouble reading from the screen. [17:14] Well this tells us that the DHCP setup is definitely not right. [17:14] jtv: hehe no worries [17:14] jtv: the DHCP server is libvirt [17:15] And it's not set up to point clients to the right TFTP path. [17:15] The right server, yes. [17:15] jtv: "" [17:15] But not the right path. [17:15] Yes. [17:15] That's wrong. [17:15] jtv: what's the right path? [17:15] We use maas//generic/pxelinux.0 [17:16] jtv: right, but in a DHCP you can't tell what patch to use based on the arch right? [17:16] You can, but it's tricky. [17:16] jtv: ah, so why are we doing that then [17:16] It's what we do in a MAAS-generated config. [17:16] jtv: right, but when we are not using MAAS DHCP [17:16] jtv: then that should not happen [17:17] But as long as you've got a single-architecture setup, you can just use the path I gave you — with either i386 or amd64 [17:17] (The paths are changing, but I don't think they've changed yet) [17:17] jtv: right, but that defeats the purpose of PXE booting doesn it? [17:17] Not really. The purpose of PXE booting is to give us control of a machine across power cycles. [17:18] jtv: right, so in a mixed network, with external DHCP., we are in trouble [17:18] Now, [17:18] jtv: on the case of an upgrade, we are screwed [17:18] the i386 image will work just fine for amd64 machines. [17:18] An upgrade? [17:19] jtv: so, we have a maas server, we upgrade precise version to latests, we don't modify external DHCP [17:19] then we have a broken maas infrastrcuture [17:19] Yes, so it seems. [17:19] that makes administrator manually modify their DHCP servers [17:19] Yes, it's a manual step if MAAS doesn't manage DHCP. [17:20] jtv: rigt, which defeats the purpose of everything [17:20] jtv: i think there should be some defaults to continue to use the older path [17:20] and default to i386 [17:20] for these cases at least [17:20] I suspect you could work around that by installing pxelinux on the server. [17:21] right, but we won't have the systems available would we? [17:21] Hmm that fixes the pxelinux.0 problem, but not the config one. [17:21] exactly [17:21] You already have pxelinux-common installed, I suspect. [17:21] yep [17:21] IMHO, this is a regressio [17:21] Gavin was working on this, but he has eow'ed. [17:22] yeah, oh well [17:22] And for me it's technically saturday now, so I won't be around for long enough to fix this tonight. [17:22] no worries [17:22] i'll try to look into it if I find the time after i finish the stuff i need to get done [17:22] Would a workaround help? [17:22] Temporary one? [17:22] Are you guys talking about a regression for precise->precise-updates, or precise->quantal? [17:23] jtv: heh, now I get a permission denied to obtain the file :) [17:23] What did you change? [17:24] jtv: i changed the pxelinux path [17:24] rbasak: I think we missed the 12.04.1 window, so I think the most urgent issue would be the latter. But I'm not too sure. [17:24] roaksoax: what did you change it to? [17:24] Hmm, ok [17:25] jtv: [17:25] Then what's biting you now, I think, is the problem that our patch for the tftp server is not in the package. [17:25] jtv: probably. I have to review that by EOD too [17:26] jtv: alright then. Thanks for the help :) [17:26] Waitwaitwait [17:26] I don't think you answered my question: would a workaround help? Temporary one? [17:26] Or would it be a waste of time? [17:26] jtv: to older way pxelinux.0 was located? I guess it would [17:27] jtv: I think sabdfl wanted to test over the weekend, but we can point out to the change in the DHCP server [17:27] Well he said he was quite keen to, so hell yeah he wants to test over the weekend! [17:28] I just wonder if this is what Gavin was doing. Let me look. [17:28] jtv: ok :) btw.. if you can point me out to the branch where the permission thing was fixed for python-tx-tftp would be great too [17:29] Ouch. I don't know that. I only heard it talked about. [17:29] Don't even know if it's on launhpad or github. [17:32] jtv: hehe no worries, I'll look it up [17:32] * roaksoax off to lunch before he gets cranky [17:32] OK. [17:32] jtv: thanks for the help though [17:32] I have something in mind for the config files, making use of something Gavin landed tonight. [17:57] Oh dear. Where do we match “default” config at all? [18:00] I remember now. We generated it as part of the filesystem tree when installing a new image. [18:04] But not now, I think... [19:05] roaksoax: having serious system problems now, not sure why. But I have a branch that you might want to try. It may solve at least part of the problem. [19:06] jtv: got the access denied issue figured out.. python-tx-tftp was with incomplete patches apparently [19:07] Hmm [19:07] but it seems that i just found another issue :) [19:08] permissions wrongly set for commissioning directory [19:08] I may drop out of the conversation at any moment. [19:08] Commissioning directory? [19:08] You mean in /var/lib/tftpboot/...? [19:08] jtv: /var/lib/tftpboot/maas/.../commissioning [19:08] jtv: yeah [19:09] We fixed that a few weeks back. [19:09] Or maybe this week? [19:09] Not very long ago. [19:09] jtv: uhmmm upgrading didn't work right then [19:10] jtv: now it doesn't enlist [19:11] Did you get past the problem of not finding the default pxe config somehow? [19:11] jtv: yeah [19:11] just edited the path to maas/pxelinux.0 [19:11] jtv: and now : NameError: name 'preseed_data' is not defined at line 14 column 3 in file /usr/share/maas/preseeds/generic at line 87 column 3 in file /usr/share/maas/preseeds/preseed_master [19:23] roaksoax: back after some system trouble… how did you get around the pxe-config problem? [19:24] jtv: [19:24] And then the machines booted into enlistment? [19:24] jtv: they boot into the commissioning environment that does the enlistment [19:25] jtv: but there's an error on the preseed [19:25] NameError: name 'preseed_data' is not defined at line 14 column 3 in file /usr/share/maas/preseeds/generic at line 87 column 3 in file /usr/share/maas/preseeds/preseed_master [19:26] Scott may know. [19:26] smoser: ^^ [19:29] roaksoax: I'm crashing. I've got a review up for a compatibility change that would solve your previous problem, but I'm not in a state where I can judge whether it's a good idea. I'll have to call it a night! [19:29] oh fool. [19:29] foo [19:29] not fool [19:29] food? [19:29] where is this? [19:30] trunk ? [19:30] jtv: thanks a lot! it used to be under maas//etc/etc/pxelinux.0 now it is under maas/pxelinux.0 so should be easier [19:30] smoser: trunk [19:30] smoser: packaging up the latest trunk [19:31] roaksoax: the branch I have up here also supports the classic pxelinux.cfg/*: https://code.launchpad.net/~jtv/maas/bug-1041318/+merge/121266 [19:31] Ubuntu bug 121266 in openoffice.org (Ubuntu) "OpenOffice does not start: abnormal early exit ..." [Undecided,Invalid] [19:31] Shut up, bot. [19:32] jtv: cool I'll take a look [19:33] i dont know from the to po fmy hhead. [19:36] smoser: ok ;) [21:44] roaksoax: Do you have some background on bug 1041327? [21:44] Launchpad bug 1041327 in MAAS "Where's the “default” PXE config?" [Critical,Triaged] https://launchpad.net/bugs/1041327 [21:54] allenap im guessing is the fact that in order to use the tftp we have to tell the external dhcp server that pxelinux.0 is in maas/pxelinux.0 [21:55] roaksoax: jtv mentioned that you were having problems with that, in a merge proposal. Is that because the boot filename cannot be set on your DHCP server? [21:55] allenap so on upgrades for those who have external dhcp maas is "broken" [21:56] becauae if you upgrade maas magically tftp fails [21:56] roaksoax: Ah, is this only pertinent to upgrades? [21:56] yrs [21:56] Righty ho. In https://code.launchpad.net/~jtv/maas/bug-1041318/+merge/121266 I ponder if we shouldn't just get rid of the maas/ prefix completely. [21:56] Ubuntu bug 121266 in openoffice.org (Ubuntu) "OpenOffice does not start: abnormal early exit ..." [Undecided,Invalid] [21:57] ubot5: Your regex is wrong. [21:58] allenap leaving it there might be benefitial if we have Nother infrastructure in pla e [21:59] the only problem i see is that since pxelinux.0 that maas is using chanves administrators will see this as broken [21:59] becauze they would need yo update the dhcp server to tsll the new location [21:59] however this could so be a regression [21:59] because an upgrade and path location haz changed [22:00] so if old pxelinux.0 is left Nd dchp server not chsnged then cinfig file iz not found [22:01] on new jnsls we have to tell the admin tha oxelinux.9 for maas has changrd location but in uogrades we cant just simply change and expect admins will figure out wjat was wrong [22:01] so it should be backwards comoatible [22:03] allrnap unless we tell tctp that root is /var/lib/tftpboot/maas [22:03] dahhh that shiuld cover it [22:03] so no need for code chsnges rught? [22:03] only config [22:08] roaksoax: I'm way too tired to understand all that right now! I think it may be best if you email the list, as I'm away until Thursday next week too. [22:08] roaksoax: Have a good weekend, thanks for putting up with us upstreamers :) [22:08] allenap will do have a good week [22:08] hehe no worries [22:09] glad to had the chance to work with u guys