/srv/irclogs.ubuntu.com/2012/09/04/#maas.txt

rbasaksmoser: ping. Need some ephemeral image debugging help when you get in please. For some reason it's not configuring DNS right even though IP-Config lists correct info (can't resolve ports.ubuntu.com later). I can't reproduce by booting the machine into a normal installation. Is there a way to get a shell onto the system via the console and/or stop it shutting down?09:10
jtvDaviey: got a moment to talk about how we run dhcpd?10:25
Davieyjtv: Hmm, i can try.. kinda knee deep atm10:31
jtvThanks.  It's just that /etc/default/isc-dhcp-server seems to be the only place where we can set which interfaces it should serve on.10:31
Davieyrbasak: are you using an apt proxy?10:31
jtvAnd so my only option to set that _if I stay within MAAS code_ is to rewrite that file, which is all sorts of ugly.10:32
rbasakDaviey: no. I've got a bit further though. Looks like cloud-init is not writing a resolv.conf at all. Some conflict with resolvconf and the nature of my image perhaps.10:32
Davieyjtv: Well, that is the proper way... but you can supply your own upstart job with a config, if that makes more sense10:32
Davieyrbasak: resolvconf awesomeness !10:32
jtvDaviey: I'm not sure it'd be good to have two upstart jobs for the same daemon side by side.  Otherwise, tempting.10:34
jtvObviously I want to avoid just appending endless lines... is there a standard reusable way to rewrite a single line with some kind of I-wrote-this marker comment in front?10:35
rbasakaugeas is the standard answer. Not sure how well adopted that is in Ubuntu. And it's probably overkill10:37
jtvDon't know what that is, but the name certainly carries an air of overkill!10:37
rbasakIt's a generic mechanism to understand different file formats and change them programatically and losslessly10:37
Davieyrbasak: yeah, we never really quite got into the augeas vibe10:37
jtvSomething else then?10:38
jtvIt's a relatively simple problem, but obviously not unique.10:38
rbasakI would use a comment as a marker and combine that with sed10:38
rbasakI think that's the usually done thing10:38
Davieyjtv: Well... having /etc/default/isc-dhcp-server - ENABLED=False.. then writing a /etc/maas/isc-dhcp-server is another option, no?10:38
jtvNot _particularly_ nice to have the comment on the same line, but...10:38
jtvBut then we're back to writing that file.  :)10:39
rbasakUsually it's the immediately preceding line10:39
rbasakWhich notes that it's an automatic thing and not to mess with the line below10:39
* jtv has not mastered inter-line editing in sed10:39
Davieywell, ## Begin MAAS entries\n .. ### End MAAS entries ?10:39
jtvA comment like that is exactly what I had in mind though.10:39
jtvI thought sed got horribly involved when you had to edit across lines.10:40
rbasakIt's not too bad.10:40
Davieysmoser would fight for awk here :)10:40
rbasakIf you use Daviey's start and end markers, then:10:40
rbasakhmm, maybe too much to do in irc10:40
rbasakYou may have a point10:40
* jtv thought so :)10:40
jtvIt's not _very_ much code but given how evil corruption in /etc would be, I'd much prefer something that's already in widespread use.10:41
jtvFailing that, I can just write some python myself.10:42
rbasakIt's against debian policy for packages to change other packages conffiles10:42
rbasakisn't it?10:42
rbasakHow would that work for maas packaging then?10:42
rbasakI think the separate upstart job would be cleaner and more policy-compliant10:43
jtvBut if it requires editing /etc/default/isc-dhcp-server to disable the original upstart job, it doesn't make things much better.10:43
rbasakLuckily the dhcp server installs disabled10:44
jtvIdeally we'd have some kind of conf.d hook here.10:44
jtvOh, yes, of course it does!10:44
jtv*facepalm*10:44
jtvOh dear.  But that's in the config, innit?10:44
jtvAnd we write to the config.10:44
rbasakHave an entirely separate config. /etc/maas/dhcp10:44
jtvYeah, that's what springs to mind but it's a lot of weight to carry around.10:45
rbasakThe maas dhcp upstart job would fire up the daemon to use that, with separate pidfiles, lockfiles, leases files and everything10:45
rbasakIt is10:45
rbasakUnfortunately there's no accepted mechanism for packages to stack up automatically in the way that maas needs to sit higher in the stack and manuiplate dhcp10:46
rbasakSo right now that's the only clean way to do it in packaging I think10:46
rbasakBut who am I to say this anyway. I'm neither a Debian nor an Ubuntu anything10:46
jtvAnd yet you cite debian policy.10:47
jtvAt least it's giving me a much better view of the problem, so I thank you for that.10:47
rbasakYou're right about the conf.d hooks btw. That's the normal way of packages getting hooks into other packages10:48
rbasakI suppose /etc/default/isc-dhcp-server could have something like [ -d /etc/default/isc-dhcp-server.d ] && for f in $(run-parts --list /etc/default/isc-dhcp-server.d); do . $f; done10:50
rbasakThat could go in as an Ubuntu delta to the dhcp package, and then maas would just need to manipulate /etc/default/isc-dhcp-server.d/maas10:51
rbasakI think that would be clean and policy compliant, but it would be unusual10:51
Daviey^ that seems better actually10:54
jtvBut it'd have to be in Precise as well.10:56
jtvDaviey: any chance of getting anything like that into Precise?10:58
jtvAs well as Quantal?10:58
Davieyjtv: well quantal is easy.. precise is so messed up with this massive change, what is more pain. :)10:59
jtv*cough*10:59
jtvWho might be able to do such a thing?10:59
Davieyjtv: I honeslty can't see the massive change going into precise at the moment...11:02
Davieythat run-parts style thing, is /possible/11:02
jtvWait... which is the massive change then?  You mean writing our own config is the impossible one?11:02
Davieyno, the massive change.. as in, pulling everything back :)11:02
jtvI don't understand that expression.11:03
Davieyjtv: The plan to SRU trunk maas back to precise.. I have reservations abou it being possible.11:04
jtvOh!  That one.11:04
jtvSorry, I had no idea you were talking about that.11:04
Davieynot saying it can11:05
jtvWell, one thing at a time then: do you know how we might achieve a runparts extension on /etc/defaults/isc-dhcp-server?11:05
Davieynot saying it can't/won't be done.. i just have reservations11:05
Davieyjtv: Well, we won't achieve it until it's required.. does that make sense?11:06
Davieyie, we won't update the precise version.. on the off-chance it might be needed in a few months.11:06
DavieyDoes that make sense?11:06
jtvOf course.11:06
jtvThe only other solution that's come up is to write our own upstart job and our own dhcpd config.11:08
jtvIt's a lot of weight to carry around, basically just to work around this relatively standard piece of config machinery not being there.11:12
Davieyjtv: right11:22
jtvrvba: you wrote the dhcp config files, right?11:29
rvbajtv: no, Julian did iirc.11:30
jtvAh11:30
jtvOh, maybe the problem you had was with the dns config.11:31
rvbaCan I help with something?11:31
jtvThe problem where you had to write into the config file.11:31
rvbaYeah, that's with the DNS config file.11:31
jtvWell, there's only one place where we can specify what interfaces dhcpd should service.  And it's in /etc/default/, so we're not really supposed to change it.11:31
jtvAlternatives include:11:31
jtv * Doing it anyway.11:32
rvba(that's what we did with the DNS stuff)11:32
jtv * Getting a "runparts" into the packaged version of that file.11:32
jtv * Creating our own upstart job, and using our own config that isn't in the usual /etc/dhcpd or wherever it normally is.11:32
rvba(That might be a problem if you add Appamor into the mix)11:33
jtvOh dear yes, there's that too.11:33
rvbaLet's hangout briefly if that's ok  with you, the decisions we made for the DNS stuff are still fresh in my memory so maybe the two of us can figure out the best way to do this.11:34
rvbajtv: ^11:34
jtvSure, thanks.  I don't have much time, but I have some.11:34
jtvI'm creating a hangout.11:34
jtvrvba: inviting...11:35
jtvhttps://plus.google.com/hangouts/_/c5f8c770e7c02a1703b226632493d84bb508ad53?authuser=0&hl=en-GB11:35
jtvDaviey: looks like my only realistic option is to write the existing config file anyway.  Thanks for  sparring with me!11:52
* jtv is off11:52
Daviey\o/11:53
smoserrbasak, what happened above?13:33
rbasaksmoser: I got a bit further13:33
smoseri'd prefer the separate upstart job myself.13:33
rbasakIt seems that resolv.conf isn't being populated for some reason13:33
smoserand i dont really think its a lot of additional things to carry. its 1 file.13:33
rbasakOh13:33
smoseroh. yeah, that too. the above comments were wrt dhcp13:33
smoserbut rbasak yeah, i suspect you have clock issues and dhcp :)13:33
rbasakThe dhcp thing is between Daviey and jtv I guess13:34
smoseror the eth0 naming bug we agreed existed.13:34
rbasakI'm not sure about clock. The rest of DHCP seems to work fine. It's just resolv.conf that's not being populated13:34
rbasakCould it be a race between eth0 and eth1?13:34
smoserrbasak, where is your image?13:34
smoserwhat does /etc/resolv. conf looki like in the pristine image?13:35
rbasakIt's a symlink13:35
smoserbecause if you're using something older on 12.04, that was busted, and i'm not sure how it would react (although it seems to work fine on intel...)13:35
smoserrbasak, ok. then i think i know.13:35
smoserah. i know.13:35
rbasakI can look up the target if you need13:35
smoserhttps://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/103106513:35
ubot5Ubuntu bug 1031065 in cloud-init (Ubuntu) "cloud-init-nonet runs 'start networking' explicitly" [Medium,Confirmed]13:36
smoseri suspect it is that.13:36
smoserand you can verify by removing the 'start networking' line from cloud-init-nonet.13:36
smoserits a race condition.13:36
smoserinterestingly, i came upon that bug when I was incorrectly not specifying 'ro' on the kernel command line13:36
rbasakVerify by removing it from where? In the image?13:37
smoserso you might want to verify that you're doing that as it massages the race differently :)13:37
smoserin the image, yes.13:37
smoserso, you *should* boot with 'ro' on the kernel command line, and that *might* "fix" your problem here.13:37
rbasakWhat's the plan for the armhf image for precise in general?13:37
roaksoaxrbasak: can you give us a short summary of the dhcp issue?13:37
roaksoaxerr13:37
roaksoaxsorry13:37
roaksoaxrvba:13:37
roaksoax^^13:37
smoserrbasak, we'll get you one if you need one.13:38
rbasakI definitely need one13:38
smoseri made a daily of the images available, and my plan was to get those down.13:38
rbasakCritical for MAAS on ARM - in time for 12.1013:38
smoserbut due to many blocking bugs in quantal maas, i've not been able to sufficiently test the images to move them to 'release'.13:38
smoseri've verified they're good for precise usage.13:38
rbasakI believe MAAS trunk is essentially working now13:39
rbasakNot sure about the packaging13:39
rvbaroaksoax: the dhcp issue that Jeroen was having?13:39
roaksoaxrvba: yeah, did you guys reach concensus on how should it be fixed?13:44
rvbaroaksoax: we basically don't have a choice here: with support for a proper conf.d directory, the only solution is to write to the config file directly. Same as what we've done for the DNS config.13:44
rvbas/with support/without support/13:45
rvbaroaksoax: can you think of another way to do this?13:46
roaksoaxrvba: so I asked the securoity team about this the other day13:46
roaksoaxrvba: and this is the response: "Perhaps it would be easier for MAAS to have a helper that spawns the"13:46
roaksoaxdhcp and bind daemons itself? This would allow specifying different13:46
roaksoaxconfiguration files, in a location that is accessible by the maas13:46
roaksoaxrvba: which I believe is probably the best way to mess up with files, without messing up with files (if you know what i mean)13:47
rvbaWe ruled that out because Apparmor restrict where, say, the named daemon can read its config file quite seriously.13:47
rvbarestricts*13:47
roaksoaxrvba: right, but we can always ship apparmor profiles13:47
rvbaRight, that would solve the Apparmor problem.13:47
rvbaBut this means adding conf.d support for the apparmor profiles.13:48
roaksoaxrvba: not really no13:48
rvbaWhich, IIRC, isn't available yet.13:48
rvbaah?13:48
roaksoaxrvba: it is the same as what happened with cobbler/maas-provisiion13:48
roaksoaxrvba: we just ship an apparmor profile which is also the same as the original one13:49
roaksoaxrvba: but we add the differences to it13:49
roaksoaxrvba: so we can just copy the dhcp apparmor profile and those to would lie their13:49
roaksoaxie. /etc/apparmor.d/usr.sbin.dhcp and /etc/apparmor/usr.sbin.dhcpd-maas13:50
roaksoaxor similar13:50
roaksoaxand the -maas profile includes the files we are touching13:50
rvbaI can't say I know Apparmor very well but I suppose this means that the profile are "cumulative"?  I mean that we would have to provide a /etc/apparmor/usr.sbin.named-maas which would sort of" extend" the default /etc/apparmor/usr.sbin.named13:52
rvbaIs that right?13:52
roaksoaxrvba: I don't know actually if it would work that way13:55
roaksoaxrvba: we shoud ask jdstrand13:55
roaksoaxrvba: cause otherwise we ship the same exact profile, plus the location of the maas configu files13:55
roaksoaxrvba: we will simply have 2 different profiles installed13:55
rbasaksmoser: definitely booting with ro. I'll try dropping the "start networking".13:58
rvbaroaksoax: if using the apparmor profile tricks works, this would also mean duplicating all the startup scripts to control our custom dns/dhcp services.  Is this really something we want to do?14:02
roaksoaxrvba: what do you mean by duplicating all the startup scripts? Isn't it just the upstart job?14:02
rvbaroaksoax: yeah, the upstart jobs.14:03
roaksoaxrvba: yeah, so only 2 upstarts jobs14:03
roaksoax1 for DHCP and 1 for DNS14:03
roaksoaxbut for now we can just try this out with DHCP14:04
smoserrvba, i +1 roaksoax's solutoin.  this is the right way especially for dhcp. we can probably find some way to get around bind, though.14:10
smoserusing groups i think.14:10
smoserbut that might be more trouble than its worth also14:10
allenaproaksoax: Is python-txtftp published to a PPA somewhere?15:45
allenapOr, where's it coming from for precise?15:46
roaksoaxallenap: for precise its on maas-trunk15:46
roaksoaxallenap: are you planing any fixes?15:46
allenaproaksoax: Where's the maas-trunk PPA?15:46
roaksoaxallenap: err there';s no precise package fro python-txtftp15:48
roaksoaxwe dont want it15:48
roaksoaxbecause we wont be able to have it in the archives15:48
roaksoaxallenap: python-tftftp is installed as part of maas15:49
allenaproaksoax: debian/control says that python-django-maas depends on python-txtftp. How will that work on precise then?15:51
roaksoaxallenap: there's a different precise branch15:51
allenapAh, right.15:51
roaksoaxallenap: the precise packaging branch is stacked on top of the quantal one15:51
rbasaksmoser: ping15:54
rbasaksmoser: I've tried the workaround you suggested, but now it cloud-init seems to failcompletely. It doesn't even attempt apt-get update15:55
rbasakcloud-init-nonet gave up waiting for a network device.15:55
rbasakThen it lists eth0 as configured correctly, and eth1 as unconfigured this time15:55
rbasakWhat is cloud-init-nonet doing here, anyway?15:56
smoserrbasak, cloud-init-nonet ensuring that network comes up before cloud-init.conf runs15:57
* rbasak doesn't understand why this isn't called cloud-init-net15:57
smoserrbasak, i can help if you can point me at something i can look at.15:58
smoserbut i'm almost certain you're seeing the bug that i pointed you at.15:58
rbasaksmoser: I've tried the workaround you suggested - I removed "start networking"15:58
smosercombined with the other bug we discussed (which does not have a launchpad bug for) of eth0 not being the device that was pxe booted from.15:58
rbasakBehaviour has changed now15:58
rbasakIt definitely has booted off eth015:58
rbasaksmoser: is it significant that it lists eth1 _before_ eth0?16:00
smoserrbasak, well are hte mac's right?16:01
smoserdid it boot off of eth0 ?16:01
rbasakYes16:01
smoserthen its not significant16:01
rbasaksmoser: am I right in thinking that this bug isn't arch specific, and not specific to having two interfaces either then?16:02
=== matsubara is now known as matsubara-lunch
smoseri dont think its specific to 2 nics.16:03
smoseri dont think its arch specific16:03
smoserif it is arch specific, it is not really because of arch, but rather because of some config somewhere that makes an assumption16:03
smoseror because of race condition16:04
rbasakI think this is release critical for maas16:04
smoserthat just happens because arm has different bottlenecks.16:04
smoserwell, yes, its clearly critical16:04
rbasakOK, where do we need to go from here?16:04
smoserrbasak, other things stoped me from poking further at bug 1031065 after i found that the bug we were working around by adding 'start networking' was not fixed otherise (ie, we still needed 'start networking' to correctly boot under lxc).16:06
ubot5Launchpad bug 1031065 in cloud-init (Ubuntu) "cloud-init-nonet runs 'start networking' explicitly" [Medium,Confirmed] https://launchpad.net/bugs/103106516:06
smoseri ran out of time the day i was poking at it nad haven't been bakc.16:06
smoserit is possible that if we debug that firuter, the root of the problem would also be the root of your issue.16:06
smoserits also possible they're unreleated.16:06
rbasakI understand that you have too much to do in too little time. I didn't mean to imply that you were slacking! I'm just not sure how to proceed right now and this issue completely blocks me16:07
smoserrbasak, i'm not complaining.16:08
smoserrbasak, so there are 3 issues here.16:08
smosera.) 1031065 documents the fact that cloud-init should not have 'start networking' as it does. but removal of that breaks booting under lxc. we need to fix that.16:09
smoserb.) your issue, which seems unrelated to me, but may be the core cause of a.16:09
smoserc.) nothing in the ephemeral images is going to force 'eth0' to be "pxe booted interface". however we assume that.16:10
smoseri dont have a bug opened for 'c', but i'd call that critical too16:10
rbasakAs an aside, there's no support for ipappend in U-Boot right now16:10
smoserrbasak, right. so i'm not sure how we'd solve that for arm, but the solutoin is possible for intel16:11
rbasakI want to propose that rather than getting that through, we instead have the MAAS dynamic TFTP server just supply the MAC address of the node it is responding for in the kernel command line, if that will work16:11
smoserrbasak, the tftp server is a IP application, no?16:11
smoserwould it necessarily (without arp hackery) know the mac of the client?16:12
smoserand if it could figure that out, i suspect it would still break any case where the client was not on the same network16:12
rbasakI think arp hackery may be needed to make that work, but I think that would be cleaner than ipappend16:12
rbasakI'd like to assume that the tftp server is on the same network if I am permitted to do that16:12
rbasakIt seems unclean to me for the node to boot and re-dhcp and then assume that the pxelinux supplied IP is the same as the one on the correct interface that it dhcp'd16:13
rbasakOh16:13
rbasakThe TFTP currently does know the mac of the client16:14
rbasakit's in the pxelinux.cfg/01-<mac>16:14
rbasakthat it tried to fetch16:14
rbasakOnly catch is that if I have it fall back to default for arch detection then that will break16:14
rbasak(without keeping some state which is horriblew)16:14
smosertftp does work generically over ip16:16
smoserso mac cannot actually be assume di dont think . unless its part of the tftp protocol16:16
smoser(ie, inside the packet)16:16
rbasakOh wait - you're using ipappend 2?16:16
rbasakfor the mac directly?16:16
smoserrbasak, i believe we use http://www.syslinux.org/wiki/index.php/SYSLINUX#IPAPPEND_flag_val_.5BPXELINUX_only.5D16:17
smoserat least previously cobbler used that, and the installer knows how to handle that.16:17
rbasakOK, just checked. It's ipappend 2 which adds bootif=<mac>16:18
smoserso yeah, 216:19
rbasakSo a workaround for U-Boot would be to supply bootif=<mac> from maas if it knows it16:19
smoserrbasak, yes. tha twuld work.16:19
smoserexcept where it is not known.16:19
rbasakyeah16:19
smoserbut, as you say, that might not be a requirement.16:19
rbasaklike enlistment, without storing state from the previous miss :-/16:19
smoserand, i'm not certain that its *not* in a tftp request16:19
smoseralthough my argument about it being IP breaks that too16:20
rbasakJust checked. TFTP doesn't include it16:20
rbasakbut it is supplied by pxelinux in a previous pxelinux.cfg/01-<mac> (what will be a) miss16:21
rbasakwould you mind if we define a missing bootif to mean eth0?16:22
smoserrbasak, ok. i just see what you were saying about 01-MAC now.16:24
smoserrbasak, well w can just make the fall through case "do nothing"16:24
smoserbut "eth0" is completely arbitrary. i think we're to the point in the kernel now that upgrades probably consistently order network adapter names on the same bus consistently16:25
smoserbut hard coding eth0 basically implies/enforces wiring in a specific way. which sucks. but we dont have a lot of other options.16:26
smoserrbasak, how does pxe work?16:26
rbasakthat's a bit of an open question!16:26
smoserit dhcp's , uses that IP to then do a tftp i guess.16:27
rbasakYes16:27
rbasakI think the NIC it uses is hardware-defined16:27
rbasakThe first one on the case16:27
rbasak(probably)16:27
rbasakI've never seen/heard of a real server trying to PXE off a second nic, but I don't usually PXE them so I may be wrong there16:28
smoserwhat is our tftp server?16:28
rbasakIt's one that's now built into maas16:28
rbasak(some twisted thing)16:28
smosercan you g+ really quick?16:28
rbasakSure16:28
=== matsubara-lunch is now known as matsubara
roaksoaxallenap: still around?18:48
=== bjf is now known as bjf[afk]
=== bjf[afk] is now known as bjf

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!