/srv/irclogs.ubuntu.com/2014/03/24/#maas.txt

bigjoolsjtv: so, adding default-driver into the maas.tgt didn't h04:02
bigjoolselp04:02
jtvCan I have a look at the config you have now?04:02
jtvIf you were running on a machine that has also run the old ephemerals script, before you added the --conf option to tgt-admin you would always have gotten a default-driver setting anyway.04:03
bigjoolsjtv: AHA04:04
bigjoolswe add that and remove the driver option in each block04:04
bigjoolsand it worked04:04
jtvYeah, it's the driver option.04:04
bigjoolsnow I get a different error, of course....04:04
jtv?04:04
bigjoolsMultiple 'readonly' definitions in 'backing-store' not allowed!04:04
jtv...04:04
bigjoolsit seems like it hates the multiple targets in the same file04:05
jtvCan I see the file?04:05
jtvGuh.  Is that root-image file seriously over a GB in size?04:06
bigjoolshttp://paste.ubuntu.com/7144556/04:06
bigjoolsit is04:06
jtvThanks.04:06
jtvI hope that's very sparse...04:06
bigjoolstgt-admin is basically as bug-ridden per-LOC as you'd expect for Perl04:06
jtvLooks like we have two identical entries in that file...  If I were tgt I wouldn't like that either.04:06
bigjoolsjtv: interesting - so the list_boot_images() had two similar entries.  Oh.... two purposes!04:07
jtvYup.04:07
* bigjools fixorates04:07
bigjoolsjtv: did you review that branch BTW?04:08
bigjoolsah you did04:08
jtvYes.04:08
bigjoolsok made the extra changes04:12
bigjoolsforgot that extra close after thinking to myself that I needed to fix it, too ...04:12
jtvWe would have caught it at some point.04:13
bigjoolsjtv: bug #3453453534324, if anything fails after the maas.meta is written, the script will never try to write the tgt files on the next run04:20
ubot5Error: Launchpad bug 3453453534324 could not be found04:20
bigjoolsit exits early as it thinks it's in sync04:20
jtvHow about we try to generate all the file contents first, and then call atomic_write to do the actual writing?04:21
jtvNot so much to get complete atomicity, as to put as much of the stuff that can fail before the part where we write files.04:22
jtvIt'll break more, but when it breaks, it'll be more likely to break in a consistent nothing-got-done state.04:22
bigjoolsjtv: I am victorious04:22
jtvPleased to meet you, Victorinox.04:23
bigjoolswe could have another state file04:23
bigjoolsbut yes atomic_write for state + data please04:23
bigjools"sudo tgt-admin -s" now shows me stuff.04:24
jtvGood!04:24
bigjoolsneed to do a proper fix for list_boot_images04:24
jtvWill it boot?04:24
bigjoolsI just hacked it04:24
bigjoolsdunno I just did all this on canonistack where it's quick to delete huges files and redownload04:24
jtvTo be clear: you don't need a fix for list_boot_images itself, just the call site, right?04:25
jtvOr is the boot purpose completely gone?04:25
bigjoolsno, a fix is needed in extract_image_params04:25
bigjoolsit is completely gone04:25
bigjoolsdo you fancy doing that while I test a real boot?04:26
bigjoolsone thing that's annoying now is that it copies a 1.4G file04:26
jtvLet me just finish up what I'm doing here...04:26
bigjoolsso there's two copies of it after download04:26
jtvWhoops.04:26
bigjoolsyeah04:27
jtvAnd will the copy support holes?04:27
bigjoolsthat's the slowest part of this on canonistack!04:27
bigjoolsno idea04:27
bigjoolssparse file you mean?04:27
jtvWhere does the copying happen?04:27
jtvYes.04:27
bigjoolsnot sure, looking04:27
bigjoolsjtv: it uses simplestreams.objectstore04:31
bigjoolsI vaguely remember something from scott about it supporting sparse files04:31
jtvIf Scott did it, I'm sure it'll support sparse files as needed.04:31
bigjoolsI don;t understand why we need to keep 2 copies of such huge files04:31
jtvBecause IIRC the boot disk image does have holes.04:31
bigjoolsit's still going to be 400Mb04:32
jtvI'm sure Scott would have used a link if he could...04:32
bigjoolsjtv: do we still need that policy hack?04:33
jtvNo.04:33
jtvShould be supported directly by the code now.04:33
bigjoolscool04:34
bigjoolsta04:34
bigjoolsjtv: how are we supposed to put an arch list in the filter in pserv.yaml?04:44
bigjoolsI tried a list and it says it needs to be a single value04:44
jtvIt does.04:45
bigjoolsI am guessing it wants a filter04:45
bigjoolswhich leaves me itching04:45
jtvIf you need multiple arches but not '*', define multiple sources.04:45
bigjools!04:45
bigjoolssrsly?04:45
jtvsrsly.  But easy enough to change.04:46
jtvI'm hammering a little bit of abstraction into that code as we speak.04:46
* bigjools adds a card04:46
jtvI'm supposed to be adding logging, but in order to log what's going on, we need to know what's going on.04:46
bigjoolsyes :)04:47
bigjoolsjtv: also the error reporting from a dodgy pserv.yaml is ugly as hell with a stacktrace04:47
* bigjools files another card04:47
bigjoolsjtv: do you know if we fixed the bug where we still need i386 when booting amd64?04:48
jtvDon't know04:50
bigjoolschrist it's slow to download05:05
bigjools82M in 15 minutes05:05
bigjoolsjtv: ok done, attempting boot05:57
bigjoolsjtv: 2014-03-24 15:57:06+1000 [TFTP (UDP)] Datagram received from ('10.0.0.104', 49155): <RRQDatagram(filename=amd64/generic/trusty/no-such-image/boot-kernel, mode=octet, options={'tsize': '0', 'blksize': '1408'})>05:59
bigjoolsdarn05:59
jtvThat means pxeconfig didn't find a matching image.05:59
bigjoolsyup06:00
bigjoolsjtv: AND it thought it was enlisting, and it was an installation boot06:01
bigjoolsthis is using FPI06:01
jtvEnlistment vs. installation is decided by... the database, right?06:01
bigjoolslet's see what  bootimages are there06:01
jtvUnknown node gets enlistment image?06:02
bigjoolsif "node" is set, yes06:02
bigjoolsbasically if there's a recognised MAC in the request06:02
jtv(For some reason my twistd stopped using up gobs of memory)06:02
bigjoolsI'll watch the console this time, one sec06:03
bigjoolsit sends the mac06:07
bigjoolscontents of BootImage look ok06:12
bigjoolsjtv: so basically this code fails to find an image:06:15
bigjools    latest_image = BootImage.objects.get_latest_image(06:15
bigjools        nodegroup, arch, subarch, series, purpose)06:15
jtvMismatch between purpose names..?06:17
bigjoolstrying to use the harness to see what it would be using06:18
jtvWhat I like to do is "open('/tmp/foo.log','a').write("Got purpose: %s\n" % purpose)"06:19
bigjoolsjtv: aaaaa06:19
bigjoolsno "xinstall"06:19
jtvxinstall!?06:19
bigjoolsyeah06:19
bigjoolssee get_boot_purpose()06:19
jtvOh, this is while installing?06:19
bigjoolsyes06:19
jtvAnd do the boot purposes in the database still say xinstall?06:19
jtvTry "sudo maas-region-admin dbshell --installed"06:20
bigjoolsno06:20
bigjoolsthey just have commissioning and install06:20
jtvAh.  Well there we go.06:20
bigjoolsI downloaded the trusty amd64 images06:20
bigjoolsand it set those up06:20
jtvWill the classic installer work?06:21
bigjoolslet's see06:21
bigjoolsjtv: working06:22
bigjoolsok so06:23
bigjoolsnot sure what to do here.  are we getting rid of "purpose" or not ...06:23
bigjoolsoh ha spoke to soon06:23
bigjoolsd-i stopped with debconf message saying "no kernel modules found"06:23
jtvMissing initrd..?06:26
jtvWell, no, it wouldn't be missing I guess or you'd fail during pivot().06:26
jtvOr so I seem to remember.06:26
bigjoolsmismatching kernel version vs modules version06:26
bigjoolsI suspect crap data coming from simplestreams06:27
jtvThat would be pretty awful.06:27
jtvBut what if it's a filtering problem somewhere?06:27
bigjoolsin what way?06:27
jtvMaybe there's an arbitrary, unstable choice between similar images.06:27
bigjoolsthere's only one image06:28
bigjoolsjtv: ok so let's go back to xinstall06:29
bigjoolsextract_image_params() only reports commissioning and install, and we want to get rid of those.  Do we now need to differentiate boot purpose here like that in get_boot_purpose?06:30
bigjoolsit looks like we still depend on that purpose to select a different preseed for the FPI06:31
jtvThere's an irony there.06:33
jtvI suppose you could just add xinstall to the list of purposes in extract_image_params..?06:36
jtvBy the way, it does look as if the files that the import script puts in the snapshots are links into the cache, as you'd expect.06:36
bigjoolsjtv: I can but we also depend on that list only having a single item for the report_boot_images call to work06:37
bigjoolssorry list_boot_images06:37
bigjoolswhich sets up the tgt targets06:37
jtvShouldn't that code just strip off the purpose and run the rest of the images' data through a set to eliminate duplicates?06:38
jtvsorted( { (img['arch'], img['subarch'], img['release'], img['label']) for img in list_boot_images(...)})06:39
jtvOr more legibly:06:40
jtvsorted({06:40
jtv    (img['arch'], img['subarch'], img['release'], img['label'])06:40
jtv    for img in list_boot_images(...)06:40
jtv    })06:40
bigjoolsthat's one way to fix it, not sure if it's the best06:40
jtvWell it's only this use of the boot images that needs them to be unique despite ignoring the purpose, right?06:41
bigjoolswe don't really need to discern the difference when selecting a boot image06:41
bigjoolsthey are all the same now06:41
jtvThen we should drop BootImage.purpose.06:41
bigjoolsindeed06:41
jtvJust... not now  :)06:41
bigjoolswell I'll take it out of the code and see how it goes06:42
bigjoolsI put "xinstall" in the list for now06:42
bigjoolslooking good so far06:43
bigjoolsboom06:47
bigjoolscloud-init crashed ot because:06:48
bigjools--2014-03-24 06:44:26--  http://10.0.0.9/MAAS/static/images/amd64/generic/trusty/xinstall/root.tar.gz06:48
bigjoolsConnecting to 10.0.0.9:80... connected.06:48
bigjoolsHTTP request sent, awaiting response... 404 Not Found06:48
bigjools2014-03-24 06:44:26 ERROR 404: Not Found.06:48
bigjoolssomewhere else that needs purpose eliminating06:48
jtvBy the way, there's some code in the new import script that checks if the JSON in the maas.meta file already matches the latest data, but it doesn't sort dict keys so that might be a little arbitrary.06:54
jtvI'm dumping the json with sorted keys instead.06:54
bigjoolsok06:55
bigjoolsjtv: ARGH.  The curtin installer expects the root.tar.gz to live under tftp, as served by apache06:56
jtvHuh?06:58
jtvYou mean that it expects to be able to access the tftp tree through http?06:59
bigjoolsthere's apache config to mirror it at /MAAS/static/images/07:01
bigjoolswe need to fix the packaging for this.07:02
bigjoolsdammit07:02
jtvNot a good time.07:04
bigjoolsjtv: also, another problem07:09
bigjoolsget_curtin_userdata() needs to know the label in the path now07:09
bigjoolsnot sure how it's going to do that07:09
jtvWe have the label in the path now, don't we?07:11
bigjoolsyes07:14
rvbabigjools: care to have a look at this tiny fix? https://code.launchpad.net/~rvb/maas-test/fix-new-option/+merge/21235607:19
bigjoolsrvba: done07:20
rvbaTa07:20
bigjoolsI am this -><- close to FPI installer working07:22
bigjoolsFPI success! \o/07:26
jtv\o/07:26
rvba\o/07:26
bigjoolsit was a *hack* though07:26
bigjoolsstill have a problem to fix as I describe in the email07:26
* bigjools hands baton to rvba07:26
rvbabigjools: question about the first item in your email:07:27
bigjoolsyarp07:27
rvbaYou said we could do away with the purpose in boot images.07:27
rvbaI don't think we want that.07:27
bigjoolsok, go on07:27
rvbaRight now, when we import stuff, for each "selection" (arch/subarch/series[/soon:label]) we import both the ephemeral and the install image.07:28
bigjoolsyes07:28
rvbaBut in the future, we thought it would be nice to let the use select what image he wants to import.07:28
rvbaThe reason being that the ephemerals are huge.07:28
rvbaAnd the install images are not.07:29
bigjoolsyes07:29
bigjoolsas things stand, purpose is useless though07:29
rvbaSo you might want to download the ephemerals for just, say, trusty/amd64/generic.07:29
rvbaWhen this is the case the purpose will be useful.07:29
bigjoolsok07:30
bigjoolsI don't disagree07:30
bigjoolsgo with the first option to fix then07:30
* bigjools heads to dinner. back for call07:30
=== ging_ is now known as ging
rvbajtv: one question about https://code.launchpad.net/~jtv/maas/write-tgt-data-at-once/+merge/21235807:54
rvbajtv: Unless I'm mistaken, you didn't solve the problem where the same entry gets written multiple times in the tgt config file… right?07:55
rvbajtv: which is fine btw, I just want to be sure.07:55
jtvNo, that's a separate problem which Julian worked on.07:55
rvbaOkay.07:55
jtvMy take was: just turn the data into tuples, without the purpose, and put them in a set to eliminate duplicates.07:56
rvbaThat's the problem I started working on.07:56
jtvAh.07:56
jtvI thought Julian said he was doing that.07:56
rvbaI didn't catch that.07:56
rvbaNo, we talked about it just now (see above).07:57
jtvAh.07:57
rvbaI'll take care of it.07:57
jtvOK07:57
rvbaOnce I'm done with your review.07:57
jtvThanks.07:57
jtvThere will be conflicts in the branch after that, but nothing serious.07:59
rvbajtv: branch approved!08:03
jtvThanks.08:03
=== CyberJacob|Away is now known as CyberJacob
rvbajtv: would you reciprocate? https://code.launchpad.net/~rvb/maas/fix-list-boot-imgs/+merge/21236308:23
jtvHonour demands it.08:24
rvbaheh08:24
jtvrvba: in trying to figure out what the code does, I did do a bit of refactoring — which will also facilitate my ongoing work.08:26
rvbajtv: I see that.08:26
jtvCan't log what's going on without knowing what the code means.  :)08:26
rvbaMakes sense.08:26
jtvI'm turning the "arch" config into an "arches" — which will benefit from the new helpers.08:27
rvbaHang on.08:28
rvbaNot sure this is appropriate.08:29
rvbaBecause the subarches list really only makes sense for a particular architecture.08:29
rvbajtv: if we turn 'arch' into a list we will end up with the same mess that we have now where the import script imports a huge product of things.08:30
jtvTrue — but what's the harm in a filter that allows 'generic' and 'highbank' for armhf and i386?08:30
rvbaThat's precisely why we have a 'selections' list.08:30
rvbajtv: In this case it would be fine but in the general case, it won't make sense.08:30
rvbaAnd can lead to import more things than you thing… which — I think — I bad.08:31
rvbas/thing/think/08:31
jtvI agree that it can be used for evil, but is it our job to prevent that?  The filters within one selections stanza form a Cartesian product, which I think is fine for the usual "i386/generic + amd64/generic"—style situation.08:32
jtvWhen the Cartesian product gets out of hand, split up the selections.08:32
jtvWouldn't a comment suffice?08:33
rvbaHum, maybe…08:33
rvbaI think that by renaming that field to 'arches' we encourage a potentially dangerous practice.08:34
jtvWe make it possible.  Potentially dangerous is not the same as bad.08:34
rvbaBut maybe a well-written comment will suffice.08:34
jtvI'll document it carefully.08:35
rvbaOkay.08:35
jtvrvba: whoops, there's no set.append().  :-)09:00
rvbajtv: whoops… fixing…09:01
jtvDid you say my logging branch was dangerous and needed testing before landing?  :-P09:01
rvbayeah :)09:02
rvbajtv: I just pushed a fix.  Sorry about that.09:03
jtvNo worries.  It'll just make me feel slightly less guilty next time I break something.  :)09:04
rvbaheh09:04
jtvI'm getting breakage on the new import script, looks related to the simplestreams data:10:46
jtv    arch, subarches = item['arch'], item['subarches']10:46
jtvKeyError: u'subarches'10:46
jtvTrying with the -v2 data.10:52
bigjoolsI unsubbed maas-maintainers from https://code.launchpad.net/~maas-maintainers/maas/new-import-script-integration.  Please add yourselves manually if you need.11:18
jtvMy last import run went suspiciously smoothly.11:53
jtvNot much luck booting with the new images yet...  unhandled error during Deferred.  :(12:23
jtvrvba: https://code.launchpad.net/~jtv/maas/multi-arch-imports/+merge/21239212:24
jtvThat's the one we discussed.  Hope you find the comments clear enough.12:25
rvbak, I'll have a look in a sec.12:25
rvbajtv: here is a review for you if you have time: https://code.launchpad.net/~rvb/maas/fix-curtin-url/+merge/21240212:33
jtvLooking.12:33
=== cmagina-away is now known as cmagina
=== cmagina is now known as cmagina-away
=== cmagina-away is now known as cmagina
=== roaksoax_ is now known as roaksoax
=== cmagina is now known as cmagina-away
=== zchander is now known as _zchander_
=== cmagina-away is now known as cmagina
=== tooth_ is now known as tooth
=== teknico__ is now known as teknico
=== cmagina is now known as cmagina-away
=== cmagina-away is now known as cmagina
allenapblake_r: Hello. How’s the UEFI refactor branch doing? I’m eager to review and land it, because my next task is to port it to the new boot resources importer.17:57
blake_rallenap: just updated it to the lastest trunk, cut the diff in half17:58
blake_rallenap: I am porting the uefi script port to the new simple streams17:58
allenapblake_r: Hehe, that’s grand. I’ll review the first bit if you want.17:59
blake_rallenap: yeah go ahead and start reviwing17:59
blake_rallenap: I will get the rest in here soon17:59
allenapblake_r: Create a merge proposal for it and I’ll do it.18:00
blake_rallenap: there is one, let me get the link18:00
blake_rallenap: its in wip, want it set to review?18:00
allenapblake_r: Yeah.18:00
blake_rallenap: done18:00
allenapTip top.18:00
=== zchander_ is now known as zchander
manjiri_Hello! Is there a way to specify cloud-init user-data for maas?19:13
rbasakmanjiri_: can you be more specific? Do you want this for enlistment, commissioning or deployment?19:26
rbasak(and if for deployment, then during install stage, or for final boot?)19:27
manjiri_rbasak: Deployment. I want to wget a file at the end.19:28
manjiri_rbasak: Install stage, I think. Can you explain what is "final boot" ?19:29
rbasakmanjiri_: what I mean by "final boot" is the boot that your MAAS client (eg. juju) will get.19:30
manjiri_rbasak: My goal is make this "wget" part of "staging" rather than during "juju deploy". But I can settle for it to be during "juju add-machine".19:33
=== roadmr_nothere_f is now known as roadmr
rbasakmanjiri_: juju controls cloud-init userdata to actually boot juju machines. I'm not aware of any mechanism to control that.19:35
rbasakmanjiri_: what's the purpose of your wget script?19:36
rbasakmanjiri_: if there's a one-off action you want on all MAAS-deployed juju nodes, you could do that in the installer.19:36
rbasakmanjiri_: either by preseeding d-i (if using d-i), or a suitable curtin hook (for the fast path installer; I know less about this)19:36
manjiri_rbasak: Yes that is it. How do I "do that in the installer". ?19:36
rbasakmanjiri_: if using curtin then I guess you could use cloud-init userdata, but I think that would apply before the installer has finished19:37
manjiri_rbasak: Got it. This is what I had surmized but I wanted to confirm. Thanks!19:38
=== roadmr is now known as roadmr_afk
=== roadmr_afk is now known as roadmr
manjiri_rbasak: Any pointers for more info on "curtin hooks" ?19:39
blake_rallenap: just pushed the install_bootloader from the UEFIBootMethod, should all be their now19:43
manjiri_rbasak: Any pointers on making the URL arg for wget a config option?19:43
rbasakmanjiri_: http://bazaar.launchpad.net/~curtin-dev/curtin/trunk/view/head:/doc/topics/overview.rst#L11220:00
rbasakmanjiri_: I'm not sure how curtin's config is generated. Probably via templates in /etc/maas or something.20:00
AskUbuntumaas user interface does not talk to or sets up dns/dhcp | http://askubuntu.com/q/43870520:09
manjiri_rbasak: Thanks! I think a late or final command should do the trick. There are no examples to parametrize the URL through GUI/CLI. For now, hardcoding seems to be the only option.20:12
rbasakmanjiri_: the templates let you execute arbitrary Python and access the data model (IIRC).20:19
rbasakmanjiri_: But, I don't know of any means to put arbitrary parameters into the data model that you can then use.20:19
rbasakmanjiri_: you can apply arbitrary tags which you could probably get to from the template, if that helps.20:20
rbasak(and of course anything you do in this area might break in a future release)20:20
manjiri_rbasak: Good advice. Thanks!20:21
=== cmagina is now known as cmagina-away
=== roadmr is now known as roadmr_afk
=== cmagina-away is now known as cmagina
allenapblake_r: Your packaging change looks good, but I’d prefer for roaksoax to see it before it lands; there may be some paperwork to do this close to release.21:52
=== cmagina is now known as cmagina-away
=== cmagina-away is now known as cmagina
Valduarehi guys - any word on maas for arm devices22:47
=== cmagina is now known as cmagina-away
bigjoolsValduare: what specifically are you looking for?23:47
Valduarelooking forward to the day when all these little mk902 style devices could be used with something like MaaS :P23:47
bigjoolsmaas supports armhf, I don't know much else about the devices themselves23:48
Valduarei dont know much about maas yet23:49
Valduaresomeone was telling me it powers down the devices, pxe wol stuff?23:49
bigjoolswol cannot power down23:49
bigjoolsbut maas supports many power typesa23:49
Valduaredoes this require a custom u-boot or anything?23:50
Valduareor can I start playing with a few of my little mk808 devices that are running an ubuntu server armhf rootfs23:50
bigjoolspossibly, yes.  The image that is served up depends on the boot request path23:50
bigjoolslet me just double check the path23:50
bigjoolsok so if uboot TFTP requests default-armhf or default.armhf it should work23:52
Valduarehmm23:54
Valduareso is it just u-boot on the device and kernel and rootfs are tftp over?23:55

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!