[04:02] <bigjools> jtv: so, adding default-driver into the maas.tgt didn't h
[04:02] <bigjools> elp
[04:02] <jtv> Can I have a look at the config you have now?
[04:03] <jtv> If you were running on a machine that has also run the old ephemerals script, before you added the --conf option to tgt-admin you would always have gotten a default-driver setting anyway.
[04:04] <bigjools> jtv: AHA
[04:04] <bigjools> we add that and remove the driver option in each block
[04:04] <bigjools> and it worked
[04:04] <jtv> Yeah, it's the driver option.
[04:04] <bigjools> now I get a different error, of course....
[04:04] <jtv> ?
[04:04] <bigjools> Multiple 'readonly' definitions in 'backing-store' not allowed!
[04:04] <jtv> ...
[04:05] <bigjools> it seems like it hates the multiple targets in the same file
[04:05] <jtv> Can I see the file?
[04:06] <jtv> Guh.  Is that root-image file seriously over a GB in size?
[04:06] <bigjools> http://paste.ubuntu.com/7144556/
[04:06] <bigjools> it is
[04:06] <jtv> Thanks.
[04:06] <jtv> I hope that's very sparse...
[04:06] <bigjools> tgt-admin is basically as bug-ridden per-LOC as you'd expect for Perl
[04:06] <jtv> Looks like we have two identical entries in that file...  If I were tgt I wouldn't like that either.
[04:07] <bigjools> jtv: interesting - so the list_boot_images() had two similar entries.  Oh.... two purposes!
[04:07] <jtv> Yup.
[04:07]  * bigjools fixorates
[04:08] <bigjools> jtv: did you review that branch BTW?
[04:08] <bigjools> ah you did
[04:08] <jtv> Yes.
[04:12] <bigjools> ok made the extra changes
[04:12] <bigjools> forgot that extra close after thinking to myself that I needed to fix it, too ...
[04:13] <jtv> We would have caught it at some point.
[04:20] <bigjools> jtv: bug #3453453534324, if anything fails after the maas.meta is written, the script will never try to write the tgt files on the next run
[04:20] <bigjools> it exits early as it thinks it's in sync
[04:21] <jtv> How about we try to generate all the file contents first, and then call atomic_write to do the actual writing?
[04:22] <jtv> Not so much to get complete atomicity, as to put as much of the stuff that can fail before the part where we write files.
[04:22] <jtv> It'll break more, but when it breaks, it'll be more likely to break in a consistent nothing-got-done state.
[04:22] <bigjools> jtv: I am victorious
[04:23] <jtv> Pleased to meet you, Victorinox.
[04:23] <bigjools> we could have another state file
[04:23] <bigjools> but yes atomic_write for state + data please
[04:24] <bigjools> "sudo tgt-admin -s" now shows me stuff.
[04:24] <jtv> Good!
[04:24] <bigjools> need to do a proper fix for list_boot_images
[04:24] <jtv> Will it boot?
[04:24] <bigjools> I just hacked it
[04:24] <bigjools> dunno I just did all this on canonistack where it's quick to delete huges files and redownload
[04:25] <jtv> To be clear: you don't need a fix for list_boot_images itself, just the call site, right?
[04:25] <jtv> Or is the boot purpose completely gone?
[04:25] <bigjools> no, a fix is needed in extract_image_params
[04:25] <bigjools> it is completely gone
[04:26] <bigjools> do you fancy doing that while I test a real boot?
[04:26] <bigjools> one thing that's annoying now is that it copies a 1.4G file
[04:26] <jtv> Let me just finish up what I'm doing here...
[04:26] <bigjools> so there's two copies of it after download
[04:26] <jtv> Whoops.
[04:27] <bigjools> yeah
[04:27] <jtv> And will the copy support holes?
[04:27] <bigjools> that's the slowest part of this on canonistack!
[04:27] <bigjools> no idea
[04:27] <bigjools> sparse file you mean?
[04:27] <jtv> Where does the copying happen?
[04:27] <jtv> Yes.
[04:27] <bigjools> not sure, looking
[04:31] <bigjools> jtv: it uses simplestreams.objectstore
[04:31] <bigjools> I vaguely remember something from scott about it supporting sparse files
[04:31] <jtv> If Scott did it, I'm sure it'll support sparse files as needed.
[04:31] <bigjools> I don;t understand why we need to keep 2 copies of such huge files
[04:31] <jtv> Because IIRC the boot disk image does have holes.
[04:32] <bigjools> it's still going to be 400Mb
[04:32] <jtv> I'm sure Scott would have used a link if he could...
[04:33] <bigjools> jtv: do we still need that policy hack?
[04:33] <jtv> No.
[04:33] <jtv> Should be supported directly by the code now.
[04:34] <bigjools> cool
[04:34] <bigjools> ta
[04:44] <bigjools> jtv: how are we supposed to put an arch list in the filter in pserv.yaml?
[04:44] <bigjools> I tried a list and it says it needs to be a single value
[04:45] <jtv> It does.
[04:45] <bigjools> I am guessing it wants a filter
[04:45] <bigjools> which leaves me itching
[04:45] <jtv> If you need multiple arches but not '*', define multiple sources.
[04:45] <bigjools> !
[04:45] <bigjools> srsly?
[04:46] <jtv> srsly.  But easy enough to change.
[04:46] <jtv> I'm hammering a little bit of abstraction into that code as we speak.
[04:46]  * bigjools adds a card
[04:46] <jtv> I'm supposed to be adding logging, but in order to log what's going on, we need to know what's going on.
[04:47] <bigjools> yes :)
[04:47] <bigjools> jtv: also the error reporting from a dodgy pserv.yaml is ugly as hell with a stacktrace
[04:47]  * bigjools files another card
[04:48] <bigjools> jtv: do you know if we fixed the bug where we still need i386 when booting amd64?
[04:50] <jtv> Don't know
[05:05] <bigjools> christ it's slow to download
[05:05] <bigjools> 82M in 15 minutes
[05:57] <bigjools> jtv: ok done, attempting boot
[05:59] <bigjools> jtv: 2014-03-24 15:57:06+1000 [TFTP (UDP)] Datagram received from ('10.0.0.104', 49155): <RRQDatagram(filename=amd64/generic/trusty/no-such-image/boot-kernel, mode=octet, options={'tsize': '0', 'blksize': '1408'})>
[05:59] <bigjools> darn
[05:59] <jtv> That means pxeconfig didn't find a matching image.
[06:00] <bigjools> yup
[06:01] <bigjools> jtv: AND it thought it was enlisting, and it was an installation boot
[06:01] <bigjools> this is using FPI
[06:01] <jtv> Enlistment vs. installation is decided by... the database, right?
[06:01] <bigjools> let's see what  bootimages are there
[06:02] <jtv> Unknown node gets enlistment image?
[06:02] <bigjools> if "node" is set, yes
[06:02] <bigjools> basically if there's a recognised MAC in the request
[06:02] <jtv> (For some reason my twistd stopped using up gobs of memory)
[06:03] <bigjools> I'll watch the console this time, one sec
[06:07] <bigjools> it sends the mac
[06:12] <bigjools> contents of BootImage look ok
[06:15] <bigjools> jtv: so basically this code fails to find an image:
[06:15] <bigjools>     latest_image = BootImage.objects.get_latest_image(
[06:15] <bigjools>         nodegroup, arch, subarch, series, purpose)
[06:17] <jtv> Mismatch between purpose names..?
[06:18] <bigjools> trying to use the harness to see what it would be using
[06:19] <jtv> What I like to do is "open('/tmp/foo.log','a').write("Got purpose: %s\n" % purpose)"
[06:19] <bigjools> jtv: aaaaa
[06:19] <bigjools> no "xinstall"
[06:19] <jtv> xinstall!?
[06:19] <bigjools> yeah
[06:19] <bigjools> see get_boot_purpose()
[06:19] <jtv> Oh, this is while installing?
[06:19] <bigjools> yes
[06:19] <jtv> And do the boot purposes in the database still say xinstall?
[06:20] <jtv> Try "sudo maas-region-admin dbshell --installed"
[06:20] <bigjools> no
[06:20] <bigjools> they just have commissioning and install
[06:20] <jtv> Ah.  Well there we go.
[06:20] <bigjools> I downloaded the trusty amd64 images
[06:20] <bigjools> and it set those up
[06:21] <jtv> Will the classic installer work?
[06:21] <bigjools> let's see
[06:22] <bigjools> jtv: working
[06:23] <bigjools> ok so
[06:23] <bigjools> not sure what to do here.  are we getting rid of "purpose" or not ...
[06:23] <bigjools> oh ha spoke to soon
[06:23] <bigjools> d-i stopped with debconf message saying "no kernel modules found"
[06:26] <jtv> Missing initrd..?
[06:26] <jtv> Well, no, it wouldn't be missing I guess or you'd fail during pivot().
[06:26] <jtv> Or so I seem to remember.
[06:26] <bigjools> mismatching kernel version vs modules version
[06:27] <bigjools> I suspect crap data coming from simplestreams
[06:27] <jtv> That would be pretty awful.
[06:27] <jtv> But what if it's a filtering problem somewhere?
[06:27] <bigjools> in what way?
[06:27] <jtv> Maybe there's an arbitrary, unstable choice between similar images.
[06:28] <bigjools> there's only one image
[06:29] <bigjools> jtv: ok so let's go back to xinstall
[06:30] <bigjools> extract_image_params() only reports commissioning and install, and we want to get rid of those.  Do we now need to differentiate boot purpose here like that in get_boot_purpose?
[06:31] <bigjools> it looks like we still depend on that purpose to select a different preseed for the FPI
[06:33] <jtv> There's an irony there.
[06:36] <jtv> I suppose you could just add xinstall to the list of purposes in extract_image_params..?
[06:36] <jtv> By the way, it does look as if the files that the import script puts in the snapshots are links into the cache, as you'd expect.
[06:37] <bigjools> jtv: I can but we also depend on that list only having a single item for the report_boot_images call to work
[06:37] <bigjools> sorry list_boot_images
[06:37] <bigjools> which sets up the tgt targets
[06:38] <jtv> Shouldn't that code just strip off the purpose and run the rest of the images' data through a set to eliminate duplicates?
[06:39] <jtv> sorted( { (img['arch'], img['subarch'], img['release'], img['label']) for img in list_boot_images(...)})
[06:40] <jtv> Or more legibly:
[06:40] <jtv> sorted({
[06:40] <jtv>     (img['arch'], img['subarch'], img['release'], img['label'])
[06:40] <jtv>     for img in list_boot_images(...)
[06:40] <jtv>     })
[06:40] <bigjools> that's one way to fix it, not sure if it's the best
[06:41] <jtv> Well it's only this use of the boot images that needs them to be unique despite ignoring the purpose, right?
[06:41] <bigjools> we don't really need to discern the difference when selecting a boot image
[06:41] <bigjools> they are all the same now
[06:41] <jtv> Then we should drop BootImage.purpose.
[06:41] <bigjools> indeed
[06:41] <jtv> Just... not now  :)
[06:42] <bigjools> well I'll take it out of the code and see how it goes
[06:42] <bigjools> I put "xinstall" in the list for now
[06:43] <bigjools> looking good so far
[06:47] <bigjools> boom
[06:48] <bigjools> cloud-init crashed ot because:
[06:48] <bigjools> --2014-03-24 06:44:26--  http://10.0.0.9/MAAS/static/images/amd64/generic/trusty/xinstall/root.tar.gz
[06:48] <bigjools> Connecting to 10.0.0.9:80... connected.
[06:48] <bigjools> HTTP request sent, awaiting response... 404 Not Found
[06:48] <bigjools> 2014-03-24 06:44:26 ERROR 404: Not Found.
[06:48] <bigjools> somewhere else that needs purpose eliminating
[06:54] <jtv> By the way, there's some code in the new import script that checks if the JSON in the maas.meta file already matches the latest data, but it doesn't sort dict keys so that might be a little arbitrary.
[06:54] <jtv> I'm dumping the json with sorted keys instead.
[06:55] <bigjools> ok
[06:56] <bigjools> jtv: ARGH.  The curtin installer expects the root.tar.gz to live under tftp, as served by apache
[06:58] <jtv> Huh?
[06:59] <jtv> You mean that it expects to be able to access the tftp tree through http?
[07:01] <bigjools> there's apache config to mirror it at /MAAS/static/images/
[07:02] <bigjools> we need to fix the packaging for this.
[07:02] <bigjools> dammit
[07:04] <jtv> Not a good time.
[07:09] <bigjools> jtv: also, another problem
[07:09] <bigjools> get_curtin_userdata() needs to know the label in the path now
[07:09] <bigjools> not sure how it's going to do that
[07:11] <jtv> We have the label in the path now, don't we?
[07:14] <bigjools> yes
[07:19] <rvba> bigjools: care to have a look at this tiny fix? https://code.launchpad.net/~rvb/maas-test/fix-new-option/+merge/212356
[07:20] <bigjools> rvba: done
[07:20] <rvba> Ta
[07:22] <bigjools> I am this -><- close to FPI installer working
[07:26] <bigjools> FPI success! \o/
[07:26] <jtv> \o/
[07:26] <rvba> \o/
[07:26] <bigjools> it was a *hack* though
[07:26] <bigjools> still have a problem to fix as I describe in the email
[07:26]  * bigjools hands baton to rvba
[07:27] <rvba> bigjools: question about the first item in your email:
[07:27] <bigjools> yarp
[07:27] <rvba> You said we could do away with the purpose in boot images.
[07:27] <rvba> I don't think we want that.
[07:27] <bigjools> ok, go on
[07:28] <rvba> Right now, when we import stuff, for each "selection" (arch/subarch/series[/soon:label]) we import both the ephemeral and the install image.
[07:28] <bigjools> yes
[07:28] <rvba> But in the future, we thought it would be nice to let the use select what image he wants to import.
[07:28] <rvba> The reason being that the ephemerals are huge.
[07:29] <rvba> And the install images are not.
[07:29] <bigjools> yes
[07:29] <bigjools> as things stand, purpose is useless though
[07:29] <rvba> So you might want to download the ephemerals for just, say, trusty/amd64/generic.
[07:29] <rvba> When this is the case the purpose will be useful.
[07:30] <bigjools> ok
[07:30] <bigjools> I don't disagree
[07:30] <bigjools> go with the first option to fix then
[07:30]  * bigjools heads to dinner. back for call
[07:54] <rvba> jtv: one question about https://code.launchpad.net/~jtv/maas/write-tgt-data-at-once/+merge/212358
[07:55] <rvba> jtv: Unless I'm mistaken, you didn't solve the problem where the same entry gets written multiple times in the tgt config file… right?
[07:55] <rvba> jtv: which is fine btw, I just want to be sure.
[07:55] <jtv> No, that's a separate problem which Julian worked on.
[07:55] <rvba> Okay.
[07:56] <jtv> My take was: just turn the data into tuples, without the purpose, and put them in a set to eliminate duplicates.
[07:56] <rvba> That's the problem I started working on.
[07:56] <jtv> Ah.
[07:56] <jtv> I thought Julian said he was doing that.
[07:56] <rvba> I didn't catch that.
[07:57] <rvba> No, we talked about it just now (see above).
[07:57] <jtv> Ah.
[07:57] <rvba> I'll take care of it.
[07:57] <jtv> OK
[07:57] <rvba> Once I'm done with your review.
[07:57] <jtv> Thanks.
[07:59] <jtv> There will be conflicts in the branch after that, but nothing serious.
[08:03] <rvba> jtv: branch approved!
[08:03] <jtv> Thanks.
[08:23] <rvba> jtv: would you reciprocate? https://code.launchpad.net/~rvb/maas/fix-list-boot-imgs/+merge/212363
[08:24] <jtv> Honour demands it.
[08:24] <rvba> heh
[08:26] <jtv> rvba: in trying to figure out what the code does, I did do a bit of refactoring — which will also facilitate my ongoing work.
[08:26] <rvba> jtv: I see that.
[08:26] <jtv> Can't log what's going on without knowing what the code means.  :)
[08:26] <rvba> Makes sense.
[08:27] <jtv> I'm turning the "arch" config into an "arches" — which will benefit from the new helpers.
[08:28] <rvba> Hang on.
[08:29] <rvba> Not sure this is appropriate.
[08:29] <rvba> Because the subarches list really only makes sense for a particular architecture.
[08:30] <rvba> jtv: if we turn 'arch' into a list we will end up with the same mess that we have now where the import script imports a huge product of things.
[08:30] <jtv> True — but what's the harm in a filter that allows 'generic' and 'highbank' for armhf and i386?
[08:30] <rvba> That's precisely why we have a 'selections' list.
[08:30] <rvba> jtv: In this case it would be fine but in the general case, it won't make sense.
[08:31] <rvba> And can lead to import more things than you thing… which — I think — I bad.
[08:31] <rvba> s/thing/think/
[08:32] <jtv> I agree that it can be used for evil, but is it our job to prevent that?  The filters within one selections stanza form a Cartesian product, which I think is fine for the usual "i386/generic + amd64/generic"—style situation.
[08:32] <jtv> When the Cartesian product gets out of hand, split up the selections.
[08:33] <jtv> Wouldn't a comment suffice?
[08:33] <rvba> Hum, maybe…
[08:34] <rvba> I think that by renaming that field to 'arches' we encourage a potentially dangerous practice.
[08:34] <jtv> We make it possible.  Potentially dangerous is not the same as bad.
[08:34] <rvba> But maybe a well-written comment will suffice.
[08:35] <jtv> I'll document it carefully.
[08:35] <rvba> Okay.
[09:00] <jtv> rvba: whoops, there's no set.append().  :-)
[09:01] <rvba> jtv: whoops… fixing…
[09:01] <jtv> Did you say my logging branch was dangerous and needed testing before landing?  :-P
[09:02] <rvba> yeah :)
[09:03] <rvba> jtv: I just pushed a fix.  Sorry about that.
[09:04] <jtv> No worries.  It'll just make me feel slightly less guilty next time I break something.  :)
[09:04] <rvba> heh
[10:46] <jtv> I'm getting breakage on the new import script, looks related to the simplestreams data:
[10:46] <jtv>     arch, subarches = item['arch'], item['subarches']
[10:46] <jtv> KeyError: u'subarches'
[10:52] <jtv> Trying with the -v2 data.
[11:18] <bigjools> I unsubbed maas-maintainers from https://code.launchpad.net/~maas-maintainers/maas/new-import-script-integration.  Please add yourselves manually if you need.
[11:53] <jtv> My last import run went suspiciously smoothly.
[12:23] <jtv> Not much luck booting with the new images yet...  unhandled error during Deferred.  :(
[12:24] <jtv> rvba: https://code.launchpad.net/~jtv/maas/multi-arch-imports/+merge/212392
[12:25] <jtv> That's the one we discussed.  Hope you find the comments clear enough.
[12:25] <rvba> k, I'll have a look in a sec.
[12:33] <rvba> jtv: here is a review for you if you have time: https://code.launchpad.net/~rvb/maas/fix-curtin-url/+merge/212402
[12:33] <jtv> Looking.
[17:57] <allenap> blake_r: Hello. How’s the UEFI refactor branch doing? I’m eager to review and land it, because my next task is to port it to the new boot resources importer.
[17:58] <blake_r> allenap: just updated it to the lastest trunk, cut the diff in half
[17:58] <blake_r> allenap: I am porting the uefi script port to the new simple streams
[17:59] <allenap> blake_r: Hehe, that’s grand. I’ll review the first bit if you want.
[17:59] <blake_r> allenap: yeah go ahead and start reviwing
[17:59] <blake_r> allenap: I will get the rest in here soon
[18:00] <allenap> blake_r: Create a merge proposal for it and I’ll do it.
[18:00] <blake_r> allenap: there is one, let me get the link
[18:00] <blake_r> allenap: its in wip, want it set to review?
[18:00] <allenap> blake_r: Yeah.
[18:00] <blake_r> allenap: done
[18:00] <allenap> Tip top.
[19:13] <manjiri_> Hello! Is there a way to specify cloud-init user-data for maas?
[19:26] <rbasak> manjiri_: can you be more specific? Do you want this for enlistment, commissioning or deployment?
[19:27] <rbasak> (and if for deployment, then during install stage, or for final boot?)
[19:28] <manjiri_> rbasak: Deployment. I want to wget a file at the end.
[19:29] <manjiri_> rbasak: Install stage, I think. Can you explain what is "final boot" ?
[19:30] <rbasak> manjiri_: what I mean by "final boot" is the boot that your MAAS client (eg. juju) will get.
[19:33] <manjiri_> rbasak: My goal is make this "wget" part of "staging" rather than during "juju deploy". But I can settle for it to be during "juju add-machine".
[19:35] <rbasak> manjiri_: juju controls cloud-init userdata to actually boot juju machines. I'm not aware of any mechanism to control that.
[19:36] <rbasak> manjiri_: what's the purpose of your wget script?
[19:36] <rbasak> manjiri_: if there's a one-off action you want on all MAAS-deployed juju nodes, you could do that in the installer.
[19:36] <rbasak> manjiri_: either by preseeding d-i (if using d-i), or a suitable curtin hook (for the fast path installer; I know less about this)
[19:36] <manjiri_> rbasak: Yes that is it. How do I "do that in the installer". ?
[19:37] <rbasak> manjiri_: if using curtin then I guess you could use cloud-init userdata, but I think that would apply before the installer has finished
[19:38] <manjiri_> rbasak: Got it. This is what I had surmized but I wanted to confirm. Thanks!
[19:39] <manjiri_> rbasak: Any pointers for more info on "curtin hooks" ?
[19:43] <blake_r> allenap: just pushed the install_bootloader from the UEFIBootMethod, should all be their now
[19:43] <manjiri_> rbasak: Any pointers on making the URL arg for wget a config option?
[20:00] <rbasak> manjiri_: http://bazaar.launchpad.net/~curtin-dev/curtin/trunk/view/head:/doc/topics/overview.rst#L112
[20:00] <rbasak> manjiri_: I'm not sure how curtin's config is generated. Probably via templates in /etc/maas or something.
[20:09] <AskUbuntu> maas user interface does not talk to or sets up dns/dhcp | http://askubuntu.com/q/438705
[20:12] <manjiri_> rbasak: Thanks! I think a late or final command should do the trick. There are no examples to parametrize the URL through GUI/CLI. For now, hardcoding seems to be the only option.
[20:19] <rbasak> manjiri_: the templates let you execute arbitrary Python and access the data model (IIRC).
[20:19] <rbasak> manjiri_: But, I don't know of any means to put arbitrary parameters into the data model that you can then use.
[20:20] <rbasak> manjiri_: you can apply arbitrary tags which you could probably get to from the template, if that helps.
[20:20] <rbasak> (and of course anything you do in this area might break in a future release)
[20:21] <manjiri_> rbasak: Good advice. Thanks!
[21:52] <allenap> blake_r: Your packaging change looks good, but I’d prefer for roaksoax to see it before it lands; there may be some paperwork to do this close to release.
[22:47] <Valduare> hi guys - any word on maas for arm devices
[23:47] <bigjools> Valduare: what specifically are you looking for?
[23:47] <Valduare> looking forward to the day when all these little mk902 style devices could be used with something like MaaS :P
[23:48] <bigjools> maas supports armhf, I don't know much else about the devices themselves
[23:49] <Valduare> i dont know much about maas yet
[23:49] <Valduare> someone was telling me it powers down the devices, pxe wol stuff?
[23:49] <bigjools> wol cannot power down
[23:49] <bigjools> but maas supports many power typesa
[23:50] <Valduare> does this require a custom u-boot or anything?
[23:50] <Valduare> or can I start playing with a few of my little mk808 devices that are running an ubuntu server armhf rootfs
[23:50] <bigjools> possibly, yes.  The image that is served up depends on the boot request path
[23:50] <bigjools> let me just double check the path
[23:52] <bigjools> ok so if uboot TFTP requests default-armhf or default.armhf it should work
[23:54] <Valduare> hmm
[23:55] <Valduare> so is it just u-boot on the device and kernel and rootfs are tftp over?