[01:50] <jtv> roaksoax: you still have that 1.4 branch waiting to land...  should I land it?
[01:51] <jtv> There were some review changes to be made... don't know if you made them.
[04:51] <jtv> bigjools: could you review this one for me?  https://code.launchpad.net/~jtv/maas-test/fix-setup-error-reporting/+merge/194781
[04:54] <bigjools> jtv: sure
[04:54] <jtv> thx
[04:56] <bigjools> done
[04:56]  * bigjools goes back to CSS
[04:57] <jtv> ...UI work?
[04:57] <bigjools> aye
[04:57] <jtv> Not the Canadian Secret Service then.
[04:57] <bigjools> will show you shortly
[04:57] <bigjools> haha
[04:57] <jtv> (If they're so secret, how come they're in the GTF, huh?)
[04:57] <bigjools> you have SIS already?
[04:58] <jtv> Which one?
[04:58] <jtv> 11 entries so far.
[04:58] <bigjools> Secret Intelligence Service
[04:58] <jtv> Yup, got that.
[04:59] <bigjools> splendid
[05:26] <bigjools> jtv: as promised: https://code.launchpad.net/~julian-edwards/maas/warning-no-power/+merge/194787
[05:26] <bigjools> Azendale: still there?
[05:26] <Azendale> bigjools: still here
[05:27] <Azendale> bigjools: give me a minute to finish the project I'm in
[05:27] <bigjools> Azendale: I have been thinking about your problem and I think you might want to find some virtualisation experts.  I can't see anything wrong with any of the maas logs you showed me so I'm mostly convinced there's some error in the networking over virtio
[05:34] <Azendale> bigjools: TBH, I'm not sure where to start on that route. But I can try. I'm thinking I could try breaking it, and then running a packet capture on the MaaS server. If it's still sending and receiving packets, I would think there would be something up with the software. I think I saw that once trying to figure it out myself, but I would like to be sure of that before saying that is the case
[05:35] <bigjools> Azendale: the reason I am sceptical it's maas is because we have a physical lab of QA machines that get fired up in tandem and they are fiine.
[05:37] <Azendale> bigjools: The other thing I could think of is setting up an LXC container on my machine, setting up MaaS inside it, and then bridging it's network to virtual machines. I could also try not using virtio
[05:38] <bigjools> all worthwhile, yeah
[05:38] <Azendale> bigjools: That does make it seem more like a virtualization thing then, though. I guess I'll keep expirimenting around
[05:39] <bigjools> Azendale: also, the fact that it fails at the DHCP level means there's no maas code involved at that point, and I don't believe that isc-dhcp-server would randomly give up under load, it's pretty solid
[05:42] <Azendale> bigjools: I do agree there, that is pretty widely used. Well, thanks for taking a look at it, I'll see what I can find. I'll probably be back with questions though (if I find something weird that I can't figure out)
[05:43] <bigjools> Azendale: sure, please let me know what you find, I am very interested.
[05:43] <bigjools> I have vague recollections of virtual networking being weird like this under load, but could be wrong.
[05:46] <Azendale> bigjools: definitely possible. Though I have seen stuff pulling during the installs from the package cache I have on the host machine at 30 MiB/s. Maybe it's just UDP stuff
[05:46] <bigjools> could well be
[05:47] <Azendale> bigjools: ok, well thanks for the help, I'll let you get back to whatever you were doing before I figured out some weird way to break stuff :)
[05:48] <bigjools> Azendale: heh no problem
[06:53] <jtv> bigjools: would also appreciate your opinion on a change I'd like to make... time for a hangout?
[07:17] <bigjools> jtv: yes
[08:19] <jtv> Ahhh — when you fire up a VM with uvtool, it does an apt-get update.  If that isn't finished yet by the time we start installing maas, things break horribly.
[08:20] <jtv> Looks like we do need proper waiting (uvt-kvm wait, followed by polling for /var/lib/cloud/instance/boot-finished).
[08:25] <rvba> jtv: You're right, we need to wait until cloud-init finishes.
[08:25] <jtv> In practice it may depend on the speed you get from your mirror, but for me, that means it doesn't.
[08:25] <jtv> Shame that that can't be fast, but oh well.
[08:56] <rbasak> jtv: currently it installs avahi-daemon, but I want to drop that. But yes - we need to wait for boot-finished in the general case.
[08:57] <rbasak> (and eventually uvt-kvm wait should incorporate that)
[08:57] <jtv> It's not so much the avahi-daemon that bothers me, it's the maas.
[08:57] <rbasak> For speed?
[08:57] <jtv> No, to avoid failure.
[08:57] <rbasak> I meant that the race is a consequence of installing avahi-daemon. Without that, cloud-init won't hit apt-get on boot.
[08:58] <jtv> It looks as if, before the "apt-get update" completes, the VM has as an apt database that doesn't match what's on the local mirror.
[08:58] <jtv> So I think if we ditch avahi-daemon, we'll have to do an apt-get update anyway.
[08:58] <rbasak> Ah, I see. Yes, that's true.
[08:58] <jtv> But it'll be easier than polling.
[08:59] <jtv> Because we'll be running it synchronously to our ssh session.
[08:59] <jtv> A polling loop doesn't sound like much, but it needs testing etc.
[08:59] <rbasak> A daily image might be closer to the apt mirror than the "release" one. I'm not sure using that is a good idea for production stuff though.
[08:59] <rbasak> When you sync, you can override the mirror to http://cloud-images.ubuntu.com/daily/ I think.
[09:00] <jtv> What was that about saving time?  :)
[09:00] <jtv> For all I know, the image's built-in apt db might have more recent versions that aren't on the mirror yet...
[09:00] <jtv> Or the other way around.  So yes, would be nice to get it all off the same mirror.
[09:01] <rbasak> You could also prep an image and get uvt-kvm to use that instead of a pristine one
[09:01] <rbasak> --backing-image-file
[09:01] <rbasak> Make sure to remove /var/lib/cloud/instance/boot-finished manually if you do that and want to poll for it though.
[09:02] <jtv> Not a worry if we do it synchronously, I guess.
[09:02] <rbasak> mount-image-callback is handy for manually prepping the image if you want to do that.
[09:02] <jtv> Not just now, to be honest — that's optimisation.
[09:02] <rbasak> (or you can use a VM)
[09:02] <jtv> It actually working sort of takes priority.  :)
[09:03] <rbasak> :)
[09:03] <rbasak> Though if your development iteration will be faster, it might be worth it
[09:04] <jtv> Yes, it will help.  But if it takes longer to build, that also affects our development cycle.
[09:04] <jtv> So bit of a trade-off.
[09:05] <jtv> So... two simple alternatives:
[09:05] <jtv> 1. Wait for boot to complete, and keep doing the apt-get update even when we ditch avahi-daemon.
[09:05] <jtv> 2. Always use the central archive.
[09:06] <jtv> And #2 needs daily uvt images, right?
[09:06] <jtv> To be reliable?
[09:08] <rbasak> To be reliable, I think I'd recommend always running apt-get update
[09:08] <jtv> It does seem like the most robust option.
[09:09] <jtv> So I'll just do the waiting.
[09:09] <rbasak> It might be safe to try apt-get --download-only install first, and if that fails to download then apt-get update first before retry, and then apt-get install
[09:09] <rbasak> Doing that with a daily might give you a speedup
[09:10] <jtv> Or a slowdown I guess, if the downloads are out of date.  Is it worth it?
[09:10] <rbasak> However, I presume the purpose of a daily is that Scott or Ben can break it for a day without everyone yelling at them, so we probably shouldn't do that in production
[09:10] <rbasak> Not worth it for me, I don't think. YMMV on your connection and your end of the world.
[09:11] <jtv> Simple is beautiful.
[09:46] <jtv> rvba: I think I spotted KVMFixture.ip_address being called multiple times in a test somewhere...  Shouldn't we just make start() say self.ip_address = [...]?
[09:46] <jtv> Not a high-priority item.  :)
[09:48] <rvba> jtv: that's an optimization we can do indeed.  Like you said, not really urgent.
[09:53] <jtv> rvba: didn't you find a package a while back with little utilities like "wait until file exists"?
[09:55] <rvba> jtv: I think you're think about http://blog.dustinkirkland.com/2013/09/introducing-run-one-constantly-run-one.html
[09:55] <rvba> thinking*
[09:55] <jtv> Yes, that's it, thanks.
[09:55] <jtv> There just has to be a tool like this in there.
[09:56] <jtv> Argh, no, I can't use that.  We can't apt-get install yet at that point.
[13:48] <freeflying> maas profile nodes list shows some nodes has two ip address, would it be a issue
[14:47] <jtv> freeflying: it might be, IIRC.  It shouldn't really happen.
[14:47] <freeflying> jtv, it did happen here :)
[14:47] <jtv> roaksoax: hi — any idea about this installation error in a VM?
[14:47] <jtv> /tmp/maas-region-controller.config.51653: 35: /tmp/maas-region-controller.config.51653: dbc_go: not found
[14:49] <jtv> roaksoax: that's saucy/amd64
[14:49] <freeflying> jtv,     "ip_addresses": [
[14:49] <freeflying>             "10.213.0.108",
[14:49] <freeflying>             "10.213.0.111"
[14:49] <freeflying>         ],
[14:50] <jtv> If MAAS manages DHCP, leases never expire... In principle (there's another weasel term) a node should always keep the same IP address on the same interface.
[14:50] <jtv> But sometimes, it seems...
[14:53] <freeflying> jtv, in our case, we have 1 bond nics has 2 slaves, due to some reason, the mac of bond could be either of the 2 slaves
[14:56] <freeflying> jtv, led maas has 2 ip for this nodes
[14:58] <jtv> freeflying: if the bonded interface arbitrarily picks the MAC of one of the two underlying interfaces, I guess after a few boots they'd both get IP addresses and things could get confused...  Maybe the real problem is the unpredictability of how these two interfaces come up.
[14:59] <jtv> I need to leave now, I'm afraid...  It sounds as if the problem may be at a very low level, so consider asking in #ubuntu-server.
[14:59] <freeflying> jtv, thats true, for example, in bond 1, when fail over happens, then the mac will be changed
[15:06] <rbasak> rvba: oh, you can put strings on separate lines and they get concatenated? I didn't know that - thanks!
[15:06] <rvba> rbasak: the operator % takes priority over +.
[15:09] <rbasak> rvba: I usually stick brackets in. I didn't realise I could just drop the +
[15:09] <roaksoax> jtv: yeah, but that's nothing that you should worry about
[15:09] <roaksoax> jtv: that does not cause an install error nor prevent maas from installing
[15:10] <rbasak> Can I merge directly from the MP in Launchpad somehow? Or do I need to do it locally and push?
[15:13] <rvba> rbasak: unless you have a lander like tarmac set up, then I think your only option is to merge locally and push.
[15:13] <rbasak> OK, thanks.
[15:13] <rbasak> Can you tell I'm new to this? :)
[15:13] <rvba> ;)
[15:21] <rvba> rbasak: does this error ring a bell: "libvirt.libvirtError: xml in virDomainDefineXML must not be NULL" I'm running this:http://paste.ubuntu.com/6405809/
[15:21] <rbasak> rvba: no. I've not seen that before.
[15:22]  * rbasak hopes he hasn't broken something recently
[15:22] <rbasak> I need tests :-/
[15:22] <rvba> I think it's related to my machine… I can't reproduce it on canonistack.
[15:22] <rvba> I think I fucked something up locally :/.
[15:23] <rbasak> rvba: can you drop into pdb on line 322 and print xml? Or is that what you're doing already?
[15:23] <rbasak> rvba: also type(xml) please
[15:23] <rbasak> (just as I'm paranoid)
[15:24] <rvba> rbasak: what I was printing was the result of compose_domain_xml().
[15:24] <rvba> rbasak: let me remove all the packages and start over first…
[15:28] <rvba> rbasak: I re-installed everything and now it works… I probably messed up the template… sorry for the noise.
[15:28] <rbasak> np
[16:02] <NCommander> If I get a "Tests Failed" status, where can I find what exactly failed?
[16:03] <NCommander> The console output isn't clear (I have some IPMI errors, but that should be expected, I don't have working IPMI)
[16:40] <fader> Hey folks, I installed MAAS on 12.04 from the havana cloud-archive but am getting some django errors ("ImportError: No module named simple") when hitting the MAAS webservice.  Googling around suggests that something is relying on some depricated functionality in django (http://stackoverflow.com/questions/15085626/no-module-named-simple-login-required-depreciated) but I'm kinda stuck after that point
[16:40] <fader> It's a fresh install that's up-to-date on packages... is there anything obvious I might be missing?
[16:49] <matsubara> fader, did you add the cloud tools archive? I think if you only added the cloud-archive:havana it's still pulling the MAAS packages from the main archive, not the cloud tools one.
[16:50] <fader> matsubara: Aha, that sounds like it could be my issue.  I'll give it a shot, thanks!
[16:51] <fader> (As a side note, it would be nice if these things were on the MAAS install docs page.  I went with the cloud archive as I'd heard that the main archive isn't getting updated, but didn't realize the cloud tools archive was separate.)
[16:51] <rbasak> fader: the havana cloud archive pocket is not compatible with the cloud-tools pocket, AIUI. You can only have one.
[16:52] <fader> Hmm... so is there a working way to get MAAS up and running and deploy havana to it?
[16:53] <rbasak> Yes, you can do that.
[16:53] <rbasak> Your MAAS installation will use the cloud-tools pocket
[16:53] <rbasak> Your deployed nodes running Openstack components will use the havana pocket.
[16:53] <rbasak> Just not both on the same node
[16:54] <fader> Ah, gotcha.  I thought the MAAS server would cache and serve up packages based on its configuration
[17:14] <NCommander> So where do commisioning/enlsitment scripts come from in the epimerial images?
[23:05] <bigjools> NCommander: they are in /etc/maas/
[23:05] <bigjools> not in the ephemerals