[01:50] roaksoax: you still have that 1.4 branch waiting to land... should I land it? [01:51] There were some review changes to be made... don't know if you made them. === freeflying is now known as freeflying_away === freeflying_away is now known as freeflying === vertigo is now known as Guest50260 [04:51] bigjools: could you review this one for me? https://code.launchpad.net/~jtv/maas-test/fix-setup-error-reporting/+merge/194781 [04:54] jtv: sure [04:54] thx [04:56] done [04:56] * bigjools goes back to CSS [04:57] ...UI work? [04:57] aye [04:57] Not the Canadian Secret Service then. [04:57] will show you shortly [04:57] haha [04:57] (If they're so secret, how come they're in the GTF, huh?) [04:57] you have SIS already? [04:58] Which one? [04:58] 11 entries so far. [04:58] Secret Intelligence Service [04:58] Yup, got that. [04:59] splendid [05:26] jtv: as promised: https://code.launchpad.net/~julian-edwards/maas/warning-no-power/+merge/194787 [05:26] Azendale: still there? [05:26] bigjools: still here [05:27] bigjools: give me a minute to finish the project I'm in [05:27] Azendale: I have been thinking about your problem and I think you might want to find some virtualisation experts. I can't see anything wrong with any of the maas logs you showed me so I'm mostly convinced there's some error in the networking over virtio [05:34] bigjools: TBH, I'm not sure where to start on that route. But I can try. I'm thinking I could try breaking it, and then running a packet capture on the MaaS server. If it's still sending and receiving packets, I would think there would be something up with the software. I think I saw that once trying to figure it out myself, but I would like to be sure of that before saying that is the case [05:35] Azendale: the reason I am sceptical it's maas is because we have a physical lab of QA machines that get fired up in tandem and they are fiine. [05:37] bigjools: The other thing I could think of is setting up an LXC container on my machine, setting up MaaS inside it, and then bridging it's network to virtual machines. I could also try not using virtio [05:38] all worthwhile, yeah [05:38] bigjools: That does make it seem more like a virtualization thing then, though. I guess I'll keep expirimenting around [05:39] Azendale: also, the fact that it fails at the DHCP level means there's no maas code involved at that point, and I don't believe that isc-dhcp-server would randomly give up under load, it's pretty solid [05:42] bigjools: I do agree there, that is pretty widely used. Well, thanks for taking a look at it, I'll see what I can find. I'll probably be back with questions though (if I find something weird that I can't figure out) [05:43] Azendale: sure, please let me know what you find, I am very interested. [05:43] I have vague recollections of virtual networking being weird like this under load, but could be wrong. [05:46] bigjools: definitely possible. Though I have seen stuff pulling during the installs from the package cache I have on the host machine at 30 MiB/s. Maybe it's just UDP stuff [05:46] could well be [05:47] bigjools: ok, well thanks for the help, I'll let you get back to whatever you were doing before I figured out some weird way to break stuff :) [05:48] Azendale: heh no problem [06:53] bigjools: would also appreciate your opinion on a change I'd like to make... time for a hangout? === CyberJacob|Away is now known as CyberJacob [07:17] jtv: yes [08:19] Ahhh — when you fire up a VM with uvtool, it does an apt-get update. If that isn't finished yet by the time we start installing maas, things break horribly. [08:20] Looks like we do need proper waiting (uvt-kvm wait, followed by polling for /var/lib/cloud/instance/boot-finished). [08:25] jtv: You're right, we need to wait until cloud-init finishes. [08:25] In practice it may depend on the speed you get from your mirror, but for me, that means it doesn't. [08:25] Shame that that can't be fast, but oh well. [08:56] jtv: currently it installs avahi-daemon, but I want to drop that. But yes - we need to wait for boot-finished in the general case. [08:57] (and eventually uvt-kvm wait should incorporate that) [08:57] It's not so much the avahi-daemon that bothers me, it's the maas. [08:57] For speed? [08:57] No, to avoid failure. [08:57] I meant that the race is a consequence of installing avahi-daemon. Without that, cloud-init won't hit apt-get on boot. [08:58] It looks as if, before the "apt-get update" completes, the VM has as an apt database that doesn't match what's on the local mirror. [08:58] So I think if we ditch avahi-daemon, we'll have to do an apt-get update anyway. [08:58] Ah, I see. Yes, that's true. [08:58] But it'll be easier than polling. [08:59] Because we'll be running it synchronously to our ssh session. [08:59] A polling loop doesn't sound like much, but it needs testing etc. [08:59] A daily image might be closer to the apt mirror than the "release" one. I'm not sure using that is a good idea for production stuff though. [08:59] When you sync, you can override the mirror to http://cloud-images.ubuntu.com/daily/ I think. [09:00] What was that about saving time? :) [09:00] For all I know, the image's built-in apt db might have more recent versions that aren't on the mirror yet... [09:00] Or the other way around. So yes, would be nice to get it all off the same mirror. [09:01] You could also prep an image and get uvt-kvm to use that instead of a pristine one [09:01] --backing-image-file [09:01] Make sure to remove /var/lib/cloud/instance/boot-finished manually if you do that and want to poll for it though. [09:02] Not a worry if we do it synchronously, I guess. [09:02] mount-image-callback is handy for manually prepping the image if you want to do that. [09:02] Not just now, to be honest — that's optimisation. [09:02] (or you can use a VM) [09:02] It actually working sort of takes priority. :) [09:03] :) [09:03] Though if your development iteration will be faster, it might be worth it [09:04] Yes, it will help. But if it takes longer to build, that also affects our development cycle. [09:04] So bit of a trade-off. [09:05] So... two simple alternatives: [09:05] 1. Wait for boot to complete, and keep doing the apt-get update even when we ditch avahi-daemon. [09:05] 2. Always use the central archive. [09:06] And #2 needs daily uvt images, right? [09:06] To be reliable? [09:08] To be reliable, I think I'd recommend always running apt-get update [09:08] It does seem like the most robust option. [09:09] So I'll just do the waiting. [09:09] It might be safe to try apt-get --download-only install first, and if that fails to download then apt-get update first before retry, and then apt-get install [09:09] Doing that with a daily might give you a speedup [09:10] Or a slowdown I guess, if the downloads are out of date. Is it worth it? [09:10] However, I presume the purpose of a daily is that Scott or Ben can break it for a day without everyone yelling at them, so we probably shouldn't do that in production [09:10] Not worth it for me, I don't think. YMMV on your connection and your end of the world. [09:11] Simple is beautiful. [09:46] rvba: I think I spotted KVMFixture.ip_address being called multiple times in a test somewhere... Shouldn't we just make start() say self.ip_address = [...]? [09:46] Not a high-priority item. :) [09:48] jtv: that's an optimization we can do indeed. Like you said, not really urgent. [09:53] rvba: didn't you find a package a while back with little utilities like "wait until file exists"? [09:55] jtv: I think you're think about http://blog.dustinkirkland.com/2013/09/introducing-run-one-constantly-run-one.html [09:55] thinking* [09:55] Yes, that's it, thanks. [09:55] There just has to be a tool like this in there. [09:56] Argh, no, I can't use that. We can't apt-get install yet at that point. === freeflying is now known as freeflying_away === freeflying_away is now known as freeflying [13:48] maas profile nodes list shows some nodes has two ip address, would it be a issue === freeflying is now known as freeflying_away === freeflying_away is now known as freeflying [14:47] freeflying: it might be, IIRC. It shouldn't really happen. [14:47] jtv, it did happen here :) [14:47] roaksoax: hi — any idea about this installation error in a VM? [14:47] /tmp/maas-region-controller.config.51653: 35: /tmp/maas-region-controller.config.51653: dbc_go: not found [14:49] roaksoax: that's saucy/amd64 [14:49] jtv, "ip_addresses": [ [14:49] "10.213.0.108", [14:49] "10.213.0.111" [14:49] ], [14:50] If MAAS manages DHCP, leases never expire... In principle (there's another weasel term) a node should always keep the same IP address on the same interface. [14:50] But sometimes, it seems... [14:53] jtv, in our case, we have 1 bond nics has 2 slaves, due to some reason, the mac of bond could be either of the 2 slaves [14:56] jtv, led maas has 2 ip for this nodes [14:58] freeflying: if the bonded interface arbitrarily picks the MAC of one of the two underlying interfaces, I guess after a few boots they'd both get IP addresses and things could get confused... Maybe the real problem is the unpredictability of how these two interfaces come up. [14:59] I need to leave now, I'm afraid... It sounds as if the problem may be at a very low level, so consider asking in #ubuntu-server. [14:59] jtv, thats true, for example, in bond 1, when fail over happens, then the mac will be changed [15:06] rvba: oh, you can put strings on separate lines and they get concatenated? I didn't know that - thanks! [15:06] rbasak: the operator % takes priority over +. [15:09] rvba: I usually stick brackets in. I didn't realise I could just drop the + [15:09] jtv: yeah, but that's nothing that you should worry about [15:09] jtv: that does not cause an install error nor prevent maas from installing [15:10] Can I merge directly from the MP in Launchpad somehow? Or do I need to do it locally and push? [15:13] rbasak: unless you have a lander like tarmac set up, then I think your only option is to merge locally and push. [15:13] OK, thanks. [15:13] Can you tell I'm new to this? :) [15:13] ;) [15:21] rbasak: does this error ring a bell: "libvirt.libvirtError: xml in virDomainDefineXML must not be NULL" I'm running this:http://paste.ubuntu.com/6405809/ [15:21] rvba: no. I've not seen that before. [15:22] * rbasak hopes he hasn't broken something recently [15:22] I need tests :-/ [15:22] I think it's related to my machine… I can't reproduce it on canonistack. [15:22] I think I fucked something up locally :/. [15:23] rvba: can you drop into pdb on line 322 and print xml? Or is that what you're doing already? [15:23] rvba: also type(xml) please [15:23] (just as I'm paranoid) [15:24] rbasak: what I was printing was the result of compose_domain_xml(). [15:24] rbasak: let me remove all the packages and start over first… [15:28] rbasak: I re-installed everything and now it works… I probably messed up the template… sorry for the noise. [15:28] np === Spideyman is now known as Spideyman_afk [16:02] If I get a "Tests Failed" status, where can I find what exactly failed? [16:03] The console output isn't clear (I have some IPMI errors, but that should be expected, I don't have working IPMI) [16:40] Hey folks, I installed MAAS on 12.04 from the havana cloud-archive but am getting some django errors ("ImportError: No module named simple") when hitting the MAAS webservice. Googling around suggests that something is relying on some depricated functionality in django (http://stackoverflow.com/questions/15085626/no-module-named-simple-login-required-depreciated) but I'm kinda stuck after that point [16:40] It's a fresh install that's up-to-date on packages... is there anything obvious I might be missing? [16:49] fader, did you add the cloud tools archive? I think if you only added the cloud-archive:havana it's still pulling the MAAS packages from the main archive, not the cloud tools one. [16:50] matsubara: Aha, that sounds like it could be my issue. I'll give it a shot, thanks! [16:51] (As a side note, it would be nice if these things were on the MAAS install docs page. I went with the cloud archive as I'd heard that the main archive isn't getting updated, but didn't realize the cloud tools archive was separate.) [16:51] fader: the havana cloud archive pocket is not compatible with the cloud-tools pocket, AIUI. You can only have one. [16:52] Hmm... so is there a working way to get MAAS up and running and deploy havana to it? [16:53] Yes, you can do that. [16:53] Your MAAS installation will use the cloud-tools pocket [16:53] Your deployed nodes running Openstack components will use the havana pocket. [16:53] Just not both on the same node [16:54] Ah, gotcha. I thought the MAAS server would cache and serve up packages based on its configuration === Spideyman is now known as Spideyman_afk [17:14] So where do commisioning/enlsitment scripts come from in the epimerial images? === racedo` is now known as racedo === Spideyman is now known as Spideyman_afk === Spideyman is now known as Spideyman_afk === Spideyman_afk is now known as Spideyman === Spideyman is now known as Spideyman_afk === Spideyman is now known as Spideyman_afk === Spideyman_afk is now known as Spideyman [23:05] NCommander: they are in /etc/maas/ [23:05] not in the ephemerals === CyberJacob is now known as CyberJacob|Away === freeflying is now known as freeflying_away === freeflying_away is now known as freeflying