| jtv | roaksoax: you still have that 1.4 branch waiting to land... should I land it? | 01:50 |
|---|---|---|
| jtv | There were some review changes to be made... don't know if you made them. | 01:51 |
| === freeflying is now known as freeflying_away | ||
| === freeflying_away is now known as freeflying | ||
| === vertigo is now known as Guest50260 | ||
| jtv | bigjools: could you review this one for me? https://code.launchpad.net/~jtv/maas-test/fix-setup-error-reporting/+merge/194781 | 04:51 |
| bigjools | jtv: sure | 04:54 |
| jtv | thx | 04:54 |
| bigjools | done | 04:56 |
| * bigjools goes back to CSS | 04:56 | |
| jtv | ...UI work? | 04:57 |
| bigjools | aye | 04:57 |
| jtv | Not the Canadian Secret Service then. | 04:57 |
| bigjools | will show you shortly | 04:57 |
| bigjools | haha | 04:57 |
| jtv | (If they're so secret, how come they're in the GTF, huh?) | 04:57 |
| bigjools | you have SIS already? | 04:57 |
| jtv | Which one? | 04:58 |
| jtv | 11 entries so far. | 04:58 |
| bigjools | Secret Intelligence Service | 04:58 |
| jtv | Yup, got that. | 04:58 |
| bigjools | splendid | 04:59 |
| bigjools | jtv: as promised: https://code.launchpad.net/~julian-edwards/maas/warning-no-power/+merge/194787 | 05:26 |
| bigjools | Azendale: still there? | 05:26 |
| Azendale | bigjools: still here | 05:26 |
| Azendale | bigjools: give me a minute to finish the project I'm in | 05:27 |
| bigjools | Azendale: I have been thinking about your problem and I think you might want to find some virtualisation experts. I can't see anything wrong with any of the maas logs you showed me so I'm mostly convinced there's some error in the networking over virtio | 05:27 |
| Azendale | bigjools: TBH, I'm not sure where to start on that route. But I can try. I'm thinking I could try breaking it, and then running a packet capture on the MaaS server. If it's still sending and receiving packets, I would think there would be something up with the software. I think I saw that once trying to figure it out myself, but I would like to be sure of that before saying that is the case | 05:34 |
| bigjools | Azendale: the reason I am sceptical it's maas is because we have a physical lab of QA machines that get fired up in tandem and they are fiine. | 05:35 |
| Azendale | bigjools: The other thing I could think of is setting up an LXC container on my machine, setting up MaaS inside it, and then bridging it's network to virtual machines. I could also try not using virtio | 05:37 |
| bigjools | all worthwhile, yeah | 05:38 |
| Azendale | bigjools: That does make it seem more like a virtualization thing then, though. I guess I'll keep expirimenting around | 05:38 |
| bigjools | Azendale: also, the fact that it fails at the DHCP level means there's no maas code involved at that point, and I don't believe that isc-dhcp-server would randomly give up under load, it's pretty solid | 05:39 |
| Azendale | bigjools: I do agree there, that is pretty widely used. Well, thanks for taking a look at it, I'll see what I can find. I'll probably be back with questions though (if I find something weird that I can't figure out) | 05:42 |
| bigjools | Azendale: sure, please let me know what you find, I am very interested. | 05:43 |
| bigjools | I have vague recollections of virtual networking being weird like this under load, but could be wrong. | 05:43 |
| Azendale | bigjools: definitely possible. Though I have seen stuff pulling during the installs from the package cache I have on the host machine at 30 MiB/s. Maybe it's just UDP stuff | 05:46 |
| bigjools | could well be | 05:46 |
| Azendale | bigjools: ok, well thanks for the help, I'll let you get back to whatever you were doing before I figured out some weird way to break stuff :) | 05:47 |
| bigjools | Azendale: heh no problem | 05:48 |
| jtv | bigjools: would also appreciate your opinion on a change I'd like to make... time for a hangout? | 06:53 |
| === CyberJacob|Away is now known as CyberJacob | ||
| bigjools | jtv: yes | 07:17 |
| jtv | Ahhh — when you fire up a VM with uvtool, it does an apt-get update. If that isn't finished yet by the time we start installing maas, things break horribly. | 08:19 |
| jtv | Looks like we do need proper waiting (uvt-kvm wait, followed by polling for /var/lib/cloud/instance/boot-finished). | 08:20 |
| rvba | jtv: You're right, we need to wait until cloud-init finishes. | 08:25 |
| jtv | In practice it may depend on the speed you get from your mirror, but for me, that means it doesn't. | 08:25 |
| jtv | Shame that that can't be fast, but oh well. | 08:25 |
| rbasak | jtv: currently it installs avahi-daemon, but I want to drop that. But yes - we need to wait for boot-finished in the general case. | 08:56 |
| rbasak | (and eventually uvt-kvm wait should incorporate that) | 08:57 |
| jtv | It's not so much the avahi-daemon that bothers me, it's the maas. | 08:57 |
| rbasak | For speed? | 08:57 |
| jtv | No, to avoid failure. | 08:57 |
| rbasak | I meant that the race is a consequence of installing avahi-daemon. Without that, cloud-init won't hit apt-get on boot. | 08:57 |
| jtv | It looks as if, before the "apt-get update" completes, the VM has as an apt database that doesn't match what's on the local mirror. | 08:58 |
| jtv | So I think if we ditch avahi-daemon, we'll have to do an apt-get update anyway. | 08:58 |
| rbasak | Ah, I see. Yes, that's true. | 08:58 |
| jtv | But it'll be easier than polling. | 08:58 |
| jtv | Because we'll be running it synchronously to our ssh session. | 08:59 |
| jtv | A polling loop doesn't sound like much, but it needs testing etc. | 08:59 |
| rbasak | A daily image might be closer to the apt mirror than the "release" one. I'm not sure using that is a good idea for production stuff though. | 08:59 |
| rbasak | When you sync, you can override the mirror to http://cloud-images.ubuntu.com/daily/ I think. | 08:59 |
| jtv | What was that about saving time? :) | 09:00 |
| jtv | For all I know, the image's built-in apt db might have more recent versions that aren't on the mirror yet... | 09:00 |
| jtv | Or the other way around. So yes, would be nice to get it all off the same mirror. | 09:00 |
| rbasak | You could also prep an image and get uvt-kvm to use that instead of a pristine one | 09:01 |
| rbasak | --backing-image-file | 09:01 |
| rbasak | Make sure to remove /var/lib/cloud/instance/boot-finished manually if you do that and want to poll for it though. | 09:01 |
| jtv | Not a worry if we do it synchronously, I guess. | 09:02 |
| rbasak | mount-image-callback is handy for manually prepping the image if you want to do that. | 09:02 |
| jtv | Not just now, to be honest — that's optimisation. | 09:02 |
| rbasak | (or you can use a VM) | 09:02 |
| jtv | It actually working sort of takes priority. :) | 09:02 |
| rbasak | :) | 09:03 |
| rbasak | Though if your development iteration will be faster, it might be worth it | 09:03 |
| jtv | Yes, it will help. But if it takes longer to build, that also affects our development cycle. | 09:04 |
| jtv | So bit of a trade-off. | 09:04 |
| jtv | So... two simple alternatives: | 09:05 |
| jtv | 1. Wait for boot to complete, and keep doing the apt-get update even when we ditch avahi-daemon. | 09:05 |
| jtv | 2. Always use the central archive. | 09:05 |
| jtv | And #2 needs daily uvt images, right? | 09:06 |
| jtv | To be reliable? | 09:06 |
| rbasak | To be reliable, I think I'd recommend always running apt-get update | 09:08 |
| jtv | It does seem like the most robust option. | 09:08 |
| jtv | So I'll just do the waiting. | 09:09 |
| rbasak | It might be safe to try apt-get --download-only install first, and if that fails to download then apt-get update first before retry, and then apt-get install | 09:09 |
| rbasak | Doing that with a daily might give you a speedup | 09:09 |
| jtv | Or a slowdown I guess, if the downloads are out of date. Is it worth it? | 09:10 |
| rbasak | However, I presume the purpose of a daily is that Scott or Ben can break it for a day without everyone yelling at them, so we probably shouldn't do that in production | 09:10 |
| rbasak | Not worth it for me, I don't think. YMMV on your connection and your end of the world. | 09:10 |
| jtv | Simple is beautiful. | 09:11 |
| jtv | rvba: I think I spotted KVMFixture.ip_address being called multiple times in a test somewhere... Shouldn't we just make start() say self.ip_address = [...]? | 09:46 |
| jtv | Not a high-priority item. :) | 09:46 |
| rvba | jtv: that's an optimization we can do indeed. Like you said, not really urgent. | 09:48 |
| jtv | rvba: didn't you find a package a while back with little utilities like "wait until file exists"? | 09:53 |
| rvba | jtv: I think you're think about http://blog.dustinkirkland.com/2013/09/introducing-run-one-constantly-run-one.html | 09:55 |
| rvba | thinking* | 09:55 |
| jtv | Yes, that's it, thanks. | 09:55 |
| jtv | There just has to be a tool like this in there. | 09:55 |
| jtv | Argh, no, I can't use that. We can't apt-get install yet at that point. | 09:56 |
| === freeflying is now known as freeflying_away | ||
| === freeflying_away is now known as freeflying | ||
| freeflying | maas profile nodes list shows some nodes has two ip address, would it be a issue | 13:48 |
| === freeflying is now known as freeflying_away | ||
| === freeflying_away is now known as freeflying | ||
| jtv | freeflying: it might be, IIRC. It shouldn't really happen. | 14:47 |
| freeflying | jtv, it did happen here :) | 14:47 |
| jtv | roaksoax: hi — any idea about this installation error in a VM? | 14:47 |
| jtv | /tmp/maas-region-controller.config.51653: 35: /tmp/maas-region-controller.config.51653: dbc_go: not found | 14:47 |
| jtv | roaksoax: that's saucy/amd64 | 14:49 |
| freeflying | jtv, "ip_addresses": [ | 14:49 |
| freeflying | "10.213.0.108", | 14:49 |
| freeflying | "10.213.0.111" | 14:49 |
| freeflying | ], | 14:49 |
| jtv | If MAAS manages DHCP, leases never expire... In principle (there's another weasel term) a node should always keep the same IP address on the same interface. | 14:50 |
| jtv | But sometimes, it seems... | 14:50 |
| freeflying | jtv, in our case, we have 1 bond nics has 2 slaves, due to some reason, the mac of bond could be either of the 2 slaves | 14:53 |
| freeflying | jtv, led maas has 2 ip for this nodes | 14:56 |
| jtv | freeflying: if the bonded interface arbitrarily picks the MAC of one of the two underlying interfaces, I guess after a few boots they'd both get IP addresses and things could get confused... Maybe the real problem is the unpredictability of how these two interfaces come up. | 14:58 |
| jtv | I need to leave now, I'm afraid... It sounds as if the problem may be at a very low level, so consider asking in #ubuntu-server. | 14:59 |
| freeflying | jtv, thats true, for example, in bond 1, when fail over happens, then the mac will be changed | 14:59 |
| rbasak | rvba: oh, you can put strings on separate lines and they get concatenated? I didn't know that - thanks! | 15:06 |
| rvba | rbasak: the operator % takes priority over +. | 15:06 |
| rbasak | rvba: I usually stick brackets in. I didn't realise I could just drop the + | 15:09 |
| roaksoax | jtv: yeah, but that's nothing that you should worry about | 15:09 |
| roaksoax | jtv: that does not cause an install error nor prevent maas from installing | 15:09 |
| rbasak | Can I merge directly from the MP in Launchpad somehow? Or do I need to do it locally and push? | 15:10 |
| rvba | rbasak: unless you have a lander like tarmac set up, then I think your only option is to merge locally and push. | 15:13 |
| rbasak | OK, thanks. | 15:13 |
| rbasak | Can you tell I'm new to this? :) | 15:13 |
| rvba | ;) | 15:13 |
| rvba | rbasak: does this error ring a bell: "libvirt.libvirtError: xml in virDomainDefineXML must not be NULL" I'm running this:http://paste.ubuntu.com/6405809/ | 15:21 |
| rbasak | rvba: no. I've not seen that before. | 15:21 |
| * rbasak hopes he hasn't broken something recently | 15:22 | |
| rbasak | I need tests :-/ | 15:22 |
| rvba | I think it's related to my machine… I can't reproduce it on canonistack. | 15:22 |
| rvba | I think I fucked something up locally :/. | 15:22 |
| rbasak | rvba: can you drop into pdb on line 322 and print xml? Or is that what you're doing already? | 15:23 |
| rbasak | rvba: also type(xml) please | 15:23 |
| rbasak | (just as I'm paranoid) | 15:23 |
| rvba | rbasak: what I was printing was the result of compose_domain_xml(). | 15:24 |
| rvba | rbasak: let me remove all the packages and start over first… | 15:24 |
| rvba | rbasak: I re-installed everything and now it works… I probably messed up the template… sorry for the noise. | 15:28 |
| rbasak | np | 15:28 |
| === Spideyman is now known as Spideyman_afk | ||
| NCommander | If I get a "Tests Failed" status, where can I find what exactly failed? | 16:02 |
| NCommander | The console output isn't clear (I have some IPMI errors, but that should be expected, I don't have working IPMI) | 16:03 |
| fader | Hey folks, I installed MAAS on 12.04 from the havana cloud-archive but am getting some django errors ("ImportError: No module named simple") when hitting the MAAS webservice. Googling around suggests that something is relying on some depricated functionality in django (http://stackoverflow.com/questions/15085626/no-module-named-simple-login-required-depreciated) but I'm kinda stuck after that point | 16:40 |
| fader | It's a fresh install that's up-to-date on packages... is there anything obvious I might be missing? | 16:40 |
| matsubara | fader, did you add the cloud tools archive? I think if you only added the cloud-archive:havana it's still pulling the MAAS packages from the main archive, not the cloud tools one. | 16:49 |
| fader | matsubara: Aha, that sounds like it could be my issue. I'll give it a shot, thanks! | 16:50 |
| fader | (As a side note, it would be nice if these things were on the MAAS install docs page. I went with the cloud archive as I'd heard that the main archive isn't getting updated, but didn't realize the cloud tools archive was separate.) | 16:51 |
| rbasak | fader: the havana cloud archive pocket is not compatible with the cloud-tools pocket, AIUI. You can only have one. | 16:51 |
| fader | Hmm... so is there a working way to get MAAS up and running and deploy havana to it? | 16:52 |
| rbasak | Yes, you can do that. | 16:53 |
| rbasak | Your MAAS installation will use the cloud-tools pocket | 16:53 |
| rbasak | Your deployed nodes running Openstack components will use the havana pocket. | 16:53 |
| rbasak | Just not both on the same node | 16:53 |
| fader | Ah, gotcha. I thought the MAAS server would cache and serve up packages based on its configuration | 16:54 |
| === Spideyman is now known as Spideyman_afk | ||
| NCommander | So where do commisioning/enlsitment scripts come from in the epimerial images? | 17:14 |
| === racedo` is now known as racedo | ||
| === Spideyman is now known as Spideyman_afk | ||
| === Spideyman is now known as Spideyman_afk | ||
| === Spideyman_afk is now known as Spideyman | ||
| === Spideyman is now known as Spideyman_afk | ||
| === Spideyman is now known as Spideyman_afk | ||
| === Spideyman_afk is now known as Spideyman | ||
| bigjools | NCommander: they are in /etc/maas/ | 23:05 |
| bigjools | not in the ephemerals | 23:05 |
| === CyberJacob is now known as CyberJacob|Away | ||
| === freeflying is now known as freeflying_away | ||
| === freeflying_away is now known as freeflying | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!