| bigjools | jtv: any ideas what's going on here in maas-test? http://paste.ubuntu.com/6712744/ | 03:21 |
|---|---|---|
| jtv | bigjools: no, but can you create a VM with uvtool on that same machine? | 03:27 |
| bigjools | jtv: gimme a command line for that please | 03:27 |
| jtv | May be best to start with the command line that's in the traceback, and whittle it down to find the source of the problem: | 03:28 |
| bigjools | uvtool: command not found | 03:28 |
| jtv | The command is uvt-kvm. Hang on, I'm about to paste. | 03:28 |
| jtv | Try: | 03:28 |
| jtv | sudo uvt-kvm create --ssh-public-key-file=/home/ubuntu/.config/maas-test/vm_ssh_id_rsa.pub --unsafe-caching --memory 2047 --disk 20 c3dd8de0-7810-11e3-944d-e4115b13819f arch=amd64 release=saucy | 03:28 |
| jtv | The "sudo" shouldn't be needed, so that's one of the parts you can whittle off to find the minimal reproduction of the problem. | 03:29 |
| jtv | I find that command line works for me, after a fresh sync of images, but there may still be a problem in the template. | 03:29 |
| bigjools | jtv: it seems to be working ... | 03:30 |
| bigjools | yay? | 03:30 |
| jtv | Same here. Let me try maas-test here. | 03:30 |
| jtv | You're running this on saucy i386? | 03:30 |
| bigjools | amd64 | 03:31 |
| jtv | By the way, clean up the VM afterwards using "uvt-kvm destroy <name>" | 03:31 |
| jtv | The name in this case is c3dd8de0-7810-11e3-944d-e4115b13819f | 03:31 |
| jtv | Feel free to use more convenient names... | 03:31 |
| bigjools | it fails quickly in maas-test | 03:31 |
| bigjools | libvirt: QEMU Driver error : Domain not found: no domain with matching name 'c3dd8de0-7810-11e3-944d-e4115b13819f' | 03:31 |
| bigjools | so it didn't get near to making it | 03:32 |
| jtv | Try a "uvt-kvm list" to see which VMs have been defined. | 03:32 |
| bigjools | empty | 03:32 |
| jtv | OK, nothing to clean up then. But then how and why did that command I pasted earlier seem to work!? | 03:33 |
| bigjools | :/ | 03:33 |
| bigjools | libvirt.libvirtError: unknown OS type hvm | 03:34 |
| bigjools | is the error | 03:34 |
| jtv | Yes, saw that. | 03:34 |
| jtv | Hmm... the template mentions that. | 03:34 |
| jtv | My guess is that nomenclature has changed, or this "hvm" OS type has been replaced, and we need to update the code. | 03:35 |
| jtv | Looks like a question for rbasak. | 03:36 |
| jtv | The one difference between the command line I pasted you and the one maas-test actually runs, is that maas-test passes in a "template" of settings. | 03:36 |
| jtv | The template is in XML. The part that seems to be causing trouble now is: | 03:37 |
| jtv | <domain type='kvm'> | 03:37 |
| jtv | <os> | 03:37 |
| jtv | <type>hvm</type> | 03:37 |
| jtv | (Can you really officially use single quotes for XML attribute values?) | 03:37 |
| bigjools | I am running on saucy BTW | 03:39 |
| bigjools | oh said that | 03:39 |
| jtv | Graham made a change to that one line recently, so he may know. | 03:40 |
| bigjools | the "uvt-kvm create" is still running | 03:40 |
| jtv | Ah | 03:40 |
| bigjools | but nothing for "list" coming up | 03:40 |
| jtv | That's taking pretty long. | 03:40 |
| bigjools | this is my command, not maas-test | 03:40 |
| jtv | You can run "uvt-kvm wait <name>" to block until it's created. | 03:40 |
| bigjools | it fails instantly on maas-test | 03:40 |
| jtv | Yes, because we pass an unknown "OS type" (whatever that means in this context) through that template. | 03:42 |
| jtv | I'm zeroing in on the branch that made that change. | 03:42 |
| bigjools | jtv: ok thanks. I am deliberately staying out of code for this so I can experience it as a normal user and make informed feedback :) | 03:45 |
| jtv | bigjools: not much luck identifying a change there, but other things I'm finding on the intertubes suggest that it may be libvirt's horrible way of saying "this hardware does not support hvm." | 03:49 |
| jtv | Where according to the GTF, HVM means Hybrid Virtual Machine. | 03:49 |
| bigjools | jtv: it';s running on my microserver, I can run other VMs on it fine | 03:50 |
| bigjools | all hail the GPL TLA FAQ | 03:50 |
| bigjools | let me look at cpuinfo | 03:50 |
| jtv | The problem may be that we specify a particular *type* of VM which requires absent hardware support. | 03:50 |
| bigjools | ah there's no hvm flag in /proc/cpuinfo | 03:51 |
| bigjools | fuck | 03:51 |
| bigjools | this is a bad error then, it needs to fail better | 03:51 |
| jtv | I don't have it here either. | 03:51 |
| bigjools | and your maas-test fails? | 03:51 |
| jtv | Do we know that the CPU capability would actually read "hvm"? | 03:51 |
| jtv | I'll try it. | 03:52 |
| bigjools | good question | 03:52 |
| jtv | "hvm" may be too obvious and hard to trademark, so it's probably something like "spx" | 03:52 |
| jtv | or "ghvl" | 03:52 |
| bigjools | vmx I think | 03:52 |
| jtv | Whaaaa. Shouldn't have grepped for that... so much output! | 03:53 |
| jtv | And thanks to whoever committed lint so that "make" now fails... | 03:53 |
| bigjools | ah it's svm on AMD | 03:54 |
| bigjools | vmx on intel | 03:54 |
| bigjools | so I am ok, I have svm | 03:54 |
| jtv | Damn, can't run on my dev machine. Will try laptop. | 03:55 |
| jtv | Okay, that's syncing images. I'll leave it to do that while I go fix my mouse. | 03:56 |
| bigjools | I'll check my bios | 03:57 |
| bigjools | bios is ok | 04:00 |
| bigjools | jtv: found the problem | 04:03 |
| bigjools | I rebooted and it worked after | 04:03 |
| bigjools | I suspect there's some kernel stuff missing after installing kvm packages | 04:04 |
| bigjools | we need to work out what it is | 04:04 |
| bigjools | jtv: also I was pending a reboot after installing a new kernel. That, I expect, buggered it. | 04:05 |
| jtv | bigjools: think we should have a maas-test section on maas.ubuntu.com where we can provide advice about problems like the one you just ran into? | 07:06 |
| bigjools | jtv: mmmm maybe | 07:06 |
| bigjools | I have more problems | 07:06 |
| bigjools | my node is not enlisting | 07:07 |
| bigjools | no idea why | 07:07 |
| bigjools | output is useless :/ | 07:07 |
| jtv | No errors logged? | 07:08 |
| bigjools | sending email | 07:08 |
| jtv | bigjools: I guess you saw this post about the hvm problem... http://blog.loftninjas.org/2009/05/04/libvirt-unknown-os-type-hvm/ | 07:09 |
| bigjools | I did | 07:12 |
| bigjools | email sent | 07:12 |
| jtv | Did we change how we generate our manpage? Suddenly there's loads of errors from the man page. | 07:17 |
| bigjools | gmb: can you look quickly at the email I just sent as well please? It looks pretty bad. | 07:18 |
| gmb | bigjools: Yowza. | 07:18 |
| gmb | Yeah, that's extremely very not good. | 07:18 |
| gmb | bigjools: If there's a log at all, it _should_ be under ~/.maas-test/logs. | 07:19 |
| gmb | Anyway, I'll reply to the email presently. | 07:20 |
| bigjools | gmb: not logs dir at all | 07:20 |
| jtv | I'm fixing up some of the superficial problems. | 07:20 |
| bigjools | no* | 07:20 |
| jtv | Mainly to get "make" back to work. | 07:20 |
| gmb | Um, that's very weird indeed. | 07:20 |
| bigjools | I wish I'd tested this earlier :/ | 07:20 |
| bigjools | but *stuff* | 07:20 |
| jtv | gmb: looks like you created some lint in the code, and accidentally pasted an extraneous heading into the man page's source. | 07:21 |
| gmb | jtv: Which revision? | 07:21 |
| gmb | I thought I'd fixed that. | 07:21 |
| gmb | (You picked it up in a review, IIRC) | 07:21 |
| jtv | gmb: r100 | 07:22 |
| gmb | Damn. | 07:22 |
| * bigjools filing bugs | 07:23 | |
| bigjools | I have enbuggerated | 07:38 |
| jtv | And I'm putting up a first branch. | 07:38 |
| bigjools | happy to engage review mode | 07:40 |
| jtv | bigjools: engage! https://code.launchpad.net/~jtv/maas-test/housekeeping/+merge/200776 | 07:41 |
| gmb | jtv: Interesting that make test didn't work for you in trunk (and almost definitely my fault has I had to manually land some things the other day). I don't see that behaviour, however. | 07:43 |
| jtv | gmb: could it be that you did a "make test" at some point shortly after a successful sudo authentication? | 07:44 |
| jtv | Because it seems to boil down to it writing into /usr/local/lib/python2.7/dist-packages/... | 07:45 |
| gmb | jtv: Aaaaah. Yes, testing it in a new byobu window and I hit it. | 07:45 |
| gmb | That's... suboptimal. | 07:45 |
| jtv | I'm filing. | 07:45 |
| bigjools | jtv: approved (via email, give it a minute) | 07:45 |
| jtv | Thanks. | 07:45 |
| gmb | jtv: Oh, actually, no, thats something else. | 07:46 |
| jtv | gmb: bug filed — https://bugs.launchpad.net/maas-test/+bug/1267004 | 07:48 |
| ubot5 | Ubuntu bug 1267004 in maas-test "Test suite writes into /usr/local/lib/python2.7/dist-packages/" [Critical,Triaged] | 07:48 |
| bigjools | !!! | 07:48 |
| gmb | I'm not even clear how the sweet fuck that happened. | 07:48 |
| gmb | And I can't reproduce it. | 07:48 |
| gmb | Hmm, unless... | 07:49 |
| jtv | Unless you already have that directory. | 07:49 |
| gmb | Right. | 07:50 |
| jtv | Which would have happened if you ever ran this as root. | 07:50 |
| gmb | Which I may have done(?!) | 07:50 |
| jtv | And with all the sudo'ing we do, I'm suspicious even of running "make check" while a previous sudo authentication is still valid. | 07:50 |
| jtv | See if you have a /usr/local/lib/python2.7/dist-packages/maas-test. | 07:51 |
| jtv | I don't. | 07:51 |
| gmb | jtv: I don't. | 07:52 |
| jtv | Interesting. | 07:52 |
| gmb | jtv: But this is not with a completely clean trunk (i.e. I've run make in this directory before and its worked, presumably doing pip and virtualenv-y stuff) | 07:53 |
| jtv | Did you do a "make clean," or run this in a fresh branch? | 07:53 |
| gmb | I ran a make clean. | 07:53 |
| gmb | Or did I? | 07:53 |
| gmb | hang on. | 07:53 |
| jtv | For me, it worked in trunk *until* I did the "make clean." | 07:53 |
| jtv | Leftovers from an older build. | 07:53 |
| gmb | Hmm. | 07:53 |
| * gmb is writing on one machine, testing on another | 07:53 | |
| gmb | jtv: Nope, I don't see that behaviour on my dev machine. | 07:55 |
| gmb | make clean | 07:55 |
| gmb | make test | 07:55 |
| * bigjools doing another test run with the node console in view | 07:55 | |
| gmb | Just seems to work. | 07:56 |
| jtv | I wonder if it's a matter of installing some package that isn't in the requirements list. | 07:56 |
| jtv | gmb: do you have python-tox installed? | 07:57 |
| bigjools | PROTIP: when you press ctrl-alt-delete make sure the keyboard is plugged into the machine you think it is | 08:00 |
| gmb | jtv: Nope. | 08:00 |
| gmb | jtv: Huh, now I'm getting failures. | 08:03 |
| gmb | Wonder what changed. | 08:03 |
| gmb | s/failures/errors | 08:03 |
| * gmb goes to brew more coffee | 08:06 | |
| bigjools | jtv: dhcp detection failed to pick up my existing dhcp server :( | 08:06 |
| jtv | Damn. It picked up mine. | 08:06 |
| jtv | ...Was the network cable plugged into the machine it was? :) | 08:06 |
| bigjools | I'm running maas-test on my maas server and had forgotten to kill the maas-dhcp-server | 08:06 |
| jtv | How repeatable is it? | 08:07 |
| bigjools | I suspect that's a mode that's not catered for | 08:07 |
| bigjools | every time | 08:07 |
| jtv | Is the problem that the DCHP server is running on the same machine as maas-test? | 08:09 |
| bigjools | it's on the same machine | 08:09 |
| bigjools | not sure why that's a problem | 08:09 |
| bigjools | or the problem | 08:10 |
| jtv | dhcpd might ignore discovery requests coming from an interface that it's serving on itself. | 08:11 |
| bigjools | food | 08:15 |
| gmb | jtv: Can you review https://code.launchpad.net/~gmb/maas-test/fix-test-blowup/+merge/200783 | 08:44 |
| rvba | gmb: hang on, why would we need tox to be installed as a package? | 08:46 |
| rvba | It's something that is only used in tests. | 08:46 |
| gmb | rvba: I can't give a better reason than "because it fixes https://bugs.launchpad.net/maas-test/+bug/1267004" | 08:47 |
| ubot5 | Ubuntu bug 1267004 in maas-test "Test suite writes into /usr/local/lib/python2.7/dist-packages/" [Critical,In progress] | 08:47 |
| gmb | rvba: If there's a way to fix that without installing Tox, I'm all for it. | 08:47 |
| rvba | gmb: I just ran the test suite a canonistack instance and didn't see any error. | 08:47 |
| rvba | on* a canonistack instance | 08:47 |
| rvba | I wonder what I'm missing. | 08:47 |
| gmb | rvba: Was this in an absolutely clean trunk? | 08:47 |
| rvba | Yeah | 08:48 |
| gmb | Hrm. | 08:48 |
| gmb | rvba: When the test is running on the canonistack instance, does the user have a leftover sudo session? | 08:49 |
| rvba | Let me check… | 08:49 |
| rvba | A leftover sudo session from running "make install-dependencies" I suppose? | 08:50 |
| gmb | Yes | 08:50 |
| * rvba runs "make install-dependencies" in a separate shell this time… | 08:50 | |
| gmb | rvba: sudo -k should kill any existing sessions | 08:51 |
| gmb | and sudo -K _really really_ kills them | 08:51 |
| rvba | I just used two completely separate shells. | 08:51 |
| rvba | No, the tests passed fine again. | 08:51 |
| gmb | Um. | 08:52 |
| bigjools | o/ rvba | 08:52 |
| rvba | gmb: care to watch me? ssh ubuntu@10.55.60.177 | 08:52 |
| gmb | Sure | 08:52 |
| rvba | \o bigjools | 08:53 |
| gmb | rvba: U;n ub,] | 08:53 |
| gmb | Er | 08:53 |
| gmb | I'm in | 08:53 |
| gmb | FencePostError | 08:53 |
| gmb | rvba: Ok, I'm officially confused. | 08:56 |
| rvba | heh | 08:56 |
| gmb | rvba: Jeroen spotted this first, maybe he knows something more about reproducing it. | 08:56 |
| * gmb waits for thailand to come back online | 08:57 | |
| gmb | Ah, jtv. Just the man! | 09:01 |
| gmb | jtv: So, rvba (and I) can't reproduce https://bugs.launchpad.net/maas-test/+bug/1267004 in a canonistack instance; could you lend your brain to help us find what we're missing? | 09:02 |
| ubot5 | Ubuntu bug 1267004 in maas-test "Test suite writes into /usr/local/lib/python2.7/dist-packages/" [Critical,In progress] | 09:02 |
| jtv | gmb: absolutely. | 09:03 |
| jtv | But why a canonistack instance when we have uvtool? | 09:03 |
| gmb | rvba: ^^? | 09:03 |
| rvba | I like using someone else's resources. | 09:04 |
| gmb | :) | 09:04 |
| rvba | :) | 09:04 |
| jtv | Fair enough. | 09:04 |
| rvba | It's also useful when you want to invite someone else. | 09:05 |
| rvba | Like I did with gmb | 09:05 |
| jtv | If you don't mind I'll stick to uvtool though — saves me from having to type halfway across the world, and freezing up every time my connection blinks. | 09:05 |
| jtv | It's doing that today. | 09:05 |
| bigjools | rvba: my enlistment failed in the test | 09:06 |
| bigjools | console shows failed to find pxe config | 09:07 |
| gmb | jtv: Can you give us an exact series of steps to reproduce the problem? | 09:10 |
| jtv | gmb: I'm working on one, from a clean system. | 09:16 |
| gmb | Cool beans. | 09:16 |
| jtv | But "make install-dependencies clean test" seems to do the trick on my own system. | 09:16 |
| jtv | Systems, I should say. | 09:17 |
| jtv | I hope it's not another locale sensitivity... | 09:18 |
| === tryggvil_ is now known as tryggvil | ||
| gmb | jtv: LMK if you want to pair on the breakage stuff; I can't really go far without that fixed anyway. | 09:59 |
| gmb | *being fixed | 09:59 |
| jtv | gmb: perhaps that would be best... At this point I only have guesses, and we need some system to this. | 10:00 |
| rvba | gmb: does you upcoming fix include something to make sure the images are cached or it that something I should look into right now? | 10:13 |
| rvba | your* | 10:13 |
| rvba | s/it/is/ | 10:13 |
| rvba | rarg | 10:13 |
| gmb | rvba: That's something that's worth looking into; all I do is make sure that everything gets to use https proxies (except things that will obviously break). | 10:13 |
| rvba | I'm not sure what you mean by "make sure that everything gets to use https proxies"… is there a diff I could look at somewhere? | 10:14 |
| gmb | rvba: I'll push it for you | 10:16 |
| jtv | Well, at least the problem with the test suite has been resolved: uninstalling python-pip (and re-building the branch) made tests pass again. | 10:20 |
| rvba | gmb: btw, this bug is particularly puzzling: https://bugs.launchpad.net/maas-test/+bug/1266996. Maybe you'll have an idea what's going on. | 10:24 |
| ubot5 | Ubuntu bug 1266996 in maas-test "All test runs ending in "AttributeError: 'file' object has no attribute 'buffer'"" [Critical,Triaged] | 10:24 |
| gmb | rvba: Yes, that's an encoding error somewhere. I'll look at it in a sec. | 10:25 |
| gmb | rvba: Diff for you (includes test changes that you probably don't need): http://paste.ubuntu.com/6714063/ | 10:25 |
| rvba | Ta. | 10:25 |
| gmb | rvba: What revision are you on there? That AttributeError isn't happening for me as of r101. | 10:26 |
| rvba | No idea, that's a bug Julian filed. I guess he was using the package from the daily PPA. | 10:27 |
| bigjools | I am | 10:28 |
| gmb | rvba: Ah, no, I see what he means. I misunderstood "test run". | 10:28 |
| gmb | I'll fix that. | 10:28 |
| bigjools | 0.1+bzr58+100+8~ppa0~ubuntu13.10.1 | 10:28 |
| bigjools | rvba: ok I have a breakpoint! | 10:29 |
| * gmb grumbles about big chunks of maas-test being untested. | 10:29 | |
| rvba | gmb: about your diff: does this means that the images (downloaded over https) will be cached as well from now on? | 10:31 |
| rvba | (I think it does if the proxy is configured properly.) | 10:31 |
| bigjools | another one https://bugs.launchpad.net/maas-test/+bug/1267056 | 10:32 |
| ubot5 | Ubuntu bug 1267056 in maas-test "maas-test leaves root-owned files in user's home dir" [High,Triaged] | 10:32 |
| bigjools | rvba: is it normal to have two VMs up? | 10:33 |
| bigjools | I am guessing not :) | 10:33 |
| rvba | No | 10:34 |
| rvba | uvt-kvm list / uvt-kvm destroy <vm-name> | 10:34 |
| bigjools | ubuntu@maas:~$ ssh admin@192.168.122.144 | 10:34 |
| bigjools | ssh: connect to host 192.168.122.144 port 22: No route to host | 10:34 |
| bigjools | which one? | 10:34 |
| gmb | rvba: Is there a separate https_proxy config option for MAAS? | 10:35 |
| bigjools | gmb: no, it sets https the same as http | 10:35 |
| bigjools | iirc | 10:35 |
| rvba | gmb: no | 10:35 |
| gmb | Okay. | 10:35 |
| rvba | Like Julian said. | 10:36 |
| gmb | rvba: When you say "the images" do you mean the PXE images? | 10:36 |
| rvba | Yeah. The stuff that import_pxe_files downloads. | 10:37 |
| bigjools | rvba: so at the breakpoint I can't ssh the VM (as above) | 10:37 |
| bigjools | eth0 has not changed IP | 10:38 |
| gmb | rvba: Yes, with this, it should cache those too. | 10:38 |
| rvba | gmb: nice, it will speed up the testing considerably. | 10:38 |
| gmb | I'm testing that now. | 10:38 |
| bigjools | \o/ | 10:38 |
| rvba | bigjools: hum… | 10:38 |
| bigjools | I am going to kill it and start again .... :/ | 10:39 |
| bigjools | gmb: what's your hack? I'll test it too! | 10:39 |
| gmb | bigjools: Branch lp:~gmb/maas-test/fix-proxy-problems or diff: http://paste.ubuntu.com/6714157/ | 10:40 |
| bigjools | ta | 10:40 |
| gmb | Not so much a hack as a general fix. | 10:40 |
| * bigjools ponders patch hackery | 10:42 | |
| gmb | ASS BASTARD | 10:50 |
| jtv | ? | 10:51 |
| gmb | jtv: Problem with logging/reporting. | 10:51 |
| gmb | Fixing it... | 10:51 |
| * gmb -> tea | 11:03 | |
| rvba | Is there an easy way to get the version of the package that will be installed if you type 'apt-get install <packagename>'. I mean, 'apt-cache policy <packagename>' gives you that information but it contains a lot of other stuff… is there a way to just get the version (other than parsing the return of 'apt-cache policy')? | 11:08 |
| rbasak | bigjools: looks like you found the problem. uvt-kvm does try to check and print a helpful message, but it relies on kvm-ok which apparently doesn't always do it. https://bugs.launchpad.net/uvtool/+bug/1246786 | 11:12 |
| ubot5 | Ubuntu bug 1246786 in uvtool "kvm-ok is insufficient" [Undecided,New] | 11:12 |
| bigjools | rvba: don't think there is | 11:21 |
| bigjools | unless there's a library for python | 11:21 |
| bigjools | lib to apt I mean | 11:21 |
| bigjools | rbasak: okeydoke. I just needed to reboot and it worked anyway. | 11:22 |
| rvba | Yeah, but we can't use that in maas-test. Because we would need to install python-apt on the VM, and then run a python script there. | 11:22 |
| bigjools | rvba: also having maas-test write the root-owned files means: | 11:22 |
| bigjools | Warning: Identity file /home/ubuntu/.config/maas-test/vm_ssh_id_rsa not accessible: Permission denied. | 11:22 |
| bigjools | :) | 11:22 |
| rvba | bigjools: like I said on the bug: 'sudo service libvirt-bin restart' usually fixes this. | 11:23 |
| bigjools | rvba: right, ta | 11:23 |
| bigjools | rvba: ok I am at my breakpoint and ssh'ed into the VM | 11:24 |
| bigjools | not sure what to look for at the moment, can we do a hangout | 11:24 |
| rvba | bigjools: sure | 11:24 |
| bigjools | rvba: is it stuck? | 11:26 |
| bigjools | rvba: ARGH | 11:43 |
| bigjools | "2014-01-08 21:42:59,645 INFO Network interface eth1 has no IP address. Can't scan for DHCP servers." | 11:43 |
| bigjools | bang | 11:43 |
| rvba | :/ | 11:44 |
| bigjools | how is this working elsewhere?! | 11:44 |
| rvba | I'm wondering the same thing. | 11:44 |
| rvba | Hang on. | 11:44 |
| rvba | This is just a warning. | 11:44 |
| rvba | Nothing more. | 11:44 |
| bigjools | oh oops | 11:45 |
| bigjools | Exception: Found pid file /home/ubuntu/.config/maas-test/proxy.pid for running proxy process. | 11:45 |
| bigjools | the root-owned dir is frustrating | 11:45 |
| bigjools | aaaand it was transient, the proxy was not there when I checked | 11:46 |
| bigjools | rvba: hmm when you think about it, not having an IP address is no reason to not scan for DHCP servers... :) | 11:47 |
| bigjools | jtv ^ | 11:47 |
| bigjools | but it's because of the way we do it | 11:47 |
| rvba | Yeah, that's a good point. The DHCP scanning code just can't cope with having no IP address. | 11:48 |
| rvba | But that's just a limitation of how we do that. | 11:48 |
| bigjools | gmb your diff for caching https doesn't seem to help me here :( Did it work for you? | 11:55 |
| gmb | bigjools: I thought it did, but now I'm not so sure. Let me clear the cache and retry. | 11:56 |
| bigjools | it's been going 5 minutes | 11:56 |
| gmb | Hmm. | 11:57 |
| bigjools | gmb: having said that the router is not flashing like mad as it normally does at this stage | 11:57 |
| bigjools | oh dear all my ssh sessions to it are frozen | 11:58 |
| gmb | Well, we never said that maas-test would leave you with a useable system, did we? | 11:58 |
| bigjools | verily, it says beware of destruction :) | 11:58 |
| bigjools | I wonder if the proxy has run out of memory | 11:59 |
| bigjools | it's not copying things in memory is it ... | 11:59 |
| gmb | Shouldn't be. | 11:59 |
| bigjools | oo my keypresses finally appeared | 12:00 |
| bigjools | it's swapping like buggery | 12:00 |
| bigjools | gmb: so yeah: | 12:06 |
| bigjools | 10523 libvirt- 20 0 3687m 1.7g 1452 D 29.2 89.4 5:23.11 qemu-system-x86 | 12:06 |
| bigjools | 1.7g | 12:06 |
| bigjools | ... | 12:06 |
| gmb | WAAAAT. | 12:06 |
| bigjools | I sense a problem :) | 12:06 |
| gmb | That's not the proxy, though... I hope. Unless you're committing inception. | 12:06 |
| bigjools | I keel eet | 12:06 |
| gmb | Si. | 12:07 |
| bigjools | and come back tomorrow | 12:07 |
| bigjools | since it's way-past-bed-o-clock | 12:07 |
| bigjools | I'll pick up your lovely updated packages | 12:07 |
| bigjools | I guess something inside the vm went wrong | 12:08 |
| bigjools | nn | 12:08 |
| * gmb lunches | 13:07 | |
| gmb | bigjools: (Don't know if you'll see this in your scroll back, but anyway): caching is definitely working for PXE files — with caching it takes 5mins, without, ~20mins. | 13:17 |
| rvba | gmb: my testing also shows that some sort of caching is indeed going on. | 13:41 |
| rvba | gmb: but, uploading stuff into launchpad failed for some reason :/ | 13:44 |
| rvba | gmb: did you test that successfully? | 13:44 |
| gmb | rvba: No, I'm seeing the same. I'm guessing it's to do with https proxying, confirming that now. | 14:14 |
| rvba | k | 14:14 |
| rvba | It's a bit weird though, because even if the call to LP does not go trough the proxy, it shouldn't fail. | 14:15 |
| gmb | Yeah. | 14:15 |
| gmb | rvba: maas-import-pxe-files runs maas-import-ephemerals, right? | 14:15 |
| gmb | Or am I having post-lunch brain fog? | 14:16 |
| rvba | That's correct. | 14:16 |
| rvba | (Not the post-lunch brain fog part ;)) | 14:16 |
| gmb | rvba: I'm wondering if the 5minutes for PXE import is just down to copying from the cache to the VM, or whether maas-import-ephemerals isn't getting the proxy settings. | 14:17 |
| rvba | The import script also does other stuff: it copies the images around more than it should (there is a bug for that). | 14:19 |
| gmb | Gaah. | 14:20 |
| gmb | Well, that would account for Julian's VM hitting 1.7GB resident. | 14:20 |
| rvba | gmb: I also noticed that, when the download actually happens (download from the proxy), my machine slows down, all the memory is sucked up. | 14:21 |
| gmb | Right. It's in that phase now on my test machine. I'll take a look and see what's happening... | 14:21 |
| gmb | PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND | 14:23 |
| gmb | 14898 libvirt- 20 0 2268m 1.2g 8124 S 23.4 31.7 3:00.88 qemu-system-x86 | 14:23 |
| gmb | It's certainly keeping busy. | 14:23 |
| gmb | 14898 libvirt- 20 0 2268m 2.0g 7576 S 22.4 50.9 6:14.68 qemu-system-x86 | 14:33 |
| gmb | rvba: Well, with the proxy disabled Launchpad reporting works fine. Which means that something daft is going on. | 14:42 |
| rvba | Damn. | 14:42 |
| rvba | Do we have easy access to the proxy's log? | 14:42 |
| gmb | rvba: yes, but it's largely useless. The only think that *might* be relevant is this entry: Couldn't read from client: Connection reset by peer. | 15:02 |
| gmb | I'll see if we can jack up the logging. | 15:02 |
| gmb | Yes we can. | 15:03 |
| * gmb turns it up to 11 | 15:03 | |
| gmb | Well, 0xFF, to be exact. | 15:05 |
| gmb | rvba: The log's worse than useless, even when turned up to its maximum level. Whats weird is that this used to work, so it must be something to do with my proxy-fix branch. I'll try some tweaking. | 15:21 |
| gmb | rvba: Yep, just confirmed it, disabling https_proxy makes LP reporting work again. That's odd, but not a _huge_ deal. Let me check out a couple of things; this could drastically simplify my proxy branch. | 15:36 |
| bjorne | why this happend? [21/Dec/2013:00:49:57 +0100] "GET /MAAS/metadata//2012-03-01/user-data HTTP/1.1" 404 181 "-" "python-requests/1.2.3 CPython/2.7.5+ Linux/3.11.0-12-generic" | 16:25 |
| bjorne | get only error 404 from webservern.. the cloent can't find user-data information... from server | 16:25 |
| bjorne | is that no one here? that have some information? | 16:26 |
| === CyberJacob|Away is now known as CyberJacob | ||
| === CyberJacob is now known as CyberJacob|Away | ||
| === CyberJacob|Away is now known as CyberJacob | ||
| bigjools | gmb: yeah so, weirdness. I expect there is some daft code *somwhere* that is copying files via Python memory (ie reading the whole thing in before writing it out again) | 23:18 |
| === CyberJacob is now known as CyberJacob|Away | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!