| jtv | Why isn't the packaging building bin/maas-probe-dhcp!? | 03:29 |
|---|---|---|
| jtv | bigjools: any ideas? ^ | 03:30 |
| bigjools | jtv: where should it live do you reckon? cluster? | 04:47 |
| bigjools | maas-dhcp? | 04:47 |
| jtv | I was thinking maas-dhcp. | 04:47 |
| bigjools | k | 04:47 |
| jtv | I already have a branch installing it as part of that package. The problem is getting it built. | 04:47 |
| jtv | But feel free to write up your own branch; it's not exactly a lot of diff. | 04:47 |
| bigjools | jtv: hmmm it needs to go in cluster controller | 04:52 |
| bigjools | well doesn't *have* to but makes more sense to me | 04:53 |
| bigjools | ah sod it maas-dhcp | 04:53 |
| jtv | Well I had been wondering, but you weren't around: did we have a particular purpose in mind for it? | 04:53 |
| jtv | Were we going to run it during install? Or before starting our DHCP server..? | 04:54 |
| bigjools | jtv: show me your diff | 04:58 |
| jtv | I added to debian/maas-dhcp.install: | 05:02 |
| jtv | debian/extras/maas-probe-dhcp /usr/sbin | 05:02 |
| jtv | (Or various other things I tried). | 05:03 |
| jtv | I also had a lintian-overrides, but I reverted that already. | 05:03 |
| bigjools | well you need a bit more than that | 05:03 |
| bigjools | MP coming | 05:03 |
| bigjools | jtv: https://code.launchpad.net/~julian-edwards/maas/packaging-add-maas-probe-dhcp/+merge/201538 | 05:12 |
| jtv | Thanks. | 05:12 |
| * jtv shall review forthwith | 05:12 | |
| bigjools | buildout scripts are a PITA | 05:12 |
| bigjools | jtv: it's untested! | 05:12 |
| bigjools | I figured since you were desperate ... | 05:12 |
| bigjools | I can do it if you want | 05:13 |
| jtv | I see that you managed to include bash. :) | 05:13 |
| bigjools | sadly | 05:13 |
| bigjools | cargo culted existing ones | 05:13 |
| jtv | What exactly do you mean by untested — "not covered by test suite" or "I haven't tried this yet"? | 05:13 |
| bigjools | I haven't tried it | 05:14 |
| jtv | Ah. Well, it'll sort of come naturally with my manual testing, so don't worry about it. | 05:14 |
| jtv | As long as the package builds... | 05:14 |
| bigjools | ok | 05:14 |
| bigjools | well let's do a test build first | 05:14 |
| bigjools | I'll do it on cstack | 05:15 |
| jtv | OK | 05:17 |
| * jtv reminds self to write wrapper script for uvt-kvm | 05:17 | |
| jtv | bigjools: it's surprising to me that you can't just use the maas-prove-dhcp tool that buildout builds for us. | 05:23 |
| bigjools | jtv: buildout doesn't run when packaging | 05:23 |
| jtv | Ahhh. | 05:24 |
| bigjools | it's a development convenience | 05:24 |
| jtv | ...Because package-building prefers setup.py over make. | 05:24 |
| jtv | If we didn't have setup.py, it'd have run "make," and then buildout would have run. | 05:24 |
| jtv | Silly me. | 05:25 |
| bigjools | jtv: nearly done testing, so far so good | 05:44 |
| bigjools | jtv: and it works | 05:48 |
| bigjools | jtv: branch merged | 06:01 |
| jtv | Thanks bigjools | 06:20 |
| rawang | hi anyone knows why I deploy service, every time I got "agent-state-info: '(error: cannot run instances: gomaasapi: got error back from server: 409 CONFLICT)", by looking at maas.log, I gotOAuthUnauthorized exception | 07:17 |
| jtv | rawang: the two sound like they're different problems... MAAS returns 409: Conflict in particular if no nodes are available that match your request. | 07:20 |
| jtv | Unfortunately older versions of juju don't return the full error message. | 07:20 |
| rawang | jtv, there are 3 available server in maas's pool | 07:21 |
| jtv | And you specified no constraints? | 07:21 |
| rawang | jtv, and i have use tag to bootstrap on one of them | 07:21 |
| jtv | And the servers are all in Ready state, I guess... | 07:21 |
| rawang | jtv, well for juju bootstrap, i have constais | 07:22 |
| rawang | jtv, oh , you remind me maybe constraints is still there when boostrap the juju ... | 07:22 |
| jtv | You could also try simulating juju's request directly against the MAAS API (e.g. using maas-cli) and seeing what the actual error message is. | 07:23 |
| rawang | jtv, oh really, could you please give me a example? :) | 07:24 |
| rawang | jtv, after I remove the constraints, it works, thanks :) | 07:25 |
| jtv | Phew. :) | 07:42 |
| jtv | bigjools: any chance you could re-test my branch lp:~jtv/maas-test/second-dhcp-check against your hardware? | 07:47 |
| jtv | I'd be particularly interested in these scenarios: | 07:47 |
| jtv | 1. A regular, successful run (to see that nothing is broken). | 07:48 |
| jtv | 2. There's a dhcpd running, but on the same interface you're running maas-test against. | 07:48 |
| jtv | I'm trying it against scenarios where the network interface has no IP address. | 07:49 |
| jtv | bigjools: scenario 2 needs your updated package inside the VM. Easiest way to do that I suppose is to edit maastest/maasfixture.py and replace the apt-get with some other way of installing the package. | 08:08 |
| === CyberJacob|Away is now known as CyberJacob | ||
| rvba | jtv: you can test scenario 1 in the lab yourself, that's precisely why the manual job is made for. | 08:37 |
| jtv | rvba: what is "the manual job"? | 08:47 |
| rvba | jtv: I was referring to the email I sent yesterday "Manual maas-test Jenkins job in the lab" | 08:48 |
| jtv | Ah, I'll have a look at that. | 08:48 |
| rvba | jtv: with that you can get the lab to test a maas-test branch for you. | 08:49 |
| jtv | Great. | 08:49 |
| rvba | That will test your scenario 1. | 08:49 |
| bjorne | I have a problem, i have try 12.04 and 13.10 and get same error | 09:34 |
| bjorne | [21/Dec/2013:00:49:57 +0100] "GET /MAAS/metadata//2012-03-01/user-data HTTP/1.1" 404 | 09:34 |
| === mwhudson is now known as zz_mwhudson | ||
| jtv | rvba: looks like that lab run failed... Do we get the details that the tests attached anywhere? | 10:22 |
| rvba | bigjools: when I comment out the broker's details (in the config), I still get a segfault… so maybe the rabbit connection is not part of the problem after all. | 10:31 |
| bigjools | rvba: interesting | 10:31 |
| bigjools | rvba: can you run the celeryd itself up with minimal args? | 10:31 |
| rvba | bigjools: that's precisely what I'm doing. | 10:32 |
| bigjools | cool | 10:32 |
| bigjools | so this smacks of a build/dependency problem | 10:32 |
| bigjools | rvba: can I ssh into your machine? | 10:32 |
| rvba | But it cannot run without some kind of config… | 10:32 |
| rvba | bigjools: sure, hang on. | 10:32 |
| rvba | bigjools: ssh ubuntu@10.55.60.167 | 10:33 |
| bigjools | rvba: did you add my key? | 10:34 |
| rvba | Yes | 10:34 |
| bigjools | it won't let me in | 10:34 |
| rvba | bigjools: ah, I'm logged in as root, just one sec. | 10:35 |
| rvba | My bad | 10:35 |
| rvba | bigjools: please try again | 10:35 |
| bigjools | still can't get in | 10:35 |
| rvba | hum… | 10:36 |
| rvba | bigjools: can you try one more time (I'm watching the logs) | 10:37 |
| rvba | ? | 10:37 |
| bigjools | there | 10:37 |
| rvba | Nothing | 10:37 |
| bigjools | you gave me the right IP address? | 10:38 |
| rvba | ssh ubuntu@10.55.60.167 | 10:38 |
| rvba | Yes | 10:38 |
| bigjools | PEBKAC | 10:38 |
| rvba | bigjools: when celeryd isn't configured to read MAAS' tasks, it doesn't blow up (!). | 10:43 |
| bigjools | rvba: If I run "celeryd" on its own it crashes | 10:43 |
| bigjools | no args | 10:43 |
| bigjools | rvba: gah remind me how to remove core dump restriction | 10:44 |
| jtv | bigjools: ulimit | 10:46 |
| bigjools | jtv: tried it already | 10:46 |
| bigjools | -bash: ulimit: core file size: cannot modify limit: Operation not permitted | 10:46 |
| jtv | Hnyug | 10:46 |
| rvba | bigjools: what core dump restriction | 10:47 |
| jtv | Strange thing is, I can do that. | 10:47 |
| rvba | ? | 10:47 |
| bigjools | rvba: size | 10:47 |
| bigjools | you don't get 'em by default | 10:47 |
| jtv | Did you make it too big perhaps? Unit of size is 512KB blocks. | 10:47 |
| bigjools | jtv: "1" doesn't even work | 10:48 |
| jtv | wha | 10:48 |
| bigjools | I used "unlimited", doesn't work | 10:48 |
| jtv | Are you sudo'ing it perhaps? I guess that might refuse. | 10:48 |
| bigjools | ulimit -S -c unlimited works | 10:49 |
| bigjools | need the -S | 10:49 |
| bigjools | gah no core file still | 10:49 |
| bigjools | wtf | 10:49 |
| bigjools | grah it's sending it to apport | 10:51 |
| jtv | /o\ | 10:51 |
| rvba | bigjools: I was wrong, it's just that sometimes it takes a long time to crash. | 10:51 |
| bigjools | rvba: writing the core file no doubt | 10:51 |
| rvba | bigjools: I noticed that python-celery (3.1.6-1ubuntu1) is now released in Trusty's cloud archive. I started a new canonistack instance and installed python-celery. When I run celeryd there is doesn't crash. Maybe the package from our ppa is faulty? | 10:57 |
| rvba | s/is doesn't/it doesn't/ | 10:57 |
| bigjools | rvba: quite likely, as I said, I suspect build or dependency problems | 10:58 |
| rvba | Trying to install MAAS with python-celery from the cloud archive. | 10:59 |
| rvba | (https://launchpad.net/ubuntu/trusty/+source/celery/3.1.6-1ubuntu1 says it was published 1 hour ago) | 10:59 |
| bigjools | rvba: can I upgrade your other instance? | 10:59 |
| rvba | bigjools: sure | 10:59 |
| rvba | But I tweaked the configs. | 11:00 |
| rvba | So it's better to start from a fresh instance. | 11:00 |
| === CyberJacob is now known as CyberJacob|Away | ||
| rvba | bigjools: arg, same crash with the python-celery from the cloud archive :/ | 11:05 |
| gmb | Why oh why oh why does Canonistack have the shits today? | 11:06 |
| rvba | bigjools: it's really weird, look: http://paste.ubuntu.com/6749871/ vs http://paste.ubuntu.com/6749868/ | 11:07 |
| bigjools | rvba: did it work with the one in the main archive? | 11:07 |
| bigjools | I still see a core | 11:07 |
| rvba | bigjools: two similar Trusty machines, one one celeryd crahsed, on the other it does not. | 11:08 |
| bigjools | ! | 11:08 |
| bigjools | this is python crashing | 11:08 |
| * rvba installs maas on the machine where python-celery does not crash… | 11:09 | |
| rvba | gmb: what do you mean? | 11:10 |
| gmb | rvba: Instances keep dying on me. lcy01 must be out of RAM. Current one is staying up though, so far... | 11:11 |
| bigjools | I gave up with 01 | 11:11 |
| rvba | gmb: hum, might be related to the problem we're seeing then… | 11:12 |
| rvba | bigjools: these are all 01 machines | 11:12 |
| gmb | rvba: Hmm, possibly. That said, if your instances are staying up, probably not. Nova is pretty proactive about pausing instances when it runs out of resources. | 11:13 |
| gmb | So you'd know if it was affecting you. | 11:14 |
| rvba | ls | 11:14 |
| rvba | Segmentation fault | 11:14 |
| rvba | I think it's affecting me :) | 11:14 |
| bigjools | rvba: I have a clue | 11:15 |
| bigjools | it's falling over in librabbitmq | 11:15 |
| bigjools | #0 0x00007fb4eef716ab in amqp_pool_alloc () | 11:15 |
| bigjools | from /usr/lib/x86_64-linux-gnu/librabbitmq.so.1 | 11:15 |
| bigjools | so it's not celery's or python's fault at all | 11:15 |
| rvba | bigjools: how did you find out? In the code dump? | 11:19 |
| bigjools | yep | 11:19 |
| bigjools | it's rather annoying that someone bumped celery a whole major version | 11:23 |
| bigjools | and didn't choose a new package name | 11:24 |
| rvba | bigjools: did you see that running celeryd with strace? | 11:26 |
| bigjools | rvba: no I analysed the core | 11:26 |
| rvba | Ah okay, I see it now. | 11:27 |
| rvba | bigjools: so, technically it blew up in librabbitmq1 but this doesn't tell us why exactly does it? It might be a wrong interaction with another library. | 11:30 |
| bigjools | rvba: amqp_pool_alloc | 11:30 |
| bigjools | I suspect some esoteric part of rabbit changed | 11:30 |
| rvba | bigjools: the most recent successful run we had in the lab is this: http://d-jenkins.ubuntu-ci:8080/view/MAAS/job/trusty-adt-maas-manual/4/console | 11:32 |
| rvba | The console log gives us all the versions of all the packages used. | 11:32 |
| bigjools | rvba: old celery? | 11:34 |
| rvba | Yes | 11:35 |
| rvba | An no trace of librabbitmq. | 11:35 |
| rvba | bigjools: celery's changelog says (for the upgrade to 3.1.6-1 from 2.5.3-4ubuntu1): 'Drop depenceny on python-amqplib, it has been replaced by python-amqp/python-librabbitmq in python-kombu.' And of course python-librabbitmq depends on librabbitmq1. | 11:41 |
| bigjools | right | 11:41 |
| bigjools | it needs a build with the -dbg library and then we can get which line of code blew up | 11:43 |
| bigjools | my frantic googling is turning up nothing | 11:43 |
| rvba | Same here. | 11:43 |
| bigjools | I gotta sleep man | 11:44 |
| bigjools | good luck and good night | 11:44 |
| rvba | Yeah, I'll see if Andres can help us. Maybe by creating debugging packages like you said. | 11:44 |
| bigjools | you'll need andres | 11:45 |
| rvba | bigjools: ^ | 11:45 |
| bigjools | just install -dbg ones | 11:45 |
| bigjools | librabbitmq-dbg | 11:45 |
| bigjools | and run again | 11:45 |
| rvba | Oh, they exist already? | 11:45 |
| bigjools | then when you get the core file: | 11:45 |
| bigjools | gdb /usr/bin/python core | 11:46 |
| bigjools | > bt | 11:46 |
| rvba | Yeah, I know that part. | 11:46 |
| bigjools | ok | 11:46 |
| rvba | I'll try that. | 11:46 |
| bigjools | ok I am going to bed | 11:46 |
| bigjools | see you | 11:46 |
| rvba | night | 11:47 |
| === CyberJacob|Away is now known as CyberJacob | ||
| === zz_mwhudson is now known as mwhudson | ||
| === CyberJacob is now known as CyberJacob|Away | ||
| === CyberJacob|Away is now known as CyberJacob | ||
| === mwhudson is now known as zz_mwhudson | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!