[03:29] <jtv> Why isn't the packaging building bin/maas-probe-dhcp!?
[03:30] <jtv> bigjools: any ideas?  ^
[04:47] <bigjools> jtv: where should it live do you reckon? cluster?
[04:47] <bigjools> maas-dhcp?
[04:47] <jtv> I was thinking maas-dhcp.
[04:47] <bigjools> k
[04:47] <jtv> I already have a branch installing it as part of that package.  The problem is getting it built.
[04:47] <jtv> But feel free to write up your own branch; it's not exactly a lot of diff.
[04:52] <bigjools> jtv: hmmm it needs to go in cluster controller
[04:53] <bigjools> well doesn't *have* to but makes more sense to me
[04:53] <bigjools> ah sod it maas-dhcp
[04:53] <jtv> Well I had been wondering, but you weren't around: did we have a particular purpose in mind for it?
[04:54] <jtv> Were we going to run it during install?  Or before starting our DHCP server..?
[04:58] <bigjools> jtv: show me your diff
[05:02] <jtv> I added to debian/maas-dhcp.install:
[05:02] <jtv> debian/extras/maas-probe-dhcp /usr/sbin
[05:03] <jtv> (Or various other things I tried).
[05:03] <jtv> I also had a lintian-overrides, but I reverted that already.
[05:03] <bigjools> well you need a bit more than that
[05:03] <bigjools> MP coming
[05:12] <bigjools> jtv: https://code.launchpad.net/~julian-edwards/maas/packaging-add-maas-probe-dhcp/+merge/201538
[05:12] <jtv> Thanks.
[05:12]  * jtv shall review forthwith
[05:12] <bigjools> buildout scripts are a PITA
[05:12] <bigjools> jtv: it's untested!
[05:12] <bigjools> I figured since you were desperate ...
[05:13] <bigjools> I can do it if you want
[05:13] <jtv> I see that you managed to include bash.  :)
[05:13] <bigjools> sadly
[05:13] <bigjools> cargo culted existing ones
[05:13] <jtv> What exactly do you mean by untested — "not covered by test suite" or "I haven't tried this yet"?
[05:14] <bigjools> I haven't tried it
[05:14] <jtv> Ah.  Well, it'll sort of come naturally with my manual testing, so don't worry about it.
[05:14] <jtv> As long as the package builds...
[05:14] <bigjools> ok
[05:14] <bigjools> well let's do a test build first
[05:15] <bigjools> I'll do it on cstack
[05:17] <jtv> OK
[05:17]  * jtv reminds self to write wrapper script for uvt-kvm
[05:23] <jtv> bigjools: it's surprising to me that you can't just use the maas-prove-dhcp tool that buildout builds for us.
[05:23] <bigjools> jtv: buildout doesn't run when packaging
[05:24] <jtv> Ahhh.
[05:24] <bigjools> it's a development convenience
[05:24] <jtv> ...Because package-building prefers setup.py over make.
[05:24] <jtv> If we didn't have setup.py, it'd have run "make," and then buildout would have run.
[05:25] <jtv> Silly me.
[05:44] <bigjools> jtv: nearly done testing, so far so good
[05:48] <bigjools> jtv: and it works
[06:01] <bigjools> jtv: branch merged
[06:20] <jtv> Thanks bigjools
[07:17] <rawang> hi anyone knows why I deploy service, every time I got "agent-state-info: '(error: cannot run instances: gomaasapi: got error back from  server: 409 CONFLICT)",   by looking at maas.log, I gotOAuthUnauthorized exception
[07:20] <jtv> rawang: the two sound like they're different problems...  MAAS returns 409: Conflict in particular if no nodes are available that match your request.
[07:20] <jtv> Unfortunately older versions of juju don't return the full error message.
[07:21] <rawang> jtv, there are 3 available server in maas's pool
[07:21] <jtv> And you specified no constraints?
[07:21] <rawang> jtv, and i have use tag to bootstrap on one of them
[07:21] <jtv> And the servers are all in Ready state, I guess...
[07:22] <rawang> jtv, well for juju bootstrap, i have constais
[07:22] <rawang> jtv, oh , you remind me maybe constraints is still there when boostrap the juju ...
[07:23] <jtv> You could also try simulating juju's request directly against the MAAS API (e.g. using maas-cli) and seeing what the actual error message is.
[07:24] <rawang> jtv, oh really, could you please give me a example? :)
[07:25] <rawang> jtv, after I remove the constraints, it works, thanks :)
[07:42] <jtv> Phew.  :)
[07:47] <jtv> bigjools: any chance you could re-test my branch lp:~jtv/maas-test/second-dhcp-check against your hardware?
[07:47] <jtv> I'd be particularly interested in these scenarios:
[07:48] <jtv> 1. A regular, successful run (to see that nothing is broken).
[07:48] <jtv> 2. There's a dhcpd running, but on the same interface you're running maas-test against.
[07:49] <jtv> I'm trying it against scenarios where the network interface has no IP address.
[08:08] <jtv> bigjools: scenario 2 needs your updated package inside the VM.  Easiest way to do that I suppose is to edit maastest/maasfixture.py and replace the apt-get with some other way of installing the package.
[08:37] <rvba> jtv: you can test scenario 1 in the lab yourself, that's precisely why the manual job is made for.
[08:47] <jtv> rvba: what is "the manual job"?
[08:48] <rvba> jtv: I was referring to the email I sent yesterday "Manual maas-test Jenkins job in the lab"
[08:48] <jtv> Ah, I'll have a look at that.
[08:49] <rvba> jtv: with that you can get the lab to test a maas-test branch for you.
[08:49] <jtv> Great.
[08:49] <rvba> That will test your scenario 1.
[09:34] <bjorne> I have a problem, i have try 12.04 and 13.10 and get same error
[09:34] <bjorne> [21/Dec/2013:00:49:57 +0100] "GET /MAAS/metadata//2012-03-01/user-data HTTP/1.1" 404
[10:22] <jtv> rvba: looks like that lab run failed...  Do we get the details that the tests attached anywhere?
[10:31] <rvba> bigjools: when I comment out the broker's details (in the config), I still get a segfault… so maybe the rabbit connection is not part of the problem after all.
[10:31] <bigjools> rvba: interesting
[10:31] <bigjools> rvba: can you run the celeryd itself up with minimal args?
[10:32] <rvba> bigjools: that's precisely what I'm doing.
[10:32] <bigjools> cool
[10:32] <bigjools> so this smacks of a build/dependency problem
[10:32] <bigjools> rvba: can I ssh into your machine?
[10:32] <rvba> But it cannot run without some kind of config…
[10:32] <rvba> bigjools: sure, hang on.
[10:33] <rvba> bigjools: ssh ubuntu@10.55.60.167
[10:34] <bigjools> rvba: did you add my key?
[10:34] <rvba> Yes
[10:34] <bigjools> it won't let me in
[10:35] <rvba> bigjools: ah, I'm logged in as root, just one sec.
[10:35] <rvba> My bad
[10:35] <rvba> bigjools: please try again
[10:35] <bigjools> still can't get in
[10:36] <rvba> hum…
[10:37] <rvba> bigjools: can you try one more time (I'm watching the logs)
[10:37] <rvba> ?
[10:37] <bigjools> there
[10:37] <rvba> Nothing
[10:38] <bigjools> you gave me the right IP address?
[10:38] <rvba> ssh ubuntu@10.55.60.167
[10:38] <rvba> Yes
[10:38] <bigjools> PEBKAC
[10:43] <rvba> bigjools: when celeryd isn't configured to read MAAS' tasks, it doesn't blow up (!).
[10:43] <bigjools> rvba: If I run "celeryd" on its own it crashes
[10:43] <bigjools> no args
[10:44] <bigjools> rvba: gah remind me how to remove core dump restriction
[10:46] <jtv> bigjools: ulimit
[10:46] <bigjools> jtv: tried it already
[10:46] <bigjools> -bash: ulimit: core file size: cannot modify limit: Operation not permitted
[10:46] <jtv> Hnyug
[10:47] <rvba> bigjools: what core dump restriction
[10:47] <jtv> Strange thing is, I can do that.
[10:47] <rvba> ?
[10:47] <bigjools> rvba: size
[10:47] <bigjools> you don't get 'em by default
[10:47] <jtv> Did you make it too big perhaps?  Unit of size is 512KB blocks.
[10:48] <bigjools> jtv: "1" doesn't even work
[10:48] <jtv> wha
[10:48] <bigjools> I used "unlimited", doesn't work
[10:48] <jtv> Are you sudo'ing it perhaps?  I guess that might refuse.
[10:49] <bigjools> ulimit -S -c unlimited works
[10:49] <bigjools> need the -S
[10:49] <bigjools> gah no core file still
[10:49] <bigjools> wtf
[10:51] <bigjools> grah it's sending it to apport
[10:51] <jtv>  /o\
[10:51] <rvba> bigjools: I was wrong, it's just that sometimes it takes a long time to crash.
[10:51] <bigjools> rvba: writing the core file no doubt
[10:57] <rvba> bigjools: I noticed that python-celery (3.1.6-1ubuntu1) is now released in Trusty's cloud archive.  I started a new canonistack instance and installed python-celery.  When I run celeryd there is doesn't crash.  Maybe the package from our ppa is faulty?
[10:57] <rvba> s/is doesn't/it doesn't/
[10:58] <bigjools> rvba: quite likely, as I said, I suspect build or dependency problems
[10:59] <rvba> Trying to install MAAS with python-celery from the cloud archive.
[10:59] <rvba> (https://launchpad.net/ubuntu/trusty/+source/celery/3.1.6-1ubuntu1 says it was published 1 hour ago)
[10:59] <bigjools> rvba: can I upgrade your other instance?
[10:59] <rvba> bigjools: sure
[11:00] <rvba> But I tweaked the configs.
[11:00] <rvba> So it's better to start from a fresh instance.
[11:05] <rvba> bigjools: arg, same crash with the python-celery from the cloud archive :/
[11:06] <gmb> Why oh why oh why does Canonistack have the shits today?
[11:07] <rvba> bigjools: it's really weird, look: http://paste.ubuntu.com/6749871/ vs http://paste.ubuntu.com/6749868/
[11:07] <bigjools> rvba: did it work with the one in the main archive?
[11:07] <bigjools> I still see a core
[11:08] <rvba> bigjools: two similar Trusty machines, one one celeryd crahsed, on the other it does not.
[11:08] <bigjools> !
[11:08] <bigjools> this is python crashing
[11:09]  * rvba installs maas on the machine where python-celery does not crash…
[11:10] <rvba> gmb: what do you mean?
[11:11] <gmb> rvba: Instances keep dying on me. lcy01 must be out of RAM. Current one is staying up though, so far...
[11:11] <bigjools> I gave up with 01
[11:12] <rvba> gmb: hum, might be related to the problem we're seeing then…
[11:12] <rvba> bigjools: these are all 01 machines
[11:13] <gmb> rvba: Hmm, possibly. That said, if your instances are staying up, probably not. Nova is pretty proactive about pausing instances when it runs out of resources.
[11:14] <gmb> So you'd know if it was affecting you.
[11:14] <rvba>  ls
[11:14] <rvba> Segmentation fault
[11:14] <rvba> I think it's affecting me :)
[11:15] <bigjools> rvba: I have a clue
[11:15] <bigjools> it's falling over in librabbitmq
[11:15] <bigjools> #0  0x00007fb4eef716ab in amqp_pool_alloc ()
[11:15] <bigjools>    from /usr/lib/x86_64-linux-gnu/librabbitmq.so.1
[11:15] <bigjools> so it's not celery's or python's fault at all
[11:19] <rvba> bigjools: how did you find out?  In the code dump?
[11:19] <bigjools> yep
[11:23] <bigjools> it's rather annoying that someone bumped celery a whole major version
[11:24] <bigjools> and didn't choose a new package name
[11:26] <rvba> bigjools: did you see that running celeryd with strace?
[11:26] <bigjools> rvba: no I analysed the core
[11:27] <rvba> Ah okay, I see it now.
[11:30] <rvba> bigjools: so, technically it blew up in librabbitmq1 but this doesn't tell us why exactly does it?  It might be a wrong interaction with another library.
[11:30] <bigjools> rvba: amqp_pool_alloc
[11:30] <bigjools> I suspect some esoteric part of rabbit changed
[11:32] <rvba> bigjools: the most recent successful run we had in the lab is this: http://d-jenkins.ubuntu-ci:8080/view/MAAS/job/trusty-adt-maas-manual/4/console
[11:32] <rvba> The console log gives us all the versions of all the packages used.
[11:34] <bigjools> rvba: old celery?
[11:35] <rvba> Yes
[11:35] <rvba> An no trace of librabbitmq.
[11:41] <rvba> bigjools: celery's changelog says (for the upgrade to 3.1.6-1 from 2.5.3-4ubuntu1): 'Drop depenceny on python-amqplib, it has been replaced by python-amqp/python-librabbitmq in python-kombu.'  And of course python-librabbitmq depends on librabbitmq1.
[11:41] <bigjools> right
[11:43] <bigjools> it needs a build with the -dbg library and then we can get which line of code blew up
[11:43] <bigjools> my frantic googling is turning up nothing
[11:43] <rvba> Same here.
[11:44] <bigjools> I gotta sleep man
[11:44] <bigjools> good luck and good night
[11:44] <rvba> Yeah, I'll see if Andres can help us.  Maybe by creating debugging packages like you said.
[11:45] <bigjools> you'll need andres
[11:45] <rvba> bigjools: ^
[11:45] <bigjools> just install -dbg ones
[11:45] <bigjools> librabbitmq-dbg
[11:45] <bigjools> and run again
[11:45] <rvba> Oh, they exist already?
[11:45] <bigjools> then when you get the core file:
[11:46] <bigjools> gdb /usr/bin/python core
[11:46] <bigjools> > bt
[11:46] <rvba> Yeah, I know that part.
[11:46] <bigjools> ok
[11:46] <rvba> I'll try that.
[11:46] <bigjools> ok I am going to bed
[11:46] <bigjools> see you
[11:47] <rvba> night