[02:25] <roaksoax> bigjools: howdy! when did the change to update-dns stuff landed and when did the integration tests start failing?
[02:25] <bigjools> roaksoax: not sure, raphael did the investigation work
[02:25] <bigjools> but before xmas at least
[02:27] <roaksoax> bigjools: err upstream_dns stuff
[02:27] <bigjools> I thought it was your changes for the charm?
[02:28] <roaksoax> bigjools: nope, I never made that change
[02:28] <bigjools> oh
[02:28] <roaksoax> bigjools: oh it is the DNS forwarders stuff
[02:28] <bigjools> don't think so, that's not changed in packaging
[02:29] <bigjools> roaksoax: revno 215 on 2013-12-17 is the problem IIRC
[02:29] <bigjools> your packaging change
[02:29] <bigjools> "Split maas-region-controller into maas-region-controller-min"
[02:29] <roaksoax> bigjools: i was talking about the DNS forwarders changes
[02:30] <bigjools> roaksoax: oh sorry didn't understand what you were talking about
[02:30] <bigjools> no, those are not even started yet
[02:32] <roaksoax> bigjools: http://pastebin.ubuntu.com/6742345/
[02:33] <bigjools> roaksoax: that's the upstream change - I thought you were talking about packaging changes?
[02:33] <roaksoax> bigjools: so the issue hasn't really been a packaging change... thanks to the packaging change is that the issue was discovered
[02:33] <bigjools> anyway that change didn't make the integration tests start failing AFAIK
[02:34] <bigjools> ah cool, you found out what's up?
[02:34] <roaksoax> bigjools: the issue is that 'maas set_up_dns' is accessing the DB, when before it didn't use to do that
[02:34] <bigjools> ah
[02:34] <bigjools> makes sense
[02:34] <roaksoax> bigjools: so, http://paste.ubuntu.com/6742346/ -> So this is a hack I use to "fix" this
[02:34] <bigjools> do you have a recommendation?
[02:35] <roaksoax> bigjools: combined with http://paste.ubuntu.com/6742350/ I think we can solve the problem
[02:35] <bigjools> yeah we don't need it in postinst for sure, the job will deal with it
[02:35] <roaksoax> bigjools: obviously for the upstream change, instead of a try/except, we can simply add a parameter to specify /'maas set_up_dns --upstream-dns=None
[02:36] <bigjools> yeah I'd prefer that over a try/except
[02:36] <bigjools> param to override it fetching from config
[02:36] <bigjools> it's amazing what innocuous changes will break elsewhere ...
[02:37] <bigjools> thanks for figuring it out dude
[02:37] <roaksoax> bigjools: yeah! But anyways, adding the parameter (which should be fairly simply), and removing the line in maas-dns.postinst, we have things back to normal (without having to revert changes back)
[02:37] <bigjools> +1
[02:37] <bigjools> if that's definitely the cause, I'm good with it
[02:38] <roaksoax> bigjools: yeah I installed maas-cluster-controller-min , and then started running DNS commands manually (those in the psotinst) and that's how I found out what was wrong
[02:38] <bigjools> and bazinga
[02:39] <roaksoax> indeed
[02:52] <roaksoax> bigjools: so do we want maas set_up_dns to receieve a boolean parameter to not try to obtain the value from the DB, or do we want a parameter where we can actually set the forwarder?
[02:53] <bigjools> roaksoax: set value is better
[02:53] <roaksoax> bigjools: as in. 1. set_up_dns --no-upstream-dns or 2. set_up_dns --upstream-dns={None,1.1.1.1}
[02:53] <roaksoax> ack
[02:53] <bigjools> roaksoax: in fact do we need the code to fetch it from config at all?
[02:53] <bigjools> the postinst is the only place that calls it IIRC
[02:54] <roaksoax> bigjools: uhmmm I don't think so, cause celery tasks would also configure that, wouldn't they?
[02:54] <bigjools> celery uses a different code path
[02:55] <bigjools> this stuff is only a "maas XXX" command
[02:56] <roaksoax> bigjools: right, but besides from maas set_up_dns, who sets the upstream_dns when found in the DB?
[02:56] <bigjools> roaksoax: celery jobs
[02:57] <roaksoax> bigjools: ok, so in reality, we don't really need it in set_up_dns, because the celery job will update that once it is manually added on maas then
[02:57] <bigjools> roaksoax: indeed
[02:57] <bigjools> that's what I was starting to think
[02:59] <roaksoax> bigjools: ok, so this should do it: http://paste.ubuntu.com/6742417/
[02:59] <roaksoax> bigjools: I'll get that tested alongside the packaging change
[02:59] <bigjools> +1
[03:44] <roaksoax> bigjools: ok it works : https://code.launchpad.net/~andreserl/maas/no_upstream_dns_on_setup/+merge/201332
[05:49] <bigjools> jtv: super quick pre-imp?
[05:55] <jtv> bigjools: sure...  firing up laptop.
[05:56] <bigjools> jtv: plug it in this time? :)
[07:02] <jtv> bigjools: any chance you could give this version of second DHCP detection a whirl?  https://code.launchpad.net/~jtv/maas-test/second-dhcp-check/+merge/201232
[07:02] <jtv> This is meant to solve the problem you had detecting the DHCP server you were still running on the same machine you ran maas-test on.
[07:02] <jtv> It should _also_ help with the case where the test interface has no IP address.
[10:05] <rvba> jtv: I get a failure when I run maas-test (trunk)…
[10:05] <rvba> it seems related to the DHCP stuff:
[10:05] <rvba> http://paste.ubuntu.com/6743775/
[10:06] <rvba> It looks (again, I might be wrong) as if we're running 'uvt-kvm ip <>' before the vm is even created…
[10:06] <rvba> jtv: does that sound possible?
[10:07] <jtv> It seems unlikely — AFAIK we do a "uvt-kvm wait" first.
[10:08] <rvba> Ok, I'll investigate…
[10:08] <jtv> Looks like the problem you pasted is in get_maas_version, no?
[10:08] <jtv> Ah!
[10:08] <jtv> I think I know what it is.
[10:09] <jtv> More or less.
[10:09] <jtv> I made the install_maas() happen earlier than it used to.
[10:09] <jtv> Shouldn't be before the "uvt-kvm wait" though.
[10:09] <rvba> Looks like the MAAS fixture is setup before the KVM fixture :/
[10:10] <jtv> No, but maas gets installed before the maas fixture is set up.
[10:10] <jtv> Crossing changes.  :(
[10:10] <rvba> (Revision 218 works)
[10:11] <jtv> The exception seems to happen the second time install_maas runs.
[10:12] <jtv> During MAASFixture.setUp().
[10:13] <jtv> At any rate, if the traceback is correct, everything looks wrong.  Why would "uvt-kvm ip" fail that late in the game!?
[10:14] <rvba> What second time?  install_maas should only be run once.
[10:14] <jtv> Ah.  That is a change I made, but I guess it's still in the branch I'm working on.
[10:17] <jtv> Note that we seem to be running "uvt-kvm ip" for every single command we run on the VM.  So you'd expect it to have succeeded at least once before this point.
[10:17] <rvba> The vm doesn't seem to exist at all when we run this command.
[10:18] <jtv> Missing sudo somewhere maybe?
[10:18] <jtv> ← desperate guess
[10:18] <rvba> Frankly, I'm really confused by the reorganisation revision 119 introduced.
[10:19] <jtv> It just removes some class fixtures and makes them variables in main().
[10:19] <rvba> No, I put a debugging statement in run_command() and we simply don't create the VM at all.
[10:19] <rvba> It's a bit more involved than that :).
[10:20] <rvba> The result is that KVMFixture.setUp() is not called (or rather, we blow up in install_maas before it's even called).
[10:21] <jtv> Ah!
[10:22] <jtv> Now I remember.  This was a fix I had wanted to have landed, but it got held up by the current packaging trouble.
[10:22] <jtv> It's a missing setUp call.
[11:22] <rvba> gmb: btw, what I had in mind when I spoke about using run_command to include hardware information in the report is something along the lines of http://paste.ubuntu.com/6744069/
[11:22] <rvba> gmb: of course, we'd want to format the output a bit better.
[11:23] <gmb> rvba: Ah, thanks for the tip! I was looking for a way to do that as an API call in the mass-test code, but that seems straightforward enough (formatting notwithstanding).
[11:24] <rvba> An API call will also work, but since we have everything setup to shell out on the VM, the solution above might be a bit more easier.
[11:24] <rvba> s/more//
[11:24] <gmb> Yep, that seems perfectly sane to me.
[11:26] <rvba> gmb: for your viewing pleasure (I've always been very found of that expression), here is the result of that script on my test machine: http://paste.ubuntu.com/6744089/
[11:27] <gmb> rvba: Lovely, thank you. Although when I say lovely I may actually mean "gaaaagh." But I'm not to worried about making it nice as I am about making it functional right now.
[11:28] <rvba> Getting it to work sounds like a good first step to me :).
[11:31] <rvba> gmb: actually, now that I think about it, it might be ever more simple to just do a "maas dumpdata."
[11:31] <gmb> rvba: Does that basically give us the whole DB?
[11:31] <rvba> Yeah.
[11:31] <gmb> More data == more better.
[11:31] <rvba> But the DB won't contain anything big so it's safe.
[11:31] <gmb> Yep.
[11:31] <gmb> Excellent.
[11:31] <rvba> Plus we're sure we're not missing anything.
[11:32] <rvba> self.maas.kvm_fixture.run_command(['sudo', 'maas', 'dumpdata'])
[11:32] <gmb> Brill.
[11:32]  * gmb does that.
[11:34] <rvba> I'll give you an example of what you get back in a minute…
[11:34] <rvba> (I killed my vm :/)
[11:52] <rvba> gmb: http://paste.ubuntu.com/6744188/
[11:53] <gmb> rvba: Why all the leading whitespace?
[11:54] <rvba> gmb: don't know… looks like it's an artifact created by paste of something, here is the file verbatim: http://people.canonical.com/~rvb/dump
[11:55] <gmb> *eyebleed*
[11:55] <gmb> rvba: My feeling is we should just dump it, attach it, and care about cleaning it up later. Thoughts?
[11:56] <rvba> gmb: I agree.  It's pure JSON so it's easy to parse.
[11:57] <rvba> gmb: as a second step, if we still want to do that, it's easy to exact identifiers from the lshw output.
[11:58] <gmb> rvba: Okay. I'll finish this branch for gathering it, then I'll look at extracting stuff that's useful.
[11:58] <rvba> Sounds good to me.
[12:28]  * gmb -> lunch
[17:10] <tomixxx> hi
[17:10] <tomixxx> i have the problem that my nodes stay in state "commissioning"
[17:10] <tomixxx> they never get "ready"