=== CyberJacob is now known as CyberJacob|Away [02:25] bigjools: howdy! when did the change to update-dns stuff landed and when did the integration tests start failing? [02:25] roaksoax: not sure, raphael did the investigation work [02:25] but before xmas at least [02:27] bigjools: err upstream_dns stuff [02:27] I thought it was your changes for the charm? [02:28] bigjools: nope, I never made that change [02:28] oh [02:28] bigjools: oh it is the DNS forwarders stuff [02:28] don't think so, that's not changed in packaging [02:29] roaksoax: revno 215 on 2013-12-17 is the problem IIRC [02:29] your packaging change [02:29] "Split maas-region-controller into maas-region-controller-min" [02:29] bigjools: i was talking about the DNS forwarders changes [02:30] roaksoax: oh sorry didn't understand what you were talking about [02:30] no, those are not even started yet [02:32] bigjools: http://pastebin.ubuntu.com/6742345/ [02:33] roaksoax: that's the upstream change - I thought you were talking about packaging changes? [02:33] bigjools: so the issue hasn't really been a packaging change... thanks to the packaging change is that the issue was discovered [02:33] anyway that change didn't make the integration tests start failing AFAIK [02:34] ah cool, you found out what's up? [02:34] bigjools: the issue is that 'maas set_up_dns' is accessing the DB, when before it didn't use to do that [02:34] ah [02:34] makes sense [02:34] bigjools: so, http://paste.ubuntu.com/6742346/ -> So this is a hack I use to "fix" this [02:34] do you have a recommendation? [02:35] bigjools: combined with http://paste.ubuntu.com/6742350/ I think we can solve the problem [02:35] yeah we don't need it in postinst for sure, the job will deal with it [02:35] bigjools: obviously for the upstream change, instead of a try/except, we can simply add a parameter to specify /'maas set_up_dns --upstream-dns=None [02:36] yeah I'd prefer that over a try/except [02:36] param to override it fetching from config [02:36] it's amazing what innocuous changes will break elsewhere ... [02:37] thanks for figuring it out dude [02:37] bigjools: yeah! But anyways, adding the parameter (which should be fairly simply), and removing the line in maas-dns.postinst, we have things back to normal (without having to revert changes back) [02:37] +1 [02:37] if that's definitely the cause, I'm good with it [02:38] bigjools: yeah I installed maas-cluster-controller-min , and then started running DNS commands manually (those in the psotinst) and that's how I found out what was wrong [02:38] and bazinga [02:39] indeed [02:52] bigjools: so do we want maas set_up_dns to receieve a boolean parameter to not try to obtain the value from the DB, or do we want a parameter where we can actually set the forwarder? [02:53] roaksoax: set value is better [02:53] bigjools: as in. 1. set_up_dns --no-upstream-dns or 2. set_up_dns --upstream-dns={None,1.1.1.1} [02:53] ack [02:53] roaksoax: in fact do we need the code to fetch it from config at all? [02:53] the postinst is the only place that calls it IIRC [02:54] bigjools: uhmmm I don't think so, cause celery tasks would also configure that, wouldn't they? [02:54] celery uses a different code path [02:55] this stuff is only a "maas XXX" command [02:56] bigjools: right, but besides from maas set_up_dns, who sets the upstream_dns when found in the DB? [02:56] roaksoax: celery jobs [02:57] bigjools: ok, so in reality, we don't really need it in set_up_dns, because the celery job will update that once it is manually added on maas then [02:57] roaksoax: indeed [02:57] that's what I was starting to think [02:59] bigjools: ok, so this should do it: http://paste.ubuntu.com/6742417/ [02:59] bigjools: I'll get that tested alongside the packaging change [02:59] +1 [03:44] bigjools: ok it works : https://code.launchpad.net/~andreserl/maas/no_upstream_dns_on_setup/+merge/201332 [05:49] jtv: super quick pre-imp? [05:55] bigjools: sure... firing up laptop. [05:56] jtv: plug it in this time? :) [07:02] bigjools: any chance you could give this version of second DHCP detection a whirl? https://code.launchpad.net/~jtv/maas-test/second-dhcp-check/+merge/201232 [07:02] This is meant to solve the problem you had detecting the DHCP server you were still running on the same machine you ran maas-test on. [07:02] It should _also_ help with the case where the test interface has no IP address. === CyberJacob|Away is now known as CyberJacob [10:05] jtv: I get a failure when I run maas-test (trunk)… [10:05] it seems related to the DHCP stuff: [10:05] http://paste.ubuntu.com/6743775/ [10:06] It looks (again, I might be wrong) as if we're running 'uvt-kvm ip <>' before the vm is even created… [10:06] jtv: does that sound possible? [10:07] It seems unlikely — AFAIK we do a "uvt-kvm wait" first. [10:08] Ok, I'll investigate… [10:08] Looks like the problem you pasted is in get_maas_version, no? [10:08] Ah! [10:08] I think I know what it is. [10:09] More or less. [10:09] I made the install_maas() happen earlier than it used to. [10:09] Shouldn't be before the "uvt-kvm wait" though. [10:09] Looks like the MAAS fixture is setup before the KVM fixture :/ [10:10] No, but maas gets installed before the maas fixture is set up. [10:10] Crossing changes. :( [10:10] (Revision 218 works) [10:11] The exception seems to happen the second time install_maas runs. [10:12] During MAASFixture.setUp(). [10:13] At any rate, if the traceback is correct, everything looks wrong. Why would "uvt-kvm ip" fail that late in the game!? [10:14] What second time? install_maas should only be run once. [10:14] Ah. That is a change I made, but I guess it's still in the branch I'm working on. [10:17] Note that we seem to be running "uvt-kvm ip" for every single command we run on the VM. So you'd expect it to have succeeded at least once before this point. [10:17] The vm doesn't seem to exist at all when we run this command. [10:18] Missing sudo somewhere maybe? [10:18] ← desperate guess [10:18] Frankly, I'm really confused by the reorganisation revision 119 introduced. [10:19] It just removes some class fixtures and makes them variables in main(). [10:19] No, I put a debugging statement in run_command() and we simply don't create the VM at all. [10:19] It's a bit more involved than that :). [10:20] The result is that KVMFixture.setUp() is not called (or rather, we blow up in install_maas before it's even called). [10:21] Ah! [10:22] Now I remember. This was a fix I had wanted to have landed, but it got held up by the current packaging trouble. [10:22] It's a missing setUp call. [11:22] gmb: btw, what I had in mind when I spoke about using run_command to include hardware information in the report is something along the lines of http://paste.ubuntu.com/6744069/ [11:22] gmb: of course, we'd want to format the output a bit better. [11:23] rvba: Ah, thanks for the tip! I was looking for a way to do that as an API call in the mass-test code, but that seems straightforward enough (formatting notwithstanding). [11:24] An API call will also work, but since we have everything setup to shell out on the VM, the solution above might be a bit more easier. [11:24] s/more// [11:24] Yep, that seems perfectly sane to me. [11:26] gmb: for your viewing pleasure (I've always been very found of that expression), here is the result of that script on my test machine: http://paste.ubuntu.com/6744089/ [11:27] rvba: Lovely, thank you. Although when I say lovely I may actually mean "gaaaagh." But I'm not to worried about making it nice as I am about making it functional right now. [11:28] Getting it to work sounds like a good first step to me :). [11:31] gmb: actually, now that I think about it, it might be ever more simple to just do a "maas dumpdata." [11:31] rvba: Does that basically give us the whole DB? [11:31] Yeah. [11:31] More data == more better. [11:31] But the DB won't contain anything big so it's safe. [11:31] Yep. [11:31] Excellent. [11:31] Plus we're sure we're not missing anything. [11:32] self.maas.kvm_fixture.run_command(['sudo', 'maas', 'dumpdata']) [11:32] Brill. [11:32] * gmb does that. [11:34] I'll give you an example of what you get back in a minute… [11:34] (I killed my vm :/) [11:52] gmb: http://paste.ubuntu.com/6744188/ [11:53] rvba: Why all the leading whitespace? [11:54] gmb: don't know… looks like it's an artifact created by paste of something, here is the file verbatim: http://people.canonical.com/~rvb/dump [11:55] *eyebleed* [11:55] rvba: My feeling is we should just dump it, attach it, and care about cleaning it up later. Thoughts? [11:56] gmb: I agree. It's pure JSON so it's easy to parse. [11:57] gmb: as a second step, if we still want to do that, it's easy to exact identifiers from the lshw output. [11:58] rvba: Okay. I'll finish this branch for gathering it, then I'll look at extracting stuff that's useful. [11:58] Sounds good to me. === CyberJacob is now known as CyberJacob|Away [12:28] * gmb -> lunch === matsubara_ is now known as matsubara [17:10] hi [17:10] i have the problem that my nodes stay in state "commissioning" [17:10] they never get "ready" === CyberJacob|Away is now known as CyberJacob === CyberJacob is now known as CyberJacob|Away