| === CyberJacob is now known as CyberJacob|Away | ||
| roaksoax | bigjools: howdy! when did the change to update-dns stuff landed and when did the integration tests start failing? | 02:25 |
|---|---|---|
| bigjools | roaksoax: not sure, raphael did the investigation work | 02:25 |
| bigjools | but before xmas at least | 02:25 |
| roaksoax | bigjools: err upstream_dns stuff | 02:27 |
| bigjools | I thought it was your changes for the charm? | 02:27 |
| roaksoax | bigjools: nope, I never made that change | 02:28 |
| bigjools | oh | 02:28 |
| roaksoax | bigjools: oh it is the DNS forwarders stuff | 02:28 |
| bigjools | don't think so, that's not changed in packaging | 02:28 |
| bigjools | roaksoax: revno 215 on 2013-12-17 is the problem IIRC | 02:29 |
| bigjools | your packaging change | 02:29 |
| bigjools | "Split maas-region-controller into maas-region-controller-min" | 02:29 |
| roaksoax | bigjools: i was talking about the DNS forwarders changes | 02:29 |
| bigjools | roaksoax: oh sorry didn't understand what you were talking about | 02:30 |
| bigjools | no, those are not even started yet | 02:30 |
| roaksoax | bigjools: http://pastebin.ubuntu.com/6742345/ | 02:32 |
| bigjools | roaksoax: that's the upstream change - I thought you were talking about packaging changes? | 02:33 |
| roaksoax | bigjools: so the issue hasn't really been a packaging change... thanks to the packaging change is that the issue was discovered | 02:33 |
| bigjools | anyway that change didn't make the integration tests start failing AFAIK | 02:33 |
| bigjools | ah cool, you found out what's up? | 02:34 |
| roaksoax | bigjools: the issue is that 'maas set_up_dns' is accessing the DB, when before it didn't use to do that | 02:34 |
| bigjools | ah | 02:34 |
| bigjools | makes sense | 02:34 |
| roaksoax | bigjools: so, http://paste.ubuntu.com/6742346/ -> So this is a hack I use to "fix" this | 02:34 |
| bigjools | do you have a recommendation? | 02:34 |
| roaksoax | bigjools: combined with http://paste.ubuntu.com/6742350/ I think we can solve the problem | 02:35 |
| bigjools | yeah we don't need it in postinst for sure, the job will deal with it | 02:35 |
| roaksoax | bigjools: obviously for the upstream change, instead of a try/except, we can simply add a parameter to specify /'maas set_up_dns --upstream-dns=None | 02:35 |
| bigjools | yeah I'd prefer that over a try/except | 02:36 |
| bigjools | param to override it fetching from config | 02:36 |
| bigjools | it's amazing what innocuous changes will break elsewhere ... | 02:36 |
| bigjools | thanks for figuring it out dude | 02:37 |
| roaksoax | bigjools: yeah! But anyways, adding the parameter (which should be fairly simply), and removing the line in maas-dns.postinst, we have things back to normal (without having to revert changes back) | 02:37 |
| bigjools | +1 | 02:37 |
| bigjools | if that's definitely the cause, I'm good with it | 02:37 |
| roaksoax | bigjools: yeah I installed maas-cluster-controller-min , and then started running DNS commands manually (those in the psotinst) and that's how I found out what was wrong | 02:38 |
| bigjools | and bazinga | 02:38 |
| roaksoax | indeed | 02:39 |
| roaksoax | bigjools: so do we want maas set_up_dns to receieve a boolean parameter to not try to obtain the value from the DB, or do we want a parameter where we can actually set the forwarder? | 02:52 |
| bigjools | roaksoax: set value is better | 02:53 |
| roaksoax | bigjools: as in. 1. set_up_dns --no-upstream-dns or 2. set_up_dns --upstream-dns={None,1.1.1.1} | 02:53 |
| roaksoax | ack | 02:53 |
| bigjools | roaksoax: in fact do we need the code to fetch it from config at all? | 02:53 |
| bigjools | the postinst is the only place that calls it IIRC | 02:53 |
| roaksoax | bigjools: uhmmm I don't think so, cause celery tasks would also configure that, wouldn't they? | 02:54 |
| bigjools | celery uses a different code path | 02:54 |
| bigjools | this stuff is only a "maas XXX" command | 02:55 |
| roaksoax | bigjools: right, but besides from maas set_up_dns, who sets the upstream_dns when found in the DB? | 02:56 |
| bigjools | roaksoax: celery jobs | 02:56 |
| roaksoax | bigjools: ok, so in reality, we don't really need it in set_up_dns, because the celery job will update that once it is manually added on maas then | 02:57 |
| bigjools | roaksoax: indeed | 02:57 |
| bigjools | that's what I was starting to think | 02:57 |
| roaksoax | bigjools: ok, so this should do it: http://paste.ubuntu.com/6742417/ | 02:59 |
| roaksoax | bigjools: I'll get that tested alongside the packaging change | 02:59 |
| bigjools | +1 | 02:59 |
| roaksoax | bigjools: ok it works : https://code.launchpad.net/~andreserl/maas/no_upstream_dns_on_setup/+merge/201332 | 03:44 |
| bigjools | jtv: super quick pre-imp? | 05:49 |
| jtv | bigjools: sure... firing up laptop. | 05:55 |
| bigjools | jtv: plug it in this time? :) | 05:56 |
| jtv | bigjools: any chance you could give this version of second DHCP detection a whirl? https://code.launchpad.net/~jtv/maas-test/second-dhcp-check/+merge/201232 | 07:02 |
| jtv | This is meant to solve the problem you had detecting the DHCP server you were still running on the same machine you ran maas-test on. | 07:02 |
| jtv | It should _also_ help with the case where the test interface has no IP address. | 07:02 |
| === CyberJacob|Away is now known as CyberJacob | ||
| rvba | jtv: I get a failure when I run maas-test (trunk)… | 10:05 |
| rvba | it seems related to the DHCP stuff: | 10:05 |
| rvba | http://paste.ubuntu.com/6743775/ | 10:05 |
| rvba | It looks (again, I might be wrong) as if we're running 'uvt-kvm ip <>' before the vm is even created… | 10:06 |
| rvba | jtv: does that sound possible? | 10:06 |
| jtv | It seems unlikely — AFAIK we do a "uvt-kvm wait" first. | 10:07 |
| rvba | Ok, I'll investigate… | 10:08 |
| jtv | Looks like the problem you pasted is in get_maas_version, no? | 10:08 |
| jtv | Ah! | 10:08 |
| jtv | I think I know what it is. | 10:08 |
| jtv | More or less. | 10:09 |
| jtv | I made the install_maas() happen earlier than it used to. | 10:09 |
| jtv | Shouldn't be before the "uvt-kvm wait" though. | 10:09 |
| rvba | Looks like the MAAS fixture is setup before the KVM fixture :/ | 10:09 |
| jtv | No, but maas gets installed before the maas fixture is set up. | 10:10 |
| jtv | Crossing changes. :( | 10:10 |
| rvba | (Revision 218 works) | 10:10 |
| jtv | The exception seems to happen the second time install_maas runs. | 10:11 |
| jtv | During MAASFixture.setUp(). | 10:12 |
| jtv | At any rate, if the traceback is correct, everything looks wrong. Why would "uvt-kvm ip" fail that late in the game!? | 10:13 |
| rvba | What second time? install_maas should only be run once. | 10:14 |
| jtv | Ah. That is a change I made, but I guess it's still in the branch I'm working on. | 10:14 |
| jtv | Note that we seem to be running "uvt-kvm ip" for every single command we run on the VM. So you'd expect it to have succeeded at least once before this point. | 10:17 |
| rvba | The vm doesn't seem to exist at all when we run this command. | 10:17 |
| jtv | Missing sudo somewhere maybe? | 10:18 |
| jtv | ← desperate guess | 10:18 |
| rvba | Frankly, I'm really confused by the reorganisation revision 119 introduced. | 10:18 |
| jtv | It just removes some class fixtures and makes them variables in main(). | 10:19 |
| rvba | No, I put a debugging statement in run_command() and we simply don't create the VM at all. | 10:19 |
| rvba | It's a bit more involved than that :). | 10:19 |
| rvba | The result is that KVMFixture.setUp() is not called (or rather, we blow up in install_maas before it's even called). | 10:20 |
| jtv | Ah! | 10:21 |
| jtv | Now I remember. This was a fix I had wanted to have landed, but it got held up by the current packaging trouble. | 10:22 |
| jtv | It's a missing setUp call. | 10:22 |
| rvba | gmb: btw, what I had in mind when I spoke about using run_command to include hardware information in the report is something along the lines of http://paste.ubuntu.com/6744069/ | 11:22 |
| rvba | gmb: of course, we'd want to format the output a bit better. | 11:22 |
| gmb | rvba: Ah, thanks for the tip! I was looking for a way to do that as an API call in the mass-test code, but that seems straightforward enough (formatting notwithstanding). | 11:23 |
| rvba | An API call will also work, but since we have everything setup to shell out on the VM, the solution above might be a bit more easier. | 11:24 |
| rvba | s/more// | 11:24 |
| gmb | Yep, that seems perfectly sane to me. | 11:24 |
| rvba | gmb: for your viewing pleasure (I've always been very found of that expression), here is the result of that script on my test machine: http://paste.ubuntu.com/6744089/ | 11:26 |
| gmb | rvba: Lovely, thank you. Although when I say lovely I may actually mean "gaaaagh." But I'm not to worried about making it nice as I am about making it functional right now. | 11:27 |
| rvba | Getting it to work sounds like a good first step to me :). | 11:28 |
| rvba | gmb: actually, now that I think about it, it might be ever more simple to just do a "maas dumpdata." | 11:31 |
| gmb | rvba: Does that basically give us the whole DB? | 11:31 |
| rvba | Yeah. | 11:31 |
| gmb | More data == more better. | 11:31 |
| rvba | But the DB won't contain anything big so it's safe. | 11:31 |
| gmb | Yep. | 11:31 |
| gmb | Excellent. | 11:31 |
| rvba | Plus we're sure we're not missing anything. | 11:31 |
| rvba | self.maas.kvm_fixture.run_command(['sudo', 'maas', 'dumpdata']) | 11:32 |
| gmb | Brill. | 11:32 |
| * gmb does that. | 11:32 | |
| rvba | I'll give you an example of what you get back in a minute… | 11:34 |
| rvba | (I killed my vm :/) | 11:34 |
| rvba | gmb: http://paste.ubuntu.com/6744188/ | 11:52 |
| gmb | rvba: Why all the leading whitespace? | 11:53 |
| rvba | gmb: don't know… looks like it's an artifact created by paste of something, here is the file verbatim: http://people.canonical.com/~rvb/dump | 11:54 |
| gmb | *eyebleed* | 11:55 |
| gmb | rvba: My feeling is we should just dump it, attach it, and care about cleaning it up later. Thoughts? | 11:55 |
| rvba | gmb: I agree. It's pure JSON so it's easy to parse. | 11:56 |
| rvba | gmb: as a second step, if we still want to do that, it's easy to exact identifiers from the lshw output. | 11:57 |
| gmb | rvba: Okay. I'll finish this branch for gathering it, then I'll look at extracting stuff that's useful. | 11:58 |
| rvba | Sounds good to me. | 11:58 |
| === CyberJacob is now known as CyberJacob|Away | ||
| * gmb -> lunch | 12:28 | |
| === matsubara_ is now known as matsubara | ||
| tomixxx | hi | 17:10 |
| tomixxx | i have the problem that my nodes stay in state "commissioning" | 17:10 |
| tomixxx | they never get "ready" | 17:10 |
| === CyberJacob|Away is now known as CyberJacob | ||
| === CyberJacob is now known as CyberJacob|Away | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!