=== jfarschman is now known as MilesDenver === jfarschman is now known as MilesDenver === jfarschman is now known as MilesDenver === jfarschman is now known as MilesDenver === jfarschman is now known as MilesDenver [04:50] Hey guys, im having issues with nodes failing to fully PXE boot. They get stuck at IP-Config requesting an IP address from the MAAS server but never getting a response. The cluster controller doesn’t ever see these requests either. [04:52] A bug report was opened for similar issue https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1327412 [04:52] Launchpad bug 1327412 in initramfs-tools (Ubuntu) "Delay during PXE Boot, IP-Config gives up" [High,New] === jfarschman is now known as MilesDenver === CyberJacob|Away is now known as CyberJacob === jfarschman is now known as MilesDenver === CyberJacob is now known as CyberJacob|Away === jfarschman is now known as MilesDenver === jfarschman is now known as MilesDenver === alexpilotti_ is now known as alexpilotti === jfarschman is now known as MilesDenver === Guest30538 is now known as wallyworld [10:34] rvba: I wonder if that migration (1) shouldn't break in the database anyway because it updates a column that was added in the same transaction, and (2) won't cause trouble updating a field that _on the model_ hasn't been added yet. [10:36] In SQL terms, what I'd expect that migration to do is: update the node table using a left outer join with the tag table where the tag is for the fastpath installer; and set the newly added column to one value or the other depending. [10:37] Instead, it loops over all nodes, queries the tags table for each, updates a model field that hasn't been fully added yet, then writes back that one node. [10:44] It was this MP wot did it: https://code.launchpad.net/~blake-rouse/maas/add-boot-type-to-node/+merge/227234 === jfarschman is now known as MilesDenver [10:56] rvba: I'm working on that broken migration. [10:57] thanks jtv [10:57] rvba: I'll review your branch if you review mind :) [10:57] mine* === jfarschman is now known as MilesDenver === jfarschman is now known as MilesDenver [12:53] bigjools: well, it's always the same question I have with twisted: how to *start* the damn thing. What am I doing wrong here: http://paste.ubuntu.com/7841796/ ? [12:53] rvba: looking [12:54] rvba: aiee [12:55] rvba: you don't need the callLater [12:55] bigjools: well, I tried without first [12:56] bigjools: what you see now is the hole I ended up in after trying different things [12:56] basically it's going to be hard the way the code is written [12:56] because of inline callbacks [12:56] the function won't exit until it's finished [12:56] but you need to do assertions while it's running [12:57] Indeed, but the function yields at various points. [12:57] I forgot what it's calling [12:57] Isn't that enough to let us do the checks we want to do? [12:57] power_action, right [12:58] no, because it yields to the reactor [12:58] let me show you something I did recently [13:02] rvba: also fwiw I think you need some test refactoring to reduce the boilerplate [13:08] rvba: move the clock.advance() to the first statement after the "for" [13:09] advance just runs the reactor for one iteration so you need to do that to get your callLater to run [13:09] the test now fails because it's actually incorrect :) === jfarschman is now known as MilesDenver === jfarschman is now known as MilesDenver [13:57] bigjools: doens't seem to work. [13:57] doesn't* [13:57] rvba: it doesn't work because your test is wrong [13:57] ah :) [13:57] it needs updating to cope with this [13:58] rvba: you could try advance(0), I've never done that though [13:58] for the first iteration I mean [13:59] bigjools: I've tried that first actually. Ni dice. [13:59] rvba: 0.1 then [13:59] :) [13:59] you just need to spin the reactor once to get the callLater to run [13:59] but not enough to get the retries to start yet [14:01] I've tried that too :) [14:01] rvba: what happened? === jfarschman is now known as MilesDenver [14:05] bigjools: the first check failed because nothing had been called. [14:06] rvba: weird. did you try 1? [14:07] bigjools: with 1 it fails during the second check. Maybe my test has a problem… let me have another look… [14:07] rvba: yes, there is a problem, it's not taking into account the initial calls I think [14:36] bigjools: it doesn't look as if clock.advance is doing its job: the code inside the loop in change_power_state() only gets called once. [14:36] rvba: the balance of probability is that your code is buggy then :) [14:37] rvba: give me a few minutes to finish my branch and then you can have my full attention [14:37] bigjools: either that or I'm not using the right magic formula in my test. [14:38] raise mismatch_error [14:38] MismatchError: !=: [14:38] reference = [call(context-key-4XklLw=u'context-val-TYmWJk', power_change=u'on')] [14:38] actual = [call(context-key-4XklLw=u'context-val-TYmWJk', power_change=u'on')] [14:38] : after on : calls do not match [14:38] Nice [14:39] wat === Solution-X is now known as Solution-X|AFK [15:05] bigjools: I've pushed the latest version of the code to lp:~rvb/maas/retry-power-changes [15:05] ./bin/test.maas src/provisioningserver/rpc/tests/test_power.py:TestPowerHelpers.test_change_power_state_pauses_in_between_retries [15:08] rvba: it works now? [15:08] Obviously not :) [15:08] rvba: ok I'll take a look [15:09] The clock.advance doesn't seem to have any effect on the tested method resuming its execution. [15:09] rvba: test.maas for pserv tests? [15:09] The call* to clock.advance [15:09] use test.pserv! [15:09] It works. [15:09] it's slow [15:10] Can you run just one test with test.pserv? [15:10] rvba: provisioningserver.rpc.tests.test_power:TestPowerHelpers.test_change_power_state_pauses_in_between_retries [15:10] rvba: bin/test.pserv provisioningserver.rpc.tests.test_power:TestPowerHelpers.test_change_power_state_pauses_in_between_retries [15:10] sigfh [15:10] notice how quick it is ;) [15:10] Okay, cool. [15:11] True, it's quick [15:13] rvba: would your DNS CNAME removal branch easily backport to 1.5? [15:14] bigjools: I think so; the interface hasn't been changed, only the internals. [15:14] yeah I thought as much, thanks [15:20] bigjools: if I debug what happens when I reach the clock.advance in the test, I can see that right after existing the twisted code, the code runs the second execution of the loop… in other words, *nothing* calls the method under test again. [15:22] rvba: put a breakpoint at the end of the method and see if it's reached [15:22] bigjools: it's not [15:22] did that [15:22] pok [15:22] so it exits at the yield [15:23] bigjools: reading the twisted code, I wonder if Clock is the right tool to use here. It seems it's only meant to write unit tests that use callLater [15:23] nah it's fine [15:23] hmmm so when I tested something similar in my own code I patched out deferToThread [15:24] so you're creating threads in here [15:25] Damn, that did the trick. [15:25] haha [15:25] * rvba can't believe it. Let me put a breakpoint and see it with my own eyes. [15:26] I suspect Clock() won't work with threads [15:32] rvba: I would patch out deferToThread so it just returns defer.succeed [15:32] self.patch(power, "deferToThread", maybeDeferred) is what I did [15:34] same thing [15:34] nearly [15:44] rvba: so did you get it working? [15:44] bigjools: the test passes [15:44] hurrah [15:45] I'm seeing if I can't break that giant function into smaller pieces that could be unit tested separately. [15:45] Well, I'm sure I can but the question: will the resulting code still be readable? [15:45] the question is* [15:45] I am sure an engineer of your high calibre will do just fine :) [15:46] rvba: I was thinking, you might not need the callLater. [15:46] make the test use inlineCallbacks [15:46] question on the virsh power settings for a node can anyone help? i configured them like this 'power address' qemu+ssh://ubuntu@192.168.25.14/system and 'power id' vm1 but it does not power on the vm. running 'sudo -u schegi virsh -c qemu+ssh://ubuntu@192.168.25.14/system start vm1' where schegi is my maas user works. [15:47] schegi_: http://maas.ubuntu.com/docs/nodes.html#virtual-machine-nodes [15:48] rvba: then you can yield the first result, followed by advancing the clock after that. [15:49] bigjools: yeah, I'll try… let me see if I can break the methods into smaller pieces first… [15:49] method* [15:49] rvba: good call. [15:49] will be much much easier to test [15:49] bigjools, already found that but the problem is that running the command from the console powers on the machine. starting the vm by maas doesnt work [15:50] schegi_: two things to check then, 1. is the cluster celery process running? 2. look in its log for errors [15:54] bigjools, seems that the login to the virsh console somehow fails http://pastebin.com/QfH9Ybqa [15:55] maas has a broken piece in trusty. [15:55] /var/log/maas/rsyslog has the wrong permissions by default. [15:57] schegi_: can't help you with that, sorry [15:57] jpds: someone filed a bug the other day [16:01] bigjools: http://paste.ubuntu.com/7842616/ (a first version with a tiny bit extracted); I think it can be done better. Is there a better pattern that having sub-methods like what I've done with doer() and checker()… I could go all the way and get rid of the inlineCallbacks entirely… [16:02] I'm starting to see why twisted is fun :). [16:03] bigjools: Do you have the bug handy/ [16:06] jpds: sorry, not to hand [16:07] busy atm, will look in a bit [16:07] rvba: one mo [16:16] rvba: ok I can look now [16:17] rvba: you could enhance retries() to take a sequence [16:17] but up to you [16:18] bigjools: what do you mean by that? [16:18] rvba: the retries util that gavin wrote just takes fixed intervals [16:19] a fixed interval, I mean [16:20] bigjools: I'm doing something different here: there is no global timeout. [16:20] ok [16:22] bigjools: the thing I don't like in my code is the need to encapsulate actions into tiny methods that take no arguments (checker and doer). [16:23] rvba: presumably you're doing that so it's easier to monkey patch? [16:23] bigjools: no, only to have something contained to pass to retry() [16:30] rvba: my brain is starting to shut down for the day, I might not be much more help! [16:31] bigjools: you've been very helpful thus far! I'll refine my code a bit more so that it marks the node as broken when the template execution fails. [16:31] sweet [16:32] rvba: TDD is generally even more important when writing twisted code [16:32] bigjools: right, I can see that now. [16:32] rvba: and sometimes I eschew inlinecallbacks in favour of passing deferreds around just to make tests easier [16:33] it's quite nice being able to add a test callback on one of the deferreds from the code being tested [16:33] bigjools: not just that. In this particular case, not using inlinecallbacks means dealing with errors that need to be treated the same way but are happening at various point in the callback tree is easier. [16:34] points* [16:34] right [16:34] rvba: the downside means you end up writing lambdas or inline functions [16:34] Indeed. [16:49] bigjools: another thing is that using the Deferred pattern (vs inlineCallback) forces you to think in terms or success vs. failure. It structure the code paths around that distinction. [16:49] s/terms or/terms of/ [16:50] rvba: well yield is more like normal python [16:50] exceptions and so on, so not sure about that. [16:50] bigjools: if you compare my code vs the previous version, handling failures in the "deferred" version comes more naturally. [17:14] bigjools: Do you know why the MAAS Daily Builds PPA appears to be fairly out of date? [17:14] Or am I misunderstanding? [17:15] I want a package built from trunk. [17:15] For Trusty. [17:15] hang on [17:15] I /could/ build it myself. [17:16] allenap: it got switched to utopic [17:16] bigjools: Oh. [17:17] allenap: dailies for trusty are in here https://launchpad.net/~maas-maintainers/+archive/ubuntu/experimental/+packages [17:17] bigjools: Can we build trunk packages for Trusty and Utopic? [17:17] allenap: it's complicated :) can't upload the same thing twice to a PPA, so we'd have to reversion stuff in the recipes to include the series info [17:18] this situation needs reviewing I think [17:19] Yeah. My lack of knowledge means I resent the arcane dark art of packaging even more. [17:19] allenap: I've been building my own lately [17:19] since I made it *so* easy ;) [17:21] allenap: talking of which, can you remember why, after installing packages, pserv fails to start with: [17:21] /usr/bin/twistd: Unknown command: maas-pserv [17:22] plugin not found, obviously, but why [17:23] bigjools: No, sorry, but I can help diagnose. Not now - kids have just returned and need feeding - but after 8pm. [17:23] allenap: at 8pm I'll be semi-comatose. better leave it to tomorrow [17:24] bigjools: Okay. [17:24] thanks [17:24] Bye! [17:24] look at the time! [17:24] I keep thinking "team call in 5 minutes..." [17:29] allenap: oh ah I see what's up, plugin failed to load [17:29] exceptions.AttributeError: 'Settings' object has no attribute 'WORKER_QUEUE_DNS' [17:29] but why... question for tomorrow! good night === CyberJacob|Away is now known as CyberJacob [18:12] got still problems with powercontrol of an virsh vm node. already configured my maas user to be able to perform pwd-less virsh command in the terminal, but doing the same via maas is not possible. [18:15] e.g commissioning node would not start node with power settings qemu+ssh://ubuntu@192.168.25.14/system id: vm1, but performing virsh -c qemu+ssh://ubuntu@192.168.25.14/system console vm1 in the terminal starts node. celeryd is running and log says provisioningserver.custom_hardware.virsh.VirshError: Failed to login to virsh console.. === jfarschman is now known as MilesDenver === jfarschman is now known as MilesDenver [19:42] hi. I have a newly installed on a mainly vanilla 14.04, but maas does not import images. pserv.log tells me "AssertionError: MAAS_URL is not set. This probably means that the script which started this program failed to source maas_cluster.conf", but access rights look fine to me [19:44] I also see repeatedly two(!) instances of HttpClientFactory (http://localhost/MAAS/rpc/) booting up and being stopped immediately [20:08] Waruii: I remember seeing the same error with my maas install but I don't remember how I fixed it. From looking at https://maas.ubuntu.com/docs/troubleshooting.html, sudo dpkg-reconfigure maas-region-controller may be what you need. [20:09] Waruii: on my install, I have maas_cluster.conf:MAAS_URL=http://localhost/MAAS [20:10] sorry, that's in /etc/maas/maas_cluster.conf [20:21] jseutter: config says the same for me. just did the dpkg-reconfigure to no avail [20:36] Waruii: Are you trying to install maas from packages on an existing 14.04 install? [20:38] Waruii: If so, check your installed packages against mine - http://pastebin.ubuntu.com/7843901/ [20:39] installed the "maas" metapackage after a vanilla 14.04 installation [20:43] Waruii: did you do the sudo add-apt-repository cloud-archive:tools before installing the maas metapackage? [20:44] does not ring a bell [20:45] Waruii: I had a couple of failed attempts, it finally worked for me when I followed https://maas.ubuntu.com/docs/install.html closely. It could be that the maas packages in the standard archive do not work as well as the ones in the cloud archive [20:46] Waruii: actually [20:47] Waruii: On my system, the cloud-tools archive has been disabled by distUpgrade, so I must have gotten maas working on 12.04 and then done a dist-upgrade to 14.04. [20:47] nuts. === jfarschman is now known as MilesDenver [20:52] I thought I followed it but I will give it another try … tomorrow ;) [20:52] thx though! [20:55] Waruii: That add-apt-repository command is described as optional in the docs, so if one way doesn't work you could try the othre [20:56] other === CyberJacob is now known as CyberJacob|Away