=== jfarschman is now known as MilesDenver
=== jfarschman is now known as MilesDenver
=== jfarschman is now known as MilesDenver
=== jfarschman is now known as MilesDenver
=== jfarschman is now known as MilesDenver
alexmcwhirterHey guys, im having issues with nodes failing to fully PXE boot. They get stuck at IP-Config requesting an IP address from the MAAS server but never getting a response. The cluster controller doesn’t ever see these requests either.04:50
alexmcwhirterA bug report was opened for similar issue https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/132741204:52
ubot5Launchpad bug 1327412 in initramfs-tools (Ubuntu) "Delay during PXE Boot, IP-Config gives up" [High,New]04:52
=== jfarschman is now known as MilesDenver
=== CyberJacob|Away is now known as CyberJacob
=== jfarschman is now known as MilesDenver
=== CyberJacob is now known as CyberJacob|Away
=== jfarschman is now known as MilesDenver
=== jfarschman is now known as MilesDenver
=== alexpilotti_ is now known as alexpilotti
=== jfarschman is now known as MilesDenver
=== Guest30538 is now known as wallyworld
jtvrvba: I wonder if that migration (1) shouldn't break in the database anyway because it updates a column that was added in the same transaction, and (2) won't cause trouble updating a field that _on the model_ hasn't been added yet.10:34
jtvIn SQL terms, what I'd expect that migration to do is: update the node table using a left outer join with the tag table where the tag is for the fastpath installer; and set the newly added column to one value or the other depending.10:36
jtvInstead, it loops over all nodes, queries the tags table for each, updates a model field that hasn't been fully added yet, then writes back that one node.10:37
jtvIt was this MP wot did it: https://code.launchpad.net/~blake-rouse/maas/add-boot-type-to-node/+merge/22723410:44
=== jfarschman is now known as MilesDenver
jtvrvba: I'm working on that broken migration.10:56
bigjoolsthanks jtv10:57
bigjoolsrvba: I'll review your branch if you review mind :)10:57
=== jfarschman is now known as MilesDenver
=== jfarschman is now known as MilesDenver
rvbabigjools: well, it's always the same question I have with twisted: how to *start* the damn thing. What am I doing wrong here: http://paste.ubuntu.com/7841796/ ?12:53
bigjoolsrvba: looking12:53
bigjoolsrvba: aiee12:54
bigjoolsrvba: you don't need the callLater12:55
rvbabigjools: well, I tried without first12:55
rvbabigjools: what you see now is the hole I ended up in after trying different things12:56
bigjoolsbasically it's going to be hard the way the code is written12:56
bigjoolsbecause of inline callbacks12:56
bigjoolsthe function won't exit until it's finished12:56
bigjoolsbut you need to do assertions while it's running12:56
rvbaIndeed, but the function yields at various points.12:57
bigjoolsI forgot what it's calling12:57
rvbaIsn't that enough to let us do the checks we want to do?12:57
bigjoolspower_action, right12:57
bigjoolsno, because it yields to the reactor12:58
bigjoolslet me show you something I did recently12:58
bigjoolsrvba: also fwiw I think you need some test refactoring to reduce the boilerplate13:02
bigjoolsrvba: move the clock.advance() to the first statement after the "for"13:08
bigjoolsadvance just runs the reactor for one iteration so you need to do that to get your callLater to run13:09
bigjoolsthe test now fails because it's actually incorrect :)13:09
=== jfarschman is now known as MilesDenver
=== jfarschman is now known as MilesDenver
rvbabigjools: doens't seem to work.13:57
bigjoolsrvba: it doesn't work because your test is wrong13:57
rvbaah :)13:57
bigjoolsit needs updating to cope with this13:57
bigjoolsrvba: you could try advance(0), I've never done that though13:58
bigjoolsfor the first iteration I mean13:58
rvbabigjools: I've tried that first actually.  Ni dice.13:59
bigjoolsrvba: 0.1 then13:59
bigjoolsyou just need to spin the reactor once to get the callLater to run13:59
bigjoolsbut not enough to get the retries to start yet13:59
rvbaI've tried that too :)14:01
bigjoolsrvba: what happened?14:01
=== jfarschman is now known as MilesDenver
rvbabigjools: the first check failed because nothing had been called.14:05
bigjoolsrvba: weird.  did you try 1?14:06
rvbabigjools: with 1 it fails during the second check.  Maybe my test has a problem… let me have another look…14:07
bigjoolsrvba: yes, there is a problem, it's not taking into account the initial calls I think14:07
rvbabigjools: it doesn't look as if clock.advance is doing its job: the code inside the loop in change_power_state() only gets called once.14:36
bigjoolsrvba: the balance of probability is that your code is buggy then :)14:36
bigjoolsrvba: give me a few minutes to finish my branch and then you can have my full attention14:37
rvbabigjools: either that or I'm not using the right magic formula in my test.14:37
rvba    raise mismatch_error14:38
rvbaMismatchError: !=:14:38
rvbareference = [call(context-key-4XklLw=u'context-val-TYmWJk', power_change=u'on')]14:38
rvbaactual    = [call(context-key-4XklLw=u'context-val-TYmWJk', power_change=u'on')]14:38
rvba: after <function get_mock_calls> on <Mock name='mock().execute' id='140383727837328'>: calls do not match14:38
=== Solution-X is now known as Solution-X|AFK
rvbabigjools: I've pushed the latest version of the code to lp:~rvb/maas/retry-power-changes15:05
rvba./bin/test.maas src/provisioningserver/rpc/tests/test_power.py:TestPowerHelpers.test_change_power_state_pauses_in_between_retries15:05
bigjoolsrvba: it works now?15:08
rvbaObviously not :)15:08
bigjoolsrvba: ok I'll take a look15:08
rvbaThe clock.advance doesn't seem to have any effect on the tested method resuming its execution.15:09
bigjoolsrvba: test.maas for pserv tests?15:09
rvbaThe call* to clock.advance15:09
bigjoolsuse test.pserv!15:09
rvbaIt works.15:09
bigjoolsit's slow15:09
rvbaCan you run just one test with test.pserv?15:10
bigjoolsrvba: provisioningserver.rpc.tests.test_power:TestPowerHelpers.test_change_power_state_pauses_in_between_retries15:10
bigjoolsrvba: bin/test.pserv provisioningserver.rpc.tests.test_power:TestPowerHelpers.test_change_power_state_pauses_in_between_retries15:10
bigjoolsnotice how quick it is ;)15:10
rvbaOkay, cool.15:10
rvbaTrue, it's quick15:11
bigjoolsrvba: would your DNS CNAME removal branch easily backport to 1.5?15:13
rvbabigjools: I think so; the interface hasn't been changed, only the internals.15:14
bigjoolsyeah I thought as much, thanks15:14
rvbabigjools: if I debug what happens when I reach the clock.advance in the test, I can see that right after existing the twisted code, the code runs the second execution of the loop… in other words, *nothing* calls the method under test again.15:20
bigjoolsrvba: put a breakpoint at the end of the method and see if it's reached15:22
rvbabigjools: it's not15:22
rvbadid that15:22
bigjoolsso it exits at the yield15:22
rvbabigjools: reading the twisted code, I wonder if Clock is the right tool to use here.  It seems it's only meant to write unit tests that use callLater15:23
bigjoolsnah it's fine15:23
bigjoolshmmm so when I tested something similar in my own code I patched out deferToThread15:23
bigjoolsso you're creating threads in here15:24
rvbaDamn, that did the trick.15:25
* rvba can't believe it. Let me put a breakpoint and see it with my own eyes.15:25
bigjoolsI suspect Clock() won't work with threads15:26
bigjoolsrvba: I would patch out deferToThread so it just returns defer.succeed15:32
rvbaself.patch(power, "deferToThread", maybeDeferred) is what I did15:32
bigjoolssame thing15:34
bigjoolsrvba: so did you get it working?15:44
rvbabigjools: the test passes15:44
rvbaI'm seeing if I can't break that giant function into smaller pieces that could be unit tested separately.15:45
rvbaWell, I'm sure I can but the question: will the resulting code still be readable?15:45
rvbathe question is*15:45
bigjoolsI am sure an engineer of your high calibre will do just fine :)15:45
bigjoolsrvba: I was thinking, you might not need the callLater.15:46
bigjoolsmake the test use inlineCallbacks15:46
schegi_question on the virsh power settings for a node can anyone help? i configured them like this 'power address' qemu+ssh://ubuntu@ and 'power id' vm1 but it does not power on the vm. running 'sudo -u schegi virsh -c qemu+ssh://ubuntu@ start vm1' where schegi is my maas user works.15:46
bigjoolsschegi_: http://maas.ubuntu.com/docs/nodes.html#virtual-machine-nodes15:47
bigjoolsrvba: then you can yield the first result, followed by advancing the clock after that.15:48
rvbabigjools: yeah, I'll try… let me see if I can break the methods into smaller pieces first…15:49
bigjoolsrvba: good call.15:49
bigjoolswill be much much easier to test15:49
schegi_bigjools, already found that but the problem is that running the command from the console powers on the machine. starting the vm by maas doesnt work15:49
bigjoolsschegi_: two things to check then, 1. is the cluster celery process running? 2. look in its log for errors15:50
schegi_bigjools, seems that the login to the virsh console somehow fails http://pastebin.com/QfH9Ybqa15:54
jpdsmaas has a broken piece in trusty.15:55
jpds /var/log/maas/rsyslog has the wrong permissions by default.15:55
bigjoolsschegi_: can't help you with that, sorry15:57
bigjoolsjpds: someone filed a bug the other day15:57
rvbabigjools: http://paste.ubuntu.com/7842616/ (a first version with a tiny bit extracted); I think it can be done better.  Is there a better pattern that having sub-methods like what I've done with doer() and checker()… I could go all the way and get rid of the inlineCallbacks entirely…16:01
rvbaI'm starting to see why twisted is fun :).16:02
jpdsbigjools: Do you have the bug handy/16:03
bigjoolsjpds: sorry, not to hand16:06
bigjoolsbusy atm, will look in a bit16:07
bigjoolsrvba: one mo16:07
bigjoolsrvba: ok I can look now16:16
bigjoolsrvba: you could enhance retries() to take a sequence16:17
bigjoolsbut up to you16:17
rvbabigjools: what do you mean by that?16:18
bigjoolsrvba: the retries util that gavin wrote just takes fixed intervals16:18
bigjoolsa fixed interval, I mean16:19
rvbabigjools: I'm doing something different here: there is no global timeout.16:20
rvbabigjools: the thing I don't like in my code is the need to encapsulate actions into tiny methods that take no arguments (checker and doer).16:22
bigjoolsrvba: presumably you're doing that so it's easier to monkey patch?16:23
rvbabigjools: no, only to have something contained to pass to retry()16:23
bigjoolsrvba: my brain is starting to shut down for the day, I might not be much more help!16:30
rvbabigjools: you've been very helpful thus far!  I'll refine my code a bit more so that it marks the node as broken when the template execution fails.16:31
bigjoolsrvba: TDD is generally even more important when writing twisted code16:32
rvbabigjools: right, I can see that now.16:32
bigjoolsrvba: and sometimes I eschew inlinecallbacks in favour of passing deferreds around just to make tests easier16:32
bigjoolsit's quite nice being able to add a test callback on one of the deferreds from the code being tested16:33
rvbabigjools: not just that.  In this particular case, not using inlinecallbacks means dealing with errors that need to be treated the same way but are happening at various point in the callback tree is easier.16:33
bigjoolsrvba: the downside means you end up writing lambdas or inline functions16:34
rvbabigjools: another thing is that using the Deferred pattern (vs inlineCallback) forces you to think in terms or success vs. failure.  It structure the code paths around that distinction.16:49
rvbas/terms or/terms of/16:49
bigjoolsrvba: well yield is more like normal python16:50
bigjoolsexceptions and so on, so not sure about that.16:50
rvbabigjools: if you compare my code vs the previous version, handling failures in the "deferred" version comes more naturally.16:50
allenapbigjools: Do you know why the MAAS Daily Builds PPA appears to be fairly out of date?17:14
allenapOr am I misunderstanding?17:14
allenapI want a package built from trunk.17:15
allenapFor Trusty.17:15
bigjoolshang on17:15
allenapI /could/ build it myself.17:15
bigjoolsallenap: it got switched to utopic17:16
allenapbigjools: Oh.17:16
bigjoolsallenap: dailies for trusty are in here https://launchpad.net/~maas-maintainers/+archive/ubuntu/experimental/+packages17:17
allenapbigjools: Can we build trunk packages for Trusty and Utopic?17:17
bigjoolsallenap: it's complicated :)  can't upload the same thing twice to a PPA, so we'd have to reversion stuff in the recipes to include the series info17:17
bigjoolsthis situation needs reviewing I think17:18
allenapYeah. My lack of knowledge means I resent the arcane dark art of packaging even more.17:19
bigjoolsallenap: I've been building my own lately17:19
bigjoolssince I made it *so* easy ;)17:19
bigjoolsallenap: talking of which, can you remember why, after installing packages, pserv fails to start with:17:21
bigjools/usr/bin/twistd: Unknown command: maas-pserv17:21
bigjoolsplugin not found, obviously, but why17:22
allenapbigjools: No, sorry, but I can help diagnose. Not now - kids have just returned and need feeding - but after 8pm.17:23
bigjoolsallenap: at 8pm I'll be semi-comatose.  better leave it to tomorrow17:23
allenapbigjools: Okay.17:24
bigjoolslook at the time!17:24
bigjoolsI keep thinking "team call in 5 minutes..."17:24
bigjoolsallenap: oh ah I see what's up, plugin failed to load17:29
bigjoolsexceptions.AttributeError: 'Settings' object has no attribute 'WORKER_QUEUE_DNS'17:29
bigjoolsbut why... question for tomorrow!  good night17:29
=== CyberJacob|Away is now known as CyberJacob
schegi_got still problems with powercontrol of an virsh vm node. already configured my maas user to be able to perform pwd-less virsh command in the terminal, but doing the same via maas is not possible.18:12
schegi_e.g commissioning node would not start node with power settings qemu+ssh://ubuntu@ id: vm1, but performing virsh -c qemu+ssh://ubuntu@ console vm1 in the terminal starts node. celeryd is running and log says provisioningserver.custom_hardware.virsh.VirshError: Failed to login to virsh console..18:15
=== jfarschman is now known as MilesDenver
=== jfarschman is now known as MilesDenver
Waruiihi. I have a newly installed on a mainly vanilla 14.04, but maas does not import images. pserv.log tells me "AssertionError: MAAS_URL is not set.  This probably means that the script which started this program failed to source maas_cluster.conf", but access rights look fine to me19:42
WaruiiI also see repeatedly two(!) instances of HttpClientFactory (http://localhost/MAAS/rpc/) booting up and being stopped immediately19:44
jseutterWaruii: I remember seeing the same error with my maas install but I don't remember how I fixed it.  From looking at https://maas.ubuntu.com/docs/troubleshooting.html, sudo dpkg-reconfigure maas-region-controller may be what you need.20:08
jseutterWaruii: on my install, I have maas_cluster.conf:MAAS_URL=http://localhost/MAAS20:09
jseuttersorry, that's in /etc/maas/maas_cluster.conf20:10
Waruiijseutter: config says the same for me. just did the dpkg-reconfigure to no avail20:21
jseutterWaruii: Are you trying to install maas from packages on an existing 14.04 install?20:36
jseutterWaruii: If so, check your installed packages against mine - http://pastebin.ubuntu.com/7843901/20:38
Waruiiinstalled the "maas" metapackage after a vanilla 14.04 installation20:39
jseutterWaruii: did you do the sudo add-apt-repository cloud-archive:tools before installing the maas metapackage?20:43
Waruiidoes not ring a bell20:44
jseutterWaruii: I had a couple of failed attempts, it finally worked for me when I followed https://maas.ubuntu.com/docs/install.html closely.  It could be that the maas packages in the standard archive do not work as well as the ones in the cloud archive20:45
jseutterWaruii: actually20:46
jseutterWaruii: On my system, the cloud-tools archive has been disabled by distUpgrade, so I must have gotten maas working on 12.04 and then done a dist-upgrade to 14.04.20:47
=== jfarschman is now known as MilesDenver
WaruiiI thought I followed it but I will give it another try … tomorrow ;)20:52
Waruiithx though!20:52
jseutterWaruii: That add-apt-repository command is described as optional in the docs, so if one way doesn't work you could try the othre20:55
=== CyberJacob is now known as CyberJacob|Away

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!