/srv/irclogs.ubuntu.com/2014/09/11/#maas.txt

mbruzekhello bigjools04:00
mbruzekDo you have a minute to help with a maas question?04:00
mbruzek/etc/maas/dhcp.conf is not being populated and i can not start the maas-dhcp-server04:01
=== CyberJacob|Away is now known as CyberJacob
=== kickinz1|afk is now known as kickinz1
=== marcoceppi is now known as rosales
=== rosales is now known as marcoceppi
bigjoolsrvba: I looked at the power error log thing07:23
bigjoolsthe only solution for this, apart from a python rewrite, is to write the generated template to a file and execute it07:24
rvbabigjools: can't we use our tiny wrapper that we use everywhere to execute shell scripts and specialize the way errors are built?07:25
bigjoolswhat tiny wrapper?07:25
rvbaOr rather, can't we tweak how we build the PowerActionFail error (http://paste.ubuntu.com/8316602/)?07:27
rvbaThat's the wrapper I had in mind, it's only a wrapper (so to speak) around the error.07:28
bigjoolsoh and CI is failing :(07:29
bigjoolsjuju bootstrap07:29
rvbaDamn.07:29
rvbaConsistently?07:29
* rvba checks07:29
bigjoolsyes07:29
bigjoolsrvba: no we cannot tweak it, the error text is returned from check_output07:29
rvbabigjools: well, the error we get from check_output has all the information we need stored on it.  Only it's __str__ method produces garbage in our case.07:33
rvbabigjools: subprocess.py http://paste.ubuntu.com/8316633/07:33
bigjoolsrvba: it's not garbage, check output does something like "command: error text"07:33
bigjoolsand command in our case is the template07:33
bigjoolsit's abusing the check_output stuff really07:34
rvbaYeah, I know; but for power command the template is too much information to put in an event :).07:34
rvbacommands*07:34
bigjoolswhat I might do is use subprocess.communicate and grab stderr separately07:34
bigjoolsalthough the power templates (I'm looking at AMT) produce  way more output than required07:35
rvbaExactly07:35
rvbabigjools: it looks to me that the problem is line 6 here http://paste.ubuntu.com/8316645/07:35
rvbaBecause we call the exception's __str__ method.07:35
rvbaAnd for errors generated by power templates failing, it's guaranteed that it contains the whole template.07:36
bigjoolsrvba: we can cheat and grab e.output I suppose07:36
bigjoolsbut the template needs fixing07:36
rvbaIndeed.07:37
rvbabigjools: but yeah, we could use e.output and e.returncode and not use e.cmd (as e.__str__ does).07:38
bigjoolsrvba: oh god look at what it's doing07:38
bigjoolsline 6 does the __str__07:38
bigjoolsthen line 10 does e.output *again*07:38
rvbaheh, true :)07:39
bigjoolsI'll propose a change07:39
bigjoolseasier than I thought07:39
bigjoolsin the meantime, please review my branch :)07:40
rvbaI'd love to.07:40
rvbaAlready approved by Dr. jtv it seems.07:41
jtvSorry.  :-P07:42
jtvAnd by the way I never finished that PhD...07:42
=== CyberJacob is now known as CyberJacob|Away
bigjoolsaha07:43
rvbajtv: I know… it was more of honorific title.  For some reason I was thinking about 'Doc' in "Back to the Future". ;)07:46
jtvI'll just take that as a compliment.  :)07:47
bigjoolsrvba: I'm the one with unruly grey hair, not jtv08:24
jtvI don't know how to break this to you but... it's not unthinkable that I'm slightly weirder.08:25
cariboujtv: I'm preparing the SRU for bug #134670310:15
ubot5`bug 1346703 in maas (Ubuntu) "/var/log/maas/rsyslog has incorrect permission" [Medium,In progress] https://launchpad.net/bugs/134670310:15
jtvHi caribou.  Excellent.10:16
cariboushould I do a debdiff against trusty-proposed or another MP ?10:16
caribouI would say it depens on who sponsor my upload10:16
jtvcaribou: I don't recall having done any backports on the packaging branch myself, so not sure.10:17
cariboujtv: I'll attach a debdiff & add sponsors & SRU team; this is what I usually do10:18
cariboujtv: if needed I'll change it later no big deal10:18
jtvYou might ask bigjools, but I think it's past the end of his day now.10:18
gmballenap, rvba, jtv: https://code.launchpad.net/~gmb/maas/enlist-mscm-to-RPC/+merge/234279 and https://code.launchpad.net/~gmb/maas/enlist-uscm-to-RPC/+merge/234285 need reviewing when you’ve got a sec.10:19
* jtv has a sec10:19
rvbagmb: I'll review your other branch.10:19
cariboujtv: will do, I should be able to catch him later10:19
gmbjtv, rvba: ta10:19
jtvRace conditions ftw10:21
jtvAt last I find out what happens when you try to claim a review that someone else has just claimed.10:21
rvbaallenap: I wonder if what's seeing in the lab is not two bugs combined.10:36
rvbawhat we are*10:36
rvbaallenap: I see two errors:10:36
rvbamaas-integration.TestMAASIntegration.test_check_nodes_declared ... ERROR10:37
rvbaOr10:37
rvbamaas-integration.TestMAASIntegration.test_juju_bootstrap ... ERROR10:37
allenaprvba: Where are you seeing that? On the run I kicked off test_check_nodes_declared is ok...10:38
rvbaallenap: we landed a bunch of branches since the first failure.10:39
rvbaAnd when I go through them, I see two different type of failures.10:39
rvbaallenap: I wonder if you haven't reverted one problem, only to get the failure from the other problem.10:40
rvbaallenap: makes sense?10:40
allenaprvba: Ah ha, yes :)10:42
=== ubot5` is now known as ubot5
=== Guest67627 is now known as wallyworld
=== kickinz1 is now known as kickinz1|afk
gmballenap, rvba: Another branch for you: https://code.launchpad.net/~gmb/maas/useful-noconnectionerrors/+merge/23431914:34
rvbagmb: I'll take it.14:35
gmbTa14:36
=== kickinz1|afk is now known as kickinz1
rvbagmb: question for you on the MP.14:39
=== roadmr is now known as roadmr_afk
blake_rallenap: https://bugs.launchpad.net/maas/+bug/136826914:45
ubot5Ubuntu bug 1368269 in MAAS "internal server error when deleting a node" [Critical,Confirmed]14:45
roaksoaxrvba: ^^14:49
rvbaroaksoax: blake_r: yeah, ugly bug. The exception is a bit confusing though.  allenap will probably have an idea.14:51
blake_rrvba: yeah rpc error14:51
gmbrvba: Good point! Updated and pushed.14:57
gmbrvba: hang on; test in the wrong place… fixing14:58
rvbagmb: is it worth changing src/maasserver/rpc/regionservice.py:getClientFor as well?14:58
gmbrvba: Definitely. I hadn’t spotted that one.14:59
roaksoaxgmb: so all the probe-and-enlist are finished already right?15:09
=== roadmr_afk is now known as roadmr
gmbroaksoax: Yes, they’re finished now.15:34
gmbrvba: I’ve updated that branch again.15:35
roaksoaxgmb: awesome!15:35
gmbrvba: Oh, you already approved. Ta :)15:36
gmbroaksoax: Indeed :).15:36
blake_rrvba: https://bugs.launchpad.net/maas/+bug/136826915:39
ubot5Ubuntu bug 1368269 in MAAS "internal server error when deleting a node" [Critical,Confirmed]15:39
blake_rrvba: that breaks juju bootstrap15:39
blake_rrvba: that is what your seing in CI15:39
blake_rrvba: i have a fix, for the enlistment issue15:39
rvbablake_r: okay, well sleuthed.  allenap will have a look at this in a bit.15:40
roaksoaxrvba: heh...so this wasn't related to blake_r 's branches after all15:41
blake_rroaksoax: enlsitment was!15:41
rvbaroaksoax: well, yes and no.15:41
roaksoaxhehe ok :)15:41
roaksoaxok, let's get this fixed asap since we are releasing today15:42
blake_rrvba: https://code.launchpad.net/~blake-rouse/maas/fix-enlistment/+merge/23432615:43
blake_rrvba: one liner!15:43
rvbablake_r: which means you're missing a test :)15:44
blake_rrvba: naw, its twisted!15:44
blake_rrvba: haha!15:44
rvbablake_r: do we really want to ignore all failures like that?15:44
blake_rrvba: we do for windows boot method15:44
rvbaI mean don't you want to only silence No Content error errors15:45
blake_rrvba: i want to silence all errors, because if windows boot method can't be used, that is fine15:46
blake_rrvba: this is only used for the deprecated windows install, that is not supported anymore15:46
blake_rrvba: it is unrelated to curtin15:46
blake_rrvba: we might remove it15:46
rvbablake_r: okay, makes sense;  probably worth a comment in the code though :)15:47
blake_rrvba: okay added comment15:50
rvbaTa15:51
=== kickinz1 is now known as kickinz1|afk
=== CyberJacob|Away is now known as CyberJacob
=== roadmr is now known as roadmr_afk
=== jfarschman is now known as MilesDenver
=== roadmr_afk is now known as roadmr
plarsmatsubara: got a sec? trying to sort out a maas issue21:09
matsubaraplars, yep21:09
matsubarawhat's up?21:09
plarsmatsubara: I have an install here on trusty that I haven't messed with in a while, but it was previously working.  When I powered it back up to try something I tried to go to the /MAAS page on my server and got a 500 error21:10
plarsmatsubara: so I updated to the latest in trusty and rebooted, still no luck21:10
plarsmatsubara: I'm now on the one in ppa:maas-maintainers/stable, but it's doing the same to me21:10
matsubaraplars, are you using any version from the PPAs?21:10
plarsmatsubara: sec and I'll post the oops21:10
matsubarathanks21:10
plarsmatsubara: previously I wasn't, but I tried the ppa one as a last effort21:10
plarsmatsubara: http://paste.ubuntu.com/8322024/21:11
matsubarathe only thing I see in that pastebin is a $21:12
matsubaraplars, ^21:13
plarsmatsubara: hmm21:13
plarssec21:13
plarsmatsubara: try http://paste.ubuntu.com/8322034/21:13
matsubaraDo you have the full traceback for that oops in /var/log/maas/oops? Are there any other tracebacks in /var/log/maas/maas.log or /var/log/maas/celery.log (assuming you are using 1.6 from the stable PPA)21:15
matsubaraplars, ^21:16
matsubaraplars, also worth checking if all services for maas are running: maas-pserv, maas-txlongpoll, maas-cluster-celery and maas-region-celery21:19
matsubaraplars, rabbitmq-server too21:20
plarsmatsubara: sec, phone21:20
matsubarabut there's probably a more informative traceback in somewhere in /var/log/maas or /var/log/apache2/21:21
=== cmagina_ is now known as cmagina
plarsmatsubara: don't see anything that looks like a real traceback, but still looking, one moment21:34
matsubaraplars, ok. Another thing, did you upgrade from 1.5? If you did you'll likely have to re-import the boot images (unrelated to the 500 error you're seeing, just a heads up)21:49
plarsmatsubara: good to know, but I can't even get that far at the moment :)21:49
plarsmatsubara: found some possible stuff http://paste.ubuntu.com/8322192/21:49
plarsmatsubara: that's from error.log21:50
matsubaraplars, it's taking forever to load that pastebin, is it a huge paste?21:52
plarsmatsubara: yes21:53
plarsmatsubara: I can chop it up if you like21:53
plarsmatsubara: there's a lot of 'error: [Errno 113] No route to host' in it21:54
plarsI'm not sure which host, perhaps the node that's turned off at the moment?21:54
plarsmatsubara: try http://paste.ubuntu.com/8322257/ for a short one21:54
matsubaraplars, is rabbitmq-server running? I'd say the region controller is trying to connect to it but failing21:55
=== sebas538_ is now known as sebas5384
matsubaraplars, when you upgrade did maas restart the services after the update?21:55
plarsmatsubara: yes, it's running21:55
plarsmatsubara: the first update to trusty latest, for certain it did, I even rebooted the whole box to be sure21:56
plarsmatsubara: I'm not sure what all services need to be restarted, I trusted the package update to take care of that but I can reboot again after the upgrade to the ppa version21:56
matsubara maas-pserv, maas-txlongpoll, maas-cluster-celery and maas-region-celery should be all up21:57
matsubaraas well as rabbitmq-server21:57
beisnerplars, matsubara - i think the one time i had 500 issue in maas, it ended up being an mq auth issue.   i think there were also some bugs where the rabbitmq pwd was reset during an upgrade.21:57
* beisner thinks back21:57
matsubarabeisner, good point21:57
matsubaraplars, another thing worth checking is the config files in /etc/maas/ and see if DEFAULT_MAAS_URL and MAAS_URL look sane22:04
matsubaraas in, are they pointing to the URL/ip you'd expect MAAS to be running?22:05
plarsmatsubara: hmm, no one of them points at localhost/MAAS22:07
matsubaraplars, where are they pointing to? I'd expect to be one of the IPs for that machine.22:09
plarsmatsubara: it is, but....22:10
plarsmatsubara: when grepping through there I think I may have found the problem22:11
plarsmatsubara: somehow the celery broker url seems to be pointing to the wrong IP22:11
matsubaraplars, there's code in the package to auto detect the default route for the given system and use that IP address as the DEFAULT_MAAS_URL which in turn the MAAS_URL would infer its value from. The package would respect the values if they're set into the debconf db but if the configs were changed manually directly in the file they might be overwritten.22:15
matsubaraplars, but if you can reproduce the issue or describe what you did, I think it's worth filing a bug. It's helpful to have this kind of upgrade feedback.22:16
plarsmatsubara: seems to be working now, somehow I think I just had a bad ip. Thanks!22:19
matsubaraplars, cool! You're welcome.22:19
=== CyberJacob is now known as CyberJacob|Away

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!