| racedo | roaksoax: we are going to keep working from the hotel now | 00:01 |
|---|---|---|
| racedo | catch you later | 00:01 |
| roaksoax | alright! | 00:02 |
| === matsubara-afk is now known as matsubara | ||
| eazel7 | hi =) | 12:53 |
| eazel7 | how do I provision a new node for maas? can I create VM with libvirt and "enlist" it as a node? | 12:54 |
| === kentb-out is now known as kentb | ||
| njin | hallo, it is expected that today's build show 'It Works' message instead maas interface ? thanks | 15:47 |
| roaksoax | rvba: howdy!! | 15:51 |
| roaksoax | njin: maybe you are missing the /MAAS ? | 15:51 |
| rvba | roaksoax: Hi | 15:51 |
| roaksoax | rvba: so I tested the django thing... seems to be working just fine | 15:51 |
| roaksoax | rvba: have you been able to test it too? | 15:51 |
| rvba | roaksoax: yeah, all fine. Tested on precise and quantal. | 15:52 |
| njin | roaksoax, with /MAAS it return error 200 | 15:56 |
| roaksoax | rvba: cool then i'll move that to /stable | 16:00 |
| roaksoax | njin: check your apache2 logs? and maas logs? | 16:00 |
| === andreas__ is now known as ahasenack | ||
| njin | roaksoax, thanks apache2 log is full of errors /usr/share/maas/start_up...Lock timeout, missed mod_wgsi and others | 16:19 |
| === matsubara is now known as matsubara-lunch | ||
| racedo | rvba: thanks for looking at lp 1131418 | 16:49 |
| ubot5 | Launchpad bug 1131418 in MAAS "Nodes don't go to ready, after commissioning they get a 500 error when reporting back to maas" [Critical,Triaged] https://launchpad.net/bugs/1131418 | 16:49 |
| racedo | we reinstalled everything last night as we were onsite and had to finish a delivery | 16:50 |
| rvba | racedo: np. As I said on the bug, I suspect the tag definition is buggy. | 16:51 |
| racedo | we don't seem to be hitting this after reproducing exactly the same environment two times | 16:51 |
| rvba | That's very weird. | 16:51 |
| racedo | yes, it all started when using --constraint maas-name=name-of-the-host wasn't recognising the host and the juju log in zookeeper said no such name or no matching or something similar | 16:52 |
| racedo | with a name-of-the-host that existed | 16:52 |
| racedo | if we see anything today we'll update the lp | 16:52 |
| rvba | Ok, cool. | 16:53 |
| racedo | roaksoax: melmoth: have you seen in your engagements maas nodes rebooting after being deployed with a service or even after being commissioned and go to grub rescue prompt? | 17:04 |
| melmoth | racedo, not really, but i never understood the powermanagement thingy woith maas | 17:06 |
| melmoth | i guess when you deploy a service, it pick a ready node, and boots ip automatically | 17:06 |
| melmoth | i never experienced it, but i guess this is what should happen, right ? | 17:06 |
| roaksoax | racedo: never | 17:07 |
| racedo | roaksoax: melmoth: it's exactly what Destreyf explains here at 19:20: http://irclogs.ubuntu.com/2012/05/15/%23juju.txt | 17:07 |
| melmoth | i wont have time to read today. | 17:08 |
| racedo | we reported the pattern in lp 1131737 | 17:09 |
| ubot5 | Launchpad bug 1131737 in MAAS "Nodes stay at grub rescue prompt after being redeployed with juju" [Undecided,New] https://launchpad.net/bugs/1131737 | 17:09 |
| roaksoax | racedo: can you pastebin the celery logs? | 17:11 |
| roaksoax | var/log/maas/celery.log region-celery.log | 17:11 |
| roaksoax | racedo: what it looks to me is that when you tell it to deploy again it is not actually telling it to PXE boot | 17:12 |
| racedo | roaksoax: it picks a pxe image and the cfg says to boot from the disk apparently | 17:12 |
| racedo | thats what we understand from the boot screen | 17:12 |
| racedo | the funny thing is that it doesn't happen always | 17:13 |
| * racedo working on getting the logs | 17:13 | |
| roaksoax | racedo: might it be that when you juju destroy and then juju deploy to that machine, it is not being set to use the PXE image to deploy? | 17:13 |
| racedo | roaksoax: https://pastebin.canonical.com/85444/ celety.log | 17:14 |
| roaksoax | rvba: ^^ | 17:15 |
| racedo | roaksoax: well, but the last pattern was: delete zookeeper from maas, reboot, enlist, accept and commission, node reboots fine, juju bootstrap to node, bootstraps fine, reboots after it finishes, pxe boots, gets pxe config that tells to boot from disk, goes to grub rescue | 17:17 |
| roaksoax | racedo: ahh I see | 17:17 |
| roaksoax | racedo: so that's a grub issue | 17:18 |
| roaksoax | it seems | 17:18 |
| racedo | roaksoax: it could well be | 17:18 |
| roaksoax | racedo: what's the disk it should be booting from? | 17:18 |
| racedo | the funny thing is that it's kind of random | 17:18 |
| roaksoax | racedo: do you happen to know what it is been told by PXE when it tells it to PXE boot? | 17:19 |
| roaksoax | say LOCALBOOT 0 | 17:19 |
| roaksoax | or KERNEL chain.c32 | 17:19 |
| roaksoax | racedo: ok you are gonna have to do something to test | 17:20 |
| racedo | roaksoax: we have 4 disks, 3 in raid 5 and one as spare disk, hw raid | 17:20 |
| roaksoax | racedo: right, so maybe that's the problem | 17:20 |
| roaksoax | but since this is HW raid | 17:20 |
| roaksoax | it should | 17:20 |
| roaksoax | not affect | 17:20 |
| racedo | they are presented as 1 disk to the os | 17:20 |
| racedo | ok | 17:21 |
| roaksoax | racedo: ok so go to /usr/share/pyshared/provisioningserver/pxe | 17:21 |
| racedo | roaksoax: ok, we will test that if we hit it this time (rebuilding) | 17:21 |
| roaksoax | racedo: ok, but go there | 17:21 |
| racedo | yep | 17:21 |
| === matsubara-lunch is now known as matsubara | ||
| roaksoax | racedo: you will find 2 files of interest | 17:21 |
| roaksoax | config.local.template | 17:22 |
| roaksoax | config.local.x86.template | 17:22 |
| roaksoax | you need to figure out which one it is using | 17:22 |
| racedo | ok | 17:22 |
| racedo | oh i see | 17:22 |
| roaksoax | i'm guessing it is using config.local.x86.template since that should be used for most of the hw | 17:22 |
| racedo | ok, once i figure which one what do i do | 17:23 |
| roaksoax | racedo: something like this: http://paste.ubuntu.com/5555710/ | 17:23 |
| roaksoax | racedo: so LOCALBOOT -1 if it is the one been used | 17:23 |
| roaksoax | racedo: or APPEND hd0 or APPEND hd0,1 | 17:23 |
| racedo | perfect | 17:23 |
| racedo | Im documenting this now | 17:24 |
| racedo | thanks roaksoax | 17:24 |
| roaksoax | racedo: the latter APPEND hd0 is basically telling it boot for hd0 | 17:24 |
| roaksoax | or boot from hd0,1 | 17:24 |
| roaksoax | that *might* be the issue | 17:24 |
| racedo | got you | 17:24 |
| roaksoax | racedo: if this doesn'y solve it, maybe grub messed things up | 17:24 |
| roaksoax | racedo: but someone who might help in grub related stuff if cjwatson | 17:24 |
| roaksoax | rvba: this is a weird thing: https://pastebin.canonical.com/85444/ (omshell issues) | 17:26 |
| racedo | i've seen it over and over ^^ | 17:27 |
| roaksoax | racedo: did it work? the boot thing? | 18:02 |
| racedo | roaksoax: we haven't hit it again, it's 10 nodes so far and nothing | 19:06 |
| racedo | roaksoax: it's very random... but i fully agree with you it looks like a grub issue with their hw raid setup | 19:07 |
| racedo | roaksoax: we are hitting the issue right now | 19:36 |
| racedo | we are at the grub rescue prompt | 19:36 |
| racedo | we don't really now how to deal with the grub rescue prompt but negronjl is changing the what you suggested in maas: http://paste.ubuntu.com/5555710/ | 19:38 |
| negronjl | roaksoax, where ( in the maas server ) are the files that need modifying ... I want to try changing that | 19:39 |
| racedo | roaksoax: we got it /usr/share/pyshared/provisioningserver/pxe | 19:41 |
| roaksoax | yeah | 19:43 |
| roaksoax | sorry | 19:43 |
| roaksoax | racedo: so do that andjust reboot the machine manually and tell it to PXE boot | 19:43 |
| racedo | roaksoax: ok, we are on it now, just modified the grub templates and we are going to pxe boot a node that fails | 19:44 |
| roaksoax | cool | 19:45 |
| roaksoax | racedo: try to see what the ouput of the PXE boot says | 19:45 |
| racedo | we will see it | 19:45 |
| racedo | roaksoax: when you say to tell it to pxe boot | 19:47 |
| roaksoax | racedo: ipmi | 19:47 |
| racedo | do you mean by ipmi pxe boot it, not reenlist it and start again? will pxeboot apply the new grub commands just by rebooting? | 19:47 |
| roaksoax | racedo: yes | 19:48 |
| roaksoax | racedo: so you need to do what I told you yesterday | 19:48 |
| racedo | yes | 19:48 |
| racedo | we got that scripted :) | 19:48 |
| roaksoax | cool | 19:48 |
| racedo | roaksoax: does this mean that every time a node pxe boots grub get installed through these templates then? | 19:51 |
| roaksoax | racedo: nope, this means that when the node PXE boot, it is telling the node "boot from your localdisk, rather than tftp" | 19:52 |
| roaksoax | racedo: so chainc.32 should have automatically determined where to pxe boot from, but since it didn't, we are telling it to specifically look for hd0 to pxe from | 19:53 |
| racedo | oh, yes but that's happening i think, and then it will go to grub rescue when trying to boot from the disk | 19:54 |
| racedo | we are trying to grab a screenshot | 19:54 |
| roaksoax | racedo: ok cool | 19:57 |
| roaksoax | racedo: or you can do a video | 19:57 |
| roaksoax | racedo: with recordmydesktop | 19:58 |
| === matsubara is now known as matsubara-afk | ||
| racedo | only our customer has the access from a windows desktop :( | 19:58 |
| roaksoax | racedo: boomer | 19:59 |
| racedo | yep | 19:59 |
| roaksoax | yeah just try to get screenshots | 19:59 |
| roaksoax | racedo: did you check with cjwatson? | 20:00 |
| racedo | not yet, so we have a plan b if this fails, we have like 2 hours to get this working, the plan is to work with the customer remotely on this if needed | 20:01 |
| racedo | but today we have a plan b which is to just not use these boxes | 20:01 |
| roaksoax | ack | 20:02 |
| racedo | roaksoax: we couldn't catch the screen but i saw: SAY Booting under MAAS direction.. which seems to be on config.install.template | 20:07 |
| racedo | but as we deleted it and started again it might be part of this randomness... | 20:08 |
| roaksoax | racedo: yeah that's installation | 20:11 |
| roaksoax | or ocmmissioning | 20:11 |
| roaksoax | racedo: how are things going? | 21:26 |
| racedo | roaksoax: we deployed everything with what we had up and running and we might try to debug with the customer and support the grub issue next week if the customer has the time | 21:55 |
| racedo | roaksoax: thanks again man, it's been an interesting week ;) | 21:56 |
| racedo | roaksoax: btw if you are interested i captured the tcpdump of yesterday's internal server error and today extracted the xml file that the node posts to maas with wireshark: https://launchpadlibrarian.net/132080481/maas_xml_post | 22:18 |
| racedo | this is lp 1131418 | 22:19 |
| ubot5 | Launchpad bug 1131418 in MAAS "Nodes don't go to ready, after commissioning they get a 500 error when reporting back to maas" [Critical,Triaged] https://launchpad.net/bugs/1131418 | 22:19 |
| === kentb is now known as kentb-afk | ||
| roy-feldman | I have found a bug on the MAAS site. Should I report it on the Ubuntu Issue tracker? I ask because it is actually not a bug with a Ubuntu distro. | 23:06 |
| roy-feldman | Basically, the search feature for the online documentation is broken. It simply hangs when you do a search. | 23:07 |
| roy-feldman | Search works on the rest of the MAAS site. | 23:07 |
| roy-feldman | But searches on other parts of the MAAS site don't seem to report matches in the MAAS docs. | 23:08 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!