| roaksoax | bigjools: yo! so we have some serious DNS issues | 00:23 |
|---|---|---|
| smoser | bigjools, i'm not able to reproduce roaksoax's problem locally | 02:02 |
| smoser | bigjools, see what you think about my comment on https://bugs.launchpad.net/maas/+bug/1116700 | 02:44 |
| ubot5 | Launchpad bug 1116700 in MAAS "MAAS 1.2 no longer auto generates a MAC based hostname" [Critical,Incomplete] | 02:44 |
| EntropyWorks | once I add node and it MAC I chose the Power type IPMI. filled in the parameters and saved. should the node reboot right? It never sees to do that. If I manually force it to boot off the network it grabs the pxelinux.0 but doesn't seem to have a pxelinux.cfg/default | 02:49 |
| bigjools | EntropyWorks: maas is responsible for starting the node, if you start it manually it won't work | 03:43 |
| bigjools | because the tftp server won't realise it's getting booted as the node is not in the right state | 03:44 |
| === shang__ is now known as shang | ||
| === matsubara-afk is now known as matsubara | ||
| jtv | Code review needed! https://code.launchpad.net/~jtv/maas/bug-1116700/+merge/146835 | 12:26 |
| rvba | jtv: I'll have a look presently. | 12:27 |
| jtv | Thanks. | 12:27 |
| jtv | I see you have replaced the word "momentarily" in your lexicon. :) | 12:27 |
| rvba | Well, I still use it, just to piss Julian off. | 12:28 |
| rvba | :) | 12:28 |
| jtv | :) | 12:29 |
| jtv | I had a call with another tall man called Julian earlier. | 12:30 |
| rvba | I hope you managed to piss him off too. | 12:30 |
| jtv | I almost got his name wrong because Edwards somehow claims the mental monopoly on that slot in my brain. | 12:31 |
| jtv | So yes, I almost pissed him off. | 12:31 |
| jtv | I should add "British expat" to round out the similarities. | 12:31 |
| === matsubara is now known as matsubara-lunch | ||
| smoser | rvba, https://code.launchpad.net/~smoser/maas/lp1103716/+merge/146863 if you'd like | 15:27 |
| rvba | smoser: sure, I'll have a look in a sec | 15:27 |
| smoser | allenap, wanted to folow up a bit on fast path installer... | 15:28 |
| allenap | smoser: What's up? | 16:00 |
| smoser | how are we doing there.. do we/i/roaksoax need to do anything? | 16:00 |
| allenap | smoser: We've not done anything on it, so, no, you don't need to do anything I don't think, we do. Is there a deadline? | 16:01 |
| smoser | i dont have a deaadline, no | 16:03 |
| smoser | but should we set one ? | 16:03 |
| smoser | feature freeze is march 7. | 16:04 |
| smoser | lets say i'd like to have necessary bits into maas by feb 21 ? | 16:05 |
| allenap | smoser: That sounds sane to me. I'll email Julian, and CC you, as this is his decision. | 16:06 |
| === matsubara-lunch is now known as matsubara | ||
| GTFr0 | I'm having a problem with MAAS, where nodes stay in the "commissioning" state. I think it maybe a NIC issue (not sure), but is there any way of verifying what NIC drivers are in the PXE booted kernel? | 17:11 |
| GTFr0 | (I originally through the issue was oath-related time skew, but I updated the ephemeral disk image to run ntpdate on boot, and that didn't seem to help) | 17:11 |
| GTFr0 | the kicker is that I can't find any indication in any logs on the MAAS server that any of the commissioned nodes are connecting | 17:14 |
| smoser | GTFr0, are you able to see console logs ? | 17:48 |
| smoser | if you enlisted correctly, then that is the same initramfs that is used for commissioning. | 17:49 |
| smoser | (ie, same modules and dthe like) | 17:49 |
| smoser | but it is quite possible that you're just missing nodes | 17:49 |
| smoser | er... missing modules | 17:49 |
| smoser | https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1115710 | 17:49 |
| ubot5 | Launchpad bug 1115710 in MAAS "Mellanox mlx4_en network driver is not automatically loaded" [Critical,Triaged] | 17:49 |
| smoser | GTFr0, http://paste.ubuntu.com/1617332/ is how you can get yourself a new initramfs | 17:50 |
| GTFr0 | smoser: I added the nodes manually on the MAAS server | 18:02 |
| smoser | then i suspect you are right. | 18:03 |
| smoser | about the netowkr drivers | 18:03 |
| smoser | if you can see console (serial consoel or vga) you can probably verify that | 18:03 |
| GTFr0 | hrm, not using Mellanox ethernet cards. Oddly enough, these are Proliant Gen8 servers with i350 ethernet onboard | 18:04 |
| GTFr0 | afaik, i350 uses the standard igb driver, and if I do the standard Ubuntu server install, it sees the ethernet interfaces just fine | 18:05 |
| GTFr0 | and I don't think it's a DHCP issue, because the nodes will boot PXE | 18:06 |
| GTFr0 | smoser: when they boot PXE, I can see the console, but all I see is a ubuntu login prompt and am not able to login to get dmesg or other diagnostic info | 18:07 |
| smoser | ah. | 18:09 |
| smoser | GTFr0, ok. so if you pxeboot in commissioning and get to a ubuntu login prompt | 18:09 |
| smoser | then you're definitely getting networking, as that is coming over iscsi | 18:10 |
| smoser | you can "backdoor" the ephemeral image so you can login | 18:10 |
| smoser | see info on how to do that at https://lists.launchpad.net/maas-devel/msg00817.html | 18:11 |
| smoser | and https://code.launchpad.net/~matsubara/maas/ephemeral-img-debugging-doc/+merge/143927 | 18:11 |
| matsubara | thanks smoser, that ping reminded me I need to chase evilnickveitch for a review :-) | 18:12 |
| evilnickveitch | oops! I'll do it tomorrow matsubara ! | 18:13 |
| matsubara | evilnickveitch, thanks! | 18:13 |
| === matsubara is now known as matsubara-afk | ||
| michaela_ | what exactly is a node? | 19:45 |
| EntropyWorks | bigjools: that makes sense. but it never sends the reboot command via IPMI. so I'm stuck now with 4 nodes waiting saying Commissioning. | 19:53 |
| EntropyWorks | Nice to see the Mellanox mlx4_en is going to be added. that has been a major headache | 20:03 |
| smoser | EntropyWorks, did you ever test that for me ? | 20:09 |
| smoser | EntropyWorks, https://code.launchpad.net/~smoser/ubuntu/raring/kmod/lp-1115710/+merge/146760 . put that mlx4.conf file in /etc/modprobe.d and then un-do any of the hacks you'vae done to get it loaded | 20:10 |
| smoser | then reboot and make sure networking works as expected | 20:10 |
| smoser | if we dont get that fix in, then you'll have the modules in your initramfs but you'll still not have networking because the _en wont get loaded. | 20:16 |
| EntropyWorks | smoser: haven't tried that but in precise and quanta the initrd.gz didn't actually contain the drivers so just adding that to /etc/modprobe.d wouldn't have done it. | 20:17 |
| smoser | understood | 20:17 |
| smoser | i'm asking you to test an installed system, really. | 20:18 |
| smoser | just to see if it comes up right if you dont have your custom udev rule or entry in /etc/modules (or some other manual way) | 20:18 |
| EntropyWorks | my fix was this for straight pxe installing. http://goo.gl/PfOZd now I'm looking at MAAS but haven't had much luck getting it to work. | 20:21 |
| smoser | EntropyWorks, right. | 20:21 |
| smoser | bu tafter you install | 20:21 |
| smoser | then what? | 20:22 |
| EntropyWorks | smoser: ok let me see about doing that. just grabbing your branch now bzr branch lp:~smoser/ubuntu/raring/kmod/lp-1115710 | 20:34 |
| smoser | all you really need is that mlx4.conf file into /etc/modprobe.d | 20:36 |
| smoser | that should be enough | 20:36 |
| smoser | EntropyWorks, and *thank you* for your help | 20:38 |
| EntropyWorks | I will also rem mlx4_en from /etc/modules which is something I believe i did in my late_command.sh | 20:38 |
| EntropyWorks | these machine take about 3 min to reboot. so here we go | 20:39 |
| smoser | right. | 20:42 |
| EntropyWorks | humm. mlx4_core 000:05:00.0: comand 0xc failed: fw status 0x40 | 20:44 |
| smoser | EntropyWorks, where do you see that ? | 20:46 |
| EntropyWorks | on the console | 20:47 |
| smoser | dmesg ? are the modules loaded? (lsmod | grep mlx4) | 20:47 |
| EntropyWorks | mlx4_core loaded but nothing else | 20:50 |
| EntropyWorks | I have a meeting so be back after that and will chat again | 20:50 |
| smoser | ok. can you try one other thing for me ? | 20:50 |
| smoser | ok. thank you. | 20:50 |
| EntropyWorks | smoser: back, what would you want next to be done? | 21:45 |
| smoser | hooray | 21:46 |
| smoser | ok | 21:46 |
| smoser | so you rebooted and mlx4_en was not loaded, right? | 21:46 |
| EntropyWorks | correct | 21:46 |
| smoser | so lets replace that mlx4.conf with | 21:46 |
| smoser | # mlx4_core should load mlx4_en (LP: #1115710). | 21:47 |
| smoser | install mlx4_core /sbin/modprobe --ignore-install mlx4_core; /sbin/modprobe mlx4_en; /sbin/modprobe mlx4_ib | 21:47 |
| ubot5 | Launchpad bug 1115710 in MAAS "Mellanox mlx4_en network driver is not automatically loaded" [Critical,Triaged] https://launchpad.net/bugs/1115710 | 21:47 |
| smoser | and then reboot and see how we fare. | 21:47 |
| EntropyWorks | so just that whole line. and remove the softdep line? | 21:48 |
| smoser | remove the soft dep | 21:49 |
| smoser | and, yeah, one full line | 21:49 |
| EntropyWorks | ok rebooting again | 21:54 |
| EntropyWorks | nope. btw I wish I could see more than just this on the serial console http://paste.ubuntu.com/1618052/ . It hangs there until I get a login prompt | 22:04 |
| EntropyWorks | still not luck loading the mlx4_en | 22:05 |
| EntropyWorks | and in desg [ 8.031717] mlx4_core 0000:05:00.0: command 0xc failed: fw status = 0x40 | 22:10 |
| EntropyWorks | no mention of mlx4_en in desg | 22:10 |
| EntropyWorks | s/desg/dmesg/ | 22:11 |
| smoser | EntropyWorks, hm.. | 22:12 |
| smoser | so to see more on the console there, i tihnk you'll need to append console=ttyS0 | 22:13 |
| smoser | and that should function | 22:13 |
| smoser | but then you wont see the stuff on the vga console | 22:13 |
| smoser | can you try now rebooting after removing ml4x.conf entirely? | 22:14 |
| smoser | err... first can you just try running 'modprobe mlx4_core' | 22:14 |
| smoser | and seeing if networking _en gets loaded | 22:14 |
| EntropyWorks | let me unload mlx4_core and mlx4_en again ( did that by hand just a moment ago ) | 22:17 |
| smoser | i just dont get it. | 22:18 |
| smoser | with either of those mlx4.conf in place, i get both modules loaded in my vm. | 22:18 |
| smoser | (granted i dont have the hardware, but ... ) | 22:18 |
| EntropyWorks | doing that loaded both and the mlx4_ib and some other ib_mad ib_core | 22:25 |
| EntropyWorks | but doesn't seem to do the trick when rebooting | 22:25 |
| smoser | right. | 22:26 |
| smoser | and if we remove that, we *do* get mlx4_core loaded, right? | 22:26 |
| smoser | EntropyWorks, ^ . lets try | 22:29 |
| smoser | removing the mlx4.conf file entirely | 22:29 |
| smoser | and rebooting | 22:29 |
| smoser | see if we still see the message in dmesg | 22:29 |
| EntropyWorks | Ok removing that conf completely and will reboot | 22:30 |
| smoser | EntropyWorks, i really appreciate your time | 22:30 |
| smoser | i'm gonna have to run for a while. | 22:30 |
| smoser | if you can just fill me in here. | 22:30 |
| EntropyWorks | np, I'll be back tomorrow just /msg me and I will get it | 22:30 |
| EntropyWorks | this mlx4 issue has been a problem for a long time for me. | 22:32 |
| EntropyWorks | now if I could get MAAS to reboot a node... LOL | 22:32 |
| EntropyWorks | rebooted and mlx4_core was loaded but not the others | 22:40 |
| EntropyWorks | smoser: without /etc/modprobe.d/mlx4.conf only mlx4_core loaded | 22:41 |
| EntropyWorks | humm... | 22:48 |
| EntropyWorks | The directory containing the ephemeral images/info is missing (u'/var/lib/maas/ephemeral/quantal/ephemeral/amd64'). Make sure to run the script 'maas-import-pxe-files'. | 22:48 |
| EntropyWorks | but I do have a dir in there /var/lib/maas/ephemeral/quantal/ephemeral/amd64/20121017/ with disk.img, info, initrd.gz, linux, tgt.conf | 22:49 |
| EntropyWorks | and have run maas-import-pxe-files | 22:49 |
| EntropyWorks | what gives | 22:49 |
| bigjools | it takes a while for the re-scan of the files to happen | 22:50 |
| bigjools | the message should disappear within a few minutes | 22:50 |
| EntropyWorks | ok, I'll let it sit for a but and watch the logs. but why won't MAAS actually send the IPMI to the node to reboot? I have HP servers with iLO3 which is on a network reachable from the MAAS box. | 23:01 |
| bigjools | EntropyWorks: maas doesn't do "reboot", it does power on and power off | 23:08 |
| bigjools | power on when allocated and power off when deallocating | 23:08 |
| EntropyWorks | ok so I need to power off the machine so it can power it on then... | 23:10 |
| bigjools | EntropyWorks: what are you trying to achieve, at a higher level? | 23:15 |
| EntropyWorks | higher lever. get a rack of HP ProLiant SL390s into MAAS | 23:16 |
| bigjools | why do you need to reboot outside of the normal commissioning/allocation cycle? | 23:17 |
| EntropyWorks | well I just installed MAAS on one machine in the rack. I then went to add a node. but it never gets past Commissioning | 23:18 |
| smoser | EntropyWorks, yeah, that message doesn't like to go away some times. | 23:19 |
| smoser | EntropyWorks, and without any mlx4.conf, do we still see the "command 0xc failed" message in dmesg? | 23:19 |
| EntropyWorks | I know the machine work and could see each other because I had my own PXE setup that with a patch initrd.gz for the mlx4 | 23:19 |
| EntropyWorks | smoser: I think you do [ 7.955801] mlx4_core 0000:05:00.0: command 0xc failed: fw status = 0x40 | 23:21 |
| smoser | ok. so thats just a red herring then. | 23:22 |
| smoser | but our changes to that ocnfig file seem not to be doing anything. | 23:22 |
| smoser | just to be sure, you're p utting that file into '/etc/modprobe.d/mlx4.conf' | 23:22 |
| smoser | right? | 23:22 |
| EntropyWorks | bigjools: I need MAAS to actually do the commissioning/allocation cycle, it doesn't want to add any of the nodes. 2 nodes in this MAAS, 2 nodes offline | 23:22 |
| bigjools | EntropyWorks: oh is this part of ther trouble you're having with the mlx4? | 23:23 |
| EntropyWorks | bigjools: nope different problem | 23:23 |
| bigjools | smoser: we probably ought to put together a faq "why is my node not commissioning?" | 23:24 |
| EntropyWorks | smoser: when the file /etc/modprobe.d/mlx4.conf contains "install mlx4_core /sbin/modprobe --ignore-install mlx4_core; /sbin/modprobe mlx4_en; /sbin/modprobe mlx4_ib" it does not load the modules at boot. it will load the modules when "modprob mlx4_core" | 23:25 |
| EntropyWorks | by hand | 23:25 |
| smoser | right. | 23:26 |
| smoser | i just dont understand that. | 23:26 |
| EntropyWorks | the way I've been making life work is adding mlx4_en to /etc/modules | 23:26 |
| smoser | hm.. | 23:26 |
| smoser | maybe your initramfs has the 'modules' entry in it? | 23:26 |
| smoser | and the driver is then getting loaded in initramfs when it doesn't have that file | 23:27 |
| smoser | (i'm kind of grasping at this point, but i really dont understand it otherwise) | 23:27 |
| smoser | i just did this: | 23:27 |
| smoser | sudo modprobe "pci:v000015B3d00001010svfsdfbcfscfif" | 23:27 |
| smoser | and i get the mlx4_en loaded | 23:27 |
| smoser | (that matches one of the aliases) | 23:28 |
| EntropyWorks | so I could open my initrd.img-3.5.0-23-generic and poke inside I guess | 23:28 |
| EntropyWorks | but that should be stock. | 23:29 |
| smoser | EntropyWorks, well, if you update-initramfs | 23:29 |
| smoser | well, since you had that /etc/modules entry | 23:29 |
| smoser | it could/might get pulled into there | 23:29 |
| smoser | on an update-initramfs | 23:29 |
| EntropyWorks | I have not run that command yet | 23:29 |
| smoser | right | 23:29 |
| smoser | but it happens on kenrle install and such | 23:29 |
| EntropyWorks | humm. the kernel is installed before I add to /etc/modules | 23:30 |
| EntropyWorks | I'm betting | 23:30 |
| smoser | well, yes, but it gets updated lots of times. | 23:31 |
| smoser | do this: | 23:31 |
| EntropyWorks | yep I do it in my late_command.sh | 23:31 |
| smoser | lsinitramfs /boot/initrd.... | grep mlx | 23:31 |
| smoser | and i suspect/hope we see it there. | 23:31 |
| smoser | my theory is that if its there, then when modprobe comes up in root, it doesn't find it. | 23:32 |
| smoser | well, its already been loaded | 23:32 |
| smoser | so nothing tries to load it | 23:32 |
| EntropyWorks | it is there. both mlx4_core.ko and mlx4_en.ko | 23:32 |
| smoser | right. but you do not have the mlx4.conf in there. | 23:34 |
| smoser | so, put mlx4.conf in place | 23:34 |
| EntropyWorks | correct | 23:34 |
| smoser | then 'update-initramfs -u -k $(uname -r)' | 23:34 |
| smoser | and reboot | 23:34 |
| smoser | and again, thank you | 23:34 |
| smoser | i have to run | 23:34 |
| smoser | will check in later. | 23:34 |
| EntropyWorks | so I should put the file back in /etc/modprobe.d/mlx4.conf and then just run the command 'update-initramfs -u -k $(uname -r)' | 23:36 |
| EntropyWorks | bigjools: having some info for why a node is not Commissioning would be wonderful | 23:45 |
| bigjools | EntropyWorks: yes! It's a complicated area, a whole lot of stuff can go wrong :( | 23:46 |
| EntropyWorks | doing a tcpdump I know MAAS is sending something to the IP of the IPMI device. but what ever it is doesn't power on the server. | 23:47 |
| bigjools | was the bmc detected at enlistment? | 23:47 |
| bigjools | or are you configuring manually? | 23:47 |
| EntropyWorks | bigjools: using ipmitool -I lanplus -H 10.X.X.X -U Administrator -P foobar chassis power on | 23:48 |
| EntropyWorks | does turn the server on. | 23:49 |
| bigjools | ok compare with the power script that maas is using | 23:49 |
| EntropyWorks | where should I look for that :-) | 23:50 |
| bigjools | provisioningserver/power/templates IIRC | 23:56 |
| EntropyWorks | humm. I wonder if MAAS is expecting IPMI to be on the same interface as the machines network card. my IPMI is in iLO which is an out of bandwidth daughter card. which is on a different netwok than the machines NIC. the iLO has an IP address already and doesn't need one from MAAS | 23:56 |
| bigjools | no it doesn't expect that | 23:56 |
| bigjools | it just stores IP address | 23:56 |
| EntropyWorks | ok cool | 23:56 |
| bigjools | I expect it's a v1 / v2 problem | 23:56 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!