| haasn | Hmm, it seems MAAS uses `ipmipower` to interact with IPMI 2.0 clients, but `ipmipower` does not seem to work on my server (always answers “invalid password”) | 00:27 |
|---|---|---|
| haasn | ipmitool on the other hand works fine | 00:27 |
| haasn | I wonder if it would be possible to get this changed | 00:28 |
| haasn | Seems like ipmitool works and FreeIPMI does not | 00:42 |
| haasn | Hmm, it seems like there used to be an ipmi.template in use but it was changed to a “native” ipmi controller written in python in some version that I can't quite discern | 01:23 |
| haasn | The bazaar web interface is virtually unusable for trying to figure out when this change happened, anybody got a clue? I'm trying to pull the “last” version of ipmi.template out of the history so I can rewrite it using ipmitool instead of freeipmi | 01:24 |
| haasn | Ah, I can do a bzr checkout then convert that to git format using bzr fast-export | git fast-import in order to get it in a usable format :) | 01:26 |
| haasn | incidentally, I tried setting up a RAID6 with 4 disks using the new storage config stuff in 1.9 and it flat-out doesn't work at all: https://0x0.st/XWQ.txt | 01:30 |
| haasn | I tried it again and it (inexplicably) got farther, this time erroring out in GRUB | 02:28 |
| haasn | but that seems to be a grub issue | 02:28 |
| haasn | https://0x0.st/XWg.txt head /dev/zero > /dev/disk ? | 03:12 |
| haasn | Isn't that horrendeously slow compared to dd if=/dev/zero of=/dev/disk bs=1M ? | 03:12 |
| mup | Bug #1542982 opened: maas http_proxy does not accept username and password for proxy <MAAS:New> <https://launchpad.net/bugs/1542982> | 05:46 |
| mup | Bug #1542982 changed: maas http_proxy does not accept username and password for proxy <MAAS:New> <https://launchpad.net/bugs/1542982> | 05:49 |
| mup | Bug #1542982 opened: maas http_proxy does not accept username and password for proxy <MAAS:New> <https://launchpad.net/bugs/1542982> | 05:52 |
| mup | Bug #1542982 changed: maas http_proxy does not accept username and password for proxy <MAAS:New> <https://launchpad.net/bugs/1542982> | 05:55 |
| mup | Bug #1542982 opened: maas http_proxy does not accept username and password for proxy <MAAS:New> <https://launchpad.net/bugs/1542982> | 05:58 |
| haasn | Hmm. I'm at the point where I've run into so many issues since switching from 1.8 to 1.9 that I just want to downgrade to see if the universe still makes sense at all, what ppa can I add to give me the 1.8 versions? The maas/stable one only contains 1.7 and 1.9 (and a few others) | 06:48 |
| haasn | (turns out upgrading other stuff solved at least some of the most important issues) | 07:10 |
| BlackDex | Hello there. I'm using maas 1.9, and i need to have two different vlan's on a single bond. is this possible? | 11:17 |
| BlackDex | Ah, i think i have found it. Adding the vlan via the maas cli, and after that i'm able to use that in the GUI ;) | 11:26 |
| mup | Bug #1541640 changed: MaaS region controller 1.9 fails on Ubuntu 15.10 <MAAS:Invalid> <https://launchpad.net/bugs/1541640> | 14:50 |
| mup | Bug #1542761 changed: MAAS will not import images and there is no error as to why <MAAS:Invalid> <https://launchpad.net/bugs/1542761> | 14:50 |
| jamespage | blake_r, hey - time for a bit of maas 1.9 network tuning? | 15:23 |
| jamespage | I need to figure out how to set MTU on MAAS configured interfaces | 15:23 |
| blake_r | jamespage: only one interface or the whole vlan? | 15:33 |
| jamespage | blake_r, well mtu applies at the physical link level - I'm not actually using vlan's here | 15:34 |
| blake_r | jamespage: yes but if you set it on the vlan that all the interfaces are connected to, then when you deploy those nodes, any that have a connection to that vlan will have the mtu set to that value | 15:34 |
| blake_r | jamespage: that way you don't have to go to every node and set the mtu in MAAS | 15:35 |
| jamespage | blake_r, ah right ok | 15:35 |
| blake_r | jamespage: so if that is what you want | 15:35 |
| blake_r | jamespage: just do this "maas my-maas-session vlan update 1 mtu=1500" | 15:35 |
| jamespage | blake_r, I absolitely want everything connected to have the same mtu | 15:35 |
| blake_r | jamespage: 1 being the vlan id | 15:35 |
| blake_r | jamespage: not the vid | 15:36 |
| blake_r | jamespage: use "maas my-maas-session vlans read" to get the list of vlans" | 15:36 |
| jamespage | blake_r, hmm - not liking that command | 15:37 |
| blake_r | jamespage: the update? | 15:37 |
| jamespage | vlans read | 15:38 |
| jamespage | blake_r, using 1.9.0 from the stable PPA | 15:38 |
| blake_r | jamespage: do "maas refresh" first | 15:38 |
| blake_r | jamespage: that way the maas client has the updated API interface | 15:38 |
| jamespage | blake_r, gotcha | 15:39 |
| jamespage | can I apply mtu to the untagged vlan? | 15:39 |
| blake_r | jamespage: yes its jsut like another vlan in MAAS | 15:40 |
| blake_r | jamespage: you will see it in the vlans listing | 15:40 |
| jamespage | blake_r, {"__all__": ["Cannot modify the default VLAN for a fabric."]} | 15:40 |
| jamespage | eek | 15:40 |
| blake_r | jamespage: ah crap | 15:42 |
| blake_r | jamespage: that should be allowed for the mtu value, crap | 15:42 |
| blake_r | jamespage: file a bug please, will get that fixed in 1.9.1 | 15:42 |
| blake_r | jamespage: no you have 2 options | 15:42 |
| blake_r | jamespage: now* | 15:42 |
| jamespage | blake_r, will do | 15:42 |
| blake_r | jamespage: either set it on each interface | 15:42 |
| blake_r | jamespage: or I can show you how to do it manually using the maas shell | 15:43 |
| jamespage | either is good with me - its only 9 interfaces in total | 15:43 |
| blake_r | jamespage: then lets just do it over the API | 15:43 |
| blake_r | jamespage: "maas my-maas-session interface update {node-system-id} {interface-id} mtu=1500" | 15:44 |
| jamespage | blake_r, ok - trying that out now | 15:52 |
| jamespage | maas applied to all interfaces on all nodes... | 15:52 |
| blake_r | jamespage: ?? | 15:58 |
| jamespage | blake_r, testung now | 16:03 |
| jamespage | blake_r, http://paste.ubuntu.com/14993910/ | 16:03 |
| blake_r | jamespage: looks good | 16:05 |
| jamespage | blake_r, 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master juju-br0 state UP group default qlen 1000 | 16:13 |
| jamespage | looking good! | 16:13 |
| blake_r | jamespage: cool | 16:16 |
| blake_r | jamespage: please file that bug | 16:16 |
| jamespage | blake_r, promise I will! | 16:17 |
| blake_r | jamespage: thanks | 16:17 |
| jamespage | blake_r, https://bugs.launchpad.net/maas/+bug/1543195 | 16:25 |
| mup | Bug #1543195 opened: unable to set mtu on default VLAN <api> <networking> <MAAS:Triaged> <MAAS 1.9:Triaged> <https://launchpad.net/bugs/1543195> | 16:38 |
| jamespage | blake_r, hey - so I think xfs support for disks formats is on the roadmap right? | 17:15 |
| jamespage | apologies for the questions - so close to 0 post deploy hacks... | 17:15 |
| roaksoax | jamespage: xfs will be added yes | 17:18 |
| jamespage | roaksoax, awesome - thanks for confirming | 17:18 |
| roaksoax | jamespage: in fact, i think we can even get it to 1.9 | 17:19 |
| jamespage | \o/ | 17:19 |
| mup | Bug #1531843 changed: [Xenial 1.10] IPMI query fails <1.10> <ipmi> <python3> <xenial> <MAAS:Incomplete> <https://launchpad.net/bugs/1531843> | 17:42 |
| haasn | the power status for one of my machines is stuck on “power error”, with the last machine event being: Failed to query node's BMC - Power state could not be queried: [Errno 8] Exec format error | 20:16 |
| haasn | I have fixed the problem in question and verified it works by manually opening a python instance and running os.execl on the file in question | 20:16 |
| haasn | (/usr/sbin/ipmipower) | 20:16 |
| haasn | But for some reason maas doesn't seem to get this. There are no machine events since then, despite me hammering the “check now” button. I don't see any ipmipower processes being spawned either. I tried rebooting the machine, restarting apache2 etc. but maas just doesn't seem to want to retry | 20:17 |
| roaksoax | haasn: what maas version are you using? What hardware are you using? Are you sure the power credentials in MAAS are the correct ones? | 20:19 |
| roaksoax | haasn: is there anything more on the logs (pastebin) | 20:19 |
| haasn | roaksoax: maas v1.9, hardware of the machine in question is a sun fire x4200, I am sure the power credentials in MAAS are the correct ones because I can copy/paste them from the webUI to my own ipmipower invocation and turn the machine on and off | 20:20 |
| haasn | e.g. | 20:20 |
| haasn | su - maas | 20:20 |
| haasn | /usr/sbin/ipmipower -D LAN_2_0 -h X.Y.Z.Z -u maas -p <snip> --stat # this works fine | 20:20 |
| roaksoax | haasn: is anything shown on the logs ? | 20:20 |
| haasn | Hmm, seems there's a new entry as of 20:18: “password invalid” | 20:21 |
| haasn | why it took half an hour for it to switch from “exec format error” to “password invalid” I do not know | 20:21 |
| haasn | This seems like it might be executing the wrong version of ipmipower. I replaced ipmipower by my own version which contains a patch for the “password invalid” bug on X4200 servers and moved it to /usr/sbin/ipmipower and the first one to /usr/sbin/ipmipower-orig | 20:22 |
| haasn | Is it possible it's executing /usr/sbin/ipmipower-orig somehow? | 20:22 |
| roaksoax | haasn: not at all | 20:22 |
| haasn | I can't find the exact location of the command it's executing in src/provisioningserver/drivers/power/ipmi.py anywhere | 20:23 |
| haasn | Okay, some update: It *is* spawning new ipmipower processes, they just exit so ridiculously fast that I couldn't see them in `watch`. With -n0.1 I see them popping up for a split second | 20:24 |
| haasn | Maybe it's time to install auditd and log all process spawns so I can figure out what it's executing? Unless there's a simpler way | 20:24 |
| haasn | Why it can't include the command it executed in the log files is beyond my understanding | 20:25 |
| haasn | Well, then again, so is “why can't it log every failure to the log files instead of only every 30 minutes?” | 20:25 |
| roaksoax | uhmm | 20:29 |
| roaksoax | haasn: you could try hacking ipmi.py to log out what IPMI command is being send | 20:29 |
| roaksoax | in MAAS' logs | 20:29 |
| koaps | heya | 20:33 |
| koaps | does anyone know what happened to MAAS 1.8.3? I don't see it available in stable anymore | 20:34 |
| roaksoax | koaps: it has been superseeded by 1.9 | 20:35 |
| koaps | ya, but shouldn't it still be avail to get if I use the version name? I see 1.7.6 and 1.7.3 there | 20:35 |
| haasn | Heh, auditd didn't want to work but I found a beautifully simple way to accomplish what I wanted: while true; do pgrep -a ipmipower; sleep 0.1; done | 20:36 |
| haasn | this gave me a nice log of (virtually) all executed ipmipower commands | 20:36 |
| haasn | Okay, I found the error and solved it | 20:37 |
| haasn | The problem was that it was adding -W opensesspriv, and the -W flag overrides previous flags - so my “patched” version of ipmipower (which was really just ipmipower-orig -W sun20 "$@") didn't really do anything. I changed it to ipmipower-orig "$@" -W sun20 and that way my sun20 workaround flag overrides the previous opensesspriv work-around flag | 20:38 |
| haasn | Now it works | 20:38 |
| haasn | And I can control the power status of all my servers just fine | 20:38 |
| haasn | maas desperately needs some mechanism for adding your own IPMI workaround flags to nodes | 20:39 |
| roaksoax | koaps: i wonder if the PPA removed it, since we have not made the removal of the version from the PPA itself | 20:47 |
| roaksoax | haasn: you can always file a bug :) | 20:47 |
| haasn | roaksoax: there is already a bug on the issue, I commented on it | 20:48 |
| haasn | But I don't really want to be spending the next few months manually running ipmipower commands until it gets fixed | 20:48 |
| roaksoax | haasn: well, that may seem like an issue with the firmware rather than MAAS itself | 20:51 |
| roaksoax | haasn: since the firmware requires non-standard paramters | 20:51 |
| haasn | roaksoax: Yes okay but that doesn't really help admins who want to provision servers with MAAS. FreeIPMI contains these work-arounds for a reason, not exposing the ability to use them to the admins in charge of running the machines when it would literally be nothing more than one input box and one line of code for appending it to the command line is just being annoying for the sake of annoying, IMO | 20:53 |
| roaksoax | haasn: right, but in situations like this, providing ability to do work around is not always something that will improve MAAS' robustness | 20:59 |
| roaksoax | haasn: as this is an isolated issue | 20:59 |
| mup | Bug #1543286 opened: Exception: 'Node' object is not iterable <MAAS:New> <MAAS 1.10:New> <https://launchpad.net/bugs/1543286> | 21:03 |
| mup | Bug #1543301 opened: MAAS should only allow creating 15 partition <storage> <MAAS:Triaged> <https://launchpad.net/bugs/1543301> | 22:03 |
| mup | Bug #1543301 changed: MAAS should only allow creating 15 partition <storage> <MAAS:Triaged> <https://launchpad.net/bugs/1543301> | 22:06 |
| mup | Bug #1543301 opened: MAAS should only allow creating 15 partition <storage> <MAAS:Triaged> <https://launchpad.net/bugs/1543301> | 22:09 |
| haasn | This is weird. Often, when I try deploying to a VM, I end up in the weird situation where the deploy process seems to be complete but the machine then just gets stuck in an offline state after it was supposed to reboot | 22:53 |
| haasn | if I ssh into the machine, I see The system is going down for reboot NOW! | 22:53 |
| haasn | But.. it never reboots. It just powers off | 22:53 |
| haasn | And MAAS also still thinks it's turned on until I manually hit “check now” | 22:53 |
| haasn | Power control is via virsh | 22:54 |
| haasn | What could possibly be going wrong here? Evidently the machine thinks its rebooting, it just.. doesn't | 22:54 |
| haasn | Hmm, seems to be an inherent part of KVM; “sudo reboot” has the same issue for some reason | 22:59 |
| fluxcore | I use plenty of KVM stuff outside of maas/openstack and don't have any such problems... | 23:03 |
| haasn | I've used dozens of KVM VMs on this same vm host, and it's running like 5 other VMs right now that all work fine | 23:05 |
| haasn | as in I can reboot them | 23:05 |
| haasn | but the ones used by maas have this inexplicable failure | 23:06 |
| roaksoax | haasn: Is this during deployment ? | 23:08 |
| roaksoax | haasn: also, maas doesn't query the VM's/BMC's every second. WE do it ever few minutes to ensure BMC's dont go crazy | 23:08 |
| roaksoax | haasn: i deploy about ~100 to ~200 vm's every few weeks for testing, and have not seen such behavior. However, I've seen a few occurrences where they all fail to PXE at once because there are too many VM's on the same host | 23:10 |
| roaksoax | haasn: but that's how the host's bridge was configured | 23:10 |
| haasn | roaksoax: Yes, this is during deployment | 23:13 |
| haasn | I just reproduced it again: clicked the machine, hit “deploy” in the web UI, chose ubuntu 14.04 lts and default kernel | 23:14 |
| haasn | it started deploying up until the point where rebooted the machine | 23:14 |
| haasn | and now it's just offline | 23:14 |
| haasn | I'll try removing the machine and re-adding/commissioning it in kvm. Maybe something got messed up because I used an xml dump to batch-create a bunch of VMs | 23:14 |
| haasn | destroyed the VM and recreated it and it seems to be working now | 23:35 |
| haasn | must be some weird failure to do with the way it was created, idk | 23:35 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!