haasnHmm, it seems MAAS uses `ipmipower` to interact with IPMI 2.0 clients, but `ipmipower` does not seem to work on my server (always answers “invalid password”)00:27
haasnipmitool on the other hand works fine00:27
haasnI wonder if it would be possible to get this changed00:28
haasnSeems like ipmitool works and FreeIPMI does not00:42
haasnHmm, it seems like there used to be an ipmi.template in use but it was changed to a “native” ipmi controller written in python in some version that I can't quite discern01:23
haasnThe bazaar web interface is virtually unusable for trying to figure out when this change happened, anybody got a clue? I'm trying to pull the “last” version of ipmi.template out of the history so I can rewrite it using ipmitool instead of freeipmi01:24
haasnAh, I can do a bzr checkout then convert that to git format using bzr fast-export | git fast-import in order to get it in a usable format :)01:26
haasnincidentally, I tried setting up a RAID6 with 4 disks using the new storage config stuff in 1.9 and it flat-out doesn't work at all: https://0x0.st/XWQ.txt01:30
haasnI tried it again and it (inexplicably) got farther, this time erroring out in GRUB02:28
haasnbut that seems to be a grub issue02:28
haasnhttps://0x0.st/XWg.txt head /dev/zero > /dev/disk ?03:12
haasnIsn't that horrendeously slow compared to dd if=/dev/zero of=/dev/disk bs=1M ?03:12
mupBug #1542982 opened: maas http_proxy does not accept username and password for proxy <MAAS:New> <https://launchpad.net/bugs/1542982>05:46
mupBug #1542982 changed: maas http_proxy does not accept username and password for proxy <MAAS:New> <https://launchpad.net/bugs/1542982>05:49
mupBug #1542982 opened: maas http_proxy does not accept username and password for proxy <MAAS:New> <https://launchpad.net/bugs/1542982>05:52
mupBug #1542982 changed: maas http_proxy does not accept username and password for proxy <MAAS:New> <https://launchpad.net/bugs/1542982>05:55
mupBug #1542982 opened: maas http_proxy does not accept username and password for proxy <MAAS:New> <https://launchpad.net/bugs/1542982>05:58
haasnHmm. I'm at the point where I've run into so many issues since switching from 1.8 to 1.9 that I just want to downgrade to see if the universe still makes sense at all, what ppa can I add to give me the 1.8 versions? The maas/stable one only contains 1.7 and 1.9 (and a few others)06:48
haasn(turns out upgrading other stuff solved at least some of the most important issues)07:10
BlackDexHello there. I'm using maas 1.9, and i need to have two different vlan's on a single bond. is this possible?11:17
BlackDexAh, i think i have found it. Adding the vlan via the maas cli, and after that i'm able to use that in the GUI ;)11:26
mupBug #1541640 changed: MaaS region controller 1.9 fails on Ubuntu 15.10 <MAAS:Invalid> <https://launchpad.net/bugs/1541640>14:50
mupBug #1542761 changed: MAAS will not import images and there is no error as to why <MAAS:Invalid> <https://launchpad.net/bugs/1542761>14:50
jamespageblake_r, hey - time for a bit of maas 1.9 network tuning?15:23
jamespageI need to figure out how to set MTU on MAAS configured interfaces15:23
blake_rjamespage: only one interface or the whole vlan?15:33
jamespageblake_r, well mtu applies at the physical link level - I'm not actually using vlan's here15:34
blake_rjamespage: yes but if you set it on the vlan that all the interfaces are connected to, then when you deploy those nodes, any that have a connection to that vlan will have the mtu set to that value15:34
blake_rjamespage: that way you don't have to go to every node and set the mtu in MAAS15:35
jamespageblake_r, ah right ok15:35
blake_rjamespage: so if that is what you want15:35
blake_rjamespage: just do this "maas my-maas-session vlan update 1 mtu=1500"15:35
jamespageblake_r, I absolitely want everything connected to have the same mtu15:35
blake_rjamespage: 1 being the vlan id15:35
blake_rjamespage: not the vid15:36
blake_rjamespage: use "maas my-maas-session vlans read" to get the list of vlans"15:36
jamespageblake_r, hmm - not liking that command15:37
blake_rjamespage: the update?15:37
jamespagevlans read15:38
jamespageblake_r, using 1.9.0 from the stable PPA15:38
blake_rjamespage: do "maas refresh" first15:38
blake_rjamespage: that way the maas client has the updated API interface15:38
jamespageblake_r, gotcha15:39
jamespagecan I apply mtu to the untagged vlan?15:39
blake_rjamespage: yes its jsut like another vlan in MAAS15:40
blake_rjamespage: you will see it in the vlans listing15:40
jamespageblake_r, {"__all__": ["Cannot modify the default VLAN for a fabric."]}15:40
blake_rjamespage: ah crap15:42
blake_rjamespage: that should be allowed for the mtu value, crap15:42
blake_rjamespage: file a bug please, will get that fixed in 1.9.115:42
blake_rjamespage: no you have 2 options15:42
blake_rjamespage: now*15:42
jamespageblake_r, will do15:42
blake_rjamespage: either set it on each interface15:42
blake_rjamespage: or I can show you how to do it manually using the maas shell15:43
jamespageeither is good with me - its only 9 interfaces in total15:43
blake_rjamespage: then lets just do it over the API15:43
blake_rjamespage: "maas my-maas-session interface update {node-system-id} {interface-id} mtu=1500"15:44
jamespageblake_r, ok - trying that out now15:52
jamespagemaas applied to all interfaces on all nodes...15:52
blake_rjamespage: ??15:58
jamespageblake_r, testung now16:03
jamespageblake_r, http://paste.ubuntu.com/14993910/16:03
blake_rjamespage: looks good16:05
jamespageblake_r, 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master juju-br0 state UP group default qlen 100016:13
jamespagelooking good!16:13
blake_rjamespage: cool16:16
blake_rjamespage: please file that bug16:16
jamespageblake_r, promise I will!16:17
blake_rjamespage: thanks16:17
jamespageblake_r, https://bugs.launchpad.net/maas/+bug/154319516:25
mupBug #1543195 opened: unable to set mtu on default VLAN <api> <networking> <MAAS:Triaged> <MAAS 1.9:Triaged> <https://launchpad.net/bugs/1543195>16:38
jamespageblake_r, hey - so I think xfs support for disks formats is on the roadmap right?17:15
jamespageapologies for the questions - so close to 0 post deploy hacks...17:15
roaksoaxjamespage: xfs will be added yes17:18
jamespageroaksoax, awesome - thanks for confirming17:18
roaksoaxjamespage: in fact, i think we can even get it to 1.917:19
mupBug #1531843 changed: [Xenial 1.10] IPMI query fails <1.10> <ipmi> <python3> <xenial> <MAAS:Incomplete> <https://launchpad.net/bugs/1531843>17:42
haasnthe power status for one of my machines is stuck on “power error”, with the last machine event being: Failed to query node's BMC - Power state could not be queried: [Errno 8] Exec format error20:16
haasnI have fixed the problem in question and verified it works by manually opening a python instance and running os.execl on the file in question20:16
haasnBut for some reason maas doesn't seem to get this. There are no machine events since then, despite me hammering the “check now” button. I don't see any ipmipower processes being spawned either. I tried rebooting the machine, restarting apache2 etc. but maas just doesn't seem to want to retry20:17
roaksoaxhaasn: what maas version are you using? What hardware are you using? Are you sure the power credentials in MAAS are the correct ones?20:19
roaksoaxhaasn: is there anything more on the logs (pastebin)20:19
haasnroaksoax: maas v1.9, hardware of the machine in question is a sun fire x4200, I am sure the power credentials in MAAS are the correct ones because I can copy/paste them from the webUI to my own ipmipower invocation and turn the machine on and off20:20
haasnsu - maas20:20
haasn/usr/sbin/ipmipower -D LAN_2_0 -h X.Y.Z.Z -u maas -p <snip> --stat  # this works fine20:20
roaksoaxhaasn: is anything shown on the logs ?20:20
haasnHmm, seems there's a new entry as of 20:18: “password invalid”20:21
haasnwhy it took half an hour for it to switch from “exec format error” to “password invalid” I do not know20:21
haasnThis seems like it might be executing the wrong version of ipmipower. I replaced ipmipower by my own version which contains a patch for the “password invalid” bug on X4200 servers and moved it to /usr/sbin/ipmipower and the first one to /usr/sbin/ipmipower-orig20:22
haasnIs it possible it's executing /usr/sbin/ipmipower-orig somehow?20:22
roaksoaxhaasn: not at all20:22
haasnI can't find the exact location of the command it's executing in src/provisioningserver/drivers/power/ipmi.py anywhere20:23
haasnOkay, some update: It *is* spawning new ipmipower processes, they just exit so ridiculously fast that I couldn't see them in `watch`. With -n0.1 I see them popping up for a split second20:24
haasnMaybe it's time to install auditd and log all process spawns so I can figure out what it's executing? Unless there's a simpler way20:24
haasnWhy it can't include the command it executed in the log files is beyond my understanding20:25
haasnWell, then again, so is “why can't it log every failure to the log files instead of only every 30 minutes?”20:25
roaksoaxhaasn: you could try hacking ipmi.py to log out what IPMI command is being send20:29
roaksoaxin MAAS' logs20:29
koapsdoes anyone know what happened to MAAS 1.8.3? I don't see it available in stable anymore20:34
roaksoaxkoaps: it has been superseeded by 1.920:35
koapsya, but shouldn't it still be avail to get if I use the version name? I see 1.7.6 and 1.7.3 there20:35
haasnHeh, auditd didn't want to work but I found a beautifully simple way to accomplish what I wanted: while true; do pgrep -a ipmipower; sleep 0.1; done20:36
haasnthis gave me a nice log of (virtually) all executed ipmipower commands20:36
haasnOkay, I found the error and solved it20:37
haasnThe problem was that it was adding -W opensesspriv, and the -W flag overrides previous flags - so my “patched” version of ipmipower (which was really just ipmipower-orig -W sun20 "$@") didn't really do anything. I changed it to ipmipower-orig "$@" -W sun20 and that way my sun20 workaround flag overrides the previous opensesspriv work-around flag20:38
haasnNow it works20:38
haasnAnd I can control the power status of all my servers just fine20:38
haasnmaas desperately needs some mechanism for adding your own IPMI workaround flags to nodes20:39
roaksoaxkoaps: i wonder if the PPA removed it, since we have not made the removal of the version from the PPA itself20:47
roaksoaxhaasn: you can always file a bug :)20:47
haasnroaksoax: there is already a bug on the issue, I commented on it20:48
haasnBut I don't really want to be spending the next few months manually running ipmipower commands until it gets fixed20:48
roaksoaxhaasn: well, that may seem like an issue with the firmware rather than MAAS itself20:51
roaksoaxhaasn: since the firmware requires non-standard paramters20:51
haasnroaksoax: Yes okay but that doesn't really help admins who want to provision servers with MAAS. FreeIPMI contains these work-arounds for a reason, not exposing the ability to use them to the admins in charge of running the machines when it would literally be nothing more than one input box and one line of code for appending it to the command line is just being annoying for the sake of annoying, IMO20:53
roaksoaxhaasn: right, but in situations like this, providing ability to do work around is not always something that will improve MAAS' robustness20:59
roaksoaxhaasn: as this is an isolated issue20:59
mupBug #1543286 opened: Exception: 'Node' object is not iterable <MAAS:New> <MAAS 1.10:New> <https://launchpad.net/bugs/1543286>21:03
mupBug #1543301 opened: MAAS should only allow creating 15 partition <storage> <MAAS:Triaged> <https://launchpad.net/bugs/1543301>22:03
mupBug #1543301 changed: MAAS should only allow creating 15 partition <storage> <MAAS:Triaged> <https://launchpad.net/bugs/1543301>22:06
mupBug #1543301 opened: MAAS should only allow creating 15 partition <storage> <MAAS:Triaged> <https://launchpad.net/bugs/1543301>22:09
haasnThis is weird. Often, when I try deploying to a VM, I end up in the weird situation where the deploy process seems to be complete but the machine then just gets stuck in an offline state after it was supposed to reboot22:53
haasnif I ssh into the machine, I see The system is going down for reboot NOW!22:53
haasnBut.. it never reboots. It just powers off22:53
haasnAnd MAAS also still thinks it's turned on until I manually hit “check now”22:53
haasnPower control is via virsh22:54
haasnWhat could possibly be going wrong here? Evidently the machine thinks its rebooting, it just.. doesn't22:54
haasnHmm, seems to be an inherent part of KVM; “sudo reboot” has the same issue for some reason22:59
fluxcoreI use plenty of KVM stuff outside of maas/openstack and don't have any such problems...23:03
haasnI've used dozens of KVM VMs on this same vm host, and it's running like 5 other VMs right now that all work fine23:05
haasnas in I can reboot them23:05
haasnbut the ones used by maas have this inexplicable failure23:06
roaksoaxhaasn: Is this during deployment ?23:08
roaksoaxhaasn: also, maas doesn't query the VM's/BMC's every second. WE do it ever few minutes to ensure BMC's dont go crazy23:08
roaksoaxhaasn: i deploy about ~100 to ~200 vm's every few weeks for testing, and have not seen such behavior. However, I've seen a few occurrences where they all fail to PXE at once because there are too many VM's on the same host23:10
roaksoaxhaasn: but that's how the host's bridge was configured23:10
haasnroaksoax: Yes, this is during deployment23:13
haasnI just reproduced it again: clicked the machine, hit “deploy” in the web UI, chose ubuntu 14.04 lts and default kernel23:14
haasnit started deploying up until the point where rebooted the machine23:14
haasnand now it's just offline23:14
haasnI'll try removing the machine and re-adding/commissioning it in kvm. Maybe something got messed up because I used an xml dump to batch-create a bunch of VMs23:14
haasndestroyed the VM and recreated it and it seems to be working now23:35
haasnmust be some weird failure to do with the way it was created, idk23:35

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!