[05:30] Bug #1733847 opened: same ip got assigned to two different container === frankban|afk is now known as frankban [09:18] Im trying to use MAAS on an IBM Bladecenter H, but im stuck at a strange error. Sometimes, when i PXE boot the server is stuck at a blank screen with a blinking "_" cursor. [09:19] I can see the DHCP negotiations in a tcpdump on the MAAS controller, but i seems like the syslinux is never downloaded and executed. [09:19] Any suggestions on how to fix this? [10:34] Hi all! We got a situation that took some debugging, and just wanted to know if this sounds like a bug in MAAS or not. We have a bunch of rack servers with 2 NICs. In MAAS we've bonded them together as one. We deploy Ubuntu 17.04 hosts on these and run multiple KVM virtual machines on the host. Our situation was such that one of the servers had one of the ethernet ports disconnected. It had no physical wire going in to it. The symptom was packet [10:34] loss on one of the KVM virtual machines, but not on all of them. While running tshark on the host we found out that packages are not even there. That lead us to look at the NICs and realized that one was disconnected. When we removed the bounding in MAAS, the traffic went back to being OK. The funny part was that this didn't affect the traffic on the host itself, nor on all the KVM guests. Just this one (at least. Could be been more if we [10:34] cloned more) [10:34] and it was/is MAAS 2.2.2 we are using [11:32] anyone with experience in proxying maas with nginx? [14:28] mornign === torontoyes is now known as heyya === roaksoax changed the topic of #maas to: World's best bare-metal provisioning tool | Docs: http://maas.io/docs | Mailing list: https://lists.ubuntu.com/mailman/listinfo/maas-devel | MAAS 2.3.0 now released! === roaksoax changed the topic of #maas to: World's best bare-metal provisioning tool | MAAS 2.3.0 now released! | Docs: http://maas.io/docs | Mailing list: https://lists.ubuntu.com/mailman/listinfo/maas-devel [17:14] roaksoax: can we restart a regiond worker? maas-regiond-worker@1 [17:14] its hung importing images but I cant restart the entire regiond service right now. [17:26] vogelc_: you can === frankban is now known as frankban|afk [18:50] In oreder for me to deploy windows 10 image with MAAS, I will need to create a sysprep image if I want unattended install, correct? Is that the only way? [19:07] Bug #1711203 changed: Deployments fail when Secure Boot enabled

[20:20] hey whats up all [20:21] does 2.3 not auto-detect power type and bmc details? [20:21] just trying 2.3 for the first time in the wild [20:22] ultimate fail due to the power type configuration not being picked up/autofilled [20:22] dah [20:23] bdx: do you have cloud-init logs ? [20:24] bdx: ther'es only been one change on ipmi auto detect that I can recall and should not impact you at all [20:25] roaksoax: yeah, omp [20:26] roaksoax: http://paste.ubuntu.com/26060222/ [20:26] ooh my b [20:26] thats regiond [20:27] bdx: was that a fresh install ? [20:27] yes [20:27] http://paste.ubuntu.com/26060226/ [20:27] clou-init.log^ [20:27] ill add these to the bug [20:28] bdx: can you ssh into the machine [20:28] bdx: while it is "commissioning " [20:28] it wont even let me commission [20:28] bdx: change the power type to manual [20:28] I get blocked in the gui from it even happening [20:29] ok [20:29] done [20:29] bdx: so where did you get that log form ? [20:29] the cloud-init one ? [20:30] var/log/cloud-init.log on my maas-region-rackd-00 [20:30] I'm assuming you wanted the one from the machine [20:30] bdx: yes, so do something else [20:30] that Im trying to commission [20:30] yeah, ok [20:30] bdx: /var/log/maas/rsyslog/maas-enlisting-node/ [20:30] bdx: /var/log/maas/rsyslog/maas-enlisting-node//messages [20:30] i think that's the path [20:30] its snap install [20:31] but I see it [20:31] ok nm [20:31] I thought I did [20:31] Bug #1734763 opened: bmc discovery/auto-detect broken in 2.3 [20:31] I have no var/log/maas bc the snap install [20:32] I wonder if the snap is proxying those logs to syslog [20:32] heres the syslog from the maas-region-rack-00 http://paste.ubuntu.com/26060259/ [20:33] bdx: check /var/snap/maas/common/log/ [20:34] roaksoax: http://paste.ubuntu.com/26060271/ [20:35] bdx: ok, so just change the machine to manual power type, commission it with the option to allow ssh, manually power it on [20:35] bdx: and ssh into the environment [20:36] bdx: wait for the commissioning to finish [20:36] ok [20:36] and gather /var/log/cloud-init-ouput.log [20:36] bdx: the issue i think you are having is that the ephemeral image cannot access the internet, hence it cannot download ipmi related tools and fails to configure the BMC [20:36] bdx: because on the hardware we have, it all works just fine [20:37] really [20:37] so from my region-rack-controller [20:38] http://paste.ubuntu.com/26060285/ [20:38] possibly its more contrived then that though [20:40] well, I didnt think I would be here (at the datacenter) for more then a few moments, totally spaced grabbing my power adapter for the macbook [20:40] power is on 5% [20:40] stupid [20:40] anyway [20:41] hopefully we can get some info from this node before my rig dies [20:41] otherwise I'll be back online in an hourish [20:41] just fyi [20:44] https://www.dropbox.com/s/oxx5xfv6bf5dv3h/Screen%20Shot%202017-11-27%20at%2012.42.58%20PM.png?dl=0 [20:44] roaksoax: I cant see what ip the node has [20:44] I dont know by what means I can get at it [20:45] ahh nm [20:45] got in [20:46] roaksoax: cloud-init.log http://paste.ubuntu.com/26060310/ [20:46] cloud-init-output.log http://paste.ubuntu.com/26060311/ [20:47] ahh [20:48] well shoot === newell_ is now known as newell === newell is now known as newell_ [20:50] bdx: there you go :) === newell_ is now known as newell [21:01] Bug #1734763 changed: bmc discovery/auto-detect broken in 2.3 [21:01] Bug #1734765 opened: [2.x] MAAS snap does not store rsyslog from pxe booting machines [21:18] roaksoax: nothing changed in my environment/network that would cause this though [21:21] roaksoax: so where to go next? [21:22] part of me just wants to revert to 2.2 so I can get my stack back [21:22] but I know we should work this out [21:22] I see you have marked the bug invalid, meaning it must be something on my end [21:23] just scrambled wondering what it could be [21:23] ill do some more digging [21:24] but I mean, I went from working 2.2, wiped the slate, installed 2.3 [21:24] now my nodes wont commission [21:24] that cant be user error [21:24] possibly somehow it is [21:26] bdx: is proxy running ? [21:26] yeah [21:28] bdx: ok, i'm gonna re-deploy mt env using the snap [21:28] ok, thx [21:29] bdx: is this MAAS though? 10.10.0.113:8000 [21:30] oooh [21:30] no [21:30] I see the issue [21:30] I'm using haproxy infront of maas [21:30] proxying port 80 -> 5240 [21:31] I'm guessing I need 8000 too then [21:32] so, I reset my maas-url to be maas-url=http://10.10.0.113/MAAS [21:32] right [21:32] but masa will always add :8000 for the proxy [21:32] so maas is effectively being told "your proxy is in 10.10.0.113:8000" [21:32] ahhh [21:32] ok [21:32] well [21:33] bdx: but might as well be haproxy [21:33] yeah, but I'm not proxying port 8000 [21:33] so its probably hitting 10.10.0.113:8000 and nothing is there [21:34] so, the answer here is to forward port 8000 too then [21:34] got it [21:34] that makes sense [21:35] so in a non-snap world, pointing rackd.conf to maas_url=http://10.10.0.113/MAAS means that you are telling it that the region is on that IP [21:35] got it [21:35] so when deploying machines, maas tells them "here's your metadata server (10.10.0.113) and your proxy is on :8000" [21:36] on the snap world, the same happens, but you just dont modify rackd.conf anymore [21:36] totally [21:36] so the only way the maas-url can change, is if the new ip accounts for proxying 8000 as well as 5240? [21:37] 5240 -> 80, 8000 -> 8000 [21:37] bleh [21:37] 80 -> 5240, 8000 -> 8000 [21:37] right, so if you tell maas is on "http://XXXX:2020/MAAS" then the maas proxy is still running in :8000 in the same XXXX [21:37] totally