/srv/irclogs.ubuntu.com/2018/04/11/#maas.txt

anankemy first time with maas, and a stack of dell PE R710 systems. all systems are set to boot from pxe, and they enlist then power off. mass doesn't seem to be able to power them back on for comissioning though00:11
anankeit seems the nodes are shown in maas to use ipmi 2.0 over lan with IP 192.168.0.120, which is the default idrac setting. however, there are no interfaces on the maas controller that would talk to that subnet. what gives? shouldn't maas set the IPs for idrac during enlisting?00:12
mupBug #1754335 changed: [2.4, UI] Node action form takes a long time to disappear <ui> <MAAS:Invalid> <https://launchpad.net/bugs/1754335>01:06
Hey__smartctl validate empty file  what does this mean?02:19
Hey__I am hitting the ipmi successfully.02:19
Hey__it says power on and its green02:19
Hey__but when it tries to comission.. nothing happens.. or the result is it fails.02:20
Hey__I manually added the node as I didn't see it in the Dashboared.02:20
mupBug #1754335 opened: [2.4, UI] Node action form takes a long time to disappear <ui> <MAAS:Incomplete> <https://launchpad.net/bugs/1754335>03:21
=== frankban|afk is now known as frankban
mupBug #1763010 opened: Block devices not discovered during commissioning <MAAS:New> <https://launchpad.net/bugs/1763010>12:37
mupBug #1763010 changed: Block devices not discovered during commissioning <MAAS:Invalid> <https://launchpad.net/bugs/1763010>12:40
parlos_Good Morning13:52
mupBug #1763010 opened: Block devices not discovered during commissioning <MAAS:Incomplete> <https://launchpad.net/bugs/1763010>13:52
anankeis maas supposed to set up automatically NAT between the controller and fabrics? my enlisting and comissioning fails, while the logs indicate that the nodes can't reach outside world13:58
anankedocumentation is a bit sparse, or perhaps I'm not looking in the right place14:00
roaksoaxananke: nope14:02
roaksoaxananke: maas won't setup NAT14:02
anankethank you, that may explain a lot of the issues I'm having14:03
anankelooks like eventually the nodes fetch stuff directly from maas, presumably using it as a proxy. however, once enlisted, maas doesn't show any relevant data as to the amount of cores/mem/etc on the nodes. it does have the right IPMI setup, and it can power cycle them14:04
parlos_Can/Does an deployed MAAS change/update how nodes are commissioned? Got a challenge... Suddenly nodes could not be commissioned, ended up in busybox... complaining some missing driver. But the nodes were commissioned before...14:06
parlos_No change on my behalf.. and now, when testing the issue again, they commission just fine..14:07
anankeparlos_: hah. I have yet been able to comission a single node14:16
anankeeach node boots, loads the image, but then logs on maas show a metric ton of ureadahead errors14:16
parlos_Took me a while before I got it working too. Had a way too complicated network environment, >2 nics14:16
anankeeg: Apr 11 14:13:45 fast-stork ureadahead[1052]: ureadahead:/usr/lib/tmpfiles.d/x11.conf: Error retrieving chunk extents: Operation not supported14:16
anankeparlos_: I just have two nics: one for external network, another for internal (where all the nodes reside)14:17
parlos_Is this at the first boot? I.e. pre-commisson?14:17
anankethis is during comission. however, I'm not convinced enlisting works correctly either14:17
anankebecause the nodes don't show cores/cpu/etc in the maas interface14:18
parlos_during enlistment it will not show any hw specs...(AFAIK)14:18
anankeahh, ok14:18
anankemaas UI shows comissioning failed, and then for each module it has an empty log file14:21
parlos_:(14:21
parlos_do you have access to the console of the devices?14:22
parlos_my experience from first MAAS deployment, was that viewing the console helped.14:22
anankeI do, but I honestly am not sure what I would be looking at. for example the system just spent 5 mins waiting for something, then mass comissioning image proceeded with shutdown14:23
parlos_in my case, the device, me and maas disagreed on what was the first interface. Hence, it did the enlistment on interface X, then during commisioning it thought X was now Y...14:24
parlos_If it waits, then i guess it tries to reach an IP and it cant...14:24
anankeso here's full dump of /var/log/maas/rsyslog/<sample node>/date/messages: http://ix.io/17wV14:25
anankeI'm not sure if that's the right place to look at to determine what actual aspects of comissioning failed or not14:26
roaksoaxananke: yes maas has a caching proxy and by default apt would attempt to use it unless you disabled it14:27
roaksoaxparlos_: could be a kernel issue14:27
parlos_did not change kernel...14:28
anankeroaksoax: nope, didn't disable it. however, it seems to try direct route first, before it tries the proxy. this is a fresh install of ubuntu 16.04 with maas 2.3.014:28
parlos_ananke, are there multiple boots in that log?14:30
anankeparlos_: just one14:30
ananke~9 minutes from start to finish14:31
parlos_Ok, its a bit confusing.. at 14:12:58 its looks that its the kernel boot, but prior to this we got ssh keys (before kernel??) could be wrong..14:33
anankeparlos_: indeed, that does look confusing. however, that's how the rsyslog on the maas controller seems to have recorded this14:34
anankewhat user can I ssh as to the given node while it's being comissioned?14:38
parlos_nope....14:39
parlos_can you get a console via the BMC?14:39
anankeso what's the point of having 'Allow SSH access and prevent machine from powering off' check box for comissioning?14:40
parlos_dunno, have never tried it..14:41
parlos_I have the luxury to use iDrac so I have console access..14:41
anankeahh: 'As long as you've added your SSH key to MAAS, you can simply connect with SSH to the node's IP with a username of ubuntu.'14:41
anankeparlos_: these systems have idrac express, so no remote console14:42
parlos_i got those too.. then I walk into the noisy room, and the KVM...14:43
parlos_What is your network cfg?14:43
parlos_for the nodes?14:43
anankeI have a dozen R710s that we were going to surplus, and instead I figured I can try maas/openstack/openshift/whatever on them14:44
parlos_:) got R715s..14:44
anankeparlos_: i have one system to act as the maas controller. it has two NICs: external and internal. internal is connected to a basic switch with a flat network14:44
parlos_and the nodes are connected to the switch with one nic, where is the BMC connecteD?14:45
anankethe rest of the r710s are then connected to that switch, with their primary interface. i set them to use only pxe boot, from that first nic. idrac is set to be shared lan mode14:45
anankecorrect14:45
parlos_did you disable the other nics?14:46
parlos_(ok on idrac)14:46
anankenope, since my plan was to eventually use those other nics for something else (perhaps external network)14:46
anankeand clearly, they do use that one interface, since they boot, get the initial image, and the maas controller receives logs from them14:47
parlos_ok, my setup is similar. But nic2 is connected to another switch. 3+4 are disabled.14:48
anankeso now the question is what exactly fails during the comissioning14:48
parlos_agree, but the log is not clear...14:49
parlos_Do you have some other HW  platform that you could test? as to see if there is an MAAS kernel to R710 issue?14:50
anankeparlos_: unfortunately, not in that data center. I have another rack full of gear in another location, but I haven't finished the setup yet14:51
parlos_I would however be surprised it it was a kernel-hw issue...  How are the discs configed?14:52
parlos_hw raid?14:52
anankeyes. perc 6i, two disks in each node with raid 114:52
anankeso a very basic setup14:54
anankeI'll see if logging into the nodes while they're in the process of comissioning will yield any clues14:55
parlos_not r710 directly, but another guy had an issue with HP dl380, and it was a bios issue..to new..14:56
anankeI got all of the r710s up to bios 6.4.0/6.5.0, and tried to get all of the idracs updated too14:57
parlos_There is an issue/bug at https://bugs.launchpad.net/ubuntu/+source/ureadahead/+bug/162843814:58
parlos_In MAAS does it list "Commision failed?"14:58
anankeyes15:00
anankeand I saw that bug earlier, sadly it leads to nowhere15:01
parlos_I'd try to get console access, and view the output. From the syslog we do not see the thing that caused the Error that resulted in a fail...15:02
anankeparlos_: I'm not sure I can even login to the console though15:05
parlos_you dont have to login, just watch the output..15:05
anankeas in, it's not like there is a login prompt15:05
anankethat's the thing. there's nothing out of the ordinary. and comissioning failed errors appear on the maas controller long time before the nodes finish and shut down15:06
parlos_It sounds to me that then node cannot properly talk to the maas server.. (for some reason).15:08
parlos_afaik, so it boots, starts some actions (based on the tftp/pexe info), then as some point it need to talk to the maas. The maas waits for this, and if this does not happen15:09
parlos_MAAS calls it failed, while the node timesout and tries again...eventually it gives up and shuts down.15:10
parlos_ok,. have to go. Have a nice day, and good luck!15:13
srihashi guys, currently the network configuration on the depoloyed node is in /etc/network/interfaces.d/*.cfg rather than /etc/network/interfaces. Is there a way to tell MAAS to do it at  /etc/network/interfaces? thank you15:14
mupBug #1763059 opened: [2.4] DHCP is being configured on a rack controller that is not set to run DHCP <MAAS:In Progress by blake-rouse> <https://launchpad.net/bugs/1763059>15:16
roaksoaxsrihas: no, network config is done by cloud-init and does it in interfaces.d/*.cfg15:29
roaksoaxparlos: hceck that rackd.conf:maas_url has the IP of the region instead of localhost15:30
srihasroaksoax: I saw a bug that JUJU is looking at interfaces file, will it be a problem if I am going to dpeloy OpenStack with JUJU later on this node?15:32
roaksoaxsrihas: juju should be handling e/n/i.d/*.cfg just fine15:50
srihasroaksoax: thank you :)15:50
anankeis there a way to login from the console of a system that's in the process of being comissioned, other than the ssh?16:02
anankeahh ffs, I see one of the potential problems16:16
anankewhen I hit 'comission', maas powers on the system. before that system has a chance to even fully POST, maas issues a forced reboot via the ipmi16:17
anankewtf16:17
anankethen it claims they failed comissioning, while the nodes are booted into some maas image16:19
anankethat's insane16:19
anankewhy would maas wait so little time for them to post? is that a configurable option?16:20
=== frankban is now known as frankban|afk
anankeit power cycles them after roughly 60 seconds. that's crazy16:28
anankeI feel like this is a bug, since I never configured any timeout settings in maas16:30
anankeahh ffs: https://bugs.launchpad.net/maas/+bug/163510716:35
roaksoaxananke: /win 416:59
roaksoaxerr16:59
roaksoaxsry16:59
mupBug #1763093 opened: Gateway can be choose in wrong subnet <MAAS:New> <https://launchpad.net/bugs/1763093>17:04
Hey__when I add a physical interface to a node I'm about to comission, I see Error: node must be connected to a network.18:28
Hey__Does the node need to have internet access?18:28
Hey__I mean.. its connected to an internal network with no internet access18:29
roaksoaxHey__: no18:36
roaksoaxHey__: if it is to *commission* no18:36
roaksoaxif it is to deploy, yes18:36
mupBug #1763147 opened: [2.4, UI] Overall service status' not updating correctly <MAAS:Triaged by blake-rouse> <https://launchpad.net/bugs/1763147>18:47
=== iatrou_ is now known as iatrou
=== icey_ is now known as icey
=== aimeeu__ is now known as aimeeu
mupBug #1763169 opened: [2.4, enhancement] Add UI option to allow/disallow proxy usage <MAAS:New> <https://launchpad.net/bugs/1763169>20:35
mupBug #1763169 changed: [2.4, enhancement] Add UI option to allow/disallow proxy usage <MAAS:Triaged> <https://launchpad.net/bugs/1763169>20:44
mupBug #1763169 opened: [2.4, enhancement] Add UI option to allow/disallow proxy usage <MAAS:Triaged> <https://launchpad.net/bugs/1763169>20:47
Hey__roaksoax, under Nodes > Interfaces I geat an error it says Error: Node  must be connected to a network.  but the node is connected21:29
bladernrroaksoax, blake_r, newell_ do you guys remember what file is handed out when a Power8 box PXE boots via MAAS?  is it the same pxelinux.0 file that x86 gets?21:30
bladernror does it get a different file in /var/lib/maas/boot-resources/*21:30
newell_bladernr: power8 uses powernv afair21:40
newell_bladernr: which uses petitboot...which is the binary bootloader so no pxelinux.0 file needs to be downloaded.21:41
bladernrhrmmm... yeah, that's what I recall.  I'm looking at an openpower box (well, looking at a dump of the petitboot menu) and it's grabbing pxelinux.0 from MAAS.21:42
bladernrand then complains that some temp file is not a valid ELF binary21:42
bladernrmeh, was just checking, the whole thing's a bit of a mess.21:42
bladernrthanks!21:42
roaksoax bladernr: /var/lib/maas/dhcpd.conf will tell you what file is for power 821:44
bladernrahhh thanks roaksoax that's the confirmation I needed.21:45
roaksoaxbladernr: https://pastebin.ubuntu.com/p/fKNqvd9v7G/21:46
Hey__I am having problems commissioning nodes.  I don't see the node in the Dashboard, I only see it in Observed under subnet. So I add it manually. adding the ipmi interfaces22:19
Hey__When I Select Commission, it runs for a while then fails.22:20
Hey__What logs do I check to see what the issue is?22:20
Hey__Events show Queried node's BMC - Power state quried o:on22:21
Hey__what power type do I use for hyper-v?22:30
Hey__ohh..i see. for that VM, I had to do it manually22:33
mupBug #1763214 opened: [2.4, UI, vanilla]  Zone details page not formatted correctly <vanilla-transition> <MAAS:Triaged> <https://launchpad.net/bugs/1763214>23:42
mupBug #1763215 opened: [2.4, UI, vanilla] Group by on 'Subnets' tab is wrapped <vanilla-transition> <MAAS:Triaged> <https://launchpad.net/bugs/1763215>23:42
mupBug #1763216 opened: [2.4, UI, vanilla] Subnet in interfaces table is gone <vanilla-transition> <MAAS:Triaged> <https://launchpad.net/bugs/1763216>23:45
mupBug #1763217 opened: [2.4, UI, vanilla] Delete subnet text is wrapped and missing warning icon <MAAS:Triaged> <https://launchpad.net/bugs/1763217>23:45
mupBug #1763218 opened: [2.4, UI, vanilla] Delete range (inside subnet) text is wrapped <MAAS:Triaged> <https://launchpad.net/bugs/1763218>23:45
mupBug #1763219 opened: [2.4, UI, vanilla] Delete fabric confirmation text is misplaced <vanilla-transaition> <MAAS:Triaged> <https://launchpad.net/bugs/1763219>23:48
mupBug #1763220 opened: [2.4, UI, vanilla] Compose pod action form has misplaced buttons <MAAS:New> <https://launchpad.net/bugs/1763220>23:48

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!