[00:02] <mup> Bug #1727551 opened: [2.3] Commissioning shows incorrect results <MAAS:New> <https://launchpad.net/bugs/1727551>
[00:17] <mup> Bug #1727551 changed: [2.3] Commissioning shows results from script that no longer exists <MAAS:Triaged> <https://launchpad.net/bugs/1727551>
[00:20] <mup> Bug #1727551 opened: [2.3] Commissioning shows incorrect results <MAAS:New> <https://launchpad.net/bugs/1727551>
[01:03] <mup> Bug #1727561 opened: [2.3, HA] Machines fails commissioning and testing and MAAS shows no errors <MAAS:Triaged> <https://launchpad.net/bugs/1727561>
[01:06] <mup> Bug #1727568 opened: [2.3, HWTv2, regression] Hardware Tests tab does not show what tests are running <MAAS:New> <https://launchpad.net/bugs/1727568>
[01:15] <mup> Bug #1727568 changed: [2.3, HWTv2, regression] Hardware Tests tab does not show what tests are running <MAAS:New> <https://launchpad.net/bugs/1727568>
[01:30] <mup> Bug #1727568 opened: [2.3, HWTv2, regression] Hardware Tests tab does not show what tests are running <MAAS:Triaged> <https://launchpad.net/bugs/1727568>
[02:12] <mup> Bug #1727575 opened: [2.3, HWTv2, regression] Hardware Test tab doesn't show latest results of newer runs and only show an older aborted run <MAAS:New> <https://launchpad.net/bugs/1727575>
[02:12] <mup> Bug #1727576 opened: [2.3, HWTv2] When test timesout there's no log <MAAS:Triaged> <https://launchpad.net/bugs/1727576>
[02:12] <mup> Bug #1727577 opened: [2.3, HWTv2] Hardware Test tab updates UI when tests selected but not running <MAAS:Triaged> <https://launchpad.net/bugs/1727577>
[12:46] <mup> Bug #1727721 opened: Show errors when adding a chassis <MAAS:New> <https://launchpad.net/bugs/1727721>
[15:07] <mup> Bug #1727754 opened: [2.3, API] Creating a bridge doesn't use the parent's mac address by default <MAAS:Triaged> <https://launchpad.net/bugs/1727754>
[15:30] <ybaumy> roaksoax: thanks for bumping this. its like forever now
[15:39] <roaksoax> ybaumy: which one?
[15:39] <ybaumy> roaksoax: the resolv.conf
[15:39] <ybaumy> roaksoax: that was you bumping right?
[15:41] <roaksoax> ybaumy: yup, i've been waiting forever to get that one through
[15:42] <ybaumy> roaksoax: its ridiculous how long we have to wait. do they even know that name resolution is not a small thing when using cloud init and stuff ;) i mean common
[15:42] <roaksoax> well to be fair this only affeects 2.3 :)
[15:43] <ybaumy> roaksoax: thats what im using.. currently
[15:43] <ybaumy> though
[15:44] <roaksoax> ybaumy: we put a work around in it
[15:44] <ybaumy> roaksoax: really i missed and havent even tried in weeks
[15:45] <roaksoax> ybaumy: but you need to uyse internal proxy
[15:45] <ybaumy> roaksoax: well i have a squid running i can try that
[15:45] <ybaumy> roaksoax: how to do? just set the proxy in settings page?
[15:46] <roaksoax> ybaumy: maas, by defualt runs internal proxy
[15:46] <roaksoax> ybaumy: so if it is using that no changes needed
[15:47] <roaksoax> then upgrade to the latest 2.3
[15:47] <ybaumy> roaksoax: will try tonight when im home thanks
[16:43] <shadoxx> Has anyone in here ever experienced an issue where the interface you use for pxe provisioning, which has a static ip, occasionally loses its ip and needs to be manually reset after a bit
[16:43] <shadoxx> From what I can tell, it's trying to renew a dhcp lease, when it doesn't have one
[16:43] <shadoxx> a quick `ifdown eno1; ifup eno1` fixes the problem until the next time it happens
[16:43] <shadoxx> But I hate having to loging to the ilo every 2 or 3 hours
[16:55] <roaksoax> shadoxx: hold on, you have a machine that has a static ip in /e/n/i, but then it loses its ip ?
[16:55] <shadoxx> roaksoax: correct
[16:55] <shadoxx> There's no other DHCP server on the network. In fact, that interface is hosting the DHCP server
[16:56] <shadoxx> And I can't get this freaking node to commission in the first place and I'm about to tear my hair out
[17:00] <shadoxx> roaksoax: it only has the address and netmask defined in /e/n/i. there's no gateway since it's all just on the same switch port group/vlan
[17:08] <roaksoax> shadoxx: that's strange... something muyst be restarting networking for that to happen ?
[17:15] <shadoxx> roaksoax: not entirely sure...i'm going to keep an eye on it. like I said, at least I have iLO so I can remote in a fix it that way
[17:16] <shadoxx> Just, it's getting annoying.
[17:21] <shadoxx> Ok, next question. Does anyone know how to load kernel drivers in rescue mode? Having an issue where I'm trying to boot the node so I can upgrade the iLO firmware, but the hpilo module is nowhere to be found
[17:22] <shadoxx> In regular mode it works fine
[17:32] <fabi_> hi @ll
[17:35] <fabi_> I installed a lxd container with MAAS, running the region-controller and rack-controller in it.
[17:37] <fabi_> The other day I ran out of disk space which caused me to increase my default zfs.img from 15G to 50G. That issue is fixed for now, however, while having the maas container down I noticed that all nodes that are managed by this container and its controllers were down.
[17:38] <fabi_> I couldn't reach them via network. connecting via IPMI revealed that the interfaces (they get the ip via dhcp from the controller) suddenly were down.
[17:39] <fabi_> the moment my maas container was working again, the interfaces came back up.
[17:40] <fabi_> I can't seem to find in the documentation this behavior and googling the issue shows me the opposite saying if I loose maas I do not loose any services.
[17:40] <fabi_> does anyone know more about this?
[17:46] <shadoxx> fabi: sounds like the nodes tried to renew the lease and couldn't find the dhcp server, so they go of the lease, even though that's not really supposed to happen
[17:46] <shadoxx> it sounds like you're running into a similar issue I am
[17:47] <shadoxx> I have a static IP set for an interface, but it loses it for some reason. This is on the MaaS controller itself though
[17:47] <fabi_> do your nodes loose connectivity then as well?
[17:48] <shadoxx> fabi_: i haven't, as of yet, been able to commission any nodes. i'm definitely still in the "discovery" phase of my deployment
[17:48] <fabi_> using lxd?
[17:48] <shadoxx> Haven't even gotten that far. :]
[17:48] <shadoxx> I have a MaaS controller, some VLANS, and a lot of green around the gills
[17:49] <shadoxx> I'm a week into trying to bootstrap the cluster.
[17:49] <fabi_> let me know if you need help, but I find this one here excellent: https://docs.ubuntu.com/maas/2.1/en/installconfig-lxd-install
[17:49] <fabi_> https://docs.ubuntu.com/maas/2.2/en/installconfig-lxd-install
[17:50] <shadoxx> Thanks! I was actually planning on doing ESXi 6.5 and then installing MaaS on that. But lxd seems like a good alternative too. I'm running HP DL385p Gen8s who don't like to take an OS or boot
[17:50] <shadoxx> Getting Ubuntu on it required me to factory reset everything. It reassigned all of its SCSI ids and suddenly booted off the network
[17:51] <fabi_> shouldn't be big of an issue using ESXi and just install it straight from the package https://docs.ubuntu.com/maas/2.2/en/installconfig-package-install (its essentially what you would do in the LXD container anyways). However, the tipping point is the configuration of your interface in the hypervisor to make sure it is in the right vlan(s).
[17:52] <shadoxx> I'm handling VLANs at Layer 2 on the switch, so luckily I don't have to worry about that. :]
[17:53] <shadoxx> Any issues to look out for you've encountered? This MAAS cluster is completely segregated with no WAN access. I'm running a full apt mirror internally and that's what I'm bootstrapping everything off of
[17:54] <fabi_> never had an airgapped maas cluster running but a starting point would probably be ntp and a mismatch of time in the hypervisor/vm and nodes?
[17:55] <shadoxx> Got NTP all settled
[17:55] <fabi_> error messages in journalctl -f ?
[17:55] <shadoxx> as soon as this node fails to comission, i'll be able to see what's going on
[17:55] <shadoxx> If I had to guess, it's probably related to how crappy these machines are to configure. I bet it's an HP issue and not an MaaS/Environment issue
[17:56] <shadoxx> The one thing I wish I could do is install the ilo kernel module in rescue mode so I can flash the iLOs to the latest verion in the nodes
[17:56] <shadoxx> When I load into rescue mode, it's barebones
[17:57] <fabi_> I just had failure of commissioning because of mismatching times of the controller and the node itself, so the api which serves the node was throwing authentication errors
[17:57] <shadoxx> The node I was trying to do that with just failed, so I'll check the times now
[17:57] <fabi_> https://docs.ubuntu.com/maas/2.2/en/troubleshoot-faq Nodes hang on "Commissioning" Timing issues
[17:59] <shadoxx> So, once a node has failed, is it possible to login via ssh to figure out what went wrong? Or does it have to be in rescue mode?
[18:00] <fabi_> when he failed during commissioning or while deploying?
[18:00] <shadoxx> During commissioning
[18:02] <fabi_> cloud-init reports logs to the controller which can be viewed in the UI under the node's tab commissioning/events or on the controller in the corresponding rsyslog folder in /var/log/maas/rsyslog/<hostname>
[18:12] <shadoxx> So, the controller doesn't provide ntp by default?
[18:20] <fabi_>  By default, the MAAS region controller uses ntp.ubuntu.com. As you use an air-gapped cluster that won't work
[18:21] <fabi_> you can reconfigure it to the ntp of your air-gapped network: https://docs.ubuntu.com/maas/2.2/en/installconfig-network-ntp
[18:24] <shadoxx> Yeah, I figured that part out. I have a bastion host that lets me access it from the wider network. Just threw an NTP server on there and went wild
[18:25] <shadoxx> It's commissioning right now. We'll see if that was the silver bullet I needed
[18:26] <fabi_> I cross the fingers :)
[18:29] <roaksoax> fabi_: if you have set static networking, which is default then there should be no networking issue
[18:29] <roaksoax> unless your maas in the gateway
[18:29] <roaksoax> which would prevent you from accessing them
[18:36] <fabi_> roaksoax: my maas is not the gateway. my subnet is managed so all aspects of IP address allocation should be handled by maas, including the dhcp-leases and assigned static addresses.
[18:40] <fabi_> roaksoax: when you say static networking you mean unmanaged using just reserved IP ranges? I have nothing in my static routes
[18:41] <fabi_> roaksoax: so adding my gateway and dns to the static route should fix the problem if my maas controller goes down the interfaces on each node are down too?
[19:28] <fabi_> roaksoax: my IP mode for this interface is also Static assign and I just tested to see what happens if I take down the maas-controller - in particular if the dhcp server is not responding. looks like every 260 seconds the /usr/sbin/dhclient-script is running and if it does not receive any leases it just sets everything down, making the dhcp-server a single point of failure for all deployed nodes using dhcp...
[19:28] <shadoxx> that seems like the issue that i'm running into as well
[19:28] <fabi_> I dont find documentation of that or how to change it
[19:28] <shadoxx> I think that might be an Ubuntu vs MaaS thing
[19:31] <shadoxx> ok, i have an error now
[19:31] <shadoxx> Node commissioning failure - 'cloudinit' running modules for config
[19:32] <shadoxx> https://bugs.launchpad.net/maas/+bug/1629578
[19:34] <roaksoax> fabi_: i mean statically configured in e/n/i, if you are using dhcp to configure interfaces, and the dhcp service goes away, then the machine will definitely lose its lease
[19:36] <fabi_> roaksoax: is there a way to have it keep the lease until it gains connection back to dhcp server?\
[19:36] <roaksoax> fabi_: you can increase the lease time
[19:36] <roaksoax> but other than that, no
[19:37] <fabi_> roaksoax: lease time increase as a custom dhcp snippet or where?
[19:37] <roaksoax> fabi_: yes, per subnet on 2.2, 2.3 you will be able to do that globally
[19:41] <fabi_> roaksoax: okay, and the only other workaround I can think of would be to Static assign an IP, however is that supported for non-ubuntu e.g. Centos installations or does it throw an error like "Custom network configuration only supported on Ubuntu. Using OS default configuration." ?
[19:54] <roaksoax> fabi_: it is supported, starting from maas 2.3
[20:25] <shadoxx> Looks like my error is related to MaaS not being able to find the required packages in my repo mirror
[20:25] <shadoxx> I guess the folders in my mirror aren't setup properly or something
[20:26] <shadoxx> Is there a way to supply a custom sources.list to my Nodes when commissioning?
[20:29] <roaksoax> shadoxx: go to settings > package repositories, you can only change mirror/add repositories to use
[20:32] <shadoxx> roaksoax: that's what I did. it doesn't like the way my mirror is setup for some reason
[20:35] <shadoxx> Other machines running the same OS update from it fine. This requires more investigationg, but at least I have a path forward now
[20:52] <fabi_> cant wait for maas 2.3 to be released ;)
[22:07] <shadoxx> i added the source repositories to my apt mirror and that seems have to fixed the missing package errors at least
[22:37] <shadoxx> Got my nodes commissioned and ready
[22:37] <shadoxx> What a journey it's been
[22:37] <shadoxx> Now to figure out exactly what to do with them...
[23:21] <roaksoax> shadoxx: uhmmm can you file a bug for that plz ?
[23:21] <shadoxx> roaksoax: if the deb-src repos are missing the machine fails to provision?
[23:23] <roaksoax> shadoxx: yeah, although I thought we dont set those anymore
[23:24] <roaksoax> shadoxx: what version of MAAS ?
[23:24] <shadoxx> 2.2
[23:44] <mup> Bug #1727884 opened: [2.3] Discoveries added as devices with dynamic IPs should display their last-seen IP in the device listing/details pages <MAAS:Triaged> <https://launchpad.net/bugs/1727884>
[23:50] <mup> Bug #1727884 changed: [2.3] Discoveries added as devices with dynamic IPs should display their last-seen IP in the device listing/details pages <MAAS:Triaged> <https://launchpad.net/bugs/1727884>
[23:53] <shadoxx> How do I troubleshoot a failed deployment?
[23:54] <shadoxx> It's in the FAQ. Let me read that firat...
[23:54] <shadoxx> first*
[23:56] <mup> Bug #1727884 opened: [2.3] Discoveries added as devices with dynamic IPs should display their last-seen IP in the device listing/details pages <MAAS:Triaged> <https://launchpad.net/bugs/1727884>