[08:32] admin0: there are some ways by adding/changing the curtin installer scripts if i'm correct. But not easy. [08:36] admin0: You can check this article, maybe it will help you: https://insights.ubuntu.com/2017/06/02/customising-maas-installs/ [10:11] BlackDex, thanks [10:11] checking [10:45] BlackDex, exactly what i needed [10:45] many thanks [12:31] admin0: yw :) [15:12] Bug #1749210 opened: node powered off after reboot from rescue mode

[15:14] Bug #1749210 opened: node powered off after reboot from rescue mode

[16:09] Bug #1741013 changed: [Wishlist] Ability that can add custom cloud-init configuration [17:09] Bug #1749246 opened: nodes boot into 4.13 (hwe kernel) with no minimum kernel set

[17:12] roaksoax: ping [17:15] Bug #1749246 opened: nodes boot into 4.13 (hwe kernel) with no minimum kernel set

[17:21] Bug #1749246 changed: nodes boot into 4.13 (hwe kernel) with no minimum kernel set

[17:39] Bug #1749246 opened: nodes boot into 4.13 (hwe kernel) with no minimum kernel set

[18:48] mpontillo: ping [18:52] xygnal: pong [18:58] roaksoax: json outputs in that bug report for you. needs review and next stepzn [18:58] 1744765 [18:59] xygnal: I saw, thank you. Were you able to try a test environment in hardware? [19:01] no. my team is not convinced you have enough evidence to prove it as a vmware issue so they have not been in a hurry to try it. [19:02] do you have a technical theory WHYn [19:02] ? [19:03] we dont see any indication of perf issues on the box itself. [19:03] just the app, which by itself, consumes all memory regularly. [19:04] even with 8 threads now, no change. [19:05] cli commands are still quick [19:05] only UI impacted [19:11] xygnal: are you running xenial? or are you running a newer version ? [19:16] 16.04.3 [19:16] yes xenial [19:19] Does anyone know why a MAAS controller would fail to set a static IP on an interface? Once I set the interface to an IP, the physical adapter is never changed (verified with ip addr show) and upon reboot, the controller shows the interface as unconfigured. What's wrong? [19:21] xygnal: sorry for slow responses; I'm swamped with other work [19:21] xygnal: when you say "by itself", are you saying that even if there are zero users of the web UI, there are still issues? [19:22] jtcressy: at what point does it change? how are you setting the static IP? [19:23] Nodes>Controllers>(my only controller)>Interfaces>(second ethernet adapter)> edit physical [19:24] Am I wrong in assuming that the MAAS server would change the IP configuration of interfaces on the server it's running on? [19:24] jtcressy: the maas controller doesn't change interface configuration [19:25] jtcressy: is this a physical interface ? have you restarted maas-rackd again ? [19:25] It is a physical interface [19:25] So, then what I need to do is configure the interface via /etc/network/interfaces and then maas will be able to use it? [19:26] (fyi the way I have this setup is one interface is management network that provides dhcp and the other interface is the rack that the controller should provide DHCP to) [19:27] i double checked this and even maas cli takes ages to complete a machines read. [19:28] er no.. [19:28] not ages.... 120 seconds [19:29] xygnal: yeah that's a non issue with the cli [19:30] known*& [19:32] jtcressy: yes, for controllers MAAS uses the interfaces as-configured. [19:33] only the ui is slow. monsterously slow. even takes several minutes to see a node page. like, just a single node's page. [19:41] yaaay stuff works after doing all the static assignment stuff in etc/network/interfaces [19:51] How can I reconfigure dhcp? machines timeout on trying to get tftp [19:59] PXE E32: TFTP open timeout [19:59] commissioning node just sits there spinning and doing nothing [20:00] have wireshark open and I see the TFTP requests on the wire. maas server just doesn't reply atall [20:01] Bug #1749281 opened: MAAS 2.3 does not revert system proxy back to original state when MAAS is removed [20:06] Wait, do I have to manually install an OS on machines before I can commission it? [20:14] jtcressy: once you enable DHCP in MAAS, machines that boot from your DHCP-enabled VLAN should enlist automatically. are you saying MAAS is responding to DHCP but not TFTP requests? are you using any kind of network filtering, DHCP forwarding, or third-party DHCP server, or are your machines just booting directly onto a DHCP-enabled VLAN? [20:15] the machines are directly getting DHCP from the maas machine [20:16] absolutely zero network filtering. I have a gateway at the '.1' address of the network which is declared accordingly on the maas server [20:16] maas server is at '.2' with a static address [20:16] and that '.2' address is declared as the dhcp address for the maas controller [20:16] jtcressy: can you pastebin the output of /usr/lib/maas/maas-test-enlistment? [20:18] mpontillo: https://pastebin.com/wJUjUYNL [20:19] jtcressy: er, I meant for you to execute the script so I could see its output; sorry if that wasn't clear [20:19] oh ok [20:20] mpontillo: https://pastebin.com/1C70p445 [20:22] mpontillo: FYI, I just checked 'netstat -tnap | grep LISTEN' and port 69 does NOT show up in the output!!!! [20:23] jtcressy: well, looks like the TFTP server is responding and serving up bootloaders - at least when the test script runs. you can see that TFTP is indeed responding, so not sure why your command doesn't see it. [20:23] jtcressy: I would try `sudo dpkg-reconfigure -plow maas-rack-controller`and make sure that the URL specified is a URL that the booting machines will be able to reach [20:24] jtcressy: by default it uses `localhost`, which tries to determine the URL for enlisting nodes to use automatically - this doesn't always work [20:26] ok, i set the api address accordingly [20:27] rebooting a node still results in tftp timeout and a bunch of node -> maas controller tftp read requests on the wire [20:27] What should I do when TFTP is apparently *non-functional* ? [20:29] jtcressy: is there anything else about your setup that might be unusual? your pastebin proved that TFTP works to some extent; the `maas-test-enlistment` script basically uses `curl`to mimic what the bootloader does, and you can see the MAAS TFTP server responding [20:30] but this is weird.... netstat shows that port 69 is NOT listening, so how did the script verify that tftp was working? [20:31] jtcressy: well, that's why I was wondering if there was something unusual in your setup. ;-) [20:31] I installed this maas server from the "install maas region controller" option at the boot prompt of the installer ISO [20:32] and running this all in vmware shouldnt affect it as long as i have the networking in order, which I've verified using wireshark [20:33] and the fact that they can communicate with maas via dhcp [20:33] jtcressy: netstat doesn't show udp ports in LISTEN mode; if you grep your netstat for "udp" you will most likely see the open ports [20:34] jtcressy: for example, I see `udp 0 0 127.0.0.1:69 0.0.0.0:* 10842/python3` on my test MAAS [20:34] just 'netstat -na | grep udp' ? [20:34] jtcressy: I used `netstat -anp | grep udp` [20:35] jtcressy: did you enable `ufw` when you installed the server? if so, try turning it off? [20:36] ok, 69 shows up in netstat with udp now [20:37] didnt touch ufw [20:37] jtcressy: ok. well, you could copy the maas-test-enlistment script to another machine to test if it works from somewhere else across the network. if you pass it a hostname or IP address as a parameter, it will query a remote MAAS [20:38] jtcressy: so far you've proved that TFTP is up and running on the MAAS server, but you haven't proven you can talk to it over the network [20:39] working on it. rebooting it for good measure, then going to try tftp from my host system [20:41] tftp works from my host system [20:43] *and* this is to the IP that the commissionable nodes are supposed to be talking to [20:47] I just don't understand why '.191' is completely ignored from tftp requests, when '.1' is answered for tftp requests. [20:50] Ok, i'm noticing one thing. I filtered my wireshark for just tftp packets to compare it directly. When running tftp from my host machine, it initially talks to the maas server on port 64 [20:50] but the node keeps trying port 69 [20:50] the node also trys port 74 [20:50] oh, forget what I said. I was reading the wrong field [20:51] both of them are reading port 69 [20:57] mpontillo: I see another discrepancy: My host is requesting type "netascii" while the node is requesting type "octet" [20:57] so, netascii works and octet does not [21:00] jtcressy: netascii vs. octet shouldn't matter; MAAS will always return the same response regardless. [21:00] jtcressy: when you say .191 is ignored and .1 is answered, what do you mean? [21:00] .1 i.e "192.168.1.1" [21:03] supposedly, there is a hosts.allow file somewhere on the ubuntu server but i don't know where it is [21:15] mpontillo: ran tcpdump on the maas controller for port 69 and watched the node request pxelinux.0 and the maas controller just never responds https://pastebin.com/rQ08vDvq [21:19] jtcressy: weird, and yet when you use the script (or something like `curl -s`?) it replies? [21:20] yep [21:20] jtcressy: is the MAC seen by the PXE address known to MAAS? you might check /var/log/maas/*.log for any clues [21:20] I just ran tftp 172.16.207.2, then get pxelinux.0 from my macbook and it also works [21:20] i'll check the maas log. the mac address is known to maas [21:21] jtcressy: is 172.16.207.2 configured on ens38? [21:22] yes [21:22] 172.16.207.1 is my macbook [21:22] and 172.16.207.119 is the node, and the node got this IP from the maas server via dhcp [21:23] jtcressy: what is the output of `ip r g 172.16.207.191` on the MAAS server? [21:23] 172.16.207.191 dev ens38 src 172.16.207.2 cache [21:24] neither /var/log/maas/maas.log or ''/rackd.log say anything when I reboot my node and do pxe again [21:26] jtcressy: anything unusual in regiond.log that corresponds? [21:26] Nope just a bunch of HTTP 200 logs [21:43] jtcressy: if you run the tcpdump while one of the working nodes on that network requests the file, how does the output differ? I know tftp switches ports for the data transfer, so it's not clear to me if the 'port 69' filter would catch everything [21:44] No, tcpdump does not catch whether or not tftp actually works, I just used it to verify that the maas server was receiving it from the network [21:45] what DOES verify when tftp work, is when i see about 200 packets flow across the network that I can see via wireshark [21:46] What I need to know is if the tftp server that maas runs ever discriminates. This is the only thing I can think of, and I cant find ANY tftp logs at all [21:47] mpontillo: Found this, reading through it... https://bugs.launchpad.net/maas/+bug/1465000 [21:47] jtcressy: it doesn't discriminate with regard to the boot loader, but it does return a custom pxelinux.cfg depending on which node booted [21:49] but it fails to even send pxelinux.0 so that can't be it. [21:49] jtcressy: that was why I asked about interface configuration and routing; setups with the same subnet configured on more than one interface could cause this type of issue [21:49] But that's exactly what I don't have. ens33 has 172.16.97.145/24 and ens28 has 172.16.207.2/24. They are completely different networks [21:50] ens38* [21:50] jtcressy: what version of MAAS are you running? [21:51] 2.3 [21:51] MAAS version: 2.3.0 (6434-gd354690-0ubuntu1~16.04.1) [22:03] what it would be interesting to see is what the virtual network in esxi is doing [22:03] also, jtcressy can you connect to the tftp server? [22:03] manually ? [22:04] and get files from it? [22:05] from the internal vmware network ? [22:05] also, are there anu other dhcp servers in the vlan ? [23:20] roaksoax: Sorry, i'm back. Yes, i can contact tftp via the virtual network using my host machine. maas is the only machine giving out dhcp on the lan.