[08:32] <BlackDex> admin0: there are some ways by adding/changing the curtin installer scripts if i'm correct. But not easy.
[08:36] <BlackDex> admin0: You can check this article, maybe it will help you: https://insights.ubuntu.com/2017/06/02/customising-maas-installs/
[10:11] <admin0> BlackDex, thanks
[10:11] <admin0> checking
[10:45] <admin0> BlackDex, exactly what i needed
[10:45] <admin0> many thanks
[12:31] <BlackDex> admin0: yw :)
[15:12] <mup> Bug #1749210 opened: node powered off after reboot from rescue mode <amd64> <apport-bug> <uec-images> <xenial> <MAAS:New> <maas (Ubuntu):New> <https://launchpad.net/bugs/1749210>
[15:14] <mup> Bug #1749210 opened: node powered off after reboot from rescue mode <amd64> <apport-bug> <uec-images> <xenial> <MAAS:New> <maas (Ubuntu):New> <https://launchpad.net/bugs/1749210>
[16:09] <mup> Bug #1741013 changed: [Wishlist] Ability that can add custom cloud-init configuration <cpe-onsite> <MAAS:Invalid> <https://launchpad.net/bugs/1741013>
[17:09] <mup> Bug #1749246 opened: nodes boot into 4.13 (hwe kernel) with no minimum kernel set <amd64> <apport-bug> <uec-images> <xenial> <maas (Ubuntu):New> <https://launchpad.net/bugs/1749246>
[17:12] <xygnal> roaksoax: ping
[17:15] <mup> Bug #1749246 opened: nodes boot into 4.13 (hwe kernel) with no minimum kernel set <amd64> <apport-bug> <uec-images> <xenial> <MAAS:Incomplete> <maas (Ubuntu):Incomplete> <https://launchpad.net/bugs/1749246>
[17:21] <mup> Bug #1749246 changed: nodes boot into 4.13 (hwe kernel) with no minimum kernel set <amd64> <apport-bug> <uec-images> <xenial> <MAAS:Incomplete> <maas (Ubuntu):Incomplete> <https://launchpad.net/bugs/1749246>
[17:39] <mup> Bug #1749246 opened: nodes boot into 4.13 (hwe kernel) with no minimum kernel set <amd64> <apport-bug> <uec-images> <xenial> <MAAS:Incomplete> <maas (Ubuntu):Incomplete> <https://launchpad.net/bugs/1749246>
[18:48] <xygnal> mpontillo: ping
[18:52] <roaksoax> xygnal: pong
[18:58] <xygnal> roaksoax:  json outputs in that bug report for you. needs review and next stepzn
[18:58] <xygnal> 1744765
[18:59] <roaksoax> xygnal: I saw, thank you. Were you able to try a test environment in hardware?
[19:01] <xygnal> no. my team is not convinced you have enough evidence to prove it as a vmware issue so they have not been in a hurry to try it.
[19:02] <xygnal> do you have a technical theory WHYn
[19:02] <xygnal> ?
[19:03] <xygnal> we dont see any indication of perf issues on the box itself.
[19:03] <xygnal> just the app, which by itself, consumes all memory regularly.
[19:04] <xygnal> even with 8 threads now,  no change.
[19:05] <xygnal> cli commands are still quick
[19:05] <xygnal> only UI impacted
[19:11] <roaksoax> xygnal: are you running xenial? or are you running a newer version ?
[19:16] <xygnal> 16.04.3
[19:16] <xygnal> yes xenial
[19:19] <jtcressy> Does anyone know why a MAAS controller would fail to set a static IP on an interface? Once I set the interface to an IP, the physical adapter is never changed (verified with ip addr show) and upon reboot, the controller shows the interface as unconfigured. What's wrong?
[19:21] <mpontillo> xygnal: sorry for slow responses; I'm swamped with other work
[19:21] <mpontillo> xygnal: when you say "by itself", are you saying that even if there are zero users of the web UI, there are still issues?
[19:22] <mpontillo> jtcressy: at what point does it change? how are you setting the static IP?
[19:23] <jtcressy> Nodes>Controllers>(my only controller)>Interfaces>(second ethernet adapter)> edit physical
[19:24] <jtcressy> Am I wrong in assuming that the MAAS server would change the IP configuration of interfaces on the server it's running on?
[19:24] <roaksoax> jtcressy: the maas controller doesn't change interface configuration
[19:25] <roaksoax> jtcressy: is this a physical interface ? have you restarted maas-rackd again ?
[19:25] <jtcressy> It is a physical interface
[19:25] <jtcressy> So, then what I need to do is configure the interface via /etc/network/interfaces and then maas will be able to use it?
[19:26] <jtcressy> (fyi the way I have this setup is one interface is management network that provides dhcp and the other interface is the rack that the controller should provide DHCP to)
[19:27] <xygnal> i double  checked this and even maas cli takes ages to complete a machines read.
[19:28] <xygnal> er no..
[19:28] <xygnal> not ages.... 120 seconds
[19:29] <roaksoax> xygnal: yeah that's a non issue with the cli
[19:30] <roaksoax> known*&
[19:32] <mpontillo> jtcressy: yes, for controllers MAAS uses the interfaces as-configured.
[19:33] <xygnal> only the ui is slow.  monsterously slow. even takes several minutes to see a node page. like, just a single node's page.
[19:41] <jtcressy> yaaay stuff works after doing all the static assignment stuff in etc/network/interfaces
[19:51] <jtcressy> How can I reconfigure dhcp? machines timeout on trying to get tftp
[19:59] <jtcressy> PXE E32: TFTP open timeout
[19:59] <jtcressy> commissioning node just sits there spinning and doing nothing
[20:00] <jtcressy> have wireshark open and I see the TFTP requests on the wire. maas server just doesn't reply atall
[20:01] <mup> Bug #1749281 opened: MAAS 2.3 does not revert system proxy back to original state when MAAS is removed <MAAS:New> <https://launchpad.net/bugs/1749281>
[20:06] <jtcressy> Wait, do I have to manually install an OS on machines before I can commission it?
[20:14] <mpontillo> jtcressy: once you enable DHCP in MAAS, machines that boot from your DHCP-enabled VLAN should enlist automatically. are you saying MAAS is responding to DHCP but not TFTP requests? are you using any kind of network filtering, DHCP forwarding, or third-party DHCP server, or are your machines just booting directly onto a DHCP-enabled VLAN?
[20:15] <jtcressy> the machines are directly getting DHCP from the maas machine
[20:16] <jtcressy> absolutely zero network filtering. I have a gateway at the '.1' address of the network which is declared accordingly on the maas server
[20:16] <jtcressy> maas server is at '.2' with a static address
[20:16] <jtcressy> and that '.2' address is declared as the dhcp address for the maas controller
[20:16] <mpontillo> jtcressy: can you pastebin the output of /usr/lib/maas/maas-test-enlistment?
[20:18] <jtcressy> mpontillo: https://pastebin.com/wJUjUYNL
[20:19] <mpontillo> jtcressy: er, I meant for you to execute the script so I could see its output; sorry if that wasn't clear
[20:19] <jtcressy> oh ok
[20:20] <jtcressy> mpontillo: https://pastebin.com/1C70p445
[20:22] <jtcressy> mpontillo: FYI, I just checked 'netstat -tnap | grep LISTEN' and port 69 does NOT show up in the output!!!!
[20:23] <mpontillo> jtcressy: well, looks like the TFTP server is responding and serving up bootloaders - at least when the test script runs. you can see that TFTP is indeed responding, so not sure why your command doesn't see it.
[20:23] <mpontillo> jtcressy: I would try `sudo dpkg-reconfigure -plow maas-rack-controller`and make sure that the URL specified is a URL that the booting machines will be able to reach
[20:24] <mpontillo> jtcressy: by default it uses `localhost`, which tries to determine the URL for enlisting nodes to use automatically - this doesn't always work
[20:26] <jtcressy> ok, i set the api address accordingly
[20:27] <jtcressy> rebooting a node still results in tftp timeout and a bunch of node -> maas controller tftp read requests on the wire
[20:27] <jtcressy> What should I do when TFTP is apparently *non-functional* ?
[20:29] <mpontillo> jtcressy: is there anything else about your setup that might be unusual? your pastebin proved that TFTP works to some extent; the `maas-test-enlistment` script basically uses `curl`to mimic what the bootloader does, and you can see the MAAS TFTP server responding
[20:30] <jtcressy> but this is weird.... netstat shows that port 69 is NOT listening, so how did the script verify that tftp was working?
[20:31] <mpontillo> jtcressy: well, that's why I was wondering if there was something unusual in your setup. ;-)
[20:31] <jtcressy> I installed this maas server from the "install maas region controller" option at the boot prompt of the installer ISO
[20:32] <jtcressy> and running this all in vmware shouldnt affect it as long as i have the networking in order, which I've verified using wireshark
[20:33] <jtcressy> and the fact that they can communicate with maas via dhcp
[20:33] <mpontillo> jtcressy: netstat doesn't show udp ports in LISTEN mode; if you grep your netstat for "udp" you will most likely see the open ports
[20:34] <mpontillo> jtcressy: for example, I see `udp        0      0 127.0.0.1:69            0.0.0.0:*                           10842/python3` on my test MAAS
[20:34] <jtcressy> just 'netstat -na | grep udp' ?
[20:34] <mpontillo> jtcressy: I used `netstat -anp | grep udp`
[20:35] <mpontillo> jtcressy: did you enable `ufw` when you installed the server? if so, try turning it off?
[20:36] <jtcressy> ok, 69 shows up in netstat with udp now
[20:37] <jtcressy> didnt touch ufw
[20:37] <mpontillo> jtcressy: ok. well, you could copy the maas-test-enlistment script to another machine to test if it works from somewhere else across the network. if you pass it a hostname or IP address as a parameter, it will query a remote MAAS
[20:38] <mpontillo> jtcressy: so far you've proved that TFTP is up and running on the MAAS server, but you haven't proven you can talk to it over the network
[20:39] <jtcressy> working on it. rebooting it for good measure, then going to try tftp from my host system
[20:41] <jtcressy> tftp works from my host system
[20:43] <jtcressy> *and* this is to the IP that the commissionable nodes are supposed to be talking to
[20:47] <jtcressy> I just don't understand why '.191' is completely ignored from tftp requests, when '.1' is answered for tftp requests.
[20:50] <jtcressy> Ok, i'm noticing one thing. I filtered my wireshark for just tftp packets to compare it directly. When running tftp from my host machine, it initially talks to the maas server on port 64
[20:50] <jtcressy> but the node keeps trying port 69
[20:50] <jtcressy> the node also trys port 74
[20:50] <jtcressy> oh, forget what I said. I was reading the wrong field
[20:51] <jtcressy> both of them are reading port 69
[20:57] <jtcressy> mpontillo: I see another discrepancy: My host is requesting type "netascii" while the node is requesting type "octet"
[20:57] <jtcressy> so, netascii works and octet does not
[21:00] <mpontillo> jtcressy: netascii vs. octet shouldn't matter; MAAS will always return the same response regardless.
[21:00] <mpontillo> jtcressy: when you say .191 is ignored and .1 is answered, what do you mean?
[21:00] <jtcressy> <network address>.1 i.e "192.168.1.1"
[21:03] <jtcressy> supposedly, there is a hosts.allow file somewhere on the ubuntu server but i don't know where it is
[21:15] <jtcressy> mpontillo: ran tcpdump on the maas controller for port 69 and watched the node request pxelinux.0 and the maas controller just never responds https://pastebin.com/rQ08vDvq
[21:19] <mpontillo> jtcressy: weird, and yet when you use the script (or something like `curl -s`?) it replies?
[21:20] <jtcressy> yep
[21:20] <mpontillo> jtcressy: is the MAC seen by the PXE address known to MAAS? you might check /var/log/maas/*.log for any clues
[21:20] <jtcressy> I just ran tftp 172.16.207.2, then get pxelinux.0 from my macbook and it also works
[21:20] <jtcressy> i'll check the maas log. the mac address is known to maas
[21:21] <mpontillo> jtcressy: is 172.16.207.2 configured on ens38?
[21:22] <jtcressy> yes
[21:22] <jtcressy> 172.16.207.1 is my macbook
[21:22] <jtcressy> and 172.16.207.119 is the node, and the node got this IP from the maas server via dhcp
[21:23] <mpontillo> jtcressy: what is the output of `ip r g 172.16.207.191` on the MAAS server?
[21:23] <jtcressy> 172.16.207.191 dev ens38  src 172.16.207.2      cache
[21:24] <jtcressy> neither /var/log/maas/maas.log or ''/rackd.log say anything when I reboot my node and do pxe again
[21:26] <mpontillo> jtcressy: anything unusual in regiond.log that corresponds?
[21:26] <jtcressy> Nope just a bunch of HTTP 200 logs
[21:43] <mpontillo> jtcressy: if you run the tcpdump while one of the working nodes on that network requests the file, how does the output differ? I know tftp switches ports for the data transfer, so it's not clear to me if the 'port 69' filter would catch everything
[21:44] <jtcressy> No, tcpdump does not catch whether or not tftp actually works, I just used it to verify that the maas server was receiving it from the network
[21:45] <jtcressy> what DOES verify when tftp work, is when i see about 200 packets flow across the network that I can see via wireshark
[21:46] <jtcressy> What I need to know is if the tftp server that maas runs ever discriminates. This is the only thing I can think of, and I cant find ANY tftp logs at all
[21:47] <jtcressy> mpontillo: Found this, reading through it... https://bugs.launchpad.net/maas/+bug/1465000
[21:47] <mpontillo> jtcressy: it doesn't discriminate with regard to the boot loader, but it does return a custom pxelinux.cfg depending on which node booted
[21:49] <jtcressy> but it fails to even send pxelinux.0 so that can't be it.
[21:49] <mpontillo> jtcressy: that was why I asked about interface configuration and routing; setups with the same subnet configured on more than one interface could cause this type of issue
[21:49] <jtcressy> But that's exactly what I don't have. ens33 has 172.16.97.145/24 and ens28 has 172.16.207.2/24. They are completely different networks
[21:50] <jtcressy> ens38*
[21:50] <mpontillo> jtcressy: what version of MAAS are you running?
[21:51] <jtcressy> 2.3
[21:51] <jtcressy> MAAS version: 2.3.0 (6434-gd354690-0ubuntu1~16.04.1)
[22:03] <roaksoax> what it would be interesting to see is what the virtual network in esxi is doing
[22:03] <roaksoax> also, jtcressy can you connect to the tftp server?
[22:03] <roaksoax> manually ?
[22:04] <roaksoax> and get files from it?
[22:05] <roaksoax> from the internal vmware network ?
[22:05] <roaksoax> also, are there anu other dhcp servers in the vlan ?
[23:20] <jtcressy> roaksoax: Sorry, i'm back. Yes, i can contact tftp via the virtual network using my host machine. maas is the only machine giving out dhcp on the lan.