| BlackDex | admin0: there are some ways by adding/changing the curtin installer scripts if i'm correct. But not easy. | 08:32 |
|---|---|---|
| BlackDex | admin0: You can check this article, maybe it will help you: https://insights.ubuntu.com/2017/06/02/customising-maas-installs/ | 08:36 |
| admin0 | BlackDex, thanks | 10:11 |
| admin0 | checking | 10:11 |
| admin0 | BlackDex, exactly what i needed | 10:45 |
| admin0 | many thanks | 10:45 |
| BlackDex | admin0: yw :) | 12:31 |
| mup | Bug #1749210 opened: node powered off after reboot from rescue mode <amd64> <apport-bug> <uec-images> <xenial> <MAAS:New> <maas (Ubuntu):New> <https://launchpad.net/bugs/1749210> | 15:12 |
| mup | Bug #1749210 opened: node powered off after reboot from rescue mode <amd64> <apport-bug> <uec-images> <xenial> <MAAS:New> <maas (Ubuntu):New> <https://launchpad.net/bugs/1749210> | 15:14 |
| mup | Bug #1741013 changed: [Wishlist] Ability that can add custom cloud-init configuration <cpe-onsite> <MAAS:Invalid> <https://launchpad.net/bugs/1741013> | 16:09 |
| mup | Bug #1749246 opened: nodes boot into 4.13 (hwe kernel) with no minimum kernel set <amd64> <apport-bug> <uec-images> <xenial> <maas (Ubuntu):New> <https://launchpad.net/bugs/1749246> | 17:09 |
| xygnal | roaksoax: ping | 17:12 |
| mup | Bug #1749246 opened: nodes boot into 4.13 (hwe kernel) with no minimum kernel set <amd64> <apport-bug> <uec-images> <xenial> <MAAS:Incomplete> <maas (Ubuntu):Incomplete> <https://launchpad.net/bugs/1749246> | 17:15 |
| mup | Bug #1749246 changed: nodes boot into 4.13 (hwe kernel) with no minimum kernel set <amd64> <apport-bug> <uec-images> <xenial> <MAAS:Incomplete> <maas (Ubuntu):Incomplete> <https://launchpad.net/bugs/1749246> | 17:21 |
| mup | Bug #1749246 opened: nodes boot into 4.13 (hwe kernel) with no minimum kernel set <amd64> <apport-bug> <uec-images> <xenial> <MAAS:Incomplete> <maas (Ubuntu):Incomplete> <https://launchpad.net/bugs/1749246> | 17:39 |
| xygnal | mpontillo: ping | 18:48 |
| roaksoax | xygnal: pong | 18:52 |
| xygnal | roaksoax: json outputs in that bug report for you. needs review and next stepzn | 18:58 |
| xygnal | 1744765 | 18:58 |
| roaksoax | xygnal: I saw, thank you. Were you able to try a test environment in hardware? | 18:59 |
| xygnal | no. my team is not convinced you have enough evidence to prove it as a vmware issue so they have not been in a hurry to try it. | 19:01 |
| xygnal | do you have a technical theory WHYn | 19:02 |
| xygnal | ? | 19:02 |
| xygnal | we dont see any indication of perf issues on the box itself. | 19:03 |
| xygnal | just the app, which by itself, consumes all memory regularly. | 19:03 |
| xygnal | even with 8 threads now, no change. | 19:04 |
| xygnal | cli commands are still quick | 19:05 |
| xygnal | only UI impacted | 19:05 |
| roaksoax | xygnal: are you running xenial? or are you running a newer version ? | 19:11 |
| xygnal | 16.04.3 | 19:16 |
| xygnal | yes xenial | 19:16 |
| jtcressy | Does anyone know why a MAAS controller would fail to set a static IP on an interface? Once I set the interface to an IP, the physical adapter is never changed (verified with ip addr show) and upon reboot, the controller shows the interface as unconfigured. What's wrong? | 19:19 |
| mpontillo | xygnal: sorry for slow responses; I'm swamped with other work | 19:21 |
| mpontillo | xygnal: when you say "by itself", are you saying that even if there are zero users of the web UI, there are still issues? | 19:21 |
| mpontillo | jtcressy: at what point does it change? how are you setting the static IP? | 19:22 |
| jtcressy | Nodes>Controllers>(my only controller)>Interfaces>(second ethernet adapter)> edit physical | 19:23 |
| jtcressy | Am I wrong in assuming that the MAAS server would change the IP configuration of interfaces on the server it's running on? | 19:24 |
| roaksoax | jtcressy: the maas controller doesn't change interface configuration | 19:24 |
| roaksoax | jtcressy: is this a physical interface ? have you restarted maas-rackd again ? | 19:25 |
| jtcressy | It is a physical interface | 19:25 |
| jtcressy | So, then what I need to do is configure the interface via /etc/network/interfaces and then maas will be able to use it? | 19:25 |
| jtcressy | (fyi the way I have this setup is one interface is management network that provides dhcp and the other interface is the rack that the controller should provide DHCP to) | 19:26 |
| xygnal | i double checked this and even maas cli takes ages to complete a machines read. | 19:27 |
| xygnal | er no.. | 19:28 |
| xygnal | not ages.... 120 seconds | 19:28 |
| roaksoax | xygnal: yeah that's a non issue with the cli | 19:29 |
| roaksoax | known*& | 19:30 |
| mpontillo | jtcressy: yes, for controllers MAAS uses the interfaces as-configured. | 19:32 |
| xygnal | only the ui is slow. monsterously slow. even takes several minutes to see a node page. like, just a single node's page. | 19:33 |
| jtcressy | yaaay stuff works after doing all the static assignment stuff in etc/network/interfaces | 19:41 |
| jtcressy | How can I reconfigure dhcp? machines timeout on trying to get tftp | 19:51 |
| jtcressy | PXE E32: TFTP open timeout | 19:59 |
| jtcressy | commissioning node just sits there spinning and doing nothing | 19:59 |
| jtcressy | have wireshark open and I see the TFTP requests on the wire. maas server just doesn't reply atall | 20:00 |
| mup | Bug #1749281 opened: MAAS 2.3 does not revert system proxy back to original state when MAAS is removed <MAAS:New> <https://launchpad.net/bugs/1749281> | 20:01 |
| jtcressy | Wait, do I have to manually install an OS on machines before I can commission it? | 20:06 |
| mpontillo | jtcressy: once you enable DHCP in MAAS, machines that boot from your DHCP-enabled VLAN should enlist automatically. are you saying MAAS is responding to DHCP but not TFTP requests? are you using any kind of network filtering, DHCP forwarding, or third-party DHCP server, or are your machines just booting directly onto a DHCP-enabled VLAN? | 20:14 |
| jtcressy | the machines are directly getting DHCP from the maas machine | 20:15 |
| jtcressy | absolutely zero network filtering. I have a gateway at the '.1' address of the network which is declared accordingly on the maas server | 20:16 |
| jtcressy | maas server is at '.2' with a static address | 20:16 |
| jtcressy | and that '.2' address is declared as the dhcp address for the maas controller | 20:16 |
| mpontillo | jtcressy: can you pastebin the output of /usr/lib/maas/maas-test-enlistment? | 20:16 |
| jtcressy | mpontillo: https://pastebin.com/wJUjUYNL | 20:18 |
| mpontillo | jtcressy: er, I meant for you to execute the script so I could see its output; sorry if that wasn't clear | 20:19 |
| jtcressy | oh ok | 20:19 |
| jtcressy | mpontillo: https://pastebin.com/1C70p445 | 20:20 |
| jtcressy | mpontillo: FYI, I just checked 'netstat -tnap | grep LISTEN' and port 69 does NOT show up in the output!!!! | 20:22 |
| mpontillo | jtcressy: well, looks like the TFTP server is responding and serving up bootloaders - at least when the test script runs. you can see that TFTP is indeed responding, so not sure why your command doesn't see it. | 20:23 |
| mpontillo | jtcressy: I would try `sudo dpkg-reconfigure -plow maas-rack-controller`and make sure that the URL specified is a URL that the booting machines will be able to reach | 20:23 |
| mpontillo | jtcressy: by default it uses `localhost`, which tries to determine the URL for enlisting nodes to use automatically - this doesn't always work | 20:24 |
| jtcressy | ok, i set the api address accordingly | 20:26 |
| jtcressy | rebooting a node still results in tftp timeout and a bunch of node -> maas controller tftp read requests on the wire | 20:27 |
| jtcressy | What should I do when TFTP is apparently *non-functional* ? | 20:27 |
| mpontillo | jtcressy: is there anything else about your setup that might be unusual? your pastebin proved that TFTP works to some extent; the `maas-test-enlistment` script basically uses `curl`to mimic what the bootloader does, and you can see the MAAS TFTP server responding | 20:29 |
| jtcressy | but this is weird.... netstat shows that port 69 is NOT listening, so how did the script verify that tftp was working? | 20:30 |
| mpontillo | jtcressy: well, that's why I was wondering if there was something unusual in your setup. ;-) | 20:31 |
| jtcressy | I installed this maas server from the "install maas region controller" option at the boot prompt of the installer ISO | 20:31 |
| jtcressy | and running this all in vmware shouldnt affect it as long as i have the networking in order, which I've verified using wireshark | 20:32 |
| jtcressy | and the fact that they can communicate with maas via dhcp | 20:33 |
| mpontillo | jtcressy: netstat doesn't show udp ports in LISTEN mode; if you grep your netstat for "udp" you will most likely see the open ports | 20:33 |
| mpontillo | jtcressy: for example, I see `udp 0 0 127.0.0.1:69 0.0.0.0:* 10842/python3` on my test MAAS | 20:34 |
| jtcressy | just 'netstat -na | grep udp' ? | 20:34 |
| mpontillo | jtcressy: I used `netstat -anp | grep udp` | 20:34 |
| mpontillo | jtcressy: did you enable `ufw` when you installed the server? if so, try turning it off? | 20:35 |
| jtcressy | ok, 69 shows up in netstat with udp now | 20:36 |
| jtcressy | didnt touch ufw | 20:37 |
| mpontillo | jtcressy: ok. well, you could copy the maas-test-enlistment script to another machine to test if it works from somewhere else across the network. if you pass it a hostname or IP address as a parameter, it will query a remote MAAS | 20:37 |
| mpontillo | jtcressy: so far you've proved that TFTP is up and running on the MAAS server, but you haven't proven you can talk to it over the network | 20:38 |
| jtcressy | working on it. rebooting it for good measure, then going to try tftp from my host system | 20:39 |
| jtcressy | tftp works from my host system | 20:41 |
| jtcressy | *and* this is to the IP that the commissionable nodes are supposed to be talking to | 20:43 |
| jtcressy | I just don't understand why '.191' is completely ignored from tftp requests, when '.1' is answered for tftp requests. | 20:47 |
| jtcressy | Ok, i'm noticing one thing. I filtered my wireshark for just tftp packets to compare it directly. When running tftp from my host machine, it initially talks to the maas server on port 64 | 20:50 |
| jtcressy | but the node keeps trying port 69 | 20:50 |
| jtcressy | the node also trys port 74 | 20:50 |
| jtcressy | oh, forget what I said. I was reading the wrong field | 20:50 |
| jtcressy | both of them are reading port 69 | 20:51 |
| jtcressy | mpontillo: I see another discrepancy: My host is requesting type "netascii" while the node is requesting type "octet" | 20:57 |
| jtcressy | so, netascii works and octet does not | 20:57 |
| mpontillo | jtcressy: netascii vs. octet shouldn't matter; MAAS will always return the same response regardless. | 21:00 |
| mpontillo | jtcressy: when you say .191 is ignored and .1 is answered, what do you mean? | 21:00 |
| jtcressy | <network address>.1 i.e "192.168.1.1" | 21:00 |
| jtcressy | supposedly, there is a hosts.allow file somewhere on the ubuntu server but i don't know where it is | 21:03 |
| jtcressy | mpontillo: ran tcpdump on the maas controller for port 69 and watched the node request pxelinux.0 and the maas controller just never responds https://pastebin.com/rQ08vDvq | 21:15 |
| mpontillo | jtcressy: weird, and yet when you use the script (or something like `curl -s`?) it replies? | 21:19 |
| jtcressy | yep | 21:20 |
| mpontillo | jtcressy: is the MAC seen by the PXE address known to MAAS? you might check /var/log/maas/*.log for any clues | 21:20 |
| jtcressy | I just ran tftp 172.16.207.2, then get pxelinux.0 from my macbook and it also works | 21:20 |
| jtcressy | i'll check the maas log. the mac address is known to maas | 21:20 |
| mpontillo | jtcressy: is 172.16.207.2 configured on ens38? | 21:21 |
| jtcressy | yes | 21:22 |
| jtcressy | 172.16.207.1 is my macbook | 21:22 |
| jtcressy | and 172.16.207.119 is the node, and the node got this IP from the maas server via dhcp | 21:22 |
| mpontillo | jtcressy: what is the output of `ip r g 172.16.207.191` on the MAAS server? | 21:23 |
| jtcressy | 172.16.207.191 dev ens38 src 172.16.207.2 cache | 21:23 |
| jtcressy | neither /var/log/maas/maas.log or ''/rackd.log say anything when I reboot my node and do pxe again | 21:24 |
| mpontillo | jtcressy: anything unusual in regiond.log that corresponds? | 21:26 |
| jtcressy | Nope just a bunch of HTTP 200 logs | 21:26 |
| mpontillo | jtcressy: if you run the tcpdump while one of the working nodes on that network requests the file, how does the output differ? I know tftp switches ports for the data transfer, so it's not clear to me if the 'port 69' filter would catch everything | 21:43 |
| jtcressy | No, tcpdump does not catch whether or not tftp actually works, I just used it to verify that the maas server was receiving it from the network | 21:44 |
| jtcressy | what DOES verify when tftp work, is when i see about 200 packets flow across the network that I can see via wireshark | 21:45 |
| jtcressy | What I need to know is if the tftp server that maas runs ever discriminates. This is the only thing I can think of, and I cant find ANY tftp logs at all | 21:46 |
| jtcressy | mpontillo: Found this, reading through it... https://bugs.launchpad.net/maas/+bug/1465000 | 21:47 |
| mpontillo | jtcressy: it doesn't discriminate with regard to the boot loader, but it does return a custom pxelinux.cfg depending on which node booted | 21:47 |
| jtcressy | but it fails to even send pxelinux.0 so that can't be it. | 21:49 |
| mpontillo | jtcressy: that was why I asked about interface configuration and routing; setups with the same subnet configured on more than one interface could cause this type of issue | 21:49 |
| jtcressy | But that's exactly what I don't have. ens33 has 172.16.97.145/24 and ens28 has 172.16.207.2/24. They are completely different networks | 21:49 |
| jtcressy | ens38* | 21:50 |
| mpontillo | jtcressy: what version of MAAS are you running? | 21:50 |
| jtcressy | 2.3 | 21:51 |
| jtcressy | MAAS version: 2.3.0 (6434-gd354690-0ubuntu1~16.04.1) | 21:51 |
| roaksoax | what it would be interesting to see is what the virtual network in esxi is doing | 22:03 |
| roaksoax | also, jtcressy can you connect to the tftp server? | 22:03 |
| roaksoax | manually ? | 22:03 |
| roaksoax | and get files from it? | 22:04 |
| roaksoax | from the internal vmware network ? | 22:05 |
| roaksoax | also, are there anu other dhcp servers in the vlan ? | 22:05 |
| jtcressy | roaksoax: Sorry, i'm back. Yes, i can contact tftp via the virtual network using my host machine. maas is the only machine giving out dhcp on the lan. | 23:20 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!