| KingJ | I'm not sure if this is an issue with MAAS or netplan, but i'm having a lot of difficult getting a 9000 byte MTU on a 802.1ad bond. In the generated netplan config, I can see that the MTU is set to 9000 for the bond member interfaces, but when the bond comes up everything switches to 1500. | 14:11 |
|---|---|---|
| KingJ | Indeed, checking dmesg I can see a lot of "changing MTU from 9000 to 1500" messages. | 14:12 |
| roaksoax | KingJ: is that for a deployed system ? | 14:15 |
| KingJ | roaksoax: Yeah | 14:15 |
| roaksoax | KingJ: what's the resulting netplan ? | 14:15 |
| KingJ | One sec | 14:15 |
| KingJ | roaksoax: Netplan config: https://paste.ubuntu.com/p/mxp6Dkxfqg/ | 14:19 |
| xygnal | roaksoax: why is pgs10 needed for 2.4? | 14:20 |
| roaksoax | KingJ: seems you have eth0 with mtu 1500 and the rest with mtu 9000, which is your expected config, right ? | 14:21 |
| roaksoax | KingJ: but those w/ mtu9000 are really being changed to mtu 1500 ? | 14:21 |
| KingJ | roaksoax: Yep, eth0 is supposed to be at 1500 (can't support higher, and is just a management interface). enp* are meant to be at 9000 and bonded. | 14:22 |
| KingJ | Let me grab the dmesg and ip link sh output | 14:22 |
| roaksoax | KingJ: ok, so probably due to the bond not declaring MTU ? | 14:23 |
| KingJ | dmesg: https://paste.ubuntu.com/p/PQrJcKbXtv/ | 14:24 |
| KingJ | ip link sh: https://paste.ubuntu.com/p/GRP3SGWrSt/ | 14:24 |
| KingJ | roaksoax: Possibly? There was no option on the UI to set the MTU on the bond. | 14:24 |
| roaksoax | xygnal: i dont /think/ we have a strict requirement on pg10, but rather, pg10 is the default on bionic | 14:24 |
| xygnal | yes figured. we run our db separate so verifying we can keep it. | 14:25 |
| roaksoax | KingJ: ok, so IIRC, if the parent interfaces of a bond have the higher mtu, then the bond should inherit that | 14:27 |
| KingJ | I did also try assigning a VLAN with a 9000 MTU to the bond interface itself, although that doesn't appear to have made it in to the netplan config (possibly because I only assigned the VLAN and left subnets/addressing unassigned?) | 14:27 |
| roaksoax | KingJ: and it seems in this case is not | 14:27 |
| roaksoax | KingJ: try this: maas <user> machine get-curtin-config <system-id> | 14:28 |
| roaksoax | KingJ: and deploy a machine in xenial and see whether the bond ends up with the mtu defined ? | 14:28 |
| KingJ | roaksoax: Here's the curtin config (just the networking section) https://paste.ubuntu.com/p/5Tdb6xwh69/ | 14:30 |
| KingJ | I'll reclaim the machine now and re-provision on xenial | 14:31 |
| roaksoax | KingJ: https://pastebin.ubuntu.com/p/jdC68hPRJR/ | 14:31 |
| roaksoax | KingJ: seems to be that the mtu is defined | 14:31 |
| roaksoax | KingJ: so that looks like a cloud-init issue to me | 14:32 |
| KingJ | Yeah, odd. I'll admit I also have tried running interface update the6f3 153 mtu=9000 however that was done after deployment - so I wouldn't expected that to have an effect? | 14:33 |
| roaksoax | KingJ: no, so lets wait to see what e/n/i results in xenial | 14:33 |
| roaksoax | KingJ: to compare | 14:33 |
| KingJ | roaksoax: Gotcha. I'll report back once deployed on Xenial. Should be within the next half hour. | 14:34 |
| KingJ | (Thanks as always for your help here!) | 14:35 |
| roaksoax | mp! | 14:39 |
| xygnal | roaksoax: what cloudinit version is officially needed for static ip support? | 14:48 |
| xygnal | roaksoax: and look at this... https://bugs.launchpad.net/cloud-init/+bug/1712680 | 14:57 |
| roaksoax | xygnal: static ip addressing is in 17.x something | 15:00 |
| roaksoax | xygnal: the bug is fixed in cloud-init 18.2+ i think, which the centos iamges are not yet using | 15:00 |
| xygnal | they are using an older version with a patch | 15:01 |
| xygnal | and the patch appears to contain it | 15:01 |
| xygnal | as it works with your centos7 image | 15:01 |
| xygnal | however, seems that bug report implies its broken in other ways | 15:02 |
| xygnal | ty | 15:04 |
| KingJ | roaksoax: Deployed on 16.04, still seeing a 1500 MTU. dmesg still has messages about the MTU being changed from 9000 to 1500. Checking /etc/network/interfaces.d/50-cloud-init.cfg I can see that all interfaces, including the bond, have a 9000 MTU configured (paste incoming) | 15:17 |
| KingJ | https://paste.ubuntu.com/p/mr2JCwTzfX/ | 15:18 |
| KingJ | Ah wait | 15:19 |
| KingJ | My bad... it's actually all at 9000 and dmesg was about changing from 1500 to 9000 | 15:19 |
| roaksoax | xygnal: it shouldn't really, if you use maas provided config | 15:21 |
| roaksoax | KingJ: ah so both bionic/xenial doing the same thing ? | 15:21 |
| KingJ | roaksoax: No sorry. Bionic has dmesg messages about MTU changing from 9000 to 1500, xenial has messages about 1500 to 9000. Xenial's ip link sh output confirms 9000 MTU, pings with large packets work without fragmenting. So Xenial works, Bionic doesn't. | 15:22 |
| xygnal | roaks: i dont change its config. i just deploy in maas with autoassign and post deploy iatrou_nspection shows a local static config file. | 15:23 |
| roaksoax | KingJ: please file a bug and attach all relevant info (maas preseed config, the cloud-init netplan config on bionic, the e/n/i on xenial, and the kernel messages) | 15:23 |
| KingJ | e/n/i ? | 15:24 |
| roaksoax | KingJ: /etc/network/interfaces.d/50-** | 15:24 |
| KingJ | Ahhhh, yes. | 15:24 |
| KingJ | I'll get that filed now, thanks. | 15:24 |
| roaksoax | xygnal: sorry, i dont follow ? | 15:25 |
| xygnal | there was a typo in there sorry. If I deploy the stock centos7 image from MAAS in 2.4, it writes a static IP config locally on the CentOS box | 15:25 |
| xygnal | using auto-assign | 15:26 |
| xygnal | you seemed to think it should not, but i'm telling you, it does | 15:26 |
| xygnal | just FYI | 15:26 |
| roaksoax | xygnal: ah no, i think there's some confusion there then, because it should deploy correctly a static config file | 15:27 |
| roaksoax | xygnal: my bug was about modifying the config manually post deployment, cloud-init would re-write it on reboot | 15:27 |
| xygnal | roaksoax: ah you were addressing the bug specifically. gotcha. | 15:27 |
| xygnal | i only pointed that out as i noticed that is the version that you deploy | 15:28 |
| roaksoax | gotcha | 15:28 |
| xygnal | (or CentOS deploys, as it were) | 15:28 |
| mup | Bug #1774665 opened: MAAS creates GPT header and partition on one disk when creating a 2 disk raid-1 set <cdo-qa> <foundations-engine> <MAAS:New> <https://launchpad.net/bugs/1774665> | 15:29 |
| mup | Bug #1774666 opened: Bond interfaces stuck at 1500 MTU on Bionic <mtu> <netplan> <MAAS:New> <https://launchpad.net/bugs/1774666> | 15:32 |
| roaksoax | KingJ: one more thing, can you provide the version of cloud-init ? dpkg -l | grep cloud-init | 15:37 |
| chquark | Hi guys, | 15:38 |
| chquark | iḿ trying to follow this guide https://docs.openstack.org/charm-deployment-guide/latest/install-maas.html | 15:38 |
| KingJ | roaksoax: Done :) Let me know if there's anything else I need to add | 15:38 |
| chquark | using ubuntu 18 LTS, but have this error https://imgur.com/a/1lSwt4z | 15:39 |
| roaksoax | KingJ: i think that's all, i opened bugs against cloud-init as i think that's wehre the error is | 15:39 |
| roaksoax | chquark: go to the 'Logs' tab, what does it show ? | 15:39 |
| roaksoax | chquark: for the installation log | 15:39 |
| KingJ | roaksoax: Yeah, that does look to be the case. Thanks for your help here. | 15:40 |
| roaksoax | np! | 15:40 |
| chquark | Installation has failed and no output was given. | 15:40 |
| chquark | ^^ that is all it says | 15:40 |
| chquark | https://pastebin.com/yXiVDWid | 15:40 |
| chquark | https://pastebin.com/cHZM0Evz ´/var/log/maas/rsyslog/juju1/2018-06-01´ | 15:41 |
| roaksoax | chquark: in the UI you have a 'logs' tab | 15:41 |
| roaksoax | chquark: with a red | 15:42 |
| roaksoax | icon | 15:42 |
| roaksoax | it will show you the logs | 15:42 |
| roaksoax | chquark: for installation | 15:42 |
| roaksoax | chquark: does it show ? | 15:42 |
| roaksoax | chquark: looking at the rsyslog, this is the error: https://pastebin.ubuntu.com/p/xDMxpkYSV5/ | 15:42 |
| chquark | so it basically needs access to the internet | 15:43 |
| roaksoax | chquark: well yes and no, 10.0.2.15:8000 -> who owns that IP ? | 15:44 |
| roaksoax | chquark: is that MAAS itself ? | 15:44 |
| chquark | its my NAT ip.. going to internet. | 15:44 |
| chquark | MAAS is 192.168.100.1 | 15:44 |
| roaksoax | chquark: so the deployment needs to access the ubuntu archive, if you have a local archive you can use that | 15:44 |
| roaksoax | chquark: so the deployment needs to access the ubuntu archive, if you have a local archive you can use that instead | 15:44 |
| roaksoax | chquark: but that error is telling me that maas is trying to contact the internal proxy on that IP address | 15:45 |
| roaksoax | chquark: that would mean maas itself is running on a machine that has that IP ? | 15:45 |
| chquark | yes MAAS server has that IP, it is NATed to get internet access | 15:45 |
| roaksoax | chquark: and are machines on 192.168.100.0/24 ? | 15:46 |
| chquark | MAAS has 3 interfaces , 1) 192.168.100.1 , 2) 10.0.2.15, 3) 192.168.56.204 | 15:47 |
| roaksoax | chquark: ok what's in /etc/maas/rackd.conf ? and whats the network that machines are PXE booting from (i'm guessing its 192.168.100.1? ) | 15:47 |
| chquark | Juju 1) 192.168.100.0/24 -- DHCP from MAAS | 15:47 |
| roaksoax | chquark: ok, so check rackd.conf | 15:47 |
| chquark | [sudo] password for user: | 15:48 |
| chquark | cluster_uuid: ec08ffa0-1d19-4591-bd3d-a31381de5ba7 | 15:48 |
| chquark | maas_url: http://localhost:5240/MAAS | 15:48 |
| chquark | user@user:/var/log/maas/rsyslog/juju1/2018-06-01$ | 15:48 |
| roaksoax | chquark: do this, use 192.168.100.1 instead of 'localhost' | 15:48 |
| roaksoax | chquark: in rackd.conf | 15:48 |
| roaksoax | sudo service maas-rackd restart | 15:48 |
| roaksoax | chquark: and try to deploy again | 15:48 |
| chquark | ok | 15:48 |
| chquark | roaksoax: I think it is working :) | 15:58 |
| roaksoax | chquark: awesome! | 15:59 |
| mup | Bug #1774665 changed: MAAS creates GPT header and partition on one disk when creating a 2 disk raid-1 set <cdo-qa> <foundations-engine> <MAAS:New> <https://launchpad.net/bugs/1774665> | 16:05 |
| mup | Bug #1774665 opened: MAAS creates GPT header and partition on one disk when creating a 2 disk raid-1 set <cdo-qa> <foundations-engine> <MAAS:New> <https://launchpad.net/bugs/1774665> | 16:14 |
| === frankban is now known as frankban|afk | ||
| roaksoax | KingJ: fwiw, what is being asked is for you to set the MTU on the bond interface itself and see if that correctly sets the MTU | 17:52 |
| KingJ | roaksoax: Gotcha. Just tried adding it to the bond section and re-running netplan apply but to no avail - also tried a systemd-networkd restart. Just going for a full reboot now before reporting back on the bug ticket. | 18:45 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!