| Japje | roaksoax: thanks, will do | 06:22 |
|---|---|---|
| Japje | regarding debian as an custom image, im having trouble with deployment. The curtin_userdata_custom flag for preserve_sources_list seems to be getting ignored | 06:38 |
| Japje | and its overwriting the debian repo's with ubuntu repos | 06:38 |
| Japje | which ofcourse results in a failed deployment | 06:38 |
| tosaraja | I have MAAS 2.6.0 and a Ubuntu 18.04.2 server as a virtual machine getting an IP address from MAAS. But the DNS records in MAAS don't get the data from the Ubuntu VM. Sometimes restarting MAAS helps, but not always. How does the workflow go as in updating the DNS records? Is it the VM that should report it right after receiving the IP from the DHCP server? | 07:09 |
| mup | Bug #1839430 opened: Add an enum for node power types <MAAS:New> <https://launchpad.net/bugs/1839430> | 07:16 |
| mup | Bug #1839430 changed: Add an enum for node power types <MAAS:New> <https://launchpad.net/bugs/1839430> | 07:22 |
| mup | Bug #1839430 opened: Add an enum for node power types <MAAS:New> <https://launchpad.net/bugs/1839430> | 07:25 |
| ivve | blake_r: hey man, did you get any chance to look at those configuration files i sent you? | 08:19 |
| tosaraja | seems my problem has happend before as well: https://bugs.launchpad.net/maas/+bug/1761326 | 08:29 |
| tosaraja | 2.6 isn't having any of those "maasserver.rpc.leases" outputs in regiond.log | 08:30 |
| tosaraja | journal does however see that maas' dchp server gives out the ip addresses and renews leases. it's just that the dns server knows squat about what's going on | 08:42 |
| ivve | anyone here have any clue if its possible to downgrade from 2.6.0 to 2.4.2? | 10:24 |
| tosaraja | ivve: you make that sound like i should stop my work on trying to upgrade from 2.4.2 to 2.6.0 :P | 11:17 |
| ivve | tosaraja: well lots of problems with pxe | 11:30 |
| ivve | i have test and prod env. it worked in test but not in prod | 11:30 |
| ivve | so really confusing | 11:30 |
| tosaraja | oh i haven't gotten that far :D i can't get the DNS server to understand that the dhcp server just gave my opennebula server an andress so that it would update the dns record. it remains empty | 11:32 |
| tosaraja | pxe is only coming up after i get that part working | 11:32 |
| ivve | dns and dhcp seems to work no issues | 11:36 |
| ivve | and well to be honest pxe works but gives incorrect values | 11:36 |
| ivve | i.e. doesn't pass FNAME | 11:37 |
| tosaraja | well it's not the first time i solve issues by simply restarting maas. I think i'll try that _again_ | 11:39 |
| tosaraja | i didn't get any of "maasserver.rpc.leases" in regiond.log before restarting maas. now i do...perhaps it began working | 11:42 |
| tosaraja | it works! so 3 restart in 2 days fixed my problem. now on to the pxe myself :D | 11:42 |
| ivve | anyone knows what this could be? https://pastebin.com/raw/rxQJhZG3 | 12:21 |
| ivve | tcpdump shows 3x udp packets go back and forth to the LOM | 12:21 |
| ivve | i can reach it, so no issues with communications on any ports | 12:23 |
| blake_r | ivve: sorry I did not | 12:53 |
| blake_r | ivve: let me take a look now | 12:53 |
| ivve | blake_r: you have the config in priv or here | 12:53 |
| ivve | blake_r: i also noticed that it seems like there is an error in the BMC count (i think). since i have good communication tcp/ip wise but BMC doesn't talk with rackd as per https://pastebin.com/raw/rxQJhZG3 | 12:54 |
| blake_r | ivve: looking at the DHCP config I only see the bootloaders defined in 1 of the vlans and not all of them | 12:55 |
| blake_r | ivve: do you know which subnet your machine is PXE booting from? | 12:55 |
| ivve | dhcp and pxe is in 1 vlan (108) and helper to the other vlans | 12:56 |
| ivve | so that should be correct? | 12:57 |
| blake_r | ivve: whats the subnet? | 12:57 |
| ivve | 10.22.5.0/24 | 12:57 |
| blake_r | yeah that one seems okay | 12:57 |
| blake_r | that has the bootloaders defined | 12:57 |
| ivve | and i have helpers to 10.22.5.20 & 10.22.5.22 (we setup a new maas) | 12:58 |
| ivve | in HA | 12:58 |
| blake_r | yeah I see the HA | 12:58 |
| ivve | since we had issues with bnx2 drivers | 12:58 |
| blake_r | what does the BIOS give when PXE booting? | 12:58 |
| ivve | these are now virtual | 12:58 |
| ivve | just that FNAME is empty | 12:58 |
| ivve | PXE-E53 error | 12:58 |
| ivve | which it is, when dhcpdumping on the maas node(s) | 12:59 |
| ivve | i can see: FNAME: . | 12:59 |
| ivve | instead of expected: FNAME: lpxelinux.0. | 13:00 |
| ivve | we tested booting up both physical and virtual machines with just live images to test dhclient and got IP's from maas with no issue | 13:01 |
| ivve | we have multiple vendors with different nic, i.e. hp or dell and nics like broadcom, intel. which by the way worked 2.4.x but not now (exactly the same machines, just redeployed) | 13:02 |
| ivve | blake_r: i guess i can offer you a sosreport, if you think you need one? | 13:03 |
| blake_r | ivve: can you turn off DHCP on all the other VLAN's | 13:08 |
| blake_r | ivve: and just have it on for that 1 valn | 13:08 |
| blake_r | ivve: i believe isc-dhcp is giving you an IP address from another subnet that doesn't have the bootloaders defined | 13:09 |
| blake_r | ivve: you have 5 other vlans with DHCP enabled, turn those off and try that machine again | 13:09 |
| ivve | blake_r: okay going for that | 13:14 |
| ivve | blake_r: turned off all dhcp to start with, enabling on the native vlan first from the maas | 13:17 |
| ivve | systemctl status maas-dhcpd.service confirms everything is off | 13:18 |
| ivve | blake_r: same error.. :( | 13:23 |
| ivve | meaning regiond spits out https://pastebin.com/raw/rxQJhZG3 | 13:25 |
| blake_r | ivve: dhcpd should be running for that one vlan | 13:25 |
| blake_r | ivve: can you provide new paste of dhcpd.conf | 13:26 |
| ivve | yes | 13:26 |
| ivve | coming up | 13:26 |
| blake_r | ivve: maas doesn't use DHCP enable status to check if a rack controller can query a BMC | 13:26 |
| blake_r | ivve: did you disable DHCP on the vlans? or delete the vlans? | 13:27 |
| ivve | disable dhcp | 13:28 |
| ivve | did not delete the vlans | 13:28 |
| ivve | you have the new config in PM | 13:28 |
| ivve | blake_r: the thing is, i have an identical setup in a lab which works. i can provide that config also | 13:38 |
| ivve | it drives me crazy that when we upgraded test, it worked. then went with prod, and now it just stopped working | 13:38 |
| ivve | the setup is basically: only L3. so a maas vlan where dhcp happens and relays out to select vlans. the select vlans are the ones in the config, around 5-6 of them. all those vlans have helpers pointing to maas on its own vlan | 13:40 |
| ivve | the lab has the exact same setup | 13:40 |
| ivve | hmm when i removed everything and added it again i think the error disappeared when i pressed commission on a new node. waiting to see if pxe working | 14:01 |
| ivve | got different error now | 14:03 |
| ivve | PXE-E51: no dhcp or proxydhcp offers were received. | 14:03 |
| ivve | seems like it doesn't offer the file on L3 | 14:06 |
| ivve | but it works on L2 | 14:06 |
| ivve | blake_r: is it possible to populate the database with existing deployed machines? im thinking of cleaning this out and repopulating it (not dumping the DB as i think the problem lies within it) | 14:11 |
| Japje | for custom images (debian 10 in my case) is there a way to re-create the 90_dpkg* files so i can access the maas datasource? | 14:41 |
| Japje | perhaps somehow from within curtin | 14:41 |
| blake_r | ivve: are you sure there is not a duplicate DHCP server on the network? | 14:53 |
| ivve | blake_r: very sure | 14:53 |
| ivve | i can see the improper lease going out from rackd servers with a missing FNAME flags | 14:54 |
| ivve | (with tcpdump) | 14:54 |
| ivve | blake_r: im thinking something is wrong in the DB.. when looking at newly added nodes in the maasserver_node table bios_boot_method is empty, in comparison to the lab db it is populated with "pxe" | 14:56 |
| ivve | i updated the row but no change | 14:57 |
| blake_r | ivve: that is because the machine has never pxe booted from maas yet | 15:06 |
| blake_r | ivve: you are still having the issue of getting the wrong fname | 15:06 |
| blake_r | ivve: maas is not between the machine and the dhcp server | 15:07 |
| blake_r | ivve: maas just configures isc-dhcp | 15:07 |
| ivve | oh okay | 15:07 |
| ivve | yes | 15:07 |
| blake_r | ivve: something is wrong with the dhcp configuration | 15:07 |
| ivve | BMC problems disappeared after reseting DHCP (removing and adding it) | 15:07 |
| blake_r | ivve: but looking at the latest paste, it looks correct | 15:07 |
| ivve | but it still missing FNAME | 15:07 |
| mup | Bug #1839491 opened: Manully performed partitioning changes get reverted on reboot <MAAS:New> <https://launchpad.net/bugs/1839491> | 15:08 |
| ivve | im a bit afraid of removing the vlan and subnet since i have lots of machines there | 15:08 |
| blake_r | ivve: no you shouldn't need to do that | 15:08 |
| blake_r | ivve: can you send me the output of dhcpdump | 15:08 |
| ivve | but i disabled DHCP in the entire environment | 15:09 |
| ivve | blake_r: coming up | 15:09 |
| ivve | blake_r: so isc-dhcp-service is failed | 15:13 |
| ivve | blake_r: i guess that is a problem? | 15:13 |
| blake_r | ivve: yeah whats the reason for the failure? | 15:14 |
| blake_r | ivve: wait no | 15:15 |
| blake_r | ivve: you care about the maas-dhcpd service | 15:15 |
| blake_r | ivve: not tha isc-dhcp-service | 15:15 |
| blake_r | ivve: maas only controls the maas-dhcpd service | 15:15 |
| ivve | that one is working and had it working entire time | 15:16 |
| ivve | isc-dhcp-server.service loaded failed failed ISC DHCP IPv4 server | 15:18 |
| ivve | ● isc-dhcp-server6.service loaded failed failed ISC DHCP IPv6 server | 15:18 |
| ivve | disabled those and rebooted server, lets see | 15:19 |
| ivve | in the lab those are disabled.. hmm | 15:19 |
| ivve | blake_r: here is the bootpreply https://pastebin.com/raw/hp6VBZDY | 15:27 |
| ivve | i guess the request isn't very interesting | 15:30 |
| ivve | but it contains 67 (bootfile name) | 15:30 |
| blake_r | ivve: i dont see option 67 in that pase | 15:39 |
| blake_r | paste* | 15:39 |
| ivve | blake_r: in the request? | 15:40 |
| blake_r | ivve: that paste* seems to be the reply | 15:40 |
| ivve | you want the request? | 15:40 |
| ivve | https://pastebin.com/raw/NYx50kfA | 15:42 |
| blake_r | ivve: the response should be getting you: filename "lpxelinux.0"; | 15:44 |
| ivve | blake_r: i know :) | 15:44 |
| blake_r | ivve: can you check /var/log/syslog dhcpd logs in there | 15:44 |
| ivve | blake_r: during a commission? nothing in there at all | 15:46 |
| ivve | i've been checking /var/log/maas/*.log as well | 15:47 |
| ivve | no errors pop or anything of particular interest | 15:47 |
| ivve | so currently tailing /var/log/maas/*.log and syslog during a commission | 15:51 |
| blake_r | ivve: is it showing the DHCP messages? | 15:52 |
| ivve | nopes | 15:53 |
| ivve | all empty | 15:53 |
| ivve | i will check the lab environment if its the same | 15:53 |
| ivve | slow HP machine booting :P | 15:57 |
| ivve | okay i can confirm that no output to logfiles happen before the actual pxelinux file is downloaded | 15:59 |
| ivve | so nothing on a working system outputs any dhcp data | 16:00 |
| ivve | first thing that happens is: | 16:00 |
| ivve | 2019-08-08 15:59:02 provisioningserver.rackdservices.tftp: [info] lpxelinux.0 requested by 10.23.5.30 | 16:00 |
| ivve | and that is in /var/log/maas/rackd.log | 16:00 |
| blake_r | hmm okay | 16:08 |
| blake_r | anything in journalctl -u maas-dhcpd | 16:08 |
| blake_r | sorry if I am all over the place, but not really seen this issue before | 16:08 |
| blake_r | expecially when the dhcpd.conf is correct | 16:08 |
| ivve | blake_r: no worries man, we are kinda baffled here as well. ready to abandon maas due this issue | 16:09 |
| blake_r | just to be sure nothing crazy is going on can you do a "ps auxf | grep dhcpd" | 16:09 |
| blake_r | make sure no extra dhcpd are running | 16:09 |
| blake_r | well its not really MAAS that is causing the issue as much as its isc-dhcp as MAAS just configures that | 16:09 |
| ivve | just the maas process | 16:10 |
| ivve | and since i only using 1 in the HA setup the "offline" dhcp has no processes | 16:10 |
| ivve | checking journalctl now | 16:10 |
| ivve | journalctl is empty on the "offline" node | 16:11 |
| ivve | but lots of data in the active, checking now | 16:11 |
| blake_r | try turning off HA for dhcp | 16:13 |
| blake_r | lets see if that fixes it | 16:13 |
| ivve | yes it is off | 16:13 |
| blake_r | I just tested mine and got | 16:13 |
| blake_r | FNAME: lpxelinux.0. | 16:13 |
| ivve | alas the "offline" node | 16:13 |
| ivve | no dhcp process going on there | 16:13 |
| ivve | and maas is suggesting to enable HA | 16:13 |
| ivve | ok found some of the request/offers | 16:14 |
| ivve | https://pastebin.com/raw/nc8GSDbb | 16:15 |
| blake_r | ivve: okay so its making an offer | 16:17 |
| blake_r | ivve: can you check the dhcpdump of that over? you still have that running? | 16:17 |
| ivve | yes | 16:18 |
| ivve | its in a previous paste | 16:18 |
| ivve | lemme check | 16:18 |
| blake_r | ah thought that was a new offer | 16:18 |
| ivve | here is the offer https://pastebin.com/raw/hp6VBZDY | 16:18 |
| ivve | they are all the same, timestamps don't match | 16:18 |
| ivve | but they are 100% identical | 16:18 |
| blake_r | this is a physical machine booting or a vm? | 16:19 |
| ivve | physical machine comission | 16:20 |
| ivve | no errors at all | 16:20 |
| ivve | just simply offer with no fname | 16:21 |
| ivve | even tho option 67 is requested from client | 16:21 |
| ivve | well the option is there, but its empty | 16:21 |
| ivve | just . | 16:21 |
| blake_r | is this a layer2 connection between the rack controller and the physical machine? or is it using a DHCP relay? | 16:27 |
| ivve | relay | 16:28 |
| blake_r | did you configure that VLAN in maas to be relayed? | 16:28 |
| ivve | yes | 16:29 |
| ivve | i can show a picture of the configurations in lab and prod | 16:29 |
| ivve | they are identical | 16:29 |
| ivve | one works, other not | 16:29 |
| blake_r | you sure the relays are configured the same? | 16:29 |
| ivve | and i can also paste the helper configuration in the swtich, which is also identical | 16:29 |
| blake_r | well if its identical | 16:29 |
| ivve | you can see for yourself | 16:29 |
| ivve | :) | 16:29 |
| blake_r | I believe you | 16:29 |
| ivve | we are all baffled here on how it doesn't work | 16:30 |
| blake_r | :-) | 16:30 |
| blake_r | if you could provide the dhcpd.conf dhcpd-interfaces and helper configuration between both envs and I can compare | 16:30 |
| blake_r | just to see if anything pops out at me | 16:30 |
| ivve | sure | 16:30 |
| ivve | https://pasteboard.co/IrLxpI8.png | 16:33 |
| ivve | https://pasteboard.co/IrLxPlb.png | 16:34 |
| blake_r | ivve: so I am confused some because that dhcpd.conf you provided only had one vlan in there | 16:36 |
| blake_r | ivve: but for dhcp relay you must have both | 16:36 |
| ivve | there is also relay to LOM network | 16:36 |
| ivve | i didn't include that | 16:36 |
| ivve | but BMC works, i can turn on/off and check power status | 16:36 |
| blake_r | yeah so I think that is the issue then | 16:37 |
| blake_r | as I didn't see it in the config | 16:37 |
| blake_r | https://bugs.launchpad.net/maas/+bug/1836276 | 16:37 |
| blake_r | is the issue you are hitting | 16:37 |
| ivve | hmm not sure what you mean | 16:37 |
| ivve | both what? | 16:37 |
| blake_r | there is also relay to LOM network | 16:37 |
| blake_r | i didn't include that | 16:37 |
| ivve | yea getting the shot one sec | 16:38 |
| blake_r | I think that bug above is your issue | 16:38 |
| blake_r | is the MAAS version between env the same? | 16:38 |
| blake_r | or is 1 MAAS 2.5 and the other is MAAS 2.6? | 16:38 |
| ivve | https://pasteboard.co/IrLztLf.png | 16:38 |
| ivve | both are 2.6.0 | 16:38 |
| ivve | i can run a dist-upgrade on both machines and they have the stable ppa | 16:39 |
| ivve | and no updates are found | 16:39 |
| blake_r | strange | 16:39 |
| ivve | yes | 16:39 |
| blake_r | then that would not be the issue | 16:39 |
| ivve | :) | 16:39 |
| blake_r | as that bug says its a 2.6 only issue | 16:39 |
| blake_r | but your not having that issue in your staging | 16:39 |
| ivve | 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. | 16:39 |
| ivve | just tried on all 4 machines | 16:40 |
| blake_r | can you provide me the full dhcpd.conf on production and then the full dhcpd.conf on staging? | 16:40 |
| ivve | yes | 16:40 |
| ivve | production https://pastebin.com/raw/MYRTQvU8 | 16:43 |
| ivve | uhm sorry that was lab | 16:43 |
| ivve | so here is the prod for real this time https://pastebin.com/raw/eaKGuhnD | 16:45 |
| ivve | observe the lab has now HA enabled | 16:46 |
| ivve | and last config for prod was to disable HA | 16:46 |
| blake_r | i think the issue is that the subnets being relayed do not have the bootloaders defined | 16:47 |
| blake_r | but in the staging environment is showing the same behaviour | 16:47 |
| blake_r | in the config, but is working | 16:47 |
| blake_r | which seems very wierd that it works in one case but not the other | 16:47 |
| ivve | yes and also it was working before in 2.4.2 | 16:47 |
| ivve | in prod | 16:47 |
| ivve | we have a pretty large env installed as you can see | 16:48 |
| ivve | and its like 1 year old | 16:48 |
| blake_r | yeah as the bug states is a 2.6 only issue as we changed to using http instead of tftp for most of the boot process | 16:48 |
| ivve | aye | 16:49 |
| blake_r | ivve try this | 16:49 |
| ivve | switches are identical | 16:49 |
| ivve | no network has changed at least in prod | 16:49 |
| ivve | even firmwares are the same | 16:49 |
| blake_r | modify /usr/lib/python3/dist-packages/provisioningserver/templates/dhcp/dhcpd.conf.template | 16:49 |
| blake_r | add to the top | 16:49 |
| ivve | i could check if the templates are identical | 16:50 |
| blake_r | https://paste.ubuntu.com/p/hqB6nMc3Ft/ | 16:50 |
| ivve | i was looking for those before | 16:50 |
| ivve | but couldn't find | 16:50 |
| blake_r | yeah I know | 16:50 |
| ivve | i guess i could have asked you | 16:50 |
| ivve | :) | 16:50 |
| blake_r | try adding that to the top | 16:51 |
| blake_r | then restart rackd | 16:51 |
| ivve | checksummed the templates | 16:51 |
| blake_r | then try to DHCP and see if the FNAME is actually set | 16:51 |
| ivve | they are the same | 16:51 |
| ivve | do i need to remove anything else? | 16:52 |
| ivve | or just add it on the very top | 16:52 |
| ivve | i will reboot the entire machine, just for safe :P | 16:53 |
| ivve | they are virtual now so its quick | 16:54 |
| ivve | ok testing a commission | 16:55 |
| ivve | nopes | 16:58 |
| ivve | no-go | 16:58 |
| ivve | doesn't offer at all | 16:58 |
| ivve | maybe i need to check some stuff | 16:59 |
| ivve | perhaps disable all dhcp and enable it again | 17:00 |
| ivve | disabled all and enabled those 3, 1 dhcp and 2 relays | 17:02 |
| ivve | still no offers | 17:02 |
| ivve | well maybe the conf was wrong or smth | 17:05 |
| ivve | ok the template was wrong | 17:07 |
| ivve | blake_r: can't find the problem but its in the } else statement at the bottom that you wanted me to add | 17:10 |
| blake_r | ivve: paste it all at line 15 of the file and down | 17:12 |
| blake_r | ivve: you need the option arch defined at the top of the file to come first | 17:12 |
| blake_r | ivve: that is probably the issue | 17:13 |
| ivve | https://pastebin.com/raw/1k3cE7Qv | 17:16 |
| ivve | blake_r: thats just the top of the generated config | 17:16 |
| blake_r | yeah you need to move it down below the PXEClient if statement | 17:19 |
| ivve | check | 17:20 |
| ivve | blake_r: some assistance, anything i can search for to find it? | 17:24 |
| ivve | or just a paste with the full file you want me to test | 17:26 |
| blake_r | if you remove what you added | 17:27 |
| blake_r | it will be line 15 | 17:27 |
| ivve | after BOOTLOADERS? | 17:27 |
| ivve | between bootloaders and subnet dhcp snippets? | 17:28 |
| ivve | {{dhcp_subnet['bootloader']}} \n {{endif}} | 17:28 |
| ivve | replace those two lines with your paste? | 17:28 |
| blake_r | no | 17:28 |
| ivve | after class "PXE" statement or replace it? | 17:29 |
| blake_r | after it | 17:29 |
| blake_r | above | 17:29 |
| ivve | check | 17:29 |
| blake_r | Define lease time globally (can be overriden globally or per subnet | 17:29 |
| ivve | https://pastebin.com/raw/FkYaXFjX | 17:31 |
| ivve | like that? | 17:31 |
| ivve | didn't seem to work either | 17:32 |
| ivve | or could you just supply a full template or should i use like 2.4.2 template? | 17:33 |
| ivve | if thats the test | 17:33 |
| ivve | reboot solved it | 17:33 |
| ivve | testing commission | 17:35 |
| ivve | it works now | 17:37 |
| ivve | ipxe was the problem | 17:37 |
| ivve | now the question remains | 17:40 |
| ivve | why does it work in the lab | 17:40 |
| ivve | and not in prod | 17:40 |
| ivve | ?! | 17:40 |
| blake_r | did you add that snippet in production? | 17:42 |
| blake_r | and it still didn't work? | 17:42 |
| ivve | your changed fixed it | 17:42 |
| ivve | not using ipxe | 17:42 |
| blake_r | you where getting FNAME: . before | 17:43 |
| blake_r | that was not related to ipxe | 17:43 |
| blake_r | i think the issue is that relayes are not getting any bootloader selections | 17:43 |
| blake_r | that change will allow the relay subnets to fallback to the global defined bootloaders | 17:44 |
| ivve | you sure? | 17:45 |
| blake_r | yeah it should | 17:50 |
| ivve | blake_r: well huge thanks for the help, how do we ensure this is fixed for future releases? | 18:07 |
| ivve | should i bugreport? | 18:07 |
| blake_r | I have that bug report | 18:09 |
| blake_r | its that same bug | 18:09 |
| blake_r | i will work on it and get it on 2.6.1 | 18:09 |
| blake_r | so on upgrade it will stay working for you all | 18:09 |
| blake_r | sorry for the issues it caused you | 18:09 |
| ivve | no worries, big thanks for the assistance! | 18:09 |
| blake_r | np | 18:10 |
| blake_r | glad I could help | 18:10 |
| ivve | guys around here wanted to abandon all hope and use something else :) | 18:10 |
| ivve | whatever that would be.... :P | 18:10 |
| ivve | there is nothing else that is as good as maas imo | 18:10 |
| sbeattie | ahhh, blackhat, for all your rigorous talks: https://twitter.com/veorq/status/1159559785068429312 | 23:40 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
