| mup | Bug #1665104 changed: ARM64 Gigabyte server sometimes fails to enlist in CI <MAAS:Expired> <https://launchpad.net/bugs/1665104> | 04:31 |
|---|---|---|
| mup | Bug #1673525 changed: Boot from SSD in AHCI mode fails <MAAS:Expired> <https://launchpad.net/bugs/1673525> | 04:31 |
| mup | Bug #1665104 opened: ARM64 Gigabyte server sometimes fails to enlist in CI <MAAS:Expired> <https://launchpad.net/bugs/1665104> | 04:34 |
| mup | Bug #1673525 opened: Boot from SSD in AHCI mode fails <MAAS:Expired> <https://launchpad.net/bugs/1673525> | 04:34 |
| mup | Bug #1665104 changed: ARM64 Gigabyte server sometimes fails to enlist in CI <MAAS:Expired> <https://launchpad.net/bugs/1665104> | 04:40 |
| mup | Bug #1673525 changed: Boot from SSD in AHCI mode fails <MAAS:Expired> <https://launchpad.net/bugs/1673525> | 04:40 |
| === frankban|afk is now known as frankban | ||
| mup | Bug #1690466 changed: Pod manager counts all virsh pools as available disk <MAAS:Won't Fix> <https://launchpad.net/bugs/1690466> | 13:35 |
| mup | Bug #1692554 opened: doc: https://docs.ubuntu.com/maas/2.1/en/api states "bond-mode" instead of bond_mode <canonical-bootstack> <MAAS:Confirmed for petermatulis> <https://launchpad.net/bugs/1692554> | 15:59 |
| mup | Bug #1692557 opened: [2.2] Menu on mobile view does not work. <MAAS:Triaged> <https://launchpad.net/bugs/1692557> | 16:29 |
| xygnal | roaksoax I have a bit of a problem. In order to force our Dell boxes to set their raid config, it reboots the system | 17:08 |
| xygnal | roaksoax can Comission handle a reboot or will that spoil it? | 17:08 |
| === frankban is now known as frankban|afk | ||
| roaksoax | xygnal: the commission process would start all over again. So what will happen? the dell firmware will upgrade the firmware, require a reboot and that's it ? or will it do more stuff after a reboot ? | 17:26 |
| roaksoax | xygnal: actually, it may even mark the machine failed commissioning because it didn't receive a ping from the machine | 17:26 |
| xygnal | roaksoax its just reseting the raid config back to 0, so we can change it to HBA mode after that. That forces it to reboot the OS to change the RAID settings. | 17:27 |
| roaksoax | ltrager: ^^ in commissioning, whilerunning a custom script, will rebooting the machine mark it as failed commissioning ? | 17:27 |
| ltrager | roaksoax: with 2.2 it should resume on reboot | 17:27 |
| xygnal | ltrager and it would just re-run the steps? | 17:27 |
| roaksoax | xygnal: it would, yes | 17:27 |
| ltrager | xygnal: it would run scripts which havnt run | 17:28 |
| xygnal | ltrager would my script which initiates the rboot count as completed? | 17:28 |
| xygnal | or be re-run? | 17:28 |
| ltrager | no the reboot will cause no result to be sent | 17:29 |
| ltrager | so it should rerun | 17:29 |
| xygnal | alright then | 17:29 |
| xygnal | i've already started coding it to check if the raid has been reset or not | 17:29 |
| xygnal | so it will say "nope not reset", reboots | 17:29 |
| xygnal | comes back around | 17:29 |
| xygnal | checks "ok its gone now, safe to move forward" on seccond boot | 17:30 |
| mup | Bug #1692607 opened: maas CLI/api interaction silently ignores bad parameters <canonical-bootstack> <MAAS:Triaged> <https://launchpad.net/bugs/1692607> | 17:32 |
| mup | Bug #1692607 changed: maas CLI/api interaction silently ignores bad parameters <canonical-bootstack> <MAAS:Triaged> <https://launchpad.net/bugs/1692607> | 17:44 |
| mup | Bug #1692607 opened: maas CLI/api interaction silently ignores bad parameters <canonical-bootstack> <MAAS:Triaged> <https://launchpad.net/bugs/1692607> | 17:53 |
| xygnal | ltrager one quesiton - how long before comission times out? This raid operation takes 10 minutes. | 18:01 |
| ltrager | xygnal: there is a heart beat with 2.2, so that shouldnt be a problem | 18:02 |
| xygnal | ltrager heartbeat to prod interface IP, or to OOB? During this 10 minutes, OS wont be available. | 18:02 |
| ltrager | xygnal: to the rack on the metadata sevice | 18:03 |
| mup | Bug #1692660 opened: sudo maas createadmin creates username with 'login' prepended <MAAS:New> <https://launchpad.net/bugs/1692660> | 18:26 |
| mup | Bug #1692660 changed: sudo maas createadmin creates username with 'login' prepended <MAAS:Invalid> <https://launchpad.net/bugs/1692660> | 18:56 |
| xygnal | roaksoax: ltrager: not working like you think it would | 20:41 |
| xygnal | soon as the box reboots, goes into failed comission state, and powers off the box | 20:42 |
| xygnal | which then prevents it from even completing the raid change | 20:42 |
| xygnal | :/ | 20:42 |
| roaksoax | ltrager: ^^ | 20:43 |
| roaksoax | xygnal: yeah I think I kind of expected that, ltrager why would that not be the case? unless you define a script timeout ? | 20:43 |
| xygnal | if i set a sleep in the script i see it wait patiently for 5 minutes of sleep | 20:44 |
| xygnal | but as soon as the OS reboots from this command | 20:44 |
| xygnal | instantly, commission failed | 20:44 |
| roaksoax | yeah because the heartbeating mechanism is asking for status | 20:45 |
| xygnal | and there is no OS to respond :( | 20:45 |
| roaksoax | exactly | 20:45 |
| xygnal | if it did not power it off | 20:45 |
| xygnal | at least it would finish its activity | 20:45 |
| xygnal | and on second try, it would skip that part | 20:45 |
| xygnal | but since it powers off right away... nope | 20:46 |
| roaksoax | but, ltrager can confirm, there may be possible to specify a timeout for the script in question, so it doesn't immediately think it is dean | 20:46 |
| roaksoax | dead* | 20:46 |
| xygnal | ltrager let me know if I can do that, a 600 second timeout should get me past it. | 20:46 |
| xygnal | dont ask my why a PERC needs 10m to reset its RAID config, it just does :/ | 20:47 |
| mup | Bug #1692723 opened: Adding RSD Pod fails if pre-composed node has remote iSCSI storage target assigned <rsd> <MAAS:Triaged> <https://launchpad.net/bugs/1692723> | 21:35 |
| mup | Bug #1692723 changed: Adding RSD Pod fails if pre-composed node has remote iSCSI storage target assigned <rsd> <MAAS:Triaged> <https://launchpad.net/bugs/1692723> | 21:41 |
| mup | Bug #1692723 opened: Adding RSD Pod fails if pre-composed node has remote iSCSI storage target assigned <rsd> <MAAS:Triaged> <https://launchpad.net/bugs/1692723> | 21:47 |
| ltrager | xygnal: What should be happening is after you reboot the machine goes back into the ephemeral environment and any script which MAAS has not received a result for is run. What I suspect is happening is the reboot is taking to long. | 21:53 |
| ltrager | xygnal: Once commissioning/testing starts MAAS expects the heartbeat to run and allows for 10 minutes of silence. If reboot takes longer then 10 minutes it will fail | 21:53 |
| xygnal | it fails in 30 seconds | 21:53 |
| xygnal | every time | 21:54 |
| ltrager | hmm I wonder if the script runner is detecting the reboot as a failure for some reason | 21:56 |
| ltrager | xygnal: are you just running the command 'reboot'? | 21:56 |
| xygnal | ltrager: no but this is racadm issuing a command to the DRAC to wipe the RAID config. It immediately power-cycles the box and boots itself into its UEFI ui and shows a progress meter for the action. When it reaches 100% it boots back to normal. | 21:57 |
| xygnal | sadly srvadmin-idracadm8 and 7 both fail their post-scripts on install, so if I cannot cleanly uninstall them, any boot after that fails because of those 2 packages that failed to finish their post scripts. | 21:59 |
| xygnal | thinking that this step is simply going to have to happen pre-maas | 21:59 |
| ltrager | xygnal: In the event log does MAAS tell you why it failed? | 22:02 |
| xygnal | checking the maas.log | 22:03 |
| mup | Bug #1692723 changed: Adding RSD Pod fails if pre-composed node has remote iSCSI storage target assigned <rsd> <MAAS:Triaged> <https://launchpad.net/bugs/1692723> | 22:05 |
| xygnal | lol | 22:08 |
| xygnal | region or rack would have this? | 22:08 |
| xygnal | reigon has 3 lines. Status transition from READY to COMMISSIONING, Comissioning Started, Status transition from COMMISSIONING to FAILED_COMMISSIONING | 22:09 |
| ltrager | xygnal: On the machine page click on the events tab | 22:09 |
| xygnal | Node commissioning failure - 'cloudinit' running modules for final | 22:09 |
| xygnal | only that, begin the begining and ending messages for commission | 22:10 |
| xygnal | er, other than the begining and ending messages | 22:10 |
| xygnal | the last script to execute is my raid reset action, and the logs for that confirm it excuted and would have rebooted at that time. | 22:11 |
| xygnal | does cloud-init get run *after* these? | 22:12 |
| ltrager | xygnal: could you post the result of maas <profile> node-script-result read <system_id> current-commissioning | 22:12 |
| ltrager | xygnal: When the ephemeral environment boots cloud-init is given the MAAS script runner as user data which downloads, executes, and returns all commissioning and testing scripts slated to be run | 22:13 |
| xygnal | btw, it does not show my scripts having failed execution. | 22:14 |
| xygnal | 'Passed' status | 22:14 |
| xygnal | They have an immediately 'exit 0' after that racadm command to ensure it comes back as Passed. Since there would be no reason for that script to execute a second time. Its work was complete. | 22:15 |
| ltrager | xygnal: but the system never comes back after the reboot? | 22:15 |
| xygnal | ltrager system is immediately powered-off by MAAS | 22:16 |
| xygnal | after commission failed status | 22:16 |
| xygnal | there is no 'time out' waiting for it to come back | 22:16 |
| xygnal | it gives up right away | 22:16 |
| xygnal | the timestamp for starting comission and ending comission is 17:02 to 17:07 | 22:17 |
| xygnal | when it fails, that is | 22:17 |
| xygnal | ltrager I am doing this as a 99- script, should I be putting it earlier instead? | 22:17 |
| ltrager | xygnal: that shouldn't matter | 22:18 |
| xygnal | ltrager do you want it strait out of maas like that, or would you rather see my script directly on pastebin? | 22:19 |
| ltrager | xygnal: straight from MAAS is fine :) | 22:19 |
| xygnal | ltrager ah I see this is the status result. | 22:20 |
| xygnal | all of the tasks after my raid reset task | 22:21 |
| xygnal | are marked as failed | 22:21 |
| xygnal | but they were never run... | 22:21 |
| xygnal | and the timestamp for the fail on each of those items | 22:21 |
| xygnal | is identical | 22:21 |
| xygnal | like it auto-failed them all | 22:21 |
| xygnal | ltrager: check private message window | 22:26 |
| roaksoax | i think that what's happening is since the machine is rebooting mid-commissioning, a message is sent to cloud-init which then tells maas that it failed finishing doing what it needed to do | 22:30 |
| roaksoax | causing the machine to be marked as failed commissioning, as expected | 22:30 |
| ltrager | roaksoax: I think its failing when the PXE file is requested | 22:35 |
| ltrager | xygnal: The reboot is causing the failure as roaksoax mentioned I'm just trying to figure out why and how it can be fixed | 22:37 |
| xygnal | any other location I can gather data from to assist with debuggin that? | 22:37 |
| roaksoax | ltrager: because cloud-init is running the show in the ephemeral environment | 22:39 |
| roaksoax | ltrager: cloud-init captures the fact that something is breaking its own process and is trying to reboot the machine | 22:39 |
| roaksoax | ltrager: and cloud-init goes back and says "oh wait, I didn't finish running what I was told to run, and something is trying to reboot the machine, so since I didn't finish, I'm failing running everything I was told to" | 22:40 |
| roaksoax | xygnal: i have a question, your change process requires to reboot the machine. I wonder if you can simply not reboot the machine. The machine wil turn off after commissioning, and when you deploy, it will have effectively applied the changes you requested | 22:45 |
| xygnal | roaksoax unfortunately there does not appear to be a 'please dont reboot' for this option. | 22:51 |
| xygnal | and telling it to clear the config does not cause it to simply change a 1 to a 0, it boots up into its UEFI boot manager and proceeds to execute. | 22:52 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!