=== queue is now known as queue-
adacI try to install the grub again to my disks in a chrooted environment, following basically these commands here: https://blog.michael.franzl.name/2014/01/29/remote-server-hetzner-rebooting/ however for one disk I get the error grub-install: error: unable to identify a filesystem in hostdisk//dev/nvme0n1; safety check can't be performed20:54
adacthe problem then here is that my machine wont boot anymore since that fails20:55
adacany ideas what I can do about this?20:55
tomreyn"hostdisk//dev/nvme0n1" is obviously wrong, and i've never seen this. how did you install?20:58
tomreynand which version of ubuntu server is this?20:58
adactomreyn, actually at the moment I'm on the rescue system of the provider and I have chrooted into my machine wichh is ubuntu  18.0421:00
adacThis was a raid and I had to let change one disk21:00
tomreynand this recue system is compatible to ubuntu 18.04 ?21:01
tomreynoh you chrooted so you're using 18.04 tools21:01
adactomreyn, good question. usually when this happened that the grub was missing, which happend more often before, this used to work21:01
adactomreyn, how can I see if it is compatible?21:02
tomreynthere's no a simple check to tell21:02
tomreynbut if you chrooted to the existing ubuntu 18.04 installation with the same boot mode it usually boots in, and with all virtual file systems bind mounted into it, you should be able to recover it if the kernel of the recovery system is not extremely old21:03
tomreyntry this https://askubuntu.com/a/83124121:04
tomreynyou'll need to get your SW RAID ready  as the first step, though21:05
tomreyndisabling secure boot in the systems' bios and upgrading the bios can make things easier, especially if its old.21:06
adactomreyn, actually back when this happened before on other machines after a disk swap, I did not had to get the SW raid ready before. I just went into the rescue system and re-installed grub 21:07
adacnot sure why this time it fails21:07
tomreynjournalctl -b | grep DMI:    tells you about board and bios version. hetzner provides bios upgrade tools on their network share, and you can access the system bios fro their KVM (which is free for 2 hours=21:07
adacjournalctl -b21:08
adac-- Logs begin at Thu 2022-02-03 04:27:42 CET, end at Sun 2022-02-20 20:02:29 CET. --21:08
adac-- No entries --21:08
adactomreyn, ok thanks 21:08
tomreynyou must be working on the rescue system still21:09
tomreyndmesg | grep DMI:   then21:09
tomreynplease use a pastebin to share multi-line outputs21:09
tomreynfirst line of this output is your mainboard + bios version21:11
adactomreyn, ok I see. It might be worth upgrading then21:12
tomreynyou may want to check this with support, too.21:13
adactomreyn, yes I will ask them. It is strange behaviour21:13
tomreynthat's a customized (for hetzner) gigabyte desktop board you have there21:14
tomreynhttps://www.gigabyte.com/Motherboard/B360-HD3P-rev-10/support#support-dl-bios would be the closest, but the release date given for version F4 there does not match up with the build date of yours21:24
adactomreyn, I think I could ask them that they would upgrade it for me21:25
tomreynpossibly. i don't know whether it's actually needed, just suggesting it generally.21:28
tomreynit's old,though, they should probably do it without asking21:28
adactomreyn, yes I see. Yeah they usually just do when there is a problem21:28
adacIm doing a backup in the rescue system of my virtual machine images first now21:29
adacthen I need to see how to proceed21:29
tomreynyou said this problem occurred after / due to a disk swap. i'm assuming the disk that was swapped contained the (single copy of the) ESP, and you had a mirror md raid spun across other partitions on this and the other nvme storage (which you still have one leg of, just without the esp)21:32
tomreynif you re-create an esp there and re-populate it, you could possibly boot from there also without the raid restored, but in failed state.21:33
tomreynthere are ways to keep a backup copy of the active ESP on another storage (such as the one which contains the other leg of a RAID-1), if you want to consider this in the future21:35
adacYes that could be the reason21:37
adacwhat would that mean booting in failed state?21:38
adacusually the raid is off already since before the disk swap one has to remove the disk from the raid21:38
adacwhich I also did this time21:39
adacso back then I was always able to enter the rescue system, install the grub again, then reboot and it would boot21:40
adacafterwards i put the disk back into the raid21:40
tomreynif this is a raid-1 array then it has to have a minimum nuumber of 2 devices. if you only have a total of two then removing one would bring the array into failed state, from what i rmember.21:42
tomreynthe difference between earlier recoveries of this type and now is probably that earlier, the disk that was affected was the one without the ESP on it, but this time it was the one with the ESP on it, so you lost that21:43
tomreyni'm still guessing, though, don't actually know how you boot or manage the ESP21:43
adactomreyn, https://pastebin.com/R7dNyAjA see that is that array you probably mean. That was right after i  removed nvme0n1 from the raid21:44
adactomreyn, ok I see. How can I check if the ESP is actually there?21:45
tomreyn'(F)' means failed device21:45
tomreynthe _ in '[_U]' means you're missing the first leg of this two device array21:46
adacYes exactly the one that I moved out of the raid21:47
tomreynfdisk -l   should list all partitions and their flags, if there's an ESP there it should show21:47
adactomreyn, actually thisis the situation: https://pastebin.com/CS5ECmfM21:51
adacsearched for esp string. Not found21:51
tomreynhmm, do you actually boot in efi mode, though?21:53
tomreynthose are mbr/dos partition tables21:53
adactomreyn, I'm afraid I do not really know21:55
tomreyn /dev/nvme1n1 contains a 512 MB partition, nvme1n1p2, this could be /boot, could also be ESP, but not on an mbr partitioned disk, so you probably boot in bios mode21:55
tomreynis sda the recovery system?21:56
adachttps://pastebin.com/n1GKYWkZ thisis the current situation in the rescue system after the steps in https://blog.michael.franzl.name/2014/01/29/remote-server-hetzner-rebooting/21:57
adacsda is an additional disk actually that was added21:57
adacsince I run out of space for my logs21:58
tomreyni see. i guess in your situation all you may need to do is to re-create the missing partitions on nvme0n1 to match those of nvme1n1 and then to install grub to both /dev/nvme0n1 and /dev/nvme1n1 22:02
tomreynand then, either before or after reboot, to resynch the raid by re-adding the nvme0n1 raid members22:03
adactomreyn, Yes that sounds about right22:03
adactomreyn, thank you so much man22:04
tomreynyou're welcome 22:04
tomreynthis guide you're following is not bind mounting /dev/pts into the chroot. you may need to do that also22:06
tomreynhmm, no, you probably wont need it22:07
adacI just finish the backup, just to be save. I have backups, but not up to date one of the whole virtual machines. then I will try to create these partitions22:10
adaccan I somehow see the exact partitions of rhe working disk and maybe just use exactly thes?22:11
adacsfdisk -d /dev/nvme1n1 | sfdisk /dev/nvme0n122:13
adacCopy over partition table from /dev/nvme1n1 to /dev/nvme0n122:13
tomreyni guess this can work. might fail for the last partition if that's larger on the existing disk than there is space available on the new one22:17
adactomreyn, kk I will try afterwards22:18
adacI#m just wondering. this usually happened after the disk was changed and when i was already booted into the system22:19
adacthen, before putting the disk into the raid, we copied over the partition tables with a script22:19
adacmaybe on the other disks, they were already there 22:20
tomreynthat seems unlikely, unless you had been using those disks before22:20
adacnot sure tbh. I will finish the backup then lets try it out :D22:21
adacaaaaa these disk issues a are a nightmare!22:22
tomreyngood luck on that, i'll make that a fortunate outcome in my dreams.22:26
adactomreyn, hehe thanks!! Have a good night!! :-)22:28

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!