TJ-mgedmin: see "man dpkg" and the explanation of --verify-format00:00
TJ-daftykins: then there is something bad happening - check the logs "journalctl -b -1 -p warning" to check the previous boot's log00:00
daftykinshttps://termbin.com/7g3bk - here's the output00:02
daftykinsprox-xen.mount is the guest utilities process, failing to mount Xen ProcFS since the xenfs module isn't loaded00:03
TJ-daftykins: I suspect those "systemd-cryptsetup" reports are a big clue00:07
TJ-daftykins: "systemd-cryptsetup@xvda5_crypt.service: Failed with result 'exit-code'." is the LUKS container for the root file-system00:08
daftykinsi might rule out the storage by moving the VM back to the mechanical RAID and see if purging and reinstalling the modules package sticks00:08
TJ-We've been doing that kind of install today across 8 systems and not seen any errors like that, so I think there's something about your specific scenario with Xen00:09
daftykinsboot is painfully slow under Xen :(00:11
daftykinswell, i say that but i don't have any frame of reference as i've not done an encrypted install on any other setup :D00:11
daftykinscertainly unencrypted VMs are slower to boot than ones on my vmware ESXi system00:11
mgedminit took a thinkpad 31 seconds to boot to gdm (firmware + loader + kernel + userspace), and that time includes me typing the passphrase00:15
mgedminluks on an nvme ssd00:15
daftykinsi don't understand what caused these lines to be thrown, recently00:17
daftykinsThe following packages were automatically installed and are no longer required:00:17
daftykins  amd64-microcode intel-microcode iucode-tool libdbus-glib-1-2 r-cran-ellipse thermald00:17
daftykinswow that's the first time i didn't have a yellow TTY100:18
daftykinsstill gets stuck on a purple background with flashing cursor top left until i manually switch to a TTY00:18
TJ-that sounds like the vt.handoff= set on the kernel command line00:19
TJ-also sounds like it's setting GRUB in gfx mode instead of text00:19
daftykinsshould "/etc/default/grub" read GRUB_CMDLINE_LINUX_DEFAULT="quiet text" instead of quiet splash?00:20
daftykinsanyway that's separate to my main issue right now, still tinkering00:20
daftykinsok dpkg --verify came back ok00:21
daftykinsnow i'm rebooting00:21
daftykinsyep files disappeared again00:25
daftykinsOct 26 00:21:14 talkietoaster systemd[1]: Failed unmounting /lib/modules.00:27
TJ-GRUB_CMDLINE... on a server, I'd leave off quiet and no, it doesn't need 'text', instead you set "GRUB_TERMINAL=console" when using grub-pc (BIOS mode boots) to do text only00:28
TJ-unmounting /lib/modules!?!?!00:29
daftykinspass :D well it seems like the machine goes down for a reboot without honouring taking care of cryptsetup then00:29
TJ-that error suggests /lib/modules/ is in fact a mount point00:30
TJ-you need to check that out00:30
daftykinscopymods on /lib/modules type tmpfs (rw,relatime)00:31
TJ-There you go!00:32
TJ-where the heck did that come from?00:32
daftykinsnot a clue :O00:32
TJ-aha cloud-initramfs-copymods00:32
TJ-!info cloud-initramfs-copymods00:32
TJ-no ubottu!00:33
daftykinsii  cloud-initramfs-copymods             0.45ubuntu1                          all          copy initramfs modules into root filesystem for later use00:33
TJ- When booting with an external-to-root kernel and initramfs, you need to00:33
TJ- ensure that /lib/modules contains any necessary modules not already00:33
TJ- loaded.00:33
TJ- .00:33
TJ- This package arranges for the modules in the initramfs to be placed into00:33
TJ- /lib/modules after the switchroot is done.00:33
TJ-are you booting with kernel/initrd.img external to the VM?00:34
daftykinsnot external no, it's all in the VDI00:34
TJ-So my original feeling that there's a tmpfs was correct00:34
daftykinsbut given i messed up /boot - what else could i have messed up o000:34
TJ-OK, so remove/purge that package after figuring out how it got installed00:34
daftykinswell i'm not going to have a clue on that one :D00:35
daftykinshang on00:35
daftykinsis it part of ubuntu-server ... it wants to remove them together00:35
TJ-aha, it's a depends of ubuntu-server00:35
TJ-this makes no sense!00:36
daftykinsi used the ubuntu-server metapackage on your suggestion after the debootstrap, so presumably it's ok to lose?00:36
* daftykins goes ahead00:37
TJ-check the kernel command-line because , on reading the scripts in that package, they look for "copymods=" on the kernel command line at boot time00:40
TJ-in there I see: log_warning_msg "copying over existing modules! due to copymods=force"00:42
daftykinsis /proc/cmdline sufficient? i see: BOOT_IMAGE=/vmlinuz-5.4.0-52-generic root=/dev/mapper/ubuntu--vg-root ro quiet splash vt.handoff=700:42
TJ-that's fine , and we do not see copymods... let me test your cmdline with that code to see if there's a bug in it00:43
TJ-might be triggering on something other than the intended00:43
daftykinsi rebooted and amusingly now have the 5.4.0-51-generic back in place in /lib/modules/ - because that must now be the real path on /00:44
daftykinsnow reinstalling linux-modules-5.4.0-52-generic00:44
daftykinsseems to stall on "Begin: Running /scripts/init-premount ..." for ages00:50
daftykinsoh it was done, again i had to switch away00:50
daftykinsand now xenfs loads and things work as expected \o/00:51
TJ-I can't work out why that script was triggered and setting up the tmpfs00:53
daftykinsany thoughts on how to resolve those systemd-cryptsetup errors i have on boot o000:55
TJ-AHA I have it00:57
TJ-if [ -d "$rootmnt/lib/modules/$myver" ]; then00:58
TJ-that's the conditional that wrap checking for copymods=<VALUE>00:58
TJ-but there is no 'exit' if that test fails so it'll fall through to doing the tmpfs mount00:59
daftykinsso bug found?00:59
TJ-not sure of the intention but it looks likely00:59
TJ-all initrd.img that I've seen will have that path BUT IF yours was missing that path in the initrd.img then the code would continue to do the tmpfs mount01:00
TJ-so the question now is, what's actually (not) in the initrd.img on that system that triggers this, and why01:01
daftykinswould a copy of my initrd help?01:02
daftykinsthen again i've regenerated it a few times since01:02
TJ-ooooo maybe it's the move from a real /lib/ to a symlink /lib/ -> /usr/lib/ !01:02
TJ-they're doing the -d test which tests is /lib/ is a directory but if it is a symlink ...!01:03
daftykinswhen is it a symlink though?01:04
TJ-always now01:05
TJ-I've just extracted the initrd.img to check01:05
TJ-ls -l /tmp/initrd/main/lib01:06
TJ-lrwxrwxrwx 1 root root 7 Oct 26 01:04 /tmp/initrd/main/lib -> usr/lib01:06
TJ-so that'll always fall through01:06
daftykinsoic from initrd to / fs01:06
TJ-I'll check for a bug report and report it if not01:06
daftykinsi saw these folks talking similar - https://unix.stackexchange.com/questions/405146/removed-lib-modules-folder-after-every-reboot/50074201:06
daftykinsgranted it's old01:07
TJ-no, in rootfs and initrd, /lib/ is now a symlink to /usr/lib/01:07
daftykinshrmm, how come i don't see that on my 18.04 and 20.04 VMs o001:08
TJ-hmmm, no, I think I'm wrong here, since the test is on the sub-dir not the parent!01:14
TJ-can you send me a list of the content of your initrd.img? "sudo lsinitramfs /boot/initrd.img-$(uname -r) > /tmp/initrd.txt "01:15
daftykinsok if i PM the link? i think you block PMs :>01:18
TJ-I think it is due to it testing $rootmnt/lib/modules/...01:18
TJ-that means if what is on $rootmnt/ when this script triggers doesn't have a lib/modules/ then it'll fail01:19
TJ-oh!!! maybe it's the conditional that preceeds this that could create an empty $rootmnt/lib/modules !01:22
TJ-[ -d "$rootmnt/lib/modules" ] || mkdir -p "$rootmnt/lib/modules" ||01:22
TJ-trying to figure out how your rootfs could get into the state where it doesn't have /lib/modules/$VERSION though01:24
TJ-wondering if it was originally caused by how you combined the FDE instructions with the server install01:28
TJ-cannot think of anything obvious01:28
daftykinswell i feel like i've distracted you enough!01:31
daftykinsbe nice to work out those cryptsetup issues at some point, but i'm in no rush01:31
TJ-did you use the server 'live' installer?01:33
daftykinsno i couldn't, due to its' partition screen refusingto let you 'continue' via hitting next, as it claims you've not selected a device to install to... so you suggested debootstrap01:33
TJ- " systemd-cryptsetup[1778]: Device LUKS_BOOT is not active. " would be due to the missing crypttab entry01:34
daftykinshrmm will take a look at that01:36
TJ-oh yeah, and with deboostrap it would be possible to install ubuntu-server BEFORE the linux-image/linux-modules packages which would of course install cloud-initramfs-copymods and generate a new initrd.img, however that shouldn't cause this UNLESS the cloud-initramfs-copymods.postinst script creates the tmpfs !01:37
daftykinsi definitely took until late to add the kernel01:37
TJ-no, it doesn't do that in .postinst, just calls update-initramfs01:38
daftykinsweird btw, my crypttab definitely has entries...01:38
daftykinsshould the UUID be the one of the unlocked luks volume or the level above that? :D01:39
daftykinsin fact i changed the LUKS_BOOT UUID there to fix my issue of not having /boot mounted01:39
daftykinsso surely that's correct!01:39
TJ-oh, I have the same systemd-cryptsetup warnings01:40
TJ-Oct 19 06:58:37 elloe000 systemd-cryptsetup[4034]: Device LUKS_VG is still in use.01:40
TJ-Oct 19 06:58:37 elloe000 systemd-cryptsetup[4034]: Failed to deactivate: Device or resource busy01:40
TJ-Oct 19 06:58:37 elloe000 systemd[1]: systemd-cryptsetup@LUKS_VG.service: Failed with result 'exit-code'.01:40
daftykinsget the idea it's not rebooting cleanly?01:40
TJ-I think your LUKS_BOOT issue might be due to /boot/ not being mounted from fstab01:41
TJ-no - i think that is systemd-cryptsetup bugging out not realising that LUKS container with rootfs in cannot be closed due to files in the rootfs still being open01:42
daftykinsit seems ok with "/dev/mapper/LUKS_BOOT   /boot   ext4    defaults        0       0"01:42
TJ-"Device LUKS_BOOT is not active" would suggest it has already been closed01:42
TJ-if that happens every shutdown/reboot we can look at it again, but it doesn't seem to be harmful - now we know its not related to the cloud-initramfs-copymods issue01:43
daftykinsyeah! presumably all the other spinlock stuff is irrelevant01:44
TJ-not sure about those but I'd guess so01:52
daftykins:D nice work01:58
TJ-am I mis-remembering or when you built that VM was it initially missing the modules package? I vaguely recall something02:00
daftykinshrmm too long ago for me to remember now02:00
daftykinsi think it's always been present, i had a fun session discovering that /lib/modules was missing it when i was playing with adding this xe-guest-utilities package the other night02:01
daftykinspublic logs of this channel from that night could be useful02:01
TJ-looking at my logs you had some problems initially with booting, which could have triggered this, but I can't see how it would continue to do it02:08
TJ-got a bunch of read only file system errors, hrmm i never did writ02:08
TJ-e an fstab02:08
daftykinsyeah, there were a few gotchas :D02:09
TJ-thing is, this is triggered by some condition on every boot, not a config issue02:09
TJ-the only explanation that makes sense is "...hasn't been mounted to $rootmnt at the time this script executes..."02:11
daftykinsTJ-: well, thanks as always :D another mystery uncovered02:42
daftykinsi've been entertaining myself by playing with a server VM that uses subiquity's encryption02:42
lotuspsychjegood morning to all03:31
* daftykins hides03:32
daftykinsoh it's ok, no cola detected03:35
lotuspsychjeoof we can speak freely :p03:39
TJ-most powerful Xubuntu ever = HP proliant mega-server03:47
daftykinsheh, but X on a server!?03:48
TJ-all just to run gparted, to shrink an NTFS, since Windows cannot do it, because it has put the 64KB $Logfile at the end of the 500GB NTFS and can't/won't move it!03:48
daftykinsstandard :D03:49
TJ-discovered that about 40 minutes ago, after the defragger had run from 09:00 to 03:00 !03:49
daftykinshrmm i'm not sure XCP-ng is running the best, might have to record some video and ask in their channel03:49
daftykinsyeah i figured you were hoping a system file would move from the end of the drive to allow the resize, but eh - often when it does that it's not going to03:50
TJ-so now i've got xubuntu on the networked KVM (key-video-mouse)03:50
daftykinsthose servers i picked up cheap have iDRAC enterprise :D03:50
TJ-the HP has iLo but easier to use KVM right now03:54
TJ-almost done 500G > 64G03:55
TJ-just took me 10 minutes to remember how to get lvm (vgchange -ay) to activate LVs in a VG that is on an MD RAID mirror when the PVs show up on the underlying devices as well as in the MD device (edit /etc/lvm/lvm.conf and in the devices section set a filter to ignore the underlying devices; in this case filter = [ "r|/dev/nvme.*|" ]  to ignore NVME devices04:28
daftykins:D i had some fun resizing earlier since subiquity set up LVM as 50% of the disk, weird choice imo04:29
daftykinsi have no idea what i'm doing in LVM land04:29
daftykinsok time for bed, g'night!04:29
daftykinssay hi to the huskies from us \o04:29
TJ-I like that -l 50%FREE since it leaves space to either extend later, or add new LVs04:29
TJ-g'night :)04:30
lotuspsychjeTJ-: check this interesting bug from yesterday https://bugs.launchpad.net/ubuntu/+source/linux/+bug/190140105:13
lotuspsychjea lot of sdhci-pci 0000:03:00.0: error -84 requesting status in his dmesg05:14
Bashing-omYou-uns take care - 'til we meet once more - g'night!05:44
TJ-lotuspsychje: added info on potential fix06:02
lotuspsychjetnx TJ-06:04
lotuspsychjeDarkTrick: when you wakeup^^06:04
DarkTricklotuspsychje, your conversation with TJ?06:05
lotuspsychjeDarkTrick: TJ- added a new comment on your bug06:05
DarkTrick👍️ (check it later)06:06
ducassegood morning07:02
TJ-bad morning!07:28
* TJ- is suffering at the hands of subiquity07:28
ducasseTJ-: drag it out back and shoot it07:29
DarkTricklotuspsychje, with more brainpower now, I understand the messages now.07:34
DarkTrickTJ-, do you have an information when the fixes will be delivered?07:34
DarkTrick= available in latest ubuntu07:35
TJ-DarkTrick: I don't know if those are the fixes; they're just commits against the driver that applies to the SDHCI controller in your system07:43
TJ-DarkTrick: test it with a mainline kernel build of v5.907:43
TJ-!mainline | DarkTrick07:43
TJ-Where's ubottu gone?07:44
TJ-DarkTrick: see in #ubuntu07:44
ducassedunno, has been gone for a good while07:44
TJ-reported 2 bugs against subiquity already this morning, and seen one or 2 other issues that are anit-user/admin07:46
lotuspsychjeTJ-: i requested in -ops for ubottu a few times in here, nobody cares anymore07:47
TJ-lotuspsychje: sounds typical - long slow decline of Ubuntu07:51
lotuspsychjethe bot in +1 needs a respin too, showing off wrong kernels07:52
TJ-does it not do a live lookup ?07:53
lotuspsychjenot sure of the mechanism they use, but in the past we saw the bots need a reload once in a while to show the right info07:54
TJ-wonderful - subiquity encrypted install reboot fails; GRUB cannot find its root/prefix. I'm guessing due to the GRUB not being told that its /boot/ is inside a LUKS container and/or the container is using LUKSv2 whereas GRUB only supports LUKSv108:06
TJ-This just confirms my experience that using debootstrap is faster!08:06
lotuspsychjeTJ-: 101 new bugs; https://bugs.launchpad.net/subiquity/+bugs?orderby=importance&start=008:09
TJ-4 are mine!08:09
lotuspsychjeTJ-: thank you to file bug #1901488 and make ubuntu better :p08:23
lotuspsychjeanyone can loan me 45.000$ for the new system76? https://imgur.com/a/wSqZjvK08:27
marcoagpintoThe cola demon is here!11:02
=== lotuspsychje_ is now known as lotuspsychje
marcoagpintoare .04 only LTS releases?15:54
marcoagpintoI can't remember15:54
marcoagpintowill there be a 21.04 (non-LTS)?15:54
daftykinsno it's only even years15:56
daftykinscan you not google? :)15:56
marcoagpintowhy google if the core is right here?15:56
lotuspsychjemarcoagpinto: you know this well enough15:56
marcoagpintolotuspsychje: I am having memory breakages15:56
daftykinsbecause then you're not relying on people15:56
lotuspsychjemarcoagpinto: its better you start drinking ubuntu cola for your memory15:57
daftykinsteach a cola demon to fish... ;)15:57
ducasseanyone using the zfs backing store with lxc?16:10
marcoagpintoguys?! Sorry to bother again...16:11
marcoagpintowill there be a 20.10?16:11
marcoagpintoor is it even years too?16:11
marcoagpintonow I am lost16:11
marcoagpintoI meant 21.1016:11
ducassethere is a release every six months16:11
ducassethis is not news16:12
marcoagpinto[15:54] <marcoagpinto> will there be a 21.04 (non-LTS)?16:12
marcoagpinto[15:55] <daftykins> no it's only even years16:12
marcoagpintowhat will be the next release after 20.10?16:12
marcoagpintoI am getting so confused :(((((((((16:13
ducassethere will be a 21.04, but it's not lts16:13
marcoagpintobut I asked that above16:14
ducassei think he misread you16:14
marcoagpintonow my mind is getting clearer16:14
marcoagpintoyou can't imagine the pressure over me16:15
marcoagpintoI can't think properly16:15
ducassethis is documented on the ubuntu wiki, i suggest you read that for details16:15
daftykinsi didn't misread, what i said was pretty clear16:15
ducasseoh, you answered his first question, not his second :)16:16
marcoagpintosuffer the demons16:17
ducassein short: new release every six months, .04 and .10 - lts is <even number>.0416:17
marcoagpintoif I ask this again please throw some cola at me16:17
ducasseif you ask again we'll point you to the wiki :)16:23
daftykinspoison cola it is16:25
ducassecovid cola16:31
marcoagpintoCOVID is very bad :(((((16:35
marcoagpintoone of my oldest doctors died of it on June16:35
marcoagpintoI phoned to get a medical appointment and I was told he died16:35
marcoagpintofor over 6 months that I didn't have appointments with him16:35
daftykinsyikes, i wouldn't know what to say to that16:36
marcoagpintoit was  shock to me16:36
marcoagpintoit was a shock to me16:36
marcoagpintoevery time I had respiratory problems, it was him who prescribed the medicins16:38
=== akem_ is now known as Akem
Maikmarcoagpinto: LTS versions will be released every two years, the next LTS will be 22.04, then 24.04 and so on and will be support for five years. Every release in between will be regular releases which are only supported for 9 months.16:41
Maikyw :)16:47
Bashing-omUWN issue 654 is on the streets: https://wiki.ubuntu.com/UbuntuWeeklyNewsletter/Issue654 :D20:12

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!