| TJ- | mgedmin: see "man dpkg" and the explanation of --verify-format | 00:00 |
|---|---|---|
| TJ- | daftykins: then there is something bad happening - check the logs "journalctl -b -1 -p warning" to check the previous boot's log | 00:00 |
| mgedmin | ah! | 00:01 |
| daftykins | https://termbin.com/7g3bk - here's the output | 00:02 |
| daftykins | prox-xen.mount is the guest utilities process, failing to mount Xen ProcFS since the xenfs module isn't loaded | 00:03 |
| TJ- | daftykins: I suspect those "systemd-cryptsetup" reports are a big clue | 00:07 |
| TJ- | daftykins: "systemd-cryptsetup@xvda5_crypt.service: Failed with result 'exit-code'." is the LUKS container for the root file-system | 00:08 |
| daftykins | yip | 00:08 |
| daftykins | i might rule out the storage by moving the VM back to the mechanical RAID and see if purging and reinstalling the modules package sticks | 00:08 |
| TJ- | We've been doing that kind of install today across 8 systems and not seen any errors like that, so I think there's something about your specific scenario with Xen | 00:09 |
| daftykins | boot is painfully slow under Xen :( | 00:11 |
| daftykins | well, i say that but i don't have any frame of reference as i've not done an encrypted install on any other setup :D | 00:11 |
| daftykins | certainly unencrypted VMs are slower to boot than ones on my vmware ESXi system | 00:11 |
| mgedmin | it took a thinkpad 31 seconds to boot to gdm (firmware + loader + kernel + userspace), and that time includes me typing the passphrase | 00:15 |
| mgedmin | luks on an nvme ssd | 00:15 |
| daftykins | i don't understand what caused these lines to be thrown, recently | 00:17 |
| daftykins | The following packages were automatically installed and are no longer required: | 00:17 |
| daftykins | amd64-microcode intel-microcode iucode-tool libdbus-glib-1-2 r-cran-ellipse thermald | 00:17 |
| daftykins | wow that's the first time i didn't have a yellow TTY1 | 00:18 |
| daftykins | still gets stuck on a purple background with flashing cursor top left until i manually switch to a TTY | 00:18 |
| TJ- | that sounds like the vt.handoff= set on the kernel command line | 00:19 |
| TJ- | also sounds like it's setting GRUB in gfx mode instead of text | 00:19 |
| daftykins | should "/etc/default/grub" read GRUB_CMDLINE_LINUX_DEFAULT="quiet text" instead of quiet splash? | 00:20 |
| daftykins | anyway that's separate to my main issue right now, still tinkering | 00:20 |
| daftykins | ok dpkg --verify came back ok | 00:21 |
| daftykins | now i'm rebooting | 00:21 |
| daftykins | yep files disappeared again | 00:25 |
| daftykins | Oct 26 00:21:14 talkietoaster systemd[1]: Failed unmounting /lib/modules. | 00:27 |
| daftykins | o0 | 00:27 |
| TJ- | GRUB_CMDLINE... on a server, I'd leave off quiet and no, it doesn't need 'text', instead you set "GRUB_TERMINAL=console" when using grub-pc (BIOS mode boots) to do text only | 00:28 |
| TJ- | unmounting /lib/modules!?!?! | 00:29 |
| daftykins | pass :D well it seems like the machine goes down for a reboot without honouring taking care of cryptsetup then | 00:29 |
| TJ- | that error suggests /lib/modules/ is in fact a mount point | 00:30 |
| TJ- | you need to check that out | 00:30 |
| daftykins | copymods on /lib/modules type tmpfs (rw,relatime) | 00:31 |
| daftykins | o0 | 00:31 |
| TJ- | There you go! | 00:32 |
| TJ- | where the heck did that come from? | 00:32 |
| daftykins | not a clue :O | 00:32 |
| TJ- | aha cloud-initramfs-copymods | 00:32 |
| TJ- | !info cloud-initramfs-copymods | 00:32 |
| TJ- | no ubottu! | 00:33 |
| daftykins | ii cloud-initramfs-copymods 0.45ubuntu1 all copy initramfs modules into root filesystem for later use | 00:33 |
| TJ- | When booting with an external-to-root kernel and initramfs, you need to | 00:33 |
| TJ- | ensure that /lib/modules contains any necessary modules not already | 00:33 |
| TJ- | loaded. | 00:33 |
| TJ- | . | 00:33 |
| TJ- | This package arranges for the modules in the initramfs to be placed into | 00:33 |
| TJ- | /lib/modules after the switchroot is done. | 00:33 |
| TJ- | are you booting with kernel/initrd.img external to the VM? | 00:34 |
| daftykins | not external no, it's all in the VDI | 00:34 |
| TJ- | So my original feeling that there's a tmpfs was correct | 00:34 |
| daftykins | but given i messed up /boot - what else could i have messed up o0 | 00:34 |
| TJ- | OK, so remove/purge that package after figuring out how it got installed | 00:34 |
| daftykins | well i'm not going to have a clue on that one :D | 00:35 |
| daftykins | hang on | 00:35 |
| daftykins | is it part of ubuntu-server ... it wants to remove them together | 00:35 |
| TJ- | aha, it's a depends of ubuntu-server | 00:35 |
| TJ- | this makes no sense! | 00:36 |
| daftykins | i used the ubuntu-server metapackage on your suggestion after the debootstrap, so presumably it's ok to lose? | 00:36 |
| * daftykins goes ahead | 00:37 | |
| TJ- | check the kernel command-line because , on reading the scripts in that package, they look for "copymods=" on the kernel command line at boot time | 00:40 |
| TJ- | in there I see: log_warning_msg "copying over existing modules! due to copymods=force" | 00:42 |
| daftykins | is /proc/cmdline sufficient? i see: BOOT_IMAGE=/vmlinuz-5.4.0-52-generic root=/dev/mapper/ubuntu--vg-root ro quiet splash vt.handoff=7 | 00:42 |
| TJ- | that's fine , and we do not see copymods... let me test your cmdline with that code to see if there's a bug in it | 00:43 |
| TJ- | might be triggering on something other than the intended | 00:43 |
| daftykins | i rebooted and amusingly now have the 5.4.0-51-generic back in place in /lib/modules/ - because that must now be the real path on / | 00:44 |
| daftykins | now reinstalling linux-modules-5.4.0-52-generic | 00:44 |
| daftykins | seems to stall on "Begin: Running /scripts/init-premount ..." for ages | 00:50 |
| daftykins | oh it was done, again i had to switch away | 00:50 |
| daftykins | and now xenfs loads and things work as expected \o/ | 00:51 |
| TJ- | I can't work out why that script was triggered and setting up the tmpfs | 00:53 |
| daftykins | any thoughts on how to resolve those systemd-cryptsetup errors i have on boot o0 | 00:55 |
| TJ- | AHA I have it | 00:57 |
| TJ- | if [ -d "$rootmnt/lib/modules/$myver" ]; then | 00:58 |
| TJ- | that's the conditional that wrap checking for copymods=<VALUE> | 00:58 |
| TJ- | but there is no 'exit' if that test fails so it'll fall through to doing the tmpfs mount | 00:59 |
| daftykins | D: | 00:59 |
| daftykins | so bug found? | 00:59 |
| TJ- | not sure of the intention but it looks likely | 00:59 |
| TJ- | all initrd.img that I've seen will have that path BUT IF yours was missing that path in the initrd.img then the code would continue to do the tmpfs mount | 01:00 |
| TJ- | so the question now is, what's actually (not) in the initrd.img on that system that triggers this, and why | 01:01 |
| daftykins | would a copy of my initrd help? | 01:02 |
| daftykins | then again i've regenerated it a few times since | 01:02 |
| TJ- | ooooo maybe it's the move from a real /lib/ to a symlink /lib/ -> /usr/lib/ ! | 01:02 |
| TJ- | they're doing the -d test which tests is /lib/ is a directory but if it is a symlink ...! | 01:03 |
| daftykins | :O | 01:03 |
| daftykins | when is it a symlink though? | 01:04 |
| TJ- | always now | 01:05 |
| TJ- | I've just extracted the initrd.img to check | 01:05 |
| TJ- | ls -l /tmp/initrd/main/lib | 01:06 |
| TJ- | lrwxrwxrwx 1 root root 7 Oct 26 01:04 /tmp/initrd/main/lib -> usr/lib | 01:06 |
| TJ- | so that'll always fall through | 01:06 |
| daftykins | oic from initrd to / fs | 01:06 |
| TJ- | I'll check for a bug report and report it if not | 01:06 |
| daftykins | i saw these folks talking similar - https://unix.stackexchange.com/questions/405146/removed-lib-modules-folder-after-every-reboot/500742 | 01:06 |
| daftykins | granted it's old | 01:07 |
| TJ- | no, in rootfs and initrd, /lib/ is now a symlink to /usr/lib/ | 01:07 |
| daftykins | hrmm, how come i don't see that on my 18.04 and 20.04 VMs o0 | 01:08 |
| TJ- | hmmm, no, I think I'm wrong here, since the test is on the sub-dir not the parent! | 01:14 |
| TJ- | can you send me a list of the content of your initrd.img? "sudo lsinitramfs /boot/initrd.img-$(uname -r) > /tmp/initrd.txt " | 01:15 |
| daftykins | ok if i PM the link? i think you block PMs :> | 01:18 |
| TJ- | I think it is due to it testing $rootmnt/lib/modules/... | 01:18 |
| TJ- | that means if what is on $rootmnt/ when this script triggers doesn't have a lib/modules/ then it'll fail | 01:19 |
| TJ- | oh!!! maybe it's the conditional that preceeds this that could create an empty $rootmnt/lib/modules ! | 01:22 |
| TJ- | [ -d "$rootmnt/lib/modules" ] || mkdir -p "$rootmnt/lib/modules" || | 01:22 |
| TJ- | trying to figure out how your rootfs could get into the state where it doesn't have /lib/modules/$VERSION though | 01:24 |
| daftykins | hrmm | 01:25 |
| TJ- | wondering if it was originally caused by how you combined the FDE instructions with the server install | 01:28 |
| TJ- | cannot think of anything obvious | 01:28 |
| daftykins | well i feel like i've distracted you enough! | 01:31 |
| daftykins | be nice to work out those cryptsetup issues at some point, but i'm in no rush | 01:31 |
| TJ- | did you use the server 'live' installer? | 01:33 |
| daftykins | no i couldn't, due to its' partition screen refusingto let you 'continue' via hitting next, as it claims you've not selected a device to install to... so you suggested debootstrap | 01:33 |
| TJ- | " systemd-cryptsetup[1778]: Device LUKS_BOOT is not active. " would be due to the missing crypttab entry | 01:34 |
| daftykins | hrmm will take a look at that | 01:36 |
| TJ- | oh yeah, and with deboostrap it would be possible to install ubuntu-server BEFORE the linux-image/linux-modules packages which would of course install cloud-initramfs-copymods and generate a new initrd.img, however that shouldn't cause this UNLESS the cloud-initramfs-copymods.postinst script creates the tmpfs ! | 01:37 |
| daftykins | i definitely took until late to add the kernel | 01:37 |
| TJ- | no, it doesn't do that in .postinst, just calls update-initramfs | 01:38 |
| daftykins | weird btw, my crypttab definitely has entries... | 01:38 |
| daftykins | should the UUID be the one of the unlocked luks volume or the level above that? :D | 01:39 |
| daftykins | in fact i changed the LUKS_BOOT UUID there to fix my issue of not having /boot mounted | 01:39 |
| daftykins | so surely that's correct! | 01:39 |
| TJ- | oh, I have the same systemd-cryptsetup warnings | 01:40 |
| TJ- | Oct 19 06:58:37 elloe000 systemd-cryptsetup[4034]: Device LUKS_VG is still in use. | 01:40 |
| TJ- | Oct 19 06:58:37 elloe000 systemd-cryptsetup[4034]: Failed to deactivate: Device or resource busy | 01:40 |
| TJ- | Oct 19 06:58:37 elloe000 systemd[1]: systemd-cryptsetup@LUKS_VG.service: Failed with result 'exit-code'. | 01:40 |
| daftykins | get the idea it's not rebooting cleanly? | 01:40 |
| TJ- | I think your LUKS_BOOT issue might be due to /boot/ not being mounted from fstab | 01:41 |
| TJ- | no - i think that is systemd-cryptsetup bugging out not realising that LUKS container with rootfs in cannot be closed due to files in the rootfs still being open | 01:42 |
| daftykins | it seems ok with "/dev/mapper/LUKS_BOOT /boot ext4 defaults 0 0" | 01:42 |
| TJ- | "Device LUKS_BOOT is not active" would suggest it has already been closed | 01:42 |
| TJ- | if that happens every shutdown/reboot we can look at it again, but it doesn't seem to be harmful - now we know its not related to the cloud-initramfs-copymods issue | 01:43 |
| daftykins | yeah! presumably all the other spinlock stuff is irrelevant | 01:44 |
| TJ- | not sure about those but I'd guess so | 01:52 |
| TJ- | https://bugs.launchpad.net/ubuntu/+source/cloud-initramfs-tools/+bug/1901464 | 01:57 |
| daftykins | :D nice work | 01:58 |
| TJ- | am I mis-remembering or when you built that VM was it initially missing the modules package? I vaguely recall something | 02:00 |
| daftykins | hrmm too long ago for me to remember now | 02:00 |
| daftykins | i think it's always been present, i had a fun session discovering that /lib/modules was missing it when i was playing with adding this xe-guest-utilities package the other night | 02:01 |
| daftykins | public logs of this channel from that night could be useful | 02:01 |
| daftykins | https://irclogs.ubuntu.com/2020/10/23/%23ubuntu-discuss.txt | 02:02 |
| TJ- | looking at my logs you had some problems initially with booting, which could have triggered this, but I can't see how it would continue to do it | 02:08 |
| TJ- | got a bunch of read only file system errors, hrmm i never did writ | 02:08 |
| TJ- | e an fstab | 02:08 |
| TJ- | ^^^ | 02:08 |
| daftykins | yeah, there were a few gotchas :D | 02:09 |
| TJ- | thing is, this is triggered by some condition on every boot, not a config issue | 02:09 |
| TJ- | the only explanation that makes sense is "...hasn't been mounted to $rootmnt at the time this script executes..." | 02:11 |
| daftykins | TJ-: well, thanks as always :D another mystery uncovered | 02:42 |
| daftykins | i've been entertaining myself by playing with a server VM that uses subiquity's encryption | 02:42 |
| lotuspsychje | good morning to all | 03:31 |
| * daftykins hides | 03:32 | |
| lotuspsychje | heh | 03:35 |
| daftykins | oh it's ok, no cola detected | 03:35 |
| lotuspsychje | oof we can speak freely :p | 03:39 |
| daftykins | xD | 03:42 |
| TJ- | most powerful Xubuntu ever = HP proliant mega-server | 03:47 |
| daftykins | heh, but X on a server!? | 03:48 |
| TJ- | all just to run gparted, to shrink an NTFS, since Windows cannot do it, because it has put the 64KB $Logfile at the end of the 500GB NTFS and can't/won't move it! | 03:48 |
| daftykins | standard :D | 03:49 |
| TJ- | discovered that about 40 minutes ago, after the defragger had run from 09:00 to 03:00 ! | 03:49 |
| daftykins | hrmm i'm not sure XCP-ng is running the best, might have to record some video and ask in their channel | 03:49 |
| daftykins | yeah i figured you were hoping a system file would move from the end of the drive to allow the resize, but eh - often when it does that it's not going to | 03:50 |
| TJ- | so now i've got xubuntu on the networked KVM (key-video-mouse) | 03:50 |
| daftykins | those servers i picked up cheap have iDRAC enterprise :D | 03:50 |
| TJ- | the HP has iLo but easier to use KVM right now | 03:54 |
| TJ- | almost done 500G > 64G | 03:55 |
| TJ- | just took me 10 minutes to remember how to get lvm (vgchange -ay) to activate LVs in a VG that is on an MD RAID mirror when the PVs show up on the underlying devices as well as in the MD device (edit /etc/lvm/lvm.conf and in the devices section set a filter to ignore the underlying devices; in this case filter = [ "r|/dev/nvme.*|" ] to ignore NVME devices | 04:28 |
| daftykins | :D i had some fun resizing earlier since subiquity set up LVM as 50% of the disk, weird choice imo | 04:29 |
| daftykins | i have no idea what i'm doing in LVM land | 04:29 |
| daftykins | ok time for bed, g'night! | 04:29 |
| daftykins | say hi to the huskies from us \o | 04:29 |
| TJ- | I like that -l 50%FREE since it leaves space to either extend later, or add new LVs | 04:29 |
| TJ- | g'night :) | 04:30 |
| lotuspsychje | TJ-: check this interesting bug from yesterday https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1901401 | 05:13 |
| lotuspsychje | a lot of sdhci-pci 0000:03:00.0: error -84 requesting status in his dmesg | 05:14 |
| Bashing-om | You-uns take care - 'til we meet once more - g'night! | 05:44 |
| TJ- | lotuspsychje: added info on potential fix | 06:02 |
| lotuspsychje | tnx TJ- | 06:04 |
| lotuspsychje | DarkTrick: when you wakeup^^ | 06:04 |
| DarkTrick | lotuspsychje, your conversation with TJ? | 06:05 |
| lotuspsychje | DarkTrick: TJ- added a new comment on your bug | 06:05 |
| DarkTrick | 👍️ (check it later) | 06:06 |
| ducasse | good morning | 07:02 |
| TJ- | bad morning! | 07:28 |
| * TJ- is suffering at the hands of subiquity | 07:28 | |
| ducasse | TJ-: drag it out back and shoot it | 07:29 |
| DarkTrick | lotuspsychje, with more brainpower now, I understand the messages now. | 07:34 |
| DarkTrick | TJ-, do you have an information when the fixes will be delivered? | 07:34 |
| DarkTrick | = available in latest ubuntu | 07:35 |
| TJ- | DarkTrick: I don't know if those are the fixes; they're just commits against the driver that applies to the SDHCI controller in your system | 07:43 |
| TJ- | DarkTrick: test it with a mainline kernel build of v5.9 | 07:43 |
| TJ- | !mainline | DarkTrick | 07:43 |
| TJ- | Where's ubottu gone? | 07:44 |
| TJ- | DarkTrick: see in #ubuntu | 07:44 |
| ducasse | dunno, has been gone for a good while | 07:44 |
| TJ- | reported 2 bugs against subiquity already this morning, and seen one or 2 other issues that are anit-user/admin | 07:46 |
| lotuspsychje | TJ-: i requested in -ops for ubottu a few times in here, nobody cares anymore | 07:47 |
| TJ- | lotuspsychje: sounds typical - long slow decline of Ubuntu | 07:51 |
| lotuspsychje | the bot in +1 needs a respin too, showing off wrong kernels | 07:52 |
| TJ- | does it not do a live lookup ? | 07:53 |
| lotuspsychje | not sure of the mechanism they use, but in the past we saw the bots need a reload once in a while to show the right info | 07:54 |
| TJ- | wonderful - subiquity encrypted install reboot fails; GRUB cannot find its root/prefix. I'm guessing due to the GRUB not being told that its /boot/ is inside a LUKS container and/or the container is using LUKSv2 whereas GRUB only supports LUKSv1 | 08:06 |
| TJ- | This just confirms my experience that using debootstrap is faster! | 08:06 |
| lotuspsychje | TJ-: 101 new bugs; https://bugs.launchpad.net/subiquity/+bugs?orderby=importance&start=0 | 08:09 |
| TJ- | 4 are mine! | 08:09 |
| lotuspsychje | heh | 08:09 |
| lotuspsychje | TJ-: thank you to file bug #1901488 and make ubuntu better :p | 08:23 |
| lotuspsychje | anyone can loan me 45.000$ for the new system76? https://imgur.com/a/wSqZjvK | 08:27 |
| marcoagpinto | The cola demon is here! | 11:02 |
| === lotuspsychje_ is now known as lotuspsychje | ||
| marcoagpinto | guys?! | 15:54 |
| marcoagpinto | are .04 only LTS releases? | 15:54 |
| marcoagpinto | I can't remember | 15:54 |
| marcoagpinto | will there be a 21.04 (non-LTS)? | 15:54 |
| daftykins | no it's only even years | 15:56 |
| daftykins | can you not google? :) | 15:56 |
| marcoagpinto | why google if the core is right here? | 15:56 |
| marcoagpinto | :) | 15:56 |
| lotuspsychje | marcoagpinto: you know this well enough | 15:56 |
| marcoagpinto | lotuspsychje: I am having memory breakages | 15:56 |
| daftykins | because then you're not relying on people | 15:56 |
| lotuspsychje | marcoagpinto: its better you start drinking ubuntu cola for your memory | 15:57 |
| daftykins | teach a cola demon to fish... ;) | 15:57 |
| marcoagpinto | Buaaaaaaaaaaaa | 15:57 |
| ducasse | anyone using the zfs backing store with lxc? | 16:10 |
| marcoagpinto | guys?! Sorry to bother again... | 16:11 |
| marcoagpinto | will there be a 20.10? | 16:11 |
| marcoagpinto | or is it even years too? | 16:11 |
| marcoagpinto | now I am lost | 16:11 |
| marcoagpinto | 21.10* | 16:11 |
| marcoagpinto | I meant 21.10 | 16:11 |
| ducasse | there is a release every six months | 16:11 |
| ducasse | this is not news | 16:12 |
| marcoagpinto | [15:54] <marcoagpinto> will there be a 21.04 (non-LTS)? | 16:12 |
| marcoagpinto | [15:55] <daftykins> no it's only even years | 16:12 |
| marcoagpinto | so? | 16:12 |
| marcoagpinto | what will be the next release after 20.10? | 16:12 |
| marcoagpinto | I am getting so confused :((((((((( | 16:13 |
| ducasse | there will be a 21.04, but it's not lts | 16:13 |
| marcoagpinto | but I asked that above | 16:14 |
| ducasse | i think he misread you | 16:14 |
| marcoagpinto | ahhhhhhh | 16:14 |
| marcoagpinto | :) | 16:14 |
| marcoagpinto | now my mind is getting clearer | 16:14 |
| marcoagpinto | thanks | 16:14 |
| marcoagpinto | you can't imagine the pressure over me | 16:15 |
| marcoagpinto | :p | 16:15 |
| marcoagpinto | I can't think properly | 16:15 |
| ducasse | this is documented on the ubuntu wiki, i suggest you read that for details | 16:15 |
| daftykins | i didn't misread, what i said was pretty clear | 16:15 |
| ducasse | oh, you answered his first question, not his second :) | 16:16 |
| marcoagpinto | Buaaaaaa | 16:16 |
| marcoagpinto | suffer the demons | 16:17 |
| ducasse | in short: new release every six months, .04 and .10 - lts is <even number>.04 | 16:17 |
| marcoagpinto | right | 16:17 |
| marcoagpinto | thanks | 16:17 |
| marcoagpinto | :) | 16:17 |
| marcoagpinto | if I ask this again please throw some cola at me | 16:17 |
| marcoagpinto | :) | 16:17 |
| ducasse | if you ask again we'll point you to the wiki :) | 16:23 |
| marcoagpinto | right | 16:24 |
| marcoagpinto | :) | 16:24 |
| daftykins | poison cola it is | 16:25 |
| ducasse | covid cola | 16:31 |
| daftykins | xD | 16:33 |
| marcoagpinto | COVID is very bad :((((( | 16:35 |
| marcoagpinto | one of my oldest doctors died of it on June | 16:35 |
| marcoagpinto | I phoned to get a medical appointment and I was told he died | 16:35 |
| marcoagpinto | for over 6 months that I didn't have appointments with him | 16:35 |
| daftykins | yikes, i wouldn't know what to say to that | 16:36 |
| marcoagpinto | it was shock to me | 16:36 |
| marcoagpinto | it was a shock to me | 16:36 |
| marcoagpinto | every time I had respiratory problems, it was him who prescribed the medicins | 16:38 |
| === akem_ is now known as Akem | ||
| Maik | marcoagpinto: LTS versions will be released every two years, the next LTS will be 22.04, then 24.04 and so on and will be support for five years. Every release in between will be regular releases which are only supported for 9 months. | 16:41 |
| marcoagpinto | thanks | 16:43 |
| Maik | yw :) | 16:47 |
| Bashing-om | UWN issue 654 is on the streets: https://wiki.ubuntu.com/UbuntuWeeklyNewsletter/Issue654 :D | 20:12 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!