| lunaphyte | i've just upgraded from 17.10 to 18.04, and now the computer won't boot. it fails to mount the root filesystem, and drops to initramfs | 04:37 |
|---|---|---|
| lunaphyte | https://9unvsq.bn.files.1drv.com/y4m6YIxUZfXzd96cuCzcKJR22rzSeBCy-2VUrndZk5gLlOyk52M6w8oZgr-ybtLbnVvC1trcidcR8L1ndgSUzH9NSysNDl1jTUD5_wBIPQ_PwSMIrWNa69rqvVv_MgV3o2czkQebQUj27idfawPbDN5jaY40rOW5swWLpKXpiu3yMgcPKj_PP6GROCAvAJpnpn3maXS_5GPsrkocfuEEbbHQQ/residue_2019_01_24.PNG?psid=1 | 04:39 |
| lunaphyte | ugh, sorry about the awful url | 04:39 |
| lunaphyte | the upgrade process went without any issues or complaints at all | 04:40 |
| lunaphyte | what can i do to troubleshoot this? | 04:40 |
| lunaphyte | since i'd had a similar problem with another upgrade not going well, with non root filesystems that were not accessible, i tried vgchange -aay, on a whim. so far, it seems to just hang upon that command indefinitely, and not do anything | 04:45 |
| lordievader | Good morning | 07:10 |
| lordievader | lunaphyte: From that initramfs can you mount your rootfs? | 07:11 |
| === cpaelzer__ is now known as cpaelzer | ||
| jamespage | cpaelzer: looking at your dpdk/ovs work today | 09:43 |
| cpaelzer | jamespage: ok, I started to test it as well | 09:54 |
| cpaelzer | but that takes quite some time | 09:54 |
| cpaelzer | as I said, I'll ping you once I have a reasonable result | 09:54 |
| jamespage | cpaelzer: ack | 09:59 |
| ahasenack | good morning | 11:03 |
| rbasak | ahasenack, cpaelzer, kstenerud: I can hit the review queue now, but it looks like everything has been looked at for the moment? | 14:06 |
| ahasenack | rbasak: agreed | 14:07 |
| ahasenack | I'm finishing up haproxy, then I'll go after apache2 which you reviewed already, and back to the net-snmp merge | 14:07 |
| rbasak | OK | 14:07 |
| ahasenack | rbasak: can you merge that salsa haproxy mp, or are you just passing by? | 14:26 |
| rbasak | ahasenack: I was just passing by, being curious about how haproxy tests operate as I know nothing about haproxy. | 14:29 |
| ahasenack | cool | 14:29 |
| ahasenack | first attempt failed, investigating | 14:29 |
| rbasak | I am not a maintainer for haproxy in Debian so I shouldn't do anything but comment. | 14:29 |
| ahasenack | https://pastebin.ubuntu.com/p/CvVKbJC2XV/ | 14:29 |
| ahasenack | ah, I have to invert the check | 14:31 |
| rbasak | Yeah, or swap the else block. | 14:31 |
| rbasak | (I prefer the latter in general to avoid an extra inversion) | 14:34 |
| ahasenack | yeah, I also dislike "!" in ifs | 14:36 |
| ahasenack | rbasak: better, let me push: https://pastebin.ubuntu.com/p/WDtmWnZRCZ/ | 14:40 |
| MJCD | i'm trying to set up xorg+lightdm+fluxbox on a server build | 15:03 |
| MJCD | i've gotten a graphical login screen after reboot | 15:03 |
| MJCD | but it just fails to start session | 15:04 |
| MJCD | lol if I switch from fluxbox to openbox | 15:06 |
| MJCD | it lets me log in, but to just black nothing | 15:07 |
| MJCD | no right click menu even | 15:07 |
| MJCD | >.< | 15:07 |
| lordcirth_ | MJCD, anything in logs? | 15:10 |
| MJCD | its ok third reboot was the charm for whatever reason, fluxbox still doesn't work | 15:10 |
| MJCD | but i'm fine with openbox for now | 15:10 |
| ahasenack | rbasak: can I consider our salsa exchange as another +1 for the haproxy dep8 tests, provided tests pass in ubuntu? For this MP: https://code.launchpad.net/~ahasenack/ubuntu/+source/haproxy/+git/haproxy/+merge/362217 | 17:31 |
| ahasenack | cpaelzer is eod now | 17:31 |
| ahasenack | I induced some errors to see how it would behave, and it looks good: | 17:31 |
| ahasenack | - diff content: https://pastebin.com/Jcn8TE4A | 17:32 |
| ahasenack | - diff size: https://pastebin.com/4y4crD39 | 17:33 |
| rbasak | ahasenack: yes, and +1 added to the MP | 17:57 |
| ahasenack | rbasak: thanks | 17:58 |
| === Tornevall is now known as Guest36728 | ||
| lunaphyte | lordievader: thanks for responding. i can't, no. | 20:05 |
| lunaphyte | if i try to mount the root filesystem manually at the initramfs prompt, it says the same thing - "mount: mounting /dev/mapper/vg1-root on /root failed: no such device" | 20:07 |
| lunaphyte | all of the lvm pieces are active and functioning, i can see all of the physical volumes, all of the volume groups, all of the logical volumes | 20:07 |
| lunaphyte | i can run btrfs check on /dev/mapper/vg1-root and it runs properly and successfully, and finds no errors | 20:08 |
| TJ- | lunaphyte: broke it again?! | 20:08 |
| lunaphyte | TJ-: :) - hah - no, this is a different one | 20:09 |
| lunaphyte | https://9unvsq.bn.files.1drv.com/y4m6YIxUZfXzd96cuCzcKJR22rzSeBCy-2VUrndZk5gLlOyk52M6w8oZgr-ybtLbnVvC1trcidcR8L1ndgSUzH9NSysNDl1jTUD5_wBIPQ_PwSMIrWNa69rqvVv_MgV3o2czkQebQUj27idfawPbDN5jaY40rOW5swWLpKXpiu3yMgcPKj_PP6GROCAvAJpnpn3maXS_5GPsrkocfuEEbbHQQ/residue_2019_01_24.PNG?psid=1 | 20:10 |
| TJ- | lunaphyte: compare the dm-X number shown by "ls -l /dev/mapper/vg1-root" with "ls -l /dev/dm-X" to ensure it exists | 20:10 |
| TJ- | lunaphyte: oh, a VM image? is the underlying file corrupt. You shouldn't get the kernel stracktrace | 20:11 |
| lunaphyte | TJ-: i don't think anything is corrupt. i can boot off the installer, in rescue mode, and successfully mount all filesystem and chroot into them as the root | 20:12 |
| lunaphyte | yeah, it's a vmware guest | 20:12 |
| TJ- | lunaphyte: is this always a VMware guest? | 20:13 |
| lunaphyte | yes | 20:13 |
| lunaphyte | it always has been, from its inception, and always will be | 20:13 |
| TJ- | lunaphyte: any changes to the hardware profile ? | 20:13 |
| lunaphyte | no changes to hardware | 20:14 |
| lunaphyte | just an upgrade from 17.10 to 18.04, that's it | 20:14 |
| lunaphyte | no errors at all during the upgrade | 20:14 |
| lunaphyte | went perfectly typical | 20:14 |
| TJ- | lunaphyte: that screenshot suggests /root/ doesn't exist | 20:14 |
| TJ- | lunaphyte: all commands trying to set-up the real file-system on /root/ fail - either that is missing or the real rootfs really has a major problem | 20:15 |
| lunaphyte | /root/ does exist | 20:15 |
| TJ- | lunaphyte: so does the block device symlinked from /dev/mapper/vg1-root exist? | 20:16 |
| lunaphyte | /dev/md-0 ? | 20:16 |
| lunaphyte | yeah, that exists | 20:16 |
| lunaphyte | i can run fsck successfully on the device | 20:16 |
| TJ- | lunaphyte: I have no idea, that that is NOT an LVM node | 20:16 |
| lunaphyte | it properly recognizes the filesystem | 20:16 |
| TJ- | lunaphyte: as I said earlier: ---> compare the dm-X number shown by "ls -l /dev/mapper/vg1-root" with "ls -l /dev/dm-X" to ensure it exists | 20:16 |
| lunaphyte | yeah, it does | 20:17 |
| lunaphyte | bah, sorry | 20:18 |
| lunaphyte | i missed that irc ignored what i wrote because of the leading slash | 20:18 |
| lunaphyte | /dev/mapper/vg1-root points to /dev/dm-0, which exists | 20:18 |
| TJ- | lunaphyte: the stacktrace shows that get_fs_type() fails, so what does "blkid /dev/dm-0" report? | 20:19 |
| lunaphyte | https://pasteboard.co/HYJHXiJ.png | 20:19 |
| lunaphyte | one moment | 20:19 |
| lunaphyte | https://pasteboard.co/HYJIrFu.png | 20:20 |
| lunaphyte | sorry i can't just use normal text. i'm stuck inside a virtual guest console | 20:21 |
| TJ- | I know :) | 20:21 |
| TJ- | So, this is really looking like a btrfs problem. is the module available? "lsmod | grep btrfs" | 20:22 |
| lunaphyte | ugh. no lsmod in this busybox, it seems | 20:23 |
| sarnold | /proc/modules ? | 20:24 |
| TJ- | lunaphyte: "grep btrfs /pro/mocules" | 20:24 |
| sarnold | it's ugly but still mostly legible | 20:24 |
| TJ- | modules! | 20:24 |
| lunaphyte | i did check the initrd earlier with lsinitramfs and it seems to allegedly include the btrfs module | 20:24 |
| lunaphyte | ah, ok, one moment | 20:24 |
| TJ- | lunaphyte: and more importantly: "find /lib/modules -name 'btrfs*' | 20:24 |
| lunaphyte | aha, it seems the module may not be loaded | 20:26 |
| lunaphyte | it does exist though | 20:26 |
| lunaphyte | btrfs.ko | 20:26 |
| TJ- | lunaphyte: "modprobe btrfs" | 20:27 |
| TJ- | lunaphyte: that ought to be loaded by udev rules based on identifying the metadata, so there may be some issue with udev rules, or 'strange' metadata | 20:27 |
| lunaphyte | hmm, ok | 20:28 |
| lunaphyte | well, modprobe didn't complain, but it's still not appearing in /proc/modules | 20:28 |
| sarnold | dmesg | tail ? | 20:28 |
| TJ- | lunaphyte: corruption? :p | 20:29 |
| TJ- | sarnold: the kernel stack-trace makes me suspicious; get_fs_type() should not be causing that | 20:31 |
| sarnold | stack trace? I missed that part :) | 20:31 |
| sarnold | ewwwwww | 20:32 |
| lunaphyte | it's hard to tell where dmesg stops and starts between boot and modprobe | 20:32 |
| TJ- | sarnold: https://9unvsq.bn.files.1drv.com/y4m6YIxUZfXzd96cuCzcKJR22rzSeBCy-2VUrndZk5gLlOyk52M6w8oZgr-ybtLbnVvC1trcidcR8L1ndgSUzH9NSysNDl1jTUD5_wBIPQ_PwSMIrWNa69rqvVv_MgV3o2czkQebQUj27idfawPbDN5jaY40rOW5swWLpKXpiu3yMgcPKj_PP6GROCAvAJpnpn3maXS_5GPsrkocfuEEbbHQQ/residue_2019_01_24.PNG?psid=1 | 20:33 |
| lunaphyte | https://pasteboard.co/HYJNunc.png | 20:33 |
| TJ- | "request-module fs-btrfs succeeded, but still no fs?" | 20:34 |
| sarnold | funny, that message actually means "we tried to look up the fs, that failed, so we loaded the module, and that succeeded" | 20:38 |
| sarnold | is there a way to try to force btrfs to load *earlier* ? | 20:38 |
| TJ- | thing is, why isn't it loading manually | 20:41 |
| TJ- | I wonder if there are missing depends? "depends: libcrc32c,zstd_compress,raid6_pq,xor" | 20:42 |
| lunaphyte | where can i look? | 20:49 |
| lunaphyte | kernel module dependencies? | 20:49 |
| TJ- | lunaphyte: it needs the "modinfo" tool | 20:50 |
| lunaphyte | busybox needs modinfo? | 20:50 |
| TJ- | lunaphyte: no, checking the dependencies needs it | 20:51 |
| lunaphyte | oh | 20:51 |
| TJ- | lunaphyte: if the module doesn't show as loaded in /proc/modules then something is obviously going wrong | 20:52 |
| lunaphyte | yeah, makes sense | 20:53 |
| TJ- | lunaphyte: "grep btrfs /proc/modules" would show something like this: "btrfs 1155072 0 - Live 0x0000000000000000" | 20:53 |
| sdeziel | what if you modprobe all the dependencies manually? | 20:54 |
| TJ- | They're not there else modprobe would add them, that's the weird part here | 20:54 |
| TJ- | I see 5 lines with 'btrfs' in them with that grep (showing the modules btrfs depends on also loaded) | 20:56 |
| lunaphyte | TJ-: looking at the server before it's broken, it looks like btrfs has two dependencies: xor and raid6_pq | 21:08 |
| lunaphyte | i've loaded those two modules manually in initramfs, and that went ok | 21:08 |
| lunaphyte | but btrfs still doesn't load | 21:08 |
| TJ- | lunaphyte: check the hash/checksum of the module in case it is corrupt - compare against one known to load (from the same kernel version of course!) | 21:09 |
| lunaphyte | ok | 21:09 |
| lunaphyte | i just verified that nothing is appended to dmesg when doing modprobe btrfs | 21:09 |
| lunaphyte | i see messages from the raid6 module, and nothing more after that | 21:10 |
| Blueking | NIC with intel chip X550 are supported ? | 21:10 |
| TJ- | lunaphyte: rtry "grep btrfs /proc/filesystems" | 21:11 |
| sarnold | Blueking: probably, I see loads of X550 hits in the ixgbe bit of the source tree, and I've got an ixgbe.ko kernel module | 21:11 |
| Blueking | sarnold: guess I am going with Intel NIC | 21:12 |
| lunaphyte | TJ-: btrfs not present | 21:12 |
| TJ- | lunaphyte: so something is causing it to fail, silently too. Is there a blacklist!? | 21:12 |
| Blueking | but what does word 'converged' say something about NIC product ? | 21:12 |
| blackflow | lunaphyte: theory: btrfs initramfs scripts aren't expecting btrfs atop of lvm? perhaps you could try force a /etc/initramfs-tools/scripts/local-top/ script that modprobes the modules? | 21:13 |
| TJ- | lunaphyte: try "grep -rn btrfs /etc/modprobe.d" | 21:13 |
| TJ- | lunaphyte: the udev rule /lib/udev/rules.d/64-btrfs-dm.rules should trigger "ENV{DM_NAME}=="?*", RUN{builtin}+="btrfs ready /dev/mapper/$env{DM_NAME}" " | 21:16 |
| TJ- | lunaphyte: does the 'btrfs' tool exist ? | 21:16 |
| lunaphyte | i'll check shortly - just indisposed for a moment | 21:22 |
| TJ- | I've done some tests on a loopdev here and as soon as the btrfs image is connected the kernel loads the module | 21:23 |
| TJ- | I've now done the same for your scenario /dev/loop5 -> PV > vg1 > root > btrfs and as soon as the image is attached to the loopdev udev causes the module to load and the FS is available | 21:29 |
| lunaphyte | TJ-: !!!! | 21:43 |
| lunaphyte | btrfs was blacklisted! | 21:43 |
| lordcirth_ | lunaphyte, wat? why? | 21:56 |
| TJ- | pffft, sysadmins! :p | 21:56 |
| blackflow | Sounds legit. ZFS for president! *runs* | 21:59 |
| lunaphyte | yeah, it's a little bit odd, sort of | 23:12 |
| lunaphyte | i'm the one who had blacklisted it | 23:12 |
| lunaphyte | this is a template system, used to create other gusts from | 23:13 |
| lunaphyte | *guests | 23:13 |
| lunaphyte | one of the things i'd done was blacklist a bunch of various modules that served no purpose | 23:14 |
| lunaphyte | a previous incarnation of this system used ext4, not btrfs and there was no need for btrfs, so it got blacklisted | 23:14 |
| lunaphyte | then the new incarnation was built, and the blacklist file copied, not realizing that was still there | 23:15 |
| lunaphyte | so that all makes sense | 23:15 |
| lunaphyte | what i don't quite understand though is how the system was even functioning before the upgrade | 23:15 |
| lunaphyte | if the btrfs module was blacklisted, how was the system booting, and how was the module being loaded? | 23:16 |
| lunaphyte | anyway, that's very much for the help, as always :) | 23:17 |
| lunaphyte | *thanks | 23:17 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
