[04:37] <lunaphyte> i've just upgraded from 17.10 to 18.04, and now the computer won't boot.  it fails to mount the root filesystem, and drops to initramfs
[04:39] <lunaphyte> https://9unvsq.bn.files.1drv.com/y4m6YIxUZfXzd96cuCzcKJR22rzSeBCy-2VUrndZk5gLlOyk52M6w8oZgr-ybtLbnVvC1trcidcR8L1ndgSUzH9NSysNDl1jTUD5_wBIPQ_PwSMIrWNa69rqvVv_MgV3o2czkQebQUj27idfawPbDN5jaY40rOW5swWLpKXpiu3yMgcPKj_PP6GROCAvAJpnpn3maXS_5GPsrkocfuEEbbHQQ/residue_2019_01_24.PNG?psid=1
[04:39] <lunaphyte> ugh, sorry about the awful url
[04:40] <lunaphyte> the upgrade process went without any issues or complaints at all
[04:40] <lunaphyte> what can i do to troubleshoot this?
[04:45] <lunaphyte> since i'd had a similar problem with another upgrade not going well, with non root filesystems that were not accessible, i tried vgchange -aay, on a whim.  so far, it seems to just hang upon that command indefinitely, and not do anything
[07:10] <lordievader> Good morning
[07:11] <lordievader> lunaphyte: From that initramfs can you mount your rootfs?
=== cpaelzer__ is now known as cpaelzer
[09:43] <jamespage> cpaelzer: looking at your dpdk/ovs work today
[09:54] <cpaelzer> jamespage: ok, I started to test it as well
[09:54] <cpaelzer> but that takes quite some time
[09:54] <cpaelzer> as I said, I'll ping you once I have a reasonable result
[09:59] <jamespage> cpaelzer: ack
[11:03] <ahasenack> good morning
[14:06] <rbasak> ahasenack, cpaelzer, kstenerud: I can hit the review queue now, but it looks like everything has been looked at for the moment?
[14:07] <ahasenack> rbasak: agreed
[14:07] <ahasenack> I'm finishing up haproxy, then I'll go after apache2 which you reviewed already, and back to the net-snmp merge
[14:07] <rbasak> OK
[14:26] <ahasenack> rbasak: can you merge that salsa haproxy mp, or are you just passing by?
[14:29] <rbasak> ahasenack: I was just passing by, being curious about how haproxy tests operate as I know nothing about haproxy.
[14:29] <ahasenack> cool
[14:29] <ahasenack> first attempt failed, investigating
[14:29] <rbasak> I am not a maintainer for haproxy in Debian so I shouldn't do anything but comment.
[14:29] <ahasenack> https://pastebin.ubuntu.com/p/CvVKbJC2XV/
[14:31] <ahasenack> ah, I have to invert the check
[14:31] <rbasak> Yeah, or swap the else block.
[14:34] <rbasak> (I prefer the latter in general to avoid an extra inversion)
[14:36] <ahasenack> yeah, I also dislike "!" in ifs
[14:40] <ahasenack> rbasak: better, let me push: https://pastebin.ubuntu.com/p/WDtmWnZRCZ/
[15:03] <MJCD> i'm trying to set up xorg+lightdm+fluxbox on a server build
[15:03] <MJCD> i've gotten a graphical login screen after reboot
[15:04] <MJCD> but it just fails to start session
[15:06] <MJCD> lol if I switch from fluxbox to openbox
[15:07] <MJCD> it lets me log in, but to just black nothing
[15:07] <MJCD> no right click menu even
[15:07] <MJCD> >.<
[15:10] <lordcirth_> MJCD, anything in logs?
[15:10] <MJCD> its ok third reboot was the charm for whatever reason, fluxbox still doesn't work
[15:10] <MJCD> but i'm fine with openbox for now
[17:31] <ahasenack> rbasak: can I consider our salsa exchange as another +1 for the haproxy dep8 tests, provided tests pass in ubuntu? For this MP: https://code.launchpad.net/~ahasenack/ubuntu/+source/haproxy/+git/haproxy/+merge/362217
[17:31] <ahasenack> cpaelzer is eod now
[17:31] <ahasenack> I induced some errors to see how it would behave, and it looks good:
[17:32] <ahasenack> - diff content: https://pastebin.com/Jcn8TE4A
[17:33] <ahasenack> - diff size: https://pastebin.com/4y4crD39
[17:57] <rbasak> ahasenack: yes, and +1 added to the MP
[17:58] <ahasenack> rbasak: thanks
=== Tornevall is now known as Guest36728
[20:05] <lunaphyte> lordievader: thanks for responding.  i can't, no.
[20:07] <lunaphyte> if i try to mount the root filesystem manually at the initramfs prompt, it says the same thing - "mount: mounting /dev/mapper/vg1-root on /root failed: no such device"
[20:07] <lunaphyte> all of the lvm pieces are active and functioning, i can see all of the physical volumes, all of the volume groups, all of the logical volumes
[20:08] <lunaphyte> i can run btrfs check on /dev/mapper/vg1-root and it runs properly and successfully, and finds no errors
[20:08] <TJ-> lunaphyte: broke it again?!
[20:09] <lunaphyte> TJ-: :) - hah - no, this is a different one
[20:10] <lunaphyte> https://9unvsq.bn.files.1drv.com/y4m6YIxUZfXzd96cuCzcKJR22rzSeBCy-2VUrndZk5gLlOyk52M6w8oZgr-ybtLbnVvC1trcidcR8L1ndgSUzH9NSysNDl1jTUD5_wBIPQ_PwSMIrWNa69rqvVv_MgV3o2czkQebQUj27idfawPbDN5jaY40rOW5swWLpKXpiu3yMgcPKj_PP6GROCAvAJpnpn3maXS_5GPsrkocfuEEbbHQQ/residue_2019_01_24.PNG?psid=1
[20:10] <TJ-> lunaphyte: compare the dm-X number shown by "ls -l /dev/mapper/vg1-root" with "ls -l /dev/dm-X" to ensure it exists
[20:11] <TJ-> lunaphyte: oh, a VM image? is the underlying file corrupt. You shouldn't get the kernel stracktrace
[20:12] <lunaphyte> TJ-: i don't think anything is corrupt.  i can boot off the installer, in rescue mode, and successfully mount all filesystem and chroot into them as the root
[20:12] <lunaphyte> yeah, it's a vmware guest
[20:13] <TJ-> lunaphyte: is this always a VMware guest?
[20:13] <lunaphyte> yes
[20:13] <lunaphyte> it always has been, from its inception, and always will be
[20:13] <TJ-> lunaphyte: any changes to the hardware profile ?
[20:14] <lunaphyte> no changes to hardware
[20:14] <lunaphyte> just an upgrade from 17.10 to 18.04, that's it
[20:14] <lunaphyte> no errors at all during the upgrade
[20:14] <lunaphyte> went perfectly typical
[20:14] <TJ-> lunaphyte: that screenshot suggests /root/ doesn't exist
[20:15] <TJ-> lunaphyte: all commands trying to set-up the real file-system on /root/ fail - either that is missing or the real rootfs really has a major problem
[20:15] <lunaphyte>  /root/ does exist
[20:16] <TJ-> lunaphyte: so does the block device symlinked from /dev/mapper/vg1-root exist?
[20:16] <lunaphyte>  /dev/md-0 ?
[20:16] <lunaphyte> yeah, that exists
[20:16] <lunaphyte> i can run fsck successfully on the device
[20:16] <TJ-> lunaphyte: I have no idea, that that is NOT an LVM node
[20:16] <lunaphyte> it properly recognizes the filesystem
[20:16] <TJ-> lunaphyte: as I said earlier: --->  compare the dm-X number shown by "ls -l /dev/mapper/vg1-root" with "ls -l /dev/dm-X" to ensure it exists
[20:17] <lunaphyte> yeah, it does
[20:18] <lunaphyte> bah, sorry
[20:18] <lunaphyte> i missed that irc ignored what i wrote because of the leading slash
[20:18] <lunaphyte>  /dev/mapper/vg1-root points to /dev/dm-0, which exists
[20:19] <TJ-> lunaphyte: the stacktrace shows that get_fs_type() fails, so what does "blkid /dev/dm-0" report?
[20:19] <lunaphyte> https://pasteboard.co/HYJHXiJ.png
[20:19] <lunaphyte> one moment
[20:20] <lunaphyte> https://pasteboard.co/HYJIrFu.png
[20:21] <lunaphyte> sorry i can't just use normal text.  i'm stuck inside a virtual guest console
[20:21] <TJ-> I know :)
[20:22] <TJ-> So, this is really looking like a btrfs problem. is the module available? "lsmod | grep btrfs"
[20:23] <lunaphyte> ugh.  no lsmod in this busybox, it seems
[20:24] <sarnold> /proc/modules ?
[20:24] <TJ-> lunaphyte: "grep btrfs /pro/mocules"
[20:24] <sarnold> it's ugly but still mostly legible
[20:24] <TJ-> modules!
[20:24] <lunaphyte> i did check the initrd earlier with lsinitramfs and it seems to allegedly include the btrfs module
[20:24] <lunaphyte> ah, ok, one moment
[20:24] <TJ-> lunaphyte: and more importantly: "find /lib/modules -name 'btrfs*'
[20:26] <lunaphyte> aha, it seems the module may not be loaded
[20:26] <lunaphyte> it does exist though
[20:26] <lunaphyte> btrfs.ko
[20:27] <TJ-> lunaphyte: "modprobe btrfs"
[20:27] <TJ-> lunaphyte: that ought to be loaded by udev rules based on identifying the metadata, so there may be some issue with udev rules, or 'strange' metadata
[20:28] <lunaphyte> hmm, ok
[20:28] <lunaphyte> well, modprobe didn't complain, but it's still not appearing in /proc/modules
[20:28] <sarnold> dmesg | tail ?
[20:29] <TJ-> lunaphyte: corruption? :p
[20:31] <TJ-> sarnold: the kernel stack-trace makes me suspicious; get_fs_type() should not be causing that
[20:31] <sarnold> stack trace? I missed that part :)
[20:32] <sarnold> ewwwwww
[20:32] <lunaphyte> it's hard to tell where dmesg stops and starts between boot and modprobe
[20:33] <TJ-> sarnold: https://9unvsq.bn.files.1drv.com/y4m6YIxUZfXzd96cuCzcKJR22rzSeBCy-2VUrndZk5gLlOyk52M6w8oZgr-ybtLbnVvC1trcidcR8L1ndgSUzH9NSysNDl1jTUD5_wBIPQ_PwSMIrWNa69rqvVv_MgV3o2czkQebQUj27idfawPbDN5jaY40rOW5swWLpKXpiu3yMgcPKj_PP6GROCAvAJpnpn3maXS_5GPsrkocfuEEbbHQQ/residue_2019_01_24.PNG?psid=1
[20:33] <lunaphyte> https://pasteboard.co/HYJNunc.png
[20:34] <TJ-> "request-module fs-btrfs succeeded, but still no fs?"
[20:38] <sarnold> funny, that message actually means "we tried to look up the fs, that failed, so we loaded the module, and that succeeded"
[20:38] <sarnold> is there a way to try to force btrfs to load *earlier* ?
[20:41] <TJ-> thing is, why isn't it loading manually
[20:42] <TJ-> I wonder if there are missing depends? "depends:        libcrc32c,zstd_compress,raid6_pq,xor"
[20:49] <lunaphyte> where can i look?
[20:49] <lunaphyte> kernel module dependencies?
[20:50] <TJ-> lunaphyte: it needs the "modinfo" tool
[20:50] <lunaphyte> busybox needs modinfo?
[20:51] <TJ-> lunaphyte: no, checking the dependencies needs it
[20:51] <lunaphyte> oh
[20:52] <TJ-> lunaphyte: if the module doesn't show as loaded in /proc/modules then something is obviously going wrong
[20:53] <lunaphyte> yeah, makes sense
[20:53] <TJ-> lunaphyte: "grep btrfs /proc/modules" would show something like this: "btrfs 1155072 0 - Live 0x0000000000000000"
[20:54] <sdeziel> what if you modprobe all the dependencies manually?
[20:54] <TJ-> They're not there else modprobe would add them, that's the weird part here
[20:56] <TJ-> I see 5 lines with 'btrfs' in them with that grep (showing the modules btrfs depends on also loaded)
[21:08] <lunaphyte> TJ-: looking at the server before it's broken, it looks like btrfs has two dependencies: xor and raid6_pq
[21:08] <lunaphyte> i've loaded those two modules manually in initramfs, and that went ok
[21:08] <lunaphyte> but btrfs still doesn't load
[21:09] <TJ-> lunaphyte: check the hash/checksum of the module in case it is corrupt - compare against one known to load (from the same kernel version of course!)
[21:09] <lunaphyte> ok
[21:09] <lunaphyte> i just verified that nothing is appended to dmesg when doing modprobe btrfs
[21:10] <lunaphyte> i see messages from the raid6 module, and nothing more after that
[21:10] <Blueking> NIC with intel chip X550 are supported ?
[21:11] <TJ-> lunaphyte: rtry "grep btrfs /proc/filesystems"
[21:11] <sarnold> Blueking: probably, I see loads of X550 hits in the ixgbe bit of the source tree, and I've got an ixgbe.ko kernel module
[21:12] <Blueking> sarnold: guess I am going with Intel NIC
[21:12] <lunaphyte> TJ-: btrfs not present
[21:12] <TJ-> lunaphyte: so something is causing it to fail, silently too. Is there a blacklist!?
[21:12] <Blueking> but what does word 'converged' say something about NIC product ?
[21:13] <blackflow> lunaphyte: theory: btrfs initramfs scripts aren't expecting btrfs atop of lvm? perhaps you could try force a /etc/initramfs-tools/scripts/local-top/ script that modprobes the modules?
[21:13] <TJ-> lunaphyte: try "grep -rn btrfs /etc/modprobe.d"
[21:16] <TJ-> lunaphyte: the udev rule /lib/udev/rules.d/64-btrfs-dm.rules should trigger "ENV{DM_NAME}=="?*", RUN{builtin}+="btrfs ready /dev/mapper/$env{DM_NAME}"  "
[21:16] <TJ-> lunaphyte: does the 'btrfs' tool exist ?
[21:22] <lunaphyte> i'll check shortly - just indisposed for a moment
[21:23] <TJ-> I've done some tests on a loopdev here and as soon as the btrfs image is connected the kernel loads the module
[21:29] <TJ-> I've now done the same for your scenario /dev/loop5 -> PV > vg1 > root > btrfs  and as soon as the image is attached to the loopdev udev causes the module to load and the FS is available
[21:43] <lunaphyte> TJ-: !!!!
[21:43] <lunaphyte> btrfs was blacklisted!
[21:56] <lordcirth_> lunaphyte, wat? why?
[21:56] <TJ-> pffft, sysadmins! :p
[21:59] <blackflow> Sounds legit.                                ZFS for president!                                      *runs*
[23:12] <lunaphyte> yeah, it's a little bit odd, sort of
[23:12] <lunaphyte> i'm the one who had blacklisted it
[23:13] <lunaphyte> this is a template system, used to create other gusts from
[23:13] <lunaphyte> *guests
[23:14] <lunaphyte> one of the things i'd done was blacklist a bunch of various modules that served no purpose
[23:14] <lunaphyte> a previous incarnation of this system used ext4, not btrfs and there was no need for btrfs, so it got blacklisted
[23:15] <lunaphyte> then the new incarnation was built, and the blacklist file copied, not realizing that was still there
[23:15] <lunaphyte> so that all makes sense
[23:15] <lunaphyte> what i don't quite understand though is how the system was even functioning before the upgrade
[23:16] <lunaphyte> if the btrfs module was blacklisted, how was the system booting, and how was the module being loaded?
[23:17] <lunaphyte> anyway, that's very much for the help, as always :)
[23:17] <lunaphyte> *thanks