[04:28] <lotuspsychje> good morning
[04:36] <pizzaiolo> o/
[04:36] <lotuspsychje> hey pizza, what cooking :p
[04:37] <pizzaiolo> in about 18 hours, pizza! \o/
[04:37] <lotuspsychje> hehe
[08:07] <lotuspsychje> todays update borked system completly, something libc related
[08:11] <lotuspsychje> and current dailys give me mdsum errors, cant work around to setup or live
[08:13] <lotuspsychje> falling back to bionic temp
[10:01] <TR2990WX> will ubuntu 2020.04 support zfs out of the box?
[10:02] <takov751> yes indeed. As it seems
[12:06] <bkurt78> I just hit an issue running an apt upgrade on my 20.04 install.  https://paste.ubuntu.com/p/B3d3Y9hfsz/
[12:07] <lotuspsychje> bkurt78, yeah be careful with that, i had a borked system this morning after updates
[12:08] <bkurt78> lotuspsychje:  yeah it doesn't look good.
[12:08] <bkurt78> not sure how to recover from this one
[12:09] <lotuspsychje> bkurt78, after reboot, i couldnt enter my system anymore, systemd gave me error on recoverymode too
[12:09] <lotuspsychje> and cant reinstall a daily hence another mdsum error neither..
[12:13] <lotuspsychje> bkurt78, so you might wanna backup stuff before rebooting
[12:19] <tomreyn> you could download and dpkg -i https://packages.ubuntu.com/focal/libcrypt1
[12:22] <lotuspsychje> bkurt78, think at updates this morning, system also asked to start/stop systemd service, doublecheck if you get that too
[12:25] <lotuspsychje> so i tryed to clean install daily 20.04 desktop, but at boot/F1 it floods fsck/mdsum errors and a whole bunch of nvidia lines and does not continue towards setup
[12:27] <RikMills> https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1866844
[12:27] <lotuspsychje> ah tnx RikMills 
[15:38] <bkurt78> My 20.04 install got borked this morning with a failed update and now won't boot back up properly.  It looked like a glibc update that didn't succeed.  Anyone else hit this and know of a fix?
[15:38] <lotuspsychje> bkurt78: RikMills pointed us to this bug ID already, did you miss it?
[15:39] <bkurt78> Yeah, I missed it.  I was actually on the laptop with the issue at the time.  I had to walk away and when it came back it had gone to sleep and was toast from there.
[15:40] <bkurt78> Do you happen to still have the link?
[15:40] <lotuspsychje> sure thing holdon
[15:41] <bkurt78> I tried searching through launchpad but I couldn't find the bug report in there so I must have not been searching with the right info.
[15:42] <lotuspsychje> bkurt78: i reinstalled daily and it seems to work again now
[15:43] <bkurt78> I was hoping to not have to do that, but not a big deal I guess.
[15:44] <MikeRL> Having a very similar issue now on 20.04 prerelease I had on eoan with updates. I will leave a paste. Basically, libc6 errors. Specifically, "libc6:amd64 tries to upgrade and gets "/usr/bin/perl: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
[15:44] <MikeRL> "
[15:44] <bkurt78> lotuspsychje: you recall if there was a fix, or was the solution to just reload at that point?
[15:45] <lotuspsychje> bkurt78: bug #1866844
[15:45] <MikeRL> Should've used a paste but figured it was short enough. Ironically I upgraded to 20.04 early thinking I could avert that package issue.
[15:46] <MikeRL> I'll vote on the bug. That package was broken on eoan, but with a different effect.
[15:46] <bkurt78> lotuspsychje: got it.  Thank you!
[15:47] <bkurt78> I think I'll boot off some recovery media and try the repair steps and if that doesn't work then just reload
[15:48] <lotuspsychje> bkurt78: i got 2.31-0ubuntu5 libc6 now
[15:56] <bkurt78> I just did the repair steps in the bug report and they worked perfectly for me.  Back up and running.  Thank you all for the help.
[15:57] <lotuspsychje> great bkurt78 
[16:41] <murphnj> lordcirth_: As requested.....
[16:43] <murphnj> lordcirth_: Just for background, I have the ability to get all the files off of it, and it was there for goofing around and testing stuff, so I'm not recovering critical, irreplaceable data or anything.  I'd just rather learn how to fix it than wash and reload too quickly.
[16:44] <murphnj> I don't want to mess up the Fedora on the other drive though, that one is a little more inportant. 
[16:45] <lordcirth_> murphnj, I would look in auth.log first, if you can't login 
[16:47] <tomreyn> recovery may still work, since it doesn't usually need to prompt for a password
[16:50] <murphnj> Auth failure, but more interesting:  PAm unable to dlopen /lib/securoty/pam)unix.so No such file or directory.
[16:50] <murphnj> Sadly, can't cut and paste, need auth to log in, and I don't want to break everything on this machine to log in from there.  
[16:59] <lotuspsychje> for those who want to reproduce, i filed bug #1867445
[17:08] <murphnj> lordcirth_: I'm heading to the outside world for a few, will return to diagnose if you're still available.
[18:27] <lotuspsychje> welcome jkwiatko 
[18:27] <jkwiatko> hello
[18:28] <lotuspsychje> jkwiatko: you are facing bug #1866844
[18:29] <lotuspsychje> and bug #1866844
[18:31] <jkwiatko> ah thank you will read through these now!
[18:42] <murphnj> lordcirth_: I'm back.  Any ideas?
[18:43] <murphnj> Interesting, I don't seem to have a /lib/security folder at all.
[18:43] <murphnj> That seems pretty bad.
[18:46] <murphnj> I'm going to take the step of backing up everything from that machine to a disk.
[19:00] <jkwiatko> Thanks lotuspsychje, that did solve my issue!
[19:56] <murphnj> Anyone else able to help out a bit?  Login fails without asking for password.  auth.log gives an error like  'PAM unable to open /lib/security/pam_unix.so' and indeed there is NO directory /lib/security.  Worked earlier, ran an update, and there was an error, but I ran a suggested "fix-broken" sort of thing from apt.  The failed packages looked simiar to what ubottu reported at 14:28. (If I recall 
[19:56] <murphnj> correctly)
[20:00] <lordcirth_> murphnj, I'm not sure what's up
[20:01] <murphnj> The fact that /lib/security is gone is strange, seems like the failed upgrade removed something, but didn't replace it.
[20:02] <lordcirth_> Here on 18.04, I don't have a /lib/security, though
[20:02] <murphnj> Luckily, this was there to try new things.  I guess I'll give it a little more elbow grease, then I may be trying re-installing. :)
[20:02] <lordcirth_> But apt-file search says: virtualbox-guest-utils: /lib/security/pam_vbox.so
[20:03] <murphnj> It looks like PAM is looking for it.  I do, however have Virtualbox installed on it.
[20:03] <murphnj> lordcirth_: I don't ask questions that are easy most of the time.  I only bring the tricky ones.
[20:04] <lordcirth_> murphnj, does "apt-file search /lib/security/pam_unix.so" find anything on your system?
[20:05] <murphnj> I can't boot into ubuntu at all.  My view into the filesystem and logs is through Fedora on another drive.
[20:05] <lordcirth_> murphnj, You can chroot in, though
[20:06] <murphnj> I can try
[20:07] <murphnj> If I remove all of the password from /etc/shadow for that account, will the account log in without password?
[20:07] <lordcirth_> No, it will be locked, I think
[20:07] <murphnj> Where should I chroot from?  cd to the root of that drive then chroot . ?
[20:08] <lordcirth_> however, if you boot into recovery, you can normally get a root shell without a login
[20:08] <lordcirth_> Your working directory doesn't matter, just chroot to the root of the drive
[20:11] <murphnj> apt-file fails
[20:11] <murphnj> After the chroot
[20:14] <lordcirth_> fails how?
[20:29] <murphnj> I don't know if it is installed, the message is from "command-not-found"
[20:29] <murphnj> It seems apt
[20:29] <murphnj> sees apt
[20:33] <Bashing-om> !info apt-file | murphnj 
[20:36] <murphnj> I tried to install apt-file, and got the following amongst the errors.
[20:37] <murphnj> libc-bin : Depends : libc6 <2.31) but 2.31-0ubunru5 is to be installed
[20:38] <murphnj> A few others like this, ending with Unmet Dependencies. try apt --fix-broken install 
[20:38] <murphnj> Which sounds a lot like what I got on the update I ran, and then did what they suggested (the --fix....)
[20:38] <murphnj> Right before I got here.
[20:41] <murphnj> It looks like apt-file is not installed by default
[20:41] <Bashing-om> !info libc6
[20:41] <murphnj> I also don't have a /lib/security on my other Ubuntu machine.
[20:43] <murphnj> So I guess that is not the problem, despite the error message.
[20:45] <Bashing-om> murphnj: ' apt policy libc-bin ' ??
[20:48] <murphnj> Installed: 2-30-0ubuntu3
[20:48] <murphnj> Candidate: 2.31-0ubuntu5
[20:49] <murphnj> Are you looking for more?  (I have to type the results across)
[20:52] <Bashing-om> murphnj: Kinda leary to mess about here as " apt show libc-bin " indicates exercise extreme caution.
[20:53] <murphnj> I have little to lose.  Data is backed up, machine is not supporting anything important. :)
[20:53] <murphnj> If it were important, I would have backed up more thoroughly, if it was important ot get running right away, I would have washed and reloaded alreasy.
[20:57] <Bashing-om> murphnj: One can take a poke at it ' sudo apt install libc6 '. see what the package manager screams about.
[20:58] <Bashing-om> murphnj: Maybe better as ' sudo apt install --reintall libc6 ' ?
[20:58] <Bashing-om> --reinstall**
[21:01] <murphnj> it failed, libc-bin, libc-dev-bin, libc6-dbg, and libc6-dev all want libc6 2.30, but 2.31 is to be installed. 
[21:02] <murphnj> It looks like my accepting of the --fix-broken-packages move (suggested by the apt update) really did things in.
[21:05] <Bashing-om> murphnj: "2.31-0ubuntu5" is the current version that should be installed . as to what is holding the system to that old 2.30 version, remains to be found.
[21:08] <murphnj> hmmmm.  I just installed (to contribute to a team) the folding@home client
[21:08] <murphnj> I wonder if that could be it.
[21:09] <Bashing-om> murphnj: See what we can find. ' sudo apt update 2>&1 | nc termbin.com 9999 ; sudo apt upgrade 2>&1 | nc termbin.com 9999 ' pass the resulting links back here for us to looksee.
[21:12] <murphnj> apt is failing to get out on the network in the chroot.
[21:13] <TJ-> murphnj: use "5.39.93.71" instead of "termbin.com" 
[21:16] <murphnj> First, I want to see the file I'd be putting out there first, so I tried the command piped to a file.  It's not termbin, it's apt that is failing.
[21:16] <murphnj> I can ping by IP, so I could send it, but it's all apt failing name resolution.
[21:17] <TJ-> murphnj: ahh, so from *outside* the chroot do "sudo mount --bind /etc/resolv.conf /$CHROOT/etc/resolv.conf" -- setting $CHROOT appropriately
[21:18] <Bashing-om> murphnj: TJ- is very very sharp - follow his leads.
[21:23] <murphnj> https://termbin.com/8g46
[21:24] <murphnj> Thanks for all the help, by the way.
[21:24] <murphnj> I missed a '9' at the end of the upgrade one, try ...1j5q
[21:27] <TJ-> murphnj: I can see the problem there, or at least a reason. There's a series of tests that are expected to succeed and aren't doing. We can try to break it down to locate the failing test
[21:30] <TJ-> murphnj: oh! so I was looking at the wrong pastebin 
[21:30] <murphnj> TJ-: Not wrong, just half of the picture.
[21:31] <TJ-> murphnj: ahh
[21:31] <TJ-> murphnj: what version does "apt list libc6-bin" report as installed ?
[21:32] <Bashing-om> TJ-: murphnj may I suggest that libc6-dbg and libc6-dev be removed ?
[21:34] <TJ-> looking at the changelog seems like there's been some churn this week that has caused this https://changelogs.ubuntu.com/changelogs/pool/main/g/glibc/glibc_2.31-0ubuntu5/changelog
[21:36] <TJ-> murphnj: oh I see the problem, doh! You're in a chroot and haven't mounted /dev and possibly other required kernel file-systems
[21:36] <TJ-> murphnj: the error about unable to create /dev/null gives it away
[21:38] <Bashing-om> TJ-: ^^ there is a reason why I look over your shoulder: " create /dev/null" had me wondering !
[21:38] <TJ-> murphnj: when setting up a chroot of a full system install you need to do e.g. " TARGET=/target; for n in proc sys dev etc/resolv.conf; do sudo mount --rbind /$n $TARGET/$n; done; chroot $TARGET mount -a; chroot $TARGET "
[21:39] <TJ-> sometimes even need to add the /run/ directory too
[21:41] <murphnj> I'll try it.  
[21:41] <murphnj> Sorry, I've been jumping between this and the Libreplanet talks.
[21:42] <murphnj> So in the above command is it as-is, or is TARGET and target supposed to be substituted with something.  I'm not clear on that.
[21:43] <murphnj> I haven't had to do this on the regular, so I'm learning new things.
[21:43] <TJ-> murphnj: you set TARGET=/whereever/you/want/to/mount/the/rootfs 
[21:43] <TJ-> murphnj: I use /target because that is what the Debian/Ubuntu installer uses by default
[21:43] <TJ-> murphnj: of course that directory must exist/be created
[22:09] <murphnj> Ok, I'm already in a chroot, so I have to crete a new location, 
[22:09] <murphnj> create a directory, and go from there?
[22:13] <TJ-> murphnj: exit the chroot, remove any mounts you've made inside it, and start again
[22:14] <murphnj> I got a lot of fails - 
[22:14] <murphnj> Let me recheck what I typed.
[22:16] <murphnj> So I also have to create all of the mount points under target?
[22:20] <murphnj> I did that and took out /etc/resolv.conf, (becuase it failed)
[22:20] <TJ-> murphnj: no, do not have to create mountpoints
[22:21] <murphnj> I am getting "chroot failed to run command ' mount': no such file or dir  (and then the same fior /bin/bash)
[22:21] <TJ-> let's say your root-fs you want to work on is on /dev/sdz9
[22:22] <TJ-> so, you'd do "sudo mkdir /target; sudo mount /dev/sdz9 /target; for n in proc sys dev etc/resolv.conf; do sudo mount --rbind /$n /target/$n; done; sudo chroot /target mount -a; sudo chroot /target" --- now you're in the correctly mounted target as 'root' and can work
[22:22] <murphnj> I misunderstood
[22:22] <murphnj> I have to go in a few minutes, let me give this one more shot.
[22:23] <murphnj> Ok seems to have worked.
[22:24] <TJ-> murphnj: now you might be able to get "apt -f install" to solve things
[22:25] <murphnj> It is giving many scary warnings.
[22:25] <murphnj> But I fear not.
[22:25] <TJ-> murphnj: capture them :)
[22:25] <murphnj> Halted, too many errors.
[22:26] <TJ-> murphnj: you can do things like "apt -f install &| tee /tmp/apt.log" then when it stops, "pastebinit /tmp/apt.log" and give us the URL
[22:27] <murphnj> DO you mean for that & before the |
[22:28] <TJ-> Yes - it causes both stdout and sterr to be piped to 'tee' so you capture both
[22:29] <murphnj> Certificate error with pastebinit
[22:30] <murphnj> It gave an unspecified token error.
[22:30] <TJ-> sounds weird; OK do "cat /tmp/apt.log | nc termbin.com 9999 "
[22:31] <murphnj> nc command not found
[22:31] <TJ-> blimey, that root-fs you're chroot-ed into seems rather badly broken!
[22:31] <murphnj> I may have to leave, and try to pick this up later.
[22:32] <murphnj> Ubuntu's update (and subsequent suggestion to resolve it) really did me in.
[22:32] <murphnj> I'm going to be sure to keep better backups, luckily I was able to mount the partition and get everything off, and wasn't really counting on it.
[22:33] <murphnj> fdy2
[22:33] <murphnj> It was nc.openbsd
[22:33] <murphnj> There must be an alias missing somewhere
[22:34] <TJ-> yes, there should be a symlink via /etc/alternatives/ from /usr/bin/nc to it
[22:34] <TJ-> readlink -e /usr/bin/nc
[22:34] <TJ-> /usr/bin/nc.openbsd
[22:35] <murphnj> I have to leave.  I'll leave the window open, thanks for the help, if you're around later, I'll try to ping you.
[22:36] <TJ-> it's 22:30 here so someone else will likely be around, but I'll be asleep!
[22:38] <murphnj> TJ-: Thank you for all of your help.  I doubt someone is going to be able to pick this up fom the middle.
[22:38] <TJ-> well, you're in a stable chroot now which'll help. Let them know that :)
[22:38] <TJ-> grabbing that apt.log will help enormously
[22:38] <murphnj> We'll see.
[22:38] <murphnj> Thanks.