[06:37] <lordievader> Good morning
=== Guest99098 is now known as lordievader
[13:36] <coreycb> icey: there's a mysql8 + alembic issue with barbican we'll need to look into: https://bugs.launchpad.net/ubuntu/+source/barbican/+bug/1899104
[13:36] <ubottu> Launchpad bug 1899104 in barbican (Ubuntu) "barbican-manage db upgrade fails with MySQL8" [Undecided,New]
[13:37] <icey> I saw that come in this morning
[13:37] <icey> coreycb:
[13:39]  * RoyK wonders why people use mysql these days
[13:46] <coreycb> icey: it looks like that may be fixed upstream https://storyboard.openstack.org/#!/story/2007732
[13:46] <coreycb> icey: since you worked on that for groovy do you want to make sure it's fixed in groovy and handle the SRU to focal?
[13:48] <icey> sure
[16:44] <nopnotme> I have an issue when running any *apt* commands. I was originally trying to install nodejs on a Ubuntu 16.04 server that I only use via ssh, however I am getting an error using apt now regarding grub-common versions: https://pastebin.com/UwFi9fdF
[17:09] <teward> nopnotme: is this on an Amazon EC2 instance?
[17:10] <nopnotme> negative
[17:11] <teward> i only asked because the package that's installed (grub-legacy-ec2) is for EC2 instances
[17:13] <nopnotme> what it be in my interest to purge it?
[17:20] <nopnotme> even after purging i get the same error
[17:26] <mgedmin> by "it" do you mean grub-legacy-ec2 or grub2-common?
[17:27] <nopnotme> grub-legacy-ex2
[17:27] <pgnd> Running regular maintenance today on an otherwise stable Ubu bionic server, 'apt update' FAILs with: "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc9 in position 4: invalid continuation byteError in sys.excepthook:"
[17:27] <pgnd> detail -> https://pastebin.com/raw/PVegTn92
[17:27] <pgnd> What do we to workaround/fix that^ in the context of apt updates?
[17:29] <pgnd> s/we to/we do in order to/
[17:41] <mgedmin> nopnotme: how can you have a conflict with grub-legacy-ec2 if grub-legacy-ec2 is no longer installed?  pastebin the full error please
[17:46] <mgedmin> pgnd: I would first report a bug (ubuntu-bug command-not-found)
[17:47] <mgedmin> pgnd: hmm, what happens if you run `python3 -c 'import apt_pkg'` on that server?
[17:47] <mgedmin> maybe this needs to be investigated a bit to rule out local damange before assuming it's a bug
[17:49] <pgnd> mgedmin: hi.  -> https://pastebin.com/raw/pddKjvbc
[17:50] <pgnd> something's clearly not happy ...
[17:50] <mgedmin> pgnd: very interesting, I've never seen it before
[17:51] <mgedmin> pgnd: what's the output of `locale charmap`?
[17:51] <pgnd> mgedmin: "UTF-8"
[17:52] <mgedmin> I'd next try `LC_ALL=C python3 -c 'import apt_pkg'` and `LC_ALL=C.UTF-8 python3 -c 'import apk_pkg'`, to see if the locale has anything to do with it
[17:53] <mgedmin> next idea: do you have debsums installed?  check with `debsums python3-apt` if the package got corrupted somehow?
[17:55] <pgnd> mgedmin: -> https://pastebin.com/raw/zF7rFkKm
[17:56] <mgedmin> well now!
[17:56] <mgedmin>  /usr/lib/python3/dist-packages/apt_pkg.cpython-36m-x86_64-linux-gnu.so    FAILED
[17:56] <pgnd> yeah
[17:56] <mgedmin> make a copy of that file, then apt install --reinstall python3-apt
[17:56] <mgedmin> when apt is done you may want to cmp the file with the bad copy
[17:56] <mgedmin> see if maybe it differs in one bit only?  could indicate hardware problems
[17:57]  * mgedmin can stop browsing https://bugs.launchpad.net/ubuntu/+source/python-apt for similar problems
[17:58] <mgedmin> I would probably run memtest86 and/or hard disk SMART self-tests with smartctl from smartmontools
[17:58] <pgnd> mgedmin: the reinstall did the trick. thx!
[17:58] <pgnd> cmp returns:
[17:59] <pgnd> smart tests run nightly.  all's good ... will rerun here shortly, just in case
[17:59] <mgedmin> cmp returns no output?
[18:00] <pgnd> huh?  oh ... ry leading  "/"  :-(
[18:00] <pgnd> -> /usr/lib/python3/dist-packages/apt_pkg.cpython-36m-x86_64-linux-gnu.so /usr/lib/python3/dist-packages/apt_pkg.cpython-36m-x86_64-linux-gnu.so.ORIG differ: byte 258055, line 823
[18:01] <mgedmin> ah, cmp -l will show the values of the differing bytes, and won't stop at the first difference
[18:03] <pgnd> mgedmin: https://pastebin.com/jZCM7dUX (somebody, aka 'me', needs to read up on cmp!)
[18:06] <pgnd> mgedmin: in any case, mny thx!
[18:09] <mgedmin> well that's a lot
[18:09] <mgedmin> cmp -l -p will also show ascii values, not sure if that's going to be helpful
[18:10] <mgedmin> converting file offsets from decimal to hex shows the differences are in the range of 0x3f007 to 0x3ffff
[18:10] <mgedmin> that upper value is very precisely aligned
[18:11] <mgedmin> it's one corrupted 4K block
[18:11] <mgedmin> exactly one memory page
[18:12] <mgedmin> oh!  pgnd: in your place I'd run `debsums -c`, to check _all_ installed packages for corruption
[18:13] <pgnd> mgedmin: good point!  doing that now
[18:17] <pgnd> mgedmin: only one, "debsums: Error: symlink loop detected in path 'usr/share/doc/module-init-tools/changelog.Debian.gz'. Please file a bug again module-init-tools."
[18:18] <mgedmin> heh
[18:18] <mgedmin> huh, on my bionic server /usr/share/doc/module-init-tools/changelog.Debian.gz is a regular file and not a symlink
[18:19] <mgedmin> and no component in that path is a symlink
[18:19] <pgnd> re-run of smart tests ... all good here
[18:19] <mgedmin> memtest time
[18:19] <mgedmin> oh, one more trick
[18:19] <mgedmin> you can ask the kernel to drop the page caches, then compare the two files again
[18:19] <mgedmin> they'll be read from disk afresh
[18:20] <mgedmin> if they're identical, then you definitely have a memory problem
[18:20] <pgnd> mgedmin: this server has been upgraded since forever ... not a clean install.  i've seen similar links vs files issues, elsewhere, in the past
[18:20] <mgedmin> if they're still different, the wrong block got written out
[18:20] <mgedmin> you drop caches by doing echo 3 |sudo tee /proc/sys/vm/drop_caches
[18:20] <mgedmin> the server will be slow for a few minutes after that, while it reads everything back from disk
[18:21] <pgnd> mgedmin: re-cmp'd after the drop cache. similarly long output ...
[18:23] <pgnd> good news is, currently no more apt* problems that I can see.
[18:34] <pgnd> need to find Memtest86+ equivalent for UEFI: "Memtest86+ only works on systems with BIOS firmware. Support for UEFI systems is currently unavailable."
[18:36] <mgedmin> there's also apt install memtester
[18:36] <mgedmin> it runs inside linux, so it can't test _all_ memory, only the parts that are not in use
[18:36] <mgedmin> but it's better than nothing
[18:38] <pgnd> thx
[18:40] <ueberall> pgnd: What about https://www.memtest86.com/ ?
[18:42] <pgnd> ueberall: Didn't realize there was a different creature ... had just checked the pkg repos, which offer memtest86+ & memtester. thx, will take a look.
=== genii is now known as genii-o-lantern
=== genii-o-lantern is now known as genii
=== arooni_team_b is now known as arooni