[06:19] <knaccc> hey all, I just did an apt upgrade and rebooted, which took me to kernel 5.4.0-122. This caused crashes on the server, so I downgraded by doing "sudo apt remove linux-image-5.4.0-122-generic" and rebooted. It's now working. I was wondering if what I did will now have any adverse consequences for the server which will break things if I do an apt upgrade in the future? Apparently there will be
[06:19] <knaccc> a bugfix for kernel .122 released on 1 August (https://bugs.launchpad.net/ubuntu/+source/linux-hwe-5.4/+bug/1981658) 
[06:20] <knaccc> i am terrified if i do an apt upgrade in the future, there will be some kind of grub issue and i'll brick the server
[06:21] <lotuspsychje> knaccc: nobody can prevent new bugs of the future, they just occur sometimes
[06:21] <lotuspsychje> a good practice with servers is when you have a test server around to test updates on, before you update your real servers
[06:21] <lotuspsychje> so you can hold up your servers until the bug is fixed
[06:21] <knaccc> hi lotuspsychje, thanks, yes i understand. i'm specifically only worried that i did not use the correct mechanism to downgrade the kernel
[06:22] <lotuspsychje> downgrading on ubuntu is not possible
[06:22] <ravage> i think he means using the older kernel
[06:22] <lotuspsychje> you can only boot a previous kernel for testing purposes like in your case now
[06:22] <knaccc> right, yes "uname -r" reports i'm now on 5.4.0-121-generic
[06:22] <ravage> usually you dont have to uninstall it. ubuntu always keeps the older kernel and you can select it in your grub config
[06:23] <ravage> removing the kernel that does not work is a workaround too of course
[06:24] <knaccc> ravage it's a remote server and it's fiddly to attempt to get into grub with a remote KVM. was there are better approach i should have used to tell grub to automatically boot an older kernel on reboot?
[06:25] <ravage> you made it to what you want already
[06:25] <ravage> sit tight and wait for the fixed kernel
[06:26] <ravage> *made it do
[06:26] <knaccc> ravage ok great, thanks for your help. i am mainly just worried that when i do the apt upgrade in the future, it'll get confused that i manually removed that 122 kernel it was expecting to be there, causing the grub config to get trashed somehow
[06:26] <ravage> from what you wrote you did not delete  the metapackage
[06:27] <ravage> so you should get the update
[06:28] <knaccc> ravage btw after the "sudo apt remove linux-image-5.4.0-122-generic" step, there was a scary pink warning screen telling me i was doing something that could break things, and so I answered "no". i'm not sure what i answered no to, since i don't remember the exact wording of the warning
[06:28] <tomreyn> knaccc: you can use grub-set-default to choose to boot into a different than the newest kernel. but as things stand, with a fix coming up shortly, you'll best just keep things as they are.
[06:28] <knaccc> tomreyn ok great
[06:29] <tomreyn> does    apt list --installed linux-image-5.4.0-122-generic    say that this kernel image is still installed?
[06:29] <knaccc> tomrey no, it's gone. the warning was something about removing the kernel i was currently using
[06:30] <knaccc> it was warning me not to answer "yes" to the yes/no question
[06:30] <knaccc> so i said no
[06:30] <knaccc> but it was removed anyway
[06:30] <knaccc> it was some kind of pink TUI dialog
[06:30] <tomreyn> i see. well, just wait for the update, i guess
[06:31] <knaccc> cool, thanks for your help, all of you. i was pretty scared for a while until the kernel reversion stopped the crashes. i just need to take some deep breaths :)
[06:34] <tomreyn> you can still keep installing updates safely. just make sure you'll boot into a kernel which has the fix bwfore you boot next time.
[06:37] <knaccc> aug 1st is only a few days away, so i think i'll just leave it alone until then. It was such a weird bug, the system booted just fine, but there was a kernel panic as soon as i started the web server
[13:40] <rbasak> ahasenack: o/ on tomcat9, is catalina.out rsyslog's _only_ output from tomcat? I don't see anything else. Just wondering about regression risk.
[13:41] <ahasenack> yes, the other catalina files are produced by tomcat9 itself, check /etc/tomcat9/logging.something
[13:41] <rbasak> That's good enough for me. Thanks!
[13:41] <ahasenack> in fact, it's logging the same thing in multiple places (also the systemd journal)
[13:43] <ahasenack> actually, /etc/tomcat9/logging.properties doesn't help determining the filenames, one has to consult upstream documentation to find out those are the default names. It's easier to just run fuser on /var/log/tomcat9/*
[14:46] <arthurvk> Hi, I run Ubuntu Server 18.04 for a php app that (heavily) uses redis-server. After upgrading from 4.15 to 5.4 (HWE) we notice a big drop in performance (~25%), the same is true when upgrading to 20.04 (both 5.4 and 5.15). Even with mitigations=off performance is still lower than 4.15 with all mitigations enabled. I can reproduce quite consistently with https://pastebin/vHFGpy5T Even more stunning 
[14:46] <arthurvk> is the performance gap compared to Debian 11 (5.10 or 5.18), the same script runs about 60% faster than Ubuntu with kernel >= 5.4
[14:48] <arthurvk> Does anyone have an idea where this performance diff comes from after 4.15 kernels? The servers run in a qemu-kvm virtual machine btw.
[14:53] <ahasenack> arthurvk: did you change just the kernel? IIRC there was a performance bug in php that was worked on by athos 
[14:54] <ravage> i dont think it is the kernel
[14:54] <ravage> tested focal and jammy on a jammy host wit lxc
[14:55] <ravage> focal: Redis 100000 ping benchmark took 2.1695239543915 seconds
[14:55] <ravage> jammy: Redis 100000 ping benchmark took 1.9991569519043 seconds
[14:56] <ravage> but times differ. i would have to run it a few more times and create an average
[14:57] <ahasenack> ravage: this is the bug I remembered: https://bugs.launchpad.net/ubuntu/+source/php7.4/+bug/1882279
[14:58] <ahasenack> there is a ppa with test packages for kinetic and jammy: https://launchpad.net/~athos-ribeiro/+archive/ubuntu/lp1882279-php-perf/+packages
[14:58] <ravage> yes i remember that too
[14:58] <ahasenack> although it now says there are newer versions available, even for jammy, so perhaps another sru was released before
[14:58] <ravage> let me finish some background tasks 
[14:58] <ravage> then i can run the script again
[14:58] <ahasenack> as the bug tasks don't mention a jammy upload
[14:58] <ahasenack> ok
[14:59] <ahasenack> sergiodj: hi, do you remember being moderated on openldap-technical@ when first posting there?
[14:59] <rbasak> ravage: you might try https://launchpad.net/~athos-ribeiro/+archive/ubuntu/lp1882279-php-perf/+packages in a Jammy container.
[15:00] <ravage> will give it a try
[15:01] <ravage> after that damn win10 finished updates. its always doing updates :P
[15:09] <arthurvk> ravage: the performance difference is primarily between 18.04 4.15 vs all newer kernels (5.4 and newer)
[15:11] <arthurvk> ahasenack: I saw that bug athos worked on, but I don't think that's related (his fix was reverted upstream though..)
[15:18] <ravage> the PPA makes no diference
[15:18] <ravage> also times for focal and normal jammy packages are about the same
[15:19] <ahasenack> then I don't know
[15:20] <ravage> i could try to run a bionic live usb and run the same script
[15:20] <ravage> would that still have the older kernel?
[15:21] <arthurvk> I think newer iso's let you choose if you want the default or HWE kernel?
[15:42] <lotuspsychje> arthurvk: i installed a 20.04 -desktop on a customer yesterday wich pulled HWE kernel on updates aka 5.15 series
[15:47] <ravage> the latest 18.04 live desktop has 5.4
[15:48] <lotuspsychje> kk
[15:48] <ravage> and it also has no redis-server or php-redis package
[15:48] <ravage> and im really too lazy to figure all that out :)
[18:07] <ahasenack> rbasak: does the git ubuntu bot look into MPs targeted at stable releases too, or just ubuntu/devel?