=== oerheks1 is now known as oerheks [06:19] hey all, I just did an apt upgrade and rebooted, which took me to kernel 5.4.0-122. This caused crashes on the server, so I downgraded by doing "sudo apt remove linux-image-5.4.0-122-generic" and rebooted. It's now working. I was wondering if what I did will now have any adverse consequences for the server which will break things if I do an apt upgrade in the future? Apparently there will be [06:19] a bugfix for kernel .122 released on 1 August (https://bugs.launchpad.net/ubuntu/+source/linux-hwe-5.4/+bug/1981658) [06:19] Launchpad bug 1981658 in linux-hwe-5.4 (Ubuntu Bionic) "BUG: kernel NULL pointer dereference, address: 0000000000000008" [Undecided, Confirmed] [06:20] i am terrified if i do an apt upgrade in the future, there will be some kind of grub issue and i'll brick the server [06:21] knaccc: nobody can prevent new bugs of the future, they just occur sometimes [06:21] a good practice with servers is when you have a test server around to test updates on, before you update your real servers [06:21] so you can hold up your servers until the bug is fixed [06:21] hi lotuspsychje, thanks, yes i understand. i'm specifically only worried that i did not use the correct mechanism to downgrade the kernel [06:22] downgrading on ubuntu is not possible [06:22] i think he means using the older kernel [06:22] you can only boot a previous kernel for testing purposes like in your case now [06:22] right, yes "uname -r" reports i'm now on 5.4.0-121-generic [06:22] usually you dont have to uninstall it. ubuntu always keeps the older kernel and you can select it in your grub config [06:23] removing the kernel that does not work is a workaround too of course [06:24] ravage it's a remote server and it's fiddly to attempt to get into grub with a remote KVM. was there are better approach i should have used to tell grub to automatically boot an older kernel on reboot? [06:25] you made it to what you want already [06:25] sit tight and wait for the fixed kernel [06:26] *made it do [06:26] ravage ok great, thanks for your help. i am mainly just worried that when i do the apt upgrade in the future, it'll get confused that i manually removed that 122 kernel it was expecting to be there, causing the grub config to get trashed somehow [06:26] from what you wrote you did not delete the metapackage [06:27] so you should get the update [06:28] ravage btw after the "sudo apt remove linux-image-5.4.0-122-generic" step, there was a scary pink warning screen telling me i was doing something that could break things, and so I answered "no". i'm not sure what i answered no to, since i don't remember the exact wording of the warning [06:28] knaccc: you can use grub-set-default to choose to boot into a different than the newest kernel. but as things stand, with a fix coming up shortly, you'll best just keep things as they are. [06:28] tomreyn ok great [06:29] does apt list --installed linux-image-5.4.0-122-generic say that this kernel image is still installed? [06:29] tomrey no, it's gone. the warning was something about removing the kernel i was currently using [06:30] it was warning me not to answer "yes" to the yes/no question [06:30] so i said no [06:30] but it was removed anyway [06:30] it was some kind of pink TUI dialog [06:30] i see. well, just wait for the update, i guess [06:31] cool, thanks for your help, all of you. i was pretty scared for a while until the kernel reversion stopped the crashes. i just need to take some deep breaths :) [06:34] you can still keep installing updates safely. just make sure you'll boot into a kernel which has the fix bwfore you boot next time. [06:37] aug 1st is only a few days away, so i think i'll just leave it alone until then. It was such a weird bug, the system booted just fine, but there was a kernel panic as soon as i started the web server [13:40] ahasenack: o/ on tomcat9, is catalina.out rsyslog's _only_ output from tomcat? I don't see anything else. Just wondering about regression risk. [13:41] yes, the other catalina files are produced by tomcat9 itself, check /etc/tomcat9/logging.something [13:41] That's good enough for me. Thanks! [13:41] in fact, it's logging the same thing in multiple places (also the systemd journal) [13:43] actually, /etc/tomcat9/logging.properties doesn't help determining the filenames, one has to consult upstream documentation to find out those are the default names. It's easier to just run fuser on /var/log/tomcat9/* [14:46] Hi, I run Ubuntu Server 18.04 for a php app that (heavily) uses redis-server. After upgrading from 4.15 to 5.4 (HWE) we notice a big drop in performance (~25%), the same is true when upgrading to 20.04 (both 5.4 and 5.15). Even with mitigations=off performance is still lower than 4.15 with all mitigations enabled. I can reproduce quite consistently with https://pastebin/vHFGpy5T Even more stunning [14:46] is the performance gap compared to Debian 11 (5.10 or 5.18), the same script runs about 60% faster than Ubuntu with kernel >= 5.4 [14:48] Does anyone have an idea where this performance diff comes from after 4.15 kernels? The servers run in a qemu-kvm virtual machine btw. [14:53] arthurvk: did you change just the kernel? IIRC there was a performance bug in php that was worked on by athos [14:54] i dont think it is the kernel [14:54] tested focal and jammy on a jammy host wit lxc [14:55] focal: Redis 100000 ping benchmark took 2.1695239543915 seconds [14:55] jammy: Redis 100000 ping benchmark took 1.9991569519043 seconds [14:56] but times differ. i would have to run it a few more times and create an average [14:57] ravage: this is the bug I remembered: https://bugs.launchpad.net/ubuntu/+source/php7.4/+bug/1882279 [14:57] Launchpad bug 1882279 in php7.4 (Ubuntu) "PHP built from source performs much better than the Ubuntu packaged version" [Medium, Triaged] [14:58] there is a ppa with test packages for kinetic and jammy: https://launchpad.net/~athos-ribeiro/+archive/ubuntu/lp1882279-php-perf/+packages [14:58] yes i remember that too [14:58] although it now says there are newer versions available, even for jammy, so perhaps another sru was released before [14:58] let me finish some background tasks [14:58] then i can run the script again [14:58] as the bug tasks don't mention a jammy upload [14:58] ok [14:59] sergiodj: hi, do you remember being moderated on openldap-technical@ when first posting there? [14:59] ravage: you might try https://launchpad.net/~athos-ribeiro/+archive/ubuntu/lp1882279-php-perf/+packages in a Jammy container. [15:00] will give it a try [15:01] after that damn win10 finished updates. its always doing updates :P [15:09] ravage: the performance difference is primarily between 18.04 4.15 vs all newer kernels (5.4 and newer) [15:11] ahasenack: I saw that bug athos worked on, but I don't think that's related (his fix was reverted upstream though..) [15:18] the PPA makes no diference [15:18] also times for focal and normal jammy packages are about the same [15:19] then I don't know [15:20] i could try to run a bionic live usb and run the same script [15:20] would that still have the older kernel? [15:21] I think newer iso's let you choose if you want the default or HWE kernel? [15:42] arthurvk: i installed a 20.04 -desktop on a customer yesterday wich pulled HWE kernel on updates aka 5.15 series [15:47] the latest 18.04 live desktop has 5.4 [15:48] kk [15:48] and it also has no redis-server or php-redis package [15:48] and im really too lazy to figure all that out :) [18:07] rbasak: does the git ubuntu bot look into MPs targeted at stable releases too, or just ubuntu/devel?