| === oerheks1 is now known as oerheks | ||
| knaccc | hey all, I just did an apt upgrade and rebooted, which took me to kernel 5.4.0-122. This caused crashes on the server, so I downgraded by doing "sudo apt remove linux-image-5.4.0-122-generic" and rebooted. It's now working. I was wondering if what I did will now have any adverse consequences for the server which will break things if I do an apt upgrade in the future? Apparently there will be | 06:19 |
|---|---|---|
| knaccc | a bugfix for kernel .122 released on 1 August (https://bugs.launchpad.net/ubuntu/+source/linux-hwe-5.4/+bug/1981658) | 06:19 |
| ubottu | Launchpad bug 1981658 in linux-hwe-5.4 (Ubuntu Bionic) "BUG: kernel NULL pointer dereference, address: 0000000000000008" [Undecided, Confirmed] | 06:19 |
| knaccc | i am terrified if i do an apt upgrade in the future, there will be some kind of grub issue and i'll brick the server | 06:20 |
| lotuspsychje | knaccc: nobody can prevent new bugs of the future, they just occur sometimes | 06:21 |
| lotuspsychje | a good practice with servers is when you have a test server around to test updates on, before you update your real servers | 06:21 |
| lotuspsychje | so you can hold up your servers until the bug is fixed | 06:21 |
| knaccc | hi lotuspsychje, thanks, yes i understand. i'm specifically only worried that i did not use the correct mechanism to downgrade the kernel | 06:21 |
| lotuspsychje | downgrading on ubuntu is not possible | 06:22 |
| ravage | i think he means using the older kernel | 06:22 |
| lotuspsychje | you can only boot a previous kernel for testing purposes like in your case now | 06:22 |
| knaccc | right, yes "uname -r" reports i'm now on 5.4.0-121-generic | 06:22 |
| ravage | usually you dont have to uninstall it. ubuntu always keeps the older kernel and you can select it in your grub config | 06:22 |
| ravage | removing the kernel that does not work is a workaround too of course | 06:23 |
| knaccc | ravage it's a remote server and it's fiddly to attempt to get into grub with a remote KVM. was there are better approach i should have used to tell grub to automatically boot an older kernel on reboot? | 06:24 |
| ravage | you made it to what you want already | 06:25 |
| ravage | sit tight and wait for the fixed kernel | 06:25 |
| ravage | *made it do | 06:26 |
| knaccc | ravage ok great, thanks for your help. i am mainly just worried that when i do the apt upgrade in the future, it'll get confused that i manually removed that 122 kernel it was expecting to be there, causing the grub config to get trashed somehow | 06:26 |
| ravage | from what you wrote you did not delete the metapackage | 06:26 |
| ravage | so you should get the update | 06:27 |
| knaccc | ravage btw after the "sudo apt remove linux-image-5.4.0-122-generic" step, there was a scary pink warning screen telling me i was doing something that could break things, and so I answered "no". i'm not sure what i answered no to, since i don't remember the exact wording of the warning | 06:28 |
| tomreyn | knaccc: you can use grub-set-default to choose to boot into a different than the newest kernel. but as things stand, with a fix coming up shortly, you'll best just keep things as they are. | 06:28 |
| knaccc | tomreyn ok great | 06:28 |
| tomreyn | does apt list --installed linux-image-5.4.0-122-generic say that this kernel image is still installed? | 06:29 |
| knaccc | tomrey no, it's gone. the warning was something about removing the kernel i was currently using | 06:29 |
| knaccc | it was warning me not to answer "yes" to the yes/no question | 06:30 |
| knaccc | so i said no | 06:30 |
| knaccc | but it was removed anyway | 06:30 |
| knaccc | it was some kind of pink TUI dialog | 06:30 |
| tomreyn | i see. well, just wait for the update, i guess | 06:30 |
| knaccc | cool, thanks for your help, all of you. i was pretty scared for a while until the kernel reversion stopped the crashes. i just need to take some deep breaths :) | 06:31 |
| tomreyn | you can still keep installing updates safely. just make sure you'll boot into a kernel which has the fix bwfore you boot next time. | 06:34 |
| knaccc | aug 1st is only a few days away, so i think i'll just leave it alone until then. It was such a weird bug, the system booted just fine, but there was a kernel panic as soon as i started the web server | 06:37 |
| rbasak | ahasenack: o/ on tomcat9, is catalina.out rsyslog's _only_ output from tomcat? I don't see anything else. Just wondering about regression risk. | 13:40 |
| ahasenack | yes, the other catalina files are produced by tomcat9 itself, check /etc/tomcat9/logging.something | 13:41 |
| rbasak | That's good enough for me. Thanks! | 13:41 |
| ahasenack | in fact, it's logging the same thing in multiple places (also the systemd journal) | 13:41 |
| ahasenack | actually, /etc/tomcat9/logging.properties doesn't help determining the filenames, one has to consult upstream documentation to find out those are the default names. It's easier to just run fuser on /var/log/tomcat9/* | 13:43 |
| arthurvk | Hi, I run Ubuntu Server 18.04 for a php app that (heavily) uses redis-server. After upgrading from 4.15 to 5.4 (HWE) we notice a big drop in performance (~25%), the same is true when upgrading to 20.04 (both 5.4 and 5.15). Even with mitigations=off performance is still lower than 4.15 with all mitigations enabled. I can reproduce quite consistently with https://pastebin/vHFGpy5T Even more stunning | 14:46 |
| arthurvk | is the performance gap compared to Debian 11 (5.10 or 5.18), the same script runs about 60% faster than Ubuntu with kernel >= 5.4 | 14:46 |
| arthurvk | Does anyone have an idea where this performance diff comes from after 4.15 kernels? The servers run in a qemu-kvm virtual machine btw. | 14:48 |
| ahasenack | arthurvk: did you change just the kernel? IIRC there was a performance bug in php that was worked on by athos | 14:53 |
| ravage | i dont think it is the kernel | 14:54 |
| ravage | tested focal and jammy on a jammy host wit lxc | 14:54 |
| ravage | focal: Redis 100000 ping benchmark took 2.1695239543915 seconds | 14:55 |
| ravage | jammy: Redis 100000 ping benchmark took 1.9991569519043 seconds | 14:55 |
| ravage | but times differ. i would have to run it a few more times and create an average | 14:56 |
| ahasenack | ravage: this is the bug I remembered: https://bugs.launchpad.net/ubuntu/+source/php7.4/+bug/1882279 | 14:57 |
| ubottu | Launchpad bug 1882279 in php7.4 (Ubuntu) "PHP built from source performs much better than the Ubuntu packaged version" [Medium, Triaged] | 14:57 |
| ahasenack | there is a ppa with test packages for kinetic and jammy: https://launchpad.net/~athos-ribeiro/+archive/ubuntu/lp1882279-php-perf/+packages | 14:58 |
| ravage | yes i remember that too | 14:58 |
| ahasenack | although it now says there are newer versions available, even for jammy, so perhaps another sru was released before | 14:58 |
| ravage | let me finish some background tasks | 14:58 |
| ravage | then i can run the script again | 14:58 |
| ahasenack | as the bug tasks don't mention a jammy upload | 14:58 |
| ahasenack | ok | 14:58 |
| ahasenack | sergiodj: hi, do you remember being moderated on openldap-technical@ when first posting there? | 14:59 |
| rbasak | ravage: you might try https://launchpad.net/~athos-ribeiro/+archive/ubuntu/lp1882279-php-perf/+packages in a Jammy container. | 14:59 |
| ravage | will give it a try | 15:00 |
| ravage | after that damn win10 finished updates. its always doing updates :P | 15:01 |
| arthurvk | ravage: the performance difference is primarily between 18.04 4.15 vs all newer kernels (5.4 and newer) | 15:09 |
| arthurvk | ahasenack: I saw that bug athos worked on, but I don't think that's related (his fix was reverted upstream though..) | 15:11 |
| ravage | the PPA makes no diference | 15:18 |
| ravage | also times for focal and normal jammy packages are about the same | 15:18 |
| ahasenack | then I don't know | 15:19 |
| ravage | i could try to run a bionic live usb and run the same script | 15:20 |
| ravage | would that still have the older kernel? | 15:20 |
| arthurvk | I think newer iso's let you choose if you want the default or HWE kernel? | 15:21 |
| lotuspsychje | arthurvk: i installed a 20.04 -desktop on a customer yesterday wich pulled HWE kernel on updates aka 5.15 series | 15:42 |
| ravage | the latest 18.04 live desktop has 5.4 | 15:47 |
| lotuspsychje | kk | 15:48 |
| ravage | and it also has no redis-server or php-redis package | 15:48 |
| ravage | and im really too lazy to figure all that out :) | 15:48 |
| ahasenack | rbasak: does the git ubuntu bot look into MPs targeted at stable releases too, or just ubuntu/devel? | 18:07 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!