[03:17] <rellis> So two things, first I just wanted to say thanks to all of you folks for so much kernel work over the holidays, you rock.
[03:19] <rellis> Second I wanted to ask if anyone knows if the table on this page with versions is getting updated as new kernels become available? I'm just asking because my machines on 16.04 LTS linux-aws are getting 4.4.0-1048.57 which I don't see listed there. I wasn't sure if the page just hadn't been updated or I was misinterpreting what they're doing with that table.
[03:19] <rellis> When I said this page I meant: https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown
=== Elimin8r is now known as Elimin8er
[08:33] <apw> rellis, i would expect the table to have the first version it was nominally addressed in to be in the table, so as long as you have a higher one than that
[08:39] <klebers> ricotz, for which series?
[08:42] <ricotz> klebers, noticed it with xenial, but probably all
[08:50] <alkisg> Please also make it Recommends instead of Depends, as the rpi firmware in many times is provided outside of the real installation (e.g. netbooting), and linux-firmware-raspi2 postinst fails on chroots etc
[08:51] <thresh> I wonder if the tools described in https://wiki.ubuntu.com/Kernel/CrashdumpRecipe will work for me to debug that crash?
[08:51] <thresh> (with 4.13.0-21.24~16.04.1 and later)
[09:01] <klebers> ricotz, alkisg: thanks a lot for the input, I'll discuss it with the raspi2 guy
[09:01] <alkisg> Thank you klebers
[09:04] <ricotz> klebers, yw
[09:06] <thresh> I'm kinda curious right now - I've installed linux-crashdump;  I have GRUB_DEFAULT="1>2" to choose the working kernel on boot automatically (as it's a remote server);  what do I do now to force booting into the faulty kernel, and getting a dump out of it?
[09:06] <thresh> (I'm kinda stupid, too, which is why I need some guidance)
[12:03] <zioproto> If you are upgrading because of SPECTRE/Meltdown, be aware that your hardware could not boot if you hit this bug https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1742630
[12:03] <ubot5> Launchpad bug 1742630 in linux (Ubuntu) "Booting from 4.13.0-21-generic leads to Oops: NULL pointer dereference - RIP: isci_task_abort_task+0x30/0x3e0 [isci]" [Undecided,Confirmed]
[12:03] <zioproto> This is a "server thing" for us
[12:03] <thresh> hey, I have this kernel failing for me, too
[12:03] <thresh> but I wasnt that lucky to have the trace
[12:03] <zioproto> thresh: read the bug description, it is already fixed in Debian
[12:04] <zioproto> thresh: do you have Quanta hardware as well ?
[12:04] <thresh> I'm on Hetzner dedicated server, which runs AMD Ryzen and some Supermicro motherboard IIRC.
[12:04] <zioproto> thresh: it is a server thing as I mentioned
[12:04] <zioproto> I hope the bug gets the right attention. I opened it this morning
[12:04] <thresh> many thanks!
[12:05] <zioproto> I figured out the issue was already fixed in Debian but in Ubuntu community it was unknown
[12:05] <thresh> I was collecting data on the kernels I could boot before I could file a bug as well.
[12:05] <zioproto> well, the issue is well described if you follow those link on Launchpad
[12:05] <zioproto> discussion on the kernel mailing list and so on
[12:05] <thresh> yep, I'm reading it at the moment
[12:05] <zioproto> :)
[12:06] <zioproto> So, I hope the information reached the right people and the bug gets triaged :) thanks !
[12:06] <thresh> well, I'm definitely not the right person to fix it
[12:06] <thresh> just someone who was struggling with booting the hwe kernel since yesterday
[12:07] <zioproto> thresh: do you know who we can tag in this IRC channel so that the bug gets attention ?
[12:07] <zioproto> we have to upgrade a ~100 servers Openstack cluster and we are blocked by this bug :(
[12:08] <thresh> they person who offered help yesterday is not here, but he asked to file a bug on launchpad when I'm done - so I guess opening a bug is the way to get attention
[12:09] <zioproto> thresh: makes sense :)
[12:09] <zioproto> if you remember the nickname we can assign the bug on Launchpad
[12:10] <zioproto> if somebody was already working on this specific issue
[12:10] <thresh> it was jsalisbury
[12:10] <zioproto> thresh: also it would be great of you can click in Launchpad "Yes This bug affects me"
[12:10] <zioproto> so that we get an headcount of how many people have a problem with this kernel
[12:10] <TJ-> zioproto: have you tested the latest image? 4.13.0-23 ?
[12:11] <thresh> I cannot really confirm since i'm not sure this is the exact bug
[12:11] <thresh> I'm having
[12:12] <zioproto> TJ-: no I did not test it. But now I will go first to look at the changelog to see differences between 4.13.0-21 and 4.13.0-23. Thank you
[12:13] <thresh> I think the latest is -26, no?
[12:14] <thresh> and 21 vs 26 is mostly kpti patches
[12:16] <thresh> zioproto, and I have a very different SATA controller I think
[12:16] <zioproto> I cant find the changelog, do you have the link ?
[12:17] <thresh> no, I'm reading /usr/share/doc/linux-image-extra-4.13.0-26-generic/changelog.Debian.gz on the machine where I have the image installed
[12:17] <zioproto> I have this http://changelogs.ubuntu.com/changelogs/pool/main/l/linux/
[12:17] <zioproto> but the version 4.13.0-23 is not listed
[12:18] <thresh> http://changelogs.ubuntu.com/changelogs/pool/main/l/linux-hwe/linux-hwe_4.13.0-26.29~16.04.2/
[12:19] <TJ-> -26 is the latest now... keeps moving too quickly for me :D
[12:25] <zioproto> I have tested this version linux-image-generic-hwe-16.04         4.13.0.26.46
[12:25] <zioproto> this is not working for me
[12:30] <TJ-> zioproto: update the bug report so we know it's affecting the current image
[12:30] <zioproto> TJ-: ok !
[12:31] <TJ-> zioproto: I'd suggest altering the title so instead of mentioning the version, it says "PTI kernels" and list the versions tested in the description
[12:33] <zioproto> all done
[14:46] <thresh> hmm, I was able to run the mainline kernel 4.14, linux-image-4.14.13-041413-generic
[14:47] <thresh> which one is the linux-image-4.13.0-26-generic based on? 4.13.13?
[14:53] <thresh> alright, linux-image-4.13.13-041313-generic_4.13.13-041313.201711150531_amd64.deb also fails to boot
[15:00] <thresh> linux-image-4.13.16-041316-generic_4.13.16-041316.201711240901_amd64.deb works
[15:01] <thresh> (huge thanks to whoever maintains mainline kernel builds over at http://kernel.ubuntu.com/~kernel-ppa/mainline/)
[15:14] <thresh> linux-image-4.13.14-041314.201711180632 fails, linux-image-4.13.15-041315-generic boots
[15:18] <jsalisbury> thresh, do you have a bug open, so I can review all the data so far?
[15:18] <thresh> jsalisbury, no, I'm going to open it right now, now that I have some informations.  many thanks!
[15:19] <jsalisbury> thresh, great, thanks.  I'll review everything and see if we can create a test kernel or dig deeper if needed.
[15:38] <thresh> there you go jsalisbury https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1742721
[15:38] <ubot5> Launchpad bug 1742721 in linux (Ubuntu) "linux-image-4.13.0-26-generic / linux-image-extra-4.13.0-26-generic fail to boot" [Undecided,New]
[15:38] <jsalisbury> thresh, thanks!  I'll take a look at it shortly
[15:39] <thresh> \o/
[15:43] <jsalisbury> thresh, Just to confirm, upstream 4.13.14 has the bug and 4.13.15 fixes the bug?  If that is the case, I can "reverse" bisect between those two versions.
[15:43] <thresh> jsalisbury, yes, that's correct.
[15:44] <thresh> I'm thinking about setting up make-kpkg or something to bisect, too.
[15:44] <jsalisbury> thresh, perfect.  I'll start bisecting between those two versions and start buiding a test kernel.
[15:44] <thresh> I've did that for debian, I wonder if that's the same for Ubuntu?
[15:44] <jsalisbury> thresh, I can build test kernels fairly quickly if your able to test them?  There will be about 10 or so to test.
[15:44] <thresh> jsalisbury, yeah, I can do that.
[15:45] <jsalisbury> thresh, great.  Give me about 20 minutes and I'll have the first test kernel ready.
[17:08] <jsalisbury> thresh, I built the first test kernel and posted a link to it in the bug.
[17:09] <thresh> jsalisbury, thanks, Ill be able to test it in an hour (commuting home atm)
[17:09] <jsalisbury> thresh, sounds good, thanks!  Just post the results and I'll build the next test kernel based on that.
[19:19] <rellis> apw: Thank you for the reply.
[19:25] <TJ-> Had a report in #ubuntu earlier; user with ~1200 AWS instances, testing the PTI upgrade and despite grub default being  -109, -100 is booting. I've asked them to enable the GRUB serial console logging to capture anything useful, because it appears the fallback kernel is being booted.
[19:30] <Odd_Bloke> TJ-: That's a little confusing; -109 is the regular xenial kernel, not the custom AWS kernel (which is -1048, I think).
[19:30] <Odd_Bloke> TJ-: If you see them again, could you ask them to file a bug in the cloud-images LP project?
[19:30] <TJ-> Odd_Bloke: this user is installing their own kernels from what I could get from them
[19:31] <TJ-> Odd_Bloke: I've asked them to file a bug if they can collect some useful data :) Not heard anything since (about an hour)
[19:36] <TJ-> one thing I saw earlier - we've had so many PTI kernel upgrades that with package autoremove we've seen situations where all the installed kernels are PTI and leaving the system unable to boot without nopti
[20:32] <thresh> jsalisbury, given linux-image-4.13.14-041314-generic_4.13.14-041314.201801111932_amd64.deb also fails to boot, and I have sm750fb loaded on a working kernel... https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?h=v4.13.15&id=c52278a636018cb8fa39b2538c6da5d35e6515f7 is likely the commit.
[20:32] <thresh> jsalisbury, but, let's test kernels a bit more ;-)
=== rellis is now known as rkeiii
[23:24] <jsalisbury> thresh, I posted the next kernel in the bug report.  
[23:24] <jsalisbury> thresh, sorry for the delay, juggling a couple other bugs as well.