[00:06] <hallyn> thanks
[00:09] <TJ-> Do we have any tools for intelligently comparing dmesg logs ignoring timestamps and slight numeric-value differences? I'm /trying/ to debug a loss of all USB devices when kexec-ing (4.13.0-25-lowlatency - but affects all versions tested so far) 
[00:26] <TJ-> With 16.04 and hwe (4.13) I've just noticed on 1 system that the symlinks /vmlinuz -> 4.4.0-111-lowlatency and /vmlinuz.old -> 4.13.0-25-lowlatency whereas on another it's /vmlinux -> 4.15.0-041500rc7-lowlatency and /vmlinuz.old -> 4.4.0-110-lowlatency. Shouldn't the first pair be reversed so /vmlinux.old ->  4.4.0-111-lowlatency ?
[01:18] <TJ-> It seems as if the vmlinuz symlink issue was due to having the hwe-16.04-edge package instead of hwe-16.04 
=== himcesjf_ is now known as him-cesjf
[06:56] <alkisg> Good morning everyone
[06:58] <alkisg> apw, jsalisbury: I managed to reproduce the "kernel reboots immediately" issue here in my office. I semi-bisected it using the mainline ppa, and I saw that:
[06:58] <alkisg> v4.15-rc4 = good, v4.15-rc5 = bad, v4.15-rc5 +nokpti = bad (although afaik rc5 is before meltdown patches, it didn't hurt to test)
[06:59] <alkisg> Anything else I could try? Should I start git-based kernel bisection? Is it time to file a bug now even though all I have is "kernel reboots on certain hardware between rc4 and rc5"?
[07:05] <alkisg> I have an almost identical hardware in the office that is NOT affected, the difference being i5-4440 (affected) vs i3-4150 (not affected) on the same motherboard Gigabyte B85M-HD3
=== alkisg_ is now known as alakisg
=== alakisg is now known as alkisg
[07:13]  * alkisg reboots to test maxcpus=2...
[07:16] <alkisg> ....it didn't help
[07:30] <alkisg> rc5 64bit works fine...
[07:35] <alkisg> Adding acpi=off works around the issue
[07:53] <alkisg> No other kernel parameter from https://wiki.ubuntu.com/DebuggingACPI worked, only acpi=off
[07:59] <alkisg> I'm reading https://wiki.ubuntu.com/Kernel/KernelBisection, and I believe I can follow the instructions for the xenial git tree.
[08:00] <alkisg> Will this tell me the offending commit, or should I bisect using 4.15 instead, and if so, what is the git URL for that?
[08:13] <f_g> alkisg: there are only 262 non-merge commits between those RCs, bisecting should be easily possible
[08:15] <alkisg>  f_g, cool, if I only knew how :)
[08:15] <f_g> git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git is the upstream git repo
[08:15] <alkisg> I'm reading various sources, but I'm not sure which one to follow
[08:15] <alkisg> Ah, will this build me a debian package, or an initramfs anyway?
[08:16] <alkisg> I think https://wiki.ubuntu.com/Kernel/KernelBisection is using the ubuntu git trees that also contain a debian/ dir?
[08:16] <f_g> https://wiki.ubuntu.com/Kernel/KernelBisection#Bisecting_upstream_kernel_versions
[08:16] <f_g> and what is linked there
[08:17] <alkisg> f_g: so a rough idea is that git bisect will compile the kernel for me, and I'll then need to manually install it and calll update-initramfs ?
=== Elimin8r is now known as Elimin8er
[08:19] <f_g> alkisg: git bisect just tells you which commits to test, the instructions tell you how you get the kernels from those commits. the generated kernel deb will call update-initramfs automatically after installing, just like a regular kernel from Ubuntu would
[08:19] <alkisg> f_g: what I can't understand in that workflow is how a kernel.org git will know to produce a .deb...
[08:20] <alkisg> I won't have the debian/ dir there, will it?
[08:20] <f_g> alkisg: the kernel build scripts have simple support for creating packages, both for .deb and .rpm
[08:20] <alkisg> Oh very nice, thank you
[08:21] <f_g> they are nowhere near as fancy as what RH/Debian/Ubuntu do, but for quick testing with integration into the package manager (especially for clean removal) they are sufficient
[08:21] <alkisg> OK, on to my first real kernel bisection then :)
[08:22] <f_g> alkisg: if you manage to compile one kernel successfully, bisecting is basically just repeating that over and over in a strategic manner.
[08:22] <alkisg> Oh the bisection logic was very easy to understand, it was the .deb part that I was missing
[08:53] <alkisg> f_g: I'm unsure what to put in series, in "build-mainline-one tag series". I tried "xenial" but it fails.
[08:53] <alkisg> Would that be "master"?
[08:55] <alkisg> http://termbin.com/rwuf
[09:01] <f_g> alkisg: not sure what you did there. running git bisect and then following https://wiki.ubuntu.com/KernelTeam/GitKernelBuild minus the clone and checkout (git bisect already gets you each commit checked out) should "just work"
[09:02] <alkisg> f_g: with a first look, I think the previous page that you linked required having a debian/ dir, and this one doesn't... trying this method now...
[09:04] <f_g> alkisg: the KernelBisection page I linked to has the link to GitKernelBuild right at the place where I linked to ;)
[09:06] <alkisg> f_g: but if you read that page, instead of following the second link to the second page, it says to use the mainline-build-one script, and it has instructions for bisecting etc, which I supposed was what I wanted :D Ty, trying...
[09:10] <f_g> alkisg: it does say to use, it says you can use ;) mainline-build-one is more Ubuntu-ish, GitKernelBuild is closer to upstream. both should be fine to find the culprit (provided your own build of rc5 shows the issue, and the one of rc4 does not - it's always good to verify this before starting a long bisect only to find out that your kernel config for bisecting does not match what Ubuntu used to build
[09:10] <f_g> their mainline PPA kernel deb ;))
[09:10] <alkisg> Gotcha. On an i5-4440, how long a compilation would take? Half an hour? Two hours?
[09:27] <alkisg> Whoops disk space became an issue before compilation time :/ 10 GB may not be enough :(
[09:36] <f_g> alkisg: no clue, but once you built your first one you know ;) it's very consistent if your system is otherwise idle
[09:37] <alkisg> f_g: I hope I don't need to `make clean` on each bisection step, do I? (trying to save time...)
[09:38] <f_g> since you are potentially going back in time, you need to
[09:38] <alkisg> Whoops. I'll need another pc to work on while my good one compiles :D
[09:38] <alkisg> Thanks again
[09:39] <f_g> if compiling a single kernel takes > 45 minutes, maybe you can convince jsalisbury or apw to play bisect ping pong with you using Ubuntu build machines?
[09:40] <f_g> on a fast server a single kernel build is doable in a few minutes, but you need lots of cores for that ;) e.g., our EPYC with 16*2*2 threads does it in 6 minutes or so
[09:43] <apw> f_g, the exact config used is included in the patches in the output
[09:45] <alkisg> apw, I copied /boot/config...rc5 as a starting point, hope that's good enough for the bisection...
[09:45] <apw> alkisg, sound be the same indeed
[09:45] <alkisg> Cool, I hope each build will only take about half an hour on my i5, let's see...
[09:47] <alkisg> ...although now that I switched to the hdd instead of the ssd due to space issues, it seems to go much slower... :(
[09:49] <thresh> if you have 32G ram or so, the build should fit into that
[09:49] <f_g> apw: yep. IMHO still a good idea to verify the bisect assumption holds for your local build env before doing a whole bisect run for nought ;)
[09:50] <thresh> IIRC it takes me more than 20 gigabytes disk space to do a build
[09:50] <thresh> (also yeah epyc and nvme ssds are awesome)
[09:50] <alkisg> thresh: just 8 GB RAM here :/
[09:51] <alkisg> Do builds produce both amd64 and i386 debs? Can I do the build itself from either an amd64 or i386 installation?
[09:52] <alkisg> (I'm currently using i386 because I wasn't sure if it produced both or just the running one)
[10:23] <alkisg> Yey, 46 minutes :) Rebooting...
[10:24] <lorddoskias> guys i just got the following message from latest HWE kernel: https://paste.ubuntu.com/26409755/
[10:24] <lorddoskias> Linux fisk 4.13.0-26-generic #29~16.04.2-Ubuntu
[10:26] <alkisg> Yey, test kernel booted. Oh no, I misttyped 4.5 instead of 4.15 and it was a wasted effort. apw, since it takes 46 minutes, do you think it would be worth it to use the ubuntu builders for that?
=== alkisg is now known as alkisg_web
=== alkisg1 is now known as alkisg
[16:47] <alkisg> f_g: I've completed the first step of the bisection, "good". Now before continuing with the bisection, I'd like to test that "rc5=bad", any hint on how to do that without losing the current bisection progress?
[16:58] <alkisg> Hmm, although restarting the bisection doesn't mean I need to rebuild the first step, I can just answer it's good and get to step 2... :)
[19:42] <hallyn> jjohansen: just to confirm, your kernel fixed the issue - thanks!
[19:48] <jjohansen> ack
[20:40] <chiluk> Hey apw bjf, what's the plan on the Spectre_v2 mitigation?  Are you guys planning on doing IBRS/IBPB or Retpoline compilation changes or both?
[20:41] <apw> ibrs to start with, then retpoline + ibrs when upstream catches up
[20:41] <chiluk> is upstream still working on repoline patches?  I was under the impression that they had made it to stable now.
[20:42] <chiluk> yeah it really looks like retpoline is in linux-stable now.  It should get pulled in with stable updates.
[20:43] <chiluk> also isn't retpoline + ibrs duplicate effort?  even though you might have retpoline compiled in, only one gets used at run time.
[20:43] <chiluk> ^^ that's supposed to be a question 
[20:49] <chiluk> I guess I should move to hardened.
[20:51] <apw> no some things need ibrs as well, and we'll wait for that to settle out
[20:53] <chiluk> apw are you referring to unrecompiled userspace?
[20:53] <chiluk> also is there any plan to enable gcc with retpoline and do a full archive recompile?
[20:53] <apw> I Amelia referring to CPUs which are not necessarily fixed with retpoline only
[20:54] <chiluk> I was unaware such cpus existed... 
[20:55] <chiluk> retpoline is effectively simply disables speculitive execution for indirect branches
[21:51] <apw> retpoline makes it hard for the CPU speculate to useful code, skylake at least  may be able to still speculate even with retpoline forms
[22:36] <chiluk> That'd be pretty impressive given my understanding of how retpoline functions.. it basically sandboxes the speculative branch in sleeps
[22:42] <ratliff> https://lwn.net/Articles/744193/