TJ-Do we have any tools for intelligently comparing dmesg logs ignoring timestamps and slight numeric-value differences? I'm /trying/ to debug a loss of all USB devices when kexec-ing (4.13.0-25-lowlatency - but affects all versions tested so far) 00:09
TJ-With 16.04 and hwe (4.13) I've just noticed on 1 system that the symlinks /vmlinuz -> 4.4.0-111-lowlatency and /vmlinuz.old -> 4.13.0-25-lowlatency whereas on another it's /vmlinux -> 4.15.0-041500rc7-lowlatency and /vmlinuz.old -> 4.4.0-110-lowlatency. Shouldn't the first pair be reversed so /vmlinux.old ->  4.4.0-111-lowlatency ?00:26
TJ-It seems as if the vmlinuz symlink issue was due to having the hwe-16.04-edge package instead of hwe-16.04 01:18
=== himcesjf_ is now known as him-cesjf
alkisgGood morning everyone06:56
alkisgapw, jsalisbury: I managed to reproduce the "kernel reboots immediately" issue here in my office. I semi-bisected it using the mainline ppa, and I saw that:06:58
alkisgv4.15-rc4 = good, v4.15-rc5 = bad, v4.15-rc5 +nokpti = bad (although afaik rc5 is before meltdown patches, it didn't hurt to test)06:58
alkisgAnything else I could try? Should I start git-based kernel bisection? Is it time to file a bug now even though all I have is "kernel reboots on certain hardware between rc4 and rc5"?06:59
alkisgI have an almost identical hardware in the office that is NOT affected, the difference being i5-4440 (affected) vs i3-4150 (not affected) on the same motherboard Gigabyte B85M-HD307:05
=== alkisg_ is now known as alakisg
=== alakisg is now known as alkisg
* alkisg reboots to test maxcpus=2...07:13
alkisg....it didn't help07:16
alkisgrc5 64bit works fine...07:30
alkisgAdding acpi=off works around the issue07:35
alkisgNo other kernel parameter from https://wiki.ubuntu.com/DebuggingACPI worked, only acpi=off07:53
alkisgI'm reading https://wiki.ubuntu.com/Kernel/KernelBisection, and I believe I can follow the instructions for the xenial git tree.07:59
alkisgWill this tell me the offending commit, or should I bisect using 4.15 instead, and if so, what is the git URL for that?08:00
f_galkisg: there are only 262 non-merge commits between those RCs, bisecting should be easily possible08:13
alkisg f_g, cool, if I only knew how :)08:15
f_ggit://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git is the upstream git repo08:15
alkisgI'm reading various sources, but I'm not sure which one to follow08:15
alkisgAh, will this build me a debian package, or an initramfs anyway?08:15
alkisgI think https://wiki.ubuntu.com/Kernel/KernelBisection is using the ubuntu git trees that also contain a debian/ dir?08:16
f_gand what is linked there08:16
alkisgf_g: so a rough idea is that git bisect will compile the kernel for me, and I'll then need to manually install it and calll update-initramfs ?08:17
=== Elimin8r is now known as Elimin8er
f_galkisg: git bisect just tells you which commits to test, the instructions tell you how you get the kernels from those commits. the generated kernel deb will call update-initramfs automatically after installing, just like a regular kernel from Ubuntu would08:19
alkisgf_g: what I can't understand in that workflow is how a kernel.org git will know to produce a .deb...08:19
alkisgI won't have the debian/ dir there, will it?08:20
f_galkisg: the kernel build scripts have simple support for creating packages, both for .deb and .rpm08:20
alkisgOh very nice, thank you08:20
f_gthey are nowhere near as fancy as what RH/Debian/Ubuntu do, but for quick testing with integration into the package manager (especially for clean removal) they are sufficient08:21
alkisgOK, on to my first real kernel bisection then :)08:21
f_galkisg: if you manage to compile one kernel successfully, bisecting is basically just repeating that over and over in a strategic manner.08:22
alkisgOh the bisection logic was very easy to understand, it was the .deb part that I was missing08:22
alkisgf_g: I'm unsure what to put in series, in "build-mainline-one tag series". I tried "xenial" but it fails.08:53
alkisgWould that be "master"?08:53
f_galkisg: not sure what you did there. running git bisect and then following https://wiki.ubuntu.com/KernelTeam/GitKernelBuild minus the clone and checkout (git bisect already gets you each commit checked out) should "just work"09:01
alkisgf_g: with a first look, I think the previous page that you linked required having a debian/ dir, and this one doesn't... trying this method now...09:02
f_galkisg: the KernelBisection page I linked to has the link to GitKernelBuild right at the place where I linked to ;)09:04
alkisgf_g: but if you read that page, instead of following the second link to the second page, it says to use the mainline-build-one script, and it has instructions for bisecting etc, which I supposed was what I wanted :D Ty, trying...09:06
f_galkisg: it does say to use, it says you can use ;) mainline-build-one is more Ubuntu-ish, GitKernelBuild is closer to upstream. both should be fine to find the culprit (provided your own build of rc5 shows the issue, and the one of rc4 does not - it's always good to verify this before starting a long bisect only to find out that your kernel config for bisecting does not match what Ubuntu used to build09:10
f_gtheir mainline PPA kernel deb ;))09:10
alkisgGotcha. On an i5-4440, how long a compilation would take? Half an hour? Two hours?09:10
alkisgWhoops disk space became an issue before compilation time :/ 10 GB may not be enough :(09:27
f_galkisg: no clue, but once you built your first one you know ;) it's very consistent if your system is otherwise idle09:36
alkisgf_g: I hope I don't need to `make clean` on each bisection step, do I? (trying to save time...)09:37
f_gsince you are potentially going back in time, you need to09:38
alkisgWhoops. I'll need another pc to work on while my good one compiles :D09:38
alkisgThanks again09:38
f_gif compiling a single kernel takes > 45 minutes, maybe you can convince jsalisbury or apw to play bisect ping pong with you using Ubuntu build machines?09:39
f_gon a fast server a single kernel build is doable in a few minutes, but you need lots of cores for that ;) e.g., our EPYC with 16*2*2 threads does it in 6 minutes or so09:40
apwf_g, the exact config used is included in the patches in the output09:43
alkisgapw, I copied /boot/config...rc5 as a starting point, hope that's good enough for the bisection...09:45
apwalkisg, sound be the same indeed09:45
alkisgCool, I hope each build will only take about half an hour on my i5, let's see...09:45
alkisg...although now that I switched to the hdd instead of the ssd due to space issues, it seems to go much slower... :(09:47
threshif you have 32G ram or so, the build should fit into that09:49
f_gapw: yep. IMHO still a good idea to verify the bisect assumption holds for your local build env before doing a whole bisect run for nought ;)09:49
threshIIRC it takes me more than 20 gigabytes disk space to do a build09:50
thresh(also yeah epyc and nvme ssds are awesome)09:50
alkisgthresh: just 8 GB RAM here :/09:50
alkisgDo builds produce both amd64 and i386 debs? Can I do the build itself from either an amd64 or i386 installation?09:51
alkisg(I'm currently using i386 because I wasn't sure if it produced both or just the running one)09:52
alkisgYey, 46 minutes :) Rebooting...10:23
lorddoskiasguys i just got the following message from latest HWE kernel: https://paste.ubuntu.com/26409755/10:24
lorddoskiasLinux fisk 4.13.0-26-generic #29~16.04.2-Ubuntu10:24
alkisgYey, test kernel booted. Oh no, I misttyped 4.5 instead of 4.15 and it was a wasted effort. apw, since it takes 46 minutes, do you think it would be worth it to use the ubuntu builders for that?10:26
=== alkisg is now known as alkisg_web
=== alkisg1 is now known as alkisg
alkisgf_g: I've completed the first step of the bisection, "good". Now before continuing with the bisection, I'd like to test that "rc5=bad", any hint on how to do that without losing the current bisection progress?16:47
alkisgHmm, although restarting the bisection doesn't mean I need to rebuild the first step, I can just answer it's good and get to step 2... :)16:58
hallynjjohansen: just to confirm, your kernel fixed the issue - thanks!19:42
chilukHey apw bjf, what's the plan on the Spectre_v2 mitigation?  Are you guys planning on doing IBRS/IBPB or Retpoline compilation changes or both?20:40
apwibrs to start with, then retpoline + ibrs when upstream catches up20:41
chilukis upstream still working on repoline patches?  I was under the impression that they had made it to stable now.20:41
chilukyeah it really looks like retpoline is in linux-stable now.  It should get pulled in with stable updates.20:42
chilukalso isn't retpoline + ibrs duplicate effort?  even though you might have retpoline compiled in, only one gets used at run time.20:43
chiluk^^ that's supposed to be a question 20:43
chilukI guess I should move to hardened.20:49
apwno some things need ibrs as well, and we'll wait for that to settle out20:51
chilukapw are you referring to unrecompiled userspace?20:53
chilukalso is there any plan to enable gcc with retpoline and do a full archive recompile?20:53
apwI Amelia referring to CPUs which are not necessarily fixed with retpoline only20:53
chilukI was unaware such cpus existed... 20:54
chilukretpoline is effectively simply disables speculitive execution for indirect branches20:55
apwretpoline makes it hard for the CPU speculate to useful code, skylake at least  may be able to still speculate even with retpoline forms21:51
chilukThat'd be pretty impressive given my understanding of how retpoline functions.. it basically sandboxes the speculative branch in sleeps22:36

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!