| hallyn | thanks | 00:06 |
|---|---|---|
| TJ- | Do we have any tools for intelligently comparing dmesg logs ignoring timestamps and slight numeric-value differences? I'm /trying/ to debug a loss of all USB devices when kexec-ing (4.13.0-25-lowlatency - but affects all versions tested so far) | 00:09 |
| TJ- | With 16.04 and hwe (4.13) I've just noticed on 1 system that the symlinks /vmlinuz -> 4.4.0-111-lowlatency and /vmlinuz.old -> 4.13.0-25-lowlatency whereas on another it's /vmlinux -> 4.15.0-041500rc7-lowlatency and /vmlinuz.old -> 4.4.0-110-lowlatency. Shouldn't the first pair be reversed so /vmlinux.old -> 4.4.0-111-lowlatency ? | 00:26 |
| TJ- | It seems as if the vmlinuz symlink issue was due to having the hwe-16.04-edge package instead of hwe-16.04 | 01:18 |
| === himcesjf_ is now known as him-cesjf | ||
| alkisg | Good morning everyone | 06:56 |
| alkisg | apw, jsalisbury: I managed to reproduce the "kernel reboots immediately" issue here in my office. I semi-bisected it using the mainline ppa, and I saw that: | 06:58 |
| alkisg | v4.15-rc4 = good, v4.15-rc5 = bad, v4.15-rc5 +nokpti = bad (although afaik rc5 is before meltdown patches, it didn't hurt to test) | 06:58 |
| alkisg | Anything else I could try? Should I start git-based kernel bisection? Is it time to file a bug now even though all I have is "kernel reboots on certain hardware between rc4 and rc5"? | 06:59 |
| alkisg | I have an almost identical hardware in the office that is NOT affected, the difference being i5-4440 (affected) vs i3-4150 (not affected) on the same motherboard Gigabyte B85M-HD3 | 07:05 |
| === alkisg_ is now known as alakisg | ||
| === alakisg is now known as alkisg | ||
| * alkisg reboots to test maxcpus=2... | 07:13 | |
| alkisg | ....it didn't help | 07:16 |
| alkisg | rc5 64bit works fine... | 07:30 |
| alkisg | Adding acpi=off works around the issue | 07:35 |
| alkisg | No other kernel parameter from https://wiki.ubuntu.com/DebuggingACPI worked, only acpi=off | 07:53 |
| alkisg | I'm reading https://wiki.ubuntu.com/Kernel/KernelBisection, and I believe I can follow the instructions for the xenial git tree. | 07:59 |
| alkisg | Will this tell me the offending commit, or should I bisect using 4.15 instead, and if so, what is the git URL for that? | 08:00 |
| f_g | alkisg: there are only 262 non-merge commits between those RCs, bisecting should be easily possible | 08:13 |
| alkisg | f_g, cool, if I only knew how :) | 08:15 |
| f_g | git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git is the upstream git repo | 08:15 |
| alkisg | I'm reading various sources, but I'm not sure which one to follow | 08:15 |
| alkisg | Ah, will this build me a debian package, or an initramfs anyway? | 08:15 |
| alkisg | I think https://wiki.ubuntu.com/Kernel/KernelBisection is using the ubuntu git trees that also contain a debian/ dir? | 08:16 |
| f_g | https://wiki.ubuntu.com/Kernel/KernelBisection#Bisecting_upstream_kernel_versions | 08:16 |
| f_g | and what is linked there | 08:16 |
| alkisg | f_g: so a rough idea is that git bisect will compile the kernel for me, and I'll then need to manually install it and calll update-initramfs ? | 08:17 |
| === Elimin8r is now known as Elimin8er | ||
| f_g | alkisg: git bisect just tells you which commits to test, the instructions tell you how you get the kernels from those commits. the generated kernel deb will call update-initramfs automatically after installing, just like a regular kernel from Ubuntu would | 08:19 |
| alkisg | f_g: what I can't understand in that workflow is how a kernel.org git will know to produce a .deb... | 08:19 |
| alkisg | I won't have the debian/ dir there, will it? | 08:20 |
| f_g | alkisg: the kernel build scripts have simple support for creating packages, both for .deb and .rpm | 08:20 |
| alkisg | Oh very nice, thank you | 08:20 |
| f_g | they are nowhere near as fancy as what RH/Debian/Ubuntu do, but for quick testing with integration into the package manager (especially for clean removal) they are sufficient | 08:21 |
| alkisg | OK, on to my first real kernel bisection then :) | 08:21 |
| f_g | alkisg: if you manage to compile one kernel successfully, bisecting is basically just repeating that over and over in a strategic manner. | 08:22 |
| alkisg | Oh the bisection logic was very easy to understand, it was the .deb part that I was missing | 08:22 |
| alkisg | f_g: I'm unsure what to put in series, in "build-mainline-one tag series". I tried "xenial" but it fails. | 08:53 |
| alkisg | Would that be "master"? | 08:53 |
| alkisg | http://termbin.com/rwuf | 08:55 |
| f_g | alkisg: not sure what you did there. running git bisect and then following https://wiki.ubuntu.com/KernelTeam/GitKernelBuild minus the clone and checkout (git bisect already gets you each commit checked out) should "just work" | 09:01 |
| alkisg | f_g: with a first look, I think the previous page that you linked required having a debian/ dir, and this one doesn't... trying this method now... | 09:02 |
| f_g | alkisg: the KernelBisection page I linked to has the link to GitKernelBuild right at the place where I linked to ;) | 09:04 |
| alkisg | f_g: but if you read that page, instead of following the second link to the second page, it says to use the mainline-build-one script, and it has instructions for bisecting etc, which I supposed was what I wanted :D Ty, trying... | 09:06 |
| f_g | alkisg: it does say to use, it says you can use ;) mainline-build-one is more Ubuntu-ish, GitKernelBuild is closer to upstream. both should be fine to find the culprit (provided your own build of rc5 shows the issue, and the one of rc4 does not - it's always good to verify this before starting a long bisect only to find out that your kernel config for bisecting does not match what Ubuntu used to build | 09:10 |
| f_g | their mainline PPA kernel deb ;)) | 09:10 |
| alkisg | Gotcha. On an i5-4440, how long a compilation would take? Half an hour? Two hours? | 09:10 |
| alkisg | Whoops disk space became an issue before compilation time :/ 10 GB may not be enough :( | 09:27 |
| f_g | alkisg: no clue, but once you built your first one you know ;) it's very consistent if your system is otherwise idle | 09:36 |
| alkisg | f_g: I hope I don't need to `make clean` on each bisection step, do I? (trying to save time...) | 09:37 |
| f_g | since you are potentially going back in time, you need to | 09:38 |
| alkisg | Whoops. I'll need another pc to work on while my good one compiles :D | 09:38 |
| alkisg | Thanks again | 09:38 |
| f_g | if compiling a single kernel takes > 45 minutes, maybe you can convince jsalisbury or apw to play bisect ping pong with you using Ubuntu build machines? | 09:39 |
| f_g | on a fast server a single kernel build is doable in a few minutes, but you need lots of cores for that ;) e.g., our EPYC with 16*2*2 threads does it in 6 minutes or so | 09:40 |
| apw | f_g, the exact config used is included in the patches in the output | 09:43 |
| alkisg | apw, I copied /boot/config...rc5 as a starting point, hope that's good enough for the bisection... | 09:45 |
| apw | alkisg, sound be the same indeed | 09:45 |
| alkisg | Cool, I hope each build will only take about half an hour on my i5, let's see... | 09:45 |
| alkisg | ...although now that I switched to the hdd instead of the ssd due to space issues, it seems to go much slower... :( | 09:47 |
| thresh | if you have 32G ram or so, the build should fit into that | 09:49 |
| f_g | apw: yep. IMHO still a good idea to verify the bisect assumption holds for your local build env before doing a whole bisect run for nought ;) | 09:49 |
| thresh | IIRC it takes me more than 20 gigabytes disk space to do a build | 09:50 |
| thresh | (also yeah epyc and nvme ssds are awesome) | 09:50 |
| alkisg | thresh: just 8 GB RAM here :/ | 09:50 |
| alkisg | Do builds produce both amd64 and i386 debs? Can I do the build itself from either an amd64 or i386 installation? | 09:51 |
| alkisg | (I'm currently using i386 because I wasn't sure if it produced both or just the running one) | 09:52 |
| alkisg | Yey, 46 minutes :) Rebooting... | 10:23 |
| lorddoskias | guys i just got the following message from latest HWE kernel: https://paste.ubuntu.com/26409755/ | 10:24 |
| lorddoskias | Linux fisk 4.13.0-26-generic #29~16.04.2-Ubuntu | 10:24 |
| alkisg | Yey, test kernel booted. Oh no, I misttyped 4.5 instead of 4.15 and it was a wasted effort. apw, since it takes 46 minutes, do you think it would be worth it to use the ubuntu builders for that? | 10:26 |
| === alkisg is now known as alkisg_web | ||
| === alkisg1 is now known as alkisg | ||
| alkisg | f_g: I've completed the first step of the bisection, "good". Now before continuing with the bisection, I'd like to test that "rc5=bad", any hint on how to do that without losing the current bisection progress? | 16:47 |
| alkisg | Hmm, although restarting the bisection doesn't mean I need to rebuild the first step, I can just answer it's good and get to step 2... :) | 16:58 |
| hallyn | jjohansen: just to confirm, your kernel fixed the issue - thanks! | 19:42 |
| jjohansen | ack | 19:48 |
| chiluk | Hey apw bjf, what's the plan on the Spectre_v2 mitigation? Are you guys planning on doing IBRS/IBPB or Retpoline compilation changes or both? | 20:40 |
| apw | ibrs to start with, then retpoline + ibrs when upstream catches up | 20:41 |
| chiluk | is upstream still working on repoline patches? I was under the impression that they had made it to stable now. | 20:41 |
| chiluk | yeah it really looks like retpoline is in linux-stable now. It should get pulled in with stable updates. | 20:42 |
| chiluk | also isn't retpoline + ibrs duplicate effort? even though you might have retpoline compiled in, only one gets used at run time. | 20:43 |
| chiluk | ^^ that's supposed to be a question | 20:43 |
| chiluk | I guess I should move to hardened. | 20:49 |
| apw | no some things need ibrs as well, and we'll wait for that to settle out | 20:51 |
| chiluk | apw are you referring to unrecompiled userspace? | 20:53 |
| chiluk | also is there any plan to enable gcc with retpoline and do a full archive recompile? | 20:53 |
| apw | I Amelia referring to CPUs which are not necessarily fixed with retpoline only | 20:53 |
| chiluk | I was unaware such cpus existed... | 20:54 |
| chiluk | retpoline is effectively simply disables speculitive execution for indirect branches | 20:55 |
| apw | retpoline makes it hard for the CPU speculate to useful code, skylake at least may be able to still speculate even with retpoline forms | 21:51 |
| chiluk | That'd be pretty impressive given my understanding of how retpoline functions.. it basically sandboxes the speculative branch in sleeps | 22:36 |
| ratliff | https://lwn.net/Articles/744193/ | 22:42 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!