| henrybucher | Hi got a question about compiling kernel 5.15.0 on 20.04.5. Is this the right room? | 01:18 |
|---|---|---|
| sarnold | it's not wrong, but there be something better .. might as well start in on it and see what happens | 01:20 |
| sarnold | (this channel can be very quiet for a very long time) | 01:20 |
| henrybucher | It's really a newbie question. I am compiling 5.15.0-48 from git://kernel.ubuntu.com/ubuntu/ubuntu-focal.git branch origin/hwe-5.15.0-next, modifying a few configs (tickless kernel) and generating the .deb files. | 01:23 |
| henrybucher | Im using devscripts, tryign to follow by the book. I edit the changelog file with `debchange` then recompile with `debian/rules binary` | 01:24 |
| henrybucher | The first time it added the +nohz suffix that I added in the changelog. However when I recompile the suffix is not there anymore. What could I possibly be doing wrong? | 01:25 |
| henrybucher | "suffix not there" meaning that the .deb files are not being generated with the suffix anymore | 01:25 |
| sarnold | "recompile" .. was that a similar debian/rules binary run? or something else? | 01:25 |
| henrybucher | yes by the book, with `LANG=C fakeroot debian/rules binary` | 01:26 |
| henrybucher | Wait I think I used `binary-generic` instead. Does it matter? | 01:26 |
| henrybucher | I just dont want to use the lowlatency kernel | 01:27 |
| sarnold | sorry, no idea there :/ | 01:27 |
| henrybucher | I'd think this would be something that happens to everyone? | 01:28 |
| sarnold | almost no one rebuilds their kernels.. | 01:30 |
| sarnold | and most people who build kernels go with the kernel.org ones instead | 01:30 |
| sarnold | (to be clear I'm not saying you're wrong, i just know that there's a lot of extra stuff in our package builds that you don't need to deal with when using the kernel.org ones; and usually people who want their own builds are trying to get a feature from a brand new kernel that isn't in our packages yet) | 01:31 |
| henrybucher | The only reason why I'm recompiling the kernel is that Ubuntu does not offer a tickless kernel (nohz_full) and I have a few experiments that need it | 01:40 |
| sarnold | we've got something like 60 kernels I can't believe one of them isn't tickless :) heh | 01:40 |
| henrybucher | For real there is none. | 01:41 |
| henrybucher | Even the lowlatency kernel is not. | 01:41 |
| sarnold | honestly that's not a real surprise, I'd kind of expect removing ticks to increase the latency of responses | 01:42 |
| sarnold | hmm, what's the kernel config setting for this? | 01:44 |
| sarnold | I see a bunch of kernels built with: CONFIG_HAVE_TIF_NOHZ=y | 01:44 |
| henrybucher | CONFIG_NOHZ_FULL | 01:46 |
| henrybucher | The use for it is to remove every single LOC interrupt from isolated cores. | 01:46 |
| sarnold | heh, I didn't even spot that with an rg NOHZ .. I wonder if that was introduced to the kernels after my most recent config sync.. | 01:46 |
| henrybucher | CONFIG_NO_HZ_FULL | 01:48 |
| henrybucher | this is not a new feature for sure | 01:49 |
| henrybucher | This is extremely important for low latency trading | 01:49 |
| henrybucher | We basically on a machine with 24 cores, we isolate 20 of them and leave 4 to handle everything else. On the 20 isolated there must be absolutely zero interrupts | 01:52 |
| henrybucher | And preempts | 01:53 |
| sarnold | henrybucher: hmm, maybe inspect one of the azure kernels to see if it meets your needs? https://termbin.com/rzd0 | 01:57 |
| henrybucher | Hmm that means I will have to reinstall the OS on my boxes... dont want to do that | 01:58 |
| henrybucher | is that kernel available straight from apt? | 01:59 |
| henrybucher | Do I need to add any line in to apt/source? | 01:59 |
| sarnold | maybe something like apt install linux-image-azure linux-image-generic- | 02:02 |
| sarnold | (please test on an appropriate machine :D | 02:02 |
| henrybucher | like a burner | 02:03 |
| henrybucher | thing is im just so close, the kernels are compiling fine and I'm even typign this from within one of them | 02:05 |
| henrybucher | its just this damn suffix | 02:06 |
| sarnold | oh hmmm when you're *that* close to your goal.. | 02:06 |
| henrybucher | :) I will keep digging but thanks a bunch for the tips | 02:07 |
| henrybucher | I think it's something with the order of clean/distclean/debchange/binary | 02:08 |
| sarnold | I suspect most kernel package builds go through sbuild or similar, something with a bit more process behind it. getting *that* set up is of course also a bit of work.. | 02:11 |
| henrybucher | I got it to compile. I think I was running `clean` after `debchange` so my changes were being overwritten. Looks like `clean` means more than `make clean`, it regenerates everything from scratch. | 03:45 |
| sarnold | sweet!! | 03:45 |
| sarnold | thanks for reporting back :D | 03:45 |
| henrybucher | Im very thankful for you hanging in there and helping out | 03:46 |
| sarnold | my pleasure :) | 03:47 |
| henrybucher | The final product: https://lucisqr.substack.com/p/compiling-a-tickless-kernel-on-ubuntu?sd=pf | 03:54 |
| sarnold | oh that's *cool* | 03:56 |
| henrybucher | if you find anything wrong in there, please let me know | 03:57 |
| sarnold | nice writeup; I'm a bit confused how the isolated cpus are selected; does the kernel automatically set aside two? three? for handling interrupts and nothing else? | 04:02 |
| henrybucher | kernel command line | 04:03 |
| henrybucher | you pass "isol_cpus=21-23" | 04:03 |
| sarnold | aha; is that inclusive at both endpoints or inclusive at one and exclusive at the other? | 04:05 |
| sarnold | what do those cpus wind up *doing* if there's no processes on them that are submitting IOs and so on? | 04:05 |
| henrybucher | we do have special "pinned" processes that eventually run on them. We plan and pair threads and cores | 04:41 |
| henrybucher | In a real production machine the ratio would be more 4 cpus for general use and 20 cpus for trading | 04:42 |
| henrybucher | we move all IRQs to these 4 "system" CPUs | 04:42 |
| henrybucher | IO on the isolated cores is pretty much done with kernel bypass | 04:42 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!