[01:18] <henrybucher> Hi got a question about compiling kernel 5.15.0 on 20.04.5. Is this the right room?
[01:20] <sarnold> it's not wrong, but there be something better .. might as well start in on it and see what happens
[01:20] <sarnold> (this channel can be very quiet for a very long time)
[01:23] <henrybucher> It's really a newbie question. I am compiling 5.15.0-48 from git://kernel.ubuntu.com/ubuntu/ubuntu-focal.git branch origin/hwe-5.15.0-next, modifying a few configs (tickless kernel) and generating the .deb files.
[01:24] <henrybucher> Im using devscripts, tryign to follow by the book. I edit the changelog file with `debchange` then recompile with `debian/rules binary`
[01:25] <henrybucher> The first time it added the +nohz suffix that I added in the changelog. However when I recompile the suffix is not there anymore. What could I possibly be doing wrong? 
[01:25] <henrybucher> "suffix not there" meaning that the .deb files are not being generated with the suffix anymore
[01:25] <sarnold> "recompile" .. was that a similar debian/rules binary   run? or something else?
[01:26] <henrybucher> yes by the book, with `LANG=C fakeroot debian/rules binary`
[01:26] <henrybucher> Wait I think I used `binary-generic` instead. Does it matter?
[01:27] <henrybucher> I just dont want to use the lowlatency kernel
[01:27] <sarnold> sorry, no idea there :/
[01:28] <henrybucher> I'd think this would be something that happens to everyone?
[01:30] <sarnold> almost no one rebuilds their kernels..
[01:30] <sarnold> and most people who build kernels go with the kernel.org ones instead
[01:31] <sarnold> (to be clear I'm not saying you're wrong, i just know that there's a lot of extra stuff in our package builds that you don't need to deal with when using the kernel.org ones; and usually people who want their own builds are trying to get a feature from a brand new kernel that isn't in our packages yet)
[01:40] <henrybucher> The only reason why I'm recompiling the kernel is that Ubuntu does not offer a tickless kernel (nohz_full) and I have a few experiments that need it
[01:40] <sarnold> we've got something like 60 kernels I can't believe one of them isn't tickless :) heh
[01:41] <henrybucher> For real there is none. 
[01:41] <henrybucher> Even the lowlatency kernel is not.
[01:42] <sarnold> honestly that's not a real surprise, I'd kind of expect removing ticks to increase the latency of responses
[01:44] <sarnold> hmm, what's the kernel config setting for this?
[01:44] <sarnold> I see a bunch of kernels built with: CONFIG_HAVE_TIF_NOHZ=y
[01:46] <henrybucher> CONFIG_NOHZ_FULL
[01:46] <henrybucher> The use for it is to remove every single LOC interrupt from isolated cores. 
[01:46] <sarnold> heh, I didn't even spot that with an rg NOHZ .. I wonder if that was introduced to the kernels after my most recent config sync..
[01:48] <henrybucher> CONFIG_NO_HZ_FULL
[01:49] <henrybucher> this is not a new feature for sure
[01:49] <henrybucher> This is extremely important for low latency trading
[01:52] <henrybucher> We basically on a machine with 24 cores, we isolate 20 of them and leave 4 to handle everything else. On the 20 isolated there must be absolutely zero interrupts
[01:53] <henrybucher> And preempts
[01:57] <sarnold> henrybucher: hmm, maybe inspect one of the azure kernels to see if it meets your needs? https://termbin.com/rzd0
[01:58] <henrybucher> Hmm that means I will have to reinstall the OS on my boxes... dont want to do that
[01:59] <henrybucher> is that kernel available straight from apt? 
[01:59] <henrybucher> Do I need to add any line in to apt/source? 
[02:02] <sarnold> maybe something like apt install linux-image-azure linux-image-generic-
[02:02] <sarnold> (please test on an appropriate machine :D
[02:03] <henrybucher> like a burner
[02:05] <henrybucher> thing is im just so close, the kernels are compiling fine and I'm even typign this from within one of them
[02:06] <henrybucher> its just this damn suffix
[02:06] <sarnold> oh hmmm when you're *that* close to your goal..
[02:07] <henrybucher> :) I will keep digging but thanks a bunch for the tips
[02:08] <henrybucher> I think it's something with the order of clean/distclean/debchange/binary
[02:11] <sarnold> I suspect most kernel package builds go through sbuild or similar, something with a bit more process behind it. getting *that* set up is of course also a bit of work..
[03:45] <henrybucher> I got it to compile. I think I was running `clean` after `debchange` so my changes were being overwritten. Looks like `clean` means more than `make clean`, it regenerates everything from scratch.
[03:45] <sarnold> sweet!!
[03:45] <sarnold> thanks for reporting back :D
[03:46] <henrybucher> Im very thankful for you hanging in there and helping out
[03:47] <sarnold> my pleasure :)
[03:54] <henrybucher> The final product: https://lucisqr.substack.com/p/compiling-a-tickless-kernel-on-ubuntu?sd=pf
[03:56] <sarnold> oh that's *cool*
[03:57] <henrybucher> if you find anything wrong in there, please let me know
[04:02] <sarnold> nice writeup; I'm a bit confused how the isolated cpus are selected; does the kernel automatically set aside two? three? for handling interrupts and nothing else?
[04:03] <henrybucher> kernel command line
[04:03] <henrybucher> you pass "isol_cpus=21-23" 
[04:05] <sarnold> aha; is that inclusive at both endpoints or inclusive at one and exclusive at the other?
[04:05] <sarnold> what do those cpus wind up *doing* if there's no processes on them that are submitting IOs and so on?
[04:41] <henrybucher> we do have special "pinned" processes that eventually run on them. We plan and pair threads and cores
[04:42] <henrybucher> In a real production machine the ratio would be more 4 cpus for general use and 20 cpus for trading
[04:42] <henrybucher> we move all IRQs to these 4 "system" CPUs
[04:42] <henrybucher> IO on the isolated cores is pretty much done with kernel bypass