leon_peggGood morning all, I would like to apply a patch to the kernel package and would like some guidance could someone please advise.10:53
leon_peggto provide a bit more information I am looking to patch the kernel with grsecurity10:55
apwok, ask ay specific questions you have and we'll try and answer10:56
leon_peggHow do i apply a patch to the kernel source package10:57
apwleon_pegg, to be honest if applying a patch is a new thing you are going to find applying a monster like grsecurity a huge effort10:58
apwleon_pegg, but ... i would start by checking out our git repository for the release to which you intend to apply it10:59
leon_peggapw: I have applied patches to raw source before but never to packages, and have built custom kernels before just not applied patches to deb packages11:00
leon_peggapw: whats the git repo address?11:00
apwleon_pegg, was struggling to find it in our wiki *sigh* ... https://wiki.ubuntu.com/Kernel/SourceCode see the git bit there11:01
apwleon_pegg, i would then apply the patch in git and build the source package from that11:01
leon_peggapw: thanks 11:01
leon_peggapw: fingers crossed it all goes well :D11:02
apwleon_pegg, also, didn't grseurity stop producing patches in public anyhow ?11:04
leon_peggapw: they still provide the testing patches to keep supporting arch-hardened 11:05
apwleon_pegg, against what kernel versions though11:06
leon_peggapw: kernel versions 3.1-4.3.411:07
apwso you might have some luck on 4.2 in wily then, but xenial is too new11:13
leon_peggapw: using wily on the desktops in the office so we should be good, we are still is discussions about applying grsec on the servers11:19
apwleon_pegg, that is a big committment for sure, as you'll need to keep up to date with security release cadance11:19
leon_peggapw: exactly it is a huge commitment, and keeping up with security updates adds additional work. the plan at the moment it to trial on some of the desktops and progress from there after11:24
MaikZHi, is there an escalation procedure we could follow to get some more attention on https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1398465 ?13:27
ubot5Launchpad bug 1398465 in linux (Ubuntu) "Memory allocation failure, presumably in FUSE" [Medium,Invalid]13:27
MaikZOops wrong bug sorry13:27
MaikZWell actually same bug, but https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505948 is the version we reported :-)13:28
ubot5Launchpad bug 1505948 in linux (Ubuntu Wily) "Memory allocation failure crashes kernel hard, presumably related to FUSE" [High,Confirmed]13:28
MaikZThere is an out of bounds write somewhere in fuse_direct_io on 4.1 and higher that can panic the kernel (and reliably does, on our virtualization hosts)13:28
MaikZWe're working around it by staying on 3.19, but that is coming uncomfortably close to being out of support we think13:29
apwMaikZ, that doesn't look like a memory allocation failure, more like a memory arena corruption, a double free or something13:29
MaikZapw: Yes13:29
MaikZWe booted a test machine with slub_debug13:29
MaikZAnd it confirms that something in fuse_direct_io wrote zero bytes into the "poison" areas13:29
MaikZWithout slub_debug, that makes the kernel panic on some future allocation or deallocation.13:30
apwis that info in the bug (save me reading it)13:30
apwMaikZ, also are you able to test the 4.4 kernel in xenial-proposed easily for this ?13:31
MaikZIf there's a .deb that installs on trusty, we should be able to do that fairly quickly yes13:31
MaikZNot yet, our storage vendor was supposed to add it but apparently didn't. I'll attach it.13:32
apwMaikZ, well in principle those are nistallable there with lick13:32
apwMaikZ, and this won't affect "regular" consumers of ubuntu i assume, as you need to use fuse 3.0 to get async io right ?13:33
MaikZNot sure about the FUSE versioning, but I can confirm that the specific FUSE file system we use ships with its own, very recent libfuse.13:35
apwand presumably you do not see this without13:35
MaikZWe have been able to reproduce the kernel crash using ntfs-3g because fuse-devel was unhappy about a closed-source userspace part13:36
MaikZBut that was also a fresh build13:36
MaikZSo for testing 4.4, we could try grabbing .debs from this build: https://launchpad.net/ubuntu/+source/linux/4.4.0-1.15/+build/888082413:44
MaikZAnd they should sort of work on trusty because kernel packaging hasn't fundamentally changed?13:44
apwMaikZ, that would be my expectation, do get -extra if you need it13:47
MaikZOkay, one machine cleared of production workloads and 4.4 installed, let's see what happens.13:59
apwMaikZ, i think it will break just the same, but that a good data point, i have found a suspicious  looking async path13:59
apwMadkiss, ?14:07
Madkissapw: MaikZ told me there's an interesting conversation going on here about a bug we reported several months ago, so I just thought i'd join and read :)14:09
apwMaikZ, assuming that machine also goes pop, i might have a guess as to the cuase of this, if so then i might have a test kernel to try (soon)14:10
apwMaikZ, if so, 1) would you be able to test it, and 2) i would need to confirm its not leaking instead of blowing up, so it'd need some monitoring14:11
MaikZWe can install a custom build. What monitoring do you have in mind?14:12
MaikZThere is sadly no metrics collection in that cluster.14:12
MaikZSo I can't draw you pretty memory usage graphs.14:12
apwMaikZ, well if it is the right fix then the machine will not blow up and things will be fine, if it is not then the machine may work fine and leak 96 byte memory blocks for every async IO and lose memeory over time14:18
apwMaikZ, so I guess I am syaing we need to monitor the slabs are not growing out of control in this case14:18
apwMaikZ, if your screenshot is accurate then like:  while :; do grep kmalloc-96 /proc/slabinfo; sleep 5; done14:19
apwwould be something to watch is similar to other machines14:20
apwMaikZ, i assume if i am wrong then it will literally lose an entry for every single IO, so it should be obvious and spectacular14:21
MaikZSpectacular failure is our core business!14:21
MaikZI'm so far failing to reproduce the previous issue (slub_debug is silent, also no panic), but instead I/O has totally frozen on the test VM.14:33
MaikZWith the 4.4 kernel.14:33
MaikZOr looks like it...some ops are getting through according to storage metrics.14:34
gQuigsany thoughts on disabling mtrr for xenial?   https://bugs.launchpad.net/ubuntu/+source/linux/+bug/116358714:42
ubot5Launchpad bug 1163587 in linux (Ubuntu) "mtrr_cleanup: can not find optimal value, perhaps no longer needed?" [Medium,Triaged]14:42
apwMaikZ, ok, after some further reading it doesn't look like that suspicious thing is suspicious15:33
MaikZI will let the 4.4 tests run overnight, the kernel might panic after a couple of hours. It usually happens within minutes but not always.15:56
apwMaikZ, sounds good thanks16:02
apwMaikZ, put any result info in the bug and let us know too here16:02

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!