| leon_pegg | Good morning all, I would like to apply a patch to the kernel package and would like some guidance could someone please advise. | 10:53 |
|---|---|---|
| leon_pegg | to provide a bit more information I am looking to patch the kernel with grsecurity | 10:55 |
| apw | ok, ask ay specific questions you have and we'll try and answer | 10:56 |
| apw | *any | 10:56 |
| leon_pegg | How do i apply a patch to the kernel source package | 10:57 |
| apw | leon_pegg, to be honest if applying a patch is a new thing you are going to find applying a monster like grsecurity a huge effort | 10:58 |
| apw | leon_pegg, but ... i would start by checking out our git repository for the release to which you intend to apply it | 10:59 |
| leon_pegg | apw: I have applied patches to raw source before but never to packages, and have built custom kernels before just not applied patches to deb packages | 11:00 |
| leon_pegg | apw: whats the git repo address? | 11:00 |
| apw | leon_pegg, was struggling to find it in our wiki *sigh* ... https://wiki.ubuntu.com/Kernel/SourceCode see the git bit there | 11:01 |
| apw | leon_pegg, i would then apply the patch in git and build the source package from that | 11:01 |
| leon_pegg | apw: thanks | 11:01 |
| leon_pegg | apw: fingers crossed it all goes well :D | 11:02 |
| apw | leon_pegg, also, didn't grseurity stop producing patches in public anyhow ? | 11:04 |
| leon_pegg | apw: they still provide the testing patches to keep supporting arch-hardened | 11:05 |
| apw | leon_pegg, against what kernel versions though | 11:06 |
| leon_pegg | apw: kernel versions 3.1-4.3.4 | 11:07 |
| apw | so you might have some luck on 4.2 in wily then, but xenial is too new | 11:13 |
| leon_pegg | apw: using wily on the desktops in the office so we should be good, we are still is discussions about applying grsec on the servers | 11:19 |
| apw | leon_pegg, that is a big committment for sure, as you'll need to keep up to date with security release cadance | 11:19 |
| leon_pegg | apw: exactly it is a huge commitment, and keeping up with security updates adds additional work. the plan at the moment it to trial on some of the desktops and progress from there after | 11:24 |
| MaikZ | Hi, is there an escalation procedure we could follow to get some more attention on https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1398465 ? | 13:27 |
| ubot5 | Launchpad bug 1398465 in linux (Ubuntu) "Memory allocation failure, presumably in FUSE" [Medium,Invalid] | 13:27 |
| MaikZ | Oops wrong bug sorry | 13:27 |
| MaikZ | Well actually same bug, but https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505948 is the version we reported :-) | 13:28 |
| ubot5 | Launchpad bug 1505948 in linux (Ubuntu Wily) "Memory allocation failure crashes kernel hard, presumably related to FUSE" [High,Confirmed] | 13:28 |
| MaikZ | There is an out of bounds write somewhere in fuse_direct_io on 4.1 and higher that can panic the kernel (and reliably does, on our virtualization hosts) | 13:28 |
| MaikZ | We're working around it by staying on 3.19, but that is coming uncomfortably close to being out of support we think | 13:29 |
| apw | MaikZ, that doesn't look like a memory allocation failure, more like a memory arena corruption, a double free or something | 13:29 |
| MaikZ | apw: Yes | 13:29 |
| MaikZ | We booted a test machine with slub_debug | 13:29 |
| MaikZ | And it confirms that something in fuse_direct_io wrote zero bytes into the "poison" areas | 13:29 |
| MaikZ | Without slub_debug, that makes the kernel panic on some future allocation or deallocation. | 13:30 |
| apw | is that info in the bug (save me reading it) | 13:30 |
| apw | MaikZ, also are you able to test the 4.4 kernel in xenial-proposed easily for this ? | 13:31 |
| MaikZ | If there's a .deb that installs on trusty, we should be able to do that fairly quickly yes | 13:31 |
| MaikZ | Not yet, our storage vendor was supposed to add it but apparently didn't. I'll attach it. | 13:32 |
| apw | MaikZ, well in principle those are nistallable there with lick | 13:32 |
| apw | luck | 13:32 |
| apw | MaikZ, and this won't affect "regular" consumers of ubuntu i assume, as you need to use fuse 3.0 to get async io right ? | 13:33 |
| MaikZ | Not sure about the FUSE versioning, but I can confirm that the specific FUSE file system we use ships with its own, very recent libfuse. | 13:35 |
| apw | and presumably you do not see this without | 13:35 |
| MaikZ | We have been able to reproduce the kernel crash using ntfs-3g because fuse-devel was unhappy about a closed-source userspace part | 13:36 |
| MaikZ | But that was also a fresh build | 13:36 |
| MaikZ | So for testing 4.4, we could try grabbing .debs from this build: https://launchpad.net/ubuntu/+source/linux/4.4.0-1.15/+build/8880824 | 13:44 |
| MaikZ | And they should sort of work on trusty because kernel packaging hasn't fundamentally changed? | 13:44 |
| apw | MaikZ, that would be my expectation, do get -extra if you need it | 13:47 |
| MaikZ | Okay, one machine cleared of production workloads and 4.4 installed, let's see what happens. | 13:59 |
| apw | MaikZ, i think it will break just the same, but that a good data point, i have found a suspicious looking async path | 13:59 |
| Madkiss | g'day | 14:01 |
| apw | Madkiss, ? | 14:07 |
| Madkiss | apw: MaikZ told me there's an interesting conversation going on here about a bug we reported several months ago, so I just thought i'd join and read :) | 14:09 |
| apw | MaikZ, assuming that machine also goes pop, i might have a guess as to the cuase of this, if so then i might have a test kernel to try (soon) | 14:10 |
| apw | MaikZ, if so, 1) would you be able to test it, and 2) i would need to confirm its not leaking instead of blowing up, so it'd need some monitoring | 14:11 |
| MaikZ | We can install a custom build. What monitoring do you have in mind? | 14:12 |
| MaikZ | There is sadly no metrics collection in that cluster. | 14:12 |
| MaikZ | So I can't draw you pretty memory usage graphs. | 14:12 |
| apw | MaikZ, well if it is the right fix then the machine will not blow up and things will be fine, if it is not then the machine may work fine and leak 96 byte memory blocks for every async IO and lose memeory over time | 14:18 |
| apw | MaikZ, so I guess I am syaing we need to monitor the slabs are not growing out of control in this case | 14:18 |
| apw | MaikZ, if your screenshot is accurate then like: while :; do grep kmalloc-96 /proc/slabinfo; sleep 5; done | 14:19 |
| apw | would be something to watch is similar to other machines | 14:20 |
| apw | MaikZ, i assume if i am wrong then it will literally lose an entry for every single IO, so it should be obvious and spectacular | 14:21 |
| MaikZ | Spectacular failure is our core business! | 14:21 |
| MaikZ | I'm so far failing to reproduce the previous issue (slub_debug is silent, also no panic), but instead I/O has totally frozen on the test VM. | 14:33 |
| MaikZ | With the 4.4 kernel. | 14:33 |
| MaikZ | Or looks like it...some ops are getting through according to storage metrics. | 14:34 |
| gQuigs | any thoughts on disabling mtrr for xenial? https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1163587 | 14:42 |
| ubot5 | Launchpad bug 1163587 in linux (Ubuntu) "mtrr_cleanup: can not find optimal value, perhaps no longer needed?" [Medium,Triaged] | 14:42 |
| apw | MaikZ, ok, after some further reading it doesn't look like that suspicious thing is suspicious | 15:33 |
| MaikZ | I will let the 4.4 tests run overnight, the kernel might panic after a couple of hours. It usually happens within minutes but not always. | 15:56 |
| apw | MaikZ, sounds good thanks | 16:02 |
| apw | MaikZ, put any result info in the bug and let us know too here | 16:02 |
| MaikZ | Sure. | 16:04 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!