[00:05] Riddell, have start ed off the kubuntu alternates build. They should show up on the iso tracker shortly. [00:06] * skaet has updated the pad to make the rebuilds include them. [00:21] infinity, ogra_ can you look at the bug that hggdh has flagged? [00:21] bug 1028664 [00:21] Launchpad bug 1028664 in debian-installer "keyboard does not work on quantal-server-armhf+omap4.img" [Critical,New] https://launchpad.net/bugs/1028664 [00:23] Riddell, ScottK, kubuntu alternates now posted. [00:23] skaet: I can try to reproduce/confirm it, I probably don't have the spare cycles to spend time debugging and fixing it. [00:24] infinity, confirming it would be good. We can decide what to do tomorrow after Europe comes online. [00:25] slangasek, ^ FYI. [00:26] hggdh: is it only the server image that's affected? [00:26] My guess is that Oli only tested this via serial (which is how Real Men install servers), but I'm grabbing an image to see if I get console response with a USB keyboard. [00:28] If it works on serial but not a USB console, though, I wouldn't call that an Alpha blocker, just an annoyance that should be fixed before final release. [00:33] * skaet nods [00:39] Well, that's cute. I get no USB keyboard support, but I also get no serial console. [00:39] I wonder how this is meant to work. :P [00:40] hmm [00:40] this is with the all-new squashfs stuff, right? [00:40] has anyone *actually* tested it interactively up to now? :P [00:40] Yeah, I imagine so, but I'm not sure how that should matter. [00:41] The "make bootable" bits should be the same. [00:41] Which should include sane commandlines, etc. [00:41] ah, true enough [00:42] Well, the lack of serial console could/should be command-line issues. [00:43] The USB keyboard thing seems more like a "blame the kernel" deal. [00:43] But someone will need to fix A to debug B, perhaps. :P [00:43] the kernel, or the initramfs [00:43] Well, the kernel, or the (lack of) kernel modules in the initrd? [00:43] yes [00:43] That's all "kernel" to me. ;) [00:44] pretty sure the bugs in the latter are the responsibility of the foundations team to fix ;) [00:45] Well, yes. I wasn't referring to the team, but the bit that's not working. ;) [00:45] Anyhow, let me have a quick poke at this card and see why it thinks I don't deserve a serial console. [00:46] ... because it has a boot.scr-serial alongside boot.scr that isn't the default. Intuitive! [00:49] [ 3.334655] usb 1-1.3: New USB device found, idVendor=046d, idProduct=c31c [00:49] [ 3.342926] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [00:50] [ 3.350891] usb 1-1.3: Product: USB Keyboard [00:50] [ 3.355651] usb 1-1.3: Manufacturer: Logitech [00:50] Sure looks there to me... [00:50] and does it work? [00:51] Yeah, that's harder to tell, when d-i is booted to serial. [00:51] No console. [00:51] * slangasek nods [00:51] hmm, where did you see this boot.scr-serial? [00:51] * infinity assumes there must be a cake and eat it too option. [00:51] slangasek: In the vfat partition. [00:52] I just downloaded the daily server image linked from the iso tracker, and I don't see that [00:52] oh [00:52] that wasn't a server image, sigh [00:53] * slangasek tries again [01:01] infinity: by the by, where does the boot.scr get generated? [01:02] slangasek: On nusakan. debian-cd/tools/boot/quantal/post-boot.* [01:02] Ish. [01:02] ok. because I think it would be just peachy if we would also include a suitable source representation on the image [01:03] without which it's bloody annoying if you have to ever edit the commandline for booting on a panda [01:03] You mean you don't like editing out the binary bits with vim before you redo it? :) [01:03] I kinda don't [01:03] Heh. [01:03] Yeah, including the matching boot.script would be simple enough. [01:03] I can commit that right now. [01:04] I can commit that as soon as I get a gnome-terminal that works... [01:05] infinity, slangasek, cp boot.scr-serial boot.scr .... on the first partition of the SD [01:05] ogra_: Yeah, we found it. [01:05] oh, ok [01:05] then i'm actually off to bed [01:06] ogra_: Does the kernel not do sane autodetection like it can on x86? :( [01:06] autodetection of what ? [01:07] On x86, if you have a physical console attached (ie: keyboard and monitor), you get a framebuffer d-i, if you don't, it falls back to serial. [01:07] Which is much handier. [01:08] ah, no, i have never seen that working [01:08] I'm entirely prepared to believe that the omap kernel doesn't give us enough info to pull that trick, but it's also possible that we could and we just don't. [01:08] on omap i even get a broken installer if i dont force the framebuffer off when using serial [01:09] you end up with display on framebuffer and input on serial [01:10] Well, that would be better than the current situation, which is no input. ;) [01:10] that sounds like a kernel bug though [01:11] anyway, i'll happily look into that if its not 3am ... really off to bed now [01:17] slangasek: Alright, future spins should have boot.script and boot.script-serial on the SD to match the compiled versions. [01:17] infinity: ta :) [01:22] So, getting as far as installing the base system, then chrooting in and running a getty on tty1, my keyboard works. [01:23] Kinda wasn't expecting it to... [01:23] Ahh, no. There it is. [01:24] It was during the hardware detection stage of d-i, the keyboard sprung into existence as an input device. [01:24] That seems like suboptimal timing. [01:25] Yeah, the keyboard doesn't work in the initramfs. Which is kinda annoying, because it doesn't find my root device correctly, so dumps me in the initramfs... [01:26] RAOF: Well, in this case, it doesn't work until much after you need it to press the "make it work" key. :P [01:26] RAOF: Which is even worse. [01:26] Hah! [01:26] infinity: do you know why it's picked up so late? [01:27] is it not udev-driven? [01:27] slangasek: I imagine the module just isn't around. [01:27] is this not booting to the squashfs? [01:28] No, the squash is used as source media only, I think. [01:28] It's booting to a d-i initrd. [01:28] ok [01:28] At least, that's what it looks like to me. [01:28] This is the first time I've seen this new magic. :P [01:28] :) [01:29] If I had to guess, I'd just say the hid* stuff isn't available on boot. [01:29] But that requires more digging. [01:31] Actually, I guess just rebooting would make that easy to confirm. [01:31] Yeah, no /lib/modules/3.4.0-204-omap4/kernel/drivers/hid on the boot image. [01:31] So, it gets pulled in later via udebs. Which is, of course, useless. [01:32] * slangasek nods [01:32] is this initrd being generated by d-i, or on nusakan? [01:33] The latter, I suspect. But that's another "not sure". [01:33] Thankfully, debian-cd is very sane and reasonably hackab... Oh, wait. [01:34] comes from d-i [01:34] via tools/boot/quantal/boot-armhf+omap4 [01:34] ubuntu/dists/quantal/main/installer-armhf/20101020ubuntu162/images/omap4/cdrom/ [01:34] Ahh, kay. [01:34] Oh, right. The cdrom bits. [01:34] Derp. [01:34] I knew that. [01:34] And I'm betting no one's dusted those off in, like... Ever. [01:39] So, looks like I need input-modules-$ver.udeb [01:39] build/pkg-lists/cdrom/armel/omap.cfg -> build/pkg-lists/cdrom/armel/omap4.cfg? [01:39] I just got there. ;) [01:39] ok [01:39] you want to follow it through? [01:39] Yep. [01:43] netboot wanted the same treatment. [01:43] You know, I think I even remember someone once claiming that netboot didn't work with a USB NIC. [01:43] That would be why. [01:43] * infinity shakes his head. [01:47] While we're on the subject, how can I debug why my netinst install can't find it's root on the USB device? [01:50] RAOF: Also armhf+omap4? [01:50] Yes. [01:50] precise or quantal? [01:50] From the quantal netinst images, circa 3 days ago or so. [01:51] Define "find its root". [01:51] The kernel boots, and then no root? [01:51] Or no kernel? [01:51] Kernel boots, dumps me in initramfs having not found /bin/init [01:51] At this point I can't really tell why it hasn't found that, because no keyboard in the initramfs ;) [01:51] Oh, this reminds me of another d-i bug I was going to fix. But it shouldn't present that way... [01:52] Oli was telling me that flash-kernel still just doesn't work at all, because our d-i images don't have a partition table that makes flash-kernel happy. Or something. [01:53] But that would present as you ending up with the installer in an infinite loop on reboot, not what you described. [01:53] Meh, I need to wipe one of my USB drives, so I can test both of these bugs and see WTF is going on. [01:54] Or go rustle up a USB stick. [01:54] Or you could make it so that the keyboard works in the initramfs, and I could tell you what had actually gone wrong :) [01:54] RAOF: Anyhow, I'm pretty sure netboot is "known not-working", but if I can fix it in the same d-i upload with minimal effort, what the heck. [01:55] RAOF: You netbooted... Use a serial console? [01:55] I don't *have* a serial console. [01:56] netboot-fb for me. [01:56] I can see how that would be frustrating. ;) [01:56] That said, I'm not sure that dumping all the usb/hid stuff in every initramfs is the right answer. [01:56] Though, maybe it is for omap. I dunno. [01:56] I'll ponder that later. [01:59] ... as he gets halfway through the install before noticing this isn't netboot. [01:59] La la la. [02:01] Reason number 34 for a local mirror: [02:01] root@vishnu:~# zcat /srv/mirrors.0c3.net/ubuntu-ports/dists/quantal/main/installer-armhf/current/ima [02:01] ges/omap4/netboot/boot.img-serial.gz > /dev/mmcblk1 [02:16] RAOF: I'd suggest you try the new server image, but. Oops. No keyboard. ;) [02:18] kenvandine: freeze anyone? [02:19] micahg, i was fixing the ftbfs from earlier [02:20] kenvandine: ah, hrm, okies [02:20] FTBFS fixes still change the archive. :P [02:20] failed on i386 [02:22] That's a curious failure. [02:23] indeed [02:25] Okay, bored with debugging netboot, want cookies. [02:25] Uploading the d-i fix for the server keyboard issue. [02:37] kenvandine: I was able to replicate the FTBFS in an i386 quantal chroot [02:38] i could do once [02:38] but running the tests a second time they pass [02:38] it's very odd [02:38] and they pass first time outside of a package build [02:55] ... [02:55] RAOF: So, uhm. quantal/omap4 netboot works flawlessly for me. [02:56] RAOF: I'm now very confused as to why people claim it doesn't. :P [02:56] infinity: Maybe I just need to flash a new image? Has it changed in the last couple of days? [02:56] * infinity wonders if the -fb variant is somehow broken when the -serial one works, and tries that. [02:57] RAOF: Well, d-i has been rebuilt, I didn't think that any of the things that were supposedly broken have been unbroken. [02:57] RAOF: But... It all works for me. [02:58] RAOF: (on serial... Testing FB now) [02:59] RAOF: Are you doing any funky partitioning, just just doing Guided-whole-disk? [02:59] RAOF: It could be that partman is doing broken things in some cases, maybe. [03:00] Man, 24 inches of HD aubergine is kinda jarring. [03:00] The most recent attempt was guided-whole-disk, replacing the ext4 partition with a btfrs one. I've also tried a straight guided-whole-disk in the past. [03:02] RAOF: It's possible that between Luke's upload on the 20th and Oli's uploads yesterday, it all Just Works. [03:02] RAOF: But none of those address the things Oli claimed were broken, so I suspect he lied to me. :) [03:02] (But working is working, I won't complain) [03:03] Of course, someone should fix partman thinking that I want or care about a /boot partition on this thing, but that's a minor nit, if the rest works. [03:04] queuebot: That's a filthy lie and you know it, they're still building. [03:05] RAOF: So, yeah, skip the current version, cause it's obsolete in 20 minutes, but please do try a 20101020ubuntu163 netboot when it's published. [03:05] RAOF: I did make one change to it, so a confirmation that it's not buggered would be nice. [03:06] RAOF: (My 20101020ubuntu162 run will finish shortly to confirm that it worked) [03:09] infinity: Oh, it doesn't need a /boot? [03:10] RAOF: No, not that it hurts to leave the default partitioning in (which I've done in my quick test here). [03:10] RAOF: Given that, by default, the kernel ends up on the SD card, a /boot makes no sense. [03:11] RAOF: In the case of a uboot on SD using extload to grab the kernel from the hard drive, it *might* make sense, not sure if uboot has issues with huge disks. But we don't boot that way by default anyway. [03:20] infinity: You know you've still got e2fsprogs in the precise-unapproved awaiting the pleasure of your review, right? [03:20] RAOF: Yeah, xnox and I talked about it in Nicaragua. And then we took the rest of the month off. [03:21] RAOF: I should probably just reject it and work with him on it later. [03:21] There. [03:21] :) [03:21] RAOF: In exchange, review my d-i. [03:21] (Ideally not the same way I just reviewed xnox's upload) [03:22] Most assuredly! [03:23] RAOF: Actually, don't. [03:23] RAOF: Well, review, but don't accept. [03:23] RAOF: It'll just lead to a no-change upload later to rebuild against Colin's d-i-utils upload sitting there. [03:24] Shall I eat some quiche until that gets done? :) [03:24] * infinity quickly reviews that. [03:24] RAOF: Real men, quiche, etc. [03:26] RAOF: Oh, did that mesa thing go before the TB or something, or were you feeling generous? [03:26] RAOF: (I'm deliberately avoiding option C, which is that you reviewed the whole thing, but if you did, wow) [03:26] Went before the TB [03:26] Cool. [03:26] It looked like the right thing to do, IMO, but I sure as heck wasn't going to audit it. [03:28] RAOF: When 'rmadison di-utils' claims something vaguely plausible about being published on 5 arches in precise-proposed, go nuts with debian-installer. Just didn't make sense to me to upload twice in a row. [03:29] * infinity thinks he might go find a cookie before the shops all close. [03:30] Win [04:00] infinity: ^^ I think this is happy enough for proposed [04:02] infinity: maya quiche? [04:06] RAOF: FWIW, the framebuffery netboot worked for me too. [04:07] infinity: uploading to quantal-proposed? [04:07] or is it already there? [04:07] slangasek: d-i is in quantal already. I was referring to testing RAOF's issues with the previous version, though. [04:08] (Or rather, testing with the previous version, his issues were obviously with an older version) [04:14] ok [04:30] slangasek: I you want to respin an omap4 server image, the new d-i should be ready for it. [04:50] RAOF: The di-utils SRU is published on all arches -- do with that information what you will. :) [04:50] infinity: This is an ack on your debian-installer upload. I see armel and ppc are being slowpokes; if you notice that they've finished before me, go for it. [04:50] Hah! Overlap :) [04:50] :P [04:51] So, yeah. All done now on ftpmaster, which is all that matters to the buildds. [04:51] rmadison is late, but I'll go ahead and accept. [04:51] rmadison is from a mirror of a mirror, it's always laggy. [04:53] Danke. [05:34] infinity: ack, respinning - thanks [05:39] hmm, crontab and pad don't seem to have been updated for the drop of preinstall images [08:45] oh, we have a new arch ? :P [08:45] Wed Jul 25 06:11:40 UTC 2012 [08:45] No live filesystem source known for armhf-omap4 [08:54] * xnox head desk [08:54] who rejected e2fsprogs? infinity ? [08:54] if i only could find out where that comes from [08:55] xnox: I did, yes. We can chat about it tomorrow. :) [08:55] infinity: ok. =) [08:55] infinity: good night (?!) [08:55] * infinity nods. [08:55] Night indeed. [08:55] * xnox is not too sure. oh ok. [08:55] infinity, any idea ? the nusakan copy pastes also all have armhf-omap4 [08:56] (on the pad) [08:57] I don't see it in any commandlines on the pad... [08:58] it looks a bit like a cron entry that ran ... (teh mail also has no (built by ...) [08:58] infinity, at the bottom of the pad [08:58] The copy and paste for output is correct, it just does project-arch-subarch [08:59] But I'm guessing someone accidentally did a run with ARCHES=armhf-omap4 and sent you your mail, that's all. [08:59] then i dont get where that failed build comes from [08:59] Probably from vorlon re-running the server image earlier tonight. [08:59] since SUDO_USER doesnt seem to be set ... [08:59] which it should from a manual run [08:59] (and which would trigger the mail subject to be different) [09:00] cdimage@nusakan:~$ echo $SUDO_USER [09:00] cdimage@nusakan:~$ [09:00] It's not always set, depends on how you sudo. [09:00] oh, indeed [09:00] i'm indeed assuming we all use the same method :P [09:00] Old habits die hard. [09:01] heh [09:01] well, sleep well then ... [09:01] I still use "sudo su - user" because that was sane before "sudo -u user -i" existed. [09:01] It's muscle memory for old people. :P [09:01] yep, i know what you mean [09:02] Arguably, we could/should patch shadow (or pam_env, whichever is killing it) to preserve SUDO_USER. [09:02] But I dunno. [09:02] but if i get that right there was a server build supposed to happen after d-i was published, right ? [09:02] That could have weird knock-on effects I'm not thinking about. [09:02] ogra_: Right. slangasek kicked it off. [09:02] ogra_: If that never finished, that was probably the email you got. :P [09:02] the last server image is from yesterday 21:xx UTC [09:03] right [09:03] Yeah, so feel free to spin another. [09:03] * ogra_ will try [09:03] It should fix the no keyboard issue, but I'd like confirmation. [09:03] Also, netboots all work great now, thanks for fixing that. [09:04] (Well, you and TheMuso, his fix was somewhat critical) [09:04] how do they work great ? they are still unbootable after install [09:04] Are not. Try one. [09:04] I just did both -serial and -fb last night, both rebooted fine. [09:05] ?? f-k cant find /dev/mmcblk1p1 (since there is no partition) [09:05] or mmcbk0p1 or so [09:05] There is on the cards I just wrote. [09:05] * ogra_ doesnt see how that would be fixed [09:06] I'm not sure how netboot was doing anything with f-k at all before TheMuso's upload. [09:06] it uses f-k-i udeb [09:06] (The answer being that it couldn't have been) [09:06] Yeah, but the udeb wasn't marked for that subarch. :P [09:07] the images i tested yesterday definitely had lukes fix [09:07] and failed miserably in f-k-i [09:07] Very weird. Cause yeah, I just did both the omap4 images yesterday, and they were happy. [09:07] so i really dont get how it can have worked for you [09:07] f-k hardcodes /dev/mmcblk0p1 [09:07] Maybe I'm magic. [09:07] heh [09:08] did you partition the mmc using partman during install ? [09:08] No, not unless it did so behind the scenes. [09:08] I only partitioned my install media. [09:08] it doesnt usually [09:08] Err, my target media. [09:08] weird [09:09] well, i'll do a new test later, but i dont see how a working install can be possible [09:10] http://paste.ubuntu.com/1109757/ [09:10] ^-- Looks partitiony to me. [09:10] yeah [09:10] I'm not really sure why it was having a sad for you. [09:10] where would these come from ? [09:11] * ogra_ doesnt get it ... i dont think we have any code for this [09:11] I dunno, but they must have always been there. flash-kernel 2.0 didn't magically partition cards either, it had the same hardcoded logic. [09:11] they definitely werent there yesterday [09:12] and the last two weeks ... LethoThe2nd reported it to me on IRC monday last week and i'm 100% sure no d-i build had them until yesterday [09:12] boot/arm/generate-partitioned-filesystem [09:13] ^--- From the d-i netboot bits. [09:13] * ogra_ scratches head [09:13] Which does basically the same thing as debian-cd. [09:13] Cargo-culted code, even. [09:13] So, I'm puzzled as to what error was being seen. :/ [09:14] Oh. Unless it has the same off-by-one error that I fixed in debian-cd last year. [09:14] err, wait, i tested omap [09:14] Which would come and go as the initrd/kernel grows and shrinks. [09:14] * ogra_ wonders if tehre is any difference [09:14] omap and omap4 use the same code here. [09:14] But the off-by-one thing is entirely possible. [09:15] I can look at that in the morning. [09:15] Or you can dig through debian-cd logs and find the fix and compare. [09:15] Whichever. [09:15] yep, will do, i only do testing today anyway (and some paperwork) [09:18] ogra_: bzr diff -r1722..1723 [09:19] where, in d-i ? [09:19] ogra_: In debian-cd. [09:19] ogra_: Though that may not apply to d-i. There's already a spare 512 there that wasn't in debian-cd. [09:19] So, I could be barking up the wrong tree. [09:19] yeah, d-i doesnt use that [09:19] Maybe you should just re-test omap3 and see if it's still borked, and we'll debug from there. :P [09:20] (Or, if you can find an image that's broken) [09:20] all d-i build logs of the last four builds show that the partitioning code is run [09:20] ogra_: Yeah, the code is obviously being run, the question is if it's broken subtly. [09:20] ogra_: Hence my guessing at the off-by-one thing. [09:20] ogra_: Since the debian-cd partitioning bits and the d-i ones are very similar in history. [09:21] thats a weird diff btw [09:21] modified file 'tools/boot/oneiric/post-boot-armel+mx5' [09:21] You're welcome. ;) [09:21] heh [09:22] I probably should have named it "MBR" instead of "LEAD_IN", but there were enough confusing variables already, adding to them was fun. [09:22] i mean that bzr shows the first half of the patch for one link name and the second one for the actual file [09:22] ogra_: Huh? Those are two files. [09:22] ogra_: Or, they were back then, if they're not now. [09:22] ah, k [09:23] Same fix, applied twice, to two slightly different bits of code. [09:23] well, oneric, old stuff ... [09:23] Only the omap one is relevant, I imagine. [09:23] yep [09:24] But, like I said, there's a spare 512+ in d-i already anyway, so this could be a red herring. [09:24] I'd like to find an image that breaks for you, and try to reproduce. [09:24] well, all images i tested didnt have any partition after dd'ing [09:25] i usually re-plug the SD before i move it to the board [09:25] Even after pulling and re-inserting the card. [09:25] yeah [09:25] Many/most card readers won't re-scan partitions. [09:25] Kay. [09:25] They need a media change event, which some readers also don't do properly. [09:26] ogra_: But if, say, 20101020ubuntu161 was broken for you, let me know, and I'll dig into the WTF of it all. [09:26] http://paste.ubuntu.com/1109783/ [09:26] ogra_: If not, then we could have a harder time finding older ones. Unless you have one sitting around. [09:27] i still have them around, but i downloaded from the /current link (still have the tab open) ... timestamp on the page is jul 24th 07:03 [09:28] -rw-rw-r-- 1 ogra ogra 31457280 Jul 19 15:34 boot.img-fat-fb [09:28] -rw-rw-r-- 1 ogra ogra 31457280 Jul 19 15:34 boot.img-fat-fb.1 [09:28] -rw-rw-r-- 1 ogra ogra 31457280 Jul 24 09:03 boot.img-fat-serial [09:28] Oh, wait. [09:28] That's the fat. [09:28] That's not an image. [09:28] all of them are omap though [09:29] oh SH*T !!! [09:29] indeed you are right [09:29] hooray for informative naming [09:29] * ogra_ goes to stand in the corner for a while [09:29] Well, it does say "fat". As in, "hi, I'm a filesystem". ;) [09:29] yes [09:30] well, we can then release it, great, saves me one release note to write :) [09:30] *grin* [09:31] and while making a fool out of me nusakan has finished the server images [09:31] great [09:31] So I saw. [09:31] So you can test those and see if you get a USB keyboard. [09:31] And I'll sleep, honest. [09:31] will do [09:31] great, sleep tight [09:31] If it's still buggered, reopen the bug and yell at me. [09:31] Or fix it. :P [09:31] i'll do the latter :) [09:32] (But I'd like to know either way) [09:32] * ogra_ goes getting some coffee [09:32] * infinity goes to have a bedtime smoke. [09:39] infinity: i hope you have a smoke alarm in your bedroom [09:53] better than in his bedtime at least :P [10:00] k, so omap netboot is definitely a no-go ... seems the NIC driver is missing [10:10] weird, omap4 as well [10:11] hmm, in fact there are no USB devices at all [10:11] * ogra_ is confused [10:14] * ogra_ tries the framebuffer image instead [10:20] aha, that works [10:34] ogra_: Hrm, I didn't remove any drivers, just added some... [10:35] yeah, it seems to randomly not work [10:35] i had two boots of the serial image where no usb disk was found, three where it was [10:35] smells like a udev race [10:36] i have a framebuffer install working atm, lets see if it gets completely through, then i'll dig more into the usb bits [10:36] It could be the vm.min_free_kbytes thing. [10:36] though testing the normal server image first is probably more important [10:36] Though, I can't see that being an issue on boot, we haven't eaten the RAM yet. [10:36] yep [10:36] also see my last comment on that bug ... [10:37] i would actually like to try that suggested kernel fix from marvin24 [10:37] instead of having the hack in userspace again [10:39] If it works, sure. [10:39] It's not the easiest thing to test, mind you. [10:39] why ? if i can get a patched kernel ... [10:40] I mean it's not the easiest thing to reliably reproduce to the point where one can be sure it's fixed. [10:41] running apachebench seems to trigger it reliably [10:41] But we'll let the kernel team comment on it. I'll bring it up with them. [10:41] i pinged paolo about it already, lets see what he says [10:41]