=== bluefoxicy [n=bluefox@c-68-33-112-13.hsd1.md.comcast.net] has joined #ubuntu-kernel [04:58] I want oops insurance [04:59] when my kernel oopses I want it to file a bug and mail the full oops output to every kernel dev so I KNOW they're fixing it :D [05:30] bluefoxicy: Every time I commit a fix for something, I want it instantly forced on every user's machine, and I want them to be forced to test it immediately. (Also, we can't always get what we want) [05:31] infinity: it was a joke [05:31] So was mine. Sort of. ;) [05:31] infinity: although instant testing would be nice ;) === j_ack [n=nico@p508D940A.dip0.t-ipconnect.de] has joined #ubuntu-kernel === CataEnry [n=cataenry@host152-2.pool8261.interbusiness.it] has joined #ubuntu-kernel [10:05] infinity: i'd like every user to accompany their bug reports w/ patches that solve the problem. also, a pony. i want a pony. [10:09] dilinger: what colour? === CataEnry [n=cataenry@host152-2.pool8261.interbusiness.it] has joined #ubuntu-kernel [10:11] blue === dilinger works on making x86_64's printk_address print something reasonable looking [10:49] [ 123.737130] Call Trace: [ffffffff802e0e15] schedule_timeout+0x25/0xd0 [10:49] [ 123.737143] [ffffffff8014b803] prepare_to_wait+0x23/0x80 [10:49] [ 123.737148] [ffffffff802dc383] unix_stream_recvmsg+0x2c3/0x5d0 [10:49] [ 123.737155] [ffffffff8014b550] autoremove_wake_function+0x0/0x30 [10:50] hey look, it's an almost readable backtrace, just like i386 and sparc64 [11:19] ugh, is that intentional? [11:19] dmesg output: [11:19] [ 82.375220] [] __pollwait+0x0/0xf0 [11:19] syslog (kern.log): [11:19] Feb 24 05:15:42 throat kernel: [ 82.375220] [__pollwait+0/240] __pollwait+0x0/0xf0 === jane_ [n=JaneW@dsl-146-140-66.telkomadsl.co.za] has joined #ubuntu-kernel [12:16] BenC: Why the change to CONFIG_2GB? [01:17] is it intended for a server setup that the hard disk spin down after activity? and if yes, that period seems to be too short === fabbione [i=fabbione@gordian.fabbione.net] has joined #ubuntu-kernel [01:31] doko: that would be one of the startup scripts doing that, or it's your bios [01:31] mjg59: mainly to allow laptops with 1GB of ram to still be able to suspend/resume [01:31] BenC: no, did start sometime in the dapper cycle [01:32] doko: has to be a startup script then...not sure which one though [01:32] BenC: Eh? [01:32] BenC: Suspend/resume should work with highmem... [01:32] mjg59: I've had reports that it doesn't, also read it somewhere [01:32] BenC: How weird [01:32] BenC: It has the side effect of breaking sbcl [01:33] yeah, it seems to be breaking more than it fixes [01:33] when I get "recursive die() failure, output supressed", that's bad, right? [01:33] hey BenC [01:33] hey fabbione [01:33] BenC: Suspend to RAM or suspend to disk? [01:33] fabbione: I did silo last night, just forgot to upload new upstream tarball [01:33] our current live CD seems exceedingly unhappy in vmware. [01:33] mjg59: can't recall, let me check [01:34] BenC: please pull from my tree. changes: more sun4v love, config changes for sparc to move sunhme, sungem and esp from mod to inline to spare people a few errors in d-i [01:34] fabbione: those d-i problems were supposed to be fixed, I filed bugs :/ [01:35] BenC: i am talking about modprobe errors during install [01:35] fabbione: did you also make scsi/sd built-in aswell? [01:35] no i didn't [01:35] what sort of modprobe errors? [01:35] only sunhme, sungem and esp [01:35] it's problem with the drivers [01:35] when the hw is not there, they exit badly [01:35] and d-i catch the error and show a big fat red screen to the user [01:35] = scary [01:36] bu [01:36] BUT [01:36] if they are compiled in, hw is still recognized [01:36] and there is no need to modprobe them [01:36] = works [01:36] gotcha [01:37] BenC: i also added the sparc ABI files for 16.22 and 16.23 [01:38] and removed the abi checker [01:38] sorry i meant the sparc.ignore [01:38] ok [01:38] now it takes me about 20 minutes to build :) [01:39] how's the DC sparc hw coming? [01:40] 20 minutes? my machine can do it in 10 :) [01:41] btw, someone is giving me an e3.5k [01:41] not sure where I'll put it, or why I need it, but it's free [01:41] mjg59: swsusp is the one that supposedly doesn't work with > 1GB of RAM (according to reports on user lists and forums, etc), though it's worked in the past on my laptop with 2GB of RAM.. [01:42] BenC: 20 minutes including udebs and without ccache? [01:42] BenC: 20 is full dpkg-buildpackage :) [01:42] BenC: Don't suppose they want to spend ridiculous sums to ship it to Australia? [01:42] BenC: the machines are installed in a rack.. i think elmo crashed before installing anything on them [01:42] fabbione: full udebs, but ccache, yes [01:43] kill ccache :) [01:43] and we can talk ;) [01:43] infinity: maybe I can sneek it through customs in to germany in my duffle bag [01:43] BenC: You must have gotten a larger duffle bag since last we met... [01:43] I think It was 30 minutes without ccache [01:43] Feb 24 04:40:30 sunrise udevd[2686] : get_netlink_msg: unable to receive kernel netlink message: No buffer space available [01:44] HMM [01:44] kernel or udev issue? [01:44] fabbione: AHA! someone else sees it aswell [01:44] that's been fucking up my e3k for weeks now [01:44] Blame davem. [01:44] it has to be kernel, I haven't seen it anywhere else [01:45] someone the netlink foo is returning ENOBUF but I can't see how [01:45] BenC: AHHHHH [01:45] s/someone/somewhere/ [01:45] crap [01:45] i will blame davem :) [01:46] Or, blame davem's girlfriend. She's too distracting. [01:46] infinity: s/duffle bag/empty crago hold/ [01:46] because it looks like udevplug is triggering that thing exactly when it needs to probe the network driver [01:46] fabbione: for me it occurs when it needs to load sd_mod, which means I have to manually bring the machine past initrd stage [01:47] ok.. [01:47] so it's random.. [01:47] good to know [01:47] david will love to fix that :) [01:47] not really, for me sd_mod is the first thing it is trying to do [01:48] the netdriver is not the first [01:48] it's way past that i think [01:48] but i collected all the infos together [01:48] it's probably the first thing that requires a kernel event [01:50] could be [01:53] BenC: btw.. silo is up [01:53] uploaded this morning === Keybuk [n=scott@213-78-32-60.ppp.onetel.net.uk] has joined #ubuntu-kernel [02:11] yeah, saw that [02:12] cool [02:12] hey Keybuk [02:12] i was just waiting for you [02:12] Keybuk: http://people.ubuntu.com/~fabbione/sparc/ [02:12] this is the error i am getting on sparc [02:13] there is lspci, the error in syslog and the strace of both udevd and udevplug [02:13] each time i run udevplug i can reproduce the error [02:13] what i see at each reboot is network not loaded (e1000) [02:13] but BenC for instance has no sd_mod === BenC concurs [02:14] BenC: i also fixed hw-detect and Mithrandir did kbd-chooser [02:15] the latter will make that annoying error disappear when you don't have a keyboard installed [02:15] so hw-detect should find my sbus devices now? [02:15] BenC: only after you will upload the new kernel... [02:15] that has esp built in [02:15] well that's no help, new kernel will make my sbus modules built-in :P [02:16] cute, that error message isn't listed in the recv() manpage [02:16] ENOBUFS [02:16] BenC: i will look at hw-detect in debian and see what they have [02:16] Keybuk: that error is from the netlink core [02:16] BenC: how do we avoid that error? [02:16] BenC: otherwise yeah. i guess that's the solution :/ [02:16] Keybuk: I couldn't figure out why it happened [02:16] netlink code is so confusing [02:17] could it be that the kernel overflowed the netlink buffer space [02:17] fabbione: they have a working libdetect that correctly descends secondary sbus busses :) [02:17] Keybuk: seems that it some how does [02:18] BenC: what package is that? [02:19] fabbione: detect or libdetect or something like that [02:19] from what I remember, we don't use the same thing they do to detect devices [02:19] BenC: oh you mean discover? [02:19] yeah, that's it [02:19] they did get rid of it [02:20] and the fix for the double sbus was merged in breezy under my "heavy" pressure [02:20] BenC: what's BUFFER_SIZE in lib/kobject_uevent.c ? [02:21] (that's the size allowed for a single uevent) [02:21] Keybuk: no idea, but there is no ENOBUFS in that code, so I think it's elsewhere [02:22] it could be that it's trying to generate a uevent that's too big [02:22] or it could that it's queuing too many uevents, and udevd isn't getting enough cpu time to slurp them all [02:22] the BUFFER_SIZE overun case returns ENOMEM [02:22] it's probably the latter [02:22] too many events [02:23] net/netlink/genetlinks:ctrl_build_msg() returns ENOBUFS [02:24] and so does net/netlink/af_netlink:netlink_overrun() [02:24] it's probably the second one generating the error [02:24] that function's called in a few places [02:25] Keybuk: is it possible to build udevplug to wait let say half second between sending each event? [02:25] that would exclude the "too many events at once" [02:25] fabbione: you'd be in the ten-minutes-to-boot area with that delay [02:25] Keybuk: i don't need to run at it boot :) [02:25] i can test it in userland [02:25] i get the same error [02:26] in both situation [02:26] 10 minutes.. no problem ;) [02:26] run "udevplug -s" to test if that's the case [02:26] that waits for the previous event to be processed before sending the next [02:26] what does -s do? [02:26] ok [02:26] sure i can do [02:26] in a few minutes.. [02:26] i am enjoying this extremely cleaned up d-i [02:27] BenC: could be caused by nlmsg_new not returning a message, though I can't find that function [02:27] do_one_broadcast() also has a few failure points === Keybuk looks in the header files [02:27] static inline struct sk_buff *nlmsg_new(int size) [02:27] { [02:27] return alloc_skb(NLMSG_GOODSIZE, GFP_KERNEL); [02:27] } [02:28] netlink_broadcast_deliver() seems like a likely candidate [02:29] yeah, there's a whole bunch of them there [02:29] it tries to push one to the queue [02:29] if only we could tag it so we knew which one it was [02:29] if fabio's test doesn't prove anything I'll start sprinkling some printk's to find out where it is failing [02:29] ah, that pushes into an actual socket rcvbuf ... so if that was full, then it wouldn't fit [02:29] right [02:30] interesting that this has only affected sparc so far though [02:30] hmmm [02:30] hey, can you increase udev's socket bufsize? [02:30] Keybuk: it might be a cpu speed issue? [02:30] I'd've thought an amd64 would show up more [02:30] Keybuk: sparc64 is slower [02:30] ahh, of course, sparc is slower so udevd gets less time because the kernel is using it all [02:30] it's odd that it happens on a 6-way system though [02:30] BenC: remind me how to do that [02:31] my sparc64 has 6 cpu's and 6gigs of ram :/ [02:31] it's a setsockopt isn't it? [02:31] "mine" only 32 CPUs and 16GB of ram === fabbione ducks === pappan [n=ppadman@bgepxyout-02.asiapac.hp.net] has joined #ubuntu-kernel [02:31] fabbione: cat /proc/sys/net/core/rmem_max /proc/sys/net/core/rmem_default [02:32] Keybuk: you will have to wait.. i am in the middle of testing parted.. just a few minutes [02:32] Keybuk: maybe setsockopt(), let me check [02:33] rebooting now... [02:33] fabbione: maybe we have just too much memory/cpu and it's confusing udev :) === fabbione goes for a break while POST [02:33] BenC: possibly === Keybuk wonders what SO_RCVBUFFFORCE is [02:33] BenC: these udev hackers and their laptops [02:33] lol [02:33] Keybuk: IIRC, you can set the buffer to a local one [02:35] SO_RCVBUF maybe [02:37] it already sets that SO_RCVBUFFORCE thing [02:38] what does that do? [02:39] dunno [02:40] re [02:40] ah [02:40] got it, sets the rcvbuf and forces it over the maximum if necessary [02:41] const int buffersize = 16 * 1024 * 1024; === mgalvin [n=mgalvin@ubuntu/member/mgalvin] has joined #ubuntu-kernel [02:41] Keybuk: ok.. now i am without network [02:41] setsockopt(uevent_netlink_sock, SOL_SOCKET, SO_RCVBUFFORCE, &buffersize, sizeof(buffersize)); [02:41] so that forces the rcvbuf of that socket to 16MB [02:41] root@sunrise:~# cat /proc/sys/net/core/rmem_max /proc/sys/net/core/rmem_default [02:41] 131071 [02:41] 124928 [02:41] which is roughly 16,000 uevents [02:41] Keybuk: maybe that's broken on sparc [02:42] BenC: or events are bigger? [02:42] events are fixed at 1024 in the kernel and in udev [02:42] fabbione: try "udevplug -s -v | tee events.txt" then count how many lines you get :) [02:43] Keybuk: udevplug -s is running [02:43] ah, ok [02:43] i can do that later again :) [02:43] the error did always appear before.. [02:44] so one run more or less won't cahnge my life [02:45] ok [02:45] Keybuk: are you checking the return value of setsockopt()? [02:45] see whether it appears this run first [02:46] BenC: no... [02:46] it may be failing [02:46] BenC: looking at the code, it can only fail with -EPERM [02:47] yeah, but it could also be something as stupid as a signed extension or 32/64 value that is getting junked and causing it to be set at minimum bufsiz [02:47] but that wouldn't error out [02:47] amd64 isn't doing this in 32-bit [02:47] Keybuk: it looks like that the run did not generate the error, but it still doesn't bring up the network [02:48] BenC: amd64 doesn't need memory to be alligned at 64 bit [02:48] that can cause issues [02:48] like it was with apt in breezy [02:48] no, I mean amd64 isn't pushing this through a compat layer [02:48] yes i understand [02:48] are we? [02:48] yes [02:48] i think.. === fabbione is tired and confused [02:49] Keybuk: i have the events file.. [02:50] fabbione: just "wc -l" it [02:50] wc -l events.txt [02:50] 0 events.txt === zul [n=chuck@ubuntu/member/zul] has joined #ubuntu-kernel [02:50] heyl [02:50] ? [02:50] hey zul [02:50] looks like it just pushes it through as a compat, no translation [02:51] fabbione: you ran with -v ? [02:51] udevplug -s -v | tee events.txt [02:51] weird [02:51] exactly as you wrote it [02:51] dunno why that didn't give you anything [02:51] does it without the | tee ? [02:51] ie just udevplug -s -v ? [02:51] yeah [02:51] Keybuk: doesn't give me anything on my amd64 box either [02:52] udevplug -s -v | wc -l [02:52] fabbione: is /sys mounted? :) [02:52] ah, are you in a chroot? [02:52] Keybuk: you joking right? it's ubuntu running system 100% [02:52] no chroot [02:52] find /sys -name uevent [02:53] find /sys -name uevent | wc -l [02:53] 730 [02:53] BenC: it works just fine on mine, I get a huge number of /sys lines printed [02:53] I do now that /sys is mounted [02:54] I need a serial console to my sparc so I can test this stuff too [02:54] fabbione: what about just "udevplug -v" does that print anything? [02:54] I ran udevplug under linux32 in an i386 chroot on my amd64, which should use the same codepath as sparc, and it was just fine [02:55] Keybuk: checking in a sec [02:55] BenC: yeah, I've just done the same [02:55] quest scott# time udevplug -v | wc -l [02:55] 773 [02:55] udevplug -v 0.01s user 0.03s system 2% cpu 1.685 total [02:55] hmm [02:55] it's taking too long === doko [n=doko@dslb-084-059-085-222.pools.arcor-ip.net] has joined #ubuntu-kernel [02:56] fabbione: check you don't have an empty /dev/.udev/queue directory (just sudo rmdir it) [02:56] Keybuk: yeah that's where i was sticking my nose :) [02:56] that's a common failure mode of previous udevd [02:56] time udevplug -v | wc -l [02:56] 735 [02:56] real 0m1.093s [02:56] user 0m0.048s [02:56] sys 0m0.164s [02:56] could also explain why -s doesn't work (it's also waiting for that to go away first, and just times out after three minutes) [02:56] ok [02:56] so now [02:57] let's try again the -s [02:57] well, you don't have any more events than my amd64, slightly less in fact [02:57] i got the error in syslog [02:57] those should take only 735,000 bytes of memory to hold in the kernel [02:57] BenC: do you know of a way to find out the size of a socket from userspace? [02:57] lsof? [02:58] hmmm this is interesting [03:00] Keybuk: if i run udevplug -v -s [03:00] it gets to /sys/class/vc/vcsa [03:00] /sys/devices/pci0000:02 [03:00] and it stalls there [03:00] probably aborts with SIGALRM :) [03:01] no no [03:01] udevplug is still running [03:01] i had to ctrl+c [03:01] odd [03:01] what's in /dev/.udev/queue? [03:01] no queue [03:01] hmm [03:01] strace it === fabbione tries again [03:03] it's polling queue [03:03] probably udev is still processing [03:03] does queue still exist? [03:03] yes [03:04] but it's empty [03:04] ah [03:04] did you rmdir it first? [03:04] yes === fabbione will do again to be sure [03:06] like i said [03:06] udevplug did generate the queue [03:06] and waiting for it to disappear [03:06] but it's empty [03:06] Keybuk: getsockopt() [03:06] Keybuk: udevd is doing nothing [03:07] that's weird [03:07] that suggests the kernel never gave the event to udevd [03:07] well it did [03:07] can you run "udevmonitor -e" as well? [03:07] otherwise i would have no / ;) [03:07] if it had given the event to udevd, udevd would have done something and removed the queue directory [03:08] ok one sec.. [03:09] i am setting up a slightly better env [03:09] like 20 xterm [03:09] udevmonitor should give you a UEVENT and UDEV for each things udevplug prints (with -s) [03:10] root@sunrise:/dev/.udev# ls -asl [03:10] total 0 [03:10] 0 drwxr-xr-x 4 root root 80 Feb 24 06:09 . [03:10] 0 drwxr-xr-x 14 root root 13920 Feb 24 06:05 .. [03:10] 0 drwxr-xr-x 2 root root 520 Feb 24 2006 db [03:10] 0 drwxr-xr-x 2 root root 40 Feb 24 06:09 failed [03:10] root@sunrise:~# udevmonitor -e [03:10] udevmonitor prints the received event from the kernel [UEVENT] [03:10] and the event which udev sends out after rule processing [UDEV] [03:10] ok? [03:10] do we agree that it is ok? [03:10] ok [03:10] i can see the events [03:11] that's a good starting point === pappan [n=ppadman@bgepxyout-02.asiapac.hp.net] has joined #ubuntu-kernel [03:11] "udevplug -s -v" on that ... for each thing it prints you should see a UEVENT and then a UDEV for it [03:11] if you get no UEVENT, then that's bad [03:11] if you get a UEVENT and no UDEV, that's even worse [03:11] UEVENT[1140790262.767691] add@/class/vc/vcsa [03:11] UDEV [1140790262.770291] add@/class/vc/vcsa [03:11] ok? [03:11] ok [03:11] and udevplug printed /sys/class/vc/vcsa as well? [03:12] it didn't get that far??? [03:12] /sys/class/tty/tty4 [03:12] it stopped a few letters before [03:12] make that 40 [03:12] it did skip 4 [03:12] meh 0 [03:13] there is a queue [03:13] and it is empty [03:13] ok, what was the last thing udevplug printed? [03:13] yes [03:13] what was? [03:13] /sys/class/tty/tty4 [03:13] ok [03:13] now [03:13] what was the last UEVENT/UDEV combo printed? [03:14] ok one second dude [03:14] when udevplug was at /sys/class/tty/tty4 === mxpxpod [n=BryanFor@unaffiliated/mxpxpod] has joined #ubuntu-kernel [03:14] udevmonitor was printing the vcsa [03:14] that's a bit weird [03:14] there was a queue and it was empty [03:14] now one more thing [03:14] i did remove the queue [03:14] and saw it recreated [03:14] more events did pass [03:15] and udevplug did finish [03:15] (btw, worth noting that "udevplug -s" is not very well tested) [03:15] it could be just a general bug with it [03:15] so ok.. what do you want me to test next? [03:15] hmm [03:15] so udevplug completed normally [03:15] and you didn't get that error [03:15] nope [03:16] nope it didn't complete noramlly? [03:16] but know i don't know if it would have loaded the module [03:16] nope = no error [03:16] i think the problem is here: [03:16] right, so this suggests that the netlink buffer doesn't overflow if you go slowly [03:16] tty4 was way before than /sys/class/vc/vcsa [03:17] how do you know? :) [03:17] (in the print from udevplug) [03:17] the print order? [03:17] now [03:17] listen up [03:17] did udevplug never do vcsa before then? [03:17] no it didn't [03:17] it did it after [03:17] if you let me :) [03:18] udevplug was printing tty4 - udevmonitor was at vcsa [03:18] the line after vcsa in udevplug is /sys/devices/pci0000:02 [03:18] the same where it was hanging a long time before [03:18] now... [03:19] yeah, I get the same behaviour here (though udevplug actually prints that ... I suggest your stdout buffers weren't flushed ) [03:19] could it be a bug in /sys parsing of that device? [03:19] this is just a "-s" bug [03:20] next test... [03:20] queue was never deleted tho [03:20] if i run only udevplug [03:20] i can see all the events and the error [03:20] that happens very early [03:20] is there kernel debugging tool in ubuntu [03:20] almost at the beginning [03:21] right, because udevd had finished processing the event, and was waiting for the next ... where udevplug had made the queue directory and then ended up waiting on it [03:21] i am facing a problem with reboot in my laptop [03:21] but if i run it normally, the queue dir disappear [03:22] I'll have to debug that, but it's reasonably safe race :) [03:22] so ignore that for now [03:22] Keybuk: so do you think the bug is from udev itself? [03:22] fabbione: no, I think this is a kernel bug [03:23] i am not really worried about the message itself [03:23] sending the events slowly seems to not produce ENOBUFS [03:23] sending them at normal speed produces it [03:23] so, BenC, can we get some printk()s to find out which -ENOBUF that is? [03:23] it's only annoying that it doesn't bring up the ethernet [03:23] anyway i need to take off for a while now [03:23] Keybuk: thanks a lot dude [03:23] Keybuk: yeah, let me get my sparc back up [03:24] later guys [03:25] toodles [03:29] damn, that -s bug is entirely consistent at the first pci device [03:30] tickle: uevent: '/sys/devices/pci0000:00/uevent' [03:30] make_queue: directory: '/dev/.udev/queue' [03:30] create_path: stat '/dev/.udev' [03:30] wait_for_queue: directory: '/dev/.udev/queue' [03:30] oh [03:30] because tickling /sys/devices/pci0000:00/uevent DOES NOTHING [03:31] BenC: kernel bug! kernel bug! kernel bug! :) [03:31] Keybuk: stop picking on me! :) [03:32] (this is irrelevant to the ENOBUFS error, it's just amusing to find more errors along the way to debugging that one) [03:32] I think ENOBUFS is a red herring [03:33] you do? [03:33] more than likely our problem is more related to uevents not getting where they should [03:33] yeah, but if the socket buffer is full, they won't get there [03:34] the fact that pci0000:00 doesn't generate a uevent is only important when using "-s" where it waits patiently for the event ... during normal booting it's irrelevant [03:34] I can't see generic sockets getting full...lots of things would be broken [03:34] aye [03:34] udevd used to do it quite regularly until they increased the size to 16MB [03:34] were the symptoms the same? [03:35] yeah, I think so [03:35] I just can't see 16k uevents occuring, even on a sparc :) [03:35] we know there's only 730 events [03:35] so filling it would take a lot of effort [03:35] which is less than my amd64 [03:35] I'm wondering whether it's actually that there's an event bigger than 1K [03:35] an event is just an env buffer, after all [03:36] true...guess I can put some debug to see what the event size is [03:36] or just check for > 1k [03:39] I wonder ... [03:39] the fact udev wants the socket buffer to be 16MB is just a hint about how big it should never grow past [03:39] it doesn't mean it can actually grow that big, the kernel might not have any free memory to grab [03:39] so it may actually be effectively smaller than the 730K needed to do the job === __keybuk [n=scott@82.108.80.242] has joined #ubuntu-kernel === j_ack [n=nico@p508D940A.dip0.t-ipconnect.de] has joined #ubuntu-kernel === Keybuk [n=scott@82.108.80.245] has joined #ubuntu-kernel === CataEnry [n=cataenry@host152-66.pool876.interbusiness.it] has joined #ubuntu-kernel === BenC [n=bcollins@debian/developer/bcollins] has joined #ubuntu-kernel === shaya [n=spotter@user-0ccembr.cable.mindspring.com] has joined #ubuntu-kernel [07:35] anyone home? === cmvo [n=cmvo@62.225.11.174] has joined #ubuntu-kernel [07:36] just filed a bug, can help try to debug it if that would help? === cmvo [n=cmvo@62.225.11.174] has left #ubuntu-kernel ["Konversation] === cjb [n=cjb@islay.ra.phy.cam.ac.uk] has joined #ubuntu-kernel [11:07] Sorry, stupid question: dmesg/lspci dumps for lkml, should they go inline or attached? [11:07] (The only examples I can find were all attached, so I wonder if there's some differing standard for dmesg as opposed to patches.)