/srv/irclogs.ubuntu.com/2006/03/26/#ubuntu-kernel.txt

=== kirkland [n=dustin@pixpat.austin.ibm.com] has joined #ubuntu-kernel
kirklandis there a such place where one can find a nightly (or regular) tarball of bcollins/ubuntu-2.6.git tree?12:08
kirklandi see /topic, but those look a little out of date...12:09
=== JanC [n=janc@lugwv/member/JanC] has joined #ubuntu-kernel
=== j_ack [n=nico@p508D9FF9.dip0.t-ipconnect.de] has joined #ubuntu-kernel
crimsunkirkland: I don't know offhand, but you can always just crontab a ``git pull''01:00
=== JanC_ [n=janc@dD577040E.access.telenet.be] has joined #ubuntu-kernel
=== _human_blip_ [n=mike@mike.nelsonbay.com] has joined #ubuntu-kernel
=== JanC_ [n=janc@dD577040E.access.telenet.be] has joined #ubuntu-kernel
=== kirkland [n=dustin@pixpat.austin.ibm.com] has left #ubuntu-kernel []
=== JanC [n=janc@lugwv/member/JanC] has joined #ubuntu-kernel
=== j_ack [n=nico@p508D9FF9.dip0.t-ipconnect.de] has joined #ubuntu-kernel
=== _human_blip_ [n=mike@mike.nelsonbay.com] has joined #ubuntu-kernel
=== JanC_ [n=janc@dD577040E.access.telenet.be] has joined #ubuntu-kernel
=== JanC [n=janc@lugwv/member/JanC] has joined #ubuntu-kernel
=== _human_blip_ [n=mike@mike.nelsonbay.com] has joined #ubuntu-kernel
=== fabbione [i=fabbione@gordian.fabbione.net] has joined #ubuntu-kernel
=== JaneW [n=JaneW@dsl-165-197-110.telkomadsl.co.za] has joined #ubuntu-kernel
=== doko [n=doko@dslb-088-073-083-063.pools.arcor-ip.net] has joined #ubuntu-kernel
=== ubuntulog [i=ubuntulo@trider-g7.fabbione.net] has joined #ubuntu-kernel
=== Topic for #ubuntu-kernel: Ubuntu kernel development discussion ONLY | New git tree for dapper: https://wiki.ubuntu.com/KernelGitGuide | 2.6.15-19.29 uploaded (Stupid firmware) | Daily Diet of Destruction: http://people.ubuntu.com/~bcollins/kernels-daily/
=== Topic (#ubuntu-kernel): set by BenC at Mon Mar 20 16:03:20 2006
=== j_ack [n=nico@p508D9856.dip0.t-ipconnect.de] has joined #ubuntu-kernel
mjg59BenC: When you updated the bcm code, you seemed to stomp over my patches...04:56
BenCcrap, I probably did04:56
mjg59BenC: Should just reapply, with luck04:56
BenCwish bcm had a 2.6.15 git tree :/04:57
BenCyeah, I can probably re-base them in git04:57
mjg59You're pulling from the 2.6.16 one and then merging back?04:57
BenCyour changes or bcm43xx in general?05:00
mjg59bcm43xx in general05:00
BenCnah, grabbing daily-snapshot tarball and just copying05:01
BenC2.6.16 would be hard to pull from05:01
crimsunBenC: (just a minor, but s/David/Daniel/ in the changelog)05:02
BenCah, sorry :)05:02
crimsunno biggie :)05:02
BenCdone05:03
crimsungracias05:03
mjg59BenC: Ah, ok05:06
mjg59BenC: If you could reapply my diff, that would be great05:06
=== fabbione [i=fabbione@gordian.fabbione.net] has joined #ubuntu-kernel
=== Cholo [n=Cholo@pool-71-98-26-99.mdsnwi.dsl-w.verizon.net] has joined #ubuntu-kernel
=== Cholo [n=Cholo@pool-71-98-26-99.mdsnwi.dsl-w.verizon.net] has left #ubuntu-kernel []
=== fabbione [i=fabbione@gordian.fabbione.net] has joined #ubuntu-kernel
=== mgalvin [n=mgalvin@ubuntu/member/mgalvin] has joined #ubuntu-kernel
lamontBenC: isn't ide-generic still the module that acutally triggers ide subsystem initialization?07:42
=== Keybuk [n=scott@syndicate.netsplit.com] has joined #ubuntu-kernel
KeybukBenC: ping07:44
BenClamont: no07:46
KeybukBenC: it appears I missed a discussion this afternoon07:46
=== BenC_ [n=bcollins@72.169.114.90] has joined #ubuntu-kernel
makxBenC: are you sure? (i mean considering isa)07:51
BenC_ide works with ide-generic07:57
KeybukBenC_: so it appears I missed a discussion this afternoon?07:57
BenC_Keybuk: appears so :)07:57
KeybukBenC_: which is a shame, because your conclusions were all wrong07:58
=== Keybuk wishes someone had phoned him
Keybukanyway, am replying now via e-mail07:58
BenC_I don't recall having any conclusions07:58
Keybuk"wait 10s before loading ide-generic"07:58
BenC_that's not a conclusion, that's a suggestion :)07:59
Keybukit's a decision, according to your e-mail07:59
Keybuk<g>07:59
BenC_based on Mith's argument07:59
Keybukit won't help a monkey's07:59
Keybukthe kernel already has that "sleep 10" in the IDE init code07:59
Keybukwhich is in the module loading bit07:59
Keybukso blocks the return of modprobe07:59
BenC_but if the module load is blocking, so are all other modules08:00
Keybuk"all other modules" ?08:00
BenC_I didn't know ide-generic had a sleep(10) i it08:00
Keybukide-generic doesn't08:00
BenC_or is this ide-core?08:00
Keybukevery IDE driver does08:00
Keybukide-core08:00
Keybukmodprobe real-pci-driver08:00
Keybukconsistently doesn't return until udev has the events08:01
BenC_is this the "Probing later" thing?08:01
Keybukit can take a few seconds to return on most things08:01
KeybukI don't know what "probing later" is08:01
BenC_are you sure you read Mith's arguments that promoted me suggesting that?08:01
BenC_prompted08:01
BenC_he statement was that the main reason we don't use UUID more widely is that it has some dependency on the bus type08:02
Keybukright08:02
Keybukand the sleep 10 won't change that08:03
Keybukit doesn't have a dependency on the bus type08:03
Keybukthis is the bit where me being there would have helped08:03
BenC_he stated differently08:03
Keybukok, so here's how the initramfs bits work08:03
Keybukfirst we look at the ROOT= to detect the bus type08:03
Keybukeither IDE or "everything else" (which are always scsi)08:03
BenC_then it does have code that depends on ide or !ide08:04
mjg59Keybuk: Eh? That's entirely untrue, as far as I can tell08:04
KeybukAll the drivers under the scsi subsystem (which includes sata, usb, firewire) behave one way08:04
mjg59Keybuk: I can't find any sign of a sleep(10) in the ide-generic load path08:04
Keybukthe modprobe returns immediately, then at some point in the future the driver finds its devices08:04
Keybukmjg59: not a literal sleep(10); not ide-generic; the ide-core probes and binds for the REAL PCI DRIVERS before modprobe can return08:04
mjg59Keybuk: How?08:05
Keybukmjg59: please let me finish first08:05
Keybukok, so for scsi and friends; we have to iterate the bus, modprobe all the drivers, then sit back and wait for the devices to show up08:05
Keybuksit back and wait up to three minutes08:05
Keybukfor especially slow scsi controllers (fabbione delights in owning these)08:06
Keybukso we do it with a simple while loop; while the root device hasn't shown up on disc, sleep another tenth of a second08:06
Keybukif it at the end of that loop, the root device still hasn't shown up, we give up08:06
BenCok, the ide portion is more important :)08:06
Keybukfor dapper+1 I'd like to extend that to "wait forever", and have some pretty on screen message saying something along the lines of "if your root disk isn't plugged in, could you do that?"08:07
Keybukso that's scsi08:07
Keybukide we do it differently, because the ide subsystem has different quirks08:08
Keybukthe drivers behave the other way, the modprobe doesn't return until the driver has done the probe-and-bind08:08
Keybukthere's no asynchronous component to them08:08
Keybukwe also may have to load ide-generic on legacy ISA machines08:09
Keybukand in some cases with a root device on an ancient PCMCIA controller (which counts as "legacy ISA" in my mind)08:09
mjg59Keybuk: I'm still not finding anywhere where the PCI IDE drivers differ from normal PCI drivers (where there is an asynchronous component)08:09
mjg59Hm. PCMCIA ought to come under pcmcia_cs08:09
mjg59Uh, ide_cs08:09
BenCregular PCI IDE drivers use pci probing, and normal device layer callbacks08:10
makxpcmcia bridges are not incl. in latest initramfs by default08:10
Keybukmjg59: critical path.  the ide core already knows what devices are available.  so there's no issuing of a probe command, and waiting for it to return.  it just iterates an already-in-memory structure and does the magic08:10
mjg59There's a corner case where it's not actually PCMCIA, but an on-board IDE interface is connected via the PMCIA slot08:10
BenCthe difference is that IDE detects the drives at the same time it detects the controllers08:10
BenCso Keybuk is correct in how it works08:11
Keybukright08:11
Keybukif you detect the controller, you also just detected the drives08:11
BenChow do you decide to load ide-generic?08:11
Keybukscsi after detecting the controller you have to ask the controller nicely to go detect the drives and get back to you on that08:11
Keybukright08:11
Keybukso because of the way ide works in this regard, what we do is08:11
Keybukiterate each PCI device in order, and load the module for it, waiting for everything to settle in between08:11
mjg59Keybuk: You load piix. That registers itself as an IDE driver, which in turn calls pci_register_driver. After pci_register_driver, the probe routine in piix (piix_init_one) is called. If that binds to something, it does ide_setup_pci_device (which actually allocates the interface)08:12
mjg59Isn't that exactly the same as loading a network driver, except that pci_register_device is done in the IDE layer rather than in the module itself?08:12
Keybukmjg59: right, nothing there returns to userspace08:12
Keybukthanks for verifying that for us :)08:12
Keybukso we know after we've loaded the PCI drivers, if the root device still hasn't shown up, it isn't going to show up later08:12
Keybukso we load ide-generic as a last resort08:13
Keybukscsi looks like "load drivers; wait for root device to show up; abort after 3 minutes"08:13
Keybukide looks like "load drivers sequentially; if root device hasn't shown up, load ide-generic"08:13
mjg59So the original discussion appears to have been based on the false premise that we need to know whether the root device is on ide or not in order to only load ide-generic at the appropriate time?08:14
Keybukwell, no08:14
=== mxpxpod [n=bryan@unaffiliated/mxpxpod] has joined #ubuntu-kernel
BenCno, it's correct08:14
Keybukwe still need to know whether we may or may not need to load ide-generic08:14
BenCthe only incorrect info I got was the loading ide-generic was racy and could cause problems08:14
mjg59Why not load scsi drivers first, then ide drivers (at this point based on PCI IDs), then unconditionally load ide-generic if there's still no rootfs?08:14
mjg59Loading ide-generic isn't going to hurt in the case where the PCI driver has already bound the io-ports08:15
Keybukmjg59: because of FUCKING SATA08:15
mjg59Keybuk: But you've already loaded the driver08:15
KeybukSATA as far as I'm concerned is a scsi driver08:15
BenCmjg59: according to Mith, Fabio's crappy scsi controller breaks if you load ide-generic08:15
Keybukit behaves like scsi drivers08:15
Keybukmost importantly, it returns from modprobe immediately, before performing the controller-specific initialisation08:15
Keybukso modprobe piix-sata will return straight away, and udev will get the devices some few seconds later08:15
Keybukon some older SATA controllers, on a busy system, this can be 10-15s08:16
BenCKeybuk: ok, here's the problems I see....08:16
Keybukloading ide-generic at any point before the SATA SCSI driver has finished can sometimes "steal" the devices from it08:16
Keybukthe solution wwould be to only load ide-generic after the three minutes are up08:16
Keybukwhich means everyone on ISA would have to wait 3 minutes before their machine started booting08:17
mjg59Keybuk: ata_piix calls piix_init, which calls pci_module_init, which will then provide a probe method which registers the pors08:17
BenCKeybuk: IMO, we should detect that ide-generic is needed at install, and make a note of it for initramfs to use :)08:17
mjg59Keybuk: I'm still not seeing how this is different to the IDE case (though there may be something very simple I'm missing)08:17
BenCor when initramfs is built, it could detect that08:17
Keybukmjg59: right, and then it returns to userspace, and libata later calls the probe method08:17
KeybukBenC: I agree.  We can look in /sys in the installer and see whether the driver is ide-generic or not;  if not, use UUID08:18
mjg59Keybuk: Sorry, /which/ probe method?08:18
Keybukmjg59: I can't remember offhand; when I looked through the code (a few months ago) the SATA stuff was very much callback orientated08:19
Keybukand hooked into the SCSI bits of the kernel which are totally async08:19
BenCmjg59: from what I can tell, scsi device probing happens in the scsiX kernel process and not in the module probe code path08:19
Keybukwhereas IDE was all one obvious line of function call codes08:19
mjg59BenC: What are you defining as "scsi device probing"?08:19
Keybukmjg59: "devices handled by the kernel scsi subsystem", which includes SATA ide drives08:19
BenCwhere it probes for devices attached to the controller, as opposed to the controller itself08:19
mjg59BenC: But by that time it's already registered the i/o ports, surely?08:20
Keybukmjg59: not always,no08:20
Keybukloading the controller driver doesn't guarantee anything other than you've opened the controller08:20
BenCmjg59: i/o for the controller yes, but for the ports no08:20
Keybukat that point, you may be lucky enough for the ports to be registered, but not generally08:20
Keybukdevice probing on the controller happens later08:21
mjg59pci_request_regions is done in ata_pci_init_one08:21
mjg59Which is called directly from the ata_piix PCI probe routine08:21
Keybukmjg59: there's an easy way to test it btw08:21
Keybukboot with break=top08:21
mjg59Keybuk: Well, if I had any SATA machines...08:21
Keybukmodprobe -a piix-sata ide-generic08:22
mjg59(That is, SATA machines with legacy i/o)08:22
Keybukabout 7/10 times, your drives will be /dev/hda rather than /dev/sda08:22
Keybukor just trawl malone for the ~100 bugs about that08:22
mjg59Keybuk: Right, I'm willing to believe that, but I still don't see any mechanism by which this can be true for sata but not for IDE08:22
Keybukone of them from mdz :)  his laptop loves to trigger that case08:22
mjg59Uh. When did mdz get a new laptop?08:22
Keybuksorry, strike that; mdz had a different bug, I think08:23
mjg59Yeah, he had ide-generic binding before piix08:23
Keybukyeah08:23
Keybukhe doesn't have a sata laptop08:23
mjg59(Which you've just been telling me is impossible, but :) )08:23
Keybukno we loaded ide-generic first in his case08:23
Keybukthat was a clear-out bug :p08:23
Keybukmjg59: if you start with IDE from the top, you'll see that nothing returns to userspace until the drives are already probed for and bound08:24
Keybukmjg59: if you start with SCSI/SATA from the top, there are plenty of return to userspace b08:24
Keybukpoints before the drives are probed for08:24
mjg59Keybuk: As far as I can tell, in both the IDE and sata cases, the io regions are requested in the call into the core layer that occurs at the end of the pci probe08:24
BenCok, from what I'm hearing, ide-generic should really be the corner case instead of ide in general08:24
KeybukI'm not sure about registering of i/o ports, I wasn't looking for that08:24
Keybukand I don't really understand that08:24
mjg59Keybuk: It's the registration of i/o ports that's then /entire point/08:25
KeybukI was looking at pure "when does the kernel create sdXY and associate it with that driver"08:25
mjg59If the i/o ports are registered, ide-generic won't bind. Otherwise, it will.08:25
Keybukbut ide-generic does bind08:25
mjg59Keybuk: Yes, but that's not the problem08:25
BenCyeah, which driver grabs the ports from the kernel is the winner08:25
mjg59So. How can that situation arise in the sata case, but not the ide one?08:25
BenCbut from what I can tell, there not only i/o per controller, but i/o per device08:25
mjg59BenC: No08:26
mjg59One i/o region per channel08:26
BenCok, one per primary and secondary08:26
KeybukI'm quite happy to change the initramfs code to look like:08:27
Keybuk- load pci drivers08:27
Keybuk- load ide-generic08:27
Keybuk- wait for up to three minutes for the root device to show up08:27
Keybuk- abort08:27
mjg59Keybuk: Could you take a look at drivers/scsi/ata_piix.c and tell me where it enters userspace between loading and hitting line 4843 of libata-core?08:27
mjg59I'm genuinely failing to get this08:28
KeybukI can't see where it gets to libata-core08:29
Keybukah, sorry08:29
Keybukat the bottom08:29
Keybukright so piix_init calls pci_module_init calls piix_init_one calls ata_pci_init_one08:30
mjg59Yes08:30
mjg59Which is identical to the IDE case, as far as I can tell08:30
Keybukright08:31
Keybukand is this true for every single sata driver?08:31
Keybukif it's true that all the sata drivers reserve their regions in init, then I'm happy08:31
Keybukthat may not have been true earlier, or we may have had a different bug08:32
mjg59ahci seems to do it internally, but doesn't do legacy IDE anyway as far as I can tell08:32
Keybukand as long as reserving the regions guarantees ide-generic can't steal the devices08:32
BenCthat still doesn't help out issue where scsi in general is delayed device probing08:32
Keybukhow does it effect that issue?08:32
mjg59BenC: That's fine. Load all SCSI controllers, loda all PCI IDE, load ide-generic, wait up to three minutes (or whatever)08:33
BenCsata behaving like ide doesn't help anything :)08:33
Keybukthis wouldn't distinguish between SCSI and PCI IDE drivers08:33
Keybukcan anyone recall off-hand how we avoid loading both the SATA and PATA driver for a controller?08:34
Keybukdo we do that in the kernel?08:34
mjg59They have different PCI IDs08:34
BenCso basically, from my point of view is, load scsi, ide, ide-generic, loop for 3 minutes checking for device path and/or UUID08:34
mjg59BenC: Yeah08:34
mjg59Keybuk: We still need to deal with the case where ata_piix binds to the controller but can't drive it08:35
Keybukmjg59: we do?08:35
Keybukdo we deal with that nw?08:35
mjg59No08:35
mjg59Which breaks several machines08:35
Keybukoh, well, if it's not a regression... :)08:35
Keybukwas this what we talked briefly about in the DeathSprint?08:35
mjg59It's not a regression, but it's been an open bug since before breezy08:35
mjg59Think so08:35
mjg59ahci and ata_piix share PCI IDs. We should always try to bind ahci first.08:36
Keybukcan't remember what the decision was there though08:36
mjg59Currently, the reverse happens08:36
Keybukright, and some cards work for ahci, some for ata_piix08:36
Keybukand you can't tell which?08:36
mjg59ahci should always fail when it's not drivable by ahci08:36
mjg59But ata_piix will sometimes bind even when it can't drive it08:36
Keybukwhy doesn't ata_piix do the same?08:37
Keybukthat sounds like the bug to me08:37
mjg59Broken BIOS08:37
Keybukthey might have a second ide controller that needs ata_piix08:37
mjg59Or, rather, the detection code in ata_piix assumes the BIOS is sensible08:37
mjg59Keybuk: I can't see how08:37
mjg59Keybuk: You can't get them on PCI cards08:37
mjg59Though it would be more of a problem once ata_piix is needed for driving PATA devices08:38
mjg59(So not a dapper issue)08:38
Keybukyeah, my inclination with anything involving two fighting modules is to fix the modules08:38
Keybukas there's always a corner case where you may HAVE to load both modules on a given machine08:38
mjg59I'm not convinced it's possible to fix the module in this case08:38
Keybukwe have similar examples today where people have OSS and ALSA sound cards in the same machine08:39
Keybukand have filed bugs saying that ALSA OSS emulation doesn't work08:39
mjg59Right, but that's clearly a bug in alsa08:39
Keybukor there's one where they need both snd_bt87x and bt878 loaded; but snd_bt87x won't work if bttv (dep of bt878) has been loaded first, etc.08:40
BenCyeah, that's a bttv bug, IMO08:40
KeybukI08:41
KeybukI do want to think about "blacklist-if" kind of thing for dapper+108:42
mjg59Keybuk: Ok. Almost all sata drivers call pci_request_regions themselves. The only one that doesn't is ata_piix, and that's the case we've already discussed08:42
mjg59So if there's no race in ide, I can't see any race here08:43
Keybukhmm08:45
Keybukthe only thing about modifying modprobe to selectively blacklist, or force a particular order, is it only works if one alias requests both modules08:45
Keybukit fails for when you have two devices which cause both modules to be loaded08:46
Keybukwhich again, leads me back to pointing at the driver itself and saying "iz module bug"08:46
mjg59Well, I'm fairly convinced it's a BIOS bug08:48
Keybukwe can't ship updates to those via APT though :p09:02
mjg59Yeah, so the issue is whether we can work around it09:08
mjg59Based on what you were saying earlier, just loading ahci first should be sufficient, right?09:09
mjg59Because it should bind to the PCI device before you've had a chance to load ata_piix09:09
Keybukthe problem is when ata_piix and ahci are loaded due to different devices09:09
Keybukno, it's a race condition :(09:09
mjg59That's not going to be a problem in this specific case09:09
mjg59Hang on. How is that a race condition and our earlier conversation wasn't?09:10
Keybuk(on phone, will be more eloquent in a sec)09:10
mjg59If there's a race between ahci and ata_piix being loaded, surely there's a race between piix and ide-generic?09:11
mjg59(I'm ignoring the case of multiple PCI devices, because that doesn't apply in the problem case as far as I can tell)09:11
lamontBenC: I could swear that was the case in 2.6.12 for ia64 (at least...)  ISTR uploading 3 times to get it right...09:17
Keybukmjg59: sorry, off phone now09:58
Keybukdamned relationships are so hard work <g>09:58
Keybukrace condition was with multiple devices, as we call modprobe at basically the same time09:59
Keybukwe could do a modprobe patch to handle the case of both drivers being expanded from an alias for a single device09:59
Keybukwould be trivial09:59
Keybukit builds a list of them, and just have a "force-order" config option or something09:59
Keybuksorting depmod output would be another option09:59
mjg59Keybuk: Right. In the cases I've seen, there's only one device10:02
mjg59So just having it load ahci first would be fine10:02
cjbHas anyone seen problems with bluetooth on latest dapper?  I can boot 2.6.15-15, but 2.6.15-19 spawns gdm and then presumably tries to do something with hcid that never returns; I never get any gettys launched.10:31
cjbIf I hit ctrl+alt+del on the empty tty1, the shutdown gets to stopping hcid and then hangs there.10:32
lamont 4) load ide-generic for legacy ISA systems10:36
lamonthrm...10:36
mdzcjb: try to confirm that hcid is in fact getting hung up, and then see if you can find out what's causing it10:36
lamontwhat do we do on ia64 boxes?10:36
mdzlamont: the same thing?10:37
lamontif we loaded any ide modules, i think we have to load it.10:37
lamontunless they fixed 2.6.15 to not do ide init (on ia64) in ide-generic...10:37
mdzlamont: read that "for" as "for the benefit of" not "only for"10:38
lamontah, pok10:38
infinityIndeed.10:38
lamonts/p//10:38
infinityide-generic is still required for anyone with root on /dev/hd* ... More or less.10:38
infinitySo we load it last, after the rest of the world settles (or doesn't)10:38
lamontciik10:39
lamontcool10:39
mdzinfinity: this seems to have been called into question during the discussions today, but we're still loading it at the end10:39
=== lamont slaps his keyboard
infinitymdz: Yeah, I've read the (heated) backscroll a bit while attempting to wake up. :)10:42
mjg59lamont: That hasn't been the case for a while10:43
mjg59lamont: legacy IDE init is done when ide-generic is loaded, but anything else will be driven by the appropriate PCI driver10:43
mjg59Anyway, after discussing it with Scott, we've come to the conclusion that there are no races now10:44
lamontmjg59: the issue I ran into was with an ide hard drive...10:44
mjg59lamont: In the old days, you loaded PCI drivers and then had to load ide-generic in order to get the PCI drivers to bind10:44
mjg59That was a bug, and it's been fixed10:44
lamont0000:a0:02.0 IDE interface: Silicon Image, Inc. (formerly CMD Technology Inc) PCI0649 (rev 02)10:44
lamontah, ok.10:44
lamontfor old days == something including breezy's 2.6.12?10:45
mjg59Probably, yeah10:45
lamontok10:45
mjg59lamont: I don't seem to get ide-generic loaded at all now10:46
=== Keybuk [n=scott@syndicate.netsplit.com] has joined #ubuntu-kernel
=== Keybuk [n=scott@syndicate.netsplit.com] has joined #ubuntu-kernel
=== _human_blip_ [n=mike@mike.nelsonbay.com] has joined #ubuntu-kernel

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!