| pwnguin | alright. so i'm very lazy and only now upgraded my laptop to karmic | 06:04 |
|---|---|---|
| pwnguin | is 3d rendering now a requirement for rendering? ive got some odd rendering quirks with nouveau | 06:05 |
| apw | tseliot, hi, just had another occurance of the corruption and gpu hang.. updated the LP bug | 09:00 |
| tseliot | apw: ah, nice, let me update the upstream report | 09:01 |
| apw | there is another GPU dump on there | 09:02 |
| * tseliot nods | 09:02 | |
| tseliot | done | 09:04 |
| tseliot | apw: on a second thought I think I'll mark 429199 as a duplicate of 429241. After all, all the gpu dumps and all of the details are in the latter | 09:42 |
| apw | sure if you want to do that, i'd been keeping the corruption and non-corruption ones apart, but the former are the only ones i've hit since | 09:43 |
| tseliot | apw: also did you manage to reproduce 429191 and get a GPU dump? | 09:48 |
| apw | tseliot, not seen another occurance on my 10v yet | 09:55 |
| tseliot | apw: ok, mine was still very stable after 23 suspend/resume cycles... | 09:56 |
| apw | it seems much harder to trigger than the isues on the GM45 | 09:57 |
| apw | and i use the GM45 based machine more too | 09:59 |
| * tseliot has only GM45 chipsets at home | 10:05 | |
| tseliot | :-( | 10:05 |
| apw | tseliot, then you are in for a world of pain like me :/ | 10:11 |
| * tseliot nods | 10:12 | |
| tseliot | apw: isn't bug 429241 very similar to what they reported here: https://bugs.edge.launchpad.net/ubuntu/+bug/421736 ? (albeit with different chipsets) | 10:51 |
| ubottu | Launchpad bug 429241 in xserver-xorg-video-intel "[GM45E] i915 graphics corruption and hang" [Undecided,Confirmed] https://launchpad.net/bugs/429241 | 10:51 |
| ubottu | Launchpad bug 421736 in xserver-xorg-video-intel "karmic: Compositing broken on resume from suspend" [Unknown,Confirmed] | 10:51 |
| apw | tseliot, there they seem to talk about 'partially transparent things become invisible ' and definatly i've seen that as part of my symtoms at times. | 10:55 |
| apw | though they don't seem to talk about it then going on to lokc up solid generally | 10:56 |
| tseliot | apw: it's possible that you were facing two bugs, one of which could be 429241. I think I need to update the upstream bug report | 10:58 |
| apw | tseliot, give me a sec, as i think the fix for that was in | 10:59 |
| tseliot | ok | 10:59 |
| apw | the fix toted in bug #419264 was the one below: | 11:01 |
| apw | drm/i915: Fix CPU-spinning hangs related to fence usage using an LRU | 11:01 |
| ubottu | Launchpad bug 419264 in linux "Uses 100% CPU with latest mesa/libdrm update" [High,Fix released] https://launchpad.net/bugs/419264 | 11:01 |
| apw | and that at least is in the karmic kernels i was running | 11:01 |
| apw | tseliot, dunno what that means for your upstream report update :) | 11:03 |
| Ng | apw: talking of G45 hangs, do you happen to know if http://bugs.freedesktop.org/attachment.cgi?id=29294 made it into any trees? | 11:04 |
| Ng | I seem to have entirely lost the bug that that attachment came from :/ | 11:04 |
| Ng | it's referenced in https://bugs.launchpad.net/ubuntu/+source/xserver-xorg-video-intel/+bug/424613 at least | 11:05 |
| ubottu | Launchpad bug 424613 in xserver-xorg-video-intel "[i945GM] GPU lockup with intel 2:2.8.1-1ubuntu1" [Undecided,Confirmed] | 11:05 |
| apw | Ng, no i do not see that commit anywhere | 11:06 |
| Ng | I couldn't even begin to comment on its correctness, but it sounds like a good idea ;) | 11:06 |
| Ng | jbarnes wonders on http://patchwork.kernel.org/patch/45938/ if it's worthy of stable@kernel.org | 11:07 |
| tseliot | apw: sorry, I meant to say that, in addition to the GPU hangs you might be experiencing 421736. Does 419264 have anything to do with it? | 11:07 |
| apw | one referenced the other as a dup i thought, hrm perhaps it got unduped | 11:08 |
| apw | in which case you are right | 11:08 |
| apw | may be there are two issues, that one, and general hanging | 11:08 |
| Ng | aha got it, it's https://bugs.freedesktop.org/show_bug.cgi?id=22336 | 11:08 |
| ubottu | Freedesktop bug 22336 in Driver/intel "[i965] GPU hang with compiz, active system use" [Critical,New] | 11:08 |
| apw | Ng, ok it does appear to be in the drm-intel drm-intel-next branch | 11:10 |
| apw | and interestingly its GM45 specific if i am reading the leader correctly | 11:11 |
| tseliot | apw: as regards this patch http://patchwork.kernel.org/patch/45938/ , would it be possible to put it in a PPA for testing (Intel would like to get some feedback about it)? This is for Freedesktop bug 22336 , as Ng said | 11:12 |
| ubottu | Freedesktop bug 22336 in Driver/intel "[i965] GPU hang with compiz, active system use" [Critical,New] http://bugzilla.freedesktop.org/show_bug.cgi?id=22336 | 11:12 |
| Ng | apw: I have no idea what the process around this is, if upstream like it that much would we naturally cherry pick it? or wait for it to hit a 2.6.31.X? | 11:12 |
| apw | well as its has been indicated as likely stable material we would expect to see it come via 31.x when that gets going | 11:13 |
| Ng | ok :) | 11:13 |
| apw | however, sa its gm5 specific and talks about hangs, and i have GM45 with hangs ... i would be temped to pick it up and test it myself to see if it fixes _my_ issues | 11:13 |
| Ng | hehe | 11:13 |
| Ng | I too have GM45 and odd random hangs, so I'd be happy to test it too | 11:13 |
| Ng | but I can't reproduce the hangs in any meaningful way, so the best I could do is "err yeah it hasn't hung today". my suspend seems to be broken too atm, so I'm having to do shutdowns when I go home, which is ruining my long uptime tests of things ;) | 11:14 |
| apw | how it could have had this issue for so long and never hit it is a mystery | 11:14 |
| apw | Ng, whats your suspend broken symptoms? | 11:15 |
| tseliot | apw: please test it if you can | 11:15 |
| apw | tseliot, i'll put together a test kernel against my GM45 curruption and hang bug and see hwo that pans out | 11:15 |
| Ng | apw: typically my laptop just doesn't suspend at all recently. it worked with 31-9, but around the -10 upgrade I also started testing all of Keybuk's boot stuff, so it's entirely possible it's his fault | 11:15 |
| tseliot | apw: great, thanks a lot | 11:16 |
| apw | heheh | 11:16 |
| tseliot | apw: did you have the desktop effects enabled when you experienced the suspend/resume hang with the mini 10v? | 11:46 |
| apw | yes always have compiz enabled | 11:47 |
| tseliot | the netbook-remix UI tricked me into thinking that compiz was enabled... | 11:47 |
| tseliot | ok, I'll try with compiz | 11:47 |
| tseliot | apw: can you try what Chris suggested in 429241 when you can, please? | 11:59 |
| apw | tseliot, does that imply ... i need that additional file when it next occurs, or is the live data good enough? | 12:02 |
| tseliot | apw: I think you need to get it (together with the GPU dump, etc.) after you reproduce the problem | 12:04 |
| tseliot | sorry for being a pain :-P | 12:04 |
| apw | tseliot, these things happen, its a complex problem | 12:04 |
| apw | tseliot, i've also asked chris if he can tell if we have a split command in those two, and pointed him to the wrap patch i am about to test | 12:07 |
| tseliot | apw: excellent, thanks | 12:09 |
| * tseliot -> lunch | 12:18 | |
| tseliot | apw: I sent you an email. It looks like you will have some testers with i965 for the patch | 14:34 |
| apw | tseliot, thanks ... | 14:40 |
| apw | i also see Albert has an idea what it might be | 14:40 |
| tseliot | apw: Alberto = tseliot ;) | 14:41 |
| apw | Alberto == tseliot, Albert == someone else tho. | 14:41 |
| tseliot | apw: who's Albert then? | 14:42 |
| * albert23 is Albert | 14:42 | |
| apw | a good question indeed :) | 14:42 |
| apw | hello :) | 14:42 |
| apw | albert23, i do indeed have one such render error | 14:43 |
| apw | syslog.1:Sep 15 08:14:32 dm kernel: [ 7689.536305] render error detected, EIR: 0x00000010 | 14:43 |
| albert23 | Is that followed by the PBTBL_ER? | 14:43 |
| tseliot | aah, ok ;) | 14:43 |
| apw | albert23, no it doesn't seem to be | 14:43 |
| apw | doh yes it is | 14:44 |
| apw | egrep errror | 14:44 |
| apw | hmm happened in the middle of the suspend too | 14:44 |
| apw | Sep 15 08:14:32 dm kernel: [ 7689.536305] render error detected, EIR: 0x00000010 | 14:44 |
| apw | [...] | 14:44 |
| apw | Sep 15 08:14:32 dm kernel: [ 7689.536315] page table error | 14:44 |
| apw | Sep 15 08:14:32 dm kernel: [ 7689.536316] PGTBL_ER: 0x00100000 | 14:44 |
| apw | Sep 15 08:14:32 dm kernel: [ 7689.536318] [drm:i915_handle_error] *ERROR* EIR stuck: 0x00000010, masking | 14:44 |
| apw | Sep 15 08:14:32 dm kernel: [ 7689.552146] PM: suspend devices took 1.648 seconds | 14:44 |
| albert23 | Looks like it | 14:45 |
| apw | ok so i'll find and add that patch to my test kerenl | 14:45 |
| albert23 | Might be worth to try the mesa fix then. It points to a freedesktop bug that's pretty similar to our bug | 14:46 |
| albert23 | freedesktop bug 23254 | 14:46 |
| ubottu | Freedesktop bug 23254 in Drivers/DRI/i965 "Compiz doesn't survive suspend/resume cycle" [Major,Resolved: fixed] http://bugzilla.freedesktop.org/show_bug.cgi?id=23254 | 14:46 |
| apw | tseliot, ... is an updated mesa with that fix something you can spin for me | 14:51 |
| apw | i'll get this kernel made available too | 14:51 |
| tseliot | apw: ok, let me check what's changed in mesa and I'll put the package in a PPA | 14:52 |
| apw | sounds gooood to me | 14:53 |
| tseliot | hah, it's just a one line patch | 14:56 |
| apw | tseliot, i see it would fit with the patch suggested for the kernel, checking the 'relocation points to within the object' | 14:57 |
| tseliot | apw: let's hope that a combination of the two patches fixes the bug | 14:57 |
| tseliot | at least one | 14:57 |
| apw | i am hoping the mesa fix is the whole thing | 14:59 |
| apw | that the kernel change just catches it without the mesa fix ... will test that next | 14:59 |
| tseliot | sounds like a plan :-) | 14:59 |
| apw | tseliot, though both changes sound like things that should go to stable imo | 15:03 |
| tseliot | apw: I agree. I think we can upload them after the freeze | 15:04 |
| apw | if that mesa fix works we might want to see if ti can be uploaded for A6, as its a bit of a mess | 15:05 |
| tseliot | apw: slangasek might disagree ;) | 15:06 |
| apw | he might, then he might not | 15:06 |
| tseliot | heh, it's worth trying | 15:06 |
| apw | obviously we need to test it first :) | 15:07 |
| tseliot | right. I'm building it now | 15:10 |
| apw | tseliot, most excellent | 15:10 |
| tseliot | it built \o/ | 15:28 |
| apw | yay | 15:32 |
| tseliot | apw: I'm using this new PPA: https://launchpad.net/~albertomilone/+archive/x-testing | 15:33 |
| apw | tseliot, sounds great | 15:34 |
| tseliot | apw: I'll let you know when the upload is complete | 15:34 |
| apw | thanks | 15:34 |
| tseliot | apw: finished. It can take a while for it to build though | 15:39 |
| apw | yep and to publish, and ... and i need to test without first too to see if this other patch would have found it | 15:40 |
| tseliot | ok | 15:46 |
| apw | tseliot, this kernel patch is _well_ unhappy with the default mesa | 15:48 |
| apw | 111.176361] [drm:i915_gem_object_pin_and_relocate] *ERROR* Relocation beyond target object bounds: obj ffff88010895da80 target 2 delta 153751552 size 16777216. | 15:48 |
| apw | [ 111.176366] [drm:i915_gem_execbuffer] *ERROR* Failed to pin buffers -22 | 15:48 |
| apw | [ 111.249110] [drm:i915_gem_object_pin_and_relocate] *ERROR* Relocation beyond target object bounds: obj ffff88010895da80 target 2 delta 153751552 size 16777216. | 15:48 |
| apw | [ 111.249115] [drm:i915_gem_execbuffer] *ERROR* Failed to pin buffers -22 | 15:48 |
| tseliot | apw: what is it that fails? | 15:48 |
| tseliot | oh | 15:48 |
| apw | tseliot, producing about 5 on login, thats before any apparent problems | 15:49 |
| apw | half my screen is missing too :) | 15:49 |
| apw | so _if_ that check is right, its well broken :) what does the mesa fix change? | 15:49 |
| tseliot | apw: it fixes the relocation delta | 15:50 |
| tseliot | drm_intel_bo_emit_reloc(brw->wm.surf_bo[unit], | 15:50 |
| tseliot | offsetof(struct brw_surface_state, ss1), | 15:50 |
| tseliot | region_bo, | 15:50 |
| tseliot | - surf.ss1.base_addr, | 15:50 |
| tseliot | + surf.ss1.base_addr - region_bo->offset, | 15:50 |
| tseliot | I915_GEM_DOMAIN_RENDER, | 15:50 |
| tseliot | I915_GEM_DOMAIN_RENDER); | 15:50 |
| tseliot | } | 15:50 |
| apw | oh thats pretty seriously off :) | 15:50 |
| tseliot | no wonder you're getting some weird relocation problems | 15:51 |
| tseliot | ;) | 15:51 |
| superm1 | tseliot, would you mind pushing bug 385658 to your x-testing PPA so we can get some testing going on it? | 15:51 |
| ubottu | Launchpad bug 385658 in xorg-server "'nv' is selected when no xorg.conf is present even if it doesn't support the nvidia hardware" [Unknown,Confirmed] https://launchpad.net/bugs/385658 | 15:51 |
| tseliot | superm1: it's on my todo list. I haven't had the time to look at the patch yet | 15:52 |
| superm1 | ok | 15:52 |
| tseliot | it looks like there's a problem in the chroot of the PPA :-/ | 15:53 |
| apw | tseliot, ? noooooo i need that .deb :) | 15:53 |
| tseliot | apw: do you use i386? | 15:54 |
| tseliot | if so, I can upload my packages to my webspace | 15:54 |
| apw | amd64 on the box in question | 15:54 |
| apw | damn damn damn | 15:55 |
| tseliot | :-/ | 15:55 |
| tseliot | apw: do you want to build the source yourself? | 15:55 |
| apw | tseliot, can do | 15:55 |
| apw | how painful is it deps wise | 15:56 |
| tseliot | https://launchpad.net/~albertomilone/+archive/x-testing/+files/mesa_7.6.0~git20090817.7c422387-0ubuntu4.diff.gz | 15:56 |
| apw | you should report that thing too | 15:56 |
| tseliot | https://launchpad.net/~albertomilone/+archive/x-testing/+files/mesa_7.6.0~git20090817.7c422387-0ubuntu4.dsc | 15:56 |
| tseliot | apw: I've just reported that | 15:56 |
| tseliot | https://launchpad.net/~albertomilone/+archive/x-testing/+files/mesa_7.6.0~git20090817.7c422387.orig.tar.gz | 15:56 |
| tseliot | apw: don't you use pbuilder? | 15:57 |
| apw | tseliot, pbuilder no | 15:58 |
| apw | i build so very few packages i've never needed it | 15:58 |
| apw | its on my list to investigate 'when i have a moment' and has been for 4 months | 15:58 |
| tseliot | apw: that would make things easier for you. This way I can build packages for karmic, jaunty, etc. | 15:58 |
| apw | i have the chroots etc, so i can build on the right one etc | 15:59 |
| tseliot | apw: to be honest, I use "pbuild" which is a script that makes pbuilder even easier | 15:59 |
| * apw adds that tot he list | 16:01 | |
| * tseliot has a look at superm1 's patch | 16:02 | |
| apw | tseliot, that didn't build for me | 16:11 |
| apw | /usr/include/drm/radeon_cs.h:181: error: 'RADEON_GEM_DOMAIN_VRAM' undeclared (first use in this function) | 16:11 |
| tseliot | apw: how did you build it? | 16:12 |
| apw | debuild -b in a chroot | 16:12 |
| tseliot | apw: did you update your chroot and did you make sure that all the dependencies were satisfied? | 16:14 |
| apw | i installed the build deps yeah | 16:15 |
| * apw updates it more | 16:15 | |
| tseliot | weird | 16:16 |
| * apw gets grumpy ... this should be easy to test | 16:16 | |
| apw | tseliot, the range check is looking good | 16:16 |
| apw | screen is mush, but very stable | 16:16 |
| apw | 10s/r without loss | 16:16 |
| tseliot | apw: nice | 16:17 |
| apw | its puking every update of course, mesa is a bad boy | 16:17 |
| tseliot | yes, I know... | 16:17 |
| tseliot | welcome to my world | 16:17 |
| apw | as bad as the kernel ... | 16:18 |
| tseliot | heh, right | 16:18 |
| apw | and hanging X == the worst of both in the mix | 16:20 |
| tseliot | hehe | 16:21 |
| tseliot | BTW: | 16:21 |
| tseliot | +# Copyright 2007 Red Hat Inc. | 16:21 |
| tseliot | +# This crappy script written by Dave Airlie to avoid hassle of adding | 16:21 |
| tseliot | +# ids in every place. | 16:21 |
| tseliot | LOL | 16:21 |
| apw | heh | 16:22 |
| apw | tseliot, so what package do i need to install from the output of this build ... i assume not all of it is needed | 16:25 |
| albert23 | apw: I only updated libgl1-mesa-glx and libgl1-mesa-dri | 16:27 |
| apw | cool. i'll do the same ... once it finishes building yawn! | 16:28 |
| tseliot | apw: libgl1-mesa-glx libgl1-mesa-dri libglu1-mesa | 16:28 |
| apw | tseliot, is there a any docs on what the various layers actually do | 16:29 |
| apw | i assume mesa is the hardware abstraction layer for 3d | 16:29 |
| tseliot | apw: also, can you try and see if Option "AccelMethod" "EXA" (in the Device section of xorg.conf) helps with the ATI card? | 16:29 |
| jcristau | isn't exa the default? | 16:31 |
| apw | i thought so, but hey, its a heap anyway ... can't make it worse | 16:31 |
| tseliot | jcristau, apw: the log says that xaa is being used (don't ask me why) | 16:32 |
| apw | tseliot, it occurs to me that the symptoms on there included the missing borders and stuff | 16:32 |
| apw | ignore that thought | 16:33 |
| apw | tseliot, ok chaning that and enablinb compiz looks like its hung completly | 16:33 |
| tseliot | apw: as regards mesa, try http://www.mesa3d.org/ and http://dri.freedesktop.org/wiki/ | 16:33 |
| tseliot | apw: ouch, can you add a comment here, please? http://bugs.freedesktop.org/show_bug.cgi?id=23928 | 16:34 |
| ubottu | Freedesktop bug 23928 in Driver/Radeon "ATI graphics corruption with compiz" [Normal,New] | 16:34 |
| tseliot | superm1: I don't see why that shouldn't work. Did you test it? | 16:35 |
| superm1 | tseliot, Yup | 16:36 |
| superm1 | i've got some of this hardware that fails without it | 16:36 |
| tseliot | superm1: ok, let me add a comment (basically my +1) in the bug report | 16:37 |
| superm1 | i'm not really sure more widespread testing is necessary given how straightforward it is, just needed a few pairs of eyes to make sure it looked acceptable (imo) | 16:37 |
| tseliot | superm1: caution is never enough. Not that I see why your patch should fail. A bunch of testers should be more than enough. | 16:46 |
| tseliot | apw: any luck with the build? | 16:46 |
| superm1 | Ok, can you push it to a PPA then? | 16:46 |
| apw | tseliot, yep just installing and booting it now | 16:47 |
| tseliot | apw: great | 16:47 |
| tseliot | superm1: would this be ok? https://launchpad.net/~albertomilone/+archive/x-testing | 16:47 |
| superm1 | sure | 16:47 |
| apw | tseliot, ok that mesa update gets rid of the kernel bitching about the reloc | 16:51 |
| tseliot | superm1: uploaded. I hope the chroot it's not broken | 16:51 |
| tseliot | apw: and can you see the whole screen now? | 16:52 |
| apw | yep normal service is also returned for my screen ... | 16:53 |
| tseliot | :-) | 16:53 |
| * apw does some suspend/resume testing | 16:54 | |
| apw | 5 s/r's and counting | 16:55 |
| * tseliot made the dell mini 10v freeze completely (black screen) without the patches after 6 suspend/resume cycles | 16:57 | |
| * tseliot is downloading the new kernel for testing | 16:58 | |
| apw | tseliot, _nice_ | 16:59 |
| tseliot | heh | 16:59 |
| apw | my machine seems pretty stable so far ... obviously only time will tell but 4 s/r's were typically fatal | 16:59 |
| tseliot | apw: very well, I'll test the fix here too | 17:00 |
| tseliot | superm1: can you upload a package for me, please? (slangasek is ok with this) | 17:29 |
| superm1 | sure | 17:29 |
| tseliot | superm1: http://albertomilone.com/ubuntu/karmic/mesa/toupload.txt Thanks | 17:32 |
| superm1 | tseliot, you'll have to point me at it though.. :) | 17:32 |
| superm1 | Ok. cool | 17:32 |
| superm1 | is that the same one that apw just test built locally? so i can skip a test build right? | 17:33 |
| tseliot | superm1: yes, I only changed a detail in the changelog | 17:34 |
| superm1 | ok thanks | 17:34 |
| apw | tseliot, do we know why the PPA builders are sick on this one? | 17:35 |
| superm1 | tseliot, i didn't see the -nv on you x-testing ppa yet, you sure you uploaded it? | 17:35 |
| tseliot | superm1: yes, I did but the PPA is broken :-/ | 17:36 |
| superm1 | but the source didn't even publish? | 17:37 |
| tseliot | apw: I'll bug lamont again | 17:37 |
| superm1 | must be *very* broken :) | 17:37 |
| tseliot | superm1: no, sorry | 17:37 |
| superm1 | is it just your PPA, or the whole PPA system? | 17:37 |
| superm1 | i can just pop it on one of mine instead if 'ts just yours | 17:37 |
| tseliot | superm1: I hope it's just mine | 17:38 |
| apw | tseliot, is it something to do with upstart and mountall and things not installing? | 17:39 |
| apw | if so its affecting the whole build system ... someone broke it | 17:39 |
| superm1 | awesome | 17:40 |
| tseliot | apw: The following packages have unmet dependencies: sysv-rc: Breaks: initscripts (< 2.86.ds1-63) but 2.86.ds1-61ubuntu16 is to be installed E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages. | 17:40 |
| tseliot | d'oh! | 17:40 |
| apw | yeha thats the overall mess. the archive admins are applying hammers to it | 17:41 |
| apw | it sounds like something which has been laying in wait for a while, a dep circle which is only hit if you upload both packages at the same time | 17:42 |
| apw | tseliot, hows the 10v, how many s/r did it survive? | 17:42 |
| tseliot | apw: 6 or 7 without problems | 17:43 |
| apw | most unexpected | 17:43 |
| apw | now my 945 is fixed i am going to test on here too | 17:43 |
| apw | (another 10v) | 17:43 |
| tseliot | good :-) | 17:43 |
| apw | if i add the kernel only and see the same vomit from Reloc tests i think we could | 17:44 |
| apw | call my other bug a dup of the main one | 17:44 |
| apw | will do the testing now | 17:44 |
| tseliot | :-) | 17:46 |
| tseliot | 12 suspend/resume cycles and still no problem whatsoever | 17:50 |
| apw | tseliot, yay | 17:53 |
| tseliot | apw: ok, I guess that 20 suspend/resume cycles are enough ;) | 17:58 |
| tseliot | I should stop before I break the netbook lid :-P | 17:58 |
| apw | heh yeah ... you can wake it with the power button | 17:59 |
| tseliot | right | 17:59 |
| tseliot | apw, superm1: it looks like it will take a while before the PPAs are fixed | 18:07 |
| tseliot | just FYI | 18:08 |
| superm1 | archive is broke too it looks like | 18:08 |
| superm1 | that mesa upload FTBFS all across the board | 18:08 |
| tseliot | :-( | 18:11 |
| * tseliot -> dinner | 18:16 | |
| === michael__ is now known as michaellarabel | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!