bjf | jjohansen: about? | 03:13 |
---|---|---|
jjohansen | bjf: hey whats up | 04:37 |
bjf | jjohansen: i'm trying to figure out if these apparmor qrt failures are due to missing pkgs | 04:37 |
bjf | http://kernel.ubuntu.com/beta/testing/test-results/rizzo.2012-09-28_01-40-19/qrt/default/qrt.test-apparmor.py/debug/qrt.test-apparmor.py.DEBUG.html | 04:38 |
bjf | jjohansen: ^ | 04:38 |
jjohansen | bjf: they python binding failures seem to be | 04:38 |
bjf | jjohansen: i "think" i fixed those and have rerun the tests | 04:38 |
jjohansen | okay, you have a longpath failure, which machine | 04:39 |
jjohansen | so I can log in and look at it | 04:39 |
bjf | jjohansen: rizzo | 04:39 |
bjf | jjohansen: did you get your qa lab creds? | 04:39 |
jjohansen | okay, longpath is not a python binding | 04:39 |
jjohansen | I suspect it is something to do with mount/chroot | 04:39 |
jjohansen | bjf: okay, I'll look into it tonight, I haven't been able to replicate yet and I have tried on the qa infastructure | 04:40 |
bjf | jjohansen: so you can get into the qa lab machines? | 04:41 |
jjohansen | if you can leave rizzo as is, I can get to it in an hour or two, /me is trying to get kids to be | 04:41 |
jjohansen | bjf: yeah I should be able to I have been given vpn access and keys | 04:41 |
bjf | jjohansen: no problem, i'm done for the night, won't be back to it til tomorrow a.m. | 04:41 |
jjohansen | okay I should have something for you in the morning | 04:42 |
bjf | jjohansen: you want me to check back with you in about an hour to see if you are getting in? | 04:42 |
jjohansen | sorry I got distracted by other problems like the weird stack failure, that apw was dealing with earlier in the week | 04:42 |
bjf | np | 04:42 |
bjf | that took priority | 04:42 |
jjohansen | bjf: give me a couple minutes I'll try it now | 04:43 |
bjf | ack | 04:43 |
jjohansen | bjf: its going to take /me more than a couple minutes | 04:45 |
bjf | np, i'll check back in about an hour .. if that's ok | 04:45 |
=== henrix_ is now known as henrix | ||
=== smb` is now known as smb | ||
smb | morning | 08:12 |
diwic | hmm, could it be that the mainline 3.6 kernel does not output snd_printk messages by default, but the ubuntu 3.5 kernel does? | 08:16 |
smb | "Could be" could always be. Though I *know* as much as you do there... | 08:19 |
ogra_ | hmm, do kernel buillds use -Werror=maybe-uninitialized by default ? | 08:21 |
ppisati | ogra_: if we would use -Werror=maybe-uninitialized than NONE of our kernels will ever build :) | 08:29 |
ogra_ | ppisati, well, then i wonder where it comes from in the linux-ac100 build :) | 08:29 |
ogra_ | probably an upstream Makefile from firendly nvidia or so :) | 08:30 |
* ogra_ goes digging | 08:30 | |
ogra_ | ogra@anubis:~/Devel/packages/linux-ac100-3.1.10$ grep -r Werror *|grep tegra|grep Makefile|wc -l | 08:33 |
ogra_ | 6 | 08:33 |
ogra_ | ogra@anubis:~/Devel/packages/linux-ac100-3.1.10$ | 08:33 |
ogra_ | yay, fun | 08:33 |
* ogra_ wouldnt mind that they set this *if* they would actually make sure there are no warnings ... tsk | 08:36 | |
=== ivoks is now known as SiIverSpace | ||
=== SiIverSpace is now known as ivoks | ||
Nosophorus | hi | 12:08 |
* henrix -> SIGFOOD | 12:35 | |
* ppisati goes out to run an errand, brb | 13:50 | |
rtg | ogasawara, whaddya think of git://kernel.ubuntu.com/rtg/ubuntu-quantal.git ixgbe ? | 14:30 |
* ogasawara looks | 14:30 | |
rtg | ogasawara, it brings ixgbe up to 3.6-rc7 | 14:31 |
rtg | hmm, though looking at the diff I think a doc patch touched a bunch of other stuff. better figure that out. | 14:32 |
ogasawara | rtg: ah, a full update rather than just the SR-IOV patch. I could go for that. | 14:34 |
rtg | looks like its 49ce9c2cda18f62b13055dc715e7b514157c2da8 which is benign documentation and cleanup | 14:34 |
rtg | ogasawara, I need a test bed for ixgbe, any thoughts ? I wonder if tangerine or gomeisa has this. | 14:35 |
ogasawara | rtg: I was just thinking the same thing | 14:35 |
rtg | tangerine is igb | 14:35 |
rtg | so is gomeisa | 14:35 |
ogasawara | rtg: what about any of the boxes bjf has in the lab | 14:36 |
rtg | ogasawara, what is your shark bay ? unlikely since ixgbe is 10GB IIRC | 14:36 |
ogasawara | rtg: I don't think it's ixgbe, but I'll double check | 14:37 |
rtg | sconklin_, can you root around on your lab boxes to see if any are ixgbe ? | 14:37 |
sconklin_ | ack, odds are low | 14:38 |
rtg | sconklin_, you've got a newer emerald class box I think. that is a possibility. | 14:39 |
sconklin_ | rtg: it's server-only, right? No sense checking commercial laptops? | 14:39 |
rtg | sconklin_, right. ixgbe is 10GB | 14:39 |
rtg | yep, "Intel 10 Gigabit PCI Express Linux driver" | 14:40 |
sconklin_ | I don't have an emerald class box, the 'biggest' boxed I have are sandy bridge and ivy bridge desktops | 14:42 |
sconklin_ | they're both gigabit, I just checked | 14:42 |
bjf | sconklin_: he's asking about machines in the lab | 14:42 |
sconklin_ | oh hell | 14:42 |
sconklin_ | ok, stand by | 14:42 |
bjf | sconklin_: statler is igb | 14:44 |
rtg | I've got plenty of igb boxes | 14:45 |
bjf | sconklin_: rizzo is broadcom | 14:46 |
bjf | rtg, so no, we don't have any | 14:46 |
rtg | bjf, ok, thanks. maybe we'll just upload it and see if it sticks :) The intel guys are pretty good about ethernet driver testing. | 14:47 |
bjf | rtg, sorry, one more system to check | 14:47 |
jjohansen | bjf: so I hit a different bug last night, and haven't tracked down the aa problem yet | 14:47 |
ogasawara | rtg: if anything, I can ask yingying to test | 14:47 |
bjf | jjohansen: heh, np. you want me to leave that system for you? | 14:47 |
jjohansen | bjf: can I just reboot the system? | 14:47 |
rtg | ogasawara, you think she's got one ? | 14:47 |
bjf | jjohansen: yup | 14:47 |
ogasawara | rtg: she should have access to one | 14:48 |
jjohansen | bjf: so the bug I hit is that loop mounts aren't getting unmounted, something is hold a ref to them | 14:48 |
rtg | ogasawara, ok, I'll request that in the bug. | 14:48 |
rtg | ogasawara, can she build, or should I do it for her | 14:49 |
rtg | ? | 14:49 |
ogasawara | rtg: you should build for her | 14:49 |
rtg | ack | 14:49 |
bjf | rtg, nope, my other lab server is igb as well | 14:58 |
rtg | bjf, ok, thanks for looking | 14:58 |
ogasawara | smb: I've opened up 1009098 | 15:02 |
smb | ogasawara, Ok cool, then as soon as people like my source package they can upload. | 15:03 |
ogasawara | smb: sweet, thanks! | 15:03 |
smb | Daviey, ^ FYI | 15:04 |
Daviey | smb: rocking | 15:08 |
Daviey | smb: shall we break some stuff? | 15:09 |
smb | Daviey, Why not? Its Friday, close to release... perfect time to break things... | 15:09 |
Daviey | wfm | 15:10 |
smb | zul, Or what ya say? :) | 15:10 |
rtg | Daviey, smb, I think you guys should have several beers first | 15:10 |
Daviey | *glug* | 15:11 |
smb | rtg, Oh right, need to wait until beer thirty... and log off irc... | 15:11 |
rtg | smb, and only then should you upload :) | 15:12 |
smb | :D | 15:12 |
zul | smb: what are we breaking? | 15:12 |
Daviey | smb: dpkg-source: error: Version number suggests Ubuntu changes, but Maintainer: does not have Ubuntu address | 15:12 |
rtg | you know ogasawara can fix it. | 15:12 |
smb | zul, Only Xen | 15:12 |
Daviey | smb: How did you build a source package? | 15:12 |
zul | smb: ah | 15:12 |
smb | Daviey, Was just a warning for me | 15:13 |
Daviey | smb: and you thought best to ignore it? :) | 15:13 |
smb | Daviey, Of course! :) | 15:13 |
smb | canonical ... ubuntu... who cares | 15:14 |
jjohansen | bjf: okay so the failure is in the test suite, and is around the core dump pattern | 15:26 |
jgriffith | smb: So what can I do to help get to the bottom of this LVM problem? | 15:37 |
xnox | what LVM problem? | 15:37 |
jgriffith | xnox: OpenStack LVM issue | 15:37 |
smb | jgriffith, One thing would be to let me know whether the steps in the bug report will cause the problem for you | 15:38 |
jgriffith | xnox: In a nut-shell dd if=/dev/zero to an LVM snapshot hangs the system | 15:38 |
jgriffith | smb: They did | 15:38 |
smb | bug 1023755 | 15:38 |
jgriffith | smb: It's random however | 15:38 |
ubot2 | Launchpad bug 1023755 in linux "Precise kernel locks up while dd to /dev/mapper files > 1Gb (was: Unable to delete volume)" [Undecided,Confirmed] https://launchpad.net/bugs/1023755 | 15:38 |
smb | jgriffith, Hm, so I should run that in a loop (that was not completely clear in the report) | 15:39 |
jgriffith | smb: Sorry... | 15:39 |
ogasawara | ppisati: can I close bug 746137 for the linux task as well? ie set Invalid | 15:39 |
ubot2 | Launchpad bug 746137 in linux "Page allocation failure on Pandaboard and Beagle XM" [High,Confirmed] https://launchpad.net/bugs/746137 | 15:39 |
jgriffith | smb: Really even running in a loop probably won't do it for ya TBH | 15:39 |
smb | jgriffith, The other thing is, would making the snapshot storage slightly bigger than the original volume be a work-around usable? | 15:40 |
* xnox is confused why would you zero out LVM snapshot, instead of asking lvm which blocks the snapshot used and zero those out. | 15:40 | |
xnox | is this reproducible in quantal? | 15:40 |
smb | xnox, That would be another way | 15:40 |
jgriffith | smb: I can try it,but I don't see how that would help | 15:40 |
jgriffith | xnox: Not sure, I'll look at it today | 15:40 |
bjf | jjohansen: ack. you'll see that the necessary fixes are applied to the qrt tests? | 15:40 |
smb | xnox, jgriffith Unfortunately for that you need to use dmsetup commands (I believe) | 15:41 |
jgriffith | xnox: I did pull the quantal kernel into 12.04 and couldn't reproduce that way | 15:41 |
jjohansen | bjf: yep working on and then I finally get to the yama test as well | 15:41 |
bjf | jjohansen: very good. just let me know and i'll give it a spin. | 15:41 |
jgriffith | smb: What would require dmsetup commands? Quantal? | 15:41 |
xnox | interesting. precise has a very old lvm2 unfortunately (we missed opportunity to merge lvm2 on time) | 15:42 |
xnox | oh well. | 15:42 |
smb | jgriffith, No converting the snapshot area into a linear volume | 15:42 |
jgriffith | smb: Oh, I see where you're going | 15:43 |
jgriffith | Sorry missed that above | 15:43 |
smb | As well as apparently a working lvconvert solution... seems to lock up in Precise sadly | 15:43 |
jgriffith | smb: Your lvconvert locks up? Sounds familiar | 15:43 |
smb | jgriffith, Similar but I am not sure it is the same. It seems to do things but then messing up the refcounting so the snapshot cannot be removed and any other lvm command locks up as well | 15:45 |
jgriffith | smb: So that's very similar behavior at any rate | 15:45 |
smb | I did not dig down there because it was something else and the dd's were acting as they should be for me | 15:46 |
jgriffith | smb: yeah, the dd thing is interesting to try and reproduce | 15:46 |
jgriffith | smb: You'll notice I initially marked this bug as invalid/not reproducible for the same reason | 15:47 |
jgriffith | smb: But then I start hitting it more often, and for a while I'd hit it almost EVERY time I ran | 15:47 |
jgriffith | smb: Nothing different that I could determine on the systems | 15:47 |
jgriffith | smb: Using Vagrant and Virtual box so the steps/setup should be the same in every case | 15:47 |
smb | jgriffith, Right, maybe the copy on write locks up rarely... Hm, when you hit it, can you in any way cause a crashdump | 15:48 |
jgriffith | smb: Well, that *ONE* time the system was still resposive | 15:49 |
jgriffith | smb: But typically, the entire system is locked up and I can't do anything except power it off | 15:49 |
jgriffith | smb: (virtually of course) | 15:49 |
smb | jgriffith, not even sysrq keys? | 15:49 |
jgriffith | smb: Nope... nothing | 15:49 |
smb | Too bad. For a Xen guest it would be simple to get a dump. I should probably investigate how / whether that can be done for KVM guests, too | 15:50 |
smb | (of course the xen guest I tried on did not break) | 15:51 |
jgriffith | smb: Wonder if it would be worth the pain of running devstack to try and reproduce that way? | 15:51 |
smb | jgriffith, Probably first would try to run the test more often to get it to happen with basic lvm commands. | 15:53 |
jgriffith | smb: Understood | 15:53 |
jgriffith | So the things I've found to be key are the size (>= 2G) and an LVM snap | 15:54 |
jgriffith | smb: Neither LVM every being written to or used | 15:55 |
smb | jgriffith, Ok, yeah. I did set it up that way. There were a few modifications to the commands I had to do (noted in the report) but at least I am trying with a loopfile based VG as well | 15:56 |
jgriffith | smb: cool... that should do it | 15:57 |
smb | Hopefully. I will hack on a script to loop and then can let it run over some time over the weekend | 15:58 |
jgriffith | smb: great.. thanks! Keep me posted, and anything I can do to help just say the word | 15:58 |
smb | jgriffith, Sure I try to keep the bug report updated with any progress or questions | 15:59 |
smb | rtg, Btw, I reached a state with bug 1021471 where I cannot really say which of the oddities I see would be the real problem. Perhaps a bell rings for you (most of the details are in the upstream bug). Otherwise I fear this needs someone from upstream actually caring. | 16:06 |
ubot2 | Launchpad bug 1021471 in linux "clone() hang when creating new network namespace (dmesg show unregister_netdevice: waiting for lo to become free. Usage count = 2)" [High,Triaged] https://launchpad.net/bugs/1021471 | 16:06 |
rtg | smb, prolly. maybe serge ? | 16:07 |
smb | rtg, Err him being expert in net/ipv4/* ? | 16:08 |
rtg | smb, no, I was thinking he's an expert with name spaces | 16:08 |
smb | rtg, Ah ok. Well in that case it might only indirectly related to name spaces. I have not really checked that but it could be that even in the initial one there would be dangling references. Just that in that case nobody cares | 16:09 |
smb | So far it looks like an odd entry in the route cache. But whether it should not be there or be there with different values or other flags I got no clue | 16:10 |
smb | And of course upstream replaced the whole thing again | 16:11 |
* smb notes that percpu refcounting alone sounds scary enough | 16:12 | |
sforshee | bjf, I've got another test kernel for brcmsmac. I've drastically reduced the amount of buffering done by the driver, and I haven't seen the flush warning once. | 16:17 |
sforshee | bjf, http://people.canonical.com/~sforshee/lp1046507/linux-3.6.0-030600rc7+wt201209271321/ | 16:17 |
bjf | sforshee: pulling ... | 16:18 |
rtg | sforshee, do you think its a buffer bloat issue ? | 16:18 |
sforshee | rtg, I think that the bufferbloat guys definitely wouldn't like the amount of buffering brcmsmac does, but I don't think the problems we're seeing are caused by bufferbloat | 16:19 |
sforshee | rtg, currently brcmsmac can queue 200+ packets internally | 16:20 |
rtg | sforshee, oh, thats _way_ too many | 16:21 |
sforshee | rtg, I think the timeouts during flush are just because it can't send all of those in the time it's allowing flush to complete | 16:21 |
rtg | I think it shouldn't have more then 3-5 queued | 16:21 |
rtg | what is the TX rate steps down to 1MB ? Thats an enormous amount of time.... | 16:22 |
rtg | what if* | 16:22 |
sforshee | rtg, with the way the broadcom chips appear do AMPDU I think it has to queue at least one AMPDU's worth | 16:22 |
sforshee | which is 16 | 16:22 |
rtg | how does it behave if you reduce the queue depth and disable aggregation ? | 16:22 |
sforshee | I'm planning to test without aggregation but haven't gotten around to it yet. It's a different code path through the driver though. | 16:24 |
rtg | why does it flush ? | 16:24 |
sforshee | The AMPDU code path explicitly tries to wait until enough packets are queued for an AMPDU | 16:24 |
sforshee | most of the flushes I see are before switching channels for background scanning | 16:25 |
rtg | ah | 16:25 |
rtg | I think waiting is a bad idea. if there is sufficient congestion, then packets will tend to queue up naturally. | 16:26 |
sforshee | The broadcom chips require extra headers on them when they are handed off to the hardware, and the first and last AMPDU packets must be marked as such in this header. | 16:29 |
sforshee | So it appears brcmsmac needs to know how many AMPDU packets it's handing off at the time it moves them to the DMA ring. | 16:29 |
rtg | sforshee, maybe you could try that, e.g., only aggregate the packets that are in the queue when the TX is ready to run rather then waiting. | 16:30 |
sforshee | What the driver does is waits as long as there's a tx in progress. Once all pending tx finishes it starts moving packets to the DMA ring regardless of how many are queued up. | 16:31 |
sforshee | It might make more sense if it just moved packets any time there was room in the DMA ring though | 16:31 |
rtg | sforshee, how many does the DMA ring hold ? is that teh 200 number ? | 16:31 |
sforshee | no, I don't recall the exact number but it is much smaller | 16:31 |
sforshee | I think the 200 number had something to do with their expectations when operating as an AP, but brcmsmac doesn't support AP mode right now anyway iirc | 16:32 |
* ppisati -> EOW | 16:42 | |
* henrix -> EOD | 17:03 | |
=== henrix is now known as henrix_ | ||
rtg | jsalisbury, re: bug #1041883, Too much noise. TLDR. Did you narrow it down to CONFIG_X86_X32 ? | 17:28 |
ubot2 | Launchpad bug 1041883 in linux "Recent patch to asus-wmi module makes system unbootable" [High,In progress] https://launchpad.net/bugs/1041883 | 17:28 |
jsalisbury | rtg, it looks like it. Comment #97 is the original bug reporter. He states that he no longer sees the panic, which was the orignial problem. | 17:32 |
jsalisbury | rtg, I asked some of the other commenters to confirm, but who knows the kind of response we will get, based on the other comments | 17:33 |
rtg | jsalisbury, ok, I think I'll go ahead and disable it, especially after chatting with cking and apw | 17:34 |
jsalisbury | rtg, cool | 17:34 |
rtg | jsalisbury, hmm, just _how_ do you disable that bugger ? | 17:47 |
jsalisbury | rtg, I changed it to "# CONFIG_X86_X32 is not set" in ~ubuntu-quantal/debian.master/config/config.common.ubuntu | 17:49 |
rtg | jsalisbury, did you check that it stuck after running updateconfigs ? | 17:51 |
rtg | git diff | 17:51 |
=== yofel_ is now known as yofel | ||
jsalisbury | rtg, It looks like it from git log of the SHA1 generated: | 17:53 |
jsalisbury | (quantal-amd64)jsalisbury@tangerine:~/bugs/lp1041883/ubuntu-quantal$ git show fc124a4fe9eda36d9509a4c5e15945ec45839d22 | 17:53 |
jsalisbury | commit fc124a4fe9eda36d9509a4c5e15945ec45839d22 | 17:53 |
jsalisbury | Author: Joseph Salisbury <joseph.salisbury@canonical.com> | 17:53 |
jsalisbury | Date: Wed Sep 26 17:36:06 2012 +0100 | 17:53 |
jsalisbury | Disabled config.common.ubuntu | 17:53 |
jsalisbury | 17:53 | |
jsalisbury | BugLink: http://bugs.launchpad.net/bugs/1041883 | 17:53 |
ubot2 | Launchpad bug 1041883 in linux "Recent patch to asus-wmi module makes system unbootable" [High,In progress] | 17:53 |
jsalisbury | 17:53 | |
jsalisbury | Signed-off-by: Joseph Salisbury <joseph.salisbury@canonical.com> | 17:53 |
jsalisbury | diff --git a/debian.master/config/config.common.ubuntu b/debian.master/config/config.common.ubuntu | 17:53 |
jsalisbury | index 37cb4b8..e852eff 100644 | 17:53 |
jsalisbury | --- a/debian.master/config/config.common.ubuntu | 17:53 |
jsalisbury | +++ b/debian.master/config/config.common.ubuntu | 17:53 |
jsalisbury | @@ -6406,7 +6406,7 @@ CONFIG_X86_USE_PPRO_CHECKSUM=y | 17:53 |
jsalisbury | CONFIG_X86_WANT_INTEL_MID=y | 17:53 |
jsalisbury | CONFIG_X86_WP_WORKS_OK=y | 17:53 |
jsalisbury | CONFIG_X86_X2APIC=y | 17:53 |
jsalisbury | -CONFIG_X86_X32=y | 17:53 |
jsalisbury | +# CONFIG_X86_X32 is not set | 17:53 |
jsalisbury | CONFIG_X86_XADD=y | 17:53 |
jsalisbury | CONFIG_XEN=y | 17:54 |
jsalisbury | CONFIG_XENFS=m | 17:54 |
jsalisbury | rtg, is there another way I can check while the kernel is booted? | 17:54 |
rtg | jsalisbury, /boot/config* | 17:54 |
rtg | /boot/config-`uname -r` | 17:54 |
jsalisbury | rtg, ok. I'll boot up the test kernel on a test machine and check. One minute | 17:55 |
rtg | jsalisbury, you don't have to boot it up if the kernel is still installed. | 17:56 |
* rtg -> lunch. bbiab | 17:56 | |
jsalisbury | rtg, I don't have it installed, but I will now | 17:57 |
jsalisbury_ | rtg, It looks like the change stuck: | 18:09 |
jsalisbury_ | jsalisbury@samsung:~$ cat /boot/config-3.5.0-15-generic | grep CONFIG_X86_X32 | 18:09 |
jsalisbury_ | # CONFIG_X86_X32 is not set | 18:09 |
infinity | Does anyone know the status of linux-armadaxp 3.5.0 in the PPA? Is that version scrubbed of non-free badness, or am I still waiting on that? | 18:29 |
rtg | infinity, hassle vanhoof. he's more likely to know, but I seem to remember that ike was gonna remove some stuff. | 18:32 |
infinity | rtg: Yeah, I just mailed vanhoof, ike, and friends. | 18:33 |
rtg | jsalisbury, I don't think CONFIG_X86_32 was _ever_ enabled. Reset to Ubuntu-3.5.0-11.11 and have a look at the config that is generated: '# CONFIG_X86_32 is not set' | 18:38 |
rtg | jsalisbury, doh, Sarvatt pointed out that its CONFIG_X86_X32, not CONFIG_X86_32 | 18:43 |
jsalisbury | rtg, hmm, yeah, if I reset to Ubuntu-3.5.0-11.11 it isn't set. The test kernel I built was a fresh clone of 3.5.0-15 | 18:43 |
rtg | jsalisbury, I lead you astray, try CONFIG_X86_X32 | 18:43 |
jsalisbury | rtg, and I reset to Ubuntu-3.5.0-11.11 in the tree that I modified. A fresh clone and reset to 3.5.0-11.11 does show it set | 18:45 |
rtg | jsalisbury, right, I was chasing the wrong symbol | 18:45 |
jsalisbury | rtg, ack | 18:46 |
* ogasawara lunch | 19:36 | |
profiler1982 | is it someone tr kernel 3.5 on 11.10 ubuntu? am have amd APU c-60 | 19:41 |
rtg | ogasawara, your shiny new kernel is accumulating patches at an alarming rate. | 20:05 |
rtg | ogasawara, have a look at https://blueprints.launchpad.net/ubuntu/+spec/powerpc-kernel-devel to consider for approval. | 20:23 |
rtg | and with that, I'm outta here. | 20:24 |
* rtg -> EOW | 20:24 | |
infinity | ogasawara: Want to whitelist me on canonical-kernel-team@lists? | 23:03 |
infinity | ogasawara: (Assuming you moderate it) | 23:03 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!