| === rsalveti_ is now known as rsalveti | ||
| === rsalveti_ is now known as rsalveti | ||
| === MaJian is now known as BruceMa_721 | ||
| ppisati | moin | 08:00 |
|---|---|---|
| === smb` is now known as smb | ||
| mlankhorst | anyone here understands mmaps and stuff in the kernel? :> | 08:26 |
| ohsix | in what respect | 08:28 |
| smb | there is a lot of stuff in the kernel... | 08:37 |
| mlankhorst | just mm in general, I'm trying to make forced drm unbinding work, and the part that's missing now is tearing down all mmaps | 08:38 |
| smb | mlankhorst, Hm, not deeply familiar with it but from the interface I'd say the using side is responsible to remember what mappings where done... | 08:43 |
| mlankhorst | ofc, but the fun is figuring out how to make the user side kill and free everything :P | 08:44 |
| ohsix | let them fault in a canary page | 08:44 |
| * ppisati tries again the install ubuntu on his nexus7... | 09:28 | |
| ppisati | brb | 09:58 |
| tseliot | apw: what's the point of having rcu_read_unlock_special defined in kernel/rcutiny_plugin.h and defined as a function prototype in include/linux/rcupdate.h? Is it private? | 10:12 |
| apw | tseliot, it may be a left over remnant of some older time, things in that code have moved a lot recently as two of the implementations have merged | 10:28 |
| tseliot | apw: hmm... I see | 10:29 |
| apw | tseliot, it sounds very much like it has moved inline, and not had its prototype removed to me | 10:38 |
| tseliot | apw: that would make sense | 10:38 |
| apw | and i don't think it would cause any issues this way, as essentially it becomes just a prototype, though you might expect to get a warning | 10:39 |
| tseliot | apw: yes, I'm getting a warning but what happens at runtime? | 10:43 |
| apw | tseliot, the right thing as it has only that one definition | 10:44 |
| apw | but for completeness paste the warning | 10:44 |
| tseliot | apw: WARNING: "rcu_read_unlock_special" [/var/lib/dkms/bcmwl/6.20.155.1+bdcom/build/wl.ko] undefined! | 10:46 |
| apw | hmmm, that is less what i expected as a warning | 10:47 |
| tseliot | apw: the whole log: http://paste.ubuntu.com/5607386/ | 10:47 |
| apw | tseliot, i would be supprised if you can load that module with that warning | 10:50 |
| tseliot | apw: I can't do it from my chroot since modprobe insists on looking for modules.dep.bin in /lib/modules/$(uname -r) | 10:56 |
| tseliot | I'll test it on a real system | 10:58 |
| apw | yeah | 10:59 |
| lilstevie | tseliot, there is always insmod if you haven't run depmod :p | 11:17 |
| tseliot | lilstevie: you don't say... ;) | 11:21 |
| lilstevie | :) | 11:21 |
| tseliot | apw: yes, it errors out: ERROR: could not insert 'wl': Unknown symbol in module, or unknown parameter (see dmesg) | 11:22 |
| tseliot | apw: wl: Unknown symbol rcu_read_unlock_special (err 0) (from dmesg) | 11:22 |
| apw | it actually seems like one of the combinations is just broken right now in that kernel then | 11:22 |
| apw | or is this with you 'transplanting' code | 11:22 |
| tseliot | apw: I simply copied rcu_lock and unlock locally and the unlock function calls rcu_read_unlock_special | 11:27 |
| ppisati | ogra_: still completely black screen with today's img | 11:28 |
| tseliot | apw: here's the patch http://paste.ubuntu.com/5607457/ | 11:29 |
| apw | tseliot, and against which version is it failing ... 3.8 right ? | 11:30 |
| tseliot | apw: yep 3.8 | 11:30 |
| apw | tseliot, so dispite these being in .h's they are not in .h's really | 11:35 |
| apw | tseliot, they are just being vile. they are actually just included into rcutiny.c and rcutree.c ... | 11:35 |
| apw | and those in that context are then real externals and not exported at all | 11:35 |
| apw | this needs to be fixed properly upstream, as it is unreasonable to export the interface without being able to use the locking. i have discussed an acceptable solutuion with the rcu maintainer | 11:36 |
| apw | but it won't help existing instances or you ... sigh | 11:36 |
| tseliot | apw: yes, that's what I was afraid of... | 11:36 |
| tseliot | apw: right, so I can't fix it unless I copy a lot of code from rcutiny_plugin.h (i.e. rcu_read_unlock_special and its dependencies) | 11:38 |
| apw | tseliot, which is getting rediculous. now for us i could spin some accessors, given my discussions i think might be reasonable upstream and we could try those out | 11:39 |
| apw | but that doesn't help other kernels in the short term | 11:39 |
| apw | tseliot, i assume it is jsut the lowlatency kernel which is in trouble right nwo ? | 11:40 |
| tseliot | apw: yes, it's just that flavour | 11:40 |
| apw | tseliot, and only in raring, so for this monent i think there is little panic at least | 11:41 |
| tseliot | apw: true, at least for now | 11:41 |
| apw | tseliot, zap me the bug number and i'll have a poke at what we might do to fix it kernel side | 11:42 |
| tseliot | apw: let me file one | 11:42 |
| tseliot | apw: I still need to write a proper description but here it is: bug #1154036 | 11:54 |
| ubot2 | Launchpad bug 1154036 in linux-lowlatency (Ubuntu) "rcu_unlock_special is undefined" [Undecided,New] https://launchpad.net/bugs/1154036 | 11:54 |
| snadge | i updated to 13.04 from 12.10 .. stock kernel wont boot.. but if i select the 3.5 kernel, it will | 12:02 |
| snadge | intel atom acer aspire one d260 | 12:02 |
| apw | snadge, "won't boot" what actually happens when you attempt to boot that kerenl | 12:05 |
| apw | snadge, as that is about the most common box around and one we have things which are very similar to in regular use on 13.04 | 12:06 |
| snadge | i was seeing kernel panic .. its not on the screen now.. i've installed 3.8.0-12-generic.. and its now booting | 12:26 |
| snadge | but theres an unrelated broadcom wireless driver problem | 12:27 |
| snadge | !bug 1110139 | 12:30 |
| ubot2 | Launchpad bug 1110139 in linux (Ubuntu) "Broadcom BCM4313 wireless card driver not working" [Medium,Confirmed] https://launchpad.net/bugs/1110139 | 12:30 |
| ppisati | xnox: btw, did you try my overlayfs kernel? | 12:39 |
| rtg_ | ppisati, did you take that over from apw for 3.9-rc2 ? | 12:41 |
| ppisati | rtg_: nope, nexus7 kernel - 3.1.X + overlayfs from precise | 12:41 |
| rtg_ | ack | 12:41 |
| xnox | ppisati: sorry, not yet, got tangled up in things. Are you pondering to upload? | 12:43 |
| * henrix -> lunch | 12:43 | |
| ppisati | xnox: nope until anyone give it a spin | 12:43 |
| ppisati | my nexs7 is busted, so can't do myself | 12:43 |
| xnox | /o\ | 12:43 |
| xnox | ok. will get to it, as soon as i can. mine was working finish last time i tried. | 12:44 |
| apw | rtg_, i am just playing with it at the moment in fact | 12:44 |
| rtg_ | apw, cool | 12:44 |
| apw | rtg_, ... and of course ... there was an overlayfs update over night ... pulling in now | 13:36 |
| rtg_ | apw, timing is everything ... | 13:37 |
| apw | i wish i had been even more lax yesterday | 13:37 |
| rtg_ | apw, sometimes it is righteous to be a slacker | 13:40 |
| * smb wished he could use that in the goals... | 13:42 | |
| === vmesons is now known as vmeson | ||
| mlankhorst | you don't say you're a slacker, you have to make it sound like you're putting work into avoiding work :) | 13:53 |
| rtg_ | mlankhorst, ok, I am actively pursuing slackerdom :) | 13:55 |
| mlankhorst | again it's how you bring it :) | 13:56 |
| ohsix | gotta try selling pap smears of famous people | 13:59 |
| apw | :q | 14:02 |
| mlankhorst | Bad way: "I want to slack off". Good way: "increase testability and reduce time needed to isolate problems". Note that in the good way you do some work, to prevent more work. :-) | 14:05 |
| ohsix | make it your job to make iteration times lower but spend all your time testing | 14:06 |
| === kentb-out is now known as kentb | ||
| jsalisbury | ## | 14:15 |
| jsalisbury | ## Kernel team meeting today @ 17:00 UTC | 14:15 |
| jsalisbury | ## | 14:15 |
| === ogra_` is now known as ogra_ | ||
| === kentb is now known as kentb-afk | ||
| chiluk | Hey everyone, the nvidia binary blob has decided to destroy itself all over my logs... should I open the bug against kernel, x.org, or something else? | 14:57 |
| rtg_ | chiluk, talk to tseliot | 14:59 |
| tseliot | chiluk: what do you mean? | 15:01 |
| chiluk | tseliot let me get a pastebin for you. | 15:01 |
| chiluk | tseliot check out http://paste.ubuntu.com/5607938/ | 15:04 |
| chiluk | that 20k lines of log is only the first hour of issue... | 15:06 |
| tseliot | chiluk: yes, please file a bug report | 15:06 |
| tseliot | chiluk: against whatever nvidia driver you're using | 15:07 |
| chiluk | I'm on latest proposed precise kernel. | 15:07 |
| chiluk | and let me check nvidia blob. | 15:07 |
| chiluk | 310.14 | 15:07 |
| chiluk | just started happenning, and it only seems to happen in the early morning after the machine has been idle a whiel. | 15:09 |
| chiluk | I hit the same issue yesterday, but passed it off to being an power issue from the storm I had. | 15:10 |
| slangasek | jsalisbury: hey, so I just caught bug #1152736 in action here again, this time on 3.5.0-9 | 15:22 |
| ubot2 | Launchpad bug 1152736 in linux (Ubuntu) "system swapping itself to death in raring for no good reason" [Critical,Confirmed] https://launchpad.net/bugs/1152736 | 15:22 |
| slangasek | jsalisbury: this one wasn't as dire - it sorted itself out in mere seconds, instead of eating my laptop alive for a half hour - but kswap0 was definitely pegging the CPU | 15:23 |
| jsalisbury | slangasek, thanks for the update. So it doesn't seem to be a regression. Can you also run on additional test with the latest 3.9-rc2 kernel? Just to confirm it is in mainline as well. | 15:24 |
| slangasek | jsalisbury: sure. Remind me where to find the package? | 15:24 |
| jsalisbury | slangasek, http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.9-rc2-raring/ | 15:24 |
| jsalisbury | slangasek, I can post it to the bug as well. | 15:25 |
| roadmr | hey folks! I think I found a regression for the bcmwl-kernel-source driver that's in precise-updates (6.20.155.1); on a Dell 15R with BCM4312 it fails to connect to any networks. If I downgrade to the 5.100.82.38 version it works fine | 15:30 |
| chiluk | tseliot, pad.lv/1154153 has been openned | 15:33 |
| tseliot | chiluk: thanks | 15:34 |
| chiluk | it really might be a power issue, as I just noticed NVRM: GPU at 0000:01:00.0 has fallen off the bus. | 15:34 |
| * ppisati needs to reboot, brb | 15:35 | |
| chiluk | tselliot can you think of any other reason it would "fall off the bus?" | 15:39 |
| === kentb-afk is now known as kentb | ||
| brendand | rtg_, we're doing some SRU testing here and encountering what seems to be an incompatibility issue between the new broadcom driver and the 3.2 kernel | 15:55 |
| brendand | rtg_, 6.20.155.1+bdcom-0ubuntu0.0.1 doesn't work with 3.2 | 15:56 |
| apw | retired/CVE-2012-2372: break-fix: 639b321b4d8f4e412bfbb2a4a19bfebc1e68ace4 local-2012-2372 | 15:56 |
| brendand | rtg_, it works with 3.5 | 15:56 |
| rtg_ | brendand, its possible that it doesn't work with a retrograde kernel. tseliot is the owner. | 15:57 |
| tseliot | brendand: what's the error? | 15:58 |
| brendand | tseliot, roadmr has more details, i'll hand you over | 16:00 |
| tseliot | ok | 16:00 |
| brendand | tseliot, he may take a bit to respond - otp apparently | 16:01 |
| tseliot | brendand: ok, no hurry | 16:01 |
| apw | rtg_, ok i have pushed an update to overlayfs to the unstable-3.9 branch, i have also tested aufs and overlayfs (minorly) and they seem to at least let me do basic operations | 16:17 |
| rtg_ | apw, coolk | 16:17 |
| rtg_ | cool* | 16:17 |
| roadmr | tseliot: ok, I'm here now, sorry! this system does see the wireless networks but is unable to connect to them with the 6.x driver. With the 5.100.x driver, it can connect to b/g networks only, n networks are visible but can't be connected to | 16:33 |
| tseliot | roadmr: it sounds like a regression in the driver | 16:34 |
| roadmr | tseliot: yep! I'm currently testing the proposed kernel, but the 6.x driver also fails with older kernels (i tested 3.2.0-29) | 16:35 |
| rtg_ | apw, https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1147678/comments/6 ? | 16:36 |
| ubot2 | Launchpad bug 1147678 in linux "kernel 3.5.0-26.40-generic oops immediately when doing schroot w/overlayfs" [Medium,Fix committed] | 16:36 |
| tseliot | roadmr: yes, I don't think it's a kernel issue. 6.x can be buggy on some chipsets | 16:37 |
| roadmr | tseliot: yep. What to do? file a bug? we're also concerned that this will hit people who apply the driver updates and will suddenly find their wireless not working :/ | 16:40 |
| tseliot | roadmr: yes please, we should probably revert the upload | 16:41 |
| roadmr | tseliot: ok, I'll start by filing a bug | 16:41 |
| tseliot | roadmr: thanks | 16:41 |
| cking | kamal, the photo does not show any < > brackets around the oopsing opcode, which is strange, also I'd like to see the objdump with the opcodes so we can correlate that with the one in the photo | 16:42 |
| kamal | where/what is the "oopsing opcode" that you expect to see <> around? | 16:43 |
| kamal | cking: ^^ | 16:44 |
| sforshee | kamal, that's the stuff in the line prefixed with "Code:" | 16:44 |
| sforshee | normally the kernel will put angle brackets around which part of that output corresponds to the oopsing opcodes | 16:44 |
| * cking wonders if it fell off screen on the right | 16:45 | |
| kamal | sforshee, cking: ok, well the lack of <> matches up with my observation that the RIP doesn't line up with the instruction boundaries I guess | 16:45 |
| cking | maybe so, yes | 16:45 |
| kamal | cking, sforshee: do you happen to know the objdump flag to make it include the opcodes? | 16:47 |
| cking | kamal, I normally variations of objdump -d | 16:48 |
| kamal | --show-raw-insn maybe -- I'll try it | 16:48 |
| jsalisbury | ## | 16:55 |
| jsalisbury | ## Kernel team meeting in 5 minutes | 16:55 |
| jsalisbury | ## | 16:55 |
| jsalisbury | ## | 17:00 |
| jsalisbury | ## Meeting starting now | 17:00 |
| jsalisbury | ## | 17:00 |
| * smb is off | 17:06 | |
| === yofel_ is now known as yofel | ||
| kamal | cking, sforshee: ok I have added a dump with opcodes (objdump --show-raw-insn) to bug 1136699 ... aaaaand I'm still confused. just fyi! | 17:09 |
| ubot2 | Launchpad bug 1136699 in linux (Ubuntu) "xserver crash in drm_vblank_count_and_time" [High,Triaged] https://launchpad.net/bugs/1136699 | 17:09 |
| === jsalisbury changed the topic of #ubuntu-kernel to: Home: https://wiki.ubuntu.com/Kernel/ || Ubuntu Kernel Team Meeting - Tues March 19th, 2013 - 17:00 UTC || If you have a question just ask, and do wait around for an answer! | ||
| rtg_ | apw, is this the first request for inclusion ? '[PATCH 00/13] overlay filesystem: request for inclusion (v16)' | 17:12 |
| apw | rtg_, i got it from his git repo ... google wasn't showing that one when i searched | 17:12 |
| rtg_ | apw, it was 20 minutes ago | 17:13 |
| apw | oh missread, no i think this is the third time he has asked for inclusion | 17:13 |
| rtg_ | apw, has Al formally responded in the past ? | 17:14 |
| apw | "wait wait wait until atomic_open is in" | 17:14 |
| apw | which it now is | 17:14 |
| rtg_ | well then, maybe it'll happen this time | 17:14 |
| apw | doubtful... he will have some other objection i am sure | 17:16 |
| apw | but ... its worth a shot | 17:16 |
| * rtg_ -> lunch | 17:18 | |
| joshhunt | i have a question about which git tree i should be pulling from to stay current with the latest quantal kernel updates. when looking at the quantal tree I do not see a tag for 3.5.0-25.39, however looking at my precise tree I do. where tree should i be using? | 17:21 |
| bjf | joshhunt, one sec checking on that tag, however, you should be using the quantal tree | 17:22 |
| joshhunt | bjf, thanks | 17:23 |
| bjf | Ubuntu-3.5.0-25.39 | 17:24 |
| bjf | joshhunt, i see the tag in the tree | 17:24 |
| joshhunt | hmm | 17:24 |
| joshhunt | bjf, from git://kernel.ubuntu.com/ubuntu/ubuntu-quantal.git ? | 17:25 |
| apw | joshhunt, git fetch --tags origin ... perhaps | 17:25 |
| bjf | joshhunt, yes, i'm in that exact tree right now | 17:25 |
| joshhunt | bjf, ok running git tags now i have: Ubuntu-3.5.0-25.38 Ubuntu-3.5.0-26.40 Ubuntu-3.5.0-26.41 | 17:25 |
| joshhunt | apw, i will try that | 17:25 |
| joshhunt | apw, ok that seems to have added the tag | 17:26 |
| apw | joshhunt, i have that tag in my tree | 17:26 |
| bjf | joshhunt, and i just did a new clone and it's there | 17:26 |
| * bjf -> brb | 17:26 | |
| apw | bjf, if that tag was somehow not in the main branch you can miss it if you don't pull every day ... though they should be linear in that tree | 17:26 |
| joshhunt | i cloned this a while back and have been doing pulls. another person in my group has the same problem. however, the fetch tags thing worked | 17:26 |
| joshhunt | apw, do i need to do 'fetch tags' in addition to periodic pulls? | 17:28 |
| apw | ok looking at the history there is some inconstancy in the order, someone has rebased it | 17:28 |
| apw | you shouldn't in a tree like this, as it should not be being rebased | 17:28 |
| apw | bjf, somehow we rebased .39 off the mainline when making .40 ... | 17:28 |
| joshhunt | apw, ok cool thanks | 17:30 |
| apw | bjf, joshhunt, it may have been embargoed at the time, and that might explain the shenanegins | 17:30 |
| joshhunt | apw, what's that mean? | 17:30 |
| apw | that .39 carried a very high profile security fix, which may have had unusual handling which may account for the discontinuity | 17:30 |
| apw | which means when you pull 'days apart' you don't see .39 as something which is reelvant and you don't get the tag | 17:31 |
| apw | without --tags | 17:31 |
| joshhunt | apw, gotcha. thanks for your help. | 17:32 |
| joshhunt | apw, how could you tell that it had been rebased? just so i can better understand when something like this happens in the future. | 17:32 |
| apw | joshhunt, it is most obvious if you use gitk. ... if you use: gitk origin/master-next Ubuntu-3.5.0-25.39 | 17:33 |
| apw | and scroll down till you see .39 you will see that it is off the main branch | 17:33 |
| joshhunt | apw, ahh i should install gitk... :) | 17:33 |
| apw | gitk origin/master-next Ubuntu-3.5.0-25.39 ^v3.5 | 17:34 |
| apw | joshhunt, ^^ use that to avoid it making your machine swap | 17:34 |
| joshhunt | apw, thanks | 17:38 |
| joshhunt | apw, really appreciate the help | 17:38 |
| bjf | back | 17:44 |
| bjf | apw, odd, i remember struggling with it a bit but not rebasing it | 17:46 |
| bjf | apw, we rebase master-next onto it | 17:46 |
| === maxb_ is now known as maxb | ||
| xnox | ppisati: testing overlayfs kernel. seems to work ok, but I am getting some failures with direct write from upstart test-suite. | 18:15 |
| xnox | will investigate further. | 18:15 |
| apw | bjf, somehow in that rebase the three commits reposenting it also got rebased | 18:17 |
| apw | bjf, you can see it in the gitk graph: gitk origin/master-next Ubuntu-3.5.0-25.39 ^v3.5 | 18:19 |
| bjf | apw, it was obviously me | 18:20 |
| apw | bjf, *shrug* it is in the past, and no great shakes as the intent is definatly not lost in the mainline | 18:22 |
| bjf | apw, agreed, though i try to make sure master never gets rebased like that | 18:22 |
| bjf | i'll buy everyone a beer at the next vUDS | 18:22 |
| apw | yeah it is clear you were trying to achieve the opposite, to stack the master-next on top, quite what is different between the two i am unsure | 18:23 |
| apw | nope they are identicle, so it must be metadata somehow | 18:24 |
| apw | bjf, the onlyh difference is the dates, and that is odd as you would expect a rebase even asked to be done from the wrong spot, to note the first few were the same and just leave them be. very odd | 18:25 |
| bjf | apw, what can i say, it's a gift | 18:26 |
| === bjf is now known as bjf[afk] | ||
| ppisati | xnox: nice, let me what's the outcome | 18:52 |
| * ppisati would love to have a working nexus so he could do by himself... :( | 18:53 | |
| * infinity grumbles about someone removing sbsigntool from the kernel PPA. | 18:53 | |
| apw | infinity, i thought that sbsigntool was now in precise (not that i would have removed it) | 18:53 |
| infinity | apw: It's in precise-updates, the PPA builds without pockets, because of security. | 18:54 |
| apw | infinity, then it should be in -security else the pockets as they are in the release won't build the kernel | 18:54 |
| apw | infinity, though to get it in the PPA i simply pocket copied it in from -updates in the first place 'with binaries' | 18:54 |
| apw | infinity, so it is easy to fix | 18:55 |
| infinity | apw: This is a fair point, actually. | 18:55 |
| * ppisati bails out | 18:55 | |
| infinity | apw: Hold off on copying it to the PPA, I *should* copy it to -security so the kernels aren't technically FTBFS. | 18:55 |
| infinity | apw: And your PPA does build against security, so that will fix it. | 18:56 |
| infinity | apw: Thanks for talking sense into me. Should be fixed shortly. | 18:57 |
| xnox | ppisati: red herring, fails in the same way on amd64, I guess the test-suite is too strict again. So from my point of view it's good to go =) | 19:00 |
| * xnox should email that. | 19:00 | |
| apw | infinity, np | 19:03 |
| * rtg_ -> EOD | 20:02 | |
| zequence | I realized it might be smarter to put the kernels into a ~ubuntustudio-kernel owned PPA, so just created one, and uploaded to ppa:ubuntustudio-kernel/linux-lowlatency-sru | 20:16 |
| zequence | apw: ^ | 20:18 |
| zequence | apw: I could do the metas too from now on | 20:27 |
| === bjf[afk] is now known as bjf | ||
| bjf | back | 20:42 |
| bjf | infinity, if someone puts something in *my* ppa without letting me know, i just might be tempted to clean it up some time | 20:44 |
| bjf | just sayin | 20:44 |
| bjf | infinity, however, i don't remember deleting it cause i didn't know what it was for | 20:46 |
| infinity | bjf: S'all good, it doesn't need to be there anymore anyway. | 20:48 |
| infinity | bjf: But I suspect it went away when henrix was doing his spring cleaning. | 20:48 |
| bjf | infinity, it could have been me, i just don't remember doing it | 20:48 |
| henrix | bjf: so, what's missing? | 20:49 |
| bjf | henrix, heh, the sbsigntool got deleted, but it is no longer an issue | 20:49 |
| infinity | Anyhow, it highlighted a bug anyway, so all good. | 20:50 |
| * henrix reads backlog | 20:50 | |
| henrix | i can't remember deleting anything, but i have fat fingers so... | 20:51 |
| bryce | ogasawara, sforshee would one of you mind rolling a kernel with the proposed patch for bug #1153302 if you have a chance? | 22:30 |
| ubot2 | Launchpad bug 1153302 in xserver-xorg-video-intel (Ubuntu) "[g33] Non-locking GPU lockup EIR: 0x00000010 PGTBL_ER: 0x00000002 IPEHR: 0x54f00006" [Undecided,New] https://launchpad.net/bugs/1153302 | 22:30 |
| bjf | bryce, i'll spin you one | 22:33 |
| bjf | bryce, amd64, i386 or both? | 22:34 |
| bryce | bjf, thanks. amd64. | 22:34 |
| bjf | bryce, http://people.canonical.com/~bradf/lp1153302/ | 22:56 |
| bryce | bjf, thanks | 22:58 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!