benji | I think I have termbeamer installable from a PPA now. Testing would be appreciated. | 11:23 |
---|---|---|
bac | benji: cool | 11:43 |
benji | bac: sudo apt-add-repository ppa:benji/termbeamer; sudo apt-get install termbeamer if you want to try it out | 11:44 |
bac | ah, you forgot sudo apt-get update! :) | 11:44 |
gary_poster | hey bac, did you get what you needed from that ec2 instance? I have some worker lists from it if you haven't that I could save. I want to restart the instance and see if the changes to my charm work | 11:45 |
bac | gary_poster: please save. i need more data to chase a theory | 11:45 |
gary_poster | cool | 11:45 |
frankban | gary_poster: http://ec2-23-21-26-159.compute-1.amazonaws.com:8010/builders/lucid_lp/builds/1/steps/shell_8/logs/summary the first failure seems a new one, the second should be 996729 | 11:46 |
frankban | gary_poster: and thanks for the review | 11:46 |
frankban | s | 11:46 |
gary_poster | frankban, the first one is also known, and Brad has a simple egg I was supposed to test to address it (simply increasing a timeout) | 11:47 |
gary_poster | That said, frankban, I hadn't filed a bug because it was tied to some previous work Brad had done | 11:47 |
gary_poster | that was probably a mistake | 11:47 |
gary_poster | so if you file a bug, Brad or I can take it | 11:47 |
frankban | gary_poster: oh, ok | 11:48 |
benji | bac: I did forget the update. Once it's installed try "tb --help" | 11:48 |
frankban | gary_poster: do you need the worker list? | 11:48 |
gary_poster | reviews: you're welcome of course--thanks for that work | 11:48 |
gary_poster | frankban, nah thanks | 11:48 |
frankban | gary_poster: https://bugs.launchpad.net/launchpad/+bug/1002820 | 11:50 |
_mup_ | Bug #1002820: lp.testing.layers.RabbitMQLayer:setUp times out rarely/intermittently in parallel tests <paralleltest> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1002820 > | 11:50 |
frankban | adding it to the board | 11:51 |
gary_poster | perfect thanks frankban | 11:51 |
* benji rushes to make coffee before the call. | 11:51 | |
bac | benji: what happens if i have two-level auth turned on for my gmail account? | 11:54 |
bac | benji: and i do. :) | 11:54 |
benji | bac: me too, you have to create an application password: support.google.com/accounts/bin/answer.py?answer=185833 | 11:56 |
bac | great | 11:57 |
bac | benji: hey that appears to work. i did get a traceback: http://paste.ubuntu.com/1000684/ | 11:59 |
benji | bac: that's caused by not having my fixed python-vte package, which I made a PPA dependency but I guess that doesn't work the way I think it works; add the benji/vte-fixes ppa and do the update/upgrade dance and you should be good to go | 12:01 |
benji | the best part of being home is that I can reliably make a decent cup of coffee | 12:02 |
gary_poster | benji, which home is this now? :-) | 12:02 |
benji | gary_poster: heh, VA | 12:02 |
gary_poster | how/when did you get there? | 12:03 |
gary_poster | I guess when is the interesting part, assuming a teleporter wasn't involved | 12:03 |
bac | benji: you don't travel with your own stash of coffee? | 12:03 |
benji | how: Dodge Caravan, when: Sunday at about 8:30 pm local time | 12:04 |
gary_poster | ah! I thought you were still in TN yesterday | 12:04 |
bac | benji: we now have a clean launch. just like SpaceX. | 12:04 |
benji | bac: heh, maybe I should from now on; although my dad has one of those keurig one-cup-at-a-time things, they're pretty good | 12:05 |
gary_poster | frankban, bac, you are the designated default maintainers for our buildbot-slave and buildbot-master charms, respectively, because you had the most commits. Would you object if I try to see if we can make yellow the maintainer for both of those? | 12:05 |
bac | nope | 12:06 |
benji | bac: maybe after the call we can do a tb test run | 12:06 |
bac | i'll prepare the iron lung | 12:07 |
bac | or was that polio? | 12:07 |
frankban | gary_poster: no objections | 12:07 |
bac | hey you've got a name for your follow on project | 12:07 |
benji | lol | 12:08 |
gary_poster | lol | 12:08 |
gary_poster | thanks bac & frankban | 12:08 |
gary_poster | bac benji frankban gmb call in 2 | 12:08 |
gary_poster | I'll get us a room, as it were | 12:08 |
gmb | k | 12:08 |
benji | (I think it was polio. People with TB just died.) | 12:09 |
gary_poster | https://talkgadget.google.com/hangouts/_/ca56b39f7ec51a3a743513e476a5512a15a0253e | 12:10 |
gary_poster | gmb yoohoo | 12:11 |
gmb | gary_poster, Firefox in quitting hilarity. | 12:11 |
gmb | Bear with me. | 12:11 |
gary_poster | :-P | 12:12 |
gary_poster | cool | 12:12 |
benji | bac: want to try termbeamer? | 12:26 |
bac | oh sure | 12:26 |
bac | i am brad.crittenden@gmail | 12:26 |
bac | benji: what is your xmpp? | 12:27 |
benji | bac: you've been invited to chat (approving the other person is a one-time first step that we have to do) | 12:27 |
benji | benji.york@gmail | 12:27 |
bac | so i just did --xmpp-receive | 12:28 |
bac | and nothing is happening | 12:28 |
bac | tb --xmpp-user=brad.crittenden@gmail.com --xmpp-password=XXX --xmpp-receive | 12:29 |
bac | is that right benji? | 12:29 |
benji | bac: did you go to gmail and approve my request to chat? | 12:29 |
benji | bac: yep, it looks good | 12:29 |
gmb | Wups. Better remove that sweary fake test from the LP tree before I commit... | 12:29 |
bac | benji: go where? should i receive an email? | 12:30 |
benji | bac: the gmail UI should prompt you | 12:30 |
bac | er, cc.booty2? | 12:31 |
bac | ok, i authed that. still see nothing | 12:31 |
* bac -> tea | 12:32 | |
benji | bac: I invited the wrong you to chat, look for another auth message | 12:33 |
gary_poster | bac, workers 0-7 of r15280 on parallel buildbot: http://pastebin.ubuntu.com/1000718/ http://pastebin.ubuntu.com/1000719/ http://pastebin.ubuntu.com/1000720/ http://pastebin.ubuntu.com/1000722/ http://pastebin.ubuntu.com/1000724/ http://pastebin.ubuntu.com/1000727/ http://pastebin.ubuntu.com/1000728/ http://pastebin.ubuntu.com/1000729/ | 12:34 |
gary_poster | argh | 12:35 |
gary_poster | we don't have 15280 on lpbuildbot >:-( | 12:35 |
* gmb lunches | 12:35 | |
gary_poster | I'll make a new set | 12:38 |
bac | thanks gary_poster | 12:40 |
gary_poster | welcome | 12:40 |
bac | benji: done | 12:40 |
bac | benji: i can see you | 12:40 |
benji | yay! | 12:40 |
benji | bac: watch this... | 12:41 |
bac | ok | 12:41 |
bac | what/ | 12:41 |
benji | did your window get smaller when I resied mine? | 12:41 |
bac | no | 12:41 |
benji | darn | 12:41 |
bac | oh, i made my window big | 12:41 |
benji | bac: oh! I forgot to get you to use the --gui-client switch | 12:42 |
benji | restart with that | 12:42 |
bac | done | 12:42 |
bac | yes, smallerized | 12:43 |
benji | :) | 12:43 |
benji | cool | 12:43 |
bac | yes, very neat | 12:43 |
bac | can i do anything or just watch? | 12:43 |
gary_poster | watch the blinken lights | 12:43 |
benji | bac: right now just watch, the very next thing I want to add is client to server communication | 12:43 |
bac | that will be cooler but this is very neat indeed | 12:44 |
benji | bac: did you see the flashing crosshair? | 12:44 |
bac | yes | 12:44 |
benji | that's the first client-to-server thing I want to add, so we can both point things out | 12:45 |
benji | bac: do these colors look sane? | 12:46 |
bac | yes. | 12:46 |
bac | the italics and underline2 didn't do anything | 12:47 |
benji | bac: yeah, most terminals don't support that (including the terminal widget I'm using) | 12:47 |
bac | benji: also note i'm running this in an ssh session from X on OS X to precise and it still works. | 12:48 |
benji | it does support strikethough, but I don't bother sending that data, strikethrough seems rare | 12:48 |
benji | very cool | 12:48 |
benji | bac: do you want to try being the sender? | 12:48 |
bac | ok, sure | 12:49 |
bac | benji: do i use --gui-client? | 12:49 |
benji | bac: nope | 12:49 |
benji | the server is gui-only so there's no switch for that | 12:49 |
benji | bac: I see you | 12:50 |
benji | bac: heh; try your console editor of choice | 12:50 |
benji | you and your white background | 12:50 |
bac | how do i point? | 12:51 |
benji | control-click | 12:51 |
benji | bac: looks good | 12:51 |
bac | oh, you can see the crosshair? | 12:52 |
benji | yep | 12:52 |
bac | i don't see anything when i control click | 12:52 |
benji | I need to make the server show it too for feedback | 12:52 |
* benji adds to the TODO | 12:52 | |
bac | well that's a smashing success | 12:53 |
benji | cool | 12:54 |
gary_poster | bac db-devel r11611. parallel tests report 16718 tests, lpbuildbot reports 16761 tests (https://lpbuildbot.canonical.com/builders/lucid_db_lp/builds/1979). parallel buildbot test worker lists follow, 0-7: http://pastebin.ubuntu.com/1000745/ http://pastebin.ubuntu.com/1000746/ http://pastebin.ubuntu.com/1000747/ http://pastebin.ubuntu.com/1000749/ http://pastebin.ubuntu.com/1000750/ http://pastebin.ubuntu.com/10 | 12:55 |
gary_poster | 00753/ http://pastebin.ubuntu.com/1000756/ http://pastebin.ubuntu.com/1000757/ . If I add the line numbers from those eight lists, I get 16718, so it seems to represent what the parallel test buildbot is counting. Also frankban, those test lists are perfectly round-robin in counts (every list is either 2089 or 2090) which is good...except odd because it was not a first test run. :-/ Anyway, bac, that looks lik | 12:55 |
gary_poster | e something you can use. | 12:55 |
gary_poster | I'm going to shut down the instance and restart now, | 12:55 |
gary_poster | . | 12:55 |
bac | gary_poster: great. that blows my theory, though. had parallel shown more tests then i had a possible explanation | 12:57 |
gary_poster | :-/ k | 13:03 |
gary_poster | oh, bac, did you see my idea on presenting about parallel testing to that pythoncarolina thing in CH? Not sure if it fits your timeline | 13:05 |
bac | gary_poster: no | 13:15 |
* gary_poster looks for it | 13:15 | |
bac | gary_poster: found it | 13:15 |
bac | that sounds interesting | 13:16 |
gary_poster | cool | 13:16 |
bac | gary_poster: if i'm not around you can just wing it. :) | 13:16 |
gary_poster | :-) | 13:16 |
bac | gary_poster: so i ran into a guy at RDU wearing an Open Stack t-shirt. he said "yeah, i work for this company called 'Canonical'"... odd | 13:17 |
gary_poster | bac, weird. https://directory.canonical.com/list/state/NC/ ... bfox or bladernr? | 13:21 |
bac | gary_poster: no, i know those guys. some new guy named adam stokes who doesn't have his address in the directory correctly | 13:22 |
gary_poster | huh | 13:22 |
gary_poster | cool, I guess | 13:22 |
bac | and he pronounced it caneenical | 13:22 |
benji | heh | 13:25 |
benji | that's the little-used Knights Who Say Caneeenical pronunciation | 13:26 |
frankban | wow, DEBUG = True in canonicaladmin: https://directory.canonical.com/NOOOO | 13:27 |
benji | frankban: that is hilarious | 13:27 |
frankban | gary_poster: I've moved the card to done-done, AFAICT testrepository is doing the right thing | 13:27 |
bac | frankban: well, at least that is only on the directory not canonicaladmin | 13:28 |
frankban | bac: yes | 13:28 |
benji | https://directory.canonical.com/orgchart/ takes a while to respond :) | 13:29 |
benji | frankban: we should report that to whomever is responsible (I don't know who that is) | 13:33 |
frankban | benji: indeed (me either) | 13:36 |
bac | benji, frankban: probably stuartm | 13:43 |
gary_poster | frankban, awesome thanks | 14:16 |
bac | gary_poster: we have some tests (mainly doctests i think) that get run multiple times. lpbuildbot is reporting them individually where our parallel test output only lists them once. summary: http://paste.ubuntu.com/1000909/ | 14:20 |
gary_poster | bac, should we be paranoid that these tests are not actually being run multiple times as they should be? | 14:20 |
gary_poster | concerned | 14:21 |
bac | gary_poster: i'm looking at the data collection code now. i suspect it is a hash indexed by test name which throws away multiples | 14:21 |
gary_poster | bac, that would make sense, except that we have seen some duplicates, like that api test from, um, some bug we had | 14:22 |
gary_poster | benji, I see you are working on the rabbitmq bug | 14:22 |
gary_poster | did you get the egg from bac already? | 14:23 |
gary_poster | back in a sec | 14:23 |
benji | gary_poster: I feel like I should know what you're talking about because it was discussed this morning. | 14:24 |
bac | benji: last week i produced a rabbit-fixture egg that set absurdly high timeouts | 14:24 |
benji | and those timeouts were still being reached? | 14:24 |
bac | benji: no, gary never used it | 14:25 |
bac | benji: and now i can't find it. :( | 14:25 |
bac | benji: found | 14:26 |
benji | what is the intended use of that egg? | 14:27 |
bac | benji: to run a novel branch of launchpad that uses that egg under parallel testing to see if the timeouts go away | 14:28 |
gary_poster | bac, benji, this is intermittent. | 14:28 |
benji | bac: are the timeouts frequent enough that doing that would tell us anything? | 14:29 |
gary_poster | therefore, I suggest that we may need to actually land use of that egg | 14:29 |
bac | benji: it is rabbitfixture 0.3.3-bac-exp and is now in download-cache/dist | 14:29 |
gary_poster | and see if it goes away. | 14:29 |
gary_poster | We've seen it twice in about 30 or 40 runs, I'd guess | 14:29 |
benji | Meta: I think there is room for process improvment here. I feel like these things should have been written down in the bug. | 14:29 |
benji | yeah, either reproducing it or dramatically upping the timeouts seem like the only two rational approaches | 14:30 |
benji | bac: did you try to reproduce the timeout? | 14:31 |
gary_poster | benji, process improvement: Gary should have made a bug. It was kind of a follow-on to existing work, so I didn't. Lesson: always make a bug. It is not written in the bug because it is a placeholder that I asked Francesco to make. Ideally I would have at least added a "talk to me or Brad" to the bug. | 14:33 |
gary_poster | We did discuss it in this morning's meeting, but again, Gary should have written it down. Lesson: always write stuff down in the proper place. | 14:34 |
gary_poster | I know those lessons, just didn't do 'em. | 14:34 |
gary_poster | Lesson: kick Gary. | 14:34 |
gary_poster | benji, I am pretty sure bac did not try to repro this timeout. He had reproed an earlier one to determine that there was a bug in the egg. | 14:35 |
gary_poster | Someone had already fixed the code | 14:35 |
bac | and if we are to make an egg that is referenced by launchpad in production then it needs to be a full release of rabbitfixture and the code pushed | 14:35 |
gary_poster | but not released it | 14:35 |
bac | gary_poster: yes, but that 'fix' was not sufficient | 14:35 |
bac | benji: no, i have not tried to reproduce with the new experimental egg | 14:35 |
gary_poster | simply because the timeout was not enough for what we need, as we believe | 14:36 |
bac | yes. it would be the proper fix for a non-parallel environment | 14:36 |
benji | so I should make a "full release" of rabbitfixture and add that to LP, right? | 14:36 |
gary_poster | Yeah, that's my suggestion. I'm not sure what Brad means by a full release | 14:37 |
gary_poster | bac, do you think we should merge this? I was thinking we ought to have a fork, and maybe file a bug to remove the fork and merge to trunk once we get 40 or so clean runs | 14:37 |
bac | gary, i mean 0.3.4 instead of 0.3.3.-exp | 14:37 |
gary_poster | gotcha, then I understood... | 14:38 |
bac | gary_poster: a fork is fine | 14:38 |
gary_poster | ok | 14:38 |
gary_poster | benji ^^ | 14:38 |
bac | gary_poster: just don't do it via egg drops | 14:38 |
gary_poster | gotcha bac, agreed | 14:38 |
bac | per your previous edict | 14:38 |
gary_poster | i'm all about my edicts | 14:38 |
bac | i prefer hot'n'sour over egg drop | 14:38 |
benji | revision: I will fork rabbitfixture, make an egg, put that in the dependencies, and land an LP branch that uses that version | 14:38 |
gary_poster | go thou! go forth! nay, stop that villainy! | 14:39 |
bac | benji: do you want my code? | 14:39 |
benji | bac: sure (any reason we don't just use your egg as-is?) | 14:39 |
gary_poster | because it is an egg drop | 14:39 |
gary_poster | not a branch | 14:39 |
benji | ah | 14:39 |
benji | I missed that bit. | 14:39 |
frankban | aaargh gary_poster, I've written buildout instead of buildbot in the dependencies dir (slave charm) can I r=you the fix? btw, we should call the next project "buildnot" | 14:40 |
benji | what PPAs are to debs X are to eggs (we need an X) | 14:41 |
bac | benji: see lp:~/rabbitfixture/timeout-exp | 14:41 |
benji | "buildnot"! | 14:41 |
bac | benji: review the changes as you may not like them. it is a mix of just upping the timeout and trying to parameterize it | 14:41 |
bac | buildsnot | 14:42 |
benji | bac: ok, thanks | 14:42 |
benji | pfft | 14:42 |
gary_poster | frankban, sorry, of course | 14:54 |
gary_poster | this error looks really familiar... http://ec2-184-72-171-77.compute-1.amazonaws.com:8010/builders/lucid_lp/builds/0/steps/shell_8/logs/summary | 14:55 |
gary_poster | anyone recognize it? Looking for it... | 14:55 |
benji | gary_poster: that looks similar to one Brad and I fixed | 14:56 |
gary_poster | bug 993510 | 14:56 |
_mup_ | Bug #993510: lp.services.job.tests.test_runner.TestJobRunner.test_runJobHandleErrors_oops_generated_user_notify_fails fails intermittently/rarely in parallel tests <paralleltest> <Launchpad itself:Triaged> < https://launchpad.net/bugs/993510 > | 14:56 |
gary_poster | bug 992692 | 14:56 |
_mup_ | Bug #992692: lp.services.mail.tests.test_incoming.TestIncoming.test_invalid_to_addresses fails intermittenty/rarely in parallel tests <paralleltest> <qa-untestable> <Launchpad itself:Fix Released by benji> < https://launchpad.net/bugs/992692 > | 14:56 |
bac | gary_poster: do you have the full, unfiltered subunit output for the db-devel r 11611 run? | 14:57 |
benji | gary_poster: 992692 is the one I was thinking of | 14:57 |
gary_poster | bac, no, sorry. If that's necessary then I have another machine running tests now, but you'll have to start the analysis over again | 14:57 |
gary_poster | :-( | 14:57 |
bac | gary_poster: it's all scripted | 14:57 |
bac | so it is no thing | 14:58 |
bac | also, to save you work i prefer the full subunit output not the ones divided by worker | 14:58 |
gary_poster | ok cool bac, got it | 14:58 |
bac | gary_poster: eta for completion? | 14:58 |
gary_poster | bac, 5 min. | 15:30 |
gary_poster | We are getting these somewhat frequently | 15:31 |
gary_poster | http://ec2-184-72-171-77.compute-1.amazonaws.com:8010/builders/lucid_db_lp/builds/0/steps/shell_6/logs/stdio | 15:31 |
gary_poster | Don't quite understand what is going on | 15:31 |
gary_poster | I think it happens when we switch between devel and db-devel | 15:31 |
gary_poster | but could be wrong | 15:31 |
gary_poster | bac, maybe you can just download this http://ec2-184-72-171-77.compute-1.amazonaws.com:8010/builders/lucid_lp/builds/0/steps/shell_8/logs/subunit/text | 15:33 |
bac | ok | 15:33 |
frankban | gmb, am I right if I say that you can create a co-located branch from a normal one (residing in a shared repo) with "bzr switch -b foo"? | 16:10 |
gmb | frankban: I have no idea, I've never tried that :). Sounds sane, though. | 16:13 |
frankban | gmb: oh, because you started using colo-branches with the plugin I guess | 16:15 |
gary_poster | frankban, try it :-) | 16:15 |
gmb | yep | 16:15 |
frankban | ok | 16:15 |
gary_poster | it's kind of sad that I really need to use Google to search Launchpad's buugs | 16:26 |
bac | yay, i have water again! | 17:13 |
gary_poster | yay! | 17:21 |
gary_poster | water good | 17:21 |
bac | there is a big capacitor that is used at start up to give the pump a good jolt. it was nasty and the wires fell out when touched. | 17:22 |
bac | i heart my redneck plumber | 17:22 |
bac | gary_poster: do you know what revno was used for the run referenced above? | 17:23 |
gary_poster | heh | 17:23 |
gary_poster | yeah I can get it | 17:24 |
gary_poster | bac, 15282 (see http://ec2-184-72-171-77.compute-1.amazonaws.com:8010/builders/lucid_lp/builds/0/) | 17:24 |
bac | got it | 17:24 |
gary_poster | frankban, was colocation as easy as it sounded? | 17:29 |
frankban | gary_poster: yes | 17:29 |
gary_poster | frankban, cool! I'll have to try. Sounds like the advantages of lightweight checkouts but simpler to use | 17:29 |
frankban | gary_poster: yes it's cool, and actually I think each colocated branch is internally a lightweight checkout of the current branch | 17:32 |
gary_poster | makes sense | 17:32 |
bac | gary_poster: i think the duplicate tests are not being run the required number of times. each only shows up once in the unified subunit output from the latest run. the test data i had from yesterday (for r 15139) shows multiple lines for the repeated tests. | 18:04 |
gary_poster | bac, oh, so maybe a regression with the newer version of the testr and friends packages. Thank you for investigating. | 18:05 |
bac | gary_poster: yeah, i think so. what was the date of the 15139 run? | 18:06 |
bac | nm | 18:06 |
gary_poster | bac, to call this finished, then, we should file a bug. We can throw it at something like testr and see if it sticks (if Robert will investigate) but +1 if you are willing and interested to take it farther | 18:06 |
bac | april 23 | 18:06 |
gary_poster | right, exactly | 18:07 |
bac | ok, i didn't realize it was that old | 18:07 |
bac | gary_poster: dumb question, how can i do the equivalent of 'bin/test -t faqcollection.txt' with testr? | 18:25 |
bac | ok, so this appears to be what i was after: | 18:29 |
bac | bin/test --subunit -t faqcollection.txt |testr load | 18:29 |
bac | duh | 18:29 |
gary_poster | good to know | 18:31 |
gary_poster | bad stats today :-/ | 19:09 |
bac | gary_poster: have time for a chat? | 20:12 |
gary_poster | bac, supposed to be talking to flacoste, but not, so sure | 20:13 |
bac | you want to make a hangout | 20:13 |
gary_poster | yeah ok | 20:13 |
bac | since whatever i did yesterday failed | 20:13 |
gary_poster | bac https://talkgadget.google.com/hangouts/_/49af6cb2474cf14af2cb37afc56f73c7c5fc120a | 20:14 |
bac | gary_poster: i've been doing some standalone (i.e. not buildbot) tests with testr --parallel | 20:56 |
bac | if i run "testr run --parallel -- -t faqcollection.txt" then faqcollection.txt appears once in the test partitions but gets run three times as seen in e.g. .testrepository/1 | 20:56 |
bac | looking at the code i don't see any use of sets or other attempts to make the test ids unique | 20:57 |
gary_poster | bac, dunno | 20:58 |
gary_poster | bac, also, when you run them non-parallel, are they in different layers? | 20:59 |
bac | so i remain a bit confused. the subunit output from the test you showed me today only had the suspect tests listed once | 20:59 |
bac | no | 20:59 |
gary_poster | or more generally | 20:59 |
gary_poster | how are they different? are we sure we are running the, differently even when we see them multiple times across workers? | 21:00 |
bac | but, i think i may have a different version of testrepository installed locally due to the funky ppa naming | 21:00 |
gary_poster | oh | 21:00 |
bac | let me try to get the same version and see what happens | 21:00 |
gary_poster | maybe | 21:00 |
gary_poster | ok | 21:00 |
gary_poster | "the," was supposed to be "them" | 21:01 |
gary_poster | bac I have to run | 21:01 |
bac | ok, have fun. wear a helmet if you go out | 21:01 |
gary_poster | will be back later fwiw | 21:01 |
gary_poster | heh thx | 21:01 |
bac | hail hurts | 21:01 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!