/srv/irclogs.ubuntu.com/2014/02/21/#juju-dev.txt

=== andrewsmedina_ is now known as andrewsmedina
=== Daviey_ is now known as Daviey
=== TheRealMue is now known as TheMue
=== gary_poster is now known as gary_poster|away
waiganiaxw: quick question?01:51
waiganiwhy do you do this: var _ state.Policy = environStatePolicy{}01:51
axwwaigani: it's a convention in juju to ensure that the implementation continues to implement the interface01:52
axwalso, documentation as code01:52
waiganiah right. I saw that before. The type assertion will fail if the struct's methods change.01:55
axwthe code won't actually compile, because the assertion can be proven at compile time01:59
axwit's a static assertion02:00
=== mwhudson is now known as zz_mwhudson
waiganiaxw: here is my first pass. https://codereview.appspot.com/6379004503:44
waiganiaxw: No tests yet and I have not implemented ValidateEnvironConfig in any provider yet. I took a look at adding it to the manual provider, but I wanted to talk over how to rewire SetConfig with you first.03:44
* axw looks03:44
axwwaigani: you shouldn't need to implement it anywhere, just use the one on EnvironProvider03:45
axwyou can get the provider from the config03:45
davecheneyaxw: http://paste.ubuntu.com/6969152/04:50
davecheney^ this error looks pretty terminal04:50
davecheneyany suggestions ?04:51
axwwtf mate04:51
axwumm04:51
axwno04:51
axware you packages up to date?04:51
axwwallyworld was having some problems with lxc on trusty04:52
davecheneyaxw: can you tell me the commands that get exectued under the hood04:52
wallyworldaxw: the latest update an d areboot fixed it04:52
davecheneyi could try to do them manually to smoke test it04:52
* davecheney shrugs04:52
davecheneyi'll give it a go04:52
davecheneyoooh, i see lxc-templates04:53
davecheneyfingers crossed04:53
wallyworldyeah you need lxc-templates installed as well as lxc04:53
wallyworldlxc rc4 works for me now04:53
wallyworldrc3 seemed to barf04:53
davecheneyyeah, i have the templates04:55
davecheneyjust i guess not hte latest ones04:55
axwdavecheney: I'm not sure what the command is, but you want to create a container with the ubuntu-cloud template04:55
davecheneymmm kay04:55
davecheneywhich series does that assume ?04:55
davecheneylots of shit just expects precise04:56
axwyou tell it which series04:56
axwlooks like you pass "-r trusty" to lxc-create04:56
davecheneywhoop whoop!04:56
davecheneyaxw: wallyworld nope, no joy05:42
davecheneystill horribly broken05:42
wallyworldhmmm05:43
wallyworldit was broken for me until i updated everything yesterday05:43
davecheneyubuntu@winton-02:~$ sudo lxc-create -t ubuntu-cloud -n dave05:43
davecheney/usr/share/lxc/templates/lxc-ubuntu-cloud: line 276: type: ubuntu-cloudimg-query: not found05:43
davecheneylxc_container: container creation template for dave failed05:43
wallyworldand got lxc rc4 and rebooted05:43
davecheneylxc_container: Error creating container dave05:43
davecheney^ this ?05:43
davecheneyii  lxc                                    1.0.0~rc4-0ubuntu1            ppc64el      Linux Containers userspace tools05:44
wallyworldi was getting a cgroups error05:44
axwdavecheney: do you have a "ubuntu-cloudimg-query" command on your system?05:44
davecheneyii  lxc-templates                          1.0.0~rc4-0ubuntu1            ppc64el      Linux Containers userspace tools (templates)05:44
davecheneyaxw: i guess not05:44
davecheneydoes juju enforce this ?05:44
davecheneyie, is there an install dep ?05:44
axwhmm05:44
axwnot sure05:44
davecheneyubuntu@winton-02:~$ sudo apt-get install ubuntu-cloudimg-query05:44
davecheneyReading package lists... Done05:44
davecheneyBuilding dependency tree05:44
davecheneyReading state information... Done05:44
davecheneyE: Unable to locate package ubuntu-cloudimg-query05:44
axwnot a package05:45
axwjust run the command05:45
axwshould tell you what you need...05:45
davecheneynot found05:45
davecheneyand no sugestion05:45
davecheneydo you have it ?05:45
axwyeah, how do I figure out what provides it again..05:46
davecheneydpkg -S $(which ubuntu-cloudimage-query)05:46
wallyworlddavecheney: i just thought - i'm running a trusty host but am deploying precise charms and that worked for me as of late yesterday05:46
davecheneywallyworld: same05:47
davecheneythis is a trusty host05:47
wallyworldok, i was thinking you may have been trying trusy charms05:47
axwdavecheney: cloud-image-utils05:47
davecheneywell, i have to do that05:47
wallyworldyeah my host is trusty also05:47
davecheneybut that is problem 2^1705:47
davecheneyaxw: ta05:47
axwand no, no check in the local provider05:48
davecheneyaxw: cock http://paste.ubuntu.com/6969306/05:48
axwbugger05:53
davecheneyok, the tasty-taco's are on it05:55
axwwallyworld: I think there may still be a problem with local upgrades, but not sure. Looks like a container in my local env is looking for downloaded-tools.txt in the host machine path07:46
wallyworlddata dir is used for the place to look07:47
axwunit-axwtest-0: 2014-02-21 07:46:44 ERROR juju runner.go:220 worker: exited "upgrader": cannot read tools metadata in tools directory: open /home/andrew/.juju/local/tools/1.17.3.1-trusty-amd64/downloaded-tools.txt: no such file or directory07:47
axwunit-axwtest-0: 2014-02-21 07:46:44 INFO juju runner.go:254 worker: restarting "upgrader" in 3s07:47
wallyworldit comes from agent config file07:47
wallyworldthat location looks ok07:47
axwno07:47
axwthat's on machine 007:47
axwthe unit is in a container07:47
axwI mean, that path would be ok if it were on machine 007:48
wallyworldwhat does the container agent conf file have07:48
wallyworldfor data dir07:48
axwin upstart it has --data-dir /var/lib/juju as expected07:49
wallyworldhmmm.07:49
wallyworldthat value is what is passed to the unit upgrader to use07:49
wallyworldah07:49
wallyworldi mean machine agent07:49
wallyworldum07:50
wallyworldffs07:50
axwthis is a unit agent if it matters07:50
wallyworldthere's a UnitUpgraderAPI07:50
wallyworldwhich is paased the data dir07:50
wallyworldwhich comes from the agent conf07:51
wallyworldso off hand i'm not sure what is going wrong07:51
hazmatg'morning07:51
wallyworldbut i have to run away now to go to soccer07:51
axwnps, later07:51
axwhey hazmat07:51
wallyworldfwereade_: i have to go to soccer now and will miss stand up but want to talk about my latest machine agent upgrade work later, ok?07:52
wallyworldtests pass but i'm unhappy with 2 bits of the implementation07:52
hazmataxw, welcome to the world of picky certs :-) thanks for diving into that.07:54
* hazmat has been having the same sort of cert fun with mysql ssl enablement07:54
axwno worries07:54
axwit's kind of interesting, but I wish rsyslog had better logging and less horrible configuration format :(07:54
axwit's all working now, just need to think about upgrades a bit07:55
hazmataxw, there's a major version change of rsyslog from precise to trusty 5.8 to 7.607:55
hazmatmight be worth verifying on both07:56
axwI've been testing with a trusty container and saucy host, local provider07:56
axwsaucy is still 5.x07:56
axwbut thanks for hte heads up07:56
hazmataxw, the notion of a worker seems like a sane path forward re syslog manage & upgrade. it also seems like it would be good on the road towards ha if there currently immutable.07:58
axwyeah, I think I will finish up the basics and then do that. there's no decent way of upgrading this stuff at the moment07:59
=== mthaddon` is now known as mthaddon
wrtpTheMue: ping09:13
TheMuewrtp: one moment, people at the door09:17
TheMuewrtp: so, back, we've got some troubles with our wastewater09:21
wrtpTheMue: my laptop has just broken, so i need to revert back to using my old mac, and i know you use some kind of VM - i wondered what you use, and if you recommend it09:22
TheMuewrtp: I'm using VMware Fusion and it works fine, only needs some RAM. At least 8 Gigs. Advantage is that you can do snapshot, e.g. before upgrading to new releases like trusty.09:24
TheMuewrtp: But it costs. I got it cheaper when it has been released the first time.09:25
TheMuewrtp: There are also Parallels and VirtualBox.09:27
dimiternaxw, hey are you around still?09:28
wrtpTheMue: ok. i did have an old vmware version, but it suggested i upgrade it, but the upgraded version wasn't compatible with the old licence keys09:28
wrtpTheMue: so i've just bought a new version09:29
wrtpTheMue: and now i'm wondering how to get ubuntu running on it. ideally i'd like to be able to bootstrap from my laptop's hard drive, but i'm not sure that's possible09:29
axwdimitern: yup?09:30
dimiternaxw, I saw your comments on my ssh fix09:31
axwoh yeah09:31
dimiternaxw, if you have 10m I wanted to ask you to bring me up to speed with all the stuff around ssh09:31
axwsure09:31
dimiternaxw, so we're using 2 different clients? why's that?09:31
axwfor Windows09:31
dimiternah09:31
axwso, because bootstrap now requires ssh, we *need* an ssh client09:32
dimiternright09:32
axwwhereas before we could just say meh you can't have juju ssh09:32
dimiternhow do you suggest i should test the fix to make sure i didn't introduce a regression?09:32
axwso now if we don't find "ssh" in $PATH, then we fall back to an embedded go.crypt/ssh client09:32
* axw looks at the change again09:33
dimiterni suppose: ssh + scp commands, manual bootstrap + manual add-machine ssh, possibly debug-hooks.. windows?09:33
axwif ssh/scp aren't in your path, it'll behave like it would on Windows09:33
dimiterni found out scp -- <args> does not work btw09:33
dimiternah, good point, I can fake it on ubuntu then09:33
TheMuewrtp: afaik it's not possible, you have to create a new image. but then you should be able to mount the old drive09:34
axwah, so if we pass through -- then the embedded client will break09:34
dimiternwrtp, why the change of nick? going clandestine? ;)09:34
dimiternright!09:35
dimiternso basically I should test both client paths everywhere :)09:35
axwyeah, it's a pain in the bum but that's probably safest09:35
dimiternok, thanks a bunch, will do09:35
axwso you'd probably want to compare what happens when you do "juju ssh 0 -- command" and "juju ssh 0 command"09:36
dimiternwhen is the -- jobbie supposed to work?09:37
dimiternwith the embedded or the openssh client?09:37
axwopenssh09:37
dimiternok, i think that's all i need then09:38
axwthat's just to say that the args won't go to ssh, but to the command it execs09:38
dimiternright09:38
=== wrtp is now known as rogpeppe
TheMuewrtp: sorry for the breaks between my answers, have also to answer the craftsmen some questions around our problems09:40
TheMues/wrtp/rogpeppe09:40
rogpeppedimitern: wrtp used to be my nick, and i'm on my mac laptop which has my old configuration set up09:40
dimiternrogpeppe, i see09:41
rogpeppehmm, i've managed to mount my laptop drive in my NAS box, but i only see a grub partition09:43
rogpeppei wonder how i can get access to the main stuff09:44
* TheMue is reminded to do a backup of his VMs after rogs troubles09:46
TheMuerogpeppe: the usage of VMs allows me to also have pure testing VMs, with no influence of the development VM09:47
=== alexlist` is now known as alexlist
dimiternrogpeppe, standup?10:46
rogpeppedimitern: could you paste a link please?10:46
dimiternhttps://plus.google.com/hangouts/_/calendar/am9obi5tZWluZWxAY2Fub25pY2FsLmNvbQ.mf0d8r5pfb44m16v9b2n5i29ig10:46
rogpeppe"you are not allowed to join this video call"10:47
rogpeppeone mo while i get my auth key10:47
rogpeppedimitern:  ^10:47
dimiternyou need to login to the canonical account10:47
rogpeppeyeah10:48
wallyworldfwereade_: hi, do you have a minute to talk about my upgrade branch? I think i really want roger but he isn't logged on11:51
fwereade_wallyworld, I'm making an effort to program in the dark today but I'm actually already distracted, so let's11:52
wallyworldsorry :-(11:52
wallyworldhttps://plus.google.com/hangouts/_/calendar/am9obi5tZWluZWxAY2Fub25pY2FsLmNvbQ.mf0d8r5pfb44m16v9b2n5i29ig?authuser=111:52
wrtpmgz: ping13:30
wrtpat long frickin' last!13:34
wrtpi have finally managed to get my laptop drive mounted in a VM13:34
natefinchwrtp: nice!13:34
wrtpthis was the final key: http://www.tuxradar.com/answers/29613:35
wrtpbut, honestly!13:35
dimiternnatefinch, how can I build a windows installer from trunk?13:47
natefinchdimitern: you need innosetup (either in a windows VM or possibly Wine, I haven't tried Wine).  Build cmd/juju using cross compilation to make a windows binary, copy juju.exe to scripts/win-installer, then load scripts/win-installer/setup.iss into innosetup and build it13:54
natefinchdimitern: I should put that into a doc somewhjere13:54
dimiternnatefinch, right, how to do the cross compilation?13:55
natefinchdimitern: http://dave.cheney.net/2013/07/09/an-introduction-to-cross-compilation-with-go-1-113:56
natefinchdimitern: unfortunately, step 1 is: "install go from source"13:56
dimiternnatefinch, ok, tyvm13:58
natefinchdimitern: welcom13:58
mgzwrtp: yeay!14:26
mgzcan you work okay from the mac then?14:27
wrtpmgz: we'll see. it looks ok so far.14:27
wrtpmgz: much slower, but that's probably mostly the disk drive being spinning rust14:28
mgzdo you have a date idea on when the repair will be done? I'm thinking about london...14:28
=== benji_ is now known as benji
wrtpmgz: they said 6 working days14:52
wrtpmgz: so i'm crossing my fingers14:52
wrtpanyone remember how to get the required mongod when running under precise?15:24
wrtpmgz: ^15:24
mgzadd the cloud-tools pocket and install from there15:24
mgzyou can probably use the new juju-mongodb even15:25
wrtpmgz: how do you add the cloud-tools pocket?15:25
wrtpmgz: apt-add-repository cloud-tools-pocket ? :-)15:25
mgz`sudo add-apt-repository cloud-archive:tools`15:25
mgzthe apt-get update and install juju-mongodb15:26
wrtpmgz: thanks15:26
wrtpmgz: (i have the binaries, but of course i'm running precise and the binaries are trusty 'cos they're on my laptop drive)15:27
wrtpmgz: i tried fiddling with LD_LIBRARY_PATH but no joy. i love shared libraries15:28
wrtpmgz: hmm; http://paste.ubuntu.com/6971425/15:30
wrtpand /etc/apt/sources.list doesn't even have 60 lines :-\15:31
mgzwell, that's amusing.15:31
mgzlook in /etc/apt/sources.liust15:31
mgz*look in /etc/apt/sources.list.d/15:32
wrtpcloudarchive-tools.list is there15:32
mgzcan you pastebin that?15:32
wrtpmgz: does that mean it's actually worked, despite the error message?15:32
wrtpdeb http://ubuntu-cloud.archive.canonical.com/ubuntu precise-updates/cloud-tools main15:33
wrtpdeb-src http://ubuntu-cloud.archive.canonical.com/ubuntu precise-updates/cloud-tools main15:33
mgzit does if apt-get update works, if it doesn't then presumable it borked15:33
mgzthat seems fine15:33
wrtpwell, apt-get update works, so i guess it worked :-)15:34
wrtpmgz: except it doesn't seem to know about juju-mongodb15:34
mgzbug 789859 it seems15:34
_mup_Bug #789859: add-apt-repository adds invalid deb line if PPA contains a - <amd64> <apport-bug> <natty> <software-properties (Ubuntu):Fix Released> <software-properties (Ubuntu Precise):Triaged> <software-properties (Debian):Invalid> <https://launchpad.net/bugs/789859>15:34
sinzuimgz, do you have a few minutes to review https://codereview.appspot.com/6678004615:35
mgzsinzui: sure thing, but natefinch might be a better bet if he's around15:35
mgz..actually, that's trivial15:36
mgzsinzui: lgtm15:36
mgzI presume the version bump in the go code already happened?15:36
sinzuiwrtp, when this bug is fixed, we will release a new juju that is juju-mongodb aware: https://bugs.launchpad.net/juju-core/+bug/127193715:36
_mup_Bug #1271937: Use juju-mongodb when the package is available <arm64> <ci> <mongodb> <ppc64el> <trusty> <juju-core:In Progress by natefinch> <https://launchpad.net/bugs/1271937>15:36
mgzwrtp: does `grep -r "^in" /etc/apt/sources.list*` give anything?15:39
wrtpmgz: nope15:39
mgzwell, lets assume it was transient weirdness then15:39
wrtpmgz: and the only entry in sources.list.d is the cloudarchives-tools.list file15:39
wrtpmgz: except that... i can't apt-get juju-mongodb15:40
wrtpmgz: apt-cache search juju doesn't print anything that looks reminiscent15:40
mgzyeah, it seems it only has the normal mongdb packages at present15:41
mgzso, just install mongodb (or just -server)15:41
wrtpmgz: so if i apt-get install mongodb, will i get the cloud-tools mongodb15:41
wrtp?15:41
mgzyup15:42
wrtpmgz: ok, i'll try that15:42
mgzit will also upgrade if you have the precise one15:42
wrtpmgz: yeah, it looks like it's fetching from the right place, ta15:42
mgzshould be 1:2.4.6-0ubuntu5~ctools115:42
wrtpmgz: i see that15:44
wrtpmgz: at last! i have juju tests passing under my VM15:48
wrtpmgz: thanks for the help15:48
mgzace!15:48
Beretsinzui, when do you expect .3?15:54
BeretI can't remember15:54
sinzuihmm, between 2 and 20 hours.15:54
sinzuiBeret, everything in in the Lp builders as of 5 minutes ago15:54
Beretsinzui, awesome, thanks :)15:56
* wrtp uses chroot(1) for real for the first time ever15:57
=== wrtp is now known as rogpeppe
mgzrogpeppe: I've used it a fair bit from inside chromeos recently when I needed to restore my ubuntu partition :)15:59
rogpeppemgz: aw dammit, "no ptys"16:00
rogpeppemgz: i can't run a terminal emulator in it16:00
mgzreally? what's the underlying terminal?16:02
natefinchsinzui: btw I am trying to land my "use juju-mongod" branch, but hit some spurious test failures.  Going to try again in a little bit, have to run to the store real quick.16:03
sinzuithank you natefinch16:03
dimiternsinzui, sorry, it seems I won't be able to land the fix for bug 1281577 in time for 1.17.3, as I said in the xteam doc16:12
_mup_Bug #1281577: juju 1.17.2 doesn't pass command line options to ssh <docs> <landscape> <regression> <ui> <juju-core:In Progress by dimitern> <https://launchpad.net/bugs/1281577>16:12
mgzsinzui: have you picked a trunk rev already, or will you go with what we have in a few hours?16:14
sinzuidimitern, mgz, 1.17.3 tarball is released.16:14
sinzuidimitern, I will release 1.17.4 when I see natefinch's branch land and CI confirms it works with the new packaging rules16:15
sinzuidimitern, So I may release 1.17.4 tomorrow, certainly next week16:15
dimiternsinzui, ok, great16:15
rogpeppeoh bugger, my disk enclosure seems to have stopped working17:47
rogpeppei hope it hasn't killed the disk17:47
mgz>_<17:50
rogpeppemgz: ah, it seems that the VMWare USB stuff is extremely sensitive to unplugging. when i unconfigured the usb, shut everything down, and started it all up in just the right order, it's working again18:02
rogpeppephew18:02
rogpeppebut time to start a backup going, i think18:02
sinzuinatefinch, thank you. I will push the new packaging rules18:20
natefinchrogpeppe: sounds like one of those things where, once you get it working you don't. touch. anything.18:33
rogpeppenatefinch: yeah. the connectors seem dodgy... both s/w and h/w18:34
rogpepperight, i'm done18:55
rogpeppehappy weekends all18:55
rogpeppequestion to ponder: anyone know why StatePort is no longer stored in the agent config?18:57
mgzhm... nope18:58
mgzis it just a constant now?18:58
natefinchrogpeppe: I think stateport was something I added with the new machine agent  stuff and that it wasn't there before.... and since that code isn't merged yet, it's not in trunk19:05
natefinchmgz: have time for a quick review?  https://codereview.appspot.com/67080043/19:16
mgznatefinch: will do in a bit19:17
rick_h__anyone able to give me hints on getting the error "ERROR empty image-stream in environment configuration" during a deploy command?19:54
rick_h__this is on ec2 during a test of our charm and everything I see around image-stream seems based on azure19:55
rick_h__my env config has no image-stream config in it for ec2 env19:55
rick_h__http://paste.ubuntu.com/6972735/ is the command run and error returned19:55
rick_h__trying it by hand I get "ERROR juju.cmd supercommand.go:294 empty image-stream in environment configuration"20:03
dpb1`rick_h__: check .juju/environments/ec2.jenv20:11
rick_h__dpb1`: ah, good call. Regenerating that and trying again20:11
dpb1`dpb1`: k20:12
dpb1`blah20:12
dpb1`rick_h__: ok. :)20:12
rick_h__dpb1`: ok, so I get that error on the juju deploy command. There's no image-stream set in the  environments.yaml or ec2.jenv20:22
rick_h__if I try to set the image-stream to : or : "" I get errors on bootstrap about it being a bad value20:22
dpb1`rick_h__: is it possible for you to paste your environments.yaml (sanitized)?20:23
dpb1`just that section20:23
natefinchrick_h__: setting it to "released" should work, though not set should be equivalent to "released"20:24
rick_h__dpb1`: sure thing, sec20:24
rick_h__natefinch: yea, that failed for me as well. I saw that in the azure stuff20:24
rick_h__I'm on 1.17.2-trusty-amd6420:24
rick_h__http://paste.ubuntu.com/6972856/ dpb1`20:25
rogpeppenatefinch: stateport is in 1.12 serialization but not 1.1620:32
natefinchrogpeppe: ok, I just know I added stateport to something in my changes, couldn't remember what.20:37
natefinchrick_h__: add in a --show-log to the deploy command and it'll spit back a bigger log that may be helpful20:40
rick_h__natefinch: http://paste.ubuntu.com/6973005/20:46
natefinchrick_h__: well that's less than useful, huh? :/20:49
rick_h__:)20:50
rick_h__sinzui: is .3 in trusty or on the way? to test for #127763622:17
_mup_Bug #1277636: juju 1.17.x cannot destroy a 1.16.5 azure environment <azure-provider> <ci> <destroy-environment> <regression> <juju-core:Fix Released by axwalk> <https://launchpad.net/bugs/1277636>22:17
rick_h__all of us hitting the bug are trusty users on .2 currently22:17
sinzuirick_h__, It in the the public ppa. Ubuntu are responsible for putting it in trusty22:29
sinzuirick_h__, I am sorry for trusty users being forced test unstable software. My plan was to never let it devel releases go into an LTS. You can find the new release in https://launchpad.net/~juju/+archive/devel/+packages for now22:38
rick_h__sinzui: rgr, thanks22:38
sinzuirick_h__, you may already know about this http://manage.jujucharms.com/heartbeat22:58
sinzuijujucharms.com is not happy, looks like manage...is at fault22:58
rick_h__sinzui: working with webops on it and running out of time to get a fix :/22:58
rick_h__ES voted out all the shards again but the fix from last time is not working22:59
sinzuirick_h__, The fix was to put the other values back...the 3 mintues later all was good22:59
sinzuiIs that the fix tried?22:59
rick_h__sinzui: "values back"?23:00
rick_h__sinzui: last time we removed /var/lib/elasticsearch and restarted it and then restarted charmworld and reran es-update and it rebuilt itself23:00
rick_h__I'm trying to figure out how/what command to send to ES via the JSON api to get it to allocate a shard23:01
sinzuirick_h__, the first time it happen, it was caused when the charm config value was changed. It didn't need to be, if you know what the old value was, setting the config back restored it23:01
rick_h__sinzui: oh hmm, no config changed I'm aware of. Will see if we can get what's set and compare.23:02
rick_h__sinzui: should be back now23:50
sinzuithank you rick_h__23:50
rick_h__sinzui: turns out they run mojo which runs CI and part of that brings up a new ES server. Which any other ES servers on the network try to join with and caused things to go badly.23:51
rick_h__loves ES and it's auto scaling stuff23:51
sinzuirick_h__, ha ha23:51
rick_h__kind of funny that CI passes while production goes down23:51
sinzuirick_h__, that was the reason we added the shard and names parts to charmworld, to ensure the names are always unique23:52
rick_h__yep, the names in CI are getting updated now23:52
sinzuiTheir CI, has to use different names or run on different networks per ES23:52
rick_h__yep23:52
rick_h__that's the great discovery of the day23:52

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!