/srv/irclogs.ubuntu.com/2015/07/16/#juju.txt

tvansteenburghstub are you around?03:59
stubtvansteenburgh: morning04:50
=== zz_CyberJacob is now known as CyberJacob
reithSomehow juju master advertise wrong IP address for us, do you have any idea to fix it?07:00
reithIt worked for serveral months but now it sends `apiaddess` as a private ip address (10.0.*)07:01
=== CyberJacob is now known as zz_CyberJacob
jamespagethedac, feedback on mps09:39
bloodearnesthazmat, tvansteenburgh, dpb1: hello! Could one of you juju-deployers folks please merge this already-reviewed branch for deployer?10:36
bloodearnesthttps://code.launchpad.net/~bloodearnest/juju-deployer/annotate-branches/+merge/23927010:36
=== psivaa is now known as psivaa-afk
=== blahdeblah_ is now known as blahdeblah
hazmatbloodearnest: looking11:48
hazmatbloodearnest: getting an error running the tests on import annotation test11:49
hazmattest_data/wiki-branch.yaml missing11:50
tvansteenburghanyone know how to work around this problem?12:02
tvansteenburgh$ juju set postgresql install_keys="`cat ACCC4CF8.asc`"12:02
tvansteenburgh2015-07-16 04:19:50 INFO install yaml.scanner.ScannerError: mapping values are not allowed here12:02
tvansteenburgh2015-07-16 04:19:50 INFO install   in "<unicode string>", line 2, column 8:12:02
tvansteenburgh2015-07-16 04:19:50 INFO install     Version: GnuPG v112:02
tvansteenburghi get the same error if i paste the key contents into juju-gui12:03
tvansteenburghfull traceback: http://pastebin.ubuntu.com/11887422/12:10
tvansteenburghi'm just trying to figure out how to escape the colon12:10
* tvansteenburgh tries \x3A ...12:13
stubtvansteenburgh: That property is yaml encoded as a string12:13
stubtvansteenburgh: So "echo '[' "'"`cat ACC.asc` "'"']'" or similar bash monstrosity that actually works12:14
* stub wishes juju cli didn't make things difficult by trying to keep things simple12:15
tvansteenburghstub: thanks i'll try that. i'm surprised it breaks in the same way when using the gui12:17
stubtvansteenburgh: The charm is just passing the string to charm-helpers, which spits out that traceback since it isn't valid yaml.12:17
stubIt could do with better validation and error messages, until juju gives us actual structured data here and we can stop serializing things ourselves.12:19
stubWhoops, I'm late12:19
* stub wanders off12:19
tvansteenburgho/12:19
bloodearnesthazmat, hmm, ok looking12:23
bloodearnesthazmat, added and pushed12:27
bloodearnestwthanks12:27
coreycbjamespage, gnuoy: could one of you take a look?  it fixes a master branch deploy from source failure that came about yesterday.  https://code.launchpad.net/~corey.bryant/charm-helpers/upgrade-pip/+merge/26488612:33
jamey-ukCan anyone help me with the error when deploying my Rails charm? I can't recreate the problem on Heroku, Dokku or locally: https://gist.github.com/anonymous/be6132615872fde0848a. I also have tried recreating the error on the VM itself with no luck.12:39
beisnergnuoy, jamespage - fyi bug 147532013:57
mupBug #1475320: rmq next charm:  pkg install fails when reverse dns fails - Error: unable to connect to node 'rabbit@10-245-173-55': nodedown <openstack> <uosci> <rabbitmq-server (Juju Charms Collection):New> <https://launchpad.net/bugs/1475320>13:57
asanjarcory_fu: kwmonroe bigdata review q14:03
cory_fuasanjar: kwmonroe has a meeting.  I thought you did, as well14:03
cory_fuYou want to shift it 30 min?14:03
asanjarcory_fu: sure14:03
kwmonroe+1, thanks14:04
jamespagegnuoy, beisner: https://code.launchpad.net/~james-page/charm-helpers/liberty-versioning/+merge/26499814:22
jamespagebeisner, I see the problem14:28
jamespagemy fault14:28
jamespagerefactoring missed something14:28
=== natefinch is now known as natefinch-afk
jamespagebeisner, can you try this branch - lp:~james-page/charms/trusty/rabbitmq-server/fixup-configure-nodename14:32
jamespagebeisner, gnuoy: https://code.launchpad.net/~james-page/charms/trusty/rabbitmq-server/fixup-configure-nodename/+merge/26500114:34
thedacjamespage: thanks for the feedback. That all makes sense.14:38
thedacgnuoy: any chance you could comment on jjo's nrpe bug? https://bugs.launchpad.net/charms/+source/nrpe/+bug/147320514:39
mupBug #1473205: nrpe charm creates checks with _sub postfix, breaking compatibility with nrpe-external-master <canonical-bootstack> <nrpe (Juju Charms Collection):New> <https://launchpad.net/bugs/1473205>14:39
gnuoythedac, sure, otp atm14:40
thedactake your time14:40
beisnerjamespage, thanks - sure will do14:40
jamespagethedac, np14:42
asanjarlazyPower: hi there15:01
lazyPowerasanjar: greetings15:01
asanjarlazyPower: I am reviewing your AWESOME etcs charm, but bundletester fails with http://paste.ubuntu.com/11888125/. Am I missing anything?15:02
lazyPowerasanjar: hmm looks ilke one of the etcd services didn't start in the cluster15:06
asanjarlazyPower: i am using "local"15:06
asanjarlazyPower: going to try on AWS15:07
asanjarlazyPower: according to this I got all services http://paste.ubuntu.com/11888149/15:08
asanjarlazyPower: but idle15:09
lazyPowerasanjar: what i imagine happened is it race conditioned and we haven't seen it happen in our testing/deployments yet15:09
marcoceppihey stub, I noticed that cassandra only runs on the public-address, is that by design?15:24
marcoceppiit's problematic because it appears that we're hitting network latency since cassandra has to be routed out of the cloud then back in again15:24
beisnerjamespage, no love with that rmq branch.  fails @ the same place.  it looks like the main diff here is that rmq is in lxc, and the ptr record @ maas doesn't match the juju unit's hostname:  http://paste.ubuntu.com/11888267/15:35
beisner(diff between this test and our os-on-os testing that is)15:35
xoritorhello15:45
xoritordoes juju have support for scaleio?  i know it has support for ceph15:46
xoritorceph is fast enough, so scaleio is not an issue15:46
beisnerjamespage, added to bug15:46
* xoritor is a juju newb15:47
xoritori am looking here https://jujucharms.com/store15:47
xoritoris that the right place to look for supported things?15:48
* xoritor is doing a lot of reading15:48
lathiatxoritor: Charm store is the right place to look, though, there may be charms not in the store you can find elsewhere (github or something)15:49
xoritorlathiat, thanks!15:50
xoritorlathiat, or it may be named something that i am not searching for15:50
lathiatxoritor: also may simply not exist15:50
xoritorlol15:50
xoritortrue15:50
xoritorceph is what i have been using and is fast enough for me15:51
aisraellazyPower: That drone-ci stuff is really sweet. Nicely done!15:56
aisraelhttp://blog.dasroot.net/2015-continuous-integration-with-juju-and-drone-ci.html15:56
lazyPoweraisrael:  thanks! :)15:57
xoritorso when using juju if i want all machines to run all things i can manually place then all on all hosts?16:08
xoritors/then/them/16:08
xoritorin the web ui16:08
lazyPowerxoritor: using machine view, you can do unit placement, correct.16:09
xoritori need the ha-proxy to make them H/A though correct16:09
xoritorlazyPower, that seems too eazy16:09
lazyPowerJuju is good at that. Making complex things deceptively simple16:10
ddellav:D16:10
xoritorif that REALLY is that simple you may have won me over16:11
marcoceppirick_h_: is there still a delay in the charmstore16:11
rick_h_marcoceppi: not that I'm aware of.16:11
xoritorone more thing... can juju+maas run on one of the openstack nodes?16:11
rick_h_marcoceppi: it took a few hours for things to sync, but that was all.16:11
xoritoror does it need to be outside of the stack16:12
marcoceppiI pushed a charm to lp:~marcoceppi/charms/trusty/cassandra-stress/trunk a bit a go, but don't see it16:12
marcoceppirick_h_: is there anything that will reject a charm from ingestion?16:12
rick_h_marcoceppi: looking16:12
rick_h_marcoceppi: proof (we still have to keep charmworld/charmstore in sync so it first has to be in charmworld before the charmstore will load it)16:13
xoritorok i also see you need a minimal of 7 machines... will it work with 5?16:13
marcoceppirick_h_: will Warning stop from ingestion of personal namespace?16:13
rick_h_marcoceppi: no16:13
xoritorwe are a really small shop16:13
marcoceppixoritor: so, maas can be put in a virtual machine on one of the nodes, it's not really recommended however16:16
marcoceppixoritor: juju however runs from your laptop, it's a client side tool16:17
xoritorhmm16:20
xoritorcould i put it in a VM?16:20
marcoceppixoritor: could what? juju or maas?16:20
xoritormarcoceppi, juju16:20
marcoceppixoritor: sure, again, it's just a command line tool for managing a deployed juju environment. It could be installed on windows, mac, or linux, in a vm or a docker container, doesn't matter16:21
xoritorcool... multiple instances?16:21
xoritorie... one in a VM and one on my laptop?   without detriment?16:22
aisraelxoritor: You could, but it's not necessary. You can control multiple juju deployments from your laptop16:23
marcoceppixoritor: sure, again, it's just the client side tool. Juju then uses MAAS, or Amazon, or GCE, or Azure, or OpenStack, or whatever other cloud to actually provision and deploy machines16:23
xoritorAWESOME!!!16:23
lazyPowerxoritor: Juju is a client/server model - each environment gets a single state server to drive the deployment/orchestration/statemanagement of the environment.16:23
lazyPowermarcoceppi: dont forget about the state server :)16:23
marcoceppilazyPower: I'm not, but you don't install that16:23
marcoceppijuju installs that16:23
lazyPowerwell,16:23
lazyPowerok you're right16:23
lazyPowerin that context16:23
xoritorwhere is the "state server" located?16:24
rick_h_marcoceppi: getting logs pulled from IS and looking into it. Sorry for the delay. I'm not aware of any issues and lazyPower had something ingested yesterday so not sure what it doesn't like about this one16:24
lazyPowerxoritor: it occupies a single node in your environment that you're managing with juju16:25
xoritorlazyPower, you mean runs on one node beside other things or takes an entire node16:26
marcoceppixoritor: you, from your computer, run `juju bootstrap`, this connects to whatever machine provisioning tool you want to use (gce, aws, maas, openstack, etc) and creates a node on that service. Once teh environment is bootstraped you can start deploying and managing that environment. Juju can manage and control multiple environments at a time and each environment can have it's on Juju GUI so you don't even have to use the command line16:27
marcoceppixoritor: while the bootstrap node does take an entire instance, you can put other things on it16:27
xoritorok16:27
xoritorwhew16:27
xoritorjust making sure16:27
marcoceppixoritor: I've managed to deploy openstack on to just two physical nodes, so it's definitely possible to model that level of complexity but at a very small physical footprint16:28
marcoceppiwell, technically three machines if you include MAAS16:29
xoritorcan i use just 5 physical machines if i have juju+maas16:29
xoritorif they all have everything with h/a16:30
marcoceppixoritor: I'm not sure about the minimum requirements for OpenStack HA, but I think 5 nodes would be enough16:30
marcoceppithough, it would be pretty tight ;)16:31
xoritorthey are all pretty beefy (12 cores, 128 GB DDR4, 4x1 Gbit, 2x40 Gbit, 1x256 GB SSD, 1x1 TB SSD, 1x2 TB PCIe)16:33
xoritori can get more ram if we need it16:34
marcoceppixoritor: tight in the amount of services you'd have to spread out, though those are some nice machines16:34
xoritoryea the new haswell xeons are pretty nice i have to say16:35
xoritorwe are small but we are not afraid of buying what we need to get the job done16:36
xoritorthats why i was asking if scaleio was supported16:37
xoritorit is supposed to be much much faster than ceph with ssd/pcie16:37
xoritorhttp://virtualgeek.typepad.com/virtual_geek/2015/05/emc-day-3-scaleio-unleashed-for-the-world.html16:38
xoritorthat says 24 times faster with SSD only16:39
marcoceppixoritor: so, scaleio can be supported, the charms are a plugin architecture and there's a pretty easy way to build a charm for cinder/storage16:39
marcoceppiwe have a few charms for other EMC products, while it's not scaleio, they could be modifiedto add support for it16:39
xoritorhmm16:40
xoritori have my reservations on using it anyway...16:42
xoritori dont fully trust emc16:43
xoritor;-)16:43
marcoceppirick_h_: it showed up fwiw16:50
rick_h_marcoceppi: ah that's good. /me checks when you pushed it up16:50
beisnerjamespage, gnuoy - i've changed the stance of bug 1475320 to be:   The rabbitmq-server next charm works on bare metal but not in a container; the stable charm works fine on both.17:05
mupBug #1475320: rmq next charm:  pkg install fails when deployed to lxc <openstack> <uosci> <rabbitmq-server (Juju Charms Collection):New> <https://launchpad.net/bugs/1475320>17:05
beisneralso added a reproducer, and that enviro is still up @ dellstack for a bit while i collect logs.17:05
=== natefinch-afk is now known as natefinch
rick_h_NOTICE: jujucharms.com is having a webui outage due to a failed redis. Charm deploys should work as normal and the API is available.17:14
xoritoraaah redis17:14
xoritorhow i love/hate thee17:14
=== psivaa-afk is now known as psivaa
rick_h_NOTICE: jujucharms.com webui is back up17:37
xoritorcan juju and maas be run in docker containers?18:04
lazyPowerxoritor: teh juju client can18:11
lazyPowerxoritor: i haven't tested MAAS ina  docker container, but theory states you should be able to isolate the applications into containers. I know it can be driven from a LXC container - as that has been tested to drive VMAAS. setup your MAAS region/cluster controller in lxc, and drive KVM containers as the nodes.18:12
marcoceppilazyPower: maas needs at least a KVM, as it needs - or works best - with two nics18:12
lazyPowermarcoceppi: you can  create two virtual nics in a container and make it work.18:13
xoritori have found a few docker files to build maas server(s)18:13
xoritorinteresting idea18:13
xoritori am trying to figure out the easiest approach to getting maas+juju up and running with my limited hardware18:14
xoritori will run everything everywhere if i need to that does not bother me ;-)18:15
xoritorjust need to get things up and running18:16
xoritori also cant spare a machine or three for just doing maas18:16
xoritorsorry for bringing that up here not much response over in #maas when i asked18:17
jcastrowell, if you don't have a ton of hardware then you don't really need maas for anything right?18:31
xoritori would like to use it for the initial setup and the "in case i get hit by a bus" scenario18:31
xoritori like to be a bit pro active for things like that18:32
xoritorheck its the whole reason i am looking into this in the first place18:33
xoritorif it were just me i would hand roll a custom solution that fit my needs and worked exactly how i wanted it to work as minimal as i could make it18:34
xoritor;-)18:34
xoritorbut thankfully i am not immortal18:34
marcoceppixoritor: so MAAS is really geared for if you have 10 or more machines, though that's not to say you can't use it for your use case (I have 4 machines total at home, 3 I use to deploy onto and 1 for MAAS)18:41
marcoceppibut MAAS as some pretty specific requirements, the first being that it really /really/ will want two nics18:41
marcoceppibut you don't need a giant beefy machine to run maas, at home I use an old optiplex, and regularly people have used $300 little Intel NUCs18:41
xoritori have an 8 core system i can use a VM for and give it 16GB ram 200 GB SSD and up to 4 1Gb nics via passthrough would that work?18:42
xoritori just cant give it the whole machine18:42
marcoceppixoritor: yeah, that should work fine18:42
xoritorit would have to be kvm18:42
marcoceppiright18:42
marcoceppiI mean, you only need like 2-4 gb18:43
xoritorcool18:43
marcoceppiand a core18:43
marcoceppiand maybe 50-80GB18:43
marcoceppidepending on how many images you want to roll out18:43
xoritori really do not have much ;-)18:43
xoritorbut i want to be able to have something in place should i have a heart attack18:43
marcoceppiwell if it's just Ubuntu 14.04, that's one image18:44
xoritorit has happened to us before18:44
xoritorwe are overly paranoid18:44
xoritorim 43 and the YOUNGEST employee18:44
xoritor;-)18:44
xoritorheh18:44
xoritoryou would never believe how many people drop a load on that one18:45
tasdomashi18:57
tasdomasamulet does not support actions yet, does it?18:57
tvansteenburghtasdomas: no18:59
tasdomasis there a good e19:13
tasdomassorry19:13
tasdomastvansteenburgh, thanks19:13
tasdomaswhat version of juju are tests currently being run against?19:13
tvansteenburghtasdomas: which tests? :)19:14
tasdomastvansteenburgh, well, when a charm goes through the approval process19:14
tvansteenburghcharm tests run against latest, so currently 1.24.219:14
tasdomastvansteenburgh, ah great - thanks19:14
tvansteenburghbtw, fwiw, adding action support to amulet is on the todo list. should be pretty straightforward once we get a little time to do it19:15
tasdomastvansteenburgh, hm - I'll take a look at it, have a charm that uses actions extensively (lp:~tasdomas/charms/trusty/git/trunk)19:19
tvansteenburghtasdomas: sure if you wanna submit a PR that'd be awesome :)19:20
tasdomashm, running charm test -e local returns:19:28
tasdomasERROR there was an issue examining the environment: failure setting config: mkdir /.juju: permission denied19:28
marcoceppitasdomas: try using bundletester instead19:30
tasdomasmarcoceppi, bundletester?19:31
marcoceppitasdomas: yes, bundletester is actually what we use in CI19:32
marcoceppihttps://pypi.python.org/pypi/bundletester19:32
coreycbmarcoceppi: any chance you could review this?  the other guys are eod.   https://code.launchpad.net/~corey.bryant/charm-helpers/upgrade-pip/+merge/26488619:34
marcoceppicoreycb: sure, giv me a few mins19:34
coreycbmarcoceppi, thanks19:34
marcoceppicoreycb: LGTM, this may actually be the errors I was hitting with neutron19:35
marcoceppiit was something about version strings19:35
marcoceppinever thought it was because pip was too old19:35
coreycbmarcoceppi, thanks.  could be if you were using master.  could you land that if you weren't going to already?19:36
marcoceppicoreycb: merged, will this be in the 07.2015 release of the openstack charms?19:37
coreycbmarcoceppi, it will be now!19:37
marcoceppierr, rather, can you let me know when these land in trunk?19:37
marcoceppis/trunk/devel19:37
marcoceppiI'll give the deployment another go19:38
coreycbmarcoceppi, yep will do19:38
marcoceppithanks!19:38
tasdomasmarcoceppi, thenks19:40
tvansteenburghtasdomas: we also have a docker container with bundletester already installed, so you can, for example:19:42
tvansteenburghsudo docker run --rm --net=host \19:42
tvansteenburgh    -u ubuntu \19:42
tvansteenburgh    -v ${JUJU_HOME}:/home/ubuntu/.juju \19:42
tvansteenburgh    -t jujusolutions/charmbox:devel \19:42
tvansteenburgh    sudo bundletester -F -e $env -t $url -l DEBUG -v19:42
tasdomastvansteenburgh, interesting - where can I find that container?19:42
tvansteenburghjujusolutions/charmbox in the docker registry19:43
tvansteenburghtasdomas: you may also like lazyPower's new blog post on charm testing with drone (uses this same docker container) http://blog.dasroot.net/2015-continuous-integration-with-juju-and-drone-ci.html19:45
tasdomastvansteenburgh, ah, thanks19:45
lazyPowertasdomas: full rundown of context management with that container: http://blog.dasroot.net/2015-local-isolation-with-docker-and-juju.html19:47
lazyPowerbut thats extra credit :)19:47
tasdomaslazyPower, thanks ;-]19:57
tasdomaslazyPower, are unit tests for charm functionality required to get the charm approved?19:57
lazyPowerits best practice and will expedite any reviews19:57
lazyPowerbut its not strictly required, only preferred19:57
lazyPowerthe requirement is an amulet level integration test so we ensure it continues to deploy as the charm ages19:58
lazyPowerwhich involves standing up your charm, any ancilliary charms related to it, and inspecting data sent over the wire + host configuration. eg: if you alter networking settings, verify they did indeed happen.19:58
marcoceppitasdomas: unit tests are a super ++ in the review process, since it gives us an idea of how much coverage is done for the code, but like lazyPower said not required to be in the store20:02
tasdomasmarcoceppi, lazyPower: thanks20:05
xoritorthanks everyone20:35
thumpertvansteenburgh: hey there21:01
thumpertvansteenburgh: still around?21:01
tvansteenburghthumper: hey21:01
thumperhey21:01
thumperdo the python-jujuclient tests pass for you?21:02
tvansteenburghhee, hot topic!21:02
tvansteenburghi haven't tried, but whit just submitted an MP that allegedly fixes them21:02
tvansteenburghwanna review it? :P21:02
thumperI'll look too, but I need something to run it in21:03
tvansteenburghthumper: running tests now21:14
* beisner stands by waiting eagerly for rabbitmq-server next charm bare metal + lxc unbreak validation21:16
tvansteenburghthumper: 18 passed, 5 skipped21:18
tvansteenburghpy27 and 3421:19
whitthumper: ported them to py-test21:24
whityou still have to set a env var21:24
thumperwhit: is there instructions on how to run the tests anywhere?21:25
tvansteenburgh$ JUJU_TEST_ENV=local tox21:25
tvansteenburghthat's it21:25
tvansteenburghjust bootstrap local env21:25
whitthumper: should be in the readme iirc21:25
whitthumper: I've set the tests to skip some dodgy incomplete implementation and some tests with broken cleanups21:26
whitthumper: until the implementations and cleanups are complete21:26
whithazmat: do you have a commit waiting to be pushed somewhere for the facade stuff?21:27
tvansteenburghfacades are already in21:28
whittvansteenburgh: removed that print statement21:38
whittvansteenburgh: I marked several tests as skips because of what appeared to be incomplete implemenation that caused the tests to error21:39
whittvansteenburgh: but maybe there was some magic I missed21:39
whittvansteenburgh: I did add comment to the skips so it should be obvious21:40
whit<3 py.test21:40
tvansteenburghwhit: ack21:42
beisnerdear jamespage and/or gnuoy scrollback,21:45
beisnerbug 1475320 is sorted, new proposal ready for you21:46
mupBug #1475320: rmq next charm:  config-changed hook fails when deployed to lxc <openstack> <uosci> <rabbitmq-server (Juju Charms Collection):New> <https://launchpad.net/bugs/1475320>21:46
marcoceppibeisner: UHG22:02
beisnermarcoceppi, uhgtastic?22:03
marcoceppithat bug was kicking my ass22:03
beisnerditto22:04
beisnerlp:~1chb1n/charms/trusty/rabbitmq-server/fixup-configure-nodename222:04
marcoceppiI would have to ssh in, force kill rabbit, then service rabbitmq-server start, then resolved --retry22:04
beisner^ that should do it for now, until reviewed/landed22:04
marcoceppiglad this is going to be fixed!22:04
marcoceppishould be able to do a smoosh without any issues now22:04
beisneryes22:04
beisnerhappy smooshing, gotta run22:05
thumperwhit: where is the main code for python-jujuclient now?22:10
thumperwhit: I assumed it was LP22:10
thumperwhit: or is this fix just proposed ATM?22:11
thumperwhit: I hit problems trying to run tests with your branch22:19
thumperwhit: no idea what I'm doing wrong22:19
tvansteenburghthumper: sorry i saw your comment right after i merged it :/22:21
thumper:)22:21
tvansteenburghthumper: pip install -U tox22:21
thumpertvansteenburgh: I did that earlier22:21
* thumper tries again22:21
thumperI currently have 2.0.222:22
thumpertvansteenburgh:  E   EnvironmentNotBootstrapped: Environment "test" is not bootstrapped22:23
thumperlots of that22:23
tvansteenburghjuju bootstrap -e test22:23
thumperso I'm guessing it expects an environment called "test" to be available?22:23
thumpertvansteenburgh: should that exist first?22:23
tvansteenburghno you have to tell it the name of your juju env22:24
tvansteenburgh$ JUJU_TEST_ENV=local tox22:24
tvansteenburghbut you must bootstrap that env first22:24
thumpertvansteenburgh: would be nice to have that written down somewhere22:24
tvansteenburghagree, not user friendly22:24
thumpertvansteenburgh: however I thiink I have enough to get this working with JES now22:25
tvansteenburghthumper: i'll make a proper Makefile/readme when i run out of other things to do :D22:26
thumperhaha22:27
thumperyeah...22:27
thumperlike that'll happen22:27
tvansteenburghthumper: check the HACKING doc22:28
tvansteenburghturns out there are some instructions22:28
thumper:)22:29
=== menn0_ is now known as menn0

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!