/srv/irclogs.ubuntu.com/2015/01/06/#juju.txt

=== kadams54-away is now known as kadams54
=== kadams54 is now known as kadams54-away
X-RobAny way to change the default memory size of KVM guests when they're made?01:48
X-RobEverything seems to be being created with  512M or RAM no matter what I do01:48
sarnoldX-Rob: what's creating your kvm guests?01:50
X-Robsarnold: 'juju add-machine kvm:10 --constraints mem=8G'01:51
X-Robwhich various places seem to imply should work01:51
sarnoldX-Rob: interesting, I hadn't seen the kvm:10 thing before :) thanks01:51
X-Robhttps://bugs.launchpad.net/juju-core/+bug/139961301:52
mupBug #1399613: juju-core not using constraints when creating KVM  unit on maas machine <constraints> <kvm> <maas-provider> <juju-core:New> <https://launchpad.net/bugs/1399613>01:52
thumperX-Rob: hmm... almost certainly a bug01:52
thumperX-Rob: in that the requested constraints for the container aren't passed down to the host01:52
X-RobThat says it's my issue, but, it's linked to a totally different bug01:52
thumperum..01:52
thumperI'm pretty sure that memory constraints were tested when I wrote it...01:53
X-Robthumper: Well, any debugging I can do?01:53
thumperX-Rob: I agree that it should work01:53
sarnoldX-Rob: that bug looks like it describes your problem exactly, but the duplicate feels like a mistake to me01:54
X-RobAhha01:55
X-RobOK01:55
lazyPowersarnold: yeah that was supported before lxc:# :)01:55
sarnoldlazyPower: it was?? why does'nt anyone tell me these things :)01:55
lazyPowersarnold: because we stink at writing changelogs?01:55
sarnoldlooks like sinzui's not online, bugger, he might have had a good reason for duping them..01:56
sarnoldlazyPower: yeah, I might really enjoy using a kvm: option.. :)01:56
X-RobI can't seem to find where, if any, logs of this stuff would be01:56
lazyPowersarnold: whoa whoa buddy - lets not get carried away here. LXC is the hotness01:56
lazyPowerX-Rob: I think those bits are in the machine-0 log, but i might be mistaken. They coudl be going to stdout/err and by proxy discarded.01:57
X-RobLXC -is- the hotness, except, it doesn't work well with pacemaker and floating IP addresses.01:57
X-RoblazyPower: nah, nothing in machine-001:57
lazyPowerX-Rob: ah i was tongue in cheek'ing @ sarnold.01:57
thumperX-Rob: is your environment running?01:57
X-Robthumper: eeyup01:58
sarnoldlazyPower: heh, I spin up a dozen kvms all the time, {i386,amd64}x{lucid,precise,trusty,utopic,vivid,wet}, etc...01:58
sarnoldlazyPower: containers are neat enough but not sufficient for those kinds of tests :)01:58
lazyPowerthumper: are we capturing the lxc container / kvm creation logs? Its sounding like we arent.01:59
lazyPowerin which case i'm more than happy to file a bug on it01:59
thumperX-Rob, lazyPower: ok, this is going to give you a lot of info...01:59
thumperyou want to rack up the logging on a certain subset02:00
thumperso02:00
X-RobWoo, I love logs.02:00
thumperget the existing logging by running 'juju get-env logging-config'02:00
thumpertell me what you have02:00
lazyPower<root>=WARNING;unit=DEBUG02:00
X-Rob^^ same02:00
thumperok, do this:02:00
thumperjuju set-env 'logging-config=juju.container.kvm=TRACE'02:01
lazyPoweroh i love it02:01
lazyPowerthumper: <3 can we put that in the docs?02:01
thumperthis will capture all command line calls to the libvirt and their output02:01
thumperlazyPower: yes02:01
lazyPowerWOO02:01
sarnoldyay docs! :)02:02
thumperlazyPower: juju.container.lxc does the same for the lxc commands02:02
thumperX-Rob: this will tell us whether the 8G mem constraint is being passed to libvirt or not02:02
thumperif it is, it is a libvirt problem02:02
thumperif it isn't, it is a juju problem02:02
X-Robthumper: OK, lemme go do this thing.02:02
X-Robhttps://www.irccloud.com/pastebin/z0Bi2JHZ02:03
X-Robnow.. for the logs..02:03
X-Rob... um. OK, WHICH logs? on 9?02:04
X-Robyep, on 902:04
X-Rob2015-01-06 02:03:27 TRACE juju.container.kvm libvirt.go:35 uvt-kvm [create --log-console-output --user-data /var/lib/juju/containers/juju-machine-9-kvm-7/cloud-init --bridge juju-br0 --memory 512 --cpu 1 --disk 8 juju-machine-9-kvm-7 release=trusty arch=amd64]02:04
X-Robthumper: juju's fault.02:04
thumperX-Rob: add that to the bug please02:05
thumperI'll try to get someone to look at it02:05
* thumper complains that it used to work02:05
thumperactually02:05
thumpernot sure if I tested that particular use case02:05
thumperI tested kvm machines created for the local provider using memory constraints02:06
thumperprobably never worked...02:06
thumperoops02:06
thumpermy bad02:06
lazyPowerthumper: the bits to reset the logging to default?02:06
lazyPowerjuju set-env 'logging-config=<root>=WARN;unit=DEBUG' ?02:06
thumperjuju set-env 'logging-config=<root>=WARN'02:07
lazyPowerstellar02:07
lazyPowerThank you02:07
thumperlazyPower: the unit=DEBUG is added unless you explicitly turn it down02:07
thumperthis means that we always see hook output02:07
lazyPowerright02:07
thumperyou can turn it off by doing this:02:07
thumperjuju set-env 'logging-config=<root>=WARN;unit=WARN'02:07
thumperthat way your logs won't contain the hook output02:07
thumperlazyPower: levels are: TRACE, DEBUG, INFO, WARNING, ERROR02:08
thumperhighly recommended that you have at least WARNING02:08
thumperand anything under WARNING is likely to be developer related debugging and logging02:08
thumperso not necessarily intelligible02:08
thumperthe logging-config can be as long as you like02:09
X-RobI like logs.02:09
thumperjuju help logging02:09
X-Robbut.. not much use with this02:09
thumperLIKE A BOSS02:09
lazyPowerhttps://github.com/juju/docs/pull/21602:09
X-RobOMGDOCSWTF02:10
sarnoldlazyPower: <302:10
lazyPowerX-Rob: i'm like Tron - I fight for the users.02:10
sarnoldlazyPower: .. where do those logs go?02:10
lazyPowersarnold: in thumpers email, where they belong :P02:11
thumperall-machines.log02:11
sarnoldlazyPower: lol02:11
thumperwhich is looked at for 'juju debug-log'02:11
lazyPowerUpdating02:11
X-Robhttps://i.imgur.com/ZPHgGeu.jpg02:12
X-RobThat's my test environment, by the way02:12
sarnoldX-Rob: is that sixteen "blades" with two storage boxes?02:13
X-Robsarnold: yeah.02:13
lazyPowersarnold: whats your github user id? you're not sarnold @_@02:13
sarnoldX-Rob: and is that a volleyball in the back? :)02:13
X-Robsarnold: that's actually my gym 8)02:13
sarnoldlazyPower: "setharnold", I laughed at github and didn't move on it quickly enough...02:13
X-Robhttps://i.imgur.com/gn1aoRw.jpg02:14
sarnold"the whole point of git is that it is decentralized, who would want to put it all on a central server anyway?" ... heh02:14
X-Robmy exuse is that I don't use it anyway, and it's got aircond, AND I happen to have 2 15amp 240v circuits in there02:14
sarnoldX-Rob: sweet, complete with minisplit aircon! :)02:14
X-Rob(<-- au, we have 240v natively)02:15
sarnoldnice02:15
X-RobEbay, $2k02:15
thumperX-Rob: what are you testing for exactly?02:15
sarnoldX-Rob: wow, I thought the "au tax" would make that rig five times that...02:16
X-Robthumper: Well, the long term result is this stuff is going to be an openstack build for my minecraft server (I run mcau.org)02:16
X-Robsarnold: I did have to jump in my van and drive for 16 hours (each way) with 1 hour notice to my wife to go pick it up. The guy wanted it gone NOW02:16
sarnoldX-Rob: ooof :)02:17
X-Robthumper: the short term is me learning all about openstack, because fuck vmware. I'm a VCP, but I now (officially) hate them.02:17
sarnoldman australia is a big place.02:17
thumperVCP?02:17
thumpermy kids are now all into minecraft :)02:18
X-Robhttps://www.google.com.au/maps/dir/Gladstone+QLD/Caringbah+South+NSW/@-28.8680946,146.9265686,6z/data=!3m1!4b1!4m13!4m12!1m5!1m1!1s0x6bc27489ada17a29:0x500eef17f210e60!2m2!1d151.2597998!2d-23.8487083!1m5!1m1!1s0x6b12c78d09dddc5b:0x5017d6816334960!2m2!1d151.1213599!2d-34.055144402:18
X-Robthumper: vmware certified professional02:18
X-RobI did the course and stuff02:18
X-Robthumper: well, mcau will be MAAS+JUJU+OpenStack02:19
thumpernifty02:19
lazyPowersarnold: https://plus.google.com/100016143571682046224/posts/ZHG9ybVN3pR02:22
lazyPowerlots of activity on the post here - seems like one person was already +1 on this idea02:22
sarnoldX-Rob: long-ass drive, but for that kind of hardware at that price, probably makes sense...02:22
X-Robsarnold: yep, it was. It's kinda old, but, good enough for lots of small minecraft servers.02:23
lazyPowerthumper: did you see the minecraft post I made lastyear? Let me refresh you :D  https://plus.google.com/100016143571682046224/posts/PaVGh51FYCR02:23
sarnoldlazyPower: heh, I gave a +1 earlier.. is it mine? :)02:24
lazyPowersarnold: actually no - an indie security researcher02:24
sarnoldX-Rob: I'm eyeing some older server gear for home use myself, I'm more interested in a pile of storage than a bunch of vms though..02:24
sarnoldaha02:24
X-Robsarnold: it's really easy to get a pile of storage these days. ceph + a couple of cheap-arse motherboards with 6 or 8 SATA sockets.02:26
sarnoldX-Rob: I've thought a bit about going that route but think I'd rather get something like a supermicro or norco with 15~20 drive bays and do zfs instead...02:28
sarnoldX-Rob: a pal keeps suggesting getting a drive shelf instead and hook it up with external sas, but I think I'd rather avoid expanders ..02:29
sarnold.. of course, I also worry about the noise of going that route. a few desktops running ceph would probably be quieter.02:30
X-RobIck, I';m with you there, external sas is nothing but heartache02:31
X-Robsarnold: my original ceph cluster was two milk crates with motheboard|cardboard|pizza spacers|cardboard|sata disks02:31
X-Roband I cabled tied the psus to the outside02:32
sarnoldhahaha02:32
X-RobPROFESSIONAL AWW YES02:32
lazyPowerseems legit02:32
lazyPowerthe pizza expanders made it even more legit02:32
lazyPowers/expanders/spacers/02:32
sarnoldyeah I lost it at the pizza spacers :)02:32
X-Robheheh02:32
X-RobThey were needed, they gave the cardboard better structural integrity so it didn't collapse all over the motherboard02:33
sarnoldplus also, kept the pizz from getting the cardboard messy in the first place, amirite? :)02:33
X-Robheh. They weren't pizza boxes. That would have been the best idea, now I think about it02:35
catbus1Hi, I have used juju to deploy some charms, my understanding is juju would cache charms, where can I find the cached charms?02:55
sarnoldcatbus1: iirc, in ~/.juju/ there's a directory of cached charms..02:56
catbus1sarnold: I do see them in ~/.juju/, there is environments/, environments.yaml and ssh/.02:57
catbus1I don't see a list of charms I deployed in those 2 directories.02:58
sarnoldcatbus1: look under environments/, it's probably in a subdirectory named after the environment02:58
catbus1sarnold: there is just maas.jenv in environments/.02:58
lazyPowercatbus1: is this a local environment, or a cloud environment?02:59
lazyPowercatbus1: if its ac loud environment,t hose charms are then cached on the stateserver02:59
catbus1lazyPower: a local environment.02:59
lazyPowercatbus1: then sarnold is correct, you should have a local charm cache in ~/.juju/local/02:59
lazyPowerthumper: has anymore thoughts/discussions happened wrt making the stateserver an honest container on the local provider?03:00
thumperlazyPower: yeah... I'm considering doing Fun Fridays :-)03:00
thumperlazyPower: got to get through MESS first03:00
thumperor at least happy it is going in the righ tway03:00
thumperlazyPower: I know what I want to do...03:01
lazyPoweroh agreed, i wasnt pressuring by any means, just curious if we're still headed down that path in the future.03:01
thumperlazyPower: and it doesn't seem too hard03:01
thumperoh yes03:01
thumperI want it03:01
thumperand I want it soon03:01
lazyPowerbecause to be honest, it sounds really compelling. And i'd love to see that be the case.03:01
thumperso I can test MESS and HA using a local environment03:01
lazyPowerMESS?03:01
lazyPowerhaha03:01
lazyPower<303:01
thumpermulti environment state server03:01
lazyPowerOH!03:01
thumpermark hates it03:01
thumperthe name that is03:01
thumperbut I think it is cute03:01
thumperI'm just making a mess03:01
lazyPoweri just laughed so hard i got light headed03:02
lazyPowerhaha03:02
sarnoldhehe03:02
thumperlazyPower: so, in something completely unrelated03:02
=== negronjl is now known as negronjl-afk
* lazyPower smells a work item coming out of this aside03:02
thumperlazyPower: I want python-django to support django 1.7, virtual environments, and python 303:02
lazyPowercalled it03:02
thumperI've hacked my local python-django so I can deploy my django 1.7 app03:03
thumperand I wanted to use python 303:03
thumperbut gave up03:03
thumperso I'm using 2.7 and django 1.703:03
lazyPowerthat should be fairly simple. Python3 exists in the repositories03:03
lazyPoweryou should be able to warden all of that with venv's03:03
thumperyeah, but python-django doesn't use virtual envs03:03
bladernr`lazyPower: catbus was incorrect, we're using juju via MAAS03:03
lazyPowerbladernr`: ok - that makes more sense why it wasn't showing up in ~/.juju/local03:04
bladernr`not the local provider.  so you're saing the charms would be cached on the bootstrap node?03:04
lazyPowerbladernr`: the cached charms will be on your state server on the MAAS machine.03:04
lazyPowercorrect03:04
lazyPowerthumper: are those charms getting stuffed in gridfs or are they on disk on the state server? this is where i get fuzzy03:04
thumperum...03:04
thumperdeployed charms?03:04
lazyPowerand wait what? django doesn't support venvs?03:04
thumperlazyPower: nope03:04
thumperhang on03:05
lazyPoweras in the framework or the charm?03:05
thumperdjango does03:05
thumperthe charm doesn03:05
thumpertt03:05
lazyPowerah ok, the charm could be modified i would think to be a simple task03:05
lazyPowerinspect for a .venv file/dir, if so - route throught hat. using something like virtualenvwrapper03:05
thumperlazyPower: if you make it work, I'll test it03:05
lazyPowerthumper: if you file bugs, i'll put it on the list03:05
thumperha03:05
thumperfair call03:05
lazyPowerforewarning though - i'm not a django dev - so i might make a bigger mess than MESS03:06
lazyPowerso i imagine i'll be pestering you for pairing03:06
bladernr`lazyPower: thanks, /var/lib/juju/charmcache finally found it03:06
lazyPowerbladernr`: awesome! glad you got it sorted :)03:06
thumperlazyPower: perhaps we can scratch each other's back...03:09
thumperlazyPower: I may submit a patch or two to python-django03:09
lazyPowerthumper: go on...03:09
thumpermostly to clean it up03:09
lazyPowerI'd like that, and i'm sure the maintainer would too03:09
thumperit is a bit messy,03:09
lazyPoweryeah, last time i poked around in there i had @_@ these eyes03:09
lazyPowerits one of those inhereted charms from when the store was the wild west03:09
thumperbut if I just make multiple small patches03:10
lazyPowerwe're cleaning quite a bit of it up though - did you see the email that hit the last pre-break about the 30 or some odd charms slated for moving to ~unmaintained?03:10
thumperI'm a little frustrated that it doesn't support django 1.7 even though it has been released for 10 months03:10
thumperlazyPower: I don't follow the list too closely for those things03:10
lazyPowertsk tsk, but you could say the same to me for not diving through the -dev list for all your features that are landing.03:11
lazyPowerokay, belated new years resolution - i'll follow juju-hackers 10% more closely than i currently do.03:11
lazyPowerwhich will put my current level of monitoring at 10%03:11
X-Robheh03:13
X-RobActually, lazyPower, 10% more of 0% is still 0%.03:13
thumperlazyPower: and one for me, you'll get a patch for python-django in the next few weeks03:13
sarnold1.1 * 0.0 == 0.0 ... success!03:13
X-Robsnap, sarnold03:13
sarnoldX-Rob: your message got to me first though, you beat me fair and square :)03:13
X-RobSo the company I work for has been bought by another company and they want me to use exchange.03:14
lazyPowersarnold X-Rob: touche - however 1.1 * 1 = 1.103:14
* X-Rob grumbles.03:14
lazyPowerreading juju-hackers list is kind of like checking the fridge, looking around for something because you know you'r ehungry - closing the door and coming back 15 minutes later to repeat the process.03:15
lazyPoweryou shuffle around the condiments a bit, figure there's nothing in there to munch on and order a pizza03:15
sarnoldX-Rob: suuuuuck03:15
lazyPowercompletely missing the cheese cake in the back03:15
skaylazyPower: just saw you were talking about the python-django charm... and missed hte conversation.05:51
skaylazyPower: will the pure-python branch be merged soon? the ci tests and stuff got cleaned up05:52
skayI have some patches I'd like submit once that goes05:52
marcoceppilazyPower: what's the deal with this? https://bugs.launchpad.net/charms/+bug/137300612:52
mupBug #1373006: Hortonworks hdp 2.1 Storm+ZK bundle <Juju Charms Collection:Fix Committed> <https://launchpad.net/bugs/1373006>12:52
=== fuzzy_ is now known as Fuzai
=== roadmr is now known as roadmr_afk
=== kadams54 is now known as kadams54-away
=== kadams54-away is now known as kadams54
=== roadmr_afk is now known as roadmr
=== kadams54 is now known as kadams54-away
=== kadams54-away is now known as kadams54
skayany chance of getting pure-python python-django MR pushed soon? I would like to start putting together some MR and that has a lot of changes in it15:39
skayI'll be in merge hell if I don't wait15:40
marcoceppiskay: I'm not sure the state of the review, but you're welcome to do a review of it yourself if it's waiting for more eyes15:41
skaymarcoceppi: I am relatively new to juju and not sure my review would be very insightful15:42
marcoceppiskay: well, we always welcome community reviewers. If you're looking to get mroe in to charming it's a great way to look at what other people are tyring to do. If you find something comment on it! If you don't that's fine too, it'll still go through the review of a charmer before making it's way to the store15:43
skayit's been reviewed by some charmers already and has passed hte ci tests15:43
skaythe commiter made fixes based on the review cmments15:44
marcoceppisounds like it's just stuck waiting finial approval then15:44
marcoceppiI'll see if I can take a look at it today, the queue is mighty big these days :\15:44
skayis there a way for me to tell?15:44
skayat this point, I would like to provide suggestions by submitting some small merge requests rather than holding up the MR15:45
marcoceppihttp://review.juju.solutions/15:45
marcoceppiis where it is in the queue, it's pretty close to the top now15:45
skaythanks15:46
marcoceppiyeah, this is just waiting for a merge, I'll take a few mins to finalize and merge it today15:46
skay\o/15:46
skaybtw, I'm getting an error when I try to login to the site15:48
marcoceppiskay: what's the error?15:49
skaymarcoceppi: The server encountered an unexpected internal server error15:50
roadmrodd... I was able to log in just fine15:50
skayit hates me15:50
* skay sobs15:50
marcoceppimyself as well, I wonder if this is because of some weird issue with code15:51
roadmrskay: did you tell it your name? (the openid interstitial that lets you choose what to share)15:51
skayI was curious if I could log in and trigger a new jenkins job15:51
marcoceppiskay: ah, you can't, that's reserved only for charmers15:51
skayroadmr: yep, after it fafiled I went and gave it all the permissions15:51
roadmrI noticed nothing was ticked, so I ticked everything before going on15:51
roadmrskay: ok, so that's not it15:51
marcoceppithe internal server error is just the reviewq not having much good code around login since it was kind of a last min thought15:51
marcoceppiskay: try again?15:52
marcoceppiyeah, I see why it's failing15:53
marcoceppiit's a known issue15:53
skayokiedokie. not urgent15:53
marcoceppiskay: once you get a review in the review queue log in will start working for you15:54
=== negronjl-afk is now known as negronjl
=== designate is now known as designated
josehey tvansteenburgh, have a min for a PM?17:15
tvansteenburghjose: yup17:15
wesleymasonAnyone know if a lint with warnings will stop a charm getting promulgated into the store?17:19
marcoceppiwesleymason: a warning, yes. though depends on the warning17:21
marcoceppiwhich are you getting?17:21
=== kadams54 is now known as kadams54-away
=== roadmr is now known as roadmr_afk
=== kadams54-away is now known as kadams54
jcastrocory_fu, what's the closest thing you have to a homepage for DHX?18:03
cory_fujcastro: I guess the blog post: http://blog.juju.solutions/cloud/juju/2014/11/26/debug-hooks-ext-plugin.html18:03
cory_fuWhy do you ask?18:04
=== kadams54 is now known as kadams54-away
jcastroI am giving a presentation to the SA guys in like 2 weeks18:04
jcastroand I want to go over DHX18:04
cory_fuAh, cool18:04
jcastrobecause it seems they would benefit a bunch from knowing about it18:05
=== roadmr_afk is now known as roadmr
=== makyo_ is now known as Makyo
skayhey, someone is giving a juju talk at lcaconf. my spouse is on the video team for the conf19:17
skayI wish I could be there. I want to go to lcaconf one day19:18
skaythis year's is in Auckland, which would be so cool to visit19:18
X-Roblca is pretty awesome19:53
=== kadams54 is now known as kadams54-away
=== kadams54-away is now known as kadams54
wesleymasonmarcoceppi: start/stop hooks missing, it's a subordinate with a binary called on demand, so there's no service to start/stop..I know I can just add blank hooks, but feels messy given they're unnecessary.21:19
=== kadams54 is now known as kadams54-away
marcoceppithat hasnt been a warning in s long time wesleymason21:20
marcoceppiwesleymason: do you have the latest charm-tools from ppa:juju/stable ?21:20
wesleymasonmarcoceppi: nah, from universe21:22
marcoceppiwesleymason: yeah, sadly that version of charm-tools is a whole rewrite behind21:23
marcoceppiwe'll have it updated in vivid and in releases going forward21:23
* wesleymason ignores the warnings then21:23
marcoceppiuntil then ppa:juju/stable is the best place to get the most recent version21:23
wesleymasonmarcoceppi: ta21:23
marcoceppiwesleymason: you can also install it in a python virtualenv from pip if you're so inclined21:24
wesleymasonmarcoceppi: avoiding using latest until juju is upgraded in prod by IS21:24
marcoceppiwesleymason: ack, I can make a charm-tools only ppa if that helps21:24
wesleymasonmarcoceppi: tbf I could just pin it, might just install in the virtualenv anyway as using one for tests21:25
=== kadams54-away is now known as kadams54
marcoceppiwesleymason: virtualenv works pretty well and is the least amount of effort betweent the two of us21:25
wesleymasonmarcoceppi: ta muchly21:28
marcoceppinp21:28
=== kadams54 is now known as kadams54-away
=== kadams54 is now known as kadams54-away
=== kadams54-away is now known as kadams54
noise][lazyPower: https://bugs.launchpad.net/charms/+source/haproxy/+bug/1407815 - I finally tracked down the cause of that relation-changed bug I mentioned yesterday23:19
mupBug #1407815: adding a 2nd website unit fails to call reverseproxy-relation-changed <haproxy (Juju Charms Collection):New> <https://launchpad.net/bugs/1407815>23:19
lazyPowernoise][: looking23:19
noise][lazyPower: now I need to go look through the code to figure out __why__23:20
lazyPowernoise][: weird, byproduct of a debug setting?23:20
noise][right??23:20
lazyPowerthat smells23:20
=== kadams54 is now known as kadams54-away
noise][nearly drove me insane23:20
* lazyPower identifys a code smell23:20
lazyPowernoise][: well i'm glad you've got more info to go on. I'm subbing to this bug to monitor your progress23:20
lazyPowerif you need any help feel free to reach out. I'm not an expert with the charm but i can lend a second pair of eyes23:20
noise][i was just about to chuck my computer in the bin!23:21
noise][lazyPower: cool, shouldn't be too hard to find the root cause now23:21

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!