/srv/irclogs.ubuntu.com/2012/01/27/#juju.txt

adam_gm_3: oh jeez, ya..00:01
m_3there's wiring inside for the EnvironmentConfig to take another path... but the cli itself doesn't pass it thru00:06
_mup_Bug #922398 was filed: cli should accept alternate config file path <juju:New> < https://launchpad.net/bugs/922398 >01:09
=== jelmer_ is now known as jelmer
niemeyerGood Friday, Magicians12:35
jorgeHi folks! I would like some help with the same problem from yesterday: juju can't connect to instances when i call juju status. the complete description of the problem is here http://pastebin.com/KkTZkEjv ... thanks.12:44
niemeyerjorge: WOw, that's an awesome debugging session, thanks13:02
niemeyerjorge: So if you actually connect to the host, does it close on you like your verification suggests?13:03
jorgeyou're welcome. i'm going to try with the ubuntu user and other... just a moment.13:05
niemeyerjorge: There's another trick worth trying to investigate this as well13:07
niemeyerjorge: We have a little-known "open-tunnel" commnad13:07
niemeyerjorge: You can use it by opening a separate terminal, and running the command "juju open-tunnel"13:08
niemeyerjorge: This will open a connection to the host, and block13:08
niemeyerjorge: Then, on your first terminal, try running "juju status"13:08
niemeyerjorge: It should tunnel the command over the ssh session from the blocked open-tunnel command13:08
jorgeniemeyer: this way worked! but, something strange still occurs. The first attempt get error and the connection just is established on the second attempt.13:19
niemeyerjorge: There's something really bizarre going on there13:19
niemeyerjorge: Do you have anything in the logs of the server saying why sshd is closing the connection abruptly?13:20
jorgethe first attempt is always closed. the second works (using open-tunnel). really strange!13:20
niemeyerjorge: Sounds like a networking issue13:20
jorgeno, just these lines indicating opened session and the next second, closed13:20
niemeyerjorge: Hmm13:21
niemeyerjorge: Try to ssh onto the machine13:23
niemeyerjorge: juju ssh 013:23
niemeyer(that's a zero at the end)13:23
jorgeops... looks like it was just luck ... now i'm getting the same error with open-tunnel13:24
niemeyerWith the open-tunnel command live13:24
niemeyerOk13:24
niemeyerSo, close the open-tunnel13:24
niemeyerand try to ssh onto it using "juju ssh 0"13:24
jorgei'm going to try that when the open-tunnel come back hehe.13:24
jorgecurious, if I leave the machine idle for a while and test the command, works. after the first attempt, stop work.13:25
niemeyerjorge: It really looks like there's something weird going on with the machine/networking there, but I'm not sure what13:31
niemeyerjorge: Have you tried to juju ssh 0?13:31
jorgehum, juju ssh 0 not working...13:32
niemeyerjorge: What happens?13:33
jorgethe same error. i think because open-tunnel is not alive. i'm trying to restablish it13:34
niemeyerjorge: Nope13:35
niemeyerjorge: Please forget open-tunnel for the moment13:36
niemeyerjorge: It's unrelated to the problem13:36
niemeyerjorge: What is the error message you get?13:36
jorgeERROR SSH forwarding error: bind: Cannot assign requested address13:37
niemeyerjorge: Is that in the first call, or in the second one?13:40
niemeyerI suspect the TCP port is simply lingering13:41
niemeyerjorge: Do you have lsof installed?13:41
jorgesecond call13:42
jorgeyes, i do13:42
niemeyerjorge: I think I know what's up13:42
niemeyerjorge: Your kernel is likely configured to linger TCP ports in a different way13:42
niemeyerjorge: Are you using juju from source or from the package?13:43
jorgei don't know about 'lingering' ports... juju was instaled from ubuntu repo, 11.10 oneiric13:44
niemeyerjorge: Ok13:44
jorgejuju 0.5+bzr398-0ubuntu113:44
jorgeLinux  3.0.0-12-server #20-Ubuntu SMP Fri Oct 7 16:36:30 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux13:45
jorgedefault from ubuntu server 11.1013:45
niemeyerjorge: Please run "dpkg -L juju | grep 'state\/utils.py'"13:45
niemeyerjorge: and open this file for editing as root13:45
jorgehey, i'll be back in 1 hour... sorry...13:46
jorgeand i'll try this13:46
jorgethanks13:46
niemeyerjorge: Have a good lunch13:46
robbiewcan anyone tell me why I have a virbr0 *and* lxcbr0 interface now? (running precise)14:57
robbiewis that related to juju local provider somehow?14:57
m_3robbiew: juju's still using virbr0 (installed from libvirt)... I assume the other's newly installed along with precise lxc15:04
m_3they've both got their own little dnsmasqs tho15:05
robbiewinteresting15:05
robbiewseems if lxc now provides a device, we could drop virbr0, no?15:09
robbiewjust seems to be a waste of resources to have both virbr0 and lxcbr015:09
robbiewbcsaller:  hazmat: ^ ?15:10
hazmatrobbiew, precise lxc includes its own bridge?15:11
robbiewI suppose users can easily disable lxcbr0 in /etc/defaults/lxc , so maybe we have the juju package do that when installed15:11
robbiewhazmat: yeah15:11
robbiewhazmat: hallyn said it just recently landed in precise15:11
hazmatrobbiew, cool, the problem is backwards compatibility, i'll try it out.. but for now i'd rather just leave it.. unless its a problem/conflict.15:13
* hazmat fires up a precise instance15:14
robbiewjust thinking folks will get a bit pissy when they see a virbr0 and lxcbr0 created....waste of resources (though small)15:15
robbieweh...I guess it's no huge deal...just surprised me15:19
hazmatrobbiew, we shouldn't really be installing libvirt-d and lxc as recommends15:20
hazmatthey only apply to local development15:20
hazmatinstalling those packages on orchestra or ec2 machines doesn't serve any purpose15:21
hazmatlocal provider usage already checks and warns appropriately if run without required packages15:21
hazmatSpamapS, ^15:21
m_3the 'juju-origin: distro15:27
m_3seems to be the default for oneiric... and it's _old_15:28
m_3is there any way to bump that up?  or do we always need to use 'ppa' for oneiric?15:28
m_3my units are going straight to 'start_error' with version 39815:29
SpamapShazmat: we should be turning off recommends when juju installs itself actually15:41
SpamapShazmat: the idea behind recommends is that if you let your machine install recommends, as most Ubuntu users do, you get the full functionality of the package.15:42
SpamapShazmat: so for automated things, usually --no-install-recommends is advisable15:43
SpamapSin fact we should probably make most charms do that.15:43
m_3it'd be strange to have different default behavior for apt in the charm -vs- laptop... it should still be an explicit --no-install-recommends15:44
SpamapSm_3: agreed, I'm suggesting we should consider making that the recommended way to call apt15:53
bacthis page needs to be updated wrt 'charm' vs 'charms' in lp.  the bzr examples fail if you use 'charm'.  should i file a bug, if so against what?15:59
bachttps://juju.ubuntu.com/Charms16:00
SpamapSbac: done, thanks!16:02
bacgreat, thanks SpamapS16:10
jorgeHi all, i'm back.16:23
jorgei make a loop to detect what command is juju calling.16:24
jorgewhile [ 1 ]; do ps aux | grep ssh | grep juju ; ls /root/.juju/ssh/ ; sleep 1; done16:24
jorgeso, I call juju status or juju open-tunnel16:24
jorgethe result is the following process16:25
jorgeroot     29772  0.2  0.1  41176  2712 pts/0    S+   14:22   0:00 ssh -T -o ControlPath /root/.juju/ssh/master-%r@%h:%p -o ControlMaster auto -o PasswordAuthentication no -Llocalhost:33511:localhost:2181 ubuntu@172.16.0.216:25
jorgeif i try to run this command on terminal i get command-line line 0: Missing argument.16:26
jorgeSo, when juju tries to connect on localhost in a local port that should be forward to 2181 of instance, the problem occurs.16:28
jorgeif this command is not successful the local port doesn't open ... right? could this be the problem?16:29
SpamapSjorge: right16:38
SpamapSjorge: for future reference, btw, you can use 'strace -e trace=exec -f juju status'16:38
jorgegreat16:39
jorgetrace=exec works on linux? here i get one error. looking at the man page, there is no exec parameter for trace=16:42
jorge-e trace=process :)16:45
jorgeexecve("/usr/bin/ssh", ["ssh", "-T", "-o", "ControlPath /root/.juju/ssh/mast"..., "-o", "ControlMaster no", "-o", "PasswordAuthentication no", "-Llocalhost:55700:localhost:2181", "ubuntu@172.16.0.2"], [/* 36 vars */]) = 016:47
niemeyerjorge: Ok, should we continue from where we stopped?16:48
jorgeyes16:49
jorgeroot@sold016:~/.juju/ssh# dpkg -L juju | grep 'state\/utils.py' /usr/share/pyshared/juju/state/utils.py /usr/lib/python2.7/dist-packages/juju/state/utils.py /usr/lib/python2.6/dist-packages/juju/state/utils.py16:52
niemeyerjorge: So please do the suggested command, and open the file in your editor as root16:52
niemeyerjorge: Then, find the get_open_port function16:52
niemeyerjorge: Right above the line temp_sock = socket.socket(...)16:53
niemeyerjorge: Enter this, on the same indentation:16:53
niemeyer    import random; return random.randint(50000, 60000)16:53
niemeyerjorge: Then, try to run juju again, and let me know please16:53
niemeyerI'll step out to get a cup of coffee meanwhile.. brb16:54
jorgeirght16:54
jorgethe first time I ran juju status worked, now getting the same error.17:00
niemeyerjorge: Please run "juju ssh 0" for testing17:08
niemeyerjorge: You're having some fundamental issues there that don't even involve juju itself.. it's failing well before getting into more interesting logic17:09
jorgewell, the first time I ran juju ssh 0 worked. So i ran again and get the error.17:11
jorge:/17:11
jorgeyes, there is some network problem ...17:12
SpamapSjorge: is there any way you can run juju *not* on your openstack network controller?17:13
jorgeI tried from another hosts but it didn't work because the difference between versions.17:15
jorgeBut, I've won a new host just now! I need to test it ... So, i'm going to install the same version and try juju on it17:16
niemeyerjorge: Can you please run 5 times in a row and paste the output for all of them?17:16
jorgewhen I do that I come back to report here.17:16
jorgeyes17:16
jorgeany of the commands? juju ssh 0 or juju status?17:17
niemeyerjorge: juju ssh 017:19
niemeyerjorge: Is Embrapa deploying OpenStack internally?17:19
jorgeyes, this is a test of openstack ... i want to run some other scenarios, simulations etc17:20
niemeyerjorge: Very nice17:21
niemeyerjorge: Glad to see you guys are putting juju to good use there17:21
niemeyerjorge: Please step by if you have any other issues17:21
jorgedo you know embrapa?17:21
niemeyerjorge: Yeah, my Dad worked there for most his life until retiring17:21
jorgeI'm trying to use juju to create (or use charms) aiming to set up clusters quickly.17:22
jorgeniemeyer: cool!! so, you are from Brazil?17:23
niemeyerjorge: Cool, that's something it does well :)17:23
niemeyerjorge: Yep17:23
jcastroThis charm could use a review: https://bugs.launchpad.net/charms/+bug/91990717:45
_mup_Bug #919907: new-charm salt-master <new-charm> <Juju Charms Collection:New> < https://launchpad.net/bugs/919907 >17:45
jcastrohttps://bugs.launchpad.net/charms/+bug/91880317:46
_mup_Bug #918803: charm for python-oops-tools <new-charm> <Juju Charms Collection:New> < https://launchpad.net/bugs/918803 >17:46
jcastrothis one too!17:46
niemeyerHmm, curious18:06
niemeyerAha, ok..18:06
niemeyerhttps://code.launchpad.net/~yellow/charms/oneiric/buildbot-master/trunk18:07
niemeyerThere are branches with no revisions.. that's ok I suppose18:07
SpamapSWeird.18:07
niemeyerSpamapS: Hmm18:24
niemeyerNo, it's working.. I'm clearly doing something silly here18:25
niemeyerAh, and I found what it is18:26
adam_ghazmat: hey, i had a chance to point juju at openstack essex, seemed to have bootstrapped just fine.18:32
hazmatadam_g, good to know, thanks18:33
adam_gif we start testing a cloud image on this openstack CI stuff instead of ttylinux, ill probably add at least a 'juju bootstrap' test of some kind18:38
andrewsmedinasomeone can explain me what does this juju code? http://bazaar.launchpad.net/~juju/juju/trunk/view/head:/juju/providers/ec2/launch.py#L9119:00
SpamapSandrewsmedina: the comment above it is accurate as far as I can read it19:18
andrewsmedinaSpamapS: it's not works with my openstack setup =/19:21
SpamapSandrewsmedina: what version of openstack?19:23
andrewsmedinaSpamapS: diablo19:24
SpamapSandrewsmedina: we test against diablo daily.. should work19:24
andrewsmedinaSpamapS: here it's returns a unexpected error in ec2 client19:28
andrewsmedinashould be a problem in my openstack setup.19:28
andrewsmedinaI don't know where19:28
SpamapSandrewsmedina: what version of Ubuntu are you running juju on?19:31
andrewsmedina11-1019:31
SpamapSandrewsmedina: ok, that should be fine. Hrm.19:31
andrewsmedinaSpamapS: how you install your openstack setup?19:32
SpamapSandrewsmedina: do you have the juju from oneiric, or the one from our PPA?19:32
SpamapSandrewsmedina: I'm not sure, our IS team set it up for us.19:32
andrewsmedinaSpamapS: from your PPA19:33
SpamapSandrewsmedina: maybe try updating openstack from oneiric-proposed19:34
SpamapSandrewsmedina: *lots* of fixes19:34
SpamapSpython-nova | 2011.3+git20111117-0ubuntu1 | oneiric-proposed | all19:34
andrewsmedinaok19:35
andrewsmedinaI installed openstack from source19:35
SpamapSuh19:36
SpamapSwhy?19:36
andrewsmedinaSpamapS: I create a script like devstack19:41
SpamapSandrewsmedina: ah ok cool19:42
hazmatandrewsmedina, the fix for that might not have made it to the diablo release tarballs,  there was an error with the internal group authorization previously in ostack, but i thought it was fixed for diablo though20:20
jcastroSpamapS: chords, strings, we brings .... charm review pls?20:37
=== avoine_ is now known as avoine
gary_posterjcastro, hey.  I'm on a squad from the LP team using juju for a project.  We are writing a buildbot master recipe and a buildbot slave recipe ATM.  We have an internal review process, but we don't know juju well enough to be competent reviewers.  We want to get better at writing and reviewing.  Could we have a juju review mentor--someone who looks over our internal reviews of our charm branches to improve both the reviews an21:11
gary_posterd the branches?  If so...how would we set that up? :-)21:11
jcastroabsolutely21:17
jcastrom_3: SpamapS: marcoceppi:  should we just have them tag their branches/bugs with new-charm and ping pong back and forth?21:18
m_3jcastro: yeah21:19
m_3jcastro gary_poster: is there any reason for those to be separate?21:19
m_3those charms should eventually be pushed to lp:charms too I assume?21:20
gary_posterm_3 definitely21:20
gary_posterm_3, jcastro, we'll use the new-charm tag and look forward to your help.  Thank you!21:21
m_3ok, then yeah... charm review queue = { charm bugs, with 'new-charm' tag, attached branch, and New/FixCommitted status }21:21
m_3you might have to kick some of us to get through reviews a little faster :)21:22
gary_posterm_3, heh, cool, understood.  Those three people jcastro mentioned are the ones to kick?  and charm bugs, I assume you mean https://bugs.launchpad.net/charms ?21:24
m_3gary_poster: anyone in ~charmers can review, but most are done by marcoceppi, nijaba, SpamapS, and myself.  yes, bugs.lp.net/charms21:27
gary_posterawesome, thanks again m_321:27
m_3sure thank21:27
m_3s/thank/thang/21:27
gary_posterheh21:27
SpamapSjcastro: reviewing charms now, had to triage bugs this morning. :)22:18
SpamapSjcastro: cuase charming is life, and life is charming22:18
jcastro<322:20
SpamapSheh.. salt calls servers minions22:31
SpamapS+1 for mirth22:31
SpamapShrm.. people keep making the mistake of starting their charm as a subdir of the bzr branch, rather than just one branch for the whole charm22:31
* SpamapS wonders if there was a recent change that is leading people to this22:32
m_3all of our charm dev docs/presentations should really assume they've never heard of bzr22:35
m_3start from scratch22:36
SpamapShah, are you suggesting that bzr is obscure? ;)22:38
m_3I'd never suggest such a thing22:40
m_3rather that our user-base is so broad22:41
m_3:)22:41
SpamapSjuju, user base, SOOOO BIG22:41
SpamapSHUGE22:41
SpamapSbzr user base, smarr22:42
SpamapS</southpark>22:42
m_3really though... the concept of putting even configuration under revision control isn't all that new22:42
m_3sorry... logic barf... isn't all that old22:43
SpamapStrue22:43
SpamapSI wanted to make /etc a davfs mount of an SVN server on our servers back when I was wild and crazy and didn't know better....22:43
m_3ha!22:43
SpamapSI would have gotten away with it if it hadn't been for you meddling (git) kids22:44
m_3I still start long-lived hw and libvirt instances with (cd /etc; git init; git commit -m'initial revision')22:44
m_3itching to move them to juju22:46
m_3actually excited about the testing stack22:46
m_3learned a bunch of lessons about longer-term -vs- development stacks already!22:47
m_3really love to choose instance size per service... that bootstrap node can get expensive otherwise!22:49
SpamapSm_3: yeah, I'd like to have the option to consider the bootstrap node as available for deploys too22:50
SpamapSm_3: there's always my old branch which adds --machine to deploy and add-unit.. ;)22:50
SpamapSProbably out of date now22:50
m_3right22:50
m_3398's not working with 447 clients... I'd imagine your branch pre-dates that22:51
SpamapSm_3: did you report that as a bug yet? ubuntu-bug juju is all you need to do on the box with r398 on it22:51
m_3haven't taken time to really isolate it... definitely happens with O running O, and P running P in local and O running O in ec222:52
m_3there's a problem with config-get coming up empty (not even defaults)22:53
m_3but don't know what else is going on22:53
m_3juju-origin: ppa works around it22:54
SpamapSWe really need to fix the bug where you can't upgrade-charm if config-changed or install or upgrade-charm has had an error.22:56
SpamapSMy dev method for the mediagoblin charm has been juju scp'ing the new, fixed script up, --retry, then upgrade-charm again.22:57
SpamapSwhich is t3h lame22:57
m_3yup22:59
SpamapSWow the error message on ambiguous relations is *SO much better* now23:00
m_3so I'm about to do something... and I want you to talk me out of it23:00
SpamapS2012-01-27 14:58:12,782 ERROR Ambiguous relation 'oops-tools postgresql'; could refer to: 'oops-tools:db postgresql:db' (pgsql client / pgsql server) 'oops-tools:db postgresql:db-admin' (pgsql client / pgsql server)23:00
SpamapSm_3: DO IT YOU KNOW YOU WANT TO23:00
SpamapSerr.. I mean.. what is it?23:00
m_3config-changed for jenkins just calls install again... I wanna catch certain config changes, and then pass it on to install again23:00
m_3the only way to do that is to save out the config set every time the hook is called and compare changes23:01
m_3wow!  yeah, that message is just nice... makes me want to go relate things just to see the message23:02
SpamapSm_3: config-get is always available, so why even have special logic in config-changed if you're just going to run the installer again?23:04
m_3only run it conditially based on which config params are passed23:05
m_3that's what I want to know... which config params were passed to this particular call of config-changed23:05
SpamapSah23:06
SpamapSyou want to use it to kick things off. :)23:06
m_3well... and change the config without a reinstall23:06
m_3but yes, in general, I like the ability to use 'juju set' to control the service23:06
m_3add new test jobs23:07
m_3kick off tests23:07
m_3etc23:07
SpamapSjuju set charmtester runstuff=(x=$(juju get charmtester runstuff); echo $(($x+1))))  ? ;)23:07
m_3juju set charmtester add_branch=lp:my/new/charm23:07
m_3ha!23:07
SpamapSIf only it could feed data back..23:08
m_3you laugh... but there might be legitimate use for that in some relation vars23:08
m_3(varnish)23:08
m_3I totally want a 'juju get' too23:09
m_3right, like your 'runstuff' above23:11
SpamapSjuju get exists!!23:12
m_3is it coupled to config-set from within the hooks?23:13
m_3what's it getting?23:14
* m_3 digs through the code23:14
SpamapSthe values23:14
SpamapSso you can retrieve what the current value of a config option is23:14
m_3cool... never used that one before23:16
SpamapSfairly recent feature23:16
m_3we're missing config-set tho23:17
m_3_that_'d be a really useful combo23:17
SpamapSYeah actually it would23:18
SpamapSHey I had a crazy idea while walking to/from lunch23:18
m_3I could get my api_token _back_ instead of forcing one forward23:18
SpamapSwhat if zookeeper just ran on client machines..and juju open-tunnel opened a tunnel to the machines it was managing, for the agent to connect *back* through23:19
SpamapSkind of like the local provider works.. but not local. :)23:20
_mup_Bug #922900 was filed: config-set callable from hooks <juju:New> < https://launchpad.net/bugs/922900 >23:22
m_3bug filed23:22
SpamapShttp://ec2-184-73-125-5.compute-1.amazonaws.com/23:22
SpamapSoops-tools23:23
SpamapS*ALMOST* ready for promulgation23:23
m_3hmmm... my charmtester environment of xlarge's would be cheaper23:23
m_3actually when you think about it, that's a nice thing about the architecture as it stands... the zk node really could be moved around23:24
m_3might be some security-group nonsense23:24
m_3and you'd ack the fact that there might be a perf hit with really large infras23:24
SpamapSI think a service with 1000 units and 2 or 3 peer relationships is going to get ugly to scale up/down because of all the joined and departed churn that will go on.23:26
m_3that's gonna happen no matter where things live23:26
SpamapS1000 nodes running relation-list with 1000 responses all at once :)23:26
SpamapSm_3: unless it becomes decentralized23:27
m_3every peer's connected to every other, so it'd be combinatorially explosive23:27
m_3yup23:28
m_3what would that look like23:28
m_3we'd need every node to have a 'torrent map' of the whole23:28
m_3be a big change23:28
SpamapSwell unless we just used bittorent to keep the map updated ;)23:29
m_3hey, I'm a huge fan of peer-based infra23:29
SpamapSanyway, I want to get to the long living 10 unit service before I start thinking about 1000 unit services. :)23:30
m_3I just honestly think we need to make sure that services with peer relations _also_ have master-slave options so they can scale23:33
m_3there's this is cool the way it _should_ work -vs- make it work and be stable23:34
SpamapShm, should we enforce that readme always be called README or README.* ?23:39
m_3sure23:40
m_3it's really not hard to be flexible with it tho23:40
m_3I _do_ like having the option of README.rst, README.md, or just README23:41
SpamapSwhat I mean is, README(\..+)?23:42
SpamapSso, not readme.txt23:42
SpamapSor ReadMe.md23:42
m_3old gnu std used to bug me... but it was useful sometimes in looking at noise that it screamed INSTALL at you23:42
SpamapSheh, our INSTALL is hooks/install23:42
m_3but yeah, it's worth asserting emphatically that they should read it :)23:45
m_3so on this note, it bugs me to see non-hooks in hooks/23:45
SpamapSI agree on that23:45
m_3I'd rather bin/ or lib/ or scripts/23:45
SpamapSbut I'm ok with "common.py" or a single "all-hooks.sh" that is symlinked to23:46
m_3scripts, templates, files for things you're installing elsewhere23:46
m_3bin/ lib/ for things your hooks are using23:46
m_3IMO that should be symlinked into bin/all-hooks.sh or lib/common.py... with only the hook links in hooks/23:47
m_3no biggie of course23:47
m_3just style23:47
SpamapSit gets REALLY hard with things like python though23:48
m_3huh?23:48
SpamapSnow you're adding things to PYTHONPATH .. and hating the guy who kicked stuff out23:48
SpamapSimport common will import ./common.py23:48
SpamapSas relative to the current python script...23:48
m_3CWD is pretty set tho23:49
SpamapSi am not sure its not going to do a readlink if $0 is a symlink..23:49
SpamapSnot CWD23:49
SpamapSrelative to the python script23:49
m_3right, but some people write wrt __file__23:49
m_3understand23:49
SpamapSjust so complicated for a tiny win23:49
m_3yeah23:49
m_3agree... very tiny win23:50
SpamapSm_3: nice job on the charm tester thing.. hopefully next week we can work on running charm-embedded tests with it. :)23:55
m_3SpamapS: thanks!  I'll roll it out to the list once we get a real url... then we can extend it23:59

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!