/srv/irclogs.ubuntu.com/2014/04/10/#juju-dev.txt

wallyworld__dannf: yeah, seems to be working for me now. i think it was just so slow yesterday that i gave up. i also tried building juju from src and i gave up after 30 minutes. maybe the vms are ust slow00:03
davecheneythumper: wallyworld__ on my ppc systems I see the provisioning constantly (every 300 sec) polling the charm store00:13
davecheneythis fails because FIREWALLS!00:13
davecheneybut i wonder why does it poll at all ?00:14
wallyworld__davecheney: to see if charms are out of date00:14
wallyworld__so that can be shown in status00:14
davecheneywallyworld__: ok, so i should raise an RT to get access to charm store enabled00:14
davecheneylooks like the proxy is blocking it00:14
wallyworld__yeah that would be good00:14
davecheneywallyworld__: ill fix00:15
wallyworld__davecheney: so when you run status, it says "hey you have mysql version 10 installed but version 12 is available"00:15
davecheneyahh nice00:40
davecheneyhttps://bugs.launchpad.net/juju-core/+bug/130536500:40
_mup_Bug #1305365: juju 1.18.0 environment unusable after bootstrap <juju-core:Triaged> <https://launchpad.net/bugs/1305365>00:40
davecheney^ what the hell00:40
davecheneyit bootstrapped and deployed fine00:41
davecheneywhy does status have a whinge00:41
cmarsthumper, i'm trying to capture output that's been written with logger.Errorf (from within in a coretesting.RunCommand). is there a way to hook into loggo like this?01:00
cmarsi've tried coretesting.Stdout(ctx), and Stderr, but nothing there01:00
thumpercmars: hey01:08
cmarshi01:08
thumpercmars: chances are you are using a LoggingSuite01:08
thumperit caputres the logging01:08
thumperit is in the gocheck *gc.C thingy01:09
thumperas some test log01:09
thumpernormally we don't test logging01:09
thumperwhat are you after exactly?01:09
thumperhi axw01:10
axwhey thumper01:10
thumperaxw: was asked to hit you up about a critical bug01:10
thumperbug 130373501:10
axwok01:10
_mup_Bug #1303735: public-address change to internal bridge post juju-upgrade <openstack-provider> <juju-core:Triaged> <juju-core 1.18:Triaged> <https://launchpad.net/bugs/1303735>01:10
cmarsthumper, the recent error message I've added. i think that'll do it, thanks01:10
thumperapparently you were talking with rob about it01:11
axwah, was chatting with rog about that last night...01:11
thumpercmars: there are some new methods on the context object01:11
thumpercmars: if you are testing the error01:11
thumperthen you should check the error response from Run01:11
thumpercmars: testing.RunCommand returns a context and an error01:12
thumperthe error is what you want to be checking01:12
thumperaxw: do you know what the issue is there with the addresses01:13
thumper?01:13
thumperI recall work going on there01:13
thumperbut I don't know exactly what changed01:13
axwthumper: it looks like the openstack environment has no public addresses, only cloud-local and/or unknown; the SelectPublicAddress code chooses the last cloud-local/unknown address in the list... so it gets one of the unknown addresses that the machiner records01:14
thumperhmm...01:15
thumperso... how do we go about fixing it?01:15
axwthumper: so, we should probably prefer the provider addresses over machine addresses if they're all cloud-local/unknown01:15
thumperare we able to tell the difference01:16
thumper?01:16
axwyes, they're recorded in separate lists01:16
axwthumper: it would be ideal if the machiner could decide which addresses were machine-local, but it's not a straightforward thing to do01:17
axwapart from localhost addresses of course...01:17
thumperright01:18
waiganithumper: davecheney has LGTMed https://codereview.appspot.com/85710043/ did you want to take a look before I land it?02:08
thumperyes please02:08
=== alexlist` is now known as alexlist
waiganiah, I should remove loggingSuite from environ_test.go02:15
davecheneyhttps://bugs.launchpad.net/juju-core/+bug/130538602:33
_mup_Bug #1305386: state/apiserver: multiple data races <juju-core:Triaged> <https://launchpad.net/bugs/1305386>02:33
thumperdavecheney: what causes these and do you have any suggestions to fix?03:03
davecheneythumper: not sure yet03:06
davecheneyi'm hoping its something we're doing (sharing a mgo conn) not a driver bug03:06
davecheneyits not clear if these are cosmetic or serious03:06
wallyworld__thumper: awesome, found out why go install was hanging. trunk of code.google.com/p/go.crypto has changed and juju-core no longer compiles but that causes go install to just hang unless you compile with the -v option03:12
wallyworld__how good is thst03:12
thumperwut?03:12
thumperhah03:12
wallyworld__so we had better not update go.crypto03:12
thumperwhy?03:12
wallyworld__no idea03:12
thumperwallyworld__: so you should run godeps, yeah?03:12
wallyworld__seems so03:13
wallyworld__but i mean wtf03:13
wallyworld__it should have told me there was a compile error03:13
thumperi would have thought so03:13
wallyworld__i just thought it was slow cause other stuff has been slow03:14
davecheneywallyworld__: is the compiler spinning ?03:15
davecheneyagl and hanwen landed that big branch today03:15
wallyworld__davecheney: not sure, how do i tell?03:15
davecheneyi didn't see any discussion about it03:15
davecheneytop ?03:15
davecheneythat was rather unorthodox03:15
wallyworld__davecheney: strangely using the -v flag which prints aeach package as it is compiled printed out the errors also and then exited when done03:16
davecheneywallyworld__: paste03:16
davecheney?03:16
wallyworld__davecheney: https://pastebin.canonical.com/108166/ shows first attempt hanging, and then -v showing errors03:18
davecheneywallyworld__: can you use pastebin it03:18
davecheneymy 2fa is downstairs03:18
wallyworld__ok03:19
wallyworld__http://pastebin.ubuntu.com/7229253/03:19
davecheneywallyworld__: can't tell if a hang, or just took a long time03:20
davecheneytry03:20
davecheneyrm -rf $GOPATH/pkg03:20
wallyworld__davecheney: it ran for over an hour before i hit ^C03:20
davecheneywallyworld__: did you look at top ?03:20
wallyworld__no, not at the time03:20
davecheneybummer03:20
davecheneyas for tip of ssh being broken03:20
davecheneyyes, that is poor form03:20
davecheneybut we're not blameless here03:20
davecheneythe top of gomaasapi is unusable isn't it03:21
davecheneyor is it gwacl ?03:21
wallyworld__not sure03:21
wallyworld__davecheney: i ran it again and that time it exited with the compile errors even without the -v flag03:21
davecheneyi don't -v would have any impact on a hang03:22
davecheney-v is for cmd/go03:22
davecheneywhich just forks the compiler03:22
davecheneyits hard to say uless you can get it to happen again03:22
davecheneyi'd 100% believe go get hanging03:22
wallyworld__i read in help that -v is for verbose03:22
davecheneywallyworld__: yes, that is what it does03:22
davecheneygo get can get confused when hg/bzr/git fire off some program to deal with merge conflicts as the pty they run against isn't a terminal03:23
wallyworld__ok03:23
wallyworld__davecheney: right, after running godeps, install now appears to be spinning03:25
davecheneywhich process is spinning03:26
davecheneycmd/go03:26
wallyworld__yep03:26
davecheneyor the compiler ?03:26
davecheneykill cmd/go with SIGQUIT03:26
davecheneycapture the output03:27
davecheneygreat03:27
davecheneythis is some pretty shit03:27
davecheney11th hour and everyting is breaking, including the toolchain03:27
davecheneywonderful03:27
wallyworld__davecheney: http://pastebin.ubuntu.com/7229265/03:28
wallyworld__something about stack unavailable03:28
wallyworld__nuking pkg fixes it03:29
davecheneywallyworld__: whos' machine is this ?03:31
davecheneyits running gccgo03:31
wallyworld__thumper: \o/ and now i have it compiled i can't test anyway because of bug 1304742  FML03:39
_mup_Bug #1304742: version reports "armhf" on arm64 <arm64> <hs-arm64> <juju-core:Triaged> <https://launchpad.net/bugs/1304742>03:39
wallyworld__i guess i should fix that03:39
thumperyeah... guess so03:39
* wallyworld__ sighs heavily03:40
* axw joins in03:45
axwHP cloud doesn't like my change03:45
davecheneysupportedArchitectures isn't an ordered list, right ? https://bugs.launchpad.net/juju-core/+bug/130539703:49
_mup_Bug #1305397: provider/common: test failure <gccgo> <ppc64el> <juju-core:Triaged> <https://launchpad.net/bugs/1305397>03:49
* davecheney goes to lunch03:50
davecheneyin the rain03:50
wallyworld__davecheney: another one http://pastebin.ubuntu.com/7229318/04:04
wallyworld__faaaark. only took 1000 attempts but finally got the go compiler to build without panicing04:21
=== vladk|offline is now known as vladk
davecheneywallyworld__: eerk, that looks simlar to the bug we see on 64k ppc64 kernels05:17
wallyworld__davecheney: yuk. i'm not having much luck with juju per se either. machine 0 agent won't start properly05:18
davecheneywallyworld__: more nil pointers ?05:18
wallyworld__doesn't seem to get past starting the start server05:18
wallyworld__not sure yet if it's because it can't see the db or something else05:19
davecheneywallyworld__: using lxc ?05:20
wallyworld__nope, arm vms05:20
wallyworld__manual provisioning05:20
davecheneyok05:20
davecheneylxc is broken atm05:21
wallyworld__:-(05:23
davecheneywallyworld__: i know05:24
wallyworld__i'm running from trunk and it seems like it's dying inside the ensure ha stuff but there's not enough logging to know for sure05:25
davecheney:(05:25
davecheney:emoji crying:05:25
wallyworld__davecheney: i farking give up. takes 1000 retries to get a compile. then complains about missing libgo.so.5. so find that i need extra compile flags. 1000 retries later get new jujud. still libgo.so.5 missing. so i install deb package on target machine just to get that shared lib. rinse and repeat. still complains. sigh06:29
davecheneywallyworld__: see https://docs.google.com/a/canonical.com/document/d/1m9R2n6LPLNLGjdopcNkQYVG8D5V4FTyvc1vvn-9ZifM/edit06:32
davecheneyat the bottom06:32
davecheneythe gb alias06:32
davecheneywallyworld__: can you paste me anything intersting from dmesg06:32
wallyworld__davecheney: i used those flags06:32
davecheneyi want to compare with the ppc64 problems06:32
davecheneyif it's complaining that libgo.so.5 is missing06:33
wallyworld__binaries were about 30% bigger after06:33
davecheneythen it didn't work06:33
davecheneywallyworld__: i think you should cut your losses06:33
wallyworld__so then i tried installing the lib directly on the target machine06:33
davecheneylooks like juju doesn't work on arm6406:33
davecheneyi don't know if anyone is working on that atm06:33
wallyworld__we're close06:34
wallyworld__dmesg doesn't have anything that interesting06:34
davecheneywallyworld__: ldd $(which jujud)06:35
wallyworld__ubuntu@ms01a:~/juju/src/launchpad.net/juju-core/cmd/jujud$ ldd $(which jujud)06:35
wallyworld__        linux-vdso.so.1 =>  (0x0000007fafbdd000)06:35
wallyworld__        libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000007fafba7000)06:35
wallyworld__        libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000007fafb07000)06:35
wallyworld__        libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000007fafae4000)06:35
wallyworld__        libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000007faf999000)06:35
wallyworld__        /lib/ld-linux-aarch64.so.1 (0x0000005595496000)06:35
wallyworld__hmm, so libgo.so.5 is missing06:36
wallyworld__i built with06:36
wallyworld__go install -a -v -gccgoflags -static-libgo launchpad.net/juju-core/...06:36
wallyworld__ah missing =06:36
wallyworld__fark06:37
davecheneywallyworld__: nope06:37
davecheneyyou got it right06:37
davecheneyif you got it wrong06:37
davecheneythere would be a line showing06:37
davecheneylibgo.so.5 => (missing)06:37
wallyworld__ok06:37
davecheneybut you should probably use the =06:38
wallyworld__used the =, same result06:39
rogpeppe2mornin' all06:40
davecheneyin this case it will be06:41
davecheneybut if you wanted to pass a second option06:41
davecheneyyou'd need = and to quote it06:41
davecheneyrogpeppe2: o/06:41
rogpeppe2davecheney: yo!06:41
axwmorning rogpeppe206:44
rogpeppe2axw: hiya06:45
axwrogpeppe2: I've got a CL for the addresses bug, about to propose (just finishing live test) - got time to have a look?06:45
rogpeppe2axw: sure06:45
axwrogpeppe2: I made the changes we talked about, but also changed instance.NewAddress derive scope from IPv4 address range06:46
axwso 10.*, 192.168.*, etc. are recorded as cloud-local06:46
rogpeppe2axw: that seems reasonable to me06:47
rogpeppe2axw: although that's not really true in the case of this bug06:49
rogpeppe2axw: the 192.168.* addresses were machine-local06:49
axwrogpeppe2: I think that's okay, because we'll still choose provider cloud-local over machine cloud-local06:50
axwmachiner*06:50
rogpeppe2axw: yeah, it should be ok06:51
* axw taps finger, waits for lbox06:51
wallyworld__davecheney: success. i blew away *everything* and started again. bootstrapped manual provider on arm64. gotta run to soccer, but will add another machine later and deploy a charm06:54
axwrogpeppe2: https://codereview.appspot.com/85590044/06:59
axwmgz: would appreciate a review from you too, in case there's some subtlety I've missed07:00
axwrogpeppe2: on a completely unrelated note, I was thinking about the EnsureAvailability task some more this morning. Is there a reason why we shouldn't move a non-voting state server back to voting if it becomes accessible again?07:01
rogpeppe2axw: absolutely not.07:02
rogpeppe2axw: i think we should do that07:02
axwgoodo07:02
axwthat's what I thought too07:02
rogpeppe2axw: that's why we keep 'em around07:02
=== vladk is now known as vladk|offline
fwereadeanyone know anything about the replicaset failure in https://code.launchpad.net/~fwereade/juju-core/uniter-relation-states/+merge/215003 ?07:21
fwereadeare we seeing it a lot?07:21
davecheneyfwereade: wrt your previous comment07:27
davecheneyhttp://paste.ubuntu.com/7229693/07:27
davecheneyi am not sure if this is a real race, or an instrumentatoin error07:28
fwereadedavecheney, I will take a look at that, thanks07:28
fwereadedavecheney, although at first glance... sync.WaitGroup *ought* to be used that way, oughtn't it?07:29
davecheneyfwereade: you cannot call Add() while another goroutine calls Wait()07:30
davecheneyonly Done()07:30
rogpeppe2axw: reviewed07:31
axwta07:31
axwrogpeppe2: heh, I kept writing IPv4 in my new code and then changed it to match the existing :/07:32
axwI will fix it another time- this is going to need to be backported, don't want to create unnecessary hardship :)07:32
rogpeppe2axw: yeah, it's definitely worth changing at some point07:32
rogpeppe2axw: fair enough07:32
davecheneyhttp://golang.org/pkg/sync/#WaitGroup.Add07:33
rogpeppe2fwereade: ah, i think i know what that might be to do with07:34
rogpeppe2fwereade: i think perhaps the replicaset tests were relying on the fact that Set put the session into monotonic mode.07:35
axwrogpeppe2: likewise for NewAddress arg order if you don't mind - that will create a lot of noise07:35
rogpeppe2axw: ok07:35
rogpeppe2fwereade: (i mean Initiate, not Set)07:36
fwereadedavecheney, indeed, I see; but I'm having some difficulty mapping it onto the code07:37
davecheneyfwereade: yes07:37
fwereadedavecheney, the code looks odd fwiw, I don't see why we'd wait twice07:37
davecheneyi'm having trouble understandig if that is a real race07:37
davecheneyor a bug in the race detector07:37
davecheneyi'd use 1.3, but thre are heaps of bugs at tip07:37
fwereadeheh07:37
davecheneyas usual juju is leading the way in showig bugs in go tip07:37
fwereadego us!07:37
rogpeppe2fwereade: hmm, it doesn't look as if that is actually the case07:38
rogpeppe2davecheney: it may well be a real race07:39
davecheneyrogpeppe2: \o/, i guess07:39
rogpeppe2davecheney: something similar came up on golang-nuts recently07:39
davecheneyrogpeppe2: indeed07:39
davecheneyand the semantics may change07:40
davecheneybut under 1.207:40
davecheneythe old semantics apply, you can't call Add(positive) after someone else has called Wait()07:40
fwereadedavecheney, hey, I think it is a race, but I'm still only at single-coffee levels of incisiveness07:40
davecheneyfwereade: based on that assesment, I shall raise a bug07:41
davecheneyi have found many other races07:41
davecheneysome in the mongo driver07:41
fwereadedavecheney, cheers, we can always close it if it isn't07:41
rogpeppe2davecheney: looks like a trivial fix for that one at least07:42
rogpeppe2davecheney: though i'm not sure it will actually fix the bug we're seeing07:42
davecheneyrogpeppe2: they normally are07:42
davecheneytrue07:42
rogpeppe2davecheney: it may do though, thinking about it07:43
rogpeppe2davecheney: get that Add in!07:43
davecheney https://code.launchpad.net/~dave-cheney/juju-core/128-environs-sync-tempdir-prefix/+merge/21508707:46
davecheneyif anyone has two seconds07:46
davecheneyno07:46
davecheneyignore, that one landed07:46
davecheneyhttps://code.launchpad.net/~dave-cheney/juju-core/127-fix-lp-1305397/+merge/21507707:46
davecheneythis one is also trivial07:46
fwereaderogpeppe2, that failure in my MP has an *awful* lot of "attempting Set got error: replSetReconfig command must be sent to the current replica set primary." lines before the no-reachable-servers failure07:46
rogpeppe2fwereade: yeah, i don't know why07:47
davecheneyrogpeppe2: worker/peergrouper/worker_test.go :31807:57
davecheneyi see test failures when the servers is not the first entry in expectedAPIHostPorts(3)07:58
rogpeppe2davecheney: yeah, agreed, it's dubious. weren't we going to sort the servers slice?07:58
davecheneyrogpeppe2: i am trying07:59
davecheneybut how can I sort a set of []instance.APIPorts07:59
rogpeppe2davecheney: fairly easily08:00
davecheneydo tell08:00
rogpeppe2davecheney: it's not hard to define an ordering between two []instance.HostPort values08:00
rogpeppe2davecheney: (compare each element in order)08:01
davecheneyrogpeppe2: thta isn't what you told me to do08:01
rogpeppe2davecheney: you can probably just compare address value08:01
davecheneyyou said that []instance.HostPort is already sorted08:01
rogpeppe2davecheney: it is08:01
davecheneywright08:01
davecheneyrifht08:02
rogpeppe2davecheney: we're sorting a [][]instance.HostPort08:02
davecheneyso I need to sort a [][]instance.HostPort08:02
davecheneyyup08:02
rogpeppe2davecheney: so to do that, you need to compare two []instance.HostPorts08:02
davecheneyrogpeppe2: ok, got it08:02
rogpeppe2davecheney: cool08:02
rogpeppe2anyone know how to find out what code a given CI test is actually running?08:29
rogpeppe2(the CI test itself, that is)08:29
fwereaderogpeppe2, I think it's lp:~juju-qa/juju-core/ci-cd-scripts208:31
rogpeppe2fwereade: i just found that, but it doesn't seem to have all the scripts in it (e.g. aws-upgrade)08:32
rogpeppe2fwereade: but i'm not sure what the correspondence is between that branch and the test names we see in jenkins08:33
davecheneyrogpeppe2: https://codereview.appspot.com/86400043/08:44
rogpeppe2darn, i've just realised that we really need to support upgrading to HA08:44
davecheneyyour thoughts sir08:44
davecheneyopps08:44
davecheneytwo secs08:44
rogpeppe2davecheney: that doesn't look quite right08:46
rogpeppe2davecheney: i don't think that Less function is commutative08:46
davecheneyhttps://codereview.appspot.com/86400043/08:47
davecheneysorry, have another look08:47
rogpeppe2davecheney: i think you want: if len(a) != len(b) { return len(a) < len(b) }08:47
davecheneyok08:49
rogpeppe2davecheney: in fact, i think the other test isn't right either08:53
davecheneyyeah, that was tricky08:53
rogpeppe2davecheney: i think it needs to compare the values for equality, and only if they're not equal should it compare the less-ness of them08:53
davecheneyyeah, that makes sense08:54
rogpeppe2davecheney: i've got that wrong before08:54
rogpeppe2davecheney: so i always look closely at Less functions now :-)08:54
davecheneyrogpeppe2: this one is tricky because in our case the ports are always the same08:55
davecheneyso a < b is always false08:55
rogpeppe2davecheney: i don't think it's that tricky08:55
rogpeppe2davecheney: one mo, i'll paste a suggestion08:55
rogpeppe2davecheney: reviewed (with suggested code in the review) https://codereview.appspot.com/86400043/08:58
davecheneyrogpeppe2: ta08:58
=== rogpeppe2 is now known as rogpeppe
davecheneyrogpeppe: thanks, that is much more straight forward09:00
rogpeppedavecheney: np09:00
rogpeppedavecheney: just sent one other trivial comment09:02
axwmgz: ping09:02
rogpeppeafk09:03
davecheneyrogpeppe: et al, http://code.google.com/p/go/issues/detail?id=774909:03
davecheneyjuju really breaks go 1.3 at the moment09:03
mgzaxw: hey09:11
axwmgz: hey, would you please take a look at https://codereview.appspot.com/85590044/ when you have a moment?09:14
axwI've changed some address logic, and I'm a bit nervous about breaking everything :)09:14
mgz:)09:14
mgzaxw: had planned the pivate range stuff, that looks fine, will have a proper poke through before the standup and see if any of the subtle logic bits have got lost09:18
axwmgz: great, thanks09:21
mgzaxw: only thing that jumps out as a risk is canonistack and similar setups09:22
mgzwere we don't actually *have* a public address generally,09:23
mgzbut currently juju can lie, and return the NetworkUnknown 10. address which will work with sshuttle09:23
mgzright, I need to change location now09:25
axwmgz: I did live test with canonistack (and HP); the 10. address is now recorded as cloud-local, and usable as both public and internal09:25
rogpeppedavecheney: :-(09:25
rogpeppedavecheney: can you reproduce that reliably?09:25
=== natefinch-afk is now known as natefinch
natefinchmorning all09:30
axwmorning natefinch09:31
=== vladk|offline is now known as vladk
jam1natefinch: good morning. extra early for you, isn't it ?09:43
natefinchjam1: yeah, been trying to get up early to get some more work done on HA. Also, yesterday I lost a good bit of the working day due to some emergency beekeeping tasks that came up.09:45
perrito666good morning09:55
jamjust mentioning, standup in 3 min09:57
thumpernatefinch: you have bees?10:19
thumperaxw: just wondering about 11.0.0.0/810:24
thumperaxw: it is technically public10:24
thumperaxw: but actually private10:24
thumperaxw: I wonder if anyone tries to use it10:24
thumperaxw: it is the ipaddress of the us military group10:24
thumperand is a disconnected internet type network10:25
thumperI think the current code is good, and if someone is stupid enough to use this10:25
thumperthen it can be on their head :-)10:25
axwthumper: that would be an interesting problem to have :)10:25
* axw thinks10:26
axwit will be no worse than it is now, as cloud-local will be used for public/internal as well as unknown10:26
axwoh wait10:26
axwpublic.. hrm10:26
axwwell they would be inside the cloud anyway, so it would be fine I think10:27
axwinside the cloud -> inside the network10:27
thumperaxw: I do like the 'less unknowns' :-)10:28
perrito666hey fwereade this is a wip https://codereview.appspot.com/86430043 for some reason https://codereview.appspot.com/86430043/patch/1/10003 is getting empty unit networks, care to take a quick look? (I did not upload work for the machine not changed checks to avoid clutter on what I am trying to debug)10:38
natefinchthumper: yeah, three hives. bees are awesome :)10:43
fwereadeperrito666, actually do you want to pop on, I'm not quite following everything there10:44
perrito666pop?10:45
fwereadeperrito666, sorry, I mean, quickly reenter the team meeting hangout10:45
perrito666I certainly can10:45
rogpeppei've got two branches up for review. would much appreciate if someone could take a look: https://codereview.appspot.com/86200043/ https://codereview.appspot.com/85600044/10:46
c7zaxw: properly going over address branch now, I see rog has already looked through it10:52
axwc7z: I assume that's mgz; yes he has, just thought I'd get your thoughts too, because you did the original work I think?10:57
axwbbs10:58
c7zaxw: yup, though there's a lot more complexity than the first version unfortuanately10:59
natefinchrogpeppe:  looking11:04
waiganioh noooo, daylight savings I missed the meeting11:05
natefinchhahaha11:05
waiganiI was like, why is there no one here???11:06
natefinchyep, daylight savings is annoying11:06
waiganisigh, reading notes11:06
jamespagefwereade, sorry - bug 130578011:06
_mup_Bug #1305780: juju-backup command fails against trusty bootstrap node <juju-core:New> <https://launchpad.net/bugs/1305780>11:06
natefinchwaigani: didn't miss much11:07
waiganihey, I'm in the notes!11:07
waiganinatefinch: okay, that's good. Well I'll get some sleep and be more productive tomorrow!11:08
waiganinight all11:08
natefinchwaigani: g'night11:08
fwereadejamespage, I thought we had the tools used by backup/restore in juju-mongodb11:22
jamespagefwereade, we do - but I suspect the fact they are not in the path is breaking things11:22
fwereadejamespage, gaaaah ofc11:22
jamespagefwereade, it works fine on 12.0411:22
jamespagewhere that is the case11:22
c7zaxw: commented11:23
jamespagefwereade, I suspect if I could get restore past "error: cannot restore bootstrap machine: cannot get public address of bootstrap machine: machine "0" has no public address"11:23
jamespagethen I would hit the same issue gain on 14.0411:23
c7zjamespage: https://codereview.appspot.com/85590044/11:24
c7zone of your bugs is nearlyfixed11:24
jamespagec7z, \o/11:24
jamespagewoser11:24
jamespagethat was complex11:24
jamespagec7z, I better stop finding new ones :-)11:25
c7zyeah, axw decided to start doing the right thing with deriving the network scope rather than piling on hacks11:26
rogpeppenatefinch: "could this code get moved to the loop over entity.jobs below?"11:36
rogpeppenatefinch: i don't think so11:36
rogpeppenatefinch: because newSingularRunner can fail11:36
dimiternjam, mgz, have the bot stopped landing stuff for gomaasapi?11:37
jamdimitern: I'm pretty sure the bot never landed things for gomaasapi11:37
jamonly gwacl11:38
c7zdimitern: it never did in the current iteration11:38
dimiternreally?11:38
c7zdimitern: I manually landed the last bits, I can land anything else you guys need11:38
c7zthere was once a gomaasapi bot, but it wasn't ours, and it went away11:38
jamdimitern: lp:~juju/gomaasapi/trunk11:38
c7zend of fairytail11:38
natefinchrogpeppe: yeah, that's a good point.  I guess if you need to make sure you fail early, that's valid.11:38
dimiternvladk, there's your reason ^^ c7z is your man :)11:38
jamthe bot has never been in ~juju, IIRC11:38
natefinchrogpeppe: not for this review, but that method really needs to be refactored. 110 lines is just too long.11:39
jamdimitern: I intentionally didn't want to give the bot too much access to things that weren't its, nor give ~juju direct access to bits controlled by the bot11:39
rogpeppenatefinch: yeah, it could be easily split up11:40
* jam is away for a bit11:40
dimiternjam, yep, understandable concerns11:40
* rogpeppe has just acquired a new, unbroken phone11:41
rogpeppewoo11:41
c7zrogpeppe: what did you go for?11:41
c7zalso, if it doesn't have a smashed screen by vegas, I'll be disappointed11:41
rogpeppec7z: a samsung galaxy s4 active from ebay11:41
rogpeppec7z: :-)11:41
rogpeppec7z: main reasons were the fact that it is waterproof (i killed a previous phone from water damage) and it has a replaceable battery11:42
vladkc7z: could you manually land my branch to gomaasapi: https://code.launchpad.net/~klyachin/gomaasapi/101-testserver-extensions/+merge/21496111:46
c7zvladk: on it11:47
axwc7z: just so I understand about the floating IP...11:47
axwc7z: floating-ip would be stored as NetworkUnknown as well?11:47
axwc7z: hence why we would take the last one?11:47
c7zyeah, so, the original iteration of the code understood some network names as special11:49
c7zbut hp and some others were annoying in that they had a network named 'private'... but the (public) floating ip just got appended to that network, not added to a new one11:50
* rogpeppe just realises that c7z==mgz11:50
c7zrogpeppe: sorry :P11:50
c7zbut I'm pretty sure that your code will actually pass that test (with less fiddling that you needed), because of the new scope detection code11:51
axwc7z: okay, cool. yes I think it should work then11:53
axwc7z: good catch on NetworkName. I'll fix that and land11:53
c7zbut yeah, the current/old version of openstack bits basically left everything as NetworkUnknown and used some hacks based on ordering to make the various previous cases work11:53
c7zvladk: landed11:57
rogpeppenatefinch: how's it going?11:59
natefinchrogpeppe: sorry, helping my ender daughter, Lily get ready for preschool.  Should be back in about 45 minutes, though not at full capacity for a little over an hour.12:06
natefinchs/ender/elder/12:06
rogpeppenatefinch: ok. perhaps you could just push the branch you were working on last night?12:06
rogpeppenatefinch: then i can move it forward12:06
natefinchrogpeppe: cool, just pushed it here: lp:~natefinch/juju-core/041-moremongo12:08
rogpeppenatefinch: thanks12:09
c7zevilnickveitch: had a note that the environments.yaml config option for bug 1241674 isn't added to the 1.18 docs, what branch do I need to get to put it in?12:16
_mup_Bug #1241674: juju-core broken with OpenStack Havana for tenants with multiple networks <cts-cloud-review> <openstack-provider> <juju-core:Fix Released by gz> <https://launchpad.net/bugs/1241674>12:16
evilnickveitchc7z, it should go in the master branch for now, thanks!12:19
axwc7z: sorry, will fix that test in a followup12:28
c7zaxw: I wasn't quite clear the first time around that the test should have just been s/127\./10\./12:29
axwc7z: no worries, understood now - I think the bot's already running it though12:29
jamanyone else having trouble getting to bazaar.launchpad.net ?12:31
c7zaxw: no problems, as I said, should pass that way as well12:31
c7zjam: I just did, and it was transient12:32
jamc7z: can you check if you can get to launchpad?12:32
jamc7z: k, it is still failing for me... :(12:32
c7zas in, failed to branch twice, pinged, worked, sshed, worked, branched... worked12:32
jamc7z: I can't SSH or get to the HTTP page12:32
c7zjam: apparrently lp app servers were seeing issues getting stuff through squid, it's still working for me at present12:45
jamc7z: it just worked fro me12:45
jamfor12:45
jamdimitern: ping about SCP and extra arguments12:51
jamyou seem to have a patch that made it so that scp only supports *1* extra argument12:51
jamand CI wants to use about 5 extra args12:51
c7zwhen doing a local provider deploy for the first time, how can you track the image download progress?12:51
jamc7z: iftop ?12:51
jamI wish I knew a better way, I think it is controlled underneath lxc12:52
jam(hidden from us)12:52
c7zisn't that fun12:52
dimiternjam, my changes to scp was that it can accept any number of extra args12:54
jamdimitern: not in 1.1812:54
jamdimitern: "juju scp 1:foo . -o "StrictHostKeyChecking: no"12:55
dimiternjam, if it got changed later i don't know12:55
jamcomplains that "-o" is unknown12:55
jamdimitern: vs juju scp 1:foo . -o"StrictHostKeyChecking: no"12:55
jamworks12:55
jambut it has to be *1* argument12:55
jamdimitern: the line is "if i != len(c.Args) - 1"12:57
jamsounds like it only accepts 1 extra argument, and is attributed to you (according to bzr annotate)12:57
dimiternjam, hmm.. looking at the code I see the problem12:57
dimiternjam, it was broken before - not being able to take more than 3 targets12:57
dimiternjam, but i broke that it seems - the fix should be: once we start adding extraArgs we treat all the rest as extraArgs12:58
jamso CI was trying to do:       if timeout 5m juju --show-log scp -e $ENV -- -o "StrictHostKeyChecking no" -o "UserKnownHostsFile /dev/null" -i $JUJU_HOME/staging-juju-rsa 0:/var/log/juju/all-machines.log $log_path; then12:58
jamusing "--"12:58
jamwhich I don't quite see that we ever actually supported anyway12:59
jamBut they would like to have the target late, and extra args early12:59
jamwe can move that around12:59
jam(I think)12:59
jambut we should support more args12:59
dimiternwhy targets late and extra args early?13:00
jamdimitern: it is a natural way to write it, if you were writing "SCP" code.13:00
jamas in, it is how *I* would write "scp ..."13:00
dimiternit used to be documented that extra args are passed after -- that was never implemented13:00
jamdimitern: I don't think we *have* to because SCP will let you passed them late.13:00
dimiternjam, well, initially i made it so -- can be at any place (even between targets) to specify one or more -args, but it was rejected on the review13:02
dimiternjam, yeah, passing them last is both easier to parse and not far from natural13:02
dimiternjam, I think something like this should fix the issue http://paste.ubuntu.com/7230671/13:05
bodie_I'm getting a weird error with cobzr.  I should be able to diff against trunk, right?13:14
axwrogpeppe: https://codereview.appspot.com/86490043  -- hopefully I'm at least on track here...13:15
rogpeppeaxw: looking13:15
axwrogpeppe: I'm about to sign off, so no particular rush13:16
c7zevilnickveitch: https://code.launchpad.net/~gz/juju-core/update_config_openstack_1.18/+merge/21517713:22
evilnickveitchc7z, thanks, i'll take a look13:23
axwI'm off now, will check a bit later but would appreciate if someone could poke my MP if it fails again: https://code.launchpad.net/~axwalk/juju-core/lp1303735-fix-address-logic/+merge/21508513:25
axwand I'll backport to 1.18 in the morning13:26
jamdimitern: the problem is the signal to start adding things has a "if i != len(args) -1"13:31
jamso we really just need a different check. I think actually implementing "--" would be the easiest thing, if we want to support multiple SCP targets13:31
dimiternjam, -- sgtm to13:33
dimiternjam, can't remember who was against it in the review ;)13:33
c7z...the code did use  --... which broke things apparently, hence the fix to not13:34
jamc7z: well the CI guys had written "scp -- 0:foo ." which would be broken regardless13:34
jambecause that would pass an explicit "0:blah" to scp13:34
jamWe *could* just detect if any given argument had a possible Juju identifier at the start and map it13:35
jamwith potentially an escape character?13:35
* jam doesn't like escapes very much here13:35
=== Ursinha is now known as Ursinha-afk
=== hatch__ is now known as hatch
dimiternfwereade, are you around?14:12
mrammfwereade: alexisb: I just put one of you on the hook for a 1.19 and 1.18 update at the cross-team meeting.14:24
alexisbmramm ack14:24
=== Ursinha-afk is now known as Ursinha
jammramm: I think fwereade said he had a headache, so is off for a bit. which puts alexisb on the hook :)14:32
mrammjam: cool14:32
mrammI'm sure alexisb can handle it ;)14:33
jamalexisb: here's my summary, I think cmars is already working on #1304770, but I don't have a feeling for why it isn't done already14:33
_mup_Bug #1304770: store: tests do not pass with juju-mongodb <ppc64el> <juju-core:Triaged by cmars> <https://launchpad.net/bugs/1304770>14:33
fwereademramm, jam, alexisb: thanks, I just dragged myself up here to make sure there was someone there14:33
* fwereade goes back to bed14:33
jambug #1302205 was the one that wallyworld__ sent an email on. I don't think it should block 1.19.0 if we can't fix it in time14:33
_mup_Bug #1302205: manual provisioned systems stuck in pending on arm64 <add-machine> <hs-arm64> <manual-provider> <juju-core:In Progress by wallyworld> <https://launchpad.net/bugs/1302205>14:33
jamthe toolchain is hard to work with there, so it isn't a *regression*14:33
mrammwe should definitely talk over that bug on the call14:34
jambug #1303697 I thought fwereade actually had a patch that I reviewed, so it should be just-about done as well14:34
_mup_Bug #1303697: peer relation disappears during upgrade of juju <juju-core:In Progress by fwereade> <juju-core 1.18:In Progress by fwereade> <https://launchpad.net/bugs/1303697>14:34
jamdimitern: do you have a status for bug #1304905? Presumably that is Critical because you wanted to fix the API before it becomes released?14:34
_mup_Bug #1304905: Change NetworkName to NetworkId across codebase and use network tags in the API <api> <tech-debt> <juju-core:In Progress by dimitern> <https://launchpad.net/bugs/1304905>14:34
jambug #1305386 is about "this might be a bug" being told to us by the go compiler, which is something we should try to address, but it isn't a bug that has seen actual live problems14:35
_mup_Bug #1305386: state/apiserver: multiple data races <juju-core:Triaged> <mgo:New> <https://launchpad.net/bugs/1305386>14:35
dimiternjam, i'm about to propose the CL that fixes it14:37
cmarsjam, alexisb, I have a fix for 1304770, which I can land, we'll just need to update CI to test the charm store tests that will become disabled by default14:43
jamcmars: could we invert the logic? or provide it via an ENV var and just have the build process set that ENV var?14:43
jambecause the landing bot is happy to run the tests14:43
jamand so is CI14:43
jamit is more that the build process for trusty needs to be able to disable them14:44
jamWe could even just do "if version.Current.arch == 'trusty'" if we wanted.14:44
cmarsjam, i can certainly invert the logic14:44
dimiternjam, mgz, rogpeppe, anyone.. I'd really appreciate a review on this https://codereview.appspot.com/86010044/, which also fixes bug 130490514:44
_mup_Bug #1304905: Change NetworkName to NetworkId across codebase and use network tags in the API <api> <tech-debt> <juju-core:In Progress by dimitern> <https://launchpad.net/bugs/1304905>14:44
cmarsjam, would you prefer a switch or env var?14:44
rogpeppedimitern: ok, lookin14:45
rogpeppeg14:45
cmarsor both :)14:45
dimiternrogpeppe, cheers - it looks huge, but it's not, just touches a lot of things14:45
jamcmars: *I* prefer an env var, because 'go test ./...' doesn't pass switches well, they have to be defined on all packages14:47
jamwhile an ENV var can just be picked up while running.14:47
cmarswfm14:47
natefinch+1 for me too14:48
rogpeppejam: +114:50
rogpeppejam: it would be nice to have a small set of well defined env vars though14:51
rogpeppefwereade: do you recognise this last test failure? looks like a test wasn't updated. https://code.launchpad.net/~rogpeppe/juju-core/mfoord-wrapsingletonworkers/+merge/21503514:55
perrito666rogpeppe: fwereade seems to be ill in bed :(14:57
rogpeppeperrito666: ah14:57
alexisbso jam just a general note... much of the status on bugs you are giving here is not actually in the bug, any reason for that?  Can we just ask folks to update bugs, seems like a pretty good common place to put status :)15:04
cmarsjam, updated the nomongojs option for tests, ptal https://codereview.appspot.com/8293004315:32
rogpeppedimitern: you've got a review15:48
dimiternrogpeppe, thanks!16:06
rogpeppedimitern: yw16:06
dimiternc7z, ping16:15
c7zdimitern: hey, will be free in a min16:16
dimiternc7z, rogpeppe reviewed my https://codereview.appspot.com/86010044/, but can you take a look as well please? I want to land it today, and I almost have the last bit working (cloudinit).16:17
jam1fwereade: does your patch for https://bugs.launchpad.net/juju-core/+bug/1303697 actually fix the full bug? I felt like it did, so I went ahead and marked it Fix Committed since your branch is merged.16:19
_mup_Bug #1303697: peer relation disappears during upgrade of juju <juju-core:Fix Committed by fwereade> <juju-core 1.18:In Progress by fwereade> <https://launchpad.net/bugs/1303697>16:19
jam1fwereade: I should note that if you do "bzr commit --fixes lp:12345" then Tarmac will mark things as Fix Committed when it merges them.16:19
jam1alexisb: so I updated some things, for things that "have a branch in progress ready to land" that information is captured in the bug, as long as people link their branches to the bugs. (after the Description is a section about Related Branches), see https://bugs.launchpad.net/juju-core/+bug/130369716:23
_mup_Bug #1303697: peer relation disappears during upgrade of juju <juju-core:Fix Committed by fwereade> <juju-core 1.18:In Progress by fwereade> <https://launchpad.net/bugs/1303697>16:23
jam1some of the other bits I mentioned were just stuff that I worked out while reading over the current list of critical bugs, thus existed nowhere but in IRC at that moment :)16:24
* jam1 goes to take my son to bed16:25
cmarscan someone PTAL at my mongojs notest option proposal, https://codereview.appspot.com/82930043/?16:25
dannfthus far i've only tried building juju from the deb package - what's the right mechanism for doing it w/ go gotten source?16:29
dannfnm - see the README16:30
rogpeppenatefinch: lp:~rogpeppe/juju-core/natefinch-041-moremongo16:41
natefinchrogpeppe: func fakeCmd(path string) {16:43
natefincherr := ioutil.WriteFile(path, []byte("#!/bin/bash --norc\nexit 0"), 0755)16:43
natefinchif err != nil {16:43
natefinchpanic(err)16:43
natefinch}16:43
natefinch}16:43
alexisbjam1, thanks!16:45
jam1cmars: lgtm17:14
cmarsjam1, thanks!17:15
jam1cmars: make sure to update the bug with how the build process can skip the tests if it needs to17:15
cmarsjam1, will do. i'm standing up an LXC to test it with an actual juju-mongodb right now, just to be sure. i usually develop against a stock mongodb-server17:16
jam1cmars: I also wonder if there is an obvious place in a README or HACKING to describe env flags for the test suite like this17:21
cmarsjam1, CONTRIBUTING, in the Testing section17:27
cmarsi'll add a blurb17:28
alexisbjam1, any objections to me moving the target for 1302205 to 19.1?17:28
jam1alexisb: fine with me17:31
jam1alexisb: it sounds like something that has a stakeholder, so it is important work, but not something that used to work that we broke17:31
alexisbnor something that should hold up 19.017:32
alexisbI think wallyworld__ moved it because he had 3 fixes he wanted to land but there are still additional issues that need to be investigated17:32
alexisbanyone else having issues with launchpad server?17:35
natefinchalexisb: the juju-core homepage opens ok for me, but I haven't really been on it all day17:38
alexisbnatefinch, I got logged out and am having troubles with the login page17:43
natefinchalexisb: haha, yeah, the login page looks borked: Something broke while generating the page. Please try again in a few minutes, and if the problem persists file a bug or contact customer support. Please quote OOPS-ID ['OOPS-a584d1910ee34d7d91342302107837b7']17:44
alexisbworked this last time though17:44
alexisbheh17:44
alexisbwell I am in now17:44
natefinchyep, me too17:45
alexisbinteresting got same type of oops trying to get back into email17:47
jam1alexisb: it has been flakey on and off today. You can probably go into #webops on irc.canonical.com and ask there when we have trouble like this. Apparently there is an issue with some of the Launchpad servers and some squid proxy machines17:50
alexisbgood pointer, thank you jam117:50
natefinchjam1: do we really need to support quantal?17:58
dimiternhttps://codereview.appspot.com/86600043 - reviews appreciated (VLAN cloudinit network setup for MAAS)18:00
jam1natefinch: is there a specific bug in quantal that we're trying to avoid?18:01
natefinchjam1: lack of mongo18:02
jam1natefinch: I don't have something that says "OMG, we must support Q for people". However, it sounds like more code than not to avoid it.18:03
jam1natefinch: so, given that nobody ever bothered to port mongo to Q and nobody has complained, I think its pretty clear where our support for it lise18:03
jam1lies18:03
jam1however, a *lot* of our test code uses Q as a "your not actually running the test suite on this otherwise 'supported' version"18:03
natefinchjam1: we have code that says "if you're running quantal, apt-add-repository ppa:juju/stable18:05
jam1natefinch: so if we have mongo, what is the problem?18:07
natefinchjam1: just more code to maintain, more special case tests to write18:07
natefinch(I was writing the test to check that quantal did the add-apt-repository)18:07
jam1natefinch: Quantal EOL's when Trusty is released, If I'm reading http://en.wikipedia.org/wiki/List_of_Ubuntu_releases#Ubuntu_12.10_.28Quantal_Quetzal.29 correctly18:08
jam1natefinch: so... I'd rather limp along and support it for the next release18:08
jam1but I wouldn't spend huge amounts of time on it.18:09
natefinchjam1: ok, that's fair18:10
=== vladk is now known as vladk|offline
* rogpeppe has reached eofd18:50
rogpeppeor eod even18:50
rogpeppeg'night all18:51
jam1anyone else able to "lbox propose" ? I get: error: Get https://api.launchpad.net/devel/people/+me: x509: certificate signed by unknown authority18:56
jam1I have the feeling LP changed their SSL certs due to Heartbleed, and now LBOX is refusing to let us get our work done.18:56
cmarsjam1, i just got that same x509 error w/lbox, then retried and it was successful. strange20:13
jam1cmars: maybe it depends what lp appserver you get20:14
jam1as there are like 16 of them20:14
jam1I did get it 2x in a row20:14
natefinchjam1: it worked for me this morning. haven't tried since then, though I've heard others have had problems sporadically20:14
jam1though I thought all the SSL stuff was handle in the Apache front end20:14
waiganimorning all21:24
alexisbmorning waigani21:25
davecheneymorning21:25
davecheneywaigani: thanks for trying to resubmit your branch21:25
davecheneymwhudson: do you have time for some deb hand holding ?21:26
waiganidavecheney: no problem, looks like mongod problem again?21:26
davecheneywaigani: that bot is screwed21:26
waiganihmmm21:26
davecheneyturns out our dogfood isn't fit for human consumption21:26
waiganihaha21:26
waiganidavecheney: fyi I'm working on this now: 130476721:27
mwhudsondavecheney: probably21:28
waiganihttps://bugs.launchpad.net/juju-core/+bug/130476721:28
_mup_Bug #1304767: test failure in cmd/juju <ppc64el> <juju-core:Triaged by waigani> <https://launchpad.net/bugs/1304767>21:28
davecheneywaigani: ok21:28
waiganibasically, I'll mock out a fake tarball so we are just testing tools uploading - not actually building21:29
davecheneywaigani: hmm21:30
davecheneyi don't think that is the right solution21:30
davecheneyit might eb21:30
waiganidavecheney: open to suggestions21:30
davecheneybe what I have found in is in many test cases they use the version.Current symbol21:31
davecheneybut the test then expects amd64 tools21:31
davecheneyid check that first21:31
waiganidavecheney: okay21:31
waiganidavecheney: wallyworld__and thumper pointed me in the direction of mocking out the tools tarball21:32
davecheneywaigani: thumper knows what he is doing21:32
waiganidavecheney: I'll look into both :)21:33
cmarsdavecheney, i've flipped the logic for mongojs tests and that's landed in trunk, re: 130477021:45
cmarsdavecheney, can you PTAL at landing this in  https://codereview.appspot.com/86650043/?21:45
cmars^in 1.18 i mean21:46
davecheneycmars: so be it21:48
davecheneynobody will read that21:48
davecheneyand we'll get bug reports21:48
davecheneybut so be it21:48
davecheneyuntil the store moves21:48
davecheneythat is21:48
davecheney*not very subtle hint*21:48
davecheneyit's not a big problem, nobody runs the tests but us21:49
cmarssounds like a topic for discussion at the next sprint. for now, this is a way to unblock CI21:50
davecheneycmars: roger21:51
davecheneycmars: thanks for getting that fix in21:54
cmarsnp21:54
davecheneyffs22:11
davecheneythe bot is really screwed22:11
davecheneyevery single change has to be landed several times before it sticks22:11
davecheneyhttp://paste.ubuntu.com/7232727/22:32
davecheneymongo is blowing up :(22:32
davecheneycan anyone connect to the bot and see if there are like 1,000 mongod processes leaking around22:33
thumperhazmat: where did you look for the source of juju-mongodb again?22:46
thumperhazmat: I'm looking at bug 130274722:47
_mup_Bug #1302747: mongodb fails to start with local provider  <mongodb> <juju-core:Incomplete> <https://launchpad.net/bugs/1302747>22:47
wallyworld__davecheney: not 1000, but maybe 522:47
wallyworld__i'll nuke them22:47
davecheneywallyworld__: ta22:47
davecheneyi normally find a few doen lurking around by the EOD22:47
hazmatthumper, i went to github  lsat time22:47
hazmatthumper, mongodb upstream that is22:48
thumperkk22:48
hazmatanyone seen juju machines pending forever on aws?23:15
hazmathttp://paste.ubuntu.com/7232845/ machine 8 fwiw machine-0 log23:17
hazmatpending in status for 20m23:17
* hazmat switches to stable branch tip23:22
cmarshttps://codereview.appspot.com/86650043/, can I get a look (backport to 1.18)23:25
cmarsneeds a LGTM23:25

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!