| lifeless | zopelessappserverlayer | 00:09 |
|---|---|---|
| lifeless | wgrant: you're going to laugh at this | 00:19 |
| lifeless | wgrant: in between the tears. | 00:19 |
| wgrant | I hope so. | 00:19 |
| lifeless | wgrant: ZASL runs up the codehosting server. | 00:19 |
| lifeless | wgrant: that calls into private-xmlrpc | 00:19 |
| lifeless | wgrant: thats raising oopses because of disconnected stores | 00:19 |
| wgrant | ZASL itself shouldn't start codehosting... | 00:20 |
| wgrant | Unless it's pretty different from AppServerLayer, which it shouldn't be since I made ZopelessLayer and FunctionalLayer almost equal. | 00:20 |
| lifeless | wait, there is more | 00:21 |
| lifeless | we also get a disconnectionerrror: terminating connection due to administrator command | 00:22 |
| wgrant | Where are you running this? | 00:22 |
| lifeless | wgrant: -> because the test db reset code pulls the db out from under the slave appserver | 00:22 |
| wgrant | Locally? | 00:22 |
| lifeless | wgrant: yes, totally reproducable | 00:22 |
| wgrant | You need the new storm. | 00:22 |
| lifeless | on lucid ? | 00:22 |
| wgrant | Yes. | 00:22 |
| wgrant | libpq5 8.4.9 breaks everything. | 00:23 |
| wgrant | Destroyed buildbot on Friday. | 00:23 |
| wgrant | ec2 isn't affected yet. | 00:23 |
| lifeless | this is 8.4.3-1 | 00:23 |
| wgrant | New Storm is in devel now. | 00:23 |
| wgrant | Ah. | 00:23 |
| wgrant | Maybe your analysis is correct, then. | 00:23 |
| lifeless | maybe :) | 00:23 |
| lifeless | the reason it happens only when two tests are run | 00:23 |
| lifeless | is that we have a brand new appserver if we only run one test | 00:24 |
| wgrant | Yep. | 00:24 |
| wgrant | Still, this looks similar to the libpq5 issue... | 00:24 |
| wgrant | Exactly the same thing. | 00:24 |
| wgrant | librarian was broken after the first test. | 00:25 |
| lifeless | how so ? | 00:25 |
| lifeless | I thought that was it not detecting the disconnect ? | 00:25 |
| wgrant | Right. | 00:25 |
| lifeless | this detects it fine | 00:25 |
| wgrant | If this is a DisconnectionError, it's probably working. | 00:25 |
| lifeless | its raised from storm.database line 376 _check_disconnect | 00:25 |
| lifeless | yes, it is detecting it | 00:25 |
| lifeless | we're just oopsing - correctly - because the db was yanked without telling the appserver | 00:26 |
| lifeless | so I should say, rather than 'codehosting is started' the test using bzr transports and these are backing onto xmlrpc requests in some fashion | 00:28 |
| lifeless | and bingo | 00:29 |
| lifeless | the db is being dropped after the test, rather than the sequences reset | 00:29 |
| lifeless | I'm now totally confident of the analysis, but wtf to do about it | 00:29 |
| lifeless | I think I'll expect 1 or 9 oopses | 00:30 |
| lifeless | along with an XXX about this | 00:30 |
| lifeless | wgrant: as my victim^Wreviewer, how do you feel about this | 00:30 |
| wgrant | lifeless: The DB will be dropped if it's dirty. | 00:34 |
| wgrant | I think that's probably fine. | 00:34 |
| wgrant | As long as it is really 1 or 9 :) | 00:34 |
| lifeless | bug 884036 | 00:35 |
| _mup_ | Bug #884036: db layer test resets interact badly with slave appserver instances <Launchpad itself:Triaged> < https://launchpad.net/bugs/884036 > | 00:35 |
| lifeless | see if that convinces you | 00:36 |
| lifeless | second oops added now | 00:36 |
| wgrant | Sounds good. | 00:39 |
| wgrant | lifeless: https://code.launchpad.net/~wgrant/launchpad/cte-it-up/+merge/80767 | 00:43 |
| wgrant | 2.5s->1.6s on qastaging | 00:44 |
| lifeless | wgrant: does this change +bugs as well? or just BugTask:* ? | 00:44 |
| wgrant | lifeless: +bugs as well. Do you want that checked too? | 00:45 |
| lifeless | uhm yes. | 00:45 |
| lifeless | just cause, well, EPIC TIMEOUTS | 00:46 |
| wgrant | It should in theory be a significant improvement. But let's see. | 00:46 |
| lifeless | rev 14156 of my branch is up; Could you please incremental review anything you haven't, and I'll land. | 00:46 |
| wgrant | Ermm. | 00:47 |
| wgrant | This isn't landable... | 00:47 |
| wgrant | There is no rabbit on prod, and the fallback is flaky at best. | 00:47 |
| lifeless | with no rabbit it will behave as it did befre | 00:48 |
| lifeless | *before* | 00:48 |
| wgrant | And the moment we have rabbit, all OOPSes will disappear. | 00:48 |
| wgrant | Also, BSON. | 00:48 |
| lifeless | whats flaky about the fallback ? | 00:48 |
| wgrant | If rabbit is up but amqp2disk hasn't set up its queue bindings, OOPSes go to nowhere. | 00:48 |
| wgrant | And that's just what I've noticed. | 00:48 |
| lifeless | thats how loosely coupled works. | 00:49 |
| wgrant | Yes, but it's not how production should work. | 00:49 |
| lifeless | Its not a defect, its expected. I'm hoping to get that side of it sorted later today. | 00:49 |
| wgrant | eg. qastaging and staging have rabbit, no amqp2disk. | 00:49 |
| lifeless | we can disable the config for them. Or get amqp2disk up and live. | 00:49 |
| wgrant | Do that before landing. | 00:49 |
| lifeless | sure | 00:50 |
| lifeless | but whats flaky? | 00:50 |
| lifeless | or was that code for 'new world order does not have the same guarantees' ? | 00:50 |
| wgrant | It only works if rabbit is entirely unconnectable, and it's not exactly well-tested. | 00:50 |
| lifeless | wgrant: its tested in oops-amqp; the tests there pull rabbit out from under it | 00:51 |
| lifeless | wgrant: rabbit config isn't, and shouldn't, be the business of the oops producer | 00:51 |
| wgrant | Sure. | 00:51 |
| wgrant | But as your branch is now, as soon as rabbitmq is live on production then we lose all error reporting. | 00:51 |
| lifeless | IFF the queue isn't setup | 00:52 |
| wgrant | Which it probably won't be. | 00:52 |
| lifeless | separate issue; we can toggle off the oops exchange very very easily. | 00:52 |
| lifeless | (in prodconfigs) | 00:52 |
| wgrant | lol prodconfigs easily | 00:52 |
| lifeless | more easily than landing this | 00:53 |
| lifeless | change, push, review, pqm-submit. | 00:53 |
| wgrant | lifeless: Why is UnknownRemoteError now permitted in test_runner.py? | 00:55 |
| lifeless | wgrant: because python OOM is nondeterministic | 00:56 |
| lifeless | wgrant: I ran it in a loop and didn't get MemoryError every time | 00:56 |
| wgrant | But that invalidates the test... | 00:56 |
| wgrant | Also, isn't the setUpQueue call in sync() racy? | 00:57 |
| lifeless | I can put it back, but I expect some buildbot failures as a result | 00:57 |
| wgrant | I don't see why it needs a new queue. | 00:57 |
| lifeless | wgrant: because auto_delete | 00:57 |
| wgrant | They would be legitimate failures. | 00:57 |
| wgrant | The current test is not flaky. | 00:57 |
| wgrant | Ah. | 00:58 |
| wgrant | Fair point. | 00:58 |
| lifeless | wgrant: I bet it is | 00:58 |
| wgrant | Still racy, but acceptable. | 00:58 |
| wgrant | lifeless: It's never been a problem before. | 00:58 |
| lifeless | wgrant: poolie ran into a race the other day, last happened 4 years ago. | 00:58 |
| lifeless | wgrant: doesn't make it less of a race | 00:58 |
| wgrant | Even so. | 00:58 |
| lifeless | I can put it back. | 00:58 |
| wgrant | It makes it an invalid test. | 00:58 |
| lifeless | But I think the test lies today | 00:58 |
| lifeless | the test isn't valid | 00:58 |
| poolie | lifeless: two races | 00:58 |
| wgrant | Better to have it failing and then disabled. | 00:59 |
| wgrant | Than left entirely invalid. | 00:59 |
| poolie | i guess it is the spring racing carnival in launchpad too | 00:59 |
| wgrant | lifeless: Perhaps it's because you were running it i386? | 00:59 |
| StevenK | wallyworld_: I've made the changes you mentioned for https://code.launchpad.net/~stevenk/launchpad/hide-create-new-ppa-open-team/+merge/80650 . Can you have another look so I can ec2 it? | 01:00 |
| lifeless | wgrant: yes, I am | 01:00 |
| lifeless | wgrant: I have no particular interest in this; the runner code doesn't generate MemoryError synthetically, by checking for ulimit violations etc - it depends on the python runtime only ever raising MemoryError AFAICT (grepping for MemoryError ..) | 01:01 |
| lifeless | wgrant: so pick a preferred resolution, and I'll do it. | 01:01 |
| wgrant | lifeless: Your INTERACTIVE_TESTS change concerns me. Doesn't that mean we're just not going to see any oopses from the appserver? | 01:02 |
| lifeless | wgrant: myself, I'm quite sure that we will eventually hit whatever point in CPython caused UnknownRemoteError | 01:02 |
| wgrant | lifeless: Sure, but I'd prefer to have a flaky or disabled test, rather than one that doesn't actually test. | 01:03 |
| lifeless | wgrant: no tests look for them, and its *only* yuixhr that uses this code path | 01:03 |
| lifeless | wgrant: depends on the intent of the test; I think 'raises promptly' is sufficiently narrow. | 01:03 |
| wgrant | Oh, it's in testapp, right. | 01:03 |
| lifeless | wgrant: we don't depend on catching MemoryError anywhere (damn straight) | 01:03 |
| lifeless | wgrant: so the -only- impact on broadening the exceptions is that the test suite lines up with the OOPS reports | 01:04 |
| lifeless | wgrant: [if/when we see this in prod] | 01:04 |
| wgrant | lifeless: Hm, UnknownRemoteError is probably txamqp... | 01:04 |
| lifeless | wgrant: txamp do you mean ? | 01:04 |
| wgrant | Yes. | 01:04 |
| wgrant | I fail. | 01:04 |
| wgrant | So, I'd prefer it if you reverted that change. | 01:05 |
| wgrant | If it is a problem, we can unrevert it. | 01:05 |
| lifeless | it is in Twisted AMP yes | 01:05 |
| lifeless | sure, I will (it is a problem) | 01:05 |
| wgrant | It's not a problem. | 01:06 |
| wgrant | It may be on i386, but nobody's noticed that before :) | 01:06 |
| lifeless | let me just thrash carob's pagecache | 01:08 |
| wgrant | Oh? | 01:09 |
| lifeless | launchpad.net-logs$ grep UnknownRemoteError . -r | 01:09 |
| wgrant | Ah | 01:10 |
| wgrant | That seems unwise... | 01:10 |
| wgrant | It's like 150GB or more. | 01:10 |
| wgrant | Or was it 250GB | 01:10 |
| wgrant | I can't quite remember. | 01:10 |
| lifeless | serial IO ftw | 01:10 |
| wgrant | I still contend that carob's disks are actually a NAS in NZ. | 01:10 |
| lifeless | I could grep faster if they were | 01:11 |
| lifeless | anything else ? | 01:11 |
| * StevenK prods wallyworld_. | 01:12 | |
| wallyworld_ | ouch | 01:12 |
| StevenK | [12:00] < StevenK> wallyworld_: I've made the changes you mentioned for https://code.launchpad.net/~stevenk/launchpad/hide-create-new-ppa-open-team/+merge/80650 . Can you have another look so I can ec2 it? | 01:13 |
| wallyworld_ | sure. | 01:13 |
| wgrant | lifeless: I think that's about it. | 01:13 |
| lifeless | ok, so I'll push th revert to test_memory_hob_job, toss at ec2, and we'll see what we see | 01:15 |
| wallyworld_ | StevenK: '%s/+activate-ppa' % canonical_url(team) -> canonical_url(team, view_name='+activate-ppa') | 01:15 |
| wallyworld_ | other than that, looks good thanks | 01:17 |
| lifeless | wgrant: can as approved vote? | 01:17 |
| StevenK | wallyworld_: Thank you for the Approve. | 01:20 |
| wallyworld_ | np. | 01:20 |
| wgrant | lifeless: Done. | 01:20 |
| lifeless | thanks | 01:24 |
| wgrant | lifeless: Erm. | 01:25 |
| wgrant | What did I say about not landing this. | 01:26 |
| wgrant | I approved mostly given that you were going to get at least (qa)staging fixed first :) | 01:26 |
| wgrant | (and no, "It'll be done soon" is not valid) | 01:26 |
| lifeless | wgrant: I'm working on that right now | 01:31 |
| wgrant | If it's not done by the time ec2 finishes, please kill the ec2 instance. | 01:31 |
| lifeless | that would be a waste | 01:32 |
| wgrant | Let's not get into another 2-month cart-before-the-horse situation like we did with fastdowntime. | 01:32 |
| lifeless | if its not ready I'll change prodconf to disable the key in staging | 01:32 |
| lifeless | wgrant: you exaggerate there | 01:32 |
| wgrant | No... | 01:32 |
| lifeless | wgrant: I'm not about to block the deployment pipeline for any extended period | 01:32 |
| lifeless | wgrant: I'm well aware of its impact, given I've done most of the analysis and discussion to demonstrate that | 01:33 |
| wgrant | You are about to significantly hinder the crisis analysis process, though. | 01:33 |
| lifeless | I'm about to change the exact recipe needed | 01:33 |
| lifeless | grep still works, data can still be easily view | 01:33 |
| lifeless | ed | 01:33 |
| wgrant | No, grep doesn't still work. | 01:34 |
| wgrant | I can't grep for ^Exception-Type: RequestExpired now. | 01:34 |
| lifeless | yes it does, bson stores strings as length: prefix, so you can do exactly that. | 01:34 |
| lifeless | just phrased -very slightly- differently. | 01:34 |
| wgrant | I am close to rescinding my approval. | 01:35 |
| lifeless | why? | 01:36 |
| wgrant | 1) all (qa)staging OOPSes will be lost unless you sort things out in the next few hours. | 01:36 |
| lifeless | 1) hystrionics, I've outlined 2 solid workarounds and taken responsibility for having them sorted. | 01:37 |
| wgrant | 2) oops-tools is mostly useless, and this breaks shell pipelines. | 01:37 |
| lifeless | 2) has had 3? more? weeks for folk to respond to the get-out-and-try aspect, and there are also recipes in that same thread to deal. | 01:37 |
| wgrant | We already had three expressions of disapproval. | 01:38 |
| wgrant | In addition to my initial one. | 01:38 |
| lifeless | aaron asked why; stuart noted about readability with an anecdote from ISD who are doing noone-knows-what, and gary was -0.5 accepting my argument about greppability | 01:41 |
| lifeless | I have been trying and playing around, and I'm satisfied I can do the analyses I was previously doing. | 01:43 |
| lifeless | Have you experimented, or are you just concerned about the possible impact? | 01:43 |
| wgrant | I know that my usual method for analysing performances crises now has to be rewritten in Python, or invoke Python thousands of times. | 01:44 |
| lifeless | how do you know that ? | 01:45 |
| wgrant | Because it relies on selecting on and extracting fields and parts of fields from OOPSes. | 01:45 |
| lifeless | so, its an assumption. Thats ok, but its useful to be clear. | 01:46 |
| lifeless | you are assuming, without trying, that grep and sed can't extract what you want from bson. | 01:47 |
| lifeless | you may be right, you may not be. | 01:47 |
| lifeless | I'm more worried about the url interactions than the change to bson; oops-tools has shocking understanding of urls-are-text | 01:49 |
| lifeless | for instance our appserver generated urls are unicode | 01:49 |
| wgrant | I'm not too concerned about how it interacts with oops-tools. | 01:49 |
| lifeless | so we may end up double encoding for a day or two, which is tolerable but annoying | 01:49 |
| wgrant | oops-tools isn't critical, and can be fixed. | 01:49 |
| wgrant | lifeless: +bugs seems to be much faster on qastaging, with the CTE patch. | 01:53 |
| wgrant | FTI Ubuntu searches are still timing out, but they were before too so I suspect the index is just cold. | 01:53 |
| lifeless | ok. | 01:53 |
| lifeless | uhm | 01:53 |
| lifeless | just for insurance, lets FF this | 01:53 |
| lifeless | its a particularly sensitive bit of code | 01:53 |
| wgrant | k | 01:54 |
| wgrant | It's all about to become a lot simpler, but until then... | 01:54 |
| lifeless | I'm sure the FF will be overkill | 01:55 |
| lifeless | but I'll feel a whole lot happier having it :) | 01:55 |
| wgrant | Indeed. | 01:55 |
| wgrant | test_bugtask_search is terrible :( | 02:08 |
| wgrant | A slow base test case with dozens of derivatives. | 02:08 |
| lifeless | yes | 02:09 |
| wgrant | lifeless: It's now behind a flag, with a few strategic tests duplicated to test both paths. The new function is ugly as hell, but at least it should be safe :) | 02:16 |
| lifeless | wgrant: thank you very much! | 02:16 |
| wgrant | Bah, but now I have to ec2 it :( | 02:17 |
| lifeless | what could possibly go wrong | 02:17 |
| wgrant | The FF check will probably introduce a new query sometimes :( | 02:17 |
| wgrant | Which could break some query limit tests. | 02:17 |
| lifeless | most of them have some deliberate fat | 02:18 |
| lifeless | with a generous backstop, but yes, it could | 02:18 |
| wgrant | I've been meaning to do a test run with the matcher ensuring a lower bound as well. | 02:19 |
| wgrant | Some of them are waaaaay high now. | 02:19 |
| StevenK | Bah, I can't have my change in create_initialized_view, too many things can't be canonical_url()'d. | 02:27 |
| poolie | what's a CTE? | 03:22 |
| wgrant | poolie: Common Table Expression | 03:25 |
| wgrant | Defined by the WITH clause. | 03:25 |
| poolie | ok | 03:27 |
| StevenK | wgrant: Is QA for r14209 on your list this afterrnoon? | 03:36 |
| wgrant | StevenK: That's the userCanView change? | 03:37 |
| StevenK | Yeah | 03:37 |
| wgrant | It's semi-ok. | 03:37 |
| StevenK | qa-ok-ish | 03:37 |
| StevenK | ? | 03:37 |
| wgrant | Liveable, but not ideal. I'm hoping that an improvement will land before someone tries to deploy. | 03:38 |
| StevenK | We have a large amount of European QA before that rev. | 03:38 |
| wgrant | Yes. | 03:38 |
| lifeless | http://research.microsoft.com/en-us/news/features/nagappan-100609.aspx is a good read | 04:08 |
| poolie | can someone quickly review http://readthedocs.org/docs/judge/en/latest/ for me | 04:09 |
| poolie | lifeless: that is interesting | 04:11 |
| poolie | it would be interesting to study: how good are developers at predicting which factors drive quality on their projects | 04:12 |
| lifeless | indeed | 04:22 |
| poolie | what should i do now? | 04:28 |
| lifeless | partay | 04:29 |
| nigelb | heh | 04:30 |
| nigelb | morning! :) | 04:30 |
| poolie | hi nigelb | 04:35 |
| nigelb | Hey poolie | 04:37 |
| spm | o/ nigelb | 04:47 |
| nigelb | ohai spm :) | 04:47 |
| nigelb | Ugh, I now officially hate the expiring membership email. | 05:24 |
| spm | yeah? ok then. INSERT into teams VALUES nibelb WHERE team has expiringing_membership; | 05:50 |
| spm | spelling to be optional. | 05:50 |
| StevenK | spm: Let me give you a cronjob that will reset nigelb's memberships to expire in seven days. Every day. | 05:51 |
| spm | StevenK: approved. | 05:51 |
| spm | this wouldn't be excessive abuse of power would it? :sincerity: | 05:51 |
| nigelb | hah | 06:13 |
| nigelb | StevenK, spm: Love you too :P | 06:13 |
| spm | :-) | 06:14 |
| lifeless | wgrant: you said txlongpoll was bust | 07:00 |
| lifeless | wgrant: in that it didn't accept a queue name prefix, right ? | 07:00 |
| wgrant | lifeless: Yes. | 07:00 |
| lifeless | is bigjools aware of this? And that its a blocker to the deploy ? | 07:00 |
| wgrant | I believe Red is aware. | 07:00 |
| wgrant | However, AFAIK they are unaware that anybody is using rabbit except them. | 07:01 |
| wgrant | So they are probably unaware that the fix is time-critical. | 07:01 |
| lifeless | wgrant: there is no bug on https://bugs.launchpad.net/txlongpoll | 07:06 |
| lifeless | wgrant: as you know the details, could you make one please? | 07:07 |
| wgrant | Hmm, rvba said he was going to file one. | 07:08 |
| wgrant | I left it with him, since it was well after midnight. | 07:08 |
| lifeless | the rt claims that read is only on ^longpoll.* | 07:08 |
| lifeless | so there is access control in place | 07:09 |
| wgrant | That is a start. | 07:09 |
| wgrant | Personally I'd prefer the old behaviour to be revived. | 07:10 |
| lifeless | I'm not sure what that is; thus my encouragement for you to file a bug :) | 07:11 |
| lifeless | hmm | 07:11 |
| lifeless | my branch has been running from 0140 through to 0708 | 07:12 |
| lifeless | I suspect a stupidly overloaded host | 07:12 |
| wgrant | lifeless: I just had a branch take 4:45 | 07:12 |
| wgrant | Which is longer than normal. | 07:12 |
| lifeless | hmm | 07:13 |
| lifeless | I suspect that that sync() in cleanUp is a problem | 07:13 |
| lifeless | I have another trace here that took 6.5 hours | 07:13 |
| lifeless | I'll land a branch immediately to remove the trailing sync() | 07:13 |
| lifeless | [given I only added it while debugging that rosetta zopelessappserver test issue, I know its not needed know] | 07:14 |
| wgrant | Ah. | 07:14 |
| lifeless | its the only think I can think of that would add 100% test time | 07:15 |
| lifeless | unless the rabbit layer reset is heinous, but I didn't change that, nor add it to more than a few dozen tests | 07:16 |
| lifeless | wgrant: I suspect both the sync and the existence/presence of poor ec2 loading | 07:18 |
| lifeless | wgrant: did we decide setting mandatory wouldn't work ? | 07:19 |
| wgrant | lifeless: We can't necessarily receive the response. | 07:20 |
| lifeless | wgrant: did we do an experiment ? | 07:20 |
| wgrant | No. | 07:20 |
| * StevenK has clearly forgotten how to drive subunit | 07:21 | |
| wgrant | But we'd need to asynchronously detect the basic.return, and then $something. | 07:21 |
| wgrant | But I don't think amqplib exposes basic.returns. | 07:21 |
| StevenK | I have a subunit stream and I want to see which tests failed or gave an error | 07:21 |
| lifeless | StevenK: subunit-filter --no-skip --no-passthrough | subunit-ls | 07:21 |
| lifeless | StevenK: or testr load < stream; testr failing | 07:22 |
| lifeless | (or testr failing --list) | 07:22 |
| lifeless | StevenK: or tribunal < stream | 07:22 |
| StevenK | subunit-filter --no-success --no-passthrough --error --failure < lp.log | subunit-ls | 07:22 |
| StevenK | That gives me a traceback | 07:22 |
| lifeless | StevenK: you're making it hard for yourself :) | 07:23 |
| lifeless | StevenK: 3 of those options are the defaults | 07:23 |
| lifeless | StevenK: also whats the tb you get? | 07:23 |
| StevenK | Broken pipe | 07:24 |
| lifeless | subunit-filter is probably erroring | 07:24 |
| StevenK | So one of subunit-filter or subunit-ls is misbehaving | 07:24 |
| lifeless | try one of my recipes above | 07:24 |
| === jbepsilon_ is now known as jbepsilon | ||
| lifeless | StevenK: ? | 07:29 |
| StevenK | Sorry, got distracted. Your recipe works fine | 07:30 |
| StevenK | "subunit-filter --no-success --no-passthrough --error --failure < lp.log 1>/dev/null" doesn't produce any output on stderr | 07:30 |
| lifeless | its a legitimate set of options | 07:31 |
| lifeless | I'm not sure why you had a tb | 07:31 |
| lifeless | conceivably the pipe was closed with no output I guess | 07:32 |
| lifeless | it works for me, fwiw | 07:32 |
| StevenK | lifeless: http://pastebin.ubuntu.com/724060/ | 07:33 |
| lifeless | wgrant: so, you'll file a bug| get rvba to ? | 07:33 |
| lifeless | StevenK: the first tb is the issue | 07:33 |
| wgrant | lifeless: I'll ask rvba for the latest news. | 07:33 |
| lifeless | StevenK: Can you reproduce with a current subunit ? | 07:33 |
| lifeless | StevenK: (what does dpkg -l python-subunit say) | 07:33 |
| StevenK | lifeless: 0.0.6-3 | 07:34 |
| lifeless | try 0.0.7 | 07:35 |
| StevenK | Such a pity, Mr Bond. | 07:35 |
| nigelb | heh | 07:38 |
| jelmer | lifeless: that reminds me, can you add a tag for 0.0.7 on lp:subunit? | 07:50 |
| lifeless | jelmer: probably, there isn't one ? | 07:51 |
| lifeless | jelmer: it'll be just tip | 07:51 |
| lifeless | jelmer: :!bzr log -r -1 | 07:51 |
| lifeless | ------------------------------------------------------------ | 07:51 |
| lifeless | revno: 152 | 07:51 |
| lifeless | tags: 0.0.7 | 07:51 |
| jelmer | lifeless: "bzr tags -d lp:subunit" doesn't list a 0.0.7 | 07:51 |
| jelmer | lifeless: perhaps that tag is local? | 07:51 |
| lifeless | jelmer: if so, I blame bzr | 07:52 |
| lifeless | jelmer: hmm, diverged branches. funky. | 07:53 |
| lifeless | yes, somethings up. I'm going to push --overwrite and then merge trunk back in | 07:54 |
| lifeless | jelmer: there should be a tag now | 07:57 |
| jelmer | lifeless: great, thanks! | 07:57 |
| lifeless | useoops has landed \o/ | 08:17 |
| wgrant | Has the fix landed? | 08:20 |
| bigjools | My next branch is going to be called The Eagle | 08:23 |
| nigelb | Just to announce its landing? | 08:23 |
| bigjools | correct | 08:23 |
| nigelb | I'll call dibs on Airforce one and Canonical one ;) | 08:24 |
| nigelb | Who's at UDS? francis, matt, dan..and? | 08:26 |
| bigjools | that's about it IIRC | 08:28 |
| bigjools | oh one of the US guys too | 08:28 |
| adeuring | good morning | 08:32 |
| lifeless | wgrant: its about to start ec2 | 08:32 |
| wgrant | nigelb: I thought it was Francie, Matt, Deryck and Curtis. Didn't know Dan was going too. | 08:45 |
| lifeless | new kid, doing the rounds :) | 08:46 |
| lifeless | bigjools: is rvba in today? | 08:54 |
| bigjools | lifeless: doesn't look like it | 09:01 |
| nigelb | bigjools / wgrant - Thanks :) | 09:16 |
| bigjools | I'd quite like to be there again, was the best UDS hotel in ages | 09:16 |
| nigelb | heh | 09:17 |
| nigelb | I wish I could be there. I regret skipping this UDS> | 09:17 |
| nigelb | bigjools: Will you be at the next one? :) | 09:18 |
| nigelb | Too bad I missed you last UDS. | 09:19 |
| bigjools | you walked right past me there :) | 09:19 |
| nigelb | :( | 09:19 |
| nigelb | That's the UDS where I asked Francis "Are you a launchpad guy?" :D | 09:19 |
| bigjools | as for the next one, no idea, we don't decide until shortly before who's going | 09:19 |
| nigelb | Ah! | 09:20 |
| \sh | moins | 09:20 |
| \sh | dear lp-devs how difficult will it be, to change someones launchpad id from X to Y ? | 09:21 |
| nigelb | Its easy to do the change. | 09:21 |
| nigelb | THe repurcusions might make some time to fix ^-^ | 09:21 |
| bigjools | \sh: the answer is "it depends" | 09:21 |
| nigelb | And its not a launchpad problem. Its an SSO thing. | 09:21 |
| wgrant | SSO actually steals that data from us. | 09:22 |
| wgrant | It's a Launchpad thing. | 09:22 |
| wgrant | \sh: Unless you have a PPA, you can just change it. | 09:22 |
| \sh | bigjools, let's say, I would like to request to change my launchpad-id from shermann to sadig | 09:22 |
| bigjools | \sh: then as wgrant says | 09:22 |
| nigelb | wgrant: Meh. I think most of the time its our openid client which isn't implemented properly. | 09:23 |
| \sh | wgrant, ok...my ppa is not a problem...so when I delete my ppa I can just change it? | 09:23 |
| bigjools | if you have any PPAs then they need to be deleted | 09:23 |
| bigjools | then yes | 09:23 |
| wgrant | \sh: Yes. May have to wait up to 10 minutes for it to be completely deleted. | 09:23 |
| lifeless | \sh: delete the ppa, wait for a publishing cycle, visit +edit and rename it | 09:23 |
| lifeless | I think its +edit | 09:23 |
| \sh | wgrant, lifeless , thx...I'm waiting now ;) | 09:24 |
| * bigjools has a buildd-manager that will cancel builds \\o/ | 09:24 | |
| StevenK | bigjools: Does it also impact on the production builders too :-P | 09:25 |
| jelmer | bigjools: niiice! | 09:25 |
| wgrant | bigjools: You know that's been possible for recipe builds for a year, right? :P | 09:25 |
| bigjools | whatever | 09:25 |
| lifeless | running out of memory != cancelling :) | 09:26 |
| bigjools | lol | 09:26 |
| nigelb | ZING. | 09:26 |
| wgrant | They have a cancel button which sets them to SUPERSEDED. Which actually works pretty well. | 09:26 |
| bigjools | yes but it doesn't rip the build off the builder | 09:26 |
| jelmer | wgrant: that doesn't kill the in-progress build though, right? | 09:26 |
| bigjools | we can set it to CANCELLED now | 09:27 |
| wgrant | I believe it deletes the BQ too, which would kill the in-progress build. | 09:27 |
| nigelb | So, the ones that are scheduled can be canceled? | 09:27 |
| nigelb | Or is it the ones that are building that can be canceled. | 09:27 |
| bigjools | interesting way of doing it | 09:28 |
| bigjools | lifeless: yay for the OOPS changes! Did you document what you said in the email in the dev wiki? | 10:15 |
| lifeless | bigjools: all relevant docs were updated | 10:16 |
| lifeless | bigjools: AFAIK | 10:16 |
| lifeless | bigjools: however, there weren't many ;) | 10:16 |
| bigjools | awesome | 10:16 |
| lifeless | bigjools: https://dev.launchpad.net/QA/OopsToolsSetup is perhaps the least likely known by ye old average LP dev | 10:17 |
| bigjools | well - the main thing is to note the stuff you wrote in the email, for posterity | 10:17 |
| bigjools | I'm going on a documentation jihad in the near future | 10:17 |
| lifeless | bigjools: I've given https://dev.launchpad.net/LoggingOopses a bit of a facelift too | 10:44 |
| bigjools | lifeless: great! | 10:45 |
| lifeless | ok, thats witching hour, time to EOD for a bit :) | 11:00 |
| nigelb | aaah. | 11:26 |
| nigelb | I now understand the outcome of comparing unicode incorrectly. | 11:26 |
| nigelb | The unicode entry goes to the end of the list :( | 11:27 |
| wgrant | Yep. | 11:28 |
| nigelb | Nice that danilos is our unicode tester :D | 11:28 |
| nigelb | Is there a better way to do that than what I did? | 11:28 |
| wgrant | Sadly we are left with only rvba. | 11:28 |
| wgrant | And his name is not very Unicode at all. | 11:28 |
| nigelb | heh | 11:30 |
| nigelb | Is there better way to sort unicode at the current point? | 11:30 |
| nigelb | I remember jtv's argument at that time being more of future oriented. | 11:30 |
| lifeless | need unicode local in the DB and appservers | 11:35 |
| lifeless | presumably not /hard/ | 11:35 |
| lifeless | but needs doing + qa | 11:35 |
| nigelb | ah | 11:36 |
| nigelb | Nothing I an do I suppose? | 11:36 |
| nigelb | *can | 11:37 |
| cjwatson | bigjools: cancelled> ooh, awesome | 11:40 |
| === allenap changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: allenap | Critical bugtasks: 266 | ||
| === benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: allenap, benji | Critical bugtasks: 266 | ||
| lifeless_ | nigelb: you could do it | 12:15 |
| lifeless_ | nigelb: but indirectly; you'd need to file bugs / rt's to find out the exact current config, reproduce it, see what you want, test that, get confidence etc | 12:15 |
| nigelb | ah | 12:16 |
| bigjools | cjwatson: only for virtual builds right now though, don't get too excited | 12:55 |
| bigjools | need to fix sbuild for non-virt | 12:55 |
| cjwatson | bigjools: ah well, still a step forward | 12:58 |
| bigjools | yes, 90% of work is done | 12:58 |
| bigjools | s/is/will be/ :) | 12:59 |
| bigjools | LinkedIn - just PO please | 12:59 |
| === bac` is now known as bac | ||
| cjwatson | Wow. The publisher has only just got to around cron.germinate | 14:49 |
| cjwatson | We're in serious danger of going past an hour here :-/ | 14:49 |
| bigjools | cjwatson: :( | 14:52 |
| jcsackett | sinzui: free to mumble? | 15:08 |
| sinzui | yes | 15:08 |
| jcsackett | excellent. | 15:09 |
| danhg | Hey everyone at UDS, I'm Dan, the new Usability & Communications Specialist at Launchpad. I'm going to be doing lots of usability testing today, so please ping me if you're around, and would like to get involved! | 15:23 |
| nigelb | danhg: #ubuntu-uds is also a nice channel to advertise :) | 15:25 |
| danhg | thanks nigelb, good plan! | 15:26 |
| danhg | All week actually - so let me know when you're free guys! | 15:27 |
| bigjools | cjwatson: yeah, it missed the run at :00 | 15:48 |
| bigjools | domination took 30m | 15:52 |
| bigjools | I figured this would happen with the arch-all changes | 15:52 |
| cjwatson | this is pretty bad - what can be done? | 15:56 |
| cjwatson | don't get me wrong, I appreciate the new feature, but ... | 15:56 |
| === dpm__ is now known as dpm | ||
| === mrevell__ is now known as mrevell | ||
| mrevell | jcsackett, yo | 16:06 |
| jcsackett | mrevell: hello. | 16:06 |
| mrevell | danilos, What's the rally project management session later all about? | 16:12 |
| mrevell | jcsackett, Oh hey, we're just trying to find a good time to talk to you. When suits you? | 16:12 |
| jcsackett | pretty much anytime today is fine. i imagine your schedule is a bit more bumpy than mine. | 16:12 |
| jcsackett | mrevell ^ | 16:14 |
| jcsackett | mrevell: danhg has proposed now, which works for me. | 16:16 |
| danhg | Cool | 16:17 |
| mrevell | jcsackett, Oh hey, we're just trying to find a good time to talk to you. When suits you? | 16:20 |
| jcsackett | mrevell: i think you meant perhaps to send another message, as that one's a repeat? :-P | 16:21 |
| mrevell | jcsackett, I have no idea how that happened. Hah :) | 16:21 |
| === beuno is now known as beuno-lunch | ||
| mrevell | jcsackett, I think danhg is talking to you about it. | 16:21 |
| jcsackett | he is indeed. | 16:21 |
| jml | awk: (FILENAME=lib/lp/contrib/javascript/yui3-gallery/gallery-accordion/gallery-accordion.js FNR=2916) fatal: cannot open file `lib/lp/contrib/javascript/mustache.js' for reading (No such file or directory) | 16:59 |
| jml | make: *** [lib/canonical/launchpad/icing/build/launchpad.js] Error 2 | 16:59 |
| jml | just got that running 'make schema' | 16:59 |
| bigjools | jml: update source? | 17:02 |
| lifeless_ | *yawn*, morning | 17:48 |
| jml | hello, sorry, was away | 17:48 |
| jml | really really want to get a working lp dev environ | 17:48 |
| === beuno-lunch is now known as beuno | ||
| lifeless_ | jml: utilities/update-sourcecode | 17:49 |
| jml | lifeless_: done that. | 17:49 |
| === lifeless_ is now known as lifeless | ||
| lifeless | jml: and still doesn't want to play? | 17:49 |
| jml | lifeless: ahh, I know | 17:49 |
| jml | forgot to run link-external-sourcecode | 17:50 |
| jml | which appears to make the right symlink. /me makes schema again. | 17:50 |
| jml | ... and it builds. thaks. | 17:57 |
| lifeless | heh, de nada | 17:58 |
| nigelb | Morning lifeless | 17:59 |
| jml | actually, I should say what I want to do | 18:00 |
| jml | I want to link to apps.ubuntu.com from Launchpad. Specifically getting package pages to link to relevant apps pages. | 18:00 |
| lifeless | o/ nigelb | 18:00 |
| nigelb | Well, I guess I should head to bed :) | 18:01 |
| jml | I *think* I can get by with just a text transform, no queries. | 18:01 |
| lifeless | \o/ test perf fix worked | 18:12 |
| lifeless | Tests started at approximately 2011-10-31 08:45:30.145372 | 18:12 |
| lifeless | Source: bzr+ssh://bazaar.launchpad.net/~lifeless/launchpad/useoops r14214 | 18:12 |
| lifeless | Target: bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/ r14213 | 18:12 |
| lifeless | 16646 tests run in 4:33:15.575714, 0 failures, 0 errors | 18:12 |
| lifeless | not as svelte as perhaps it was in the past, but not 7 hours either | 18:12 |
| jml | Is there a fast way of consulting Contents.gz from Launchpad code? (Is there a version in the db?) | 18:21 |
| jelmer | jml: the archivepublisher generates it and I'm pretty sure it doesn't store it in the database | 18:24 |
| jelmer | jml: what are you trying to do? | 18:25 |
| jml | jelmer: link to apps.ubuntu.com for a package from packages on Launchpad | 18:25 |
| jml | jelmer: but apps.ubuntu.com has only apps, which are packages that have desktop files | 18:26 |
| jelmer | jml: ah, and you need the section and the like? | 18:26 |
| jml | jelmer: no, just to know whether or not it has a desktop file | 18:26 |
| jml | e.g. https://apps.ubuntu.com/cat/applications/oneiric/evolution/ vs https://apps.ubuntu.com/cat/applications/oneiric/openssh/ | 18:26 |
| lifeless | jml: there is a version being populated in the DB | 18:27 |
| jelmer | jml: do .desktop files necessarily indicate applications? IIUC it's possible to have .desktop files for non-application things | 18:27 |
| lifeless | jml: StevenK knows the status of that | 18:27 |
| jml | jelmer: that's how they make the distinction in Software Center, I'm told. | 18:27 |
| lifeless | jml: My first thoughts about how to implement this are: | 18:28 |
| lifeless | - what about derived distros ? | 18:28 |
| lifeless | - and if software centre knows, why not consult with it (e.g. from client js, or even from the appserver) to see whether you can link to it safely | 18:29 |
| jml | lifeless: I only want this for Ubuntu, not for derived distributions. | 18:29 |
| james_w | software-center has some fudges to determine the list of apps I believe, some are ignored, others are added | 18:31 |
| lifeless | jml: poking at contents because thats what SC uses seems like an abstraction violation to me | 18:31 |
| jml | lifeless: yeah, that's another option. More implementation work than using a table LP already has, but maybe more feasible. Would probably have to add that API. | 18:31 |
| james_w | I wonder if apps.ubuntu.com will grow the "technical items" thing too | 18:31 |
| lifeless | jml: this is so folk can find out how to install the package or something? | 18:32 |
| jml | It's hard enough writing code for what exists now. Dealing with possible futures is going to make this impossible. | 18:32 |
| lifeless | jml: a different approach would be to embed the content you're interested in on the relevant LP pages. E.g. reviews. | 18:32 |
| jelmer | jml: is software center using contents.gz directly though? I thought all that data was pre-generated in app-install-data | 18:32 |
| lifeless | jml: In my head there are 2 services (LP and SC) and 2 UI's (LP and SC) here, so having LP's UI talk to SC's service for some stuff is totally fine. | 18:33 |
| lifeless | jml: (if SC is fast enough) | 18:33 |
| jml | lifeless: largely the motivation is to increase google juice of apps.ubuntu.com. I guess it could also help developers using LP to quickly get a sense of what their package looks like to end users. | 18:34 |
| jml | lifeless: would look at reviews stuff in a later iteration. | 18:34 |
| jml | jelmer: quite possibly. | 18:34 |
| jml | lifeless: agree re abstraction violation. Doesn't feel a bad enough one to stop me though. Difficulty of implementation is more persuasive | 18:35 |
| lifeless | jml: such violations create debt instantly | 18:36 |
| lifeless | jml: I'd rather we not link at all than do it poorly | 18:36 |
| jml | lifeless: right, but I don't even have a working prototype that I can improve at this point | 18:36 |
| jml | jelmer: does that change anything? | 18:38 |
| jelmer | jml: does what change anything? the fact the data is pregenerated? | 18:38 |
| jml | jelmer: yeah | 18:38 |
| jelmer | jml: I suspect there is more going on than just looking for .desktop files | 18:39 |
| jelmer | jml: probably filtering out things that are not for applications | 18:39 |
| lifeless | jml: I'm curious - why does apps. need google juice? I thought it was mainly a thin shell for its API used by the sc client on Ubuntu installs ? | 18:39 |
| jelmer | but that's all speculation | 18:39 |
| jml | jelmer: OK. It's definitely possible that I was misinformed. | 18:39 |
| jml | jelmer: certainly using some kind of provided API would avoid that. | 18:39 |
| jml | lifeless: It turns out that a lot of people don't actually look at the Software Center very much. People fresh from other platforms often just open a browser to search for software, rather than looking for Software Center (or an app store or what have you). When they do that search, we want them to find http://apps.ubuntu.com – the web directory for the users. | 18:40 |
| jelmer | jml: it seems like a bad idea for software center to have to download a 20Mb file every time something in the archive changes | 18:40 |
| jml | lifeless: (paragraph from an email I was drafting) | 18:40 |
| lifeless | jml: interesting | 18:40 |
| jml | lifeless: yeah. | 18:41 |
| jml | lifeless: linking from LP is not the beginning & end of our approach, but it's something that would probably help. | 18:41 |
| jml | jelmer: well, I get the very strong impression that mvo would prefer some debtags-based solution. | 18:41 |
| jelmer | jml: debtags seems like a better approach for this kind of thing than inspecting Contents.gz | 18:42 |
| jml | jelmer: I am thinking of volunteering to add debtags support. Sort of contingent on how hard it is to add a hyperlink to a web page though :) | 18:43 |
| jml | hmm. | 18:44 |
| jml | I guess a naive-but-restful API is to just GET the link and then see if it 404s | 18:45 |
| jml | there's probably a way to do that without dl'ing the whole page too | 18:45 |
| jml | wonder how late I can render the link and still get SEO | 18:46 |
| jml | (also, battery about to fail) | 18:46 |
| lifeless | jml: google execute js | 18:49 |
| lifeless | jml: so I expect quite late | 18:50 |
| === dpm__ is now known as dpm | ||
| === fjlacoste is now known as flacoste | ||
| lifeless | sinzui: hiya | 19:33 |
| sinzui | hello | 19:33 |
| lifeless | sinzui: I think we have a bug in team administration :) | 19:33 |
| lifeless | sinzui: see #launchpad for context. Seeking your opinion. | 19:34 |
| === mrevell_ is now known as mrevell | ||
| benji | sinzui: are you aware of a way I can postpone a user's team membership expiration date? This is in reference to this question: https://answers.launchpad.net/launchpad/+question/177021 | 20:01 |
| lifeless | benji: needs a duckie | 20:02 |
| benji | lifeless: come again? | 20:02 |
| lifeless | ~admin ;) | 20:03 |
| lifeless | the logo is, or was, a yellow rubber duck | 20:03 |
| benji | lifeless: I think you're saying that I should be made a member of some administrative team, ~admin doesn't seem to be it though (https://launchpad.net/~admin). If so, who can give me such membership? | 20:08 |
| lifeless | benji: I was unclear; I mean you cannot; you have to get an admin to do it. | 20:08 |
| benji | ah! (where, I presume admin == LOSA) | 20:09 |
| lifeless | yes | 20:09 |
| === fjlacoste is now known as flacoste | ||
| === benji changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 266 | ||
| === micahg_ is now known as micahg | ||
| lifeless | \o/ instant OOPSes on qastaging. | 21:36 |
| lifeless | booyah | 21:36 |
| lifeless | flacoste: \o/ instant OOPSes on qastaging. | 21:37 |
| lifeless | booyah | 21:37 |
| flacoste | lifeless: awesome | 21:37 |
| lifeless | try it, its spicy ;) | 21:38 |
| beuno | lifeless, I like the word instant. Is it using rabbit or something like that? | 21:41 |
| lifeless | beuno: yes | 21:41 |
| beuno | lifeless, I WANTS | 21:41 |
| beuno | so you've been talking to rye, right? | 21:41 |
| lifeless | yes, he's production hardening the opened code base for your environment | 21:42 |
| lifeless | there is a wsgi enhancement needed for your tx stack before you can widely migrate | 21:42 |
| beuno | awesomeness | 21:50 |
| lifeless | beuno: see b.l.n/python-oops-wsgi | 21:57 |
| poolie | morning | 22:38 |
| wallyworld_ | jcsackett: r=me but i think the template still references the old public_or_private() method name | 22:41 |
| jcsackett | wallyworld_: yes. yes it does. i'll fix that. | 22:47 |
| wallyworld_ | thanks :-) | 22:47 |
| wallyworld_ | jcsackett: i assume the stuff you will be sending through contains the necessary detail to understand what needs change wrt the searching and making it clearer when things are deleted etc? | 22:50 |
| jcsackett | wallyworld_: indeed. | 22:54 |
| wallyworld_ | jcsackett: the fwd email is missing the original material | 23:01 |
| jcsackett | wallyworld_: oh fantastic. | 23:02 |
| wallyworld_ | :-) | 23:02 |
| wgrant | lifeless: https://pastebin.canonical.com/54976/ is what we used to fix the the BPBs. | 23:17 |
| wgrant | A similar thing over SPRBs would be good. | 23:17 |
| wgrant | (I'm not here today) | 23:17 |
| lifeless | wgrant: ah, it is damage ? | 23:18 |
| wgrant | lifeless: Yes, failure counting except buggy. | 23:19 |
| lifeless | wgrant: qastaging. Rabbit. AMQP. OOPS. WIN WIN WIN | 23:27 |
| wgrant | lifeless: It works? | 23:27 |
| lifeless | hell yes | 23:27 |
| wgrant | Excellent! Have we stopped rabbit to see that it falls back? | 23:27 |
| lifeless | not yet | 23:27 |
| lifeless | been doing uds stuff | 23:27 |
| lifeless | wgrant: still, you should make qas oops and see the result.... its just nice | 23:33 |
| wgrant | lifeless: I already have :) | 23:35 |
| wgrant | It is indeed. | 23:35 |
| wgrant | oops-tools needs some major hacking, but that can be done soon. | 23:35 |
| wgrant | lifeless: Is https://pastebin.canonical.com/55154/ a yesterday-style query? | 23:55 |
| wgrant | lifeless: ie. EXPLAIN ANALYZE is instant, execution is 300ms? | 23:55 |
| lifeless | qaqs ok? | 23:55 |
| wgrant | Yeah. | 23:56 |
| lifeless | no limits ? | 23:56 |
| wgrant | None. | 23:56 |
| wgrant | Should be 1 row. | 23:56 |
| lifeless | 300ms | 23:56 |
| lifeless | 323 | 23:56 |
| wgrant | And explain analyze? | 23:56 |
| lifeless | 162 | 23:56 |
| lifeless | 162 | 23:56 |
| wgrant | It's 1ms on DF :/ | 23:56 |
| lifeless | 0.6ms to profile | 23:56 |
| wgrant | lolnot | 23:57 |
| wgrant | It's not even a very unpleasant plan at all. | 23:58 |
| lifeless | ugh | 23:58 |
| lifeless | AND (TeamParticipation.team = BugSubscription.person) | 23:58 |
| lifeless | AND (Person.id = TeamParticipation.team) | 23:58 |
| lifeless | wtf | 23:58 |
| lifeless | oh, via TP, I guess. | 23:58 |
| lifeless | not the join I expected. | 23:59 |
| wgrant | StevenK: Can you QA or ignore jtv's notification refactor? | 23:59 |
| lifeless | wgrant: in fact, that will give N rows per subscription | 23:59 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!