[00:09] <lifeless> zopelessappserverlayer
[00:19] <lifeless> wgrant: you're going to laugh at this
[00:19] <lifeless> wgrant: in between the tears.
[00:19] <wgrant> I hope so.
[00:19] <lifeless> wgrant: ZASL runs up the codehosting server.
[00:19] <lifeless> wgrant: that calls into private-xmlrpc
[00:19] <lifeless> wgrant: thats raising oopses because of disconnected stores
[00:20] <wgrant> ZASL itself shouldn't start codehosting...
[00:20] <wgrant> Unless it's pretty different from AppServerLayer, which it shouldn't be since I made ZopelessLayer and FunctionalLayer almost equal.
[00:21] <lifeless> wait, there is more
[00:22] <lifeless> we also get a disconnectionerrror: terminating connection due to administrator command
[00:22] <wgrant> Where are you running this?
[00:22] <lifeless> wgrant: -> because the test db reset code pulls the db out from under the slave appserver
[00:22] <wgrant> Locally?
[00:22] <lifeless> wgrant: yes, totally reproducable
[00:22] <wgrant> You need the new storm.
[00:22] <lifeless> on lucid ?
[00:22] <wgrant> Yes.
[00:23] <wgrant> libpq5 8.4.9 breaks everything.
[00:23] <wgrant> Destroyed buildbot on Friday.
[00:23] <wgrant> ec2 isn't affected yet.
[00:23] <lifeless> this is 8.4.3-1
[00:23] <wgrant> New Storm is in devel now.
[00:23] <wgrant> Ah.
[00:23] <wgrant> Maybe your analysis is correct, then.
[00:23] <lifeless> maybe :)
[00:23] <lifeless> the reason it happens only when two tests are run
[00:24] <lifeless> is that we have a brand new appserver if we only run one test
[00:24] <wgrant> Yep.
[00:24] <wgrant> Still, this looks similar to the libpq5 issue...
[00:24] <wgrant> Exactly the same thing.
[00:25] <wgrant> librarian was broken after the first test.
[00:25] <lifeless> how so ?
[00:25] <lifeless> I thought that was it not detecting the disconnect ?
[00:25] <wgrant> Right.
[00:25] <lifeless> this detects it fine
[00:25] <wgrant> If this is a DisconnectionError, it's probably working.
[00:25] <lifeless> its raised from storm.database line 376 _check_disconnect
[00:25] <lifeless> yes, it is detecting it
[00:26] <lifeless> we're just oopsing - correctly - because the db was yanked without telling the appserver
[00:28] <lifeless> so I should say, rather than 'codehosting is started' the test using bzr transports and these are backing onto xmlrpc requests in some fashion
[00:29] <lifeless> and bingo
[00:29] <lifeless> the db is being dropped after the test, rather than the sequences reset
[00:29] <lifeless> I'm now totally confident of the analysis, but wtf to do about it
[00:30] <lifeless> I think I'll expect 1 or 9 oopses
[00:30] <lifeless> along with an XXX about this
[00:30] <lifeless> wgrant: as my victim^Wreviewer, how do you feel about this
[00:34] <wgrant> lifeless: The DB will be dropped if it's dirty.
[00:34] <wgrant> I think that's probably fine.
[00:34] <wgrant> As long as it is really 1 or 9 :)
[00:35] <lifeless> bug 884036
[00:35] <_mup_> Bug #884036: db layer test resets interact badly with slave appserver instances <Launchpad itself:Triaged> < https://launchpad.net/bugs/884036 >
[00:36] <lifeless> see if that convinces you
[00:36] <lifeless> second oops added now
[00:39] <wgrant> Sounds good.
[00:43] <wgrant> lifeless: https://code.launchpad.net/~wgrant/launchpad/cte-it-up/+merge/80767
[00:44] <wgrant> 2.5s->1.6s on qastaging
[00:44] <lifeless> wgrant: does this change +bugs as well? or just BugTask:* ?
[00:45] <wgrant> lifeless: +bugs as well. Do you want that checked too?
[00:45] <lifeless> uhm yes.
[00:46] <lifeless> just cause, well, EPIC TIMEOUTS
[00:46] <wgrant> It should in theory be a significant improvement. But let's see.
[00:46] <lifeless> rev 14156 of my branch is up; Could you please incremental review anything you haven't, and I'll land.
[00:47] <wgrant> Ermm.
[00:47] <wgrant> This isn't landable...
[00:47] <wgrant> There is no rabbit on prod, and the fallback is flaky at best.
[00:48] <lifeless> with no rabbit it will behave as it did befre
[00:48] <lifeless> *before*
[00:48] <wgrant> And the moment we have rabbit, all OOPSes will disappear.
[00:48] <wgrant> Also, BSON.
[00:48] <lifeless> whats flaky about the fallback ?
[00:48] <wgrant> If rabbit is up but amqp2disk hasn't set up its queue bindings, OOPSes go to nowhere.
[00:48] <wgrant> And that's just what I've noticed.
[00:49] <lifeless> thats how loosely coupled works.
[00:49] <wgrant> Yes, but it's not how production should work.
[00:49] <lifeless> Its not a defect, its expected. I'm hoping to get that side of it sorted later today.
[00:49] <wgrant> eg. qastaging and staging have rabbit, no amqp2disk.
[00:49] <lifeless> we can disable the config for them. Or get amqp2disk up and live.
[00:49] <wgrant> Do that before landing.
[00:50] <lifeless> sure
[00:50] <lifeless> but whats flaky?
[00:50] <lifeless> or was that code for 'new world order does not have the same guarantees' ?
[00:50] <wgrant> It only works if rabbit is entirely unconnectable, and it's not exactly well-tested.
[00:51] <lifeless> wgrant: its tested in oops-amqp; the tests there pull rabbit out from under it
[00:51] <lifeless> wgrant: rabbit config isn't, and shouldn't, be the business of the oops producer
[00:51] <wgrant> Sure.
[00:51] <wgrant> But as your branch is now, as soon as rabbitmq is live on production then we lose all error reporting.
[00:52] <lifeless> IFF the queue isn't setup
[00:52] <wgrant> Which it probably won't be.
[00:52] <lifeless> separate issue; we can toggle off the oops exchange very very easily.
[00:52] <lifeless> (in prodconfigs)
[00:52] <wgrant> lol prodconfigs easily
[00:53] <lifeless> more easily than landing this
[00:53] <lifeless> change, push, review, pqm-submit.
[00:55] <wgrant> lifeless: Why is UnknownRemoteError now permitted in test_runner.py?
[00:56] <lifeless> wgrant: because python OOM is nondeterministic
[00:56] <lifeless> wgrant: I ran it in a loop and didn't get MemoryError every time
[00:56] <wgrant> But that invalidates the test...
[00:57] <wgrant> Also, isn't the setUpQueue call in sync() racy?
[00:57] <lifeless> I can put it back, but I expect some buildbot failures as a result
[00:57] <wgrant> I don't see why it needs a new queue.
[00:57] <lifeless> wgrant: because auto_delete
[00:57] <wgrant> They would be legitimate failures.
[00:57] <wgrant> The current test is not flaky.
[00:58] <wgrant> Ah.
[00:58] <wgrant> Fair point.
[00:58] <lifeless> wgrant: I bet it is
[00:58] <wgrant> Still racy, but acceptable.
[00:58] <wgrant> lifeless: It's never been a problem before.
[00:58] <lifeless> wgrant: poolie ran into a race the other day, last happened 4 years ago.
[00:58] <lifeless> wgrant: doesn't make it less of a race
[00:58] <wgrant> Even so.
[00:58] <lifeless> I can put it back.
[00:58] <wgrant> It makes it an invalid test.
[00:58] <lifeless> But I think the test lies today
[00:58] <lifeless> the test isn't valid
[00:58] <poolie> lifeless: two races
[00:59] <wgrant> Better to have it failing and then disabled.
[00:59] <wgrant> Than left entirely invalid.
[00:59] <poolie> i guess it is the spring racing carnival in launchpad too
[00:59] <wgrant> lifeless: Perhaps it's because you were running it i386?
[01:00] <StevenK> wallyworld_: I've made the changes you mentioned for https://code.launchpad.net/~stevenk/launchpad/hide-create-new-ppa-open-team/+merge/80650 . Can you have another look so I can ec2 it?
[01:00] <lifeless> wgrant: yes, I am
[01:01] <lifeless> wgrant: I have no particular interest in this; the runner code doesn't generate MemoryError synthetically, by checking for ulimit violations etc - it depends on the python runtime only ever raising MemoryError AFAICT (grepping for MemoryError ..)
[01:01] <lifeless> wgrant: so pick a preferred resolution, and I'll do it.
[01:02] <wgrant> lifeless: Your INTERACTIVE_TESTS change concerns me. Doesn't that mean we're just not going to see any oopses from the appserver?
[01:02] <lifeless> wgrant: myself, I'm quite sure that we will eventually hit whatever point in CPython caused UnknownRemoteError
[01:03] <wgrant> lifeless: Sure, but I'd prefer to have a flaky or disabled test, rather than one that doesn't actually test.
[01:03] <lifeless> wgrant: no tests look for them, and its *only* yuixhr that uses this code path
[01:03] <lifeless> wgrant: depends on the intent of the test; I think 'raises promptly' is sufficiently narrow.
[01:03] <wgrant> Oh, it's in testapp, right.
[01:03] <lifeless> wgrant: we don't depend on catching MemoryError anywhere (damn straight)
[01:04] <lifeless> wgrant: so the -only- impact on broadening the exceptions is that the test suite lines up with the OOPS reports
[01:04] <lifeless> wgrant: [if/when we see this in prod]
[01:04] <wgrant> lifeless: Hm, UnknownRemoteError is probably txamqp...
[01:04] <lifeless> wgrant: txamp do you mean ?
[01:04] <wgrant> Yes.
[01:04] <wgrant> I fail.
[01:05] <wgrant> So, I'd prefer it if you reverted that change.
[01:05] <wgrant> If it is a problem, we can unrevert it.
[01:05] <lifeless> it is in Twisted AMP yes
[01:05] <lifeless> sure, I will (it is a problem)
[01:06] <wgrant> It's not a problem.
[01:06] <wgrant> It may be on i386, but nobody's noticed that before :)
[01:08] <lifeless> let me just thrash carob's pagecache
[01:09] <wgrant> Oh?
[01:09] <lifeless> launchpad.net-logs$ grep UnknownRemoteError . -r
[01:10] <wgrant> Ah
[01:10] <wgrant> That seems unwise...
[01:10] <wgrant> It's like 150GB or more.
[01:10] <wgrant> Or was it 250GB
[01:10] <wgrant> I can't quite remember.
[01:10] <lifeless> serial IO ftw
[01:10] <wgrant> I still contend that carob's disks are actually a NAS in NZ.
[01:11] <lifeless> I could grep faster if they were
[01:11] <lifeless> anything else ?
[01:12]  * StevenK prods wallyworld_.
[01:12] <wallyworld_> ouch
[01:13] <StevenK> [12:00] < StevenK> wallyworld_: I've made the changes you mentioned for https://code.launchpad.net/~stevenk/launchpad/hide-create-new-ppa-open-team/+merge/80650 . Can you have another look so I can ec2 it?
[01:13] <wallyworld_> sure.
[01:13] <wgrant> lifeless: I think that's about it.
[01:15] <lifeless> ok, so I'll push th revert to test_memory_hob_job, toss at ec2, and we'll see what we see
[01:15] <wallyworld_> StevenK: '%s/+activate-ppa' % canonical_url(team) -> canonical_url(team, view_name='+activate-ppa')
[01:17] <wallyworld_> other than that, looks good thanks
[01:17] <lifeless> wgrant: can as approved vote?
[01:20] <StevenK> wallyworld_: Thank you for the Approve.
[01:20] <wallyworld_> np.
[01:20] <wgrant> lifeless: Done.
[01:24] <lifeless> thanks
[01:25] <wgrant> lifeless: Erm.
[01:26] <wgrant> What did I say about not landing this.
[01:26] <wgrant> I approved mostly given that you were going to get at least (qa)staging fixed first :)
[01:26] <wgrant> (and no, "It'll be done soon" is not valid)
[01:31] <lifeless> wgrant: I'm working on that right now
[01:31] <wgrant> If it's not done by the time ec2 finishes, please kill the ec2 instance.
[01:32] <lifeless> that would be a waste
[01:32] <wgrant> Let's not get into another 2-month cart-before-the-horse situation like we did with fastdowntime.
[01:32] <lifeless> if its not ready I'll change prodconf to disable the key in staging
[01:32] <lifeless> wgrant: you exaggerate there
[01:32] <wgrant> No...
[01:32] <lifeless> wgrant: I'm not about to block the deployment pipeline for any extended period
[01:33] <lifeless> wgrant: I'm well aware of its impact, given I've done most of the analysis and discussion to demonstrate that
[01:33] <wgrant> You are about to significantly hinder the crisis analysis process, though.
[01:33] <lifeless> I'm about to change the exact recipe needed
[01:33] <lifeless> grep still works, data can still be easily view
[01:33] <lifeless> ed
[01:34] <wgrant> No, grep doesn't still work.
[01:34] <wgrant> I can't grep for ^Exception-Type: RequestExpired now.
[01:34] <lifeless> yes it does, bson stores strings as length: prefix, so you can do exactly that.
[01:34] <lifeless> just phrased -very slightly- differently.
[01:35] <wgrant> I am close to rescinding my approval.
[01:36] <lifeless> why?
[01:36] <wgrant> 1) all (qa)staging OOPSes will be lost unless you sort things out in the next few hours.
[01:37] <lifeless> 1) hystrionics, I've outlined 2 solid workarounds and taken responsibility for having them sorted.
[01:37] <wgrant> 2) oops-tools is mostly useless, and this breaks shell pipelines.
[01:37] <lifeless> 2) has had 3? more? weeks for folk to respond to the get-out-and-try aspect, and there are also recipes in that same thread to deal.
[01:38] <wgrant> We already had three expressions of disapproval.
[01:38] <wgrant> In addition to my initial one.
[01:41] <lifeless> aaron asked why; stuart noted about readability with an anecdote from ISD who are doing noone-knows-what, and gary was -0.5 accepting my argument about greppability
[01:43] <lifeless> I have been trying and playing around, and I'm satisfied I can do the analyses I was previously doing.
[01:43] <lifeless> Have you experimented, or are you just concerned about the possible impact?
[01:44] <wgrant> I know that my usual method for analysing performances crises now has to be rewritten in Python, or invoke Python thousands of times.
[01:45] <lifeless> how do you know that ?
[01:45] <wgrant> Because it relies on selecting on and extracting fields and parts of fields from OOPSes.
[01:46] <lifeless> so, its an assumption. Thats ok, but its useful to be clear.
[01:47] <lifeless> you are assuming, without trying, that grep and sed can't extract what you want from bson.
[01:47] <lifeless> you may be right, you may not be.
[01:49] <lifeless> I'm more worried about the url interactions than the change to bson; oops-tools has shocking understanding of urls-are-text
[01:49] <lifeless> for instance our appserver generated urls are unicode
[01:49] <wgrant> I'm not too concerned about how it interacts with oops-tools.
[01:49] <lifeless> so we may end up double encoding for a day or two, which is tolerable but annoying
[01:49] <wgrant> oops-tools isn't critical, and can be fixed.
[01:53] <wgrant> lifeless: +bugs seems to be much faster on qastaging, with the CTE patch.
[01:53] <wgrant> FTI Ubuntu searches are still timing out, but they were before too so I suspect the index is just cold.
[01:53] <lifeless> ok.
[01:53] <lifeless> uhm
[01:53] <lifeless> just for insurance, lets FF this
[01:53] <lifeless> its a particularly sensitive bit of code
[01:54] <wgrant> k
[01:54] <wgrant> It's all about to become a lot simpler, but until then...
[01:55] <lifeless> I'm sure the FF will be overkill
[01:55] <lifeless> but I'll feel a whole lot happier having it :)
[01:55] <wgrant> Indeed.
[02:08] <wgrant> test_bugtask_search is terrible :(
[02:08] <wgrant> A slow base test case with dozens of derivatives.
[02:09] <lifeless> yes
[02:16] <wgrant> lifeless: It's now behind a flag, with a few strategic tests duplicated to test both paths. The new function is ugly as hell, but at least it should be safe :)
[02:16] <lifeless> wgrant: thank you very much!
[02:17] <wgrant> Bah, but now I have to ec2 it :(
[02:17] <lifeless> what could possibly go wrong
[02:17] <wgrant> The FF check will probably introduce a new query sometimes :(
[02:17] <wgrant> Which could break some query limit tests.
[02:18] <lifeless> most of them have some deliberate fat
[02:18] <lifeless> with a generous backstop, but yes, it could
[02:19] <wgrant> I've been meaning to do a test run with the matcher ensuring a lower bound as well.
[02:19] <wgrant> Some of them are waaaaay high now.
[02:27] <StevenK> Bah, I can't have my change in create_initialized_view, too many things can't be canonical_url()'d.
[03:22] <poolie> what's a CTE?
[03:25] <wgrant> poolie: Common Table Expression
[03:25] <wgrant> Defined by the WITH clause.
[03:27] <poolie> ok
[03:36] <StevenK> wgrant: Is QA for r14209 on your list this afterrnoon?
[03:37] <wgrant> StevenK: That's the userCanView change?
[03:37] <StevenK> Yeah
[03:37] <wgrant> It's semi-ok.
[03:37] <StevenK> qa-ok-ish
[03:37] <StevenK> ?
[03:38] <wgrant> Liveable, but not ideal. I'm hoping that an improvement will land before someone tries to deploy.
[03:38] <StevenK> We have a large amount of European QA before that rev.
[03:38] <wgrant> Yes.
[04:08] <lifeless> http://research.microsoft.com/en-us/news/features/nagappan-100609.aspx is a good read
[04:09] <poolie> can someone quickly review http://readthedocs.org/docs/judge/en/latest/ for me
[04:11] <poolie> lifeless: that is interesting
[04:12] <poolie> it would be interesting to study: how good are developers at predicting which factors drive quality on their projects
[04:22] <lifeless> indeed
[04:28] <poolie> what should i do now?
[04:29] <lifeless> partay
[04:30] <nigelb> heh
[04:30] <nigelb> morning! :)
[04:35] <poolie> hi nigelb
[04:37] <nigelb> Hey poolie
[04:47] <spm> o/ nigelb
[04:47] <nigelb> ohai spm :)
[05:24] <nigelb> Ugh, I now officially hate the expiring membership email.
[05:50] <spm> yeah? ok then. INSERT into teams VALUES nibelb WHERE team has expiringing_membership;
[05:50] <spm> spelling to be optional.
[05:51] <StevenK> spm: Let me give you a cronjob that will reset nigelb's memberships to expire in seven days. Every day.
[05:51] <spm> StevenK: approved.
[05:51] <spm> this wouldn't be excessive abuse of power would it? :sincerity:
[06:13] <nigelb> hah
[06:13] <nigelb> StevenK, spm: Love you too :P
[06:14] <spm> :-)
[07:00] <lifeless> wgrant: you said txlongpoll was bust
[07:00] <lifeless> wgrant: in that it didn't accept a queue name prefix, right ?
[07:00] <wgrant> lifeless: Yes.
[07:00] <lifeless> is bigjools aware of this? And that its a blocker to the deploy ?
[07:00] <wgrant> I believe Red is aware.
[07:01] <wgrant> However, AFAIK they are unaware that anybody is using rabbit except them.
[07:01] <wgrant> So they are probably unaware that the fix is time-critical.
[07:06] <lifeless> wgrant: there is no bug on https://bugs.launchpad.net/txlongpoll
[07:07] <lifeless> wgrant: as you know the details, could you make one please?
[07:08] <wgrant> Hmm, rvba said he was going to file one.
[07:08] <wgrant> I left it with him, since it was well after midnight.
[07:08] <lifeless> the rt claims that read is only on ^longpoll.*
[07:09] <lifeless> so there is access control in place
[07:09] <wgrant> That is a start.
[07:10] <wgrant> Personally I'd prefer the old behaviour to be revived.
[07:11] <lifeless> I'm not sure what that is; thus my encouragement for you to file a bug :)
[07:11] <lifeless> hmm
[07:12] <lifeless> my branch has been running from 0140 through to 0708
[07:12] <lifeless> I suspect a stupidly overloaded host
[07:12] <wgrant> lifeless: I just had a branch take 4:45
[07:12] <wgrant> Which is longer than normal.
[07:13] <lifeless> hmm
[07:13] <lifeless> I suspect that that sync() in cleanUp is a problem
[07:13] <lifeless> I have another trace here that took 6.5 hours
[07:13] <lifeless> I'll land a branch immediately to remove the trailing sync()
[07:14] <lifeless> [given I only added it while debugging that rosetta zopelessappserver test issue, I know its not needed know]
[07:14] <wgrant> Ah.
[07:15] <lifeless> its the only think I can think of that would add 100% test time
[07:16] <lifeless> unless the rabbit layer reset is heinous, but I didn't change that, nor add it to more than a few dozen tests
[07:18] <lifeless> wgrant: I suspect both the sync and the existence/presence of poor ec2 loading
[07:19] <lifeless> wgrant: did we decide setting mandatory wouldn't work ?
[07:20] <wgrant> lifeless: We can't necessarily receive the response.
[07:20] <lifeless> wgrant: did we do an experiment ?
[07:20] <wgrant> No.
[07:21]  * StevenK has clearly forgotten how to drive subunit
[07:21] <wgrant> But we'd need to asynchronously detect the basic.return, and then $something.
[07:21] <wgrant> But I don't think amqplib exposes basic.returns.
[07:21] <StevenK> I have a subunit stream and I want to see which tests failed or gave an error
[07:21] <lifeless> StevenK: subunit-filter --no-skip --no-passthrough | subunit-ls
[07:22] <lifeless> StevenK: or testr load < stream; testr failing
[07:22] <lifeless> (or testr failing --list)
[07:22] <lifeless> StevenK: or tribunal < stream
[07:22] <StevenK> subunit-filter --no-success --no-passthrough --error --failure < lp.log | subunit-ls
[07:22] <StevenK> That gives me a traceback
[07:23] <lifeless> StevenK: you're making it hard for yourself :)
[07:23] <lifeless> StevenK: 3 of those options are the defaults
[07:23] <lifeless> StevenK: also whats the tb you get?
[07:24] <StevenK> Broken pipe
[07:24] <lifeless> subunit-filter is probably erroring
[07:24] <StevenK> So one of subunit-filter or subunit-ls is misbehaving
[07:24] <lifeless> try one of my recipes above
[07:29] <lifeless> StevenK: ?
[07:30] <StevenK> Sorry, got distracted. Your recipe works fine
[07:30] <StevenK> "subunit-filter --no-success --no-passthrough --error --failure < lp.log 1>/dev/null" doesn't produce any output on stderr
[07:31] <lifeless> its a legitimate set of options
[07:31] <lifeless> I'm not sure why you had a tb
[07:32] <lifeless> conceivably the pipe was closed with no output I guess
[07:32] <lifeless> it works for me, fwiw
[07:33] <StevenK> lifeless: http://pastebin.ubuntu.com/724060/
[07:33] <lifeless> wgrant: so, you'll file a bug| get rvba to ?
[07:33] <lifeless> StevenK: the first tb is the issue
[07:33] <wgrant> lifeless: I'll ask rvba for the latest news.
[07:33] <lifeless> StevenK: Can you reproduce with a current subunit ?
[07:33] <lifeless> StevenK: (what does dpkg -l python-subunit say)
[07:34] <StevenK> lifeless: 0.0.6-3
[07:35] <lifeless> try 0.0.7
[07:35] <StevenK> Such a pity, Mr Bond.
[07:38] <nigelb> heh
[07:50] <jelmer> lifeless: that reminds me, can you add a tag for 0.0.7 on lp:subunit?
[07:51] <lifeless> jelmer: probably, there isn't one ?
[07:51] <lifeless> jelmer: it'll be just tip
[07:51] <lifeless> jelmer: :!bzr log -r -1
[07:51] <lifeless> ------------------------------------------------------------
[07:51] <lifeless> revno: 152
[07:51] <lifeless> tags: 0.0.7
[07:51] <jelmer> lifeless: "bzr tags -d lp:subunit" doesn't list a 0.0.7
[07:51] <jelmer> lifeless: perhaps that tag is local?
[07:52] <lifeless> jelmer: if so, I blame bzr
[07:53] <lifeless> jelmer: hmm, diverged branches. funky.
[07:54] <lifeless> yes, somethings up. I'm going to push --overwrite and then merge trunk back in
[07:57] <lifeless> jelmer: there should be a tag now
[07:57] <jelmer> lifeless: great, thanks!
[08:17] <lifeless> useoops has landed \o/
[08:20] <wgrant> Has the fix landed?
[08:23] <bigjools> My next branch is going to be called The Eagle
[08:23] <nigelb> Just to announce its landing?
[08:23] <bigjools> correct
[08:24] <nigelb> I'll call dibs on Airforce one and Canonical one ;)
[08:26] <nigelb> Who's at UDS? francis, matt, dan..and?
[08:28] <bigjools> that's about it IIRC
[08:28] <bigjools> oh one of the US guys too
[08:32] <adeuring> good morning
[08:32] <lifeless> wgrant: its about to start ec2
[08:45] <wgrant> nigelb: I thought it was Francie, Matt, Deryck and Curtis. Didn't know Dan was going too.
[08:46] <lifeless> new kid, doing the rounds :)
[08:54] <lifeless> bigjools: is rvba in today?
[09:01] <bigjools> lifeless: doesn't look like it
[09:16] <nigelb> bigjools / wgrant - Thanks :)
[09:16] <bigjools> I'd quite like to be there again, was the best UDS hotel in ages
[09:17] <nigelb> heh
[09:17] <nigelb> I wish I could be there. I regret skipping this UDS>
[09:18] <nigelb> bigjools: Will you be at the next one? :)
[09:19] <nigelb> Too bad I missed you last UDS.
[09:19] <bigjools> you walked right past me there :)
[09:19] <nigelb> :(
[09:19] <nigelb> That's the UDS where I asked Francis "Are you a launchpad guy?" :D
[09:19] <bigjools> as for the next one, no idea, we don't decide until shortly before who's going
[09:20] <nigelb> Ah!
[09:20] <\sh> moins
[09:21] <\sh> dear lp-devs how difficult will it be, to change someones launchpad id from X to Y ?
[09:21] <nigelb> Its easy to do the change.
[09:21] <nigelb> THe repurcusions might make some time to fix ^-^
[09:21] <bigjools> \sh: the answer is "it depends"
[09:21] <nigelb> And its not a launchpad problem. Its an SSO thing.
[09:22] <wgrant> SSO actually steals that data from us.
[09:22] <wgrant> It's a Launchpad thing.
[09:22] <wgrant> \sh: Unless you have a PPA, you can just change it.
[09:22] <\sh> bigjools, let's say, I would like to request to change my launchpad-id from shermann to sadig
[09:22] <bigjools> \sh: then as wgrant says
[09:23] <nigelb> wgrant: Meh. I think most of the time its our openid client which isn't implemented properly.
[09:23] <\sh> wgrant, ok...my ppa is not a problem...so when I delete my ppa I can just change it?
[09:23] <bigjools> if you have any PPAs then they need to be deleted
[09:23] <bigjools> then yes
[09:23] <wgrant> \sh: Yes. May have to wait up to 10 minutes for it to be completely deleted.
[09:23] <lifeless> \sh: delete the ppa, wait for a publishing cycle, visit +edit and rename it
[09:23] <lifeless> I think its +edit
[09:24] <\sh> wgrant, lifeless , thx...I'm waiting now ;)
[09:24]  * bigjools has a buildd-manager that will cancel builds \\o/
[09:25] <StevenK> bigjools: Does it also impact on the production builders too :-P
[09:25] <jelmer> bigjools: niiice!
[09:25] <wgrant> bigjools: You know that's been possible for recipe builds for a year, right? :P
[09:25] <bigjools> whatever
[09:26] <lifeless> running out of memory != cancelling :)
[09:26] <bigjools> lol
[09:26] <nigelb> ZING.
[09:26] <wgrant> They have a cancel button which sets them to SUPERSEDED. Which actually works pretty well.
[09:26] <bigjools> yes but it doesn't rip the build off the builder
[09:26] <jelmer> wgrant: that doesn't kill the in-progress build though, right?
[09:27] <bigjools> we can set it to CANCELLED now
[09:27] <wgrant> I believe it deletes the BQ too, which would kill the in-progress build.
[09:27] <nigelb> So, the ones that are scheduled can be canceled?
[09:27] <nigelb> Or is it the ones that are building that can be canceled.
[09:28] <bigjools> interesting way of doing it
[10:15] <bigjools> lifeless: yay for the OOPS changes!  Did you document what you said in the email in the dev wiki?
[10:16] <lifeless> bigjools: all relevant docs were updated
[10:16] <lifeless> bigjools: AFAIK
[10:16] <lifeless> bigjools: however, there weren't many ;)
[10:16] <bigjools> awesome
[10:17] <lifeless> bigjools: https://dev.launchpad.net/QA/OopsToolsSetup is perhaps the least likely known by ye old average LP dev
[10:17] <bigjools> well - the main thing is to note the stuff you wrote in the email, for posterity
[10:17] <bigjools> I'm going on a documentation jihad in the near future
[10:44] <lifeless> bigjools: I've given https://dev.launchpad.net/LoggingOopses a bit of a facelift too
[10:45] <bigjools> lifeless: great!
[11:00] <lifeless> ok, thats witching hour, time to EOD for a bit :)
[11:26] <nigelb> aaah.
[11:26] <nigelb> I now understand the outcome of comparing unicode incorrectly.
[11:27] <nigelb> The unicode entry goes to the end of the list :(
[11:28] <wgrant> Yep.
[11:28] <nigelb> Nice that danilos is our unicode tester :D
[11:28] <nigelb> Is there a better way to do that than what I did?
[11:28] <wgrant> Sadly we are left with only rvba.
[11:28] <wgrant> And his name is not very Unicode at all.
[11:30] <nigelb> heh
[11:30] <nigelb> Is there better way to sort unicode at the current point?
[11:30] <nigelb> I remember jtv's argument at that time being more of future oriented.
[11:35] <lifeless> need unicode local in the DB and appservers
[11:35] <lifeless> presumably not /hard/
[11:35] <lifeless> but needs doing + qa
[11:36] <nigelb> ah
[11:36] <nigelb> Nothing I an do I suppose?
[11:37] <nigelb> *can
[11:40] <cjwatson> bigjools: cancelled> ooh, awesome
[12:15] <lifeless_> nigelb: you could do it
[12:15] <lifeless_> nigelb: but indirectly; you'd need to file bugs / rt's to find out the exact current config, reproduce it, see what you want, test that, get confidence etc
[12:16] <nigelb> ah
[12:55] <bigjools> cjwatson: only for virtual builds right now though, don't get too excited
[12:55] <bigjools> need to fix sbuild for non-virt
[12:58] <cjwatson> bigjools: ah well, still a step forward
[12:58] <bigjools> yes, 90% of work is done
[12:59] <bigjools> s/is/will be/ :)
[12:59] <bigjools> LinkedIn - just PO please
[14:49] <cjwatson> Wow.  The publisher has only just got to around cron.germinate
[14:49] <cjwatson> We're in serious danger of going past an hour here :-/
[14:52] <bigjools> cjwatson: :(
[15:08] <jcsackett> sinzui: free to mumble?
[15:08] <sinzui> yes
[15:09] <jcsackett> excellent.
[15:23] <danhg> Hey everyone at UDS, I'm Dan, the new Usability & Communications Specialist at Launchpad. I'm going to be doing lots of usability testing today, so please ping me if you're around, and would like to get involved!
[15:25] <nigelb> danhg: #ubuntu-uds is also a nice channel to advertise :)
[15:26] <danhg> thanks nigelb, good plan!
[15:27] <danhg> All week actually - so let me know when you're free guys!
[15:48] <bigjools> cjwatson: yeah, it missed the run at :00
[15:52] <bigjools> domination took 30m
[15:52] <bigjools> I figured this would happen with the arch-all changes
[15:56] <cjwatson> this is pretty bad - what can be done?
[15:56] <cjwatson> don't get me wrong, I appreciate the new feature, but ...
[16:06] <mrevell> jcsackett, yo
[16:06] <jcsackett> mrevell: hello.
[16:12] <mrevell> danilos, What's the rally project management session later all about?
[16:12] <mrevell> jcsackett, Oh hey, we're just trying to find a good time to talk to you. When suits you?
[16:12] <jcsackett> pretty much anytime today is fine. i imagine your schedule is a bit more bumpy than mine.
[16:14] <jcsackett> mrevell ^
[16:16] <jcsackett> mrevell: danhg has proposed now, which works for me.
[16:17] <danhg> Cool
[16:20] <mrevell> jcsackett, Oh hey, we're just trying to find a good time to talk to you. When suits you?
[16:21] <jcsackett> mrevell: i think you meant perhaps to send another message, as that one's a repeat? :-P
[16:21] <mrevell> jcsackett, I have no idea how that happened. Hah :)
[16:21] <mrevell> jcsackett, I think danhg is talking to you about it.
[16:21] <jcsackett> he is indeed.
[16:59] <jml> awk: (FILENAME=lib/lp/contrib/javascript/yui3-gallery/gallery-accordion/gallery-accordion.js FNR=2916) fatal: cannot open file `lib/lp/contrib/javascript/mustache.js' for reading (No such file or directory)
[16:59] <jml> make: *** [lib/canonical/launchpad/icing/build/launchpad.js] Error 2
[16:59] <jml> just got that running 'make schema'
[17:02] <bigjools> jml: update source?
[17:48] <lifeless_> *yawn*, morning
[17:48] <jml> hello, sorry, was away
[17:48] <jml> really really want to get a working lp dev environ
[17:49] <lifeless_> jml: utilities/update-sourcecode
[17:49] <jml> lifeless_: done that.
[17:49] <lifeless> jml: and still doesn't want to play?
[17:49] <jml> lifeless: ahh, I know
[17:50] <jml> forgot to run link-external-sourcecode
[17:50] <jml> which appears to make the right symlink. /me makes schema again.
[17:57] <jml> ... and it builds. thaks.
[17:58] <lifeless> heh, de nada
[17:59] <nigelb> Morning lifeless
[18:00] <jml> actually, I should say what I want to do
[18:00] <jml> I want to link to apps.ubuntu.com from Launchpad. Specifically getting package pages to link to relevant apps pages.
[18:00] <lifeless> o/ nigelb
[18:01] <nigelb> Well, I guess I should head to bed :)
[18:01] <jml> I *think* I can get by with just a text transform, no queries.
[18:12] <lifeless> \o/ test perf fix worked
[18:12] <lifeless> Tests started at approximately 2011-10-31 08:45:30.145372
[18:12] <lifeless> Source: bzr+ssh://bazaar.launchpad.net/~lifeless/launchpad/useoops r14214
[18:12] <lifeless> Target: bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/ r14213
[18:12] <lifeless> 16646 tests run in 4:33:15.575714, 0 failures, 0 errors
[18:12] <lifeless> not as svelte as perhaps it was in the past, but not 7 hours either
[18:21] <jml> Is there a fast way of consulting Contents.gz from Launchpad code? (Is there a version in the db?)
[18:24] <jelmer> jml: the archivepublisher generates it and I'm pretty sure it doesn't store it in the database
[18:25] <jelmer> jml: what are you trying to do?
[18:25] <jml> jelmer: link to apps.ubuntu.com for a package from packages on Launchpad
[18:26] <jml> jelmer: but apps.ubuntu.com has only apps, which are packages that have desktop files
[18:26] <jelmer> jml: ah, and you need the section and the like?
[18:26] <jml> jelmer: no, just to know whether or not it has a desktop file
[18:26] <jml> e.g. https://apps.ubuntu.com/cat/applications/oneiric/evolution/ vs https://apps.ubuntu.com/cat/applications/oneiric/openssh/
[18:27] <lifeless> jml: there is a version being populated in the DB
[18:27] <jelmer> jml: do .desktop files necessarily indicate applications? IIUC it's possible to have .desktop files for non-application things
[18:27] <lifeless> jml: StevenK knows the status of that
[18:27] <jml> jelmer: that's how they make the distinction in Software Center, I'm told.
[18:28] <lifeless> jml: My first thoughts about how to implement this are:
[18:28] <lifeless>  - what about derived distros ?
[18:29] <lifeless>  - and if software centre knows, why not consult with it (e.g. from client js, or even from the appserver) to see whether you can link to it safely
[18:29] <jml> lifeless: I only want this for Ubuntu, not for derived distributions.
[18:31] <james_w> software-center has some fudges to determine the list of apps I believe, some are ignored, others are added
[18:31] <lifeless> jml: poking at contents because thats what SC uses seems like an abstraction violation to me
[18:31] <jml> lifeless: yeah, that's another option. More implementation work than using a table LP already has, but maybe more feasible. Would probably have to add that API.
[18:31] <james_w> I wonder if apps.ubuntu.com will grow the "technical items" thing too
[18:32] <lifeless> jml: this is so folk can find out how to install the package or something?
[18:32] <jml> It's hard enough writing code for what exists now. Dealing with possible futures is going to make this impossible.
[18:32] <lifeless> jml: a different approach would be to embed the content you're interested in on the relevant LP pages. E.g. reviews.
[18:32] <jelmer> jml: is software center using contents.gz directly though? I thought all that data was pre-generated in app-install-data
[18:33] <lifeless> jml: In my head there are 2 services (LP and SC) and 2 UI's (LP and SC) here, so having LP's UI talk to SC's service for some stuff is totally fine.
[18:33] <lifeless> jml: (if SC is fast enough)
[18:34] <jml> lifeless: largely the motivation is to increase google juice of apps.ubuntu.com. I guess it could also help developers using LP to quickly get a sense of what their package looks like to end users.
[18:34] <jml> lifeless: would look at reviews stuff in a later iteration.
[18:34] <jml> jelmer: quite possibly.
[18:35] <jml> lifeless: agree re abstraction violation. Doesn't feel a bad enough one to stop me though. Difficulty of implementation is more persuasive
[18:36] <lifeless> jml: such violations create debt instantly
[18:36] <lifeless> jml: I'd rather we not link at all than do it poorly
[18:36] <jml> lifeless: right, but I don't even have a working prototype that I can improve at this point
[18:38] <jml> jelmer: does that change anything?
[18:38] <jelmer> jml: does what change anything? the fact the data is pregenerated?
[18:38] <jml> jelmer: yeah
[18:39] <jelmer> jml: I suspect there is more going on than just looking for .desktop files
[18:39] <jelmer> jml: probably filtering out things that are not for applications
[18:39] <lifeless> jml: I'm curious - why does apps. need google juice? I thought it was mainly a thin shell for its API used by the sc client on Ubuntu installs ?
[18:39] <jelmer> but that's all speculation
[18:39] <jml> jelmer: OK. It's definitely possible that I was misinformed.
[18:39] <jml> jelmer: certainly using some kind of provided API would avoid that.
[18:40] <jml> lifeless: It turns out that a lot of people don't actually look at the Software Center very much. People fresh from other platforms often just open a browser to search for software, rather than looking for Software Center (or an app store or what have you). When they do that search, we want them to find http://apps.ubuntu.com – the web directory for the users.
[18:40] <jelmer> jml: it seems like a bad idea for software center to have to download a 20Mb file every time something in the archive changes
[18:40] <jml> lifeless: (paragraph from an email I was drafting)
[18:40] <lifeless> jml: interesting
[18:41] <jml> lifeless: yeah.
[18:41] <jml> lifeless: linking from LP is not the beginning & end of our approach, but it's something that would probably help.
[18:41] <jml> jelmer: well, I get the very strong impression that mvo would prefer some debtags-based solution.
[18:42] <jelmer> jml: debtags seems like a better approach for this kind of thing than inspecting Contents.gz
[18:43] <jml> jelmer: I am thinking of volunteering to add debtags support. Sort of contingent on how hard it is to add a hyperlink to a web page though :)
[18:44] <jml> hmm.
[18:45] <jml> I guess a naive-but-restful API is to just GET the link and then see if it 404s
[18:45] <jml> there's probably a way to do that without dl'ing the whole page too
[18:46] <jml> wonder how late I can render the link and still get SEO
[18:46] <jml> (also, battery about to fail)
[18:49] <lifeless> jml: google execute js
[18:50] <lifeless> jml: so I expect quite late
[19:33] <lifeless> sinzui: hiya
[19:33] <sinzui> hello
[19:33] <lifeless> sinzui: I think we have a bug in team administration :)
[19:34] <lifeless> sinzui: see #launchpad for context. Seeking your opinion.
[20:01] <benji> sinzui: are you aware of a way I can postpone a user's team membership expiration date?  This is in reference to this question: https://answers.launchpad.net/launchpad/+question/177021
[20:02] <lifeless> benji: needs a duckie
[20:02] <benji> lifeless: come again?
[20:03] <lifeless> ~admin ;)
[20:03] <lifeless> the logo is, or was, a yellow rubber duck
[20:08] <benji> lifeless: I think you're saying that I should be made a member of some administrative team, ~admin doesn't seem to be it though (https://launchpad.net/~admin).  If so, who can give me such membership?
[20:08] <lifeless> benji: I was unclear; I mean you cannot; you have to get an admin to do it.
[20:09] <benji> ah!  (where, I presume admin == LOSA)
[20:09] <lifeless> yes
[21:36] <lifeless> \o/ instant OOPSes on qastaging.
[21:36] <lifeless> booyah
[21:37] <lifeless> flacoste: \o/ instant OOPSes on qastaging.
[21:37] <lifeless> booyah
[21:37] <flacoste> lifeless: awesome
[21:38] <lifeless> try it, its spicy ;)
[21:41] <beuno> lifeless, I like the word instant. Is it using rabbit or something like that?
[21:41] <lifeless> beuno: yes
[21:41] <beuno> lifeless, I WANTS
[21:41] <beuno> so you've been talking to rye, right?
[21:42] <lifeless> yes, he's production hardening the opened code base for your environment
[21:42] <lifeless> there is a wsgi enhancement needed for your tx stack before you can widely migrate
[21:50] <beuno> awesomeness
[21:57] <lifeless> beuno: see b.l.n/python-oops-wsgi
[22:38] <poolie> morning
[22:41] <wallyworld_> jcsackett: r=me but i think the template still references the old public_or_private() method name
[22:47] <jcsackett> wallyworld_: yes. yes it does. i'll fix that.
[22:47] <wallyworld_> thanks :-)
[22:50] <wallyworld_> jcsackett: i assume the stuff you will be sending through contains the necessary detail to understand what needs change wrt the searching and making it clearer when things are deleted etc?
[22:54] <jcsackett> wallyworld_: indeed.
[23:01] <wallyworld_> jcsackett: the fwd email is missing the original material
[23:02] <jcsackett> wallyworld_: oh fantastic.
[23:02] <wallyworld_> :-)
[23:17] <wgrant> lifeless: https://pastebin.canonical.com/54976/ is what we used to fix the the BPBs.
[23:17] <wgrant> A similar thing over SPRBs would be good.
[23:17] <wgrant> (I'm not here today)
[23:18] <lifeless> wgrant: ah, it is damage ?
[23:19] <wgrant> lifeless: Yes, failure counting except buggy.
[23:27] <lifeless> wgrant: qastaging. Rabbit. AMQP. OOPS. WIN WIN WIN
[23:27] <wgrant> lifeless: It works?
[23:27] <lifeless> hell yes
[23:27] <wgrant> Excellent! Have we stopped rabbit to see that it falls back?
[23:27] <lifeless> not yet
[23:27] <lifeless> been doing uds stuff
[23:33] <lifeless> wgrant: still, you should make qas oops and see the result.... its just nice
[23:35] <wgrant> lifeless: I already have :)
[23:35] <wgrant> It is indeed.
[23:35] <wgrant> oops-tools needs some major hacking, but that can be done soon.
[23:55] <wgrant> lifeless: Is https://pastebin.canonical.com/55154/ a yesterday-style query?
[23:55] <wgrant> lifeless: ie. EXPLAIN ANALYZE is instant, execution is 300ms?
[23:55] <lifeless> qaqs ok?
[23:56] <wgrant> Yeah.
[23:56] <lifeless> no limits  ?
[23:56] <wgrant> None.
[23:56] <wgrant> Should be 1 row.
[23:56] <lifeless> 300ms
[23:56] <lifeless> 323
[23:56] <wgrant> And explain analyze?
[23:56] <lifeless> 162
[23:56] <lifeless> 162
[23:56] <wgrant> It's 1ms on DF :/
[23:56] <lifeless> 0.6ms to profile
[23:57] <wgrant> lolnot
[23:58] <wgrant> It's not even a very unpleasant plan at all.
[23:58] <lifeless> ugh
[23:58] <lifeless>   AND (TeamParticipation.team = BugSubscription.person)
[23:58] <lifeless>   AND (Person.id = TeamParticipation.team)
[23:58] <lifeless> wtf
[23:58] <lifeless> oh, via TP, I guess.
[23:59] <lifeless> not the join I expected.
[23:59] <wgrant> StevenK: Can you QA or ignore jtv's notification refactor?
[23:59] <lifeless> wgrant: in fact, that will give N rows per subscription