[00:03] * maxb hopes the cell towers are only metaphorically down :-) [00:08] maxb: at least logically down [00:10] Anyone have any ideas why my tests failed? http://paste.ubuntu.com/570317/ (URLError: ) [00:11] huwshimi: Both known occasional spurious failures. [00:11] huwshimi: Land the branch manually. [00:12] http://maps.google.co.nz/maps?f=q&source=s_q&hl=en&geocode=&q=http://magma.geonet.org.nz/services/quake/kml/2.2/search?externalRef=3468575 [00:14] wgrant: You mean just 'lp-land'? Is there anything else I need to do?' [00:14] huwshimi: lp-land, yeah. [00:14] lifeless: 6.3? [00:14] yeah [00:14] but shallow [00:14] so felt it more [00:14] mum, down in dunedin, felt it [00:14] thumper: did you ? [00:14] Ah, very shallow, yes. [00:15] Any major damage known yet? [00:15] http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=10707996 [00:15] water mains burst, some buildings down, more road cracks [00:16] .au news sources haven't mentioned it yet... [00:16] phone lines are down [00:16] lifeless: surely you're not asking thumper if the earth moved for him? [00:16] 5km deep!? [00:16] That's awfully shallow. [00:16] wgrant: yup [00:16] its being called an aftershock for now [00:17] lifeless: yes I felt it [00:17] Ah, I see geonet.org.nz has it as 6.3 rather than 3.1 now. [00:17] lifeless: quite a shaker here [00:17] That's quite a difference! [00:18] it was felt in wellington too [00:19] USGS says 4km deep +/- 0.5km. [00:26] lifeless: Why do you invoke PersonSet directly instead of using the utility? [00:26] * lifeless shrugs [00:26] I think we need an object for 'group of ' [00:26] the Sets look ideal for that, if parameterised [00:26] I was heading slightly in that direction at the time [00:26] Sure. [00:27] feel free to do it any which way [00:27] doing DI on these singletons seems fairly uninteresting [00:28] rumours abound in the office here that the christchurch cathedral is down [00:28] yup [00:28] police are saying that [00:29] though the degree of down isn't clear [00:29] Ah, now it's hitting the news here. [00:29] 'Witnesses said there were buildings down all around Cathedral Square in the city, with the church destroyed.' [00:29] Hmmm. [00:31] another shock now [00:31] relatively gentle [00:36] Ah, the cathedral has just lost its steeple. Not tooo destroyed. [00:38] the bugger of it is that we're still waiting on a plumber having time to fix some shit for us - but other folk are more badly in need [00:40] anyone know why adding @operation_returns_collection_of(Interface) to a method with an existing @export_read_operation() decorator results in: [00:40] UnknownEntryAdapter: No IEntry adapter found for Interface (web service version: 1.0). Encountered as a result of the entry interface None, field ''. [00:40] there seems to be several other places in the code where the same pattern is used [00:41] and i can't find anywhere where adaptors are defined for those places [00:41] wallyworld: You need to patch them in lib/canonical/launchpad/interfaces/_schema_circular_imports.py [00:42] * wallyworld looks [00:42] wgrant: ah that looks like it. thanks. [00:43] i was getting a circular import when I tried using ISourcePackageRecipe instead of Interface [00:43] wasn't sure how to fix it but now i know :-) [00:44] http://www.youtube.com/watch?v=qt0iIHXFnR0 [00:44] That's quite a rock. [00:44] * thumper stabs the compressed javascript [00:45] wallyworld: Yeah, it's a bit awful :( [00:45] awful is one way to describe it :-) [00:45] wallyworld: thats sumner [00:46] bah [00:46] wgrant: thats sumner; just around the hills from where our rental property was [00:47] * wallyworld was wondering what sumner was :-) [00:49] wallyworld: its a suburb [00:50] i know that now :-) [00:50] but had to wait for you to post a subsequent message to figure it out :-) [00:52] lifeless: Congrats! [00:53] wgrant: thanks [01:18] lifeless: Just saw your post: congrats! [01:18] thanks [01:20] lifeless: My wife is three weeks ahead of yours. [01:20] congrats [01:20] lifeless: Thanks. My first. You have others right? [01:20] nope [01:21] we're late to the sprogging game [01:21] lifeless: Oh, I'm thinking of someone else. Exciting times then [01:26] lifeless: congrats! [01:26] spiv: thanks [01:33] lifeless: you are always on irc etc, don't know where you found the time for species propagation :-P [01:34] Multitasking, perhaps. [01:34] spiv: but python is single threaded :-) [01:35] wallyworld: doesn't mean *I* am :> [01:35] lifeless: so you can type and do "other" things at the same time. must have been hard to set up your desk that way :-) [01:39] no comment, keeping the workplace clean since 1436 [01:39] bbiab, going to try and top up our supplies before supermarket shelves run dry - we have enough but its bland fare, better to have the remainder of fresh meats etc [02:16] what does this do? --> ${objects/?key/webservice:json} [02:16] the ?key has me wondering [02:18] Literally '?key'? [02:18] Ah, yes. [02:19] thumper: It uses the value of the 'key' variable to continue traversal. [02:19] ah... ok [02:19] thanks [02:19] that makes sense [02:55] i can has review? https://code.launchpad.net/~wallyworld/launchpad/recipe-webservice-api/+merge/50684 [02:57] 90 def getBuilds(): [02:57] 91 """Return a ResultSet of all the non-pending builds.""" [02:57] That's an odd method name, if the docstring is correct. [02:59] wgrant: i didn't write it, but it does make sense if you realise there's also a getPendingBuilds() method [02:59] which returns the pending builds :-) [02:59] wallyworld: We should export a sane API if at all possible, and that method name is insane. [03:00] Does the method need to not return pending builds? [03:00] wgrant: yes, it needs to not return the pending builds [03:00] Hmm. [03:00] Is that useful for API users, though? [03:01] wgrant: there was a refactoring previously which changed those method names. i can't recall the exact circumstances now since i didn't do it [03:02] i guess the method names could be changed [03:02] again [03:02] We shouldn't be exposing suboptimal interfaces to people who are going to use them forever so we can never change them. [03:02] We can rename just the exports if required. [03:03] wgrant: so given there's 2 methods - getPendingBuilds() and getBuilds() - is it just the getBuilds() name you have a problem with? [03:04] wallyworld: I think a single method with arguments may be better. But getPendingBuilds on its own is possibly OK. [03:04] wgrant: there used to be a single method with args and the refactoring converted it to 2 methods :-) [03:06] Why? TAL? [03:07] wgrant: not sure exactly. i was never involved in the decision, just noticed that it had happened [03:07] the getBuilds() method returns all builds where status != NEEDSBUILD [03:07] so that could be successful, failed, whatever [03:08] Hmm. [03:08] getNonPendingBuilds() sounds a bit funny, or? [03:09] getFinalisedBuilds? [03:09] Terminal [03:09] Complete [03:09] Ahh, it's used by builds_for_recipe. [03:09] complete implies success [03:09] I don't think exporting them as separate methods is useful. [03:09] i like finalised perhaps [03:09] wallyworld: complete does not imply success [03:09] wgrant: so we could have an api just just gets all builds [03:10] i'll do this: getBuilds() returns everything and is exported. i'll add a getCompletedBuilds() which does what getBuilds() does now [03:10] sound ok? [03:11] Maybe have getBuilds(include_pending)? [03:11] wgrant: that's what we had before the refactoring :-) [03:12] lifeless: Tracebacks in OOPSes are very enlightening. [03:12] yes [03:12] lifeless: Could not work out where some queries were coming from... turns out they were in __storm_loaded__ in various places. [03:12] Grr. [03:12] wgrant: the 'is it a team' check ? [03:13] lifeless: And is the distribution a derivative or not. [03:13] Queries for Ubuntu on every distro load. [03:23] let me just say arrrrrrrrrrrrrgh [03:25] Oh? [03:31] ahh, inTeam often issues two queries. That sort of sucks. [03:31] It's very tempting to preload all participation IDs at the start of the request... [03:32] We can get rid of a lot of queries if we just prefetch the user's participations and a few celebrities. [03:33] lifeless: Have you thought about that at all? [03:42] yes [03:42] net loss I think [03:42] getting one TP entry is very fast [03:43] some users have hundreds of TP entries and we'd only be looking at 2 or 3 [03:43] wgrant: consider lying to objects instead. [03:43] wgrant: that is, if we load an object from a context where view/edit is known a priori, grant it during the load, the way bugs _known_viewers gets data injected into it [03:44] also [03:44] bug 722447 [03:44] <_mup_> Bug #722447: checking 'well known' teams requires two queries at startup and in tests < https://launchpad.net/bugs/722447 > [03:44] lifeless: Getting 200 TPs takes 200 microseconds in the DB. [03:44] I tried with person=1 [03:45] wgrant: hot or cold [03:45] That was cold. [03:45] consider storm deserialisation - its not brillianrtly fast [03:45] This is probably the most trivial table in the DB. [03:45] wgrant: perhaps you mean milliseconds ? [03:45] It's deserialising ints... it goes nowhere near Storm. [03:45] No, 0.2ms. [03:45] nice [03:45] well, lets say I'm open to data on this [03:45] I guess the table is so small that it may have been hot enough even on mawson. [03:45] It is just two ints... [03:46] launchpad_qastaging=> select count(*) from teamparticipation where person=1; [03:47] Time: 3.323 ms [03:48] wgrant: its got 1.4m rows, so fairly small [03:48] and yes, a narrow table - like bugmessage [03:48] wgrant: I'm not saying no, I'm saying its not a no-brainer [03:50] wgrant: if you bring back team names: [03:50] Time: 1694.335 ms [03:50] wgrant: it becomes a lot slower [03:51] that was 'select team.name from person as team, teamparticipation where teamparticipation.person=317562 and teamparticipation.team=team.id;' [03:51] lifeless: Sure. But why would we want team names? [03:51] Of course a join across a wideish table is going to be slow. [03:51] wgrant: just bringing back ids is pretty useful. [03:51] s/useful/useless/ [03:53] * wallyworld wants to stab windmill through the heart [03:53] AttributeError: 'AppYUIUnitTestCase' object has no attribute 'client' [03:53] WTF. [03:54] lifeless: It's completely sufficient for inTeam, though. [03:54] Just not FF. [03:54] its only sufficient for inTeam if the team id is known a priori [03:55] wgrant: we can instead do an inTeam on PersonSet which would issue one fast query. [03:57] interestingly, teamparticipation seems to have redundant entries for code in offspring hackers - three rows [03:59] doh, bad query [04:05] lifeless: But that would require restructuring huge amounts of code. [04:06] If we prefetch participations then later prefetch relevant persons, we can make all of the permission checks free with very little code. [04:07] wgrant: my concern is three fold: [04:08] - This sort of thing eventually becomes an albatross : its hard to undo an optimisation like you're proposal [04:08] - its not inherently scalable [04:08] - its very data set specific [04:08] we should be able to get down to 4 or 5 teamparticipation checks per page without any special case [04:09] I think we should do that first - thats a whole 30ms : we're dealing with 10000ms pages at the moment, so lets reserve the special cases until they are a big win, not a little win [04:11] Very data set specific? [04:12] The inTeam thing needs to be done in a single place, and it will optimise every permission check in the project. [04:13] at the cost of loading a potentially unbounded dataset on every request [04:13] thats a risky design [04:13] now, if you were to load just the tps relevant to the teams you'd also preload, that addresses that scaling factor [04:13] but it still doesn't address the data set specific nature [04:14] by which I mean that the teams relevant to a given page depend on the pillar, the privacy settings, the bug settings etc [04:14] and we should be doing *at most* 2 queries for the same stuff today [04:14] so the maximum possible win is 30-40ms on a page. [04:15] i don't understand - given the existing caching - how you can save more than that [04:15] \o/Test results: bug-717394 => devel: SUCCESS [04:16] lifeless: BugTask:+index does several for each task and nomination. [04:17] so, you'll need to eager load all the /teams/ for that , right ? [04:18] even if we had the persons ids, we'd still trigger - in some cases - 100 queries [04:18] Right, so I could precache participations at the same time. [04:18] exactly [04:18] But getting all of them is *so* cheap for even our worst case right now. [04:18] but its not incrementally useful given that we have to do the team eager loading anyway [04:19] For this particular case, no. [04:19] But it's harder to make it specific to this case, and it is less globally useful. [04:20] mmm [04:20] I really think its better to sneak up on this [04:21] We can solve all of them ever with less work than solving this specific case, with the only risk being that someone joins a million teams, in which case we have bigger problems. [04:21] perhaps I'm misunderstanding bu that sounds completely false [04:22] Oh? [04:23] see above under having to special case this situation anyway [04:23] we've no particular reason to think that any of the late evaluation triggers would not need special casing to load their teams. [04:23] And lots of reason to think that they *will*, even with a seed-at-start-tp-list-for-the-current-user [04:25] its less work to write one eager-load-for-doing-a-inTeam-check helper and use it where we need it than to write a tp loader, change inTeam to be handle that extra cache, write a eager loader for groups of teams, and apply that in the same places [04:25] What part of my statement above was completely false? [04:25] that it was less work [04:25] If we have a global one, all we need to do is preload persons every time. [04:26] If we don't, we have to preload persons every time and then possibly preload participations if we think they will be necessary. [04:26] We *will* need at least one participation check on edge authenticated page. [04:27] I think that one check might as well do the trivial extra work that lets it fulfil *every* subsequent check. [04:27] s/edge/each/ ? [04:27] Uh, yes. [04:27] WTF [04:28] so [04:28] this isn't the way I'd do it [04:28] I think its a poor design [04:28] if you're doing the work, you get to choose [04:29] I've laid out my concerns [04:29] at this point we're clearly missing each others point, or something. [04:29] I think so. [04:30] I *will* insist that if you do it, you instrument it [04:30] * StevenK suspects dogfood is now missing recipes [04:30] so that we can see the overhead clearly in the PPR or a similar system [04:31] Where can I see feature flags? [04:31] StevenK: Watch out, DF will be a bit broken. [04:31] the one guaranteed thing is that users will do crazy things with teams and memberships and so on [04:31] It needs a new DB. [04:31] StevenK: /+feature-rules [04:31] wgrant: Rargh, I need it :-( [04:31] StevenK: Add 'code.recipes_enabled default 0 true' [04:31] StevenK: It should mostly work. [04:31] Just bugs will do very strange things. [04:31] And it can't run latest db-devel. [04:32] I merged devel into it this morning, though. [04:32] Heh [04:32] By accident? :-) [04:32] No. [04:33] wgrant: Recipes enabled, let's see [04:33] wgrant: We can probably sort out a new DB for it soonish. [04:34] if you've still got disk space [04:34] lifeless: So, I agree that unconditionally loading an unbounded and excessive dataset is a bad thing. [04:34] lifeless: We still have 110GB free, with a DB dump and one full DB. [04:34] wgrant: whats \l+ show [04:35] launchpad_dogfood | postgres | UTF8 | C | C | | 222 GB | pg_default | [04:35] so, you're going to lose 80GB of that 110 [04:35] the db is ~300GB now [04:35] I was thinking it was ~260. [04:35] But maybe that's old. [04:35] it was [04:35] it was 290GB during the last db deploy IIRC [04:37] TBH, we tend to update dogfood by scorched earthing the DB and re-importing [04:37] Which is a little ... brutal [04:37] sure [04:37] point is, you can't run a db and a dump now [04:37] Hm, I wonder if I can order \d+ by size, or if I have to go querying manually... [04:38] lifeless: If I'm reading what wgrant said correctly, we already have a full DB and a full DB dump on dogfood. So if we delete both, we can get another dump and import it [04:39] That's how it's normally done. [04:39] Since the dump compresses well. [04:39] Pity it takes mawson 40 hours to import [04:39] StevenK: you can get another dump, but the dump will be bigger too; if its 30GB bigger you'll run out of disk space mid import [04:39] well, 90% or so [04:40] lifeless: Er, we have 110GiB free? [04:40] It's not 30GB bigger, fortunately. [04:40] How does a 30GiB increase to both eat 110? [04:41] StevenK: The dump is much smaller than the full DB. [04:41] The DB itself has grown by 80GB. [04:41] (how!?) [04:41] The dump will also have grown by a few. [04:41] Either I'm missing something or lifeless is. I'm not sure who yet. [04:41] 110 - 80 - a bit is almost 0 [04:41] StevenK: you have 110GB free [04:42] StevenK: the db is 80GB bigger, that means you'll have 30GB free for the dumps increase in size. [04:42] public | branchrevision | table | postgres | 26 GB | [04:42] I think there's a good 12 or so GiB I could free in a hurry if i need to. [04:42] And then there are four revision-related indices that are about 15GB each :/ [04:43] That's like a third of the DB. [04:43] launchpad@mawson:/srv/launchpad.net$ ls -lh | head -n 2 | tail -n 1 [04:43] -rw-r--r-- 1 launchpad launchpad 8.0K Aug 27 16:50 ,x [04:43] * StevenK chuckles [04:45] Yeah, 100GB goes to branchrevision. [04:45] Awesome. [04:45] Can't we delete that monstrosity yet? [04:45] Wait. [04:46] That's 100GB of 222GB. [04:46] Not 300GB. [04:46] That is huge. [04:46] lifeless: I suspect you didn't close bugs after the deploy you requested. [04:47] StevenK: it was midnight. I suspect that too [04:47] sigh buildbot [04:48] StevenK: did you get any traction on why jenkins appeared more fragile than buildbot? [04:48] Because windmill. [04:49] It was fine until windmill :( [04:49] It's been behaving fine this week and last, mind you. [04:50] But Windmill makes me very sad. [04:50] windmill makes its mother sad [04:50] "revisionnumber_branch_id_unique" UNIQUE, btree (branch, id) [04:50] "revisionnumber_branch_sequence_unique" UNIQUE, btree (branch, sequence) [04:50] grrr [04:50] bug 1 timeout >< [04:50] <_mup_> Bug #1: Microsoft has a majority market share

That first index is a bit off, since id is the pkey... [04:51] it may be used [04:51] lifeless: Which query? [04:51] alternatively we could rely on heap joins and just do single column indices [04:51] dunno yet [04:51] OOPS-1879G332 [04:52] Here too. That's a bit upsetting. [04:53] Death by a couple of hundred Person queries, I guess. [04:53] lifeless: Thanks for the bug closures. [04:53] StevenK: thanks for the reminder; the list was at the top of LPS so easy to do (you could have done it :P) [04:53] "Meh" :-) [04:54] lifeless: I realise I'm blocking the QA pipeline, but I need to check with sinzui on the best way to QA the DMP. [04:54] And he wasn't around yesterday. [04:54] wgrant: we can't run it in staging [04:54] Exactly. [04:55] so, the answer is 'hope' [04:55] Haha [04:55] Mm, I guess. [04:55] * wgrant untestables. [04:55] same with branch mirroring [04:55] We can QA that. [04:55] Just need to find a consenting victim and disable the rest. [04:57] looks new: 23 / 1 BinaryPackageBuild:+retry [04:57] On prod? [04:57] Argh, not again. [04:57] That could well have been during the fuckage last night. [04:57] Could it possibly be fallout? [04:57] With buildd-manager and p-u holding locks open for an hour. [04:58] * StevenK high fives wgrant for having the same thought. [04:58] wheeee [04:58] Hmmm [04:59] I am really disturbed that nagios didn't notify us within a minute of the incident beginning... [04:59] it was probably swapping [05:00] And? A missing probe is a failed probe... [05:00] actually [05:00] its more nuance than that ;) [05:00] wgrant: As a thought: a way to handle cancelling distro builds: add a new status: BuildStatus.REFUSED or so which process-upload will silently reject binaries. And then instruct buildd-manager to shoot the buildd in the head. Thoughts? [05:00] StevenK: We can't cancel builds on the non-virt builders. [05:00] lifeless: It *is*, but it shouldn't be. [05:01] wgrant: talk to the netsaint designer [05:01] My Nagios days are hopefully over. [05:01] Nagios makes me sad, too. [05:01] I used that enough at my last job. [05:06] StevenK: Anyway, what were you doing to mawson? [05:06] wgrant: Testing https://code.launchpad.net/~stevenk/launchpad/link-recipe-on-ppa-page/+merge/50694 [05:06] wgrant: Can has review? :-) [05:08] StevenK: Could you QA thumper's thing while you're there? [05:08] Which 'thing'? [05:09] r12410 [05:09] Recipe FF changes. [05:09] Main thing that needs testing is r-d-b [05:10] OOPS syncing seems to be really slow. [05:10] G332 still isn't there. [05:10] wgrant: With the flag set, it runs, but there are no daily builds to build. I can reach into the db and set one if you wish [05:10] StevenK: Could remove the flag and try again? [05:11] It needs to run fine without it. [05:11] And be able to request builds. [05:12] wgrant: Same output as with the flag [05:12] StevenK: Could you staleify a recipe and try without the flag? [05:13] wgrant: you were interested in OOPS: https://dev.launchpad.net/LEP/OopsDisplay [05:15] lifeless: Excellent, looking. [05:15] Oh, bah, the recipe I forced to be stale has built recently [05:16] * StevenK reaches back into the db [05:16] Die forster die. [05:16] wgrant: we need an admin for that [05:16] thumper: Or a mawson. [05:16] And given the lack of admins this week, it is tempting to use mawson. [05:16] It's where we normally test daily builds, anyway. [05:18] thumper: Also, do you plan to remove the unused key from the production configs? [05:18] If you don't, it will make me sad. [05:18]