[00:08] sinzui is probably offline now [00:08] btw [00:08] we're getting a whole bunch of script failed to run errors [00:20] lifeless: Please hold that MP, it is broken [00:23] pcjc2: ec2 should bounce it [00:23] how is it bust? [00:24] I moved the EXISTS() into the clause generator, but stupidly put it _inside_ the clasuse which is UNION'd when you have more than one tag queried [00:24] whee [00:24] So I moved the EXISTS outside the join, but now have to return "None" for the no-tags passed case [00:24] seems like a plan [00:24] that's why I shouldn't code when tired [00:25] * pcjc2 aims clue-bat at own head [00:25] I _thought_ I had run the tests [00:45] pcjc2: have you pushed? [00:45] yes [00:45] * pcjc2 hopes this time it is not completely broken [00:45] will be worth giving it another quick review though.. as it has changed a little [00:47] Only bugtask.py has changed since the last push [00:47] I guess real LP developers have lots of irons in the fire at once.. for all the pause-time running tests etc.. [00:48] high throughput, high latency [00:51] a bit yeah [00:52] pcjc2: kicked it off [00:52] Thanks [00:52] EC2 is a cloud server instance? [00:53] amazon ec2 [00:54] ok - had no idea you used Amazon servers [00:54] is that just for testing? [00:54] =yes [00:57] ok, its away [01:15] Project db-devel build (266): STILL FAILING in 4 hr 23 min: https://hudson.wedontsleep.org/job/db-devel/266/ [01:15] * Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 12191 [01:15] included. [01:15] * Launchpad Patch Queue Manager: [r=abel, [01:15] stub][ui=none][bug=276488] Add a nameblacklist admin field that [01:15] exempty the assigned team from the restriction. [01:18] Project devel build (358): STILL FAILING in 4 hr 28 min: https://hudson.wedontsleep.org/job/devel/358/ [01:18] * Launchpad Patch Queue Manager: [r=benji, [01:18] edwin-grubbs][ui=none][bug=696913] Adds doNotSnapshot to fields [01:18] involved in OOPSes with ShortlistTooBigErrors. [01:18] * Launchpad Patch Queue Manager: [r=jml][ui=none][no-qa] Upgrade the version of Twisted that Launchpad [01:18] uses from 10.1 to 10.2, preserving the patch for Twisted bug 4395. [01:18] <_mup_> Bug #4395: wings3d: merge new debian version < https://launchpad.net/bugs/4395 > [01:20] Hmm, those two errors look gmpy-related. [01:20] wgrant: which ones? [01:26] jelmer: Hudson's. entitlement.txt and looptuner.txt [01:26] I don't know why gmpy would be installed there, though. [01:27] for codehosting ssh exchange performance [01:27] Uhoh. [01:27] wgrant: https://lpstats.canonical.com/graphs/CodehostingPerformanceStaging/ [01:29] lifeless: OK, I see your point, but test failures. [01:30] wgrant: agreed [01:30] wgrant: not sure whats up there [01:30] wgrant: given it had to go through bb etc [01:30] lifeless: I bet buildbot doesn't have python-gmpy installed. [01:30] I can reproduce locally once I install it. [01:31] perhaps it wasn't added to the deps? [01:31] or $something [01:31] It wasn't, AFAICT. [01:31] Hmm. [01:31] Why does codehosting send me the URL to the MP twice? [01:32] In MP emails, that is. [01:32] For more details, see: [01:32] https://code.launchpad.net/~wgrant/launchpad/rebuilds-without-nai/+merge/46070 [01:32] -- https://code.launchpad.net/~wgrant/launchpad/rebuilds-without-nai/+merge/46070 You are the owner of lp:~wgrant/launchpad/rebuilds-without-nai. [01:37] lifeless: that drop is pretty awesome [01:39] I'd go a bit further than that. [01:39] Although that could be because qastaging's bzr is broken. [01:40] is it? [01:40] bzr: ERROR: Server sent an unexpected error: ('error', 'No module named bzrdir') [01:40] oops [01:40] heh! [01:40] jam: ^ [01:40] * thumper chuckles [01:41] Is qastaging running the forking lp-serve thing? [01:41] yes [01:43] wgrant: well, the graph is doing "hello" which does work on qastaging [01:43] "echo hello | ssh bazaar.qastaging.launchpad.net bzr serve --inet --directory=/ --allow-writes" [01:43] however, there is, indeed, something broken [01:44] lifeless: is there a place where qastaging log files will be copied, such that I can get a look at them? [01:45] knowing who is asking for bzrdir and not finding it would be helpful [01:49] tellurium's codehosting-access.log is a couple of months out of date on carob :( [01:49] I wonder if the qastaging stuff doesn't sync yet. [01:50] the log file is at /srv/bazaar.qastaging.launchpad.net/qastaging-logs/codehosting/bzr-lp-forking-service.log [01:50] or possibly another one [01:51] Nothing like that on carob. [01:51] spm: ^^ are qastaging codehosting logs not syncing? [01:52] goodnight [02:05] jam: devpad [02:05] wgrant: well, I did check again that running the service locally works, and I can push through local codehosting [02:06] lifeless: looking at /srv/launchpad.net-logs/qastaging doesn't have much [02:06] Hmm, odd. [02:06] wgrant: yeah [02:06] check with losas in -ops [02:06] jam: There is some stuff under /srv/launchpad.net-logs/staging/tellurium/codehosting. [02:06] ah, you are [02:06] lifeless: no response yet [02:06] But nothing from qastaging, and nothing since November. [02:06] So it may have all broken when qastaging was introduced. [02:07] 811.51s OOPS-1839REPORTIFSEEN1172 None [02:07] Yay [02:13] wgrant: that means something running as 'production' [02:33] lifeless: I know. I was just surprised to see it being used for something that takes >10 minutes. [02:34] I thought it would be mostly tiny things that nobody has looked at for 5 years. [02:36] wgrant: right, so the question is do we want to bless doing this, or move them to another config [02:39] lifeless: Kill them. [02:44] I need to do a similar thing to cesium. [02:44] It uses the ftpmaster config :( [02:48] wgrant: if you're looking at this stuff, I'd love it if you file some bugs [02:48] (Or JFDI) [02:57] lifeless: Or we could adopt your OOPS naming strategy. [02:57] And then we can delete most of the configs. [02:57] spm: ^^ are qastaging codehosting logs not syncing? <== no. never been setup to [02:58] spm: :( [02:58] wgrant: I want to do that [02:58] spm: please do?! [02:59] yeah, am atm. needs rotatin' as well [02:59] fwiw, have I ever mention just how much I despise the default twistd logging? [02:59] log rotation* [03:01] Why does Australia fail at summer? [03:01] Now we are flooding too :( [03:02] * spm tries not too feel smug at 650m above sea level [03:06] spm: thanks [03:06] * wgrant throws a few buckets of water at spm. [03:06] ta, but we've had enough rain lately. thanks tho! ;-) [03:08] :( [03:09] still overcast today. yeck. [03:10] The weather today is pretty nice. [03:10] Except that the ground is so saturated that parts of my floor appear to be exuding water. [03:10] The humidity is <90%, though, which is a pleasant change. [03:17] heh, we all start dying from excessive humidity when it gets above 25% [03:35] the yak shaving on tellurium continues. just about in tears of laughter here..... [03:36] I see we have a directory, but no content. It does sound fun. [03:48] I was being excessively optimistic creating the directory, I feel [03:57] the yak, be shavéd. [03:59] Indeed, thanks. [04:01] jam: There are logs now. [04:02] /srv/launchpad.net-logs/staging/tellurium/qastaging-codehosting/codehosting/bzr-lp-forking-service.log [04:05] Hmm, did devpad's ssh host key change relatively recently? [04:06] spiv: if by relatively recently you mean months ago, yes [04:06] It changed from sodium to carob at some point over a month ago. [04:07] elmo: months ago is entirely possible. [04:07] Thanks. [05:39] Project db-devel build (267): STILL FAILING in 4 hr 23 min: https://hudson.wedontsleep.org/job/db-devel/267/ [05:39] Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 12192 [05:39] included. [05:45] Project devel build (359): STILL FAILING in 4 hr 27 min: https://hudson.wedontsleep.org/job/devel/359/ [05:45] * Launchpad Patch Queue Manager: [r=jml][ui=none][no-qa] Always use Python 2.6. [05:45] * Launchpad Patch Queue Manager: [r=julian-edwards][ui=none][no-qa] Use the uploader db user for [05:45] ppa-add-missing-builds. [05:45] * Launchpad Patch Queue Manager: [r=adeuring][ui=none][bug=697685] Adds validation to [05:45] RequestPeopleMergeView so that persons with PPAs cannot be merged. [06:05] * huwshimi is off [06:08] huwshimi: See you in Dallas. [06:08] wgrant: Yeah will do :) [06:09] have safe flights guys! === almaisan-away is now known as al-maisan [06:29] wgrant: when are you coming? [06:38] hmm [06:38] We need a way for the code import scheduler to not dispatch multiple jobs for the same svn repository to the same machine simultaneousl [06:38] y [06:39] not sure if there's a bug for that; but that's been an operational issue for us before. so +1. [06:40] Because for noteable one-repo-many-projects repositories (like Apache), we now can't start any new code imports, because they take so long to initialize the first time, another job comes along, and kills the long running one with a sqlite error [06:40] lifeless: I depart at midday on Sunday. [06:40] Of course, this requires figuring out what it means to "be from the same repository" [06:41] yes :-) [06:41] lifeless: Arriving DFW at 15:10 [06:41] for which svn has a perfectly good UUID, but I can't think how to get it available in the DB/appservers [06:42] maxb: Particularly when it is the initial imports that often fail :( [06:42] I suppose one possibility would be to cheat... add external whole-job locking by uuid to bzr-svn, such that it's any new jobs which fail, rather than new jobs killing jobs which have been running for hours [06:43] I can't think of much beyond making the import take a UUID-based lock. [06:43] Right. [06:43] Preferably making the subsequent jobs not really fail at all. [06:43] Possibly adding a new "Skipped" job status, such.... snap :-) [06:43] But just rejecting them softly, not contributing to failure counts. [06:43] maxb: I think it'd be more worthwhile to spend time on the new cache format /or/ switch to tdb. [06:43] We are of one mind :-) [06:44] jelmer: My personal experiences with tdb are that it sucks badly once the cache gets into the gigabyte range [06:44] Also the operational pain of needing to rebuild the cache for all code imports makes me weep [06:46] maxb: It's supposed to scale pretty well even in that regard [06:46] maxb: It's worked fine for larger branches for me in the past [06:46] maxb: are you on NFS? [06:46] jelmer: it, uh, doesn't :-) [06:46] I've gone back from tdb to sqlite for performance [06:46] maxb: Have you reproduced that recently? [06:47] even though it triples the size of the file [06:47] No NFS, just plain local disk [06:47] I wouldn't say *recently*. On the other hand, I've not noticed any tdb updates in Ubuntu in a while [06:48] maxb: e.g. the bucket sizes are different these days [06:48] isn't tdb a tridge/samba thing? ergo perfect already? [06:48] spm: it's a tridge/rusty thing :-) [06:48] near enough... ;-) [06:49] there's a new format on the way with some improvements, but I'd be really surprised if the current performance is worse than sqlite [06:49] jelmer: hmm. So, you think that might solve the problem of performance sucking unless the entire file was in kernel cache? [06:51] maxb: that depends entirely on the access patterns of the user [06:52] access pattern: fetching svn rev info for ~1000 new revs [06:59] that shouldn't require having the entire db in the kernel cache in the current format either [07:01] * jelmer gets some sleep [07:01] good idea :) [07:01] * maxb curses at private code imports causing the code import list pages to oops [09:02] good morning === al-maisan is now known as almaisan-away [09:38] mrevell: I'd like some advice on the PPA creation and edit UI, if you're free. [09:45] wgrant, I can be, although my day's pretty full. Is it something we could talk about next week? Perhaps with huwshimi. [09:45] mrevell: Sure. [09:45] Thanks. I'll make a note to ask you about it. [09:49] Thanks. [10:04] Project db-devel build (268): STILL FAILING in 4 hr 25 min: https://hudson.wedontsleep.org/job/db-devel/268/ [10:04] Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 12193, [10:04] 12194, 12195, 12196 included. [10:12] Project devel build (360): STILL FAILING in 4 hr 27 min: https://hudson.wedontsleep.org/job/devel/360/ [10:12] * Launchpad Patch Queue Manager: [r=lifeless][ui=none][bug=501945] Improve SQL efficiency when [10:12] searching for bugs by tag (Pre-filter by Bug.id) [10:12] * Launchpad Patch Queue Manager: [r=jcsackett, [10:12] Edwin][ui=none][bug=183372] Add a heartbeat to Mailman's xmlrpc log. [10:12] * Launchpad Patch Queue Manager: [r=thumper][ui=none][no-qa] Update loggerhead to get important bug [10:12] fixes and performance improvements. === almaisan-away is now known as al-maisan === al-maisan is now known as almaisan-away === matsubara-afk is now known as matsubara [11:57] Is anyone else finding pushing branches to LP to be really slow or is it just my connection? [12:05] Morning, all. === Ursinha-afk is now known as Ursinha === almaisan-away is now known as al-maisan === mrevell is now known as mrevell-lunch === al-maisan is now known as almaisan-away === almaisan-away is now known as al-maisan [13:15] hi === mrevell-lunch is now known as mrevell [13:48] mrevell: For the blueprints stats, did you want numbers per project (in the top 50), or just a total? [13:48] allenap, Per project, if that's okay, purlease [13:49] mrevell: It is totally okay. [13:49] I have to state that I wasn't on lunch that long ... I just dived back into work without checking my IRC nick. [13:49] Is the code for the Launchpad QA Bot available somewhere? [13:50] Looks like bazaar.qastaging.launchpad.net redirects infinitely, so I can't test loggerhead on qastaging. [13:51] pcjc2: Its not open sourced yet; it is planned to be [13:51] ok - just I was writing something similar for git commits, and wondered if I could steal some ideas, that was all [13:52] Design is going to be Git HOOK -> sqlite database with a "work queue" -> gpleda.org robot -> LP [13:52] The database in the middle is to absorb any grief which might happen if LP is down when commits come in [13:52] Possibly not an issue for the QA bot [13:52] pcjc2: I'd be surprised if there isn't already such a system for git. [13:52] I wouldn't know where to look for it.. [13:53] pcjc2: #git would know. [13:53] I'm also writing (going to write) a small patch to our gitweb instance to look for the same "Closes-bug: lp-12345" syntax we intend to use, and link to LP [13:54] lifeless: When will that merged bug tag code hit staging? [13:54] (Or has it done so already - bug is tagged qa-needstesting) [13:55] qastaging.launchpad.net has it already [13:55] I was just going to do a few searches and see how it looks [13:55] ok- I was testing on staging [13:55] inkscape fails on staging [13:55] staging is where things with db schema patches, and some specific services are qa'd [13:56] damn - also fails on qastaging [13:56] qastaging is where things that can be continuously deployed are tested [13:56] works for me [13:57] I did a search on launchpad for tag -* [13:57] ok [13:57] copied the search to production [13:57] works now [13:57] fail [13:57] you can hit cold cache effects [13:57] I had a search URL from staging, changed to "qastaging" [13:57] staging has 64GB or ram or something small like that [13:57] so cold cache -> timeout? [13:57] db size is 250GB [13:58] ok, but there is some improvement on qastaging [13:58] one can at least perform the query [13:58] at this stage in our performance overhaul we generally are happy if it works a second time [13:58] I'm amazed that on inkscape, out of 2805 open bugs, there are only 288 without tags [13:59] This was the kind of query I originally wanted to do on our bugs (but I realised afterwards that we imported everything with _some_ tag, so it doesn't help so much) [14:14] Is there a better place than a bug report to file feature requests / design ideas? [14:14] Two ideas in my head right now... [14:15] Advanced search link (from inside an existing advanced search) should keep the current search terms selected [14:15] facet based search [14:15] strongly desired, very sure its already a bug [14:15] It would be nice to do a search, then work through all those bugs - with an added "Next" "Previous" (whatever) nagivation overlayered on the bug page [14:16] So one could find all bugs with no tags (for example), then click through them one by one tagging them [14:16] without having to go back to the original search [14:16] a bit like gmail? :) I don't recall seeing a bug for that, but that could be nice [14:16] I've not used gmail in ages, so couldn't comment [14:16] I was just thinking ... "what am I trying to do.." and "how could it be easier?" [14:17] Rather than features for the sake of features [14:17] pcjc2: Are you thinking about multiple bug editing here? [14:17] That's something we've wanted for donkey's ages. [14:17] yes and no.. edit one at once, but page through them [14:17] Ah, I see. [14:17] bonus points for letting a project save particular search queries they work with often [14:18] find all bugs tagged "fruit" [14:18] That would be nice. I can definitely see myself using that. [14:18] I don't know how it would be done technically [14:19] Perhaps one could add "next bug" and "previous bug" links to every bug page [14:19] and then allow a concept of an active filter / search which reduces your view of bugs [14:20] would need to think of a UI to specify / show the collection context [14:20] IE.. am I looking at the "next bug" assigned to me? [14:20] in the project? [14:20] In the search results? [14:25] mrevell: How do you want the results ordered? By the number of specifications for each project, or the project name? [14:25] pcjc2: If you could look at a bug in the context of a specific group of bugs it could make sense. e.g.: [14:26] bugs.lp.n/~gmb/launchpad/+assignedbugs/12345 [14:26] Though the URL construction logic could be a bit torturous. [14:27] (I think that's how Gmail does it, but I'm not sure). [14:27] lots of possibilities once you start wanting to do it for arbitrary searches [14:27] I think we should support saving searches as a resource [14:27] either against a project, or person / team [14:27] pcjc2: It's on our to-do list. [14:27] Or at least our want-to-do list [14:28] kiko called it "Bug Bags" in 2008, I remember that much. [14:28] Mainly because he told me roughly what that meant and I had to explain it to a lot of people. [14:29] Actually, I think Bug Bags were arbitrary groupings of bugs to which you could add new ones if you wished. A bit like tags but personal to the user. [14:29] I think they were ordered, too. [14:29] But I don't think we heard anything about them after UDS Jaunty... [14:30] Also - is it me, or is LP missing a "related bugs" feature [14:30] where something is not quite a duplicate.. but may be relevant [14:30] wgrant: They're a kiko feature: Kind of ephemeral until lifeless decides to do it. [14:30] Heh. [14:30] pcjc2: YES. [14:30] Project db-devel build (269): STILL FAILING in 4 hr 26 min: https://hudson.wedontsleep.org/job/db-devel/269/ [14:30] A thousand times yes. [14:30] Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 12197 [14:30] included. [14:30] Rargh [14:31] And possibly "blocks bug" / "blocked by bug" relationship between 2x or more bugs [14:31] allenap, I don't think I mind. [14:31] allenap, number of specs I s'ppose [14:32] quote 'the "bug bag" functionality that will come with LP 3.0' [14:32] From https://wiki.edubuntu.org/UDSJaunty/Report/QA [14:33] pcjc2: Bug relationships are my number one thing-to-do. [14:33] awesome - then I won't feel the need to file a feature request about that === al-maisan is now known as almaisan-away [14:34] I'm considering the "bug bag" / bug collection idea though.. (currently looking for docs on the old bug bag idea) [14:34] I really hope that that feature will get assigned to my new squad when its time comes. [14:34] what is your squad? [14:34] mrevell, allenap: https://pastebin.canonical.com/41512/ is the top 50 project blueprint totals, if that's what you're after. mpt asked for it a couple of weeks back. [14:34] (Flacoste, lifeless: Take note, I will cry like a man-baby if we don't get that one). [14:35] pcjc2: The yellow squad. I'm told that we're not going to be the yellow squad for very long, though :) [14:35] (Or is it just - when the dev cycle rolls around to your turn for feature development in that area again?) [14:35] pcjc2: Well, we're reorganising, so that's the general plan. [14:36] Basically, you have N squads, N-2 of which are assigned to feature work. The other 2 deal with bugs on a day-to-day basis. [14:36] I read the posts about it, sounds like a good plan [14:36] Once a feature's complete, the feature squad becomes a Triage / bugfix squad and one of the existing squads becomes a feature squad. [14:36] pcjc2: The new structure takes effect next week when we sprint in Dallas. [14:36] wgrant: I'm getting some more fiddly stats than that, but thanks anyway :) [14:37] Ah, OK. [14:37] wgrant: go. to. sleep. [14:37] Not to be ungrateful, but search ought to be more AJAXy [14:37] Project devel build (361): STILL FAILING in 4 hr 25 min: https://hudson.wedontsleep.org/job/devel/361/ [14:37] * Launchpad Patch Queue Manager: [r=lifeless][ui=none][bug=701947][no-qa] Cull a load of unnecessary [14:37] .count()s in favour of less-expensive SQL. [14:37] * Launchpad Patch Queue Manager: [r=thumper][ui=sinzui][bug=702694] Tweak the text on the [14:37] +daily-builds page. [14:37] * Launchpad Patch Queue Manager: [r=lifeless][ui=none][bug=680733] Fix the recipe view's builds so [14:37] broken builds don't get stuck at the top of the list. [14:37] pcjc2: Again, a thousand times yes. [14:37] hmm, I need to people with PPAs to qa bug 697685 [14:37] <_mup_> Bug #697685: People with PPAs should not be allowed to merge accounts

< https://launchpad.net/bugs/697685 > [14:38] bigjools: I was just resending an ec2 instance that failed because of a conflict [14:38] ! [14:38] is that the conflict I warned you about? :) [14:38] Yes. [14:38] Shh. [14:38] gmb: Does this get you something: http://onecall.farnell.com/jsp/search/browse.jsp?N=411+500003+1000224&Ntk=gensearch_001&Ntt=100uF+25V&Ntx=mode+matchallpartial [14:38] gmb: noted [14:38] Might be tied to my session though [14:38] gmb: why won't you the yellow ones for long? [14:38] flacoste: I think there's a general sense that we want to be the Gold Squad ;) [14:39] (Or something) [14:39] I jest. [14:39] lol [14:39] yellow gold [14:39] not to be confused with yyykon gold [14:39] yukon gold [14:39] I need something better than Red, I shall be asking my team next week [14:39] bigjools: as you as you are not the Reds [14:40] I cannot possibly be associated with red :) [14:40] flacoste: the google spreadsheet says Red [14:40] gmb: That is a nice example of ajax search (even if the results are manually updated). The only missing feature is the ability to exclude based upon a given field [14:40] IE. "Not 25V DC" === matsubara is now known as matsubara-lunch [14:42] Combine that search functionality, the ability to negate the filters.. and a means to save a given query.. we would have awesome [14:42] bigjools: Come on, red is good :) [14:42] pcjc2: That's a reasonable example. I think we'd want to aim for a slightly slicker UI, but it would be a good first step. [14:42] allenap: I don't want to be Red Ed [14:42] Hahaha [14:42] It is usable - I agree you want slicker [14:42] * StevenK keep that in mind for next week [14:43] bigjools: Glitter then. [14:43] allenap: lol [14:43] that would be gary's team.... [14:43] OI. [14:43] Of course! [14:43] :-) [14:43] Silly ideas in abundance here.. what about reverse AJAX for pushing updates to bugs whilst one has the page open? [14:44] Often I find I'm working on bugs with other developers, and we have to keep hitting refresh [14:44] bigjools: any idea how to qa http://launchpad.net/bugs/680733 ? [14:44] <_mup_> Bug #680733: broken recipe build gets stuck at top of "5 latest builds" list on recipe page, forever

< https://launchpad.net/bugs/680733 > [14:44] If the Launchpad squards are going to named like that, there will be a beating next week [14:44] I was thinking Gold Lame, per gmb. Or maybe Yellow Bellies (Americanism for cowards). [14:44] lifeless: checking [14:45] StevenK: maybe something that alliterates with the squad leader's name then [14:45] Gary's Gang, for example [14:45] heh [14:46] * gary_poster tries to recall the names of the two gangs in West Side Story... [14:46] Jets and... [14:46] ah, yes, Sharks [14:47] lifeless: don't worry about bug 697685. i just qa'ed. thanks for getting my other bug, btw. :-) [14:47] <_mup_> Bug #697685: People with PPAs should not be allowed to merge accounts

< https://launchpad.net/bugs/697685 > [14:47] jcsackett: sweet, thanks [14:47] lifeless: no... :/ [14:49] lifeless: the URL is invalid on qastaging... ummm [14:49] anyone know of a team with recipes on qastaging ? [14:50] https://code.qastaging.launchpad.net/~bzr/ is repeatedly timing out [14:51] yeah, its high on the ppr but not a timeout on prod atm [14:52] well we're clear to 12201 [14:52] be nice to get 12201 and 12202 in the deploy [15:17] bac: did you start the wiki page for the review discussion of next week? [15:27]