[00:00] wgrant: just merged devel 12533, and pushed. No conflicts, fwiw. [00:00] now, time for me to go sleep [00:00] hopefully benji will comment on the proposal soon, so we can get it landed before downtime. [00:00] but, probably not today :) [00:02] jam: is this the email I replied to about 12 hours ago? [00:02] benji: right. I guess you need to put your Release Manager stamp on the merge proposal itself [00:02] at least, that is what I gleaned from wgrant's comment [00:03] I can do t [00:03] hat [00:03] (If someone can tell me how.) [00:03] don't look at me :) [00:03] heh [00:04] jam: which MP is it? [00:05] https://code.launchpad.net/~jameinel/launchpad/lp-serve-child-hangup/+merge/51893 [00:05] is the lp proposal [00:05] and https://code.launchpad.net/~jameinel/lp-production-configs/enable-forking/+merge/52297 [00:05] is the config change [00:06] ok, let me see if there's something obvious I can do to them [00:06] benji: Review with type 'rc', or 'release-critical'. I forget which. [00:06] * wgrant checks. [00:06] cool, that's along the lines of what I was thinking [00:06] jam: I'd already done that locally, so it's already in ec2. [00:06] I'll lp-land it once it succeeds. [00:07] lib/devscripts/autoland.py: if 'release-critical' in review_type: [00:09] cool, release-critical it is [00:09] Thanks. [00:09] done [00:10] I'm going away now, but much like he-who-must-not-be-named, if you mention my name I'll find you whereever you are. [00:10] Heh. [00:12] jam: I won't land the prod configs change until we have confirmed LOSA availability for the rest of the day of the deployment. [00:12] Just in case. [00:13] Since we had no spm last week. [00:13] wgrant: sounds reasonable [04:27] Project devel build #508: FAILURE in 15 min: https://hudson.wedontsleep.org/job/devel/508/ [04:41] bzrlib.errors.InvalidHttpResponse: Invalid http response for https://xmlrpc.launchpad.net/bazaar/: Unable to handle http code 502: Bad Gateway [04:41] That's not good. [04:48] Whee [04:49] Slave deleted [04:49] It wasn't a slave fault. [04:49] But OK. [04:50] I know, just want to certain it doesn't affect a later build [04:50] Ah, true. [04:50] I can check out lp:difftacular locally [04:52] Ah, it was a change. Let me schedule a build. [06:23] ola [06:27] Hi lifeless. [06:27] epic lounge time [06:28] SYD? [06:28] yeah [06:29] hmmm [06:30] I think tomorrow I'll add the new appserver instances, patch oops-tools to not truncate the lists of incidents and iterate on my bug 727560 [06:30] <_mup_> Bug #727560: Archive:EntryResource:getPublishedSources < https://launchpad.net/bugs/727560 > [06:30] List of incidents? You mean list of exceptions? [06:31] wgrant: hows our in dc latency going ? [06:31] wgrant: yeah, the 'X of Y' in the headings. [06:31] oh, and with the report not truncating, drop the soyuz specific report. [06:31] lifeless: Up to 100ms. [06:32] lifeless: Yeah, I think we can probably do that now. [06:32] mmm, still tolerable. [06:32] The main report will become a lot shorter this week. [06:32] not not brilliant [06:32] wgrant: it doesn't really matter how long it is. [06:33] lifeless: Bug #730011 [06:33] <_mup_> Bug #730011: ~registry should be able to unsuspend accounts < https://launchpad.net/bugs/730011 > [06:33] Can you see any reason not to allow ~registry to traverse to suspended accounts? [06:33] possibly [06:34] it would depend on the style of suspension [06:34] resurrect a 'banned cause a bot was spamming via your account' -> fine [06:34] suspended cause they were a dirty dirty hacker -> fine IFF we make the cause /realllllly clear/ [06:35] Right. [06:35] suspended if they used legal action to get all details removed and DPA stuff ... no [06:35] or at least, I'd want legal to comment on the third [06:35] I think if you get charlie and elmo to agree, that its pragmatically fine. [06:36] remembering that non-canonical are in ~regstry too [06:36] we may have 0 of the third case and not handle them in the normal way anyhow [06:36] which would make the concern irrelevant [06:39] * lifeless just fixed 7 oops a day [06:39] Oh? [06:39] https://wiki.ubuntu.com/UbuntuMediaCenter?action=diff&rev2=50&rev1=49 [06:40] Hah. [06:40] They're not real OOPSes. [06:40] well [06:41] they are within the set of 'canonical' web properties that its useful to report on bad links [06:41] so I don't feel comfortable deleting them [06:41] True [06:42] and another 9 [06:42] 10 NotFound: Object: , name: u'latest-bugs.atom ' [06:42] GET: 10 Robots: 0 Local: 10 Most Common Referrer: http://feeds.launchpad.net/lilregcleaner/latest-bugs.atom%20 [06:43] 10 http://feeds.launchpad.net/lilregcleaner/latest-bugs.atom%20 (Unknown) [06:43] I wonder if thats real, or bad data *within* lp [06:43] There are a few other odd ones like that. [06:43] which will be a pita to correct because its a users data [06:43] erm [06:43] wtf [06:43] http://answers.launchpad.net/index.php [06:43] GET: 12 Robots: 0 Local: 12 Most Common Referrer: http://answers.launchpad.net/index.php [06:43] eg. '%20source' instead of '+source' sometimes, as a referer. [06:44] Which is clearly fake. [06:44] But why. [06:44] hack attempts ? [06:44] Presumably. [06:44] But why. [06:44] because we're running php ... [06:45] Oh, the index.php one is clearly a hack attempt, yes. [06:45] But the other s/+/%20/ probably aren';t. [06:45] you're talking https://launchpad.net/ubuntu-russian-guide/lucid-guide/lucid-guide-1.0/%20download/ ? [06:46] ok, flight time [06:46] ciao [10:11] Yippie, build fixed! [10:11] Project devel build #509: FIXED in 5 hr 7 min: https://hudson.wedontsleep.org/job/devel/509/ [11:22] bah, frozen [11:22] we'll see how this db deploy goes, I think we're probably ready to move freeze to monday morning [11:23] wgrant: lp:~lifeless/launchpad/bug-727560 has had all the tests that failed in my ec2 run fixed [11:23] lifeless: We almost ended up having to delay it to then anyway. [11:24] because of the conflict? [11:24] Yes. [11:24] And we were running out of mbarnett. [11:24] say its not so [11:25] I think we should freeze on Monday morning and possibly aim to unfreeze by the next morning. [11:25] 5 hours should be all we nned [11:25] direct landing to devel [11:26] one buildbot [11:26] deploy to qas. [11:26] test [11:26] wgrant: I think we should freeze closer, too - but have been taking incremental steps [11:26] less risk [11:26] Sure. [11:29] I think the future will become more clear after a couple more normal releases. [11:29] 11.02 was rather anomalous. [11:30] we need to be able to handle that too though [11:30] Do we? [11:31] there will always be exceptions and surprises [11:31] we need to have a way to cope [11:31] There will. [11:31] But they can be handled by reversion, if we make the window too narrow. [11:31] be that 'don't merge that patch' or 'don't do a deploy' or whatever. [11:32] well, there's no guarantee that reverting some patch P out of a sequence N >= P items long will be itself regression free [11:32] I'm not trying to say /how/ we should handle it [11:32] but we need to think about how to handle it [11:32] Right. [11:32] avoiding the issue is one way, but tends to be rigid and expensive most of the time [11:33] having a fallback plan is a different way [11:33] like a second deploy date booked 2 days later [11:33] a third is much more regular db deploys [11:33] Lots of patches are short and suitable for regular DB deploys. [11:34] 11.02 was not. [11:34] the problem isn't (usually) the patch size [11:34] its doing the deploy itself. [11:34] readonly mode is /expensive/ to invoke [11:34] we can't do readonly mode more than once every 3 days [11:34] not reliably [11:34] Why not? [11:34] The rebuild? [11:35] because after doing readonly mode the readonly replica has to be rebuilt [11:35] True. [11:35] 11.02 was fine patch size [11:35] Although that only take 24hourish. [11:35] the problem there was stale locks on the sso replica [11:35] Estimates said that the patches would almost exhaust the window. [11:35] estimate was on crack [11:36] They were in fact much shorter, but that's not what we thought before hand. [11:36] no, they weren't. [11:36] the estimate was inflated by calculating slony slaves as linear sums [11:36] Hence they were shorter than the estimates. [11:36] but ddl propogates to the slaves same as normal commits - in paralle. [11:36] well, depends on your definition. [11:37] if you run the right formula on the original patch time, the predicted time was right no [11:37] *on8 [11:37] Sure. [11:37] But the original patch time is not the interesting one. [11:38] what is? [11:38] The final value that tells us how long the window needs to be. [11:41] I don't understand why you prefaced your prior assertion with 'but'. [11:41] given I too was talking about the predicted time. [11:42] I think we're both pretty clear on the constraints [11:47] http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/ looks nice [11:57] gnight [11:59] Night. === lionel__ is now known as lionel === almaisan-away is now known as al-maisan === al-maisan is now known as almaisan-away === elmo_ is now known as elmo === almaisan-away is now known as al-maisan === al-maisan is now known as almaisan-away [21:03] lifeless: are you able to +1 on this mp now that i've exported to the correct version as requested: https://code.launchpad.net/~wallyworld/launchpad/mp-related-bugtasks-webservice/+merge/52004 [21:13] wallyworld: actually I'm thinking we probably shouldn't export it at all [21:13] wallyworld: its entirely redundant with a bug search for branch=X [21:13] wallyworld: I dunno [21:14] lifeless: i have no real opinion on it. it just seemed like something that people would want to do. === almaisan-away is now known as al-maisan === al-maisan is now known as almaisan-away [22:00] wgrant: Have you seen the OOPS report today? [22:01] StevenK: do you mean sundays? tordays is 3 hours away [22:01] Likely [22:07] StevenK: You mean the QISEs that started on Friday? [22:07] wgrant: Indeed. [22:07] Basically, delayed copies suck. [22:08] And are completely unnecessary. [22:08] Particularly with the branch I am about to propose. [22:09] wgrant: Oh, do you have a nice suggestion of how I can take a .dsc in the tree and import it into the librarian and create publishing history for it? [22:09] If we have an spm today we can reject those two PUs. [22:09] StevenK: Why? [22:09] Testing the changelog populator? [22:09] Right [22:10] StevenK: I'd to it manually. [22:10] makeSourcePackagePublishingHistory probably doesn't create any SPRfs. [22:10] So you can add them yourse.f [22:10] And I cannot type. [22:10] Haha [22:10] More caffeine? Less caffeine? [22:21] wgrant: fyi (cause I know you're interested) - https://code.launchpad.net/~lifeless/lp-production-configs/pending/+merge/52345 [22:22] how does one get a bug notification level of nothing ? [22:23] lifeless: I think you need the advanced subscriptions UI :( [22:24] That's why I haven't already done that [22:25] ditto tims patch I guess ? [22:25] wgrant: you could give yourself that on dogfood, for the former [22:26] Ah, true, Julian's not around. [22:26] wgrant: him being around would interfere? [22:26] He would go on a rant about how it's only for Soyuz testing :P [22:26] its a team qa resource [22:30] wgrant: so, want to file a new bug via email with 32K of text or whatever? [I still haven't setup dkim on my domain yet] [22:30] lifeless: My DKIM isn't trusted, but I can gpg-sign it... [22:30] wgrant: yeah [22:31] then we can get spm to run process-mail [22:31] and we should get a reject in the staging mailbox [22:31] I think it runs regularly, actually. [22:31] Judging by the number of OOPSes it generates. [22:31] On staging, at least. [22:31] I will spam both. [22:38] Hm [22:39] You'd think we'd have a helper in the test suite for "Here, punt this file into the Librarian" [22:39] StevenK: Why do we need a helper? [22:39] The API is not that terrible :) [22:40] Clearly, I'm failing to read Python this morning, and require tea. === almaisan-away is now known as al-maisan [22:41] lifeless: I'm currently making Bugs usable on mawson again with the same hack we used on qas. [22:41] But it's taking a while. [22:41] lifeless: can I get you to mentor two reviews by wallyworld? [22:41] wgrant: what was that ? [22:41] thumper: sure [22:41] lifeless: https://code.edge.launchpad.net/~thumper/launchpad/mail-header-oops/+merge/52156 and https://code.edge.launchpad.net/~thumper/launchpad/strip-email-attachment-path/+merge/52159 [22:41] damn edge [22:42] I just won't leave my browser memory [22:42] s/I/It/ [22:42] UPDATE bugmessage SET index=id WHERE index IS NULL; [22:42] wgrant: we ran the indexer on qastaging :) [22:43] lifeless: There were 3000 unindexed messages there when we tried to upgrade it on Saturday. [22:43] Production is still fine. [22:43] So I suspect the indexer stopped too early. [22:43] >< [22:43] ok === al-maisan is now known as almaisan-away [23:18] Error message: [23:18] The description is too long. If you have lots of text to add, use an [23:18] attachment instead. [23:18] Excellent. [23:32] woo [23:32] qa-ok ? [23:32] Just waiting for a successful one to go through. [23:33] http://webnumbr.com/launchpad-oops-bugs.all [23:34] Where's spiv with his perspective graph? I liked that one better. [23:35] lifeless: qa-ok [23:36] wgrant: And you also did r12527? [23:37] StevenK: i've just given up waiting for mawson. [23:37] Going to live with the breakage. [23:39] lifeless: thanks for the review mentors. [23:39] lifeless: got any quick pointers to fake librarian usage? [23:39] also... [23:39] anyone remember the magic commands to add a loopback delay (and to clear it again) ? [23:40] Rargh, LP's icon is broken still [23:40] StevenK: where? [23:41] lifeless: On any page -- last time, wgrant and I nailed it down to the librarian returning an incorrect content-type [23:41] thumper: uhm, just grep for FakeLibrarian, its pretty self explanatory [23:41] StevenK: ah, yes [23:41] https://bugs.launchpad.net/launchpad/+bug/643224 is where I noticed it [23:41] <_mup_> Bug #643224: dkim whitelist should be in configuration

< https://launchpad.net/bugs/643224 > [23:41] ok [23:41] we should probably critical this [23:41] And does the font used have to change every day? [23:41] It's getting annoying. [23:42] At the moment, yes. [23:42] lifeless: Do eet [23:44] oh man [23:44] this is going to be fugly. [23:44] Bug heat, please die. === almaisan-away is now known as al-maisan [23:46] lifeless: We are deployable and releasable, yet disappointingly LOSA- and RM-less. [23:47] Well, it is Sunday for BjornT [23:47] DOH [23:47] * StevenK glares at irssi [23:48] I'm thinking its time to ditch rm in a month or two [23:48] I'd say so. [23:50] And do what instead? [23:50] StevenK: there's approximately zero to do. [23:50] I disagree -- given the fun wallyworld had last rollout, for example [23:51] StevenK: the only nontrivial thing remaining is blessing additional patches during freeze, and thats /only/ needed for fixes to DB patches which were incorrectly qa-ok in db-stable, or which break when combined with the tip of devel. [23:51] Last rollout is not what I meant, but you see what I mean [23:51] StevenK: that had no need to be centralised [23:51] there needs to be a point of contact that everyone knows about - as someone said to me, it's like herding cats === al-maisan is now known as almaisan-away [23:52] wallyworld: there does? [23:52] i think so [23:52] why? [23:52] if/when something goes wrong, there needs to be someone (the rm) accountable for liasing with whoever is needed to get it sorted out [23:53] also to chase up people for qa etc [23:53] to nominate what db-devel rev no will be used [23:53] point by point [23:53] etc [23:53] losas are doing the deploy, they are the only sane contact point to initiate any fubar [23:53] qa chaseup is not needed - if its not qaed, its not in the deploy. [23:54] the nomination of revisions is trivial, the report plus a quick eyeball over it can be done by anyone. [23:55] * thumper scratches his head [23:55] re qa: but what if there's a rev holding stuff up and the person just needs reminding or even the qa could just be done by someone else [23:55] wallyworld: And yet you complain when wgrant does your QA? :-) [23:55] yes but "done by anyone" usually means "done by no-one" unless "someone" is accountable [23:56] StevenK: only because there's certain things i want to test :-) [23:56] wallyworld: no, this is a false meme [23:56] its possible to have a culture where team accountability matters [23:56] StevenK: but close to a release, when stuff is being held up by a qa delay, you just want to get it done [23:56] And I'm sure everyone is aware of the deployment report. [23:56] in this specific case - [23:56] : if its not qaed, the pipeline stalls but the earlier stuff still gets released. [23:57] : if its not qaed and there is a revision after it that someone else wants included in the deploy, they have motivation to qa the earlier revision. [23:57] lifeless: Er, but that doesn't work? [23:57] Ah [23:57] StevenK: by doesn't, you mean 'does'. [23:58] : if its not qaed, and there is nothing after it and the person that landed it doesn't care and noone else cares - then clearly the change isn't important enough to worry about us not having it in the deploy. [23:58] lifeless: I was going to point out that we by definition want to deploy tip of stable since we merge db-devel into it [23:58] StevenK: That was a matter of some contention last time. [23:58] but i maintain that qa is often best done by the person who wrote the code since they know the specific in's and out's and corner cases etc that need to be tested [23:58] StevenK: we only merge teh qa-ok'd revisions of db-stable, we *do not* merge db-devel. [23:58] wallyworld: then again, all the other devs have motivation to talk to them. [23:58] wallyworld: I agree, but sometimes the pipeline is blocked and minimal QA is better than blockage. [23:59] wallyworld: and if the change looks hard to qa, others can roll it out. [23:59] *back it out* [23:59] there are lots of options that don't depend on a single person nagging