jamwgrant: just merged devel 12533, and pushed. No conflicts, fwiw.00:00
jamnow, time for me to go sleep00:00
jamhopefully benji will comment on the proposal soon, so we can get it landed before downtime.00:00
jambut, probably not today :)00:00
benjijam: is this the email I replied to about 12 hours ago?00:02
jambenji: right. I guess you need to put your Release Manager stamp on the merge proposal itself00:02
jamat least, that is what I gleaned from wgrant's comment00:02
benjiI can do t00:03
benji(If someone can tell me how.)00:03
jamdon't look at me :)00:03
benjijam: which MP is it?00:04
jamis the lp proposal00:05
jamand https://code.launchpad.net/~jameinel/lp-production-configs/enable-forking/+merge/5229700:05
jamis the config change00:05
benjiok, let me see if there's something obvious I can do to them00:06
wgrantbenji: Review with type 'rc', or 'release-critical'. I forget which.00:06
* wgrant checks.00:06
benjicool, that's along the lines of what I was thinking00:06
wgrantjam: I'd already done that locally, so it's already in ec2.00:06
wgrantI'll lp-land it once it succeeds.00:06
wgrantlib/devscripts/autoland.py:        if 'release-critical' in review_type:00:07
benjicool, release-critical it is00:09
benjiI'm going away now, but much like he-who-must-not-be-named, if you mention my name I'll find you whereever you are.00:10
wgrantjam: I won't land the prod configs change until we have confirmed LOSA availability for the rest of the day of the deployment.00:12
wgrantJust in case.00:12
wgrantSince we had no spm last week.00:13
jamwgrant: sounds reasonable00:13
LPCIBotProject devel build #508: FAILURE in 15 min: https://hudson.wedontsleep.org/job/devel/508/04:27
wgrantbzrlib.errors.InvalidHttpResponse: Invalid http response for https://xmlrpc.launchpad.net/bazaar/: Unable to handle http code 502: Bad Gateway04:41
wgrantThat's not good.04:41
StevenKSlave deleted04:49
wgrantIt wasn't a slave fault.04:49
wgrantBut OK.04:49
StevenKI know, just want to certain it doesn't affect a later build04:50
wgrantAh, true.04:50
StevenKI can check out lp:difftacular locally04:50
StevenKAh, it was a change. Let me schedule a build.04:52
wgrantHi lifeless.06:27
lifelessepic lounge time06:27
lifelessI think tomorrow I'll add the new appserver instances, patch oops-tools to not truncate the lists of incidents and iterate on my bug 72756006:30
_mup_Bug #727560: Archive:EntryResource:getPublishedSources <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/727560 >06:30
wgrantList of incidents? You mean list of exceptions?06:30
lifelesswgrant: hows our in dc latency going ?06:31
lifelesswgrant: yeah, the 'X of Y' in the headings.06:31
lifelessoh, and with the report not truncating, drop the soyuz specific report.06:31
wgrantlifeless: Up to 100ms.06:31
wgrantlifeless: Yeah, I think we can probably do that now.06:32
lifelessmmm, still tolerable.06:32
wgrantThe main report will become a lot shorter this week.06:32
lifelessnot not brilliant06:32
lifelesswgrant: it doesn't really matter how long it is.06:32
wgrantlifeless: Bug #73001106:33
_mup_Bug #730011: ~registry should be able to unsuspend accounts <chr> <Launchpad itself:Triaged> < https://launchpad.net/bugs/730011 >06:33
wgrantCan you see any reason not to allow ~registry to traverse to suspended accounts?06:33
lifelessit would depend on the style of suspension06:34
lifelessresurrect a 'banned cause a bot was spamming via your account' -> fine06:34
lifelesssuspended cause they were a dirty dirty hacker -> fine IFF we make the cause /realllllly clear/06:34
lifelesssuspended if they used legal action to get all details removed and DPA stuff ... no06:35
lifelessor at least, I'd want legal to comment on the third06:35
lifelessI think if you get charlie and elmo to agree, that its pragmatically fine.06:35
lifelessremembering that non-canonical are in ~regstry too06:36
lifelesswe may have 0 of the third case and not handle them in the normal way anyhow06:36
lifelesswhich would make the concern irrelevant06:36
* lifeless just fixed 7 oops a day06:39
wgrantThey're not real OOPSes.06:40
lifelessthey are within the set of 'canonical' web properties that its useful to report on bad links06:41
lifelessso I don't feel comfortable deleting them06:41
lifelessand another 906:42
lifeless 10 NotFound: Object: <Product at INSTANCE-ID>, name: u'latest-bugs.atom '06:42
lifeless   GET: 10 Robots: 0  Local: 10 Most Common Referrer: http://feeds.launchpad.net/lilregcleaner/latest-bugs.atom%2006:42
lifeless     10 http://feeds.launchpad.net/lilregcleaner/latest-bugs.atom%20 (Unknown)06:43
lifelessI wonder if thats real, or bad data *within* lp06:43
wgrantThere are a few other odd ones like that.06:43
lifelesswhich will be a pita to correct because its a users data06:43
lifelessGET: 12 Robots: 0  Local: 12 Most Common Referrer: http://answers.launchpad.net/index.php06:43
wgranteg. '%20source' instead of '+source' sometimes, as a referer.06:43
wgrantWhich is clearly fake.06:44
wgrantBut why.06:44
lifelesshack attempts ?06:44
wgrantBut why.06:44
lifelessbecause we're running php ...06:44
wgrantOh, the index.php one is clearly a hack attempt, yes.06:45
wgrantBut the other s/+/%20/ probably aren';t.06:45
lifelessyou're talking https://launchpad.net/ubuntu-russian-guide/lucid-guide/lucid-guide-1.0/%20download/ ?06:45
lifelessok, flight time06:46
LPCIBotYippie, build fixed!10:11
LPCIBotProject devel build #509: FIXED in 5 hr 7 min: https://hudson.wedontsleep.org/job/devel/509/10:11
lifelessbah, frozen11:22
lifelesswe'll see how this db deploy goes, I think we're probably ready to move freeze to monday morning11:22
lifelesswgrant: lp:~lifeless/launchpad/bug-727560 has had all the tests that failed in my ec2 run fixed11:23
wgrantlifeless: We almost ended up having to delay it to then anyway.11:23
lifelessbecause of the conflict?11:24
wgrantAnd we were running out of mbarnett.11:24
lifelesssay its not so11:24
wgrantI think we should freeze on Monday morning and possibly aim to unfreeze by the next morning.11:25
lifeless5 hours should be all we nned11:25
lifelessdirect landing to devel11:25
lifelessone buildbot11:26
lifelessdeploy to qas.11:26
lifelesswgrant: I think we should freeze closer, too - but have been taking incremental steps11:26
lifelessless risk11:26
wgrantI think the future will become more clear after a couple more normal releases.11:29
wgrant11.02 was rather anomalous.11:29
lifelesswe need to be able to handle that too though11:30
wgrantDo we?11:30
lifelessthere will always be exceptions and surprises11:31
lifelesswe need to have a way to cope11:31
wgrantThere will.11:31
wgrantBut they can be handled by reversion, if we make the window too narrow.11:31
lifelessbe that 'don't merge that patch' or 'don't do a deploy' or whatever.11:31
lifelesswell, there's no guarantee that reverting some patch P out of a sequence N >= P items long will be itself regression free11:32
lifelessI'm not trying to say /how/ we should handle it11:32
lifelessbut we need to think about how to handle it11:32
lifelessavoiding the issue is one way, but tends to be rigid and expensive most of the time11:32
lifelesshaving a fallback plan is a different way11:33
lifelesslike a second deploy date booked 2 days later11:33
lifelessa third is much more regular db deploys11:33
wgrantLots of patches are short and suitable for regular DB deploys.11:33
wgrant11.02 was not.11:34
lifelessthe problem isn't (usually) the patch size11:34
lifelessits doing the deploy itself.11:34
lifelessreadonly mode is /expensive/ to invoke11:34
lifelesswe can't do readonly mode more than once every 3 days11:34
lifelessnot reliably11:34
wgrantWhy not?11:34
wgrantThe rebuild?11:34
lifelessbecause after doing readonly mode the readonly replica has to be rebuilt11:35
lifeless11.02 was fine patch size11:35
wgrantAlthough that only take 24hourish.11:35
lifelessthe problem there was stale locks on the sso replica11:35
wgrantEstimates said that the patches would almost exhaust the window.11:35
lifelessestimate was on crack11:35
wgrantThey were in fact much shorter, but that's not what we thought before hand.11:36
lifelessno, they weren't.11:36
lifelessthe estimate was inflated by calculating slony slaves as linear sums11:36
wgrantHence they were shorter than the estimates.11:36
lifelessbut ddl propogates to the slaves same as normal commits - in paralle.11:36
lifelesswell, depends on your definition.11:36
lifelessif you run the right formula on the original patch time, the predicted time was right no11:37
wgrantBut the original patch time is not the interesting one.11:37
lifelesswhat is?11:38
wgrantThe final value that tells us how long the window needs to be.11:38
lifelessI don't understand why you prefaced your prior assertion with 'but'.11:41
lifelessgiven I too was talking about the predicted time.11:41
lifelessI think we're both pretty clear on the constraints11:42
lifelesshttp://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/ looks nice11:47
=== lionel__ is now known as lionel
=== almaisan-away is now known as al-maisan
=== al-maisan is now known as almaisan-away
=== elmo_ is now known as elmo
=== almaisan-away is now known as al-maisan
=== al-maisan is now known as almaisan-away
wallyworldlifeless: are you able to +1 on this mp now that i've exported to the correct version as requested: https://code.launchpad.net/~wallyworld/launchpad/mp-related-bugtasks-webservice/+merge/5200421:03
lifelesswallyworld: actually I'm thinking we probably shouldn't export it at all21:13
lifelesswallyworld: its entirely redundant with a bug search for branch=X21:13
lifelesswallyworld: I dunno21:13
wallyworldlifeless: i have no real opinion on it. it just seemed like something that people would want to do.21:14
=== almaisan-away is now known as al-maisan
=== al-maisan is now known as almaisan-away
StevenKwgrant: Have you seen the OOPS report today?22:00
lifelessStevenK: do you mean sundays? tordays is 3 hours away22:01
wgrantStevenK: You mean the QISEs that started on Friday?22:07
StevenKwgrant: Indeed.22:07
wgrantBasically, delayed copies suck.22:07
wgrantAnd are completely unnecessary.22:08
wgrantParticularly with the branch I am about to propose.22:08
StevenKwgrant: Oh, do you have a nice suggestion of how I can take a .dsc in the tree and import it into the librarian and create publishing history for it?22:09
wgrantIf we have an spm today we can reject those two PUs.22:09
wgrantStevenK: Why?22:09
wgrantTesting the changelog populator?22:09
wgrantStevenK: I'd to it manually.22:10
wgrantmakeSourcePackagePublishingHistory probably doesn't create any SPRfs.22:10
wgrantSo you can add them yourse.f22:10
wgrantAnd I cannot type.22:10
StevenKMore caffeine? Less caffeine?22:10
lifelesswgrant: fyi (cause I know you're interested) - https://code.launchpad.net/~lifeless/lp-production-configs/pending/+merge/5234522:21
lifelesshow does one get a bug notification level of nothing ?22:22
wgrantlifeless: I think you need the advanced subscriptions UI :(22:23
wgrantThat's why I haven't already done that22:24
lifelessditto tims patch I guess ?22:25
lifelesswgrant: you could give yourself that on dogfood, for the former22:25
wgrantAh, true, Julian's not around.22:26
lifelesswgrant: him being around would interfere?22:26
wgrantHe would go on a rant about how it's only for Soyuz testing :P22:26
lifelessits a team qa resource22:26
lifelesswgrant: so, want to file a new bug via email with 32K of text or whatever? [I still haven't setup dkim on my domain yet]22:30
wgrantlifeless: My DKIM isn't trusted, but I can gpg-sign it...22:30
lifelesswgrant: yeah22:30
lifelessthen we can get spm to run process-mail22:31
lifelessand we should get a reject in the staging mailbox22:31
wgrantI think it runs regularly, actually.22:31
wgrantJudging by the number of OOPSes it generates.22:31
wgrantOn staging, at least.22:31
wgrantI will spam both.22:31
StevenKYou'd think we'd have a helper in the test suite for "Here, punt this file into the Librarian"22:39
wgrantStevenK: Why do we need a helper?22:39
wgrantThe API is not that terrible :)22:39
StevenKClearly, I'm failing to read Python this morning, and require tea.22:40
=== almaisan-away is now known as al-maisan
wgrantlifeless: I'm currently making Bugs usable on mawson again with the same hack we used on qas.22:41
wgrantBut it's taking a while.22:41
thumperlifeless: can I get you to mentor two reviews by wallyworld?22:41
lifelesswgrant: what was that ?22:41
lifelessthumper: sure22:41
thumperlifeless: https://code.edge.launchpad.net/~thumper/launchpad/mail-header-oops/+merge/52156 and https://code.edge.launchpad.net/~thumper/launchpad/strip-email-attachment-path/+merge/5215922:41
thumperdamn edge22:41
thumperI just won't leave my browser memory22:42
wgrantUPDATE bugmessage SET index=id WHERE index IS NULL;22:42
lifelesswgrant: we ran the indexer on qastaging :)22:42
wgrantlifeless: There were 3000 unindexed messages there when we tried to upgrade it on Saturday.22:43
wgrantProduction is still fine.22:43
wgrantSo I suspect the indexer stopped too early.22:43
=== al-maisan is now known as almaisan-away
wgrantError message:23:18
wgrantThe description is too long. If you have lots of text to add, use an23:18
wgrantattachment instead.23:18
lifelessqa-ok ?23:32
wgrantJust waiting for a successful one to go through.23:32
StevenKWhere's spiv with his perspective graph? I liked that one better.23:34
wgrantlifeless: qa-ok23:35
StevenKwgrant: And you also did r12527?23:36
wgrantStevenK: i've just given up waiting for mawson.23:37
wgrantGoing to live with the breakage.23:37
thumperlifeless: thanks for the review mentors.23:39
thumperlifeless: got any quick pointers to fake librarian usage?23:39
thumperanyone remember the magic commands to add a loopback delay (and to clear it again) ?23:39
StevenKRargh, LP's icon is broken still23:40
lifelessStevenK: where?23:40
StevenKlifeless: On any page -- last time, wgrant and I nailed it down to the librarian returning an incorrect content-type23:41
lifelessthumper: uhm, just grep for FakeLibrarian, its pretty self explanatory23:41
lifelessStevenK: ah, yes23:41
StevenKhttps://bugs.launchpad.net/launchpad/+bug/643224 is where I noticed it23:41
_mup_Bug #643224: dkim whitelist should be in configuration <dkim> <feature-flags> <lp-foundations> <Launchpad itself:In Progress by mbp> < https://launchpad.net/bugs/643224 >23:41
lifelesswe should probably critical this23:41
StevenKAnd does the font used have to change every day?23:41
StevenKIt's getting annoying.23:41
wgrantAt the moment, yes.23:42
StevenKlifeless: Do eet23:42
lifelessoh man23:44
lifelessthis is going to be fugly.23:44
wgrantBug heat, please die.23:44
=== almaisan-away is now known as al-maisan
wgrantlifeless: We are deployable and releasable, yet disappointingly LOSA- and RM-less.23:46
StevenKWell, it is Sunday for BjornT23:47
* StevenK glares at irssi23:47
lifelessI'm thinking its time to ditch rm in a month or two23:48
wgrantI'd say so.23:48
StevenKAnd do what instead?23:50
lifelessStevenK: there's approximately zero to do.23:50
StevenKI disagree -- given the fun wallyworld had last rollout, for example23:50
lifelessStevenK: the only nontrivial thing remaining is blessing additional patches during freeze, and thats /only/ needed for fixes to DB patches which were incorrectly qa-ok in db-stable, or which break when combined with the tip of devel.23:51
StevenKLast rollout is not what I meant, but you see what I mean23:51
lifelessStevenK: that had no need to be centralised23:51
wallyworldthere needs to be a point of contact that everyone knows about - as someone said to me, it's like herding cats23:51
=== al-maisan is now known as almaisan-away
lifelesswallyworld: there does?23:52
wallyworldi think so23:52
wallyworldif/when something goes wrong, there needs to be someone (the rm) accountable for liasing with whoever is needed to get it sorted out23:52
wallyworldalso to chase up people for qa etc23:53
wallyworldto nominate what db-devel rev no will be used23:53
lifelesspoint by point23:53
lifelesslosas are doing the deploy, they are the only sane contact point to initiate any fubar23:53
lifelessqa chaseup is not needed - if its not qaed, its not in the deploy.23:53
lifelessthe nomination of revisions is trivial, the report plus a quick eyeball over it can be done by anyone.23:54
* thumper scratches his head23:55
wallyworldre qa: but what if there's a rev holding stuff up and the person just needs reminding or even the qa could just be done by someone else23:55
StevenKwallyworld: And yet you complain when wgrant does your QA? :-)23:55
wallyworldyes but "done by anyone" usually means "done by no-one" unless "someone" is accountable23:55
wallyworldStevenK: only because there's certain things i want to test :-)23:56
lifelesswallyworld: no, this is a false meme23:56
lifelessits possible to have a culture where team accountability matters23:56
wallyworldStevenK:  but close to a release, when stuff is being held up by a qa delay, you just want to get it done23:56
StevenKAnd I'm sure everyone is aware of the deployment report.23:56
lifelessin this specific case -23:56
lifeless : if its not qaed, the pipeline stalls but the earlier stuff still gets released.23:56
lifeless : if its not qaed and there is a revision after it that someone else wants included in the deploy, they have motivation to qa the earlier revision.23:57
StevenKlifeless: Er, but that doesn't work?23:57
lifelessStevenK: by doesn't, you mean 'does'.23:57
lifeless : if its not qaed, and there is nothing after it and the person that landed it doesn't care and noone else cares - then clearly the change isn't important enough to worry about us not having it in the deploy.23:58
StevenKlifeless: I was going to point out that we by definition want to deploy tip of stable since we merge db-devel into it23:58
wgrantStevenK: That was a matter of some contention last time.23:58
wallyworldbut i maintain that qa is often best done by the person who wrote the code since they know the specific in's and out's and corner cases etc that need to be tested23:58
lifelessStevenK: we only merge teh qa-ok'd revisions of db-stable, we *do not* merge db-devel.23:58
lifelesswallyworld: then again, all the other devs have motivation to talk to them.23:58
wgrantwallyworld: I agree, but sometimes the pipeline is blocked and minimal QA is better than blockage.23:58
lifelesswallyworld: and if the change looks hard to qa, others can roll it out.23:59
lifeless*back it out*23:59
lifelessthere are lots of options that don't depend on a single person nagging23:59

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!