[00:10] ugh [00:10] lazr stuff uses the whle zope stack doesn't it :( [00:15] can has review? https://code.launchpad.net/~lifeless/lazr.batchnavigator/keyoffset/+merge/56091 [00:17] thumper: ^ if you're around, its trivial [00:22] lifeless: The branch name is a lie. [00:23] it is [00:23] lifeless, what does "newest = false" do? [00:24] nevermind [00:24] lifeless, r=me [00:25] jelmer: thanks! [00:43] seeking teddy bear for short design call === lifeless_ is now known as lifeless [01:33] * wallyworld hates debugging 200+ line doc tests [01:34] me too with s/200+ line// [01:36] * wallyworld meant to type 2000 not 200 [02:41] A review for someone who has magic Monday reviewing powers: https://code.launchpad.net/~huwshimi/launchpad/ie-comment-button-414747/+merge/56094 [03:01] huwshimi: done === wgrant changed the topic of #launchpad-dev to: devel in RC until r12735 releasable | https://dev.launchpad.net/ | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews [03:01] thumper: Thanks :) [03:04] np [03:36] FFS [03:36] pgAdmin III fails on natty with trust authentication :( [03:49] thumper: Were you looking at the hwdb OOPSes? [03:49] wgrant: nope, not right now [03:49] wgrant: I'm looking at branch stacking problems on rename [03:49] Someone was talking about it on Friday. [03:49] I don't recall who. [03:49] thumper: Great! [03:50] I was talking about looking at it, but changed to look at the stacking issues [03:51] Ah, right. [04:13] wgrant: lazr restful upgrade broke some tests. i've finally got them fixed. in ec2 now. [04:14] Can haz review? https://code.launchpad.net/~stevenk/launchpad/dsd-cant-request-resolved/+merge/56097 [04:14] wallyworld: What broke/ [04:14] ? [04:14] wallyworld: Note that we're RC, so it won't make it through PQM. [04:15] wgrant: a bugs doctest. the issue was not our stuff but another change in 0.18.1 which short circuited patch requests [04:15] wgrant: np. i'll land after rc if it passes ec2 [04:15] Ah, right. [04:16] wgrant: plus, the notifications stuff i did sometimes adds up to an extra 2 queries to a request to get the notifications from the session, so some query count tests needed tweaking [04:16] wallyworld: Only for webservice requests? [04:17] wgrant: yes, afaik. only for requests come in via a WebServicePublicationMixin implementation [04:17] Great. [04:19] StevenK: i'll grab it but someone will need to mentor it === spm` is now known as spm === SpamapS_ is now known as SpamapS [05:07] StevenK: Are you going to harrass mawson? === wgrant_ is now known as wgrant [05:32] Hmmm. [05:32] It is tempting to batchnavigate BugTask:+index... [05:33] That and gandwanacide should just about finish off those timeouts. [05:33] Making it pretty and AJAXy can come later. [05:34] hi thumper [05:34] wgrant: that would be cool [05:34] hi lifeless,wgrant [05:34] wgrant: still have the milestone gangcrush to finish off bugtask:+index [05:34] hi poolie_ [05:37] lifeless: Mm, that's not so bad. [05:37] Afternoon poolie_. [05:40] wgrant: the milestone thing... the thing that was breaking cve:+index ? :) [05:41] wgrant: its pretty shallow conceptually, but a bit ugly pragmatically === almaisan` is now known as al-maisan [05:45] wgrant: Updating code now. === al-maisan is now known as almaisan-away [05:46] StevenK: Great. [05:47] wgrant: jtv's branch is merged as of r10383. [05:47] StevenK: I know,. [05:48] We avoided merging it to devel. [05:48] Am I right in thinking we don't have any javascript link checks in Launchpad? [05:49] huwshimi: What do you mean? [05:49] wgrant: You said we need to check both, so should I avoid rf-get? [05:49] StevenK: r10383 introduced a new script, but also changed the old one. [05:49] StevenK: We need to test the cron.publish-ftpmaster still works, and that publish-ftpmaster.py does too. [05:49] Speaking of the devil. [05:50] StevenK: wgrant: also see my mail (just sent) to -devel about commingling of changes during complete downtime with stuff that really only needs partial downtime. [05:50] thumper, i was wondering about your branch name on bug 377519 [05:50] <_mup_> Bug #377519: Stacked on location breaks if the stacked upon branch is renamed

< https://launchpad.net/bugs/377519 > [05:50] wgrant: make stop'ing, and merging [05:50] are you planning to fix it by exposing the branch id? [05:50] wgrant: Well during our build process or anything [05:50] wgrant: Actually build would probably be a bad place to do it [05:51] huwshimi: How does one check JavaScript links? [05:51] Or do you mean lint? [05:51] wgrant: Ah crap. Yeah lint [05:52] huwshimi: pocket-lint's JS checking is broken at the moment. sinzui was looking at that last week. [05:52] It's meant to work, but broke at some point. [05:54] wgrant: Ah I see, there is a bug reporting the exact issue I'm trying to address (bug 742619) [05:54] <_mup_> Bug #742619: JS syntax checking is broken in packaging

< https://launchpad.net/bugs/742619 > [05:56] StevenK: where do DSD.source_pub and DSD.parent_source_pub get set? [05:57] Oh, stupid question [05:57] They don't [05:57] properties [05:58] poolie_: yes, he is [06:00] :/ [06:01] i would have liked us to be involved in the preimplementation discussion of that [06:01] since we'll have to support it [06:01] wgrant: dogfood is up and running on r10388. Let's break it into tiny pieces. [06:01] StevenK: Have you run the cronscript there before? [06:01] i don't think it's an awful decision but i do wonder where the discussion happens [06:01] if any [06:01] poolie_: Why do you have to support it? [06:01] wgrant: Once, I think [06:02] poolie_: well, he was working on a complex scheme to rewrite all the branches affected by renames, and I asked if he'd considered just exposing the persistent id [06:02] poolie_: we chatted a bit, compared races, vuln, implementation overheads, and decided this was simpler [06:02] poolie_: here and in -ops for part of it [06:02] wgrant, because people will report bugs about this against bzr [06:03] just now? [06:03] poolie_: on friday [06:04] poolie_: so there are two discussions; meta and change specific. Lets have one at a time :) - which is your preference for first-up ? [06:04] either; let's do the meta one [06:05] i think, ideally, when people change an interface between two systems, they'd let people on the other side know [06:05] obviously in this case you know a fair bit about the impact on bzr [06:05] but, it's kind of disturbing [06:06] wgrant: Stale lockfile found, and removed. [06:06] i think we tend to post on the bug saying "i'm planning to fix this by responding to /+branchid/whatever and then rewriting the existing branches..." or whatever it is [06:06] it doesn't need to take a long tihme [06:07] poolie_: uhm, so I think that drives siloisation and adds a roadblock: we can debate the /size/ of the block (I'll acknowledge it could be quite low) - but what do we get for it ? [06:07] poolie_: I'd rather encourage engineers to gather enough data to be confident - and socialise things sufficiently to be confident - and then get out and do it === poolie_ is now known as poolie [06:10] poolie_: I may be abbreviating the logic here; we can switch to voice if you want more bandwidth, or tell me to expand on stuff [06:10] wgrant: set -x in this script sucks [06:10] wgrant: And there's some partner errors [06:10] i'm not saying anyone needs affirmative permission from the other team to change things [06:10] that would cause siloization and blocking [06:11] i just find it disturbing there's no communication at all [06:11] perhaps I'm categorising this differently [06:12] probably [06:12] I see it rather like bzr choosing to use a different published api from lp - something that already exists [06:12] we need to work hand in hand to give a good system here [06:13] changing it without saying anything undermines the trust we need to do that well [06:14] I'm extremely surprised now [06:16] I need to move to an English-speaking country. Someone just told me something in Indonesian and I understood far too much. [06:16] :) [06:17] uh, maybe we should go to voice, but i should finish something here first [06:17] sorry to leave you hanging [06:18] wrt the actual change, is lp going to rewrite existing branches to use numeric ids? [06:18] thats not entirely clear [06:18] the basic plan is: [06:19] - do the work to expose numeric ids safely [secure, private branches support etc] on http and sftp and bzr:// === almaisan-away is now known as al-maisan [06:19] - using a feature flag start offering the stacking targets numeric path rather than symbolic path, and see how it works [06:20] - we can look at a mass rewrite in future if it works well [06:20] ok... [06:20] by offering I mean in the policy config file that bzr reads from LP [06:20] suggesting bzr stack on the numeric id [06:20] yes === al-maisan is now known as almaisan-away [06:21] i think it's either something like this, or limiting renames [06:21] in a way, this is like introducing immutable names and then another set of names on top of them [06:22] sure [06:22] though we already have aliases for branches, so there is very little new stuff here [06:23] true [06:23] there's another bug about recipes breaking when branches are renamed [06:23] its a duplicate [06:23] is it proposed to also rewrite recipes into numeric ids? [06:23] recipes actually track branches by id in the db [06:23] the breakage is when the thing that a recipe branch is stacked on is renamed, AIUI [06:23] That's right. [06:24] the recipe branch becomes unusable [06:24] All the cases I've seen have been stacking breakage. [06:24] oh, ok [06:24] i see, it was just duped on the weekend [06:24] Yeah, I missed that last week :/ [06:24] will there be a public way to discover the id for a branch? [06:25] (or maybe there already is in the api?) [06:25] i guess this disturbs me because ... it seems like a lot of launchpad development happens behind closed doors [06:25] even from the point of view of someone about as close to the project as you can be without actually working on the team [06:26] i don't understand why people don't just say "the plan is X" on the bug [06:26] (to go back to the meta question) [06:27] it's fine for you guys to decide to just do this [06:27] it just feels wasteful to have all these "oh, when did lp start doing X?" questions, even after reading most of the bug mail and all of the devel list [06:28] poolie: FWIW, I feel somewhat disconnected on recent bzr stuff; its much harder than I expected to track it unless I read all #bzr backlog [06:29] it may be a grass-is-greener thing [06:29] poolie: I think a policy saying 'say what your intent is on the bug' is problematic in a few ways [06:29] - changing bugs leads to 'me too' and 'have you thought of X' comments [even from you!] which can be a bit of a drain to deal with [06:29] :) indeed [06:30] - the change is going to get reviewed during code review, and if its 'feature scale' jml, and if its important blogged about on the blog anyway [06:31] on the first point: that effect does happen, but it's pretty much saying "working in the open is annoying" [06:31] - plans often don't survive contact with the enemy^Wcodebase; theres some sort of balance between communicating eagerly and drowning in out of date data [06:31] I disagree that its equivalent to 'working in the open is annoying' [06:32] on the 3rd; i agree, i'm just wishing for a bit more [06:33] well [06:34] having bug reports, or talking about what you're doing at all, encourages people to comment [06:34] some of those comments will be helpful and some will be noise [06:36] so do you see any value in such a policy, or do you think it's just a waste? [06:37] by policy i mean more like 'custom' than 'legally mandated' [06:38] I think its a waste - its treating a symptom not a cause [06:38] I don't know what the cause is [06:39] anyhow, the root request is [06:39] but the fact you felt trust was betrayed by this suggests that its not at all about communication of intent [06:39] i'd like to know if launchpad's going to change the way it interacts with bzr [06:39] i realize in theory there is an abstraction, but in practice we deal with lots of things that cross that abstraction [06:40] probably several dupes of that bug were originally filed against bzr, for instance [06:40] this gets my hackles up, and I don't know quite why [06:40] some possibilities: [06:41] * it really should be a shiny abstraction and i shouldn't need to know about it [06:41] * i ought to read the lp commits too [06:41] * i ought to find out about it on demand when it becomes a problem, not ahead of time [06:42] The closest thing I am bringing to mind when I think about similar problems I face is my first statement as TA nearly a year ago: (paraphrased) I'm a resource not a control [06:42] * lp shouldn't describe this in the bug ahead of time but rather on the blog or release notes or something after it happens [06:43] yeah, interesting comparison [06:44] perhaps i should be asking myself why tim didn't choose to talk to or tell me about it [06:44] or you [06:45] ? [06:45] I didn't because right now is the first time I saw you since the discussion with Tim on Friday; and it had slipped my mind since then because its not in my critical-quadrant : its vapourware [as its not done yet and may have sandtraps] and its [in my assessment] low risk right up to the point where we start changing the policy [06:46] I was up to 4am this morning after Lynne had an asthma attack in the weekend - had to go into the 24 hour clinic. [06:46] sorry to hear that [06:46] This tends to clear ones mind of dander [06:46] shes ok, its just context. [06:46] like, if the discussion happened monday and I saw you tuesday, vs this [06:47] also we have serious scaling issues with batch nav, and most of my [shredded] brain today has been looking at that, dealing with stale trees that won't bootstrap on lucid, design etc [06:47] it's not an emergency, especially if it's not contemplated to be turned on suddenly [06:48] and coordinating the downtime on wed [which is largely a communication exercise, but we're still in the process of changing to make it faster and more robust, so requires care] [06:49] so i guess overall [06:49] I think to do our job well, the LP team needs tools and space; you need to trust that if an engineer thinks an LP change will impact the bzr *team*, that they, or their line manager, will raise it. [06:49] just as LP needs to trust that bzr won't jump CPU use by 20% in a point release [for instance] [06:50] if the LP team doesn't have that trust from you at the moment, we need to figure out why that isn't the case, and what we need to do about it. [06:51] yeah, that is the key issue [06:51] i wonder [06:52] LP needs the room to make mistakes (and fix them rapidly) - because rapidly fixing things that do break is cheaper overall than avoiding every breakage [but different changes have different risks - there is no golden rule] [06:52] i'm trying to think of any other changes in lp that have affected us since orlando and the squad structure [06:52] i can't think of any [06:53] there is the beta redirect [06:53] bah [06:53] edge [06:53] i don't think there have been a lot of changes there either way [06:53] which failed and was reverted [06:53] yes, that was .. kind of ok, but kind of messy [06:53] its queued up in RT [06:54] i guess at the moment i just come back to that being told this would change would have built more trust [06:54] if you tell me about the probably-won't-hurt thing, i know you'll tell me about the dangerous things [06:54] I want the bzr integration in LP to be not at all special: special cases have higher friction and get less traction from self-directed folk unless they have a vested interest [06:54] agree [06:55] poolie: still sounds like you have a lack of trust, or a need for control [06:55] it does [06:56] i have the impression lp has sometimes broken things in ways we could have avoided if we'd been asked [06:57] so, that causes a feeling of not trusting that they will ask quite enough [06:58] it seems to me there are many things that could cause that: [06:58] - top down management [above both bzr and lp | above the relevant engineer] deciding something [06:58] - lack of knowledge [unknown unknown: engineers involved didn't know enough to know that they didn't know enough] [06:59] - lack of attention to detail / cavalier changes / something [06:59] - deadlines [06:59] i guess, in particular, stacking has had a lot of bugs that cross both systems [06:59] so, to me, it seems obvious to check on both sides before changing it [06:59] - previous miscommunication [thought that they were doing the right thing on a prior agrement about direction] [06:59] (even a pretty shallow change, like this) [07:00] well, you've gone from general issue [trust] to specific issue [stacking is a special case] [07:01] we've probably seen all of those causes [07:01] ah, i'm saying stacking seems like a good example of the general issue [07:03] so looking at this list [07:03] so, telling you about small changes as a predictor for telling you about dangerous changes, is a poor metric [07:03] this is perhaps a case where it's easy to put in more protection to guard against previous mistakes [07:03] indeed [07:04] (easy but possibly fallacious) [07:04] so, in short, you think i should just trust lp to tell us if it is dangerous? [07:04] as its a poor metric, I don't think it would build trust - it would diminish it [because it would fail to predict, and you would feel betrayed] [07:04] seems pretty easy on both sides [07:04] how will we know if it's not working? [07:05] poolie: it? this specific change? [07:06] let's back up [07:06] > so, in short, you think i should just trust lp to tell us if it is dangerous? [07:06] i meant that [07:06] ok [07:07] that's a nice scalable algorithm [07:07] i had got the idea that it wasn't working well enough [07:09] I expect failures in the algorithm. [07:09] I think thats ok. [07:10] I expect failures everywhere. [07:10] A /huge/ chunk of my non-timeout-coding-changes work has been about making it easier for us to change, to undo things so we can get back to known good states; to reduce the number of things changed at once to make diagnosis easier etc. [07:11] The risk is that a really dangerous change [high impact, high probability of going wrong, lots of user costs and black marks for lp and bzr] will get through [07:12] the reason I say that *that* is a risk, not 'some bad changes will get through', is because that sort of change can be hard to recover from. [07:12] yeah [07:13] of course in the past changes were very hard to undo [07:13] But OTOH thats why we have the oversight we do: most changes are seen by 2 people (engineer, manager, or engineer, reviewer), larger ones have sysadmins see them, more chance of random eyeballs on the diffm etc [07:13] both through the deployment cycle, and because people would have moved on [07:13] right [07:13] thus the squad structure & nodowntime deployments [07:14] so small-medium size problems can be tolerated more [07:15] right [07:15] not appreciated, but tolerated [07:15] what we trade to get that is [we hope] less churn on engineers, more focused time looking at the problem - things that *help* with flow, with analysis and good solutions. [07:16] thanks for talking it over [07:16] I'm happy to [07:16] so the short story is, relax, poolie, trust people will ask if something really does need external input or risk-assessment or coordination [07:16] and if it turns out things are actually slipping through too much, we can debug then? [07:16] right [07:17] fair enough [07:17] I realise the outcome of a messup is likely to be dramatic and widespread with bzr [07:17] probably this new setup is different enough we should have a blank scorecard [07:17] ditto the archive / ppas though [07:17] yeah [07:17] and, rightly or wrongly, i feel we field a lot of the user support and complaints [07:17] certainly some fair fraction of it [07:17] anyhow, let's try that [07:18] I'm sure you do [07:18] the best thing we can do to reduce that is to fix bugs like this so that folk don't encounter problems ;) [07:18] indede [07:19] we have mearly 400K branches on LP [07:19] thats a looooot of room for bugs to cause support tickets [07:20] On the topic of mentioning possible solutions on bugs, I'm pretty sure I mentioned this possible option (branchid in URLs) on a bug several months ago :-) [07:21] lifeless didn't like it much at the time, I recall [07:21] i thought so too :) so that added to my surprise [07:22] i thought there was a religious prohibition against exposing database keys [07:22] though, i guess mps already break that, and perhaps also others [07:22] poolie: Talk to MPs [07:22] sometimes the simplest thing is the best thing [07:22] bug ids [07:22] merge proposals [07:22] questions [07:22] Bugs and questions are already "special" anyway [07:22] How else are you going to identify them? [07:23] StevenK: ask for a unique name [07:23] so in some ways i actually wanted to ban renames, because they break people's bzr checkouts etc in confusing ways [07:23] StevenK: like e.g. stack exchange [07:23] possibly avoiding that is better [07:23] lifeless: Oh, awesome. I just fixed bug nmnfgolwnpgwnepgnwgp [07:23] Banning renames is quite harsh [07:23] people can re-push if they want to change ownership etc [07:23] poolie: my experience with users is that when you ban something, they will do it by hand [07:23] poolie: and make the situation even worse [07:23] lifeless, stackexchange's primary key is a number [07:23] the name is just icing, i'm pretty sure [07:23] yeah, maybe so [07:24] i'm happy to give this a try [07:24] put it this way; if the lp team cannot make sensible decisions as a bzr hosting site, sourceforge and savannah have /no hope/ [07:24] or i should say happy for you to give this a try [07:24] i think the main place this comes up is in changing ownership, and perhaps we should address that more directly [07:24] :) [07:24] indede [07:24] we need to debug and fix *that* problem, if it exists. [07:25] poolie: Do you think this is actually going to break anything? Other than involving stacking at all, it seems pretty un-scary [07:25] stub: we forgot to speak on thursday [07:26] stub: is now convenient ? [07:26] maxb, i reckon it will have a couple of non-critical bugs [07:26] if i could tell you exactly what they will be i would [07:26] StevenK: you know that lp:~stevenk/launchpad/unmatched-brackets-annoyance is breaking a half-open range, right ? [07:26] anyhow, to draw a line under it, i'm not trying to veto the change; thanks for talking it over; i'll trust lp more [07:27] poolie: cool, thanks. [07:27] we can talk tomorrow if you'd like to catch up on bzr [07:27] poolie: thursday would be good actually, I have a tonne of bits to sort out [07:27] lifeless: wgrant already pointed out the fail and the branch and MP are already deleted. [07:27] lifeless: Gimme a minute... the coffee is still kicking in. [07:28] stub: ping me in a few then [07:32] hmm [07:32] UDS coming up === gmb` is now known as gmb [07:32] I suspect we'll see more of https://bugs.launchpad.net/launchpad/+bug/735970 soon [07:32] <_mup_> Bug #735970: Person:+specworkload timeouts < https://launchpad.net/bugs/735970 > [07:34] :) probably [07:38] ! 5% of open LP bugs were reported by me [07:39] I would have expected higher [07:39] Bah, only 3% are mine. [07:40] StevenK: it is, I'm rounding [07:41] 7.7M requests yesterday [07:41] Interesting. [07:41] non-opstats [07:41] previously these were spikes [07:41] Hmm. 3.1% of open bugs are mine, and 3.0% of all bugs. [07:42] Oddly stable. [07:42] I think we're either in UDS swell, or we've got some new api clients [most of the growth is api.] [07:42] But you said there were 500M new non-API requests. [07:42] yeah [07:42] we were tending to do 6M a day [07:43] so growth of 1.7M and 1/3 is non api. domains [07:43] wgrant: StevenK: wallyworld__: was my post on our capacity planning/deployment changes useful ? [07:44] should I do more such things? less? include other details? [07:44] lifeless: haven't read it yet sorry [07:44] wallyworld__: no need to apologise [07:44] lifeless: was it email? i'm a bit behind [07:44] email [07:44] * wallyworld__ looks [07:44] lifeless: Wasn't really useful for me, since I've been following it all pretty closely anyway. [07:45] lifeless: But I think it was probably generally useful. [07:46] wgrant: NullBT is truely dead right ? [07:49] lifeless: The number of instances of that string in the tree is 0. [07:49] So I presume so. [07:49] we can trim a feature rule then [07:49] Ah, forgot about that. [07:50] Assuming we get nodowntime out before tomorrow morning, we should be able to drop the timeout on Wednesday. [07:50] wgrant: thursday [07:50] wgrant: IFF it looks good [07:50] To 8s? [07:50] Oh? [07:50] Oh, right, release. [07:50] wgrant: downtime please :) [07:50] lifeless: Also, we can drop the publicrestrictedlibrarian rule after the release. [07:50] lifeless: true, true. [07:50] cool [07:51] Can we remove edge now? [07:51] StevenK: no [07:51] lifeless: i found it really useful. i had never seen that information anywhere before (it may be on the wiki but i haven't seem it). thanks for posting it [07:51] StevenK: we need to do the redirect for everything but bzr and api clients [07:51] wallyworld__: cool, thanks [07:52] lifeless: I've been thinking about it. Couldn't we hack the prod apache's to also answer for edge? [07:52] lifeless: We can drop the Archive:+index override, I think. [07:52] The latest PPR shows nothing above 12s. [07:53] And even that is a tiny fraction of 0.1%. [07:53] 99% under 6.81 [07:53] sob - src/lazr/batchnavigator/README.txt is not end user docs. [07:53] wgrant: cool! [07:53] StevenK: no [07:54] StevenK: for apis, the wadl includes the domain and thats hashed [07:54] StevenK: you cannot serve the wadl for domain A from a server that thinks its domain B [07:54] StevenK: and, LP doesn't believe in multiple root-urls, its a server wide config [07:56] stub: yo [07:57] lifeless: yo. Caffeinated, showered, human (fsvoh) [07:57] \o/ [07:58] calling [07:58] lifeless: skype just hung [07:59] wops [07:59] Wow... needed -9 [07:59]