[00:09] pqm is open === lifeless changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | firefighting: - | On call reviewer: - | https://code.launchpad.net/launchpad-project/+activereviews [00:12] lifeless: Erk [00:12] garbo-hourly is failing [00:12] -> http://launchpadlibrarian.net/63827955/dMm2kkmdygvorvKKvD9t02EQdHa.txt (duplicate key value violates unique constraint "bugmessage__bug__index__key" [00:13] thanks [00:13] unindex that bug [00:13] Hm? [00:13] a bug message index is being changed [00:13] if we unindex that bug, it will resume [00:13] Right. [00:14] we can tell the bug by running the query it uses [00:14] -> -ops and nab a losa [00:14] Ah, possibly, true. [00:14] alternatively [00:14] unindex all partly indexed bugs [00:25] where should a developer guide to feature flags live? on the wiki? under the architecture guide? [00:27] lifeless: the production environment for LD is secure, it's on Canonical's servers and managed by the sysadmins [00:27] any credentials or anything can be stored there, and even us developers wouldn't know them [00:27] wgrant, i have a patch up for multiply stacked repositories, which is +1 [00:27] i will land that, i guess into 2.3 [00:27] poolie: So I saw. [00:28] Should I patch our egg with it? [00:28] do you know of any other problems related to this? [00:28] I don't really feel like RCing 2.3 in. [00:28] i thought you had a branch not an egg? [00:28] We have an egg, not a branch. [00:28] (hm, did that change?) [00:28] anyhow, that seems like a reasonable approach to me, given my understanding of lp deployment [00:28] Not in my time, I don't think. [00:29] wgrant: I'll do a branch in the meantime, when reindexing, to set all to NULL, then flush, then assign values. [00:30] lifeless: Thanks. [00:32] poolie: Hm, we're running a patched 2.2.2 at the moment. [00:32] I suppose we should upgrade to at least 2.2.4 soonish. [00:36] wallyworld_: Hi. [00:36] hello [00:37] wallyworld_: I am about to apply https://code.launchpad.net/~mbp/bzr/715000-stacking/+merge/48886 to our bzr egg. [00:37] wallyworld_: This ideally wants to be deployed tomorrow. [00:37] Will we deploy the latest stable rev, or the one that you blessed? [00:37] * wallyworld_ looks [00:38] wgrant: the mp diff has conflict markers in it? [00:38] wgrant, since we just froze 2.3.0 i think we should go straight to that [00:39] wallyworld_, that's because it's targeted to trunk; it should be clean against 2.2 [00:39] poolie: OK, I might do that next week. [00:39] i'd be happy to help [00:40] poolie: wgrant: is there an urgent need to have to rollout the bzr update with this release or can we do it as a nodowntime deployment after release? [00:40] wallyworld_: crowberry isn't in nodowntime. [00:40] But I guess this is only realllllly needed on loganberry. [00:40] I think. [00:41] ah ok [00:41] This is also a critical issue. [00:41] last time a bzr upgrade went into lp, stuff broke so i'm being cautious [00:41] It is preventing us from starting the package importer, which blocks UDD. [00:41] Right. [00:41] But this is a very small patch :) [00:41] so was the other one :-P [00:41] Oh? [00:41] I don't recall it. [00:42] i can't recalls the specifics now [00:42] we can do it separately from the release i think [00:42] in that case, why not do it now and then do the release tomorrow? [00:42] to me, any change close to release time is risky [00:42] wallyworld_: OK, can I at least get it on stable nowish so we can deploy right after the release? [00:42] i know, but it's always release time [00:43] we need time to do proper qa, considering the potentialimpact if the bzr upgrade breaks things [00:43] poolie: true [00:43] so [00:43] this is what I suggest [00:43] a) land the patch [00:43] b) qa [00:43] c) if, at rollout time, a newrev is deployable, great. Do that. [00:44] d) if the bzr change isn't ready by then, we do scheduled downtime to codehosting to deploy it [00:44] That was my plan. [00:44] Except that loganberry runs the scanner, so we probably don't need downtime. [00:45] does bzr 2.3 have the change in it already? [00:46] No. [00:46] so in that case imho we patch our current bzr egg as suggested [00:46] great [00:47] and rollout 2.3 later once it has had the change and full qa for the other new 2.3 things etc [00:47] qa in terms of lp integration [00:48] poolie: you still ok to help with the forking lp-serve switchover tomorrow? [00:48] do you need to help? [00:48] wallyworld_: Right, that's what I thought. [00:49] wgrant: great. thanks for doing the work to get it in [00:51] wallyworld_, i shouldn't need to do anything afaik but i'll be on line if you want to talk/ask about it [00:51] i just want to make sure it goes live [00:51] you could perhaps check now whether it's sufficiently documented in the rt etc [00:51] or i guess we can just discover that tomorrow [00:52] poolie: that would be appreciated. just in case. i've not enough knowledge to know if the doc is good. has a dry run been done using the doco? if not, perhaps we should do one? [00:53] good idea; maybe we should do it with a losa [00:53] i guess spm [00:55] if he is the one doing the work on the day, then that would be my suggestion [00:55] it's not *that* scary of a change [00:57] sure, but doco needs "testing", just like code :-) [00:57] just to ensure that it is correct and when followed on the day, everything goes as expected [00:58] so a dry run often achieves this goal, especially when done by the intended implementer [01:07] sure [01:07] so when spm gets back(?) we could do a walk-through? [01:08] poolie: Can a 2.3.1 come soon with this fix, or should I prepare a patched 2.3.0? [01:08] wallyworld_, i think of you when i say "treat every gun as loaded" in bug 714043 :) [01:08] <_mup_> Bug #714043: defaulting to staging is confusing < https://launchpad.net/bugs/714043 > [01:08] wgrant, it can come soon [01:08] but where are you going to use a patched 2.3? [01:08] i thought you said lp was on 2.2. [01:09] poolie: I'm patching 2.2.2 for now. [01:09] But you say we want to be on 2.3.0 soon. [01:09] ah, by the time we get there it will be in 2.3x, and probably in 2.3.1 [01:09] i think for conceptual clarity it would be good to keep lp on actual releases as much as we can [01:09] Definitely. [01:10] huwshimi, oh, interesting re my comment about hover and tablets to see the thread about mobile browsers [01:11] How do I run bzr tests? [01:11] poolie: watch what you say or else i'll get out my uzi. i know where you live :-) [01:11] ./bzr selftest [01:12] I tried that. [01:12] bzr: ERROR: exceptions.AttributeError: 'module' object has no attribute '_WritelnDecorator' [01:12] :/ [01:12] Maybe it'll work on 2.6. [01:12] Yes. [01:12] sure, but doco needs "testing", just like code :-) <== /me awards 100 awesome points to wallyworld_ :-) [01:12] was that on 2.7? [01:12] i think that's fixed in bzr 2.3 [01:13] That was. [01:13] I forgot that 2.2 was still 2.7-hating. [01:13] FUUUUUUU..... [01:13] spm: i spent 10 years seeing the results of deployment doco used on site without doing a run through first - not pretty :-) [01:14] :) [01:14] ha [01:14] poolie knows the project i am referring to :-) [01:14] * thumper wants to stab lazr-js in the face [01:14] lifeless: if by secured you mean I can't access it yes... we have to submit an RT [01:15] * thumper takes a deep breath [01:24] yeah, and i tried to help deploy it after about 6 months working on it [01:24] i felt a bit out of my depth :) [01:25] was kind of fun though [01:28] wgrant: checkwatches; halted? in progress? [01:29] huwshimi: wontfix? really? I was hoping for a unification of shades and a fix for the two odd tables I found [01:30] lifeless: I was going to get back onto it after lunch, which is itself going to happen once I get this bzr thing proposed. [01:44] sinzui: I think those are a separate issue. If you file a bug about them I can get to them at some stage. [01:45] sinzui: I think there are bigger issues with the readability of the tables, and I'm not sure the hover is the right solution. [01:46] huwshimi: I think mpt had similar concerns in the past [01:46] sinzui: I want to think about the tables a bit more and come up with a plan. [01:46] huwshimi: The plethora of background shades may be the same issue. I wonder if I never see some of them [01:47] sinzui: yeah there is a bit of that. [01:48] sinzui: I'm always grepping through the source trying to find weird situations where things show up [01:49] I new to look at the mirrors listing because I recalled they had an exceptional style [01:50] s/new/knew/ [01:50] or gnu [01:52] * thumper is full of lazr-js sadness [01:52] * thumper branches lp:lazr-js again [01:53] thumper: whats wrong? [01:54] lifeless: I'm hacking a BinaryChoiceWidget based on lazr-js ChoiceEdit [01:55] lifeless: it is just not good, and needs fixing [01:55] I'd be happy to bitch at length, but that doesn't help it get fixed [01:55] also, your use of the ChoiceEdit for the vocabulary choosers is HORRIBLE [01:55] and needs to be widgetized [01:55] s/your/our/ [01:56] it is just a whole pile of javascript hurt in our tree [01:56] and the bug use of it is not entirely standard [01:56] which also hurts [01:57] * thumper goes to pick up the girls from school [02:08] i'm going out to lunch, maybe we can do the walkthrough after htat === Ursinha is now known as Ursinha-zzz [02:31] I need a quick teddy bear [02:31] I want to use feature flags in base-layout [02:31] but base-layout also does +opstats [02:31] which has a DisallowedStore policy [02:32] so I'm thinking that catching DisallowedStore in the flag lookup and signalling no-rule, would be ok. [02:32] anyone see a zomg thing with that ? [02:32] I don't know what the DisallowedStore policy is [02:32] lib/canonical/launchpad/webapp/dbpolicy.py [02:32] its used for pages which are nt allowed to use a given store [02:32] e.g. +opstats isn't allowed DB access at all [02:32] lifeless: I think that's OK. [02:33] lifeless: when is the exception raised? [02:33] the alternative is, in the +opstats codepath to inject a different FeatureController [02:33] lifeless: as long as it is not before the query, you should be ok [02:34] thumper: its raised in IStore [02:34] theres some nuance around this thing, I'm going to keep looking, but thats the crux of it [03:08] poolie: I cannot get bzr selftest to pass on the 2.2 branch on either of my machines, parallel or not :( [03:39] wgrant, i'll check [03:40] i'm pretty sure the whole thing passed here last night [03:42] wgrant, did you merge it into any branch, or just run mine as-is? [03:43] lifeless: got a minute to chat? [03:44] spm, hi [03:46] poolie: I merged it into our 2.2.2-ish branch. [03:46] But I also tried lp:bzr/2.2. [03:46] Without your patch. [03:46] And it fails there too. [03:46] lp:~launchpad/bzr/2.2-lp is your patch on top of our branch. [03:48] ! [03:48] thumper: sure [03:48] lifeless: skype? [03:48] also [03:48] I hate our request timeout tests [03:49] poolie: am about to fetch boy from school, I'll ping you on return? [03:49] sure [03:50] wgrant, maybe your branch is old enough that the thread leak bugs i thought were fixed are still present [03:50] but even then it never caused it to actually fail for me, just some warnings [03:52] poolie: There's nothing in 2.2 to fix them. [03:52] Since you presumably have a working test setup, could you test our branch? [03:54] yes, i will [03:56] i'm re-running my branch now [03:56] wgrant, i suppose i should test on lucid? [03:56] poolie: That would be great, if you could. [03:56] yes, i have a schroot [04:01] bug 707234 [04:01] <_mup_> Bug #707234: multiple redundant copies of person picker make_picker function in view-source:https://bugs.launchpad.net/launchpad-project/+bugs?advanced=1 < https://launchpad.net/bugs/707234 > [04:04] wgrant, ok, i do get some spurious failures due to, i think, testtools version mismatches [04:04] that's on maverick, my branch, my laptop [04:04] will re-run lp's branch under lucid shortly [04:06] Thanks. [04:07] turns out i need to bootstrap it [04:09] anyhow, my branch, aside from some things i feel safe saying are skew against testtools, passes ok [04:10] Right, thanks. [04:10] The changes in lp:bzr/2.2 since we branched are irrelevant, so it seems good. [04:10] I will push the egg. [04:11] just so i know, what do you actually do to make this happen? [04:12] python setup.py sdist, copy it into ~/launchpad/lp-branches/download-cache/dist, commit, push. [04:12] Then create an LP branch updating versions.cfg. [04:24] ahhha [04:24] finally got it all figured out [04:24] so [04:24] rendering an exception is done with db access turned off [04:24] the first few lines of handleException [04:24] and base-layout is used there, so my change to base-layout triggers the flag lookup [04:41] bug expiry just restarted [04:41] or at least, it just got around to projects i'm watching [04:41] wgrant: What were you saying the other day about not having a wrapping element on tal conditions? [04:41] poolie: I thought it was restricted to ubuntu. Maybe that changed in the last few days. [04:42] it just hit bzr [04:42] poolie: And LP. [04:42] huwshimi: Any element in the tal: namespace will be omitted from the rendering. [04:42] since we told people about it in http://blog.launchpad.net/general/enabling-automatic-bug-expiry ah about 3 months ago, maybe we should tell them now? [04:44] wgrant: ok thanks [04:45] poolie: Do you know how I create an sdist with pyrex already compiled to C? [04:46] i think so but i'm not sure [04:47] Oh. [04:48] The one in download-cache at the moment appears to not be an sdist at all. [04:48] Perhaps it is a manually created tarball. [04:48] thumper: Around? [04:58] wgrant, my run under lucid of 2.2-lp completed with some thread leaks, one failure due to not having python-dev installed, and no other problems [04:58] also i sent that branch to pqm for our 2.2 [04:58] Thanks. [05:05] poolie: [05:05] -17 04 * * * $LP_PY /srv/launchpad.net/production/launchpad/cronscripts/expire-bugtasks.py -l 200 --ubuntu >> /srv/launchpad.net/production-logs/expire-bugtasks.log 2>&1 [05:05] grr [05:05] +17 04 * * * $LP_PY /srv/launchpad.net/production/launchpad/cronscripts/expire-bugtasks.py >> /srv/launchpad.net/production-logs/expire-bugtasks.log 2>&1 [05:05] now a DoomedTransaction [05:05] Last night. [05:06] ? [05:06] they took of the size limit and the limit to ubuntu [05:06] but now it's doomed because it's too big? [05:07] poolie: yo [05:08] wgrant, that's lp-production-configs or something? [05:08] poolie: lp-production-crontabs [05:08] lifeless' doom is unrelated. [05:09] :) [05:10] Codehosting tests, please stop messing with my dev branches... [05:12] wgrant, sanity check of http://blog.launchpad.net/?p=1925 [05:12] spm, hi, i'm happy to go through the lp-serve rollout dry run whenever you are [05:12] poolie: lets make it so [05:12] poolie: I can't see it. [05:13] wgrant, how about if you click log in? [05:13] in the top right [05:13] I didn't even know there was a login link. [05:13] But so there is. [05:14] No, it still hates me. [05:14] Doesn't use LP teams. [05:14] huh [05:14] * poolie tries [05:14] I can log in. [05:14] But I have no privs. [05:14] win [05:14] now you should [05:15] So I do. [05:16] poolie: Looks good. [05:16] Could you do it last night? :) [05:17] wgrant: no, its very related. [05:17] wgrant: if a db request expires the transaction is doomed [05:17] *as well* as requestexpired getting raisde. [05:17] thanks [05:17] lifeless: This is related to the unchoking of expire-bugtasks.py? [05:17] hahaha [05:18] i can _pretend_ i did it last night, does that help? :-) [05:18] iwbn to have had whoever committed that change do it [05:18] wgrant: no, getting the render time patch landed [05:18] Yes. [05:18] lifeless: Right. [05:18] lifeless: So it's not related. [05:19] wgrant: its related to the thing I have been whining about [05:19] wgrant: when I said 'now a DoomedTransaction' it was following that thread. [05:20] lifeless: Right, but you said that right after my expire-bugtasks.py diff, which seemed to confuse poolie that it was related. [05:21] oh [05:21] poolie: do you have some docco on the process for the lp-serve rollout lying around somewhere? or is it buried in the losa wiki and I need to search harder? [05:21] you were not speaking to me in saying it was unrelated [05:21] gotcha [05:21] lifeless: Yeah, "'", not ":". [05:21] spm, there's an rt linked from unusual rollout requests [05:21] Ahh. ta [05:22] if it's missing anything i'm happy to add it but i'd rather you read it than me just tell you [05:22] so we have docs [05:22] wfm [05:24] oh. gah. face. palm. for some reason I had it fixed in my head this was the debian import system whatsist on jubany. lalalalalala [05:31] nup [05:31] we're going to do that on the 1st of March [05:31] happy to talk about how that will happen, but this other one is more pressing [05:31] Where's it moving to? [05:34] we're moving machine from jubany to pepo [05:34] and we're aiming to do all operations through remote losa arms at the same time [05:34] Excellent. [05:35] rt 39614 if you're curious [05:35] but the ticket's a bit of a saga [05:37] the good ones usually are [05:39] thumper: still around per chance ? [05:40] wgrant: I'd like to borrow your eyeballs [05:40] wgrant: can you look at the tip of lp:~lifeless/launchpad/showtimes and see if it makes sense to you [05:41] 12342 [05:42] lifeless: The tip rev in particular, or all the changes? [05:42] tip rev [05:42] you can eyeball more if you want, but thats all reviewed yada yada yada [05:43] base-layout.pt is (sadly) used in error rendering. [05:43] so changes there tend to trigger nasty interactions with publication [05:50] lifeless: You elide the features and scopes because they are now being queried? [05:50] wgrant: if the request has expired, I return no rules [05:51] lifeless: Right, I see that. [05:51] But I'm wondering why you elide them now. [05:51] Because some show up with None? [05:52] It looks fine, just want to be sure. [05:52] oh, in the other tests [05:52] because the feature visible_render_times shows up now [05:52] and thats not very interesting to those tests [05:52] other features in future would also be uninteresting [05:52] Right. [05:52] Looks fine, then. [05:53] thanks for the yeeballs [05:54] lifeless: I was just going to ask you to do that review. [05:54] But you are too efficient :( [05:55] ok, render time on page -> ec2; me -> shops [06:06] stub: hi [06:07] lifeless: You fail at shopping. [06:07] wgrant: EWIFE [06:07] Or you are exceptionally good. [06:07] wgrant: I will be shopping soon [06:07] Ah. [06:08] Yippie, build fixed! [06:08] Project db-devel build (350): FIXED in 5 hr 40 min: https://hudson.wedontsleep.org/job/db-devel/350/ [06:10] woo [06:10] At least 110 queries/external actions issued in 8.12 seconds [06:10] for a search for kpassgen on qastaging /ubuntu [06:10] Excellent. [06:10] terrible cold cache effects still [06:10] we're going to need to do something about that eventually [06:11] wgrant: did bug 709717 make any practical difference ? [06:11] <_mup_> Bug #709717: Archive:+index timeouts : ArchiveView.num_pkgs_building can be very slow

< https://launchpad.net/bugs/709717 > [06:13] lifeless: yo [06:13] lifeless: Impossible to say without a new copy archive. [06:14] stub: flacoste has suggested you and I setup a weekly catchup voice call; do you have any sort of preference for when such a call would happen? [06:15] later in the week so I might actually remember what I've done, nearing end of your official work day probably best [06:16] thursday avo? [06:16] sure [06:16] I'll send a calendar thingy in a bit [06:16] I'm happy with a flexible time too [06:16] So 'thursday arvo' is fine. [06:17] ok; I'll send a calendar thing anyhow cause that will remind me; we can always fudge it on the day [06:18] One day I'll recover that password... [06:19] heh [06:19] #is can do that [06:34] wgrant, pass/fail/don't know? [06:35] poolie: On what? [06:36] the bzr rollout [06:37] the fix for 71500 [06:37] 715000 [06:37] It's in ec2. It may not be QA'd in time for the release, but if not we can nodowntime to loganberry later tomorrow. [06:37] ok [06:38] if/once it gets through ec2, will it automatically deploy, or will a losa just deploy it? [06:38] Once it's QA'd someone will request a deployment. [06:39] Given the timing I will probably not be awake in time to QA it, so it probably won't make the downtime rollout tomorrow. [06:39] matsubara-afk is doing deploy requests, wgrant does them, I do them... we all do them. [06:39] sure [06:40] Hah, the last three have been me. [06:40] i just wondered if/when i should try to test it or request deployment [06:40] by which I mean it won't hang around long [06:40] It should be on qastaging in 10 hours. [06:41] Then it needs somehow to push up an appropriate branch configuration, and a LOSA to run scan_branches.py [06:41] s/somehow/someone/ [06:42] In the likely even that it's not QA'd before the release, I will QA it during the release and we can release again a couple of hours later :) [06:42] And then we can switch jubany back on and wait for more explosions, and then work out how to retry all the scan failures. [06:42] And the import failures. [06:43] And then resolve the next batch of wheezy-related fun. [06:44] http://searchengineland.com/google-proposes-to-make-ajax-crawlable-27408 [06:45] lifeless: EAYEARAGO [06:46] I'm old school [06:46] wgrant: linked from http://isolani.co.uk/blog/javascript/BreakingTheWebWithHashBangs [06:47] Heh [06:50] Woah. [06:52] The universal Gawker redesign is, uh, special. [07:08] indeed [07:08] I see that the ui= curse continues. [07:09] weren't you fixenating? [07:09] I fixed the commit message thing last week, yes. [07:09] do you know off hand a good branch that's nontrivially doubly stacked? [07:09] But I refer to the ui reviewer blight, which causes graduated UI reviewers to leave the LP team. [07:09] poolie: lp:ubuntu/lucid/bzr [07:10] wgrant: causes? [07:10]