[01:23] qastaging takes forever to update :( [01:35] poo [01:53] added: lib/lp/archivepublisher/tests/util.py.THIS [01:53] thumper: ^^ [01:54] eh? [01:54] arse [01:54] did that slip through? [01:54] Yes. [01:54] poo [01:54] Just landed a couple of minutes ago. [01:54] another pp [01:54] * thumper fixorates [01:54] wgrant: is it breaking things? [01:54] it passed ec2 [01:55] thumper: It shouldn't break anything, no. [02:01] pqm-sumitting the fix [02:01] Thanks, [02:21] * wgrant lunches. [03:51] Has anyone looked at the db-devel failure? [03:52] the real question is, has the failure looked at anyone? [03:52] Ah, it's already running again. [03:52] Yay buildbot. [04:49] lifeless: How do we fix a stale librarian PID issue on buildbot? Get a GSA to remove the file manually? [05:00] yes, checking that: [05:00] - the librarian is not still running [05:00] - *how* it happened - was the machine rebooted? did the librarian crash? was the test killed (and if so with what signal) [05:01] basically gather data so we can permanently fix [05:01] There was a test explosion in the last run. [05:02] zope.testrunner does not seem to rate very highly in the not sucking department. [05:02] go on, be more accurate ;) [05:06] wgrant: please make sure there is a bug on lp, high pri explaining what you figured out [05:06] wgrant: test explosions should always tear down [05:06] etc [05:07] lifeless: But Twisted tests dying with KeyboardInterrupt make me cry. [05:23] stub: Morning. [05:23] yo [05:23] stub: Can I please have you db-blessing for https://code.launchpad.net/~wgrant/launchpad/die-lucilleconfig-die/+merge/44305? [05:27] In the name of the Senate and Peoples of Rome [05:28] stub: Do I also need a TA-blessing? [05:32] Not before landing, no. Request 2 db reviews from me and lifeless. Land after you have the number and one approval. [05:33] I thought thinks might be different at the moment, given the lack of TA. [05:33] things. [05:33] Thanks. [05:33] I've put in the second review request [05:33] The two reviews thing is so we can both keep on top of things, but not block devs. [05:33] Holidays etc. [05:33] Right. === almaisan-away is now known as al-maisan [06:44] hi [06:44] stub, would you be kind enough to sponsor landing of https://code.launchpad.net/~mbp/launchpad/314507-oauth/+merge/44188 for me? [06:44] iow to send it to pqm [06:45] poolie: Has it gone through ec2 test yet? [06:45] no [06:45] i guess to send it to pqm via ec2 [06:45] Ok. Doing that now. [06:46] thanks [07:01] Bug #692872 [07:01] <_mup_> Bug #692872: Test suite fails if previous run did not tear down Librarian fully. < https://launchpad.net/bugs/692872 > [07:03] lifeless: Oh? [07:03] wgrant: did you end up landing my librarianfixture branch ? [07:03] lifeless: No. [07:03] I was told that adeuring was working on that. [07:03] But that was nearly two weeks ago. [07:04] right [07:04] well it needs landing [07:04] the cure is worse than the disease [07:04] we need to stop doing expendient things and actually make the foundations sane and reliable [07:05] Sure. [07:05] I'll take a look at that tomorrow. [07:05] Once I work out how I am going to get this regression fix deployed. [07:05] qa the intermediate patches [07:06] Difficult. [07:06] qa isn't restricted to the author [07:06] I need r12112, but r12102 is bad. [07:06] It is needed on cesium, which r12102 does not affect. [07:06] So I need manual approval. [07:06] But the relevant team lead and project lead are on leave. [07:06] whats in 12102 [07:07] A webapp formatter change. [07:07] ok [07:07] +1 [07:07] the losas should remove cesium from nodowntime [07:07] lifeless: Thanks! [07:09] danilos: Should the variants rev I tried to QA last night block a rollout? [07:09] danilos: Or does it just not fix the bug? [07:09] It looks qa-ok to me, but I'd like to be sure. [07:30] lifeless: Does that branch relateto Bug #692872 ? [07:30] <_mup_> Bug #692872: Test suite fails if previous run did not tear down Librarian fully. < https://launchpad.net/bugs/692872 > [07:30] stub: very much so [07:33] Your comments don't say if your branch fixes the issue, or if we want the issue to continue. [07:33] oh [07:34] gimme a sec to find it [07:34] I just don't want buildbot to fail if a previous run left librarian crud on well known sockets or in well known files. [07:34] (which is why we are in testfix atm) [07:35] (Hudson! Hudson!) [07:35] wgrant: Won't hudson have the same problems? [07:36] stub: It doesn't have the same catastrophic failures, plus it uses Canonicloud and recreates the instances every so often. [07:37] ok [07:37] https://code.launchpad.net/~lifeless/launchpad/librarian/+merge/39013 [07:37] Recreating the instances seems nice - pita buildbot requires manual intervention. [07:37] stub: it won't have this problem at all [07:37] because of the new instance thing [07:37] so that MP ^ [07:37] lifeless: Cool. So the bug isn't won't fix, it is pending :) [07:38] well [07:38] let me reread [07:38] right [07:38] the proposed fix is inappropriate [07:43] Argh... midair collision [07:43] Hmmm. [07:43] Something is broken. [07:44] wgrant: Is that something software? That always breaks. [07:44] 5963 things are broken in LP [07:44] It's a wonder anyone ever bothers to use software, really. [07:44] https://code.qastaging.launchpad.net/~wgrant/launchpad [07:44] - Expression: [07:44] KeyError: 'mergequeues' [07:44] I don't think that's changed lately :/ [07:45] lifeless: Anything stopping the librarian branch landing (apart from being in testfix mode because the librarian branch hasn't landed yet)? [07:45] stub: I'd love it if someone would land it [07:46] I'll stuff it through ec2 then [07:46] It has some test failures. [07:46] I suspect it will have (some) failures / may need a merge with trunk [07:46] Or did last time I looked. [07:46] I don't remember exactly which. Which may indicate that it was catastrophic enough that I was unable to get a complete list. [07:46] I'll stuff it through ec2 test to get the list. [07:47] Not if it explodes :) [07:49] That bad huh? [07:49] its altering fairly fundamental stuff [07:49] IIRC yes. [07:49] The testrunner did not end up very happy at all. [07:50] Maybe it is a post-lunch job then [07:53] stub: There shouldn't be anything quite as terrible as the databasefixture branch. [07:53] Since we now know that everything uses the right config. [07:53] we're pretty close to being able to truely parallelise [07:53] Yup. [07:53] we need to track down the cause of leaks [07:53] Although I ran into some trouble with launchpadlib tests last night. [07:53] something is either not calling cleanups, or the process is being killed hard [07:54] launchpadlib needs to be fixed [07:54] where is that concurrent use bug [07:54] It seems to enjoy always connecting to :8085 [07:54] I don't know how that can work, since AppServerLayer is meant to be dynamic now. [07:54] Oh. [07:54] I guess we don't do custom ports yet, just custom DBs. [07:54] And on Natty it decides to connect to :443 instead, just for fun. [07:55] win [07:55] So I will probably beat it to death with some monkeypatches. [07:55] wait what [07:55] we maintain it [07:55] Hmm, maybe I can get it to use a custom base URI. [07:56] lifeless: True. [07:57] But lazr.restful has at least one 3000-line doctest. [07:57] Which has sort of put me off that stack a bit. [07:57] boom shaka [07:57] Maybe I should try again. [07:58] Also, Python decided it would be amusing to change the default email header wrapping character from \t to ' '. [07:58] This breaks a lot of tests :D [07:59] OTOH the only thing that should be testing serialised forms is emaillib [08:01] lifeless: That is true, but there are a lot of other 'should' and 'should not's in LP. [08:01] wgrant: I think I mean 'cast the serialised form to an object, use a matcher, or delete the test [08:02] eg. you should probably not have a single monolithic 570KLOC Python app that can be logically split into lots of pieces :) [08:03] wgrant: I hold a rather different view on the 'logical split' thing [08:03] wgrant: I grant that there are /some/ pieces we can(and should) split out [08:04] wgrant: but I'm not convinced that there are lots; and certainly the UI->persistence->storage story is all one logical thing [08:04] At least Translations, Buildmaster and most of Soyuz have no business being in with the rest of LP. [08:05] no, yes, no [08:05] The interaction points between those three and the rest are minimal. [08:05] And will remain so. [08:06] to me there are two key tests [08:06] a) could it be written to APIs [08:07] b) would it be free of lockstep changes with the core [08:07] if the answers are yes and yes, then I think a separate piece makes sense [08:07] that isn't the case with translations or soyuz [08:08] and soyuz is (finally) getting reuse and working in better with code [08:08] translations ditto [08:09] I'm not sure that those interactions are significant enough to prevent a split. [08:10] I think they are massive impediments against splitting [08:10] They make it harder than it would be otherwise. [08:10] they are fundamental problems [08:13] that will massively outweigh any benefits from splitting them out, unless the split edge is designed to prevent those problems occuring [08:19] the latter problem in particular would mean terrible deployment and testing issues [08:24] AttributeError: type object 'BaseLayer' has no attribute 'config' [08:24] Think that means I should terminate the run :) [08:25] thats fallout from wgrants fixes to databasefixture [08:25] Indeed. [08:25] I'll look at it after lunch [08:26] wgrant: what was teh change [08:28] lifeless: I was going to tell him, but then he escaped. [08:28] I renamed config to config_fixture. [08:28] Or config_name. [08:28] Because the other one sprung into existence. [08:29] don't remember which was which. [08:40] lifeless, would you care to offer an opinion on https://code.edge.launchpad.net/~mbp/launchpad/690021-rlimit/+merge/43733 [08:40] ie about setting an rlimit in cron scripts [08:40] not an emergency, just feeling a bit stuck [08:40] or anyone else for that matter [08:41] poolie: uhm [08:41] if you're not here, you don't have to, of course [08:41] ideally we'd give each task an even slice of the machine memory [08:41] which implies knowing the machine memory [08:41] I'd look to features to configure that [08:41] as for where its set [08:41] I don't care whether its in-proc or in-parent [08:41] you could do a lot more towards this [08:42] we could have a whole lep on the idea of resource caps on jobs [08:42] istm perhaps you want a model of "this job should never need more than x GB unless something's wrong; this machine has Y GB; run y/x of them in parallel" [08:43] or perhaps for other jobs, you just want to cap them [08:43] that seems liable to fail with too many jobs for the cpus [08:43] oh, that too of course [08:43] anyhow, ideally it would be an operational knob, not in the code [08:43] there are mulitple dimensions here [08:44] the thing for this particular mp is [08:44] one is not using more resources than we have [08:45] istm it is a step improvement to at least have a cap on this job as we have on others [08:45] another is autoconfiguration [08:45] poolie: sure [08:45] poolie: why don't you land it? [08:45] and no worse than having losas manually kill them, as has happened at least once (maybe only once) [08:46] i didn't want to land it over the top of aaron and tim's concerns [08:47] fair enough [08:47] anyhow, it seems to me that you'll want to use a mock [08:47] because setting an rlimit in a test and then butting up against it would be unreliable at best [08:47] but i would like to restrict the scope [08:47] i agree [08:48] if you're testing with a mock, it really doesn't matter where you set it, does it ? [08:48] a mock of what? a monkeypatched setrlimit? [08:48] yah [08:48] oh, i mean restrict the scope to smaller than "make a new job system thx" [09:17] wgrant, hi, can you please run the rosetta-approve-imports script on dogfood please? I've made all the variant languages visible to see if that case would at least work [09:18] danilos: Does it need a fresh upload? [09:18] wgrant, no, we've got plenty of unapproved files left [09:18] danilos: Running. [09:19] It's doing stuff. [09:19] wgrant, cool, thanks [09:19] wgrant, it puts stuff in arbitrary order so I can't find it in the queue [09:20] danilos: Should I knock all the old ones to some other status? [09:20] wgrant, can you please kill the run and clean up all the TIQE entries with status != 5 (needsreview) [09:20] wgrant, or that, if it's not a big bother [09:20] wgrant, I believe status 3 is deleted so try with that :) [09:20] danilos: That's what I'm doing. [09:21] 1 -> 3 [09:21] That was quick. [09:21] wgrant, cool, thanks [09:21] Reapproving. [09:21] cool [09:24] wgrant, looks good so far, but please let it finish [09:24] Looks good, yeah. [09:26] danilos: It's finished. [09:27] wgrant, oh, is it perhaps because of a time-limit? can you run it again please? [09:27] Still lots unapproved. [09:27] Indeed, ran for a few seconds over 5 minutes. [09:27] And now is doing lots more. [09:28] wgrant, I've made "sr@ijekavian" hidden to see what will happen with those (if it works properly, they would stay unapproved, if not, they might again be approved in "serbian" directly) [09:29] danilos: The queue is not getting shorter. [09:29] It's doing dozens of transactions per second, and not running for more than 30 seconds or so. [09:30] wgrant, right, then it's surprisingly working fine with hidden languages this time around as well [09:31] danilos: Do we want to do another upload of the same file to another series and try it again from the start? [09:31] wgrant, you can perhaps kill that run and I can make "sr@ijekavian" visible again so we just make sure it works fine to clean up the queue (well, mostly: sometimes not all files have matching templates) [09:31] Sure. [09:31] It's finished again. [09:32] wgrant, not really, I am experiencing some weirdness locally as well [09:32] So unhide. [09:32] wgrant, done [09:32] I need to learn this Translations stuff. [09:32] wgrant, I want to get down to the local weirdness first, and then when I am certain why is it happening we can try it all over again :) [09:32] danilos: Right. [09:32] So, is it qa-ok? [09:32] wgrant, it's all very simple, and you have a head start compared to everybody else [09:32] danilos: It's running much slower this time. [09:32] Which may be a good sign. [09:33] wgrant, yeah, it doesn't break stuff and works fine for when languages are not hidden [09:33] Indeed, queue is decreasing in size. [09:33] wgrant, I'll file a new bug if I find more problems with it [09:33] danilos: Can you qa-ok the bug so I can request a deployment? [09:33] wgrant, I did that already [09:34] Ah, great. [09:34] Thanks. [09:34] mthaddon: Around? [09:34] wgrant: yup [09:41] mthaddon: We need r12112 deployed to cesium (to fix bug #692114). [09:41] <_mup_> Bug #692114: Recipe builds require indices for non-main PPA components < https://launchpad.net/bugs/692114 > [09:41] mthaddon: There is one qa-bad rev in front of it, but that only affects the webapp. [09:42] lifeless has given me his blessing. [09:42] erm, maybe, but that's completely non-standard... [09:43] It was done for Soyuz a couple of weeks ago... r12043, IIRC. [09:43] wgrant: it will need a incident report, I assumed you already knew that :) [09:43] that doesn't mean we're okay to do it every time... [09:44] mthaddon: will you be at the epic? [09:44] lifeless: nope - mbarnett will be there though [09:45] ok, I want to go through our exception-handling workflows face to face, can you perhaps make sure he's across all your concerns? [09:45] wgrant, fwiw, 12111 is also still "needtesting" [09:45] mthaddon: Ah, OK. [09:45] mthaddon: I was under the impression that it wouldn't be that abnormal. [09:46] I guess I will wait. [09:46] how critical is it? [09:47] lifeless: I'm not really sure I understand the scope of the conversation you're after, but we can certainly cover anything over the phone that you need to [09:47] It breaks recipe builds into new PPAs or new series in existing PPAs, which is probably most new recipes. [09:47] and how long has it been doing this for? [09:48] and how long do we expect before we can get those revisions qa-ed? [09:48] mthaddon: that would be great [09:48] * mthaddon nods [09:48] Well, buildbot is being more agreeable now, so it's not as long as I thought. Assuming that we don't get another qa-bad which immediately pushes us back another 24 hours :/ [09:48] But these look OK so far. [09:49] mthaddon: scope wise - we have cherrypicks, but we're also doing some adhoc things while I've been on leave [09:49] lifeless: Er, don't we not have cherrypicks any more? [09:49] I'd like to consolidate things a bit, and talk cost-of-execution, latency, and approvals/risk analysis [09:49] yeah, we've not done a cherry pick for a good while now [09:50] wgrant: we certainly do have them in policy [09:50] I thought we'd lost that capability. [09:50] wgrant: we haven't had to do many - but this soyuz working-around-unqaed-things is cherrypick-like [09:50] wgrant: not at all [09:51] lifeless: We need some way to roll out urgent changes that is not blocked by irrelevant bugs in the webapp, each of which blocks us for a day. [09:51] wgrant: perhaps. [09:52] wgrant: I'm not interested in drilling into this now. At the epic I'd be delighted to. [09:52] Sure. [09:53] I will say that the process simplification and frequent rollouts are great. It just seems to border on ridiculous to block another one-line critical fix for days. [09:54] so we're not meant to block for days ever [09:54] if there is something buggered, roll it back immediately. [09:54] But every time we have a qa-bad we block for 24 hours. [09:54] Which then gives us time for another one to slip in. [09:54] things are *meant* to be qa'd the same day they land. [09:54] => infinite chain of pain [09:54] wgrant: how it is taking 24 hours? [09:55] lifeless: The change is landed during the engineer's working day. [09:55] 4 hours of EC2 + 1 hour of PQM + 6 hours of buildbot takes us well after the end of their day. [09:55] and? [09:55] They QA it the following morning. [09:55] Notice that it's bad. [09:55] Send the rollback through EC2. [09:55] Another 11 hours later, it is in stable and can be QAd. [09:55] so, flaw one: don't wait for the engineer to qa it. [09:56] we're strongly encouraged to describe how to qa things in merge proposals. [09:56] 2) rollbacks go straight to pqm, no ec2. [09:57] s/rollback/fix/, then. [09:57] don't fix [09:57] rollback [09:57] Since avoiding rollbacks in DVCS\{darcs} is nice. [09:57] no, its insane. [09:57] rollbacks are how we undo mistakes rapdily. [09:58] there is no opprobobium in having something rolled back [09:59] and it removes all the latency involved in analysing and fixing the issue in the bad commit [10:03] wgrant: if something was broken, and I was working, I'd land a rollback with no hesitation at all [10:03] wgrant: if we *don't* do this, the expected result is a trunk that is broken a lot. [10:04] lifeless: So, it sounds like everyone needs to know three things: 1) QA quickly. 2) QA other people's stuff. 3) Rollback is the first resort, not the last. [10:05] wgrant: makes sense to me [10:05] wgrant: this has been communicated before, but it bears repeating. [10:05] wgrant: I'll be talking about this in my TA report too [10:05] Nobody treats QA as priority 1. [10:05] When in reality it probably should be. [10:06] wgrant: the logic behind this is simple; once a patch is in trunk, its on the critical path to deploy, anywhere. [10:06] The only thing higher priority than QA is fixing a production issue directly. [10:07] wgrant: this is why I was making such a big deal on your first? second? day about qaing when you had a blocked thing [10:18] wgrant, fwiw, I figured what the problem is with "sometimes doesn't work": if we have paths stored in pofiles in the DB (due to the previous buggy behavior) it can wrongly take up a wrong pofile early on [10:18] danilos: Ah, great. === lifeless_ is now known as lifeless [11:04] allenap, deryck: Is there anything you need me to chase RE: Contributor agreement? [11:04] (https://code.launchpad.net/~pcjc2/launchpad/allow-empty-comments/+merge/43449_ [11:04] ) [11:04] danilos: do you know what the problem might have been with bug 487137? [11:04] <_mup_> Bug #487137: Allow Rosetta admins to create custom language codes

< https://launchpad.net/bugs/487137 > [11:05] danilos: Adi had a branch ready a month before his last comment on the branch (and unassigning himself from it). [11:06] henninge, I was just looking at that and planning to get it landed [11:06] danilos: ;) [11:06] henninge, yeah :) [11:06] henninge, if you want to take it over and go through it to ensure it's all still good, go for it ;) [11:06] danilos: but what is Adi's last comment about? Is the branch flawed? [11:10] henninge, well, last comment is from Michael, but just because of the status we should basically re-review it now [11:11] danilos: sorry, Adi's last comment on the *bug* [11:11] that's a month after Michael looked at the mp [11:12] henninge, oh, I think it's easy: we can just let project owners administer their own custom language codes [11:13] ;-) [11:13] henninge, especially since there won't be any per-app teams anymore, I think that's the way to go [11:13] good point [11:13] henninge, who ever screws it up for themselves, well, they've done it themselves [11:13] henninge, fwiw, we should make them able to do translations approval as well [11:14] hah! [11:14] henninge, it's just that we won't have the people to monitor it anyway, so we should just worry about being able to clean up later [11:14] good thing it's just the two of us here ... [11:14] henninge, heh [11:15] henninge, anyway, is that how Adi's branch is working? (i.e. using TranslationsAdmin privilege directly) [11:16] henninge, (if you looked at it) [11:16] henninge, MP suggests it is [11:25] sorry, was afk [11:28] danilos: I think in general per-app responsibilities will become interrupt-team responsibilities; its not that there is noone to do it, its that its no longer such a static group of people [11:29] lifeless, and this is one of responsibilities that it'd be better to hand of to project owners than to hand of to that team, that's all I am saying [11:29] danilos: +1 [11:30] danilos: I thought that perhaps you felt there wouldn't be staff to do it at all, rather than that project owners were a better natural fit [11:30] danilos: so I was trying to clear that up, was all. [11:30] lifeless, well, the staff that will remain will not really be up to the task either, it's a complex work and you have to be careful (I am talking about import queue management) because it's easy to mess stuff up [11:32] danilos: is that an ops thing perhaps? [11:32] lifeless, ideally, we wanted to transfer that to project owners only when we made it very hard to make mistakes, but I think it's better to let owners make mistakes now and be able to fix it to at least some extent than to have owners depend on a team that might make mistakes as well and depend on them to fix them afterwards [11:32] lifeless, "ops" as in operation? yes [11:32] operational, that is [11:32] danilos: I love the idea of being able to fix things rather than obsessing about preventing them [11:32] scales better [11:32] feels easier to use [11:32] lifeless, the problem is that we can't really fix them atm [11:33] lifeless, which is why the permissions are very restricted [11:33] danilos: sure, we have some work to do to get there [11:33]