[00:00] mwhudson: Once the build finishes, buildd-manager should copy it into an upload directory and set the status to UPLOADING. [00:00] i wonder how much faster the chroot would unpack on arm if it was gzipped not bzip2ed [00:00] We then need to run a separate script to upload it. [00:00] ah right, this is the stuff that's changed recent [00:00] ly [00:00] Like 8 hours ago, yes. [00:01] So I haven't run it yet. [00:01] * wgrant fixes that. [00:01] speaking of changes [00:01] I'm now running a couple of ec2 runs with my new, improved ec2 error mail [00:05] email looks something like: http://paste.ubuntu.com/497325/ [00:06] wgrant: it's "Uploading build on Bob The Builder" now [00:06] (this looks like progress) [00:07] mwhudson: Excellent. [00:07] So, now you need to run... [00:08] scripts/process-upload.py ? [00:08] ./scripts/process-upload.py --builds -C buildd -vvv /var/tmp/builddmaster [00:08] I think that should work. [00:08] * mwhudson tries science [00:09] wgrant: http://pastebin.ubuntu.com/497326/ [00:09] mwhudson: I think it worked. [00:10] not quite sure what the bottom two bits are about, probably from the fun i had yesterday [00:10] Except that there were two other uploads from the breakage yesterday. [00:10] Does the build have binaries listed now? === matsubara is now known as matsubara-afk [00:10] wgrant: yes [00:10] lifeless, hello [00:11] process-accepted, publish-distro? [00:11] mwhudson: OK, so it's all working. [00:11] Right. [00:11] lifeless, I'm in an oops tools minisprint with matsubara-afk [00:11] wgrant: if you edit the page, you might want to scatter some -vv s around i think [00:11] Ursinha: hi [00:11] mwhudson: I might indeed. [00:11] Ursinha: I am blocked; I need a hand [00:12] Ursinha: I cannot diagnose why https://devpad.canonical.com/~lpqateam/qa_reports/launchpad-stable-deployment.txt shows 11557 as qa-notok [00:12] although wow scripts/publish-distro.py --ppa -vv is noisy [00:12] Ursinha: I *thought* we had more output than that. [00:12] mwhudson: Most of it was still written on July 22nd last year, so it's not exactly as I'd do it now! [00:12] Ursinha: and I'm wondering where it goes so that I can see it., [00:12] wgrant: :) [00:12] Ursinha: (and the rest of the canonical staff should be able to look at it too) [00:12] lifeless, take a look here: https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html [00:12] and yay, i can download a build [00:13] mwhudson: Is p-a happy? [00:13] Ursinha: that looks old [00:13] wgrant: seems so [00:13] Ursinha: because 11555 is live on production [00:13] lifeless, it's not final, just showing you what must be the report [00:14] Ursinha: let me try a different angle. [00:14] Ursinha: we *had a totally useable debug report* [00:14] Ursinha: what happened to it? [00:14] lifeless, wait, I need context, reading devpad logs [00:15] been in a totally different context the whole day [00:15] Ursinha: ok, lets start over. [00:15] lifeless, just give me a moment [00:15] anyway, see you crazy cats later. [00:15] Ursinha: hi, I've been doing CP's to lpnet based on the qatagger output; and getting folk to QA stuff as needed for ut. [00:15] *it* [00:15] Ursinha: sure; /me hits the pause button [00:15] thanks :) [00:17] wgrant: it seems to have worked [00:18] mwhudson: Great. [00:18] mwhudson: Docs updated. [00:18] With the new process-upload invocation, and liberal application of -v. [00:19] lifeless, when was 11555 deployed? [00:19] lifeless, please, refresh the html report [00:19] Ursinha: this morning [00:19] lifeless, always consider the txt report accurate [00:20] the html one is experimental [00:20] Ursinha: so, I think we want to keep the txt report as it is too - because its safe to show e.g. wgrant [00:20] Ursinha: whereas the html report can potentially report on security bugs [00:20] lifeless, not sure what you mean; txt are for losas, html for developers [00:20] Ursinha: but the html still doesn't address my question [00:20] roughly speaking [00:21] Ursinha: why is 11557 blocked? [00:21] lifeless, I need to finish some tests and will put that in production, so we'll have both [00:21] wgrant: thanks for the help [00:21] Ursinha: it wasn't mentioned by the commit message [00:21] lifeless, linked to the branch [00:22] Ursinha: its an in-progress bug later on the branch [00:22] the problem I told you reusing branches would cause [00:22] fair enough [00:22] mwhudson: np [00:22] there's a bug open to support this better though? [00:22] because neither ec2 or bzr lp-land or the tagger support this use case [00:22] lifeless, of course [00:22] Ursinha: coolio [00:22] librarian downloads are slow :( [00:22] I wonder why. [00:23] Ursinha: could you please tell the txt report to refresh ? [00:23] lifeless, the HTML report says bug 618375 has not been QA'd. the 'qa-notok' status should really be 'qa-needstesting'. However, the bug has no tags? [00:23] <_mup_> Bug #618375: BugTrackerSet:+index consistently slower than 10 seconds [00:23] lifeless, bug 638468 for ec2 land, bug 640566 [00:23] <_mup_> Bug #638468: ec2 land: do not include already fix-committed or fix-released bugs in commit message [00:23] Ursinha: I recognise I'm shooting myself in the foot [00:23] <_mup_> Bug #640566: qa-tagger should not consider linked bugs to branches when bugs are already Fix Released

[00:23] for qa-tagger [00:23] mars: it has no tags because its from later on; its further work. [00:23] yes, hehe [00:23] that's what reusing branches cause [00:23] Ursinha: the big reason to handle reused branches properly is developers shouldn't need to remember if they used the name before. [00:24] but I'm working on a workaround [00:24] Ursinha: we should 'just work' [00:24] Ursinha: cool, and thanks. [00:24] lifeless, that's fine, I'm just saying this case wasn't considered [00:24] yeah [00:24] Ursinha: theres another case I think we missed. [00:24] look at rev 11566 [00:24] and that's why such things are happening [00:24] wgrant: what does [00:24] this was one part of a 2-part branch [00:25] 2010-09-20 23:24:03 WARNING Upload was rejected: [00:25] 2010-09-20 23:24:03 WARNING nominatedarchindep is not present in legal_archseries: armel [00:25] mean? [00:25] oh i think i know [00:25] the upload contains _all packages and i only have a armel builder configured [00:25] What's nominatedarchindep set to? [00:25] lifeless, let me see [00:26] no idea [00:26] it's on distroseries? [00:26] It should be on distroseries:+edit [00:26] Or maybe +admin. [00:26] Set it to armel. [00:26] Ursinha: so this is fixable by a rollback [00:26] Ursinha: -or- by doing a fix [00:26] ah, wasn't the package i wanted to build anyway [00:26] Ursinha: so we need a matching thing like rollback=11566 fixes=11566 [00:27] Ursinha: same logic for the tagger - can only deploy 11566 with the fix-for-11566, but humans would be confused by a rollback=11566 [00:27] Ursinha: lastly, we also need to be able to annotate this things after the fact: commit messags are nice, but we make mistakes. [00:28] lifeless, if we're reusing branches, I need to rely in commit messages [00:28] Ursinha: that seems orthogonal to me. [00:28] for commit messages to be reliable, i.e. only contain valid in progress bugs, people all need to use ec2 land or bzr lp-land [00:29] wgrant: +edit and +admin look basically the same (?) and neither have nominatearchindep [00:29] lifeless, qa-tagger needs to consider something besides linked bugs to branches [00:29] Ursinha: perhaps. We're iterating here, I recognise that. [00:29] Ursinha: now that we have something working - and it is, cause we're deploying :P - we're finding out things that were not obvious up front [00:30] yes :) [00:30] that's nice [00:30] I was expecting that [00:30] I've very excited and happy with what we've achieved so far. [00:31] lifeless, me too [00:31] lifeless, next step is to make html reliable and available, after that fix this problem of reused branches (that's messing with old bugs, argh) [00:31] mwhudson: Maybe you'll need to set it with SQL. [00:32] wgrant: yeah, maybe [00:32] Ursinha: part of the issue there is the many week delay between landing and deploy, as that gap narrows it will be easier. [00:32] I'm boring and just use i386. [00:32] it wasn't the package i wanted to build though, so i don't really care [00:32] agreed [00:32] * mwhudson is building pyton2.6 so he can compare build times with the official builders [00:33] lifeless, what's the problem with 11566? that's an incremental qaable fix, right? [00:33] Ursinha: thanks for interrupting your sprint; I think I'm good now, I know hat to look for for more revs [00:33] so the bug will be qa-needstesting [00:34] lifeless, it's late now, 8h40PM [00:34] I have to take matsubara-afk home :) [00:34] Ursinha: no, the branch is broken [00:34] lifeless, is the bug qa-bad? [00:34] Ursinha: if its rolled out without its second half it will break. [00:34] hm [00:35] I put bad-revision-11566 in the bug tags [00:35] Ursinha: 5 *6* 6 , not 5 *5* 6 [00:35] lifeless, I know [00:35] (I got confused between them yesterday :P) [00:36] lifeless, so it shouldn't be committed without the second part? why only half of the fix was committed? [00:36] lifeless, hehe [00:36] I almost did now :P [00:36] Ursinha: it wasn't meant to be [00:36] Ursinha: it was reviewed in two parts, meant to be landed in one. [00:36] sorry bad-commit-11566 [00:36] lifeless, right, so we should tell developers to don't do that [00:36] hah [00:36] first of all [00:37] yes, I agree its undesirable. [00:37] lifeless, I can't make the script work without some assumptions :) [00:37] or I'll spend 90% of the time trying to teach it how to workaround bad use of the process [00:37] but I understand it has to be considered [00:38] agreed [00:38] so you think using the same mechanism of rollback tag should work [00:38] I agree [00:38] that would be nice exercise of the script, btw [00:38] https://devpad.canonical.com/~lpqateam/qa_reports/launchpad-stable-deployment.txt [00:39] Ursinha: I'd really like to see the other output from the script [00:39] Ursinha: can you get it to go to the same file, or a different file? [00:39] you know the stuff where it says what revs it looked at as it goes and so on. I know its not pretty, but its very helpful. [00:39] what's the other output? the html one? [00:40] Ursinha: no, there were debug messages [00:40] ah [00:40] AH [00:40] I see [00:40] you want the true log [00:40] :) [00:40] yes! [00:40] :) [00:40] debug log, cool [00:40] wallyworld: thumper: https://bugs.edge.launchpad.net/launchpad-code/+bug/404131 needs QA [00:40] <_mup_> Bug #404131: tales linkify create broken links [00:41] lifeless: ack [00:56] lifeless, https://devpad.canonical.com/~lpqateam/qa_reports/qa-tagger-stable.log [00:57] Ursinha: \o/ [00:57] :) [00:57] lifeless, should be there from now on [00:57] Ursinha: thank you! [00:58] lifeless, my pleasure, sir :) [00:58] now when 404131 is qa'd we should have another big batch CP'ing today [00:58] yay [01:00] hi, happy performance tuesday! [01:01] wallyworld: good question! [01:05] istr there's a way i can run process-mail.py feeding it from a test mailbox [01:05] to interactively test gpg/dkim stuff [01:05] is that true? [01:06] lib/canonical/launchpad/doc/mailbox.txt [01:10] thanks [01:58] lifeless: I've received an ec2 failure regarding runJobHandleError and with a Fake exception. Did you work on that last week or so? [02:00] bdmurray: I changed some code in that area [02:00] pastebin ? [02:03] https://pastebin.canonical.com/37437/ [02:03] lifeless, hey [02:04] lifeless, thought about one workaround for 11566 [02:04] lifeless, if we wait for the second branch to land, just leave it as qa-needstesting [02:04] when the second branch lands, it will mention the same bug, and it will be QAable [02:04] as soon as this bug gets then QAed ok, the first revision will be marked as ok to land [02:04] voila [02:04] Ursinha: right, but we need to make sure that nothing lands without both. [02:05] the bugs ensures that [02:05] Shouldn't the first branch just never land? [02:05] *bug [02:05] Ursinha: how so ? [02:05] wgrant, land in qa environment [02:05] not production [02:05] bdmurray: try with https://code.edge.launchpad.net/~lifeless/launchpad/lessGetLastOops/+merge/35994 merged in [02:05] lifeless, the first and the second branches mention the same bug [02:05] If the first branch is going to break things, it should never even hit devel alone, probably. [02:06] Ursinha: yes, but a branch in the middle might be bad [02:06] wgrant, but it's only known as breaking things because it's already in the qa instance [02:06] lifeless, what do you mean? [02:06] Ursinha: In 11566's case it was known that it would break things. [02:06] wgrant, so it shouldn't have landed in first place, but that happened and I don't know how to avoid that to happen again [02:06] Ursinha: 11566=X, 11567=Y,11568=Z [02:07] lets say that Z fixes X [02:07] if we use a bug, as you say, then X and Z can be qa-ok on the same bug [02:07] but Y may be qa-needstesting [02:07] lifeless, that's fine, but the special case of 11566 it's not bad, just incomplete [02:07] if X is qa-ok then the rules say it can be deployed, but it can't, we have to deploy nothing, or X+Y+Z [02:07] both are part of the same bug fix [02:07] Ursinha: its bad unless its complete. [02:07] Ursinha: because it will break the buildmaster. [02:08] lifeless, but it will break anyway, because it will land in edge [02:08] that's the QA instance [02:08] Ursinha: edge doesn't [yet] deploy to the buildmaster [02:08] ah [02:08] but that's not my point [02:08] for qastaging, it would break yes. [02:08] that's the point [02:08] or whatever tom is going to call the domain. [02:09] as long as we do exercise soyuz there. [02:09] thing I want to say is that's not bad, but qa-needstesting that cannot be tested without the second part [02:09] or hm well [02:10] that's bad because it landed in the first place, so it should be rolled back... [02:10] yes, but what if the rollback just contains the fix. [02:10] lifeless, in this special case the fix is the other part, right? [02:10] from the deployment story its identical. X on its own = boom, X+Z together = ok. [02:10] Ursinha: yes. [02:10] so keeping the branch qa-needstesting would block it from deployment, right? [02:10] branch no, bug [02:10] as will bad-commit-xxxxxx [02:11] but why use that [02:11] which I think is accurate. [02:11] because it is a bad commit, all commits are -meant- to keep trunk stable [02:11] it's because it's not just a partial fix, but it's breaking stuff? [02:11] right [02:11] I see [02:11] its not incremental [02:11] its damaged without its soul mate [02:12] it's bad [02:12] nothing much special about it, just bad [02:12] break stuff, should not be there [02:12] I'm proposing a pqm commit messag elike fixcommit=12345 to handle roll-forward rathr than rollback [02:12] Ursinha: yes, it shouldn't have happened, but we need to deal when it does [02:13] the only difference fixcommit and rollback would have is: [02:13] lifeless, I'm thinking out loud to check if I make sense [02:13] rollback tags in-progress again, fixcommit doesn't change the bug 'status' field. [02:13] hm [02:15] so we'd need to say that it not just is a rollback but a fix itself [02:15] yes [02:23] mwhudson: How's python2.6 going? [02:25] wgrant: it seems to be running the tests [02:25] Aha. [02:25] https://edge.launchpad.net/ubuntu/+source/python2.6/2.6.6-5ubuntu1/+build/1961090 -> (took 3 hours, 35 minutes, 1.1 seconds) [02:25] been going for close to 2 on my board, so a while yet, i expect [02:28] Hm, the xM looks rather nice. [02:28] wgrant: o [02:28] bah [02:29] wgrant: i've been blogging about it! http://voices.canonical.com/michael.hudson/ [02:31] I may have to acquire one. My previous ARM board is now acting as the home servery/routery thing, so I need something to experiment on. [02:45] anyone know the status of 'mentoring' [02:46] lifeless: i think it's "kill code on sight", not 100% sure though [02:47] "Under trial withdrawal" is the official status. [02:47] Removed from the UI and condemned, but code not yet deleted. [02:49] Although the reference to it in the feature listing seems to have now been entirely removed. [02:53] and it causes extra queries. [02:53] le sigh [02:54] thumper: I think its better to do this via a garbo patch: [02:54] - gets rid of the interrupt on spm [02:54] - long term solution [02:54] lifeless: yes, so you have said multiple times already [02:54] - same basic code as we'd be cowboying, but tested [02:54] and it will be [02:55] thumper: I've been thinking more about it [02:55] thumper: that was summary; nuff said. [02:55] oh, and man echannel too. [02:55] * lifeless hides in shame [02:56] chuckles [03:31] * thumper code dives [03:32] zomg [03:32] the celebrity code queries every time ... [03:33] I thought it cached IDs. [03:33] That's how you can get MutatedCelebrityErrors. [03:33] now when you do ).admin [03:34] or perhaps my test is just catching the cache loading [03:34] still [03:34] ECRY [03:34] It will have to load the object if it's not in the cache. [03:34] But I believe it's meant to cache the ID over transactions. [03:53] lifeless, https://devpad.canonical.com/~lpqateam/qa_reports/logs/ [03:53] I've changed them all to a separate folder [03:58] thanks for letting me know [04:00] I wonder how badly LP breaks on pg 9.0. [04:09] mwhudson: LaunchBag is spectacularly ugly [04:09] lifeless: yes [04:28] silly question [04:28] is it ok to have __init__ on SQLBase classes [04:28] or .. ah _init? [04:35] I'm done for today === Ursinha is now known as Ursinha-zzz [04:40] lifeless, could you please file a bug for the fixes-xxxxx problem? [04:43] Ursinha-zzz: go sleep [04:43] :) [04:51] * thumper goes offline briefly while IRC proxy goes to maverick [04:55] I'm making progress [04:55] viewing a single bug page with no subscribers is now down to 99 queries. [04:56] ! [04:56] from where? [04:56] 110 when I started this branch [04:56] lifeless: How many comments/tasks? [04:56] wgrant: 1/1 [04:57] ..... ew. [04:57] wgrant: I took my view scaling test (14 queries), made it render... [04:57] and blinked. [05:00] i was thinking this morning about the idea of giving sql planner logs back to the client [05:00] and it occurred to me that giving them to the web app (to put in an oops) would be nice but isn't strictly necessary [05:00] *if* it turns out that logging them is not noticeably expensive, perhaps we could log them all the time and just make them available on devpad [05:01] we have detailed logging off on the master due to load. [05:02] it does show up? [05:02] Have I done the numbers myself ? no. Ask stub :P [05:02] I presume its turned off because someone measured. [05:02] i believe you, i'm just curious [05:03] just thought it would be nice to cut out the roundtrips to get this [05:03] anyhow, I'd expect query plan serialisation to be similar overhead: totally fine from time to time but significant if on all the time [05:03] poolie: personally, ignoring performance, I'd rather have it in the oops. [05:03] nicely interleaved with the queries. [05:03] rather than in a separate file i have to go read [05:03] i agree [05:03] i was just thinking it would be less work to record it to a log [05:04] have a look at requesttimeline [05:04] its nice [05:04] shove it in there, it turns up in the oops, bada boom bada bing [05:04] where is it? [05:04] lp.services.timeline [05:06] ok, thanks [05:06] hmm, we query the tags for the js fragment twice... sigh [05:07] no, something else [05:13] wgrant: if you're free could you do a security-oriented review of https://code.edge.launchpad.net/~mbp/launchpad/dkim/+merge/35985 [05:14] nah, he charges ;P [05:15] lifeless: could you please land https://code.edge.launchpad.net/~mbp/launchpad/flags-webapp/+merge/32967 for me [05:15] Let's see. [05:18] + canonicalise_line_endings(mail.signedContent), signature) [05:18] WTF [05:18] mixedCase attributes? [05:18] Must be old code. [05:18] not my fault [05:18] I know. [05:18] Makes me cringe anyway :) [05:19] what is the standard meant to be? underscored attributes, camelcase methods? [05:19] Right. [05:19] PEP-8 with a pinch of Zope. [05:20] urk [05:20] biab [05:27] is bugtask.distribution always set when bugtask.sourcepackagename is ? [05:27] hmm [05:27] I hope so. [05:28] actually [05:28] it never is [05:28] * lifeless suspect official bug tag adaption mojo [05:28] Er. [05:29] What? [05:29] sourcepackagename on its own is useless. [05:29] distribution and distroseries should be mutually exclusive. [05:29] But sourcepackagename should always have one of those two, although it's not enforced by the DB. [05:29]