wgrantmwhudson: Once the build finishes, buildd-manager should copy it into an upload directory and set the status to UPLOADING.00:00
mwhudsoni wonder how much faster the chroot would unpack on arm if it was gzipped not bzip2ed00:00
wgrantWe then need to run a separate script to upload it.00:00
mwhudsonah right, this is the stuff that's changed recent00:00
wgrantLike 8 hours ago, yes.00:00
wgrantSo I haven't run it yet.00:01
* wgrant fixes that.00:01
jmlspeaking of changes00:01
jmlI'm now running a couple of ec2 runs with my new, improved ec2 error mail00:01
jmlemail looks something like: http://paste.ubuntu.com/497325/00:05
mwhudsonwgrant: it's "Uploading build  on Bob The Builder" now00:06
mwhudson(this looks like progress)00:06
wgrantmwhudson: Excellent.00:07
wgrantSo, now you need to run...00:07
mwhudsonscripts/process-upload.py ?00:08
wgrant./scripts/process-upload.py --builds -C buildd -vvv /var/tmp/builddmaster00:08
wgrantI think that should work.00:08
* mwhudson tries science00:08
mwhudsonwgrant: http://pastebin.ubuntu.com/497326/00:09
wgrantmwhudson: I think it worked.00:09
mwhudsonnot quite sure what the bottom two bits are about, probably from the fun i had yesterday00:10
wgrantExcept that there were two other uploads from the breakage yesterday.00:10
wgrantDoes the build have binaries listed now?00:10
=== matsubara is now known as matsubara-afk
mwhudsonwgrant: yes00:10
Ursinhalifeless, hello00:10
mwhudsonprocess-accepted, publish-distro?00:11
wgrantmwhudson: OK, so it's all working.00:11
Ursinhalifeless, I'm in an oops tools minisprint with matsubara-afk00:11
mwhudsonwgrant: if you edit the page, you might want to scatter some -vv s around i think00:11
lifelessUrsinha: hi00:11
wgrantmwhudson: I might indeed.00:11
lifelessUrsinha: I am blocked; I need a hand00:11
lifelessUrsinha: I cannot diagnose why https://devpad.canonical.com/~lpqateam/qa_reports/launchpad-stable-deployment.txt shows 11557 as qa-notok00:12
mwhudsonalthough wow scripts/publish-distro.py --ppa -vv is noisy00:12
lifelessUrsinha: I *thought* we had more output than that.00:12
wgrantmwhudson: Most of it was still written on July 22nd last year, so it's not exactly as I'd do it now!00:12
lifelessUrsinha: and I'm wondering where it goes so that I can see it.,00:12
mwhudsonwgrant: :)00:12
lifelessUrsinha: (and the rest of the canonical staff should be able to look at it too)00:12
Ursinhalifeless, take a look here: https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html00:12
mwhudsonand yay, i can download a build00:12
wgrantmwhudson: Is p-a happy?00:13
lifelessUrsinha: that looks old00:13
mwhudsonwgrant: seems so00:13
lifelessUrsinha: because 11555 is live on production00:13
Ursinhalifeless, it's not final, just showing you what must be the report00:13
lifelessUrsinha: let me try a different angle.00:14
lifelessUrsinha: we *had a totally useable debug report*00:14
lifelessUrsinha: what happened to it?00:14
Ursinhalifeless, wait, I need context, reading devpad logs00:14
Ursinhabeen in a totally different context the whole day00:15
lifelessUrsinha: ok, lets start over.00:15
Ursinhalifeless, just give me a moment00:15
jmlanyway, see you crazy cats later.00:15
lifelessUrsinha: hi, I've been doing CP's to lpnet based on the qatagger output; and getting folk to QA stuff as needed for ut.00:15
lifelessUrsinha: sure; /me hits the pause button00:15
Ursinhathanks :)00:15
mwhudsonwgrant: it seems to have worked00:17
wgrantmwhudson: Great.00:18
wgrantmwhudson: Docs updated.00:18
wgrantWith the new process-upload invocation, and liberal application of -v.00:18
Ursinhalifeless, when was 11555 deployed?00:19
Ursinhalifeless, please, refresh the html report00:19
lifelessUrsinha: this morning00:19
Ursinhalifeless, always consider the txt report accurate00:19
Ursinhathe html one is experimental00:20
lifelessUrsinha: so, I think we want to keep the txt report as it is too -  because its safe to show e.g. wgrant00:20
lifelessUrsinha: whereas the html report can potentially report on security bugs00:20
Ursinhalifeless, not sure what you mean; txt are for losas, html for developers00:20
lifelessUrsinha: but the html still doesn't address my question00:20
Ursinharoughly speaking00:20
lifelessUrsinha: why is 11557 blocked?00:21
Ursinhalifeless, I need to finish some tests and will put that in production, so we'll have both00:21
mwhudsonwgrant: thanks for the help00:21
lifelessUrsinha: it wasn't mentioned by the commit message00:21
Ursinhalifeless, linked to the branch00:21
lifelessUrsinha: its an in-progress bug later on the branch00:22
Ursinhathe problem I told you reusing branches would cause00:22
lifelessfair enough00:22
wgrantmwhudson: np00:22
lifelessthere's a bug open to support this better though?00:22
Ursinhabecause neither ec2 or bzr lp-land or the tagger support this use case00:22
Ursinhalifeless, of course00:22
lifelessUrsinha: coolio00:22
wgrantlibrarian downloads are slow :(00:22
wgrantI wonder why.00:22
lifelessUrsinha: could you please tell the txt report to refresh ?00:23
marslifeless, the HTML report says bug 618375 has not been QA'd.  the 'qa-notok' status should really be 'qa-needstesting'.  However, the bug has no tags?00:23
_mup_Bug #618375: BugTrackerSet:+index consistently slower than 10 seconds <qa-ok> <timeout> <Launchpad Bugs:Fix Committed by lifeless> <https://launchpad.net/bugs/618375>00:23
Ursinhalifeless, bug 638468 for ec2 land, bug 64056600:23
_mup_Bug #638468: ec2 land: do not include already fix-committed or fix-released bugs in commit message <new-merge-workflow> <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/638468>00:23
lifelessUrsinha: I recognise I'm shooting myself in the foot00:23
_mup_Bug #640566: qa-tagger should not consider linked bugs to branches when bugs are already Fix Released <new-merge-workflow> <qa-tagger:Triaged by ursinha> <https://launchpad.net/bugs/640566>00:23
Ursinhafor qa-tagger00:23
lifelessmars: it has no tags because its from later on; its further work.00:23
Ursinhayes, hehe00:23
Ursinhathat's what reusing branches cause00:23
lifelessUrsinha: the big reason to handle reused branches properly is developers shouldn't need to remember if they used the name before.00:23
Ursinhabut I'm working on a workaround00:24
lifelessUrsinha: we should 'just work'00:24
lifelessUrsinha: cool, and thanks.00:24
Ursinhalifeless, that's fine, I'm just saying this case wasn't considered00:24
lifelessUrsinha: theres another case I think we missed.00:24
lifelesslook at rev 1156600:24
Ursinhaand that's why such things are happening00:24
mwhudsonwgrant: what does00:24
lifelessthis was one part of a 2-part branch00:24
mwhudson2010-09-20 23:24:03 WARNING Upload was rejected:00:25
mwhudson2010-09-20 23:24:03 WARNING nominatedarchindep is not present in legal_archseries: armel00:25
mwhudsonoh i think i know00:25
mwhudsonthe upload contains _all packages and i only have a armel builder configured00:25
wgrantWhat's nominatedarchindep set to?00:25
Ursinhalifeless, let me see00:25
mwhudsonno idea00:26
mwhudsonit's on distroseries?00:26
wgrantIt should be on distroseries:+edit00:26
wgrantOr maybe +admin.00:26
wgrantSet it to armel.00:26
lifelessUrsinha: so this is fixable by a rollback00:26
lifelessUrsinha: -or- by doing a fix00:26
mwhudsonah, wasn't the package i wanted to build anyway00:26
lifelessUrsinha: so we need a matching thing like rollback=11566 fixes=1156600:26
lifelessUrsinha: same logic for the tagger - can only deploy 11566 with the fix-for-11566, but humans would be confused by a rollback=1156600:27
lifelessUrsinha: lastly, we also need to be able to annotate this things after the fact: commit messags are nice, but we make mistakes.00:27
Ursinhalifeless, if we're reusing branches, I need to rely in commit messages00:28
lifelessUrsinha: that seems orthogonal to me.00:28
Ursinhafor commit messages to be reliable, i.e. only contain valid in progress bugs, people all need to use ec2 land or bzr lp-land00:28
mwhudsonwgrant: +edit and +admin look basically the same (?) and neither have nominatearchindep00:29
Ursinhalifeless, qa-tagger needs to consider something besides linked bugs to branches00:29
lifelessUrsinha: perhaps. We're iterating here, I recognise that.00:29
lifelessUrsinha: now that we have something working - and it is, cause we're deploying :P - we're finding out things that were not obvious up front00:29
Ursinhayes :)00:30
Ursinhathat's nice00:30
UrsinhaI was expecting that00:30
lifelessI've very excited and happy with what we've achieved so far.00:30
Ursinhalifeless, me too00:31
Ursinhalifeless, next step is to make html reliable and available, after that fix this problem of reused branches (that's messing with old bugs, argh)00:31
wgrantmwhudson: Maybe you'll need to set it with SQL.00:31
mwhudsonwgrant: yeah, maybe00:32
lifelessUrsinha: part of the issue there is the many week delay between landing and deploy, as that gap narrows it will be easier.00:32
wgrantI'm boring and just use i386.00:32
mwhudsonit wasn't the package i wanted to build though, so i don't really care00:32
* mwhudson is building pyton2.6 so he can compare build times with the official builders00:32
Ursinhalifeless, what's the problem with 11566? that's an incremental qaable fix, right?00:33
lifelessUrsinha: thanks for interrupting your sprint; I think I'm good now, I know hat to look for for more revs00:33
Ursinhaso the bug will be qa-needstesting00:33
Ursinhalifeless, it's late now, 8h40PM00:34
UrsinhaI have to take matsubara-afk home :)00:34
lifelessUrsinha: no, the branch is broken00:34
Ursinhalifeless, is the bug qa-bad?00:34
lifelessUrsinha: if its rolled out without its second half it will break.00:34
lifelessI put bad-revision-11566 in the bug tags00:35
lifelessUrsinha: 5 *6* 6 , not 5 *5* 600:35
Ursinhalifeless, I know00:35
lifeless(I got confused between them yesterday :P)00:35
Ursinhalifeless, so it shouldn't be committed without the second part? why only half of the fix was committed?00:36
Ursinhalifeless, hehe00:36
UrsinhaI almost did now :P00:36
lifelessUrsinha: it wasn't meant to be00:36
lifelessUrsinha: it was reviewed in two parts, meant to be landed in one.00:36
lifelesssorry bad-commit-1156600:36
Ursinhalifeless, right, so we should tell developers to don't do that00:36
Ursinhafirst of all00:36
lifelessyes, I agree its undesirable.00:37
Ursinhalifeless, I can't make the script work without some assumptions :)00:37
Ursinhaor I'll spend 90% of the time trying to teach it how to workaround bad use of the process00:37
Ursinhabut I understand it has to be considered00:37
Ursinhaso you think using the same mechanism of rollback tag should work00:38
UrsinhaI agree00:38
Ursinhathat would be nice exercise of the script, btw00:38
lifelessUrsinha: I'd really like to see the other output from the script00:39
lifelessUrsinha: can you get it to go to the same file, or a different file?00:39
lifelessyou know the stuff where it says what revs it looked at as it goes and so on. I know its not pretty, but its very helpful.00:39
Ursinhawhat's the other output? the html one?00:39
lifelessUrsinha: no, there were debug messages00:40
UrsinhaI see00:40
Ursinhayou want the true log00:40
Ursinhadebug log, cool00:40
lifelesswallyworld: thumper: https://bugs.edge.launchpad.net/launchpad-code/+bug/404131 needs QA00:40
_mup_Bug #404131: tales linkify create broken links  <qa-needstesting> <trivial> <ui> <Launchpad Bazaar Integration:Fix Committed by wallyworld> <https://launchpad.net/bugs/404131>00:40
wallyworldlifeless: ack00:41
Ursinhalifeless, https://devpad.canonical.com/~lpqateam/qa_reports/qa-tagger-stable.log00:56
lifelessUrsinha: \o/00:57
Ursinhalifeless, should be there from now on00:57
lifelessUrsinha: thank you!00:57
Ursinhalifeless, my pleasure, sir :)00:58
lifelessnow when 404131 is qa'd we should have another big batch CP'ing today00:58
pooliehi, happy performance tuesday!01:00
pooliewallyworld: good question!01:01
poolieistr there's a way i can run process-mail.py feeding it from a test mailbox01:05
poolieto interactively test gpg/dkim stuff01:05
poolieis that true?01:05
bdmurraylifeless: I've received an ec2 failure regarding runJobHandleError and with a Fake exception.  Did you work on that last week or so?01:58
lifelessbdmurray: I changed some code in that area02:00
lifelesspastebin ?02:00
Ursinhalifeless, hey02:03
Ursinhalifeless, thought about one workaround for 1156602:04
Ursinhalifeless, if we wait for the second branch to land, just leave it as qa-needstesting02:04
Ursinhawhen the second branch lands, it will mention the same bug, and it will be QAable02:04
Ursinhaas soon as this bug gets then QAed ok, the first revision will be marked as ok to land02:04
lifelessUrsinha: right, but we need to make sure that nothing lands without both.02:04
Ursinhathe bugs ensures that02:05
wgrantShouldn't the first branch just never land?02:05
lifelessUrsinha: how so ?02:05
Ursinhawgrant, land in qa environment02:05
Ursinhanot production02:05
lifelessbdmurray: try with https://code.edge.launchpad.net/~lifeless/launchpad/lessGetLastOops/+merge/35994 merged in02:05
Ursinhalifeless, the first and the second branches mention the same bug02:05
wgrantIf the first branch is going to break things, it should never even hit devel alone, probably.02:05
lifelessUrsinha: yes, but a branch in the middle might be bad02:06
Ursinhawgrant, but it's only known as breaking things because it's already in the qa instance02:06
Ursinhalifeless, what do you mean?02:06
wgrantUrsinha: In 11566's case it was known that it would break things.02:06
Ursinhawgrant, so it shouldn't have landed in first place, but that happened and I don't know how to avoid that to happen again02:06
lifelessUrsinha: 11566=X, 11567=Y,11568=Z02:06
lifelesslets say that Z fixes X02:07
lifelessif we use a bug, as you say, then X and Z can be qa-ok on the same bug02:07
lifelessbut Y may be qa-needstesting02:07
Ursinhalifeless, that's fine, but the special case of 11566 it's not bad, just incomplete02:07
lifelessif X is qa-ok then the rules say it can be deployed, but it can't, we have to deploy nothing, or X+Y+Z02:07
Ursinhaboth are part of the same bug fix02:07
lifelessUrsinha: its bad unless its complete.02:07
lifelessUrsinha: because it will break the buildmaster.02:07
Ursinhalifeless, but it will break anyway, because it will land in edge02:08
Ursinhathat's the QA instance02:08
lifelessUrsinha: edge doesn't [yet] deploy to the buildmaster02:08
Ursinhabut that's not my point02:08
lifelessfor qastaging, it would break yes.02:08
Ursinhathat's the point02:08
lifelessor whatever tom is going to call the domain.02:08
lifelessas long as we do exercise soyuz there.02:09
Ursinhathing I want to say is that's not bad, but qa-needstesting that cannot be tested without the second part02:09
Ursinhaor hm well02:09
Ursinhathat's bad because it landed in the first place, so it should be rolled back...02:10
lifelessyes, but what if the rollback just contains the fix.02:10
Ursinhalifeless, in this special case the fix is the other part, right?02:10
lifelessfrom the deployment story its identical. X on its own = boom, X+Z together = ok.02:10
lifelessUrsinha: yes.02:10
Ursinhaso keeping the branch qa-needstesting would block it from deployment, right?02:10
Ursinhabranch no, bug02:10
lifelessas will bad-commit-xxxxxx02:10
Ursinhabut why use that02:11
lifelesswhich I think is accurate.02:11
lifelessbecause it is a bad commit, all commits are -meant- to keep trunk stable02:11
Ursinhait's because it's not just a partial fix, but it's breaking stuff?02:11
UrsinhaI see02:11
lifelessits not incremental02:11
lifelessits damaged without its soul mate02:11
Ursinhait's bad02:12
Ursinhanothing much special about it, just bad02:12
Ursinhabreak stuff, should not be there02:12
lifelessI'm proposing a pqm commit messag elike fixcommit=12345 to handle roll-forward rathr than rollback02:12
lifelessUrsinha: yes, it shouldn't have happened, but we need to deal when it does02:12
lifelessthe only difference fixcommit and rollback would have is:02:13
Ursinhalifeless, I'm thinking out loud to check if I make sense02:13
lifelessrollback tags in-progress again, fixcommit doesn't change the bug 'status' field.02:13
Ursinhaso we'd need to say that it not just is a rollback but a fix itself02:15
wgrantmwhudson: How's python2.6 going?02:23
mwhudsonwgrant: it seems to be running the tests02:25
mwhudsonhttps://edge.launchpad.net/ubuntu/+source/python2.6/2.6.6-5ubuntu1/+build/1961090 -> (took 3 hours, 35 minutes, 1.1 seconds)02:25
mwhudsonbeen going for close to 2 on my board, so a while yet, i expect02:25
wgrantHm, the xM looks rather nice.02:28
mwhudsonwgrant: o02:28
mwhudsonwgrant: i've been blogging about it! http://voices.canonical.com/michael.hudson/02:29
wgrantI may have to acquire one. My previous ARM board is now acting as the home servery/routery thing, so I need something to experiment on.02:31
lifelessanyone know the status of 'mentoring'02:45
mwhudsonlifeless: i think it's "kill code on sight", not 100% sure though02:46
wgrant"Under trial withdrawal" is the official status.02:47
wgrantRemoved from the UI and condemned, but code not yet deleted.02:47
wgrantAlthough the reference to it in the feature listing seems to have now been entirely removed.02:49
lifelessand it causes extra queries.02:53
lifelessle sigh02:53
lifelessthumper: I think its better to do this via a garbo patch:02:54
lifeless - gets rid of the interrupt on spm02:54
lifeless - long term solution02:54
thumperlifeless: yes, so you have said multiple times already02:54
lifeless - same basic code as we'd be cowboying, but tested02:54
thumperand it will be02:54
lifelessthumper: I've been thinking more about it02:55
lifelessthumper: that was summary; nuff said.02:55
lifelessoh, and man echannel too.02:55
* lifeless hides in shame02:55
* thumper code dives03:31
lifelessthe celebrity code queries every time ...03:32
wgrantI thought it cached IDs.03:33
wgrantThat's how you can get MutatedCelebrityErrors.03:33
lifelessnow when you do ).admin03:33
lifelessor perhaps my test is just catching the cache loading03:34
wgrantIt will have to load the object if it's not in the cache.03:34
wgrantBut I believe it's meant to cache the ID over transactions.03:34
Ursinhalifeless, https://devpad.canonical.com/~lpqateam/qa_reports/logs/03:53
UrsinhaI've changed them all to a separate folder03:53
lifelessthanks for letting me know03:58
wgrantI wonder how badly LP breaks on pg 9.0.04:00
lifelessmwhudson: LaunchBag is spectacularly ugly04:09
mwhudsonlifeless: yes04:09
lifelesssilly question04:28
lifelessis it ok to have __init__ on SQLBase classes04:28
lifelessor .. ah _init?04:28
UrsinhaI'm done for today04:35
=== Ursinha is now known as Ursinha-zzz
Ursinha-zzzlifeless, could you please file a bug for the fixes-xxxxx problem?04:40
lifelessUrsinha-zzz: go sleep04:43
* thumper goes offline briefly while IRC proxy goes to maverick04:51
lifelessI'm making progress04:55
lifelessviewing a single bug page with no subscribers is now down to 99 queries.04:55
pooliefrom where?04:56
lifeless110 when I started this branch04:56
wgrantlifeless: How many comments/tasks?04:56
lifelesswgrant: 1/104:56
wgrant..... ew.04:57
lifelesswgrant: I took my view scaling test (14 queries), made it render...04:57
lifelessand blinked.04:57
pooliei was thinking this morning about the idea of giving sql planner logs back to the client05:00
poolieand it occurred to me that giving them to the web app (to put in an oops) would be nice but isn't strictly necessary05:00
poolie*if* it turns out that logging them is not noticeably expensive, perhaps we could log them all the time and just make them available on devpad05:00
lifelesswe have detailed logging off on the master due to load.05:01
poolieit does show up?05:02
lifelessHave I done the numbers myself ? no. Ask stub :P05:02
lifelessI presume its turned off because someone measured.05:02
pooliei believe you, i'm just curious05:02
pooliejust thought it would be nice to cut out the roundtrips to get this05:03
lifelessanyhow, I'd expect query plan serialisation to be similar overhead: totally fine from time to time but significant if on all the time05:03
lifelesspoolie: personally, ignoring performance, I'd rather have it in the oops.05:03
lifelessnicely interleaved with the queries.05:03
lifelessrather than in a separate file i have to go read05:03
pooliei agree05:03
pooliei was just thinking it would be less work to record it to a log05:03
lifelesshave a look at requesttimeline05:04
lifelessits nice05:04
lifelessshove it in there, it turns up in the oops, bada boom bada bing05:04
pooliewhere is it?05:04
poolieok, thanks05:06
lifelesshmm, we query the tags for the js fragment twice... sigh05:06
lifelessno, something else05:07
pooliewgrant: if you're free could you do a security-oriented review of https://code.edge.launchpad.net/~mbp/launchpad/dkim/+merge/3598505:13
lifelessnah, he charges ;P05:14
poolielifeless: could you please land https://code.edge.launchpad.net/~mbp/launchpad/flags-webapp/+merge/32967 for me05:15
wgrantLet's see.05:15
wgrant+            canonicalise_line_endings(mail.signedContent), signature)05:18
wgrantmixedCase attributes?05:18
wgrantMust be old code.05:18
poolienot my fault05:18
wgrantI know.05:18
wgrantMakes me cringe anyway :)05:18
pooliewhat is the standard meant to be? underscored attributes, camelcase methods?05:19
wgrantPEP-8 with a pinch of Zope.05:19
lifelessis bugtask.distribution always set when bugtask.sourcepackagename is ?05:27
wgrantI hope so.05:27
lifelessit never is05:28
* lifeless suspect official bug tag adaption mojo05:28
wgrantsourcepackagename on its own is useless.05:29
wgrantdistribution and distroseries should be mutually exclusive.05:29
wgrantBut sourcepackagename should always have one of those two, although it's not enforced by the DB.05:29
lifelessit is05:29
lifelessCONSTRAINT bugtask_assignment_checks05:29
wgrantAh, true.05:30
thumperAssertionError: not equal:05:43
thumpera = set([u'revision-id319180', u'revision-id319182', u'revision-id319184', u'revision-id319186', u'revision-id319188', u'revision-id319198', u'revision-id319196', u'revision-id319194', u'revision-id319192', u'revision-id319190'])05:43
thumperb = set([u'revision-id319180', u'revision-id319182', u'revision-id319184', u'revision-id319186', u'revision-id319188', u'revision-id319198', u'revision-id319196', u'revision-id319194', u'revision-id319192', u'revision-id319190'])05:43
thumperis it me or is that crackful?05:43
lifelessare those actually strings or storm objects with pretty reprs?05:44
thumperlifeless: good point05:45
* thumper double-checks05:45
thumperit says unicode05:46
wgrantIt's not assertIs?05:47
lifelesswgrant: != - assertEqual in testtools prints out like that05:47
lifelessthumper: yes, thats cracked.05:48
thumpera == b in the python interpreter05:48
stubSo the repr is bogus05:48
mwhudsonwoo python2.6 built on the beagle xm05:49
mwhudsontook 4 hours, 53 minutes, 23.1 seconds05:49
mwhudsonso a bit slower than the official buildd05:49
wgrantSo not too slow.05:49
mwhudsonnot too shabby though05:49
mwhudsongiven that you can buy xm's by the crate now05:50
wgrantIs the buildd hardware publicly available yet?05:50
wgrantmwhudson: They're actually on backorder pretty much everywhere.05:50
mwhudsonyou mean, the details of what it is?05:50
wgrantBut yes.05:50
wgrantNo, the actual devices.05:50
wgrantFor a while the buildds were unreleased hardware.05:50
thumperlifeless: http://pastebin.ubuntu.com/497480/05:50
thumperlifeless: I'm in the pdb session05:50
mwhudsonwgrant: ok, well linaro has about 40 or something now05:50
mwhudsonmaybe not that many05:51
mwhudsonbut >10 anyway05:51
mwhudsonwgrant: eh, the buildds are some kind of fancy dev board i think, i'm not sure that it's ever going to be publicly available in the sense of being able to just pony up a credit card number and buy one05:52
mwhudsonicbw though05:52
mwhudsonthumper: wth!05:53
mwhudson(Pdb) print list(expected) == list(observed)05:53
mwhudson(Pdb) expected == observed05:53
mwhudsonthumper: types of the set classes?05:53
mwhudsonprint type(expected) == type(observed)05:53
lifelessone of these is not like the other05:53
wgrantMaybe it's security proxied.05:54
wgrantBut that should normally be fine.05:54
wgrantUnless set has some horrible optimisation.05:54
* wgrant throws Django's URL system into the "cruel joke" bin.05:54
mwhudsonif observed is proxied, i bet observed == expected would be true05:55
thumper(Pdb) print type(expected) == type(observed)05:55
lifelessprint type(expected), type(observed)05:56
wgrantAre they both actually sets?05:56
mwhudsonstarting to think weird ass bugs now05:56
thumperfucking proxies05:56
wgrantThey're both proxied?05:56
* lifeless chalks another one up to 'lets remove the proxies'05:56
stubThat shouldn't mess with == :-/05:57
thumperyes, both proxies05:57
thumperstub: they are both proxied sets05:57
stubYes, but the wrappers are supposed to be transparent to pretty much everything (everything but 'is' / 'is not' IIRC)05:58
thumperstub: probably the testtools implementation of assertEquals05:59
lifelessrockstar: you forgot the link05:59
wgrantNo, == says they're not equal.05:59
wgrantIt's nothing to do with testtools.06:00
thumperwgrant: good point06:01
wgrant    def __eq__(self, other):06:02
wgrant        if isinstance(other, BaseSet):06:02
wgrant            return self._data == other._data06:02
wgrant        else:06:02
wgrant            return False06:02
wgrantBut surely the proxies are smarter than that.06:03
rockstarlifeless, ?06:04
mwhudsonwgrant: return **False** !06:05
mwhudsonsurely that should be NotImplemented06:05
mwhudsonunless comparison is different06:05
wgrantHm. Except that's sets.Set, which is not set.06:05
wgrantNow, where is set...06:05
mwhudsonwgrant: Modules/setsmodule.c06:06
rockstarlifeless, ah, I see.06:06
* rockstar _should_ go to bed.06:06
wgrantI was looking at setobject06:06
wgrantmwhudson: I have no setsmodule.c06:06
mwhudsonoh right06:07
mwhudsonyes setobject06:07
lifelessso, if I've added an attribute to the Interface06:09
lifelesshow do I specify access permissions for it ?06:09
thumperlifeless: it depends06:09
lifelessForbiddenAttribute: ('official_tags', <Bug at 0xef872d0>)06:09
thumperlifeless: on how the interface itself has the permissions set06:09
wgrantmwhudson: Looks like it has the same bug as the Python one.06:09
lifelessofficial_tags = Attribute("The official bug tags relevant to this bug.")06:10
wgrantIt does the type check and returns false, rather than NotImplemented like makes sense, and others seems to do.06:10
wgrantlifeless: Bug doesn't use interfaces for security.06:12
wgrantIt lists the attributes in ZCML.06:12
thumperlifeless: lib/lp/bugs/configure.zcml line 574ish06:12
thumperlifeless: also that file has lots of references to canonical.launchpad.database.*06:12
thumperlifeless: which it shouldn't06:12
lifelessthats so bong06:13
wgrantsinzui: Why are maps dying?06:13
* thumper rings lifeless's bong06:13
thumperwgrant: cost is one reason06:13
lifelesswgrant: they want lots of money to serve on ssl06:13
thumperwgrant: and they changed the api06:13
lifelessthumper: to be fair, thats orthogonal06:13
lifelessthe primary problem was our account expired.06:14
thumperyeah, it is06:14
lifelessTypeError: [] is not JSON serializable06:15
* lifeless is not sure if a practical joke is being played or not06:15
lifelessI bet I know06:16
lifelessthe list is security proxied06:16
lifelessso the browser code which was doing a redundant sorted(list( <- and yes thats also self redundant06:16
lifelesswas hiding that06:16
wgrantIs this still BugTask:+index?06:16
lifelesswill save, about 40 queries on bug 106:16
_mup_Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:In Progress by sabdfl> <06:17
lifelessits got waay to many bugtasks.06:17
pooliewgrant: any other comments on that patch?06:17
wgrantpoolie: That getUtility(ILaunchpadBag).to_user looks like a sed gone wrong.06:17
lifelessyay, finally, squashed that one.06:17
mwhudsonlifeless: work on deleting a bug task next? :-)06:17
pooliei could make it more testable by tearing apart the methods more06:17
lifelessmwhudson: nah, I'm going to defeat this foul beast.06:18
pooliebut i anticipate an uncomfortable interval where the patch gets bigger but the tests don't get safer06:18
lifelessof course, the patch is going to be a) ugly and b) insufficiently tested.06:18
wgrantpoolie: Apart from that it looks OK. But the code needs refactoring.06:18
lifelessIt has the only test I care about right now: query scaling :)06:18
poolieah you're right about the '.to_user' well spotted06:19
pooliei think my next refactoring would be to separate "determine who this comes from and how strongly they're trusted" from "become that user"06:20
pooliecleaner to test that way i think06:20
wgrantpoolie: That suggests a missing test.06:21
wgrantOr you just didn't find it.06:21
poolieit does, doesn't it06:21
lifelessis there a debug mode to tal?06:21
wgrantI wonder if the LP favicon should be that of the current context.06:21
pooliei didn't run everything but i ran 'bin/test mail' which you'd think would catch it06:21
lifelessthat is, to print the thing being evaluated?06:21
lifelessor similar06:21
pooliei think it is a missing test and i could add one06:22
poolieand see if it would have trapped on that line06:22
lifelessdown to 8806:53
poolielifeless, wgrant, how about something like http://pastebin.ubuntu.com/497509/ as a test?06:53
lifelessseems fine to me; lines 8,9 and 10 look like they will want to be reusable06:55
pooliea lot of it should be07:01
wgrantmaxb: Is python-debian really still needed in the PPA? It's in sourcecode now.07:05
pooliewgrant i'm pretty sure there was no test for mail to help@ but there is one now07:09
wgrantpoolie: Great.07:09
pooliecan you two help me decide whether to add more tests to this or just leave it?07:16
pooliei don't want to break incoming mail but i also want to leave this and work on something else07:17
lifelesspoolie: how big is the change? is it flag controllable? that would make it easy to back out.07:21
poolieit's https://code.edge.launchpad.net/~mbp/launchpad/dkim/+merge/3598507:21
lifelesspoolie: For refactoring I tend to remove all but one old-place test, and move them to the new place, and not try to add coverage07:21
poolieit's not flag controllable atm and it's refactoring code that seems poorly tested at the moment07:22
pooliethus my discomfort07:22
wgrantIt's also rather ancient code.07:22
lifelesspoolie: do you think it needs more tests?07:23
pooliein general, yes, it does07:23
pooliedoes it need me to add them right now? i hope not :)07:23
lifelesspoolie: to be comfortable landing this change I mean?07:23
pooliei don't know07:25
pooliei have a sinking feeling that to be able to test it more i'll have to change more stuff07:26
lifelessland it, qa on staging, rather than edge, done.07:26
pooliein particular, to make it have less side effects07:26
pooliefair enough07:27
pooliestaging receives mail, even thought it doesn't send it?07:27
poolieand we can now see the mail logs from it, or we should be able to07:27
lifelessit should be able to yes07:34
lifelessif not, I'll RT that :P07:34
pooliei replied to the rt asking for logs from staging too07:38
pooliethey don't seem to be visible atm07:39
=== almaisan-away is now known as al-maisan
poolieok, so with that off the plate i'll look at a flags editor07:42
lifelessyou wanted a chat with wallyworld07:45
lifelessI'm up for that nowish07:45
poolieis he?07:45
poolie<hold music>07:45
wallyworlddah de dah dah de dah07:46
pooliewallyworld, lifeless: mumble/skype/pots?07:46
wallyworldskype is ok for me07:46
wallyworldi don't have any specific agenda07:47
wallyworldsome of my random musings have already been aired in the mailing list07:47
lifelesspoolie: skype plox07:47
wgrantIt's a countdown!09:01
wgrantBut to what?09:01
lifelessreturn list(MilestoneVocabulary(self.context)) != []09:05
wgrantAt least it is scoped.09:05
lifelessguess how many milestones bug 1 has09:06
_mup_Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:In Progress by sabdfl> <09:06
wgrantWhy's it doing that? To work out whether to display the milestone selector?09:06
lifelesswgrant: yes09:08
lifelessbrowser/bugtask.py 363209:08
bigjoolsjeez, someone added 200 builds this morning09:09
wgrantjml: Are you going to get that BuilderSlave.clean thing merged?09:10
jmldoo-di-dee-dooo, doo-di-dee-do, doo-di-dee-dooo, doo-di-dee-do-do-di, doo-do-de-doo doo-de-de-do-do-do-do-dooo...09:10
adeuringgood morning09:11
jmlwgrant, yes, I'm going to do that next thing.09:12
wgrantlifeless: My religion forbids me from opening a file that large.09:12
wgrantjml: Great, thanks. Everything seems to work well with that fixed.09:13
bigjoolswgrant: bug 64405909:14
_mup_Bug #644059: buildd upload processor explodes if source produced by binary build <Soyuz:New> <https://launchpad.net/bugs/644059>09:14
bigjoolscan you write that in English? :)09:14
wgrantbigjools: Description updated.09:15
bigjoolsgrassy ass09:15
jmlwgrant, how did mwhudson trigger the bug in first place? it would be nice to have some kind of test for that.09:16
jmlas well as a direct one for clean.09:16
wgrantjml: I triggered it too.09:16
wgrantLet me check my logs.09:16
jmlno matter09:16
jmlbigjools has just given me a few pointers09:16
jmlmrevell, hello09:16
bigjoolswgrant: was it getting called from rescueBuilderIfLost ?09:17
wgrantbigjools: No, updateBuild.09:17
wgrantAlthough rescueBuilderIfLost probably had the same issue.09:18
bigjoolshey mrevell09:18
wgrantWe also sometimes get into a situation where a BuilderSlave is used during dispatch to.09:18
wgrantNFI how.09:18
jmlif all goes well, that won't matter at the end of the week.09:19
wgrantAlthough I wonder how much of a performance boost it will be.09:22
jmlI would like someone to review this please: https://code.edge.launchpad.net/~jml/launchpad/split-mail-and-summary/+merge/3611109:22
jmlwgrant, probably not much.09:23
jmlwgrant, the sanity boost is not to be sneezed at09:23
wgrantThat's true.09:23
jmlbigjools, you might be interested in this bug: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/41968509:30
_mup_Bug #419685: Buildbot sends emails that require no action <build-infrastructure> <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/419685>09:30
lifelessthe per row stuff here is still going to be terrible.09:36
jmlIt's such a good branch.09:41
lifelessjml: ask me how many queries viewing a trivial bug page with no subscribers, 2 product tasks takes.09:43
jmllifeless, I've been wondering, ...09:44
jmllifeless, :(09:44
lifelessjml: I got pissed off enough this morning09:44
lifelessthat I'm doing something about it09:44
lifelessI'm down to 7209:44
jmllifeless, woot!09:45
lifeless1 15 files changed, 157 insertions(+), 86 deletions(-)09:45
lifelessjml: I may want a very pragmatic review for this :P09:45
jmllifeless, ok. :)09:45
jmllifeless, swap you.09:45
lifelesssure, what up09:45
jmlbetter mail from ec209:45
jmllifeless, thanks.09:56
jml(also, wuuu)09:56
wgrantlifeless: I suppose it also now scales better?09:56
bigjoolshave you seen how many HTTP requests there are for a bug page with a cold browser cache?09:57
wgrantWhat are they all?09:58
wgrantI can't think of too many.09:58
bigjoolstry it in .dev09:58
wgrant.dev is special.09:58
wgrantThe JS isn't merged, is it?09:58
bigjoolsummm I dunno10:01
* bigjools -> sprint10:01
lifelesswgrant: somewhat10:02
lifelesswgrant: I haven't reached the last query yet.10:02
lifelessit will still do 4 queries per nomination.10:02
wgrantlifeless: Was your mass-CP rolled out everywhere?10:02
wgrantEven Soyuz?10:03
lifelesscheckwatches + appservers10:04
lifelessnot quite everywhere10:04
lifelessthere was a singular CP done at the same time10:04
wgrantLooks like it did make it to cocoplum anyway.10:05
* lifeless headdesks10:05
lifelessSELECT COUNT(*) FROM Bug WHERE Bug.duplicateof = %s10:05
bigjoolsummm that's bad if you updated cocoplum10:05
wgrantbigjools: Why?10:05
wgrantThe buildd-manager thing wasn't in the last batch.10:05
bigjoolsif the b-m change made it there10:05
bigjoolsah ok10:05
wgrantAnd b-m is OK now, anyway.10:06
bigjoolswell no, it needs a cronjob10:06
wgrantJust needs the extra cron job set up before the next CP.10:06
=== bigjools is now known as bigjools-afk
lifelesswgrant: we have a denormalised field to avoid that particular query10:06
lifelesswgrant: ...10:06
wgrantHah, so we do.10:07
wgrantIt shouldn't be too slow, though.10:07
lifelesswgrant: hah10:08
lifelesswgrant: hah10:08
lifelesswgrant: hah10:08
lifelesscount(*) is one of the slower operations you can do10:08
wgrantNo bug ever has more than a few hundred.10:08
wgrantSo there's not a huge number of rows to go through.10:09
lifelesswgrant: right, but it still requires a index operation that is not significantly cheaper than select..10:09
lifeless(emphasis mine)10:10
lifelesswgrant: hey, guess how bug privacy duplicates interact10:13
lifelesswell, and I'm just guess10:18
lifelessone query per private dupe10:18
lifelessmaybe 210:18
wgrantOnly 1?10:19
lifelessguess what happens when apport backlogs10:19
jmlhave those windmill failures been addressed?10:20
lifelessnot that I know of10:24
lifelessmars says he's oging to look at bb soon10:24
lifelessalso \u/ for duplicate methods that do exactly the same thing10:24
lifelesspersonIsDirectSubscriber == isSubscribed10:25
jmllifeless, hey, your fixture thing is an lp dep?10:27
lifelessjml: yes10:31
jmllifeless, ok. should we deprecate the lp.testing.fixtures business?10:32
lifelessjml: its mostly moved over to context managers already; I moved fakelibrarian10:32
lifelessthere's one use remaining I think.10:32
jmllifeless, looking at lp.testing.fixtures, it's not using ctxmgrs at all10:32
lifelessI may be confused. ELate+BugTask:+index is consuming me10:33
wgrant8 to go!10:35
jmllifeless, fixtures... sourcedeps, package or eggs?10:35
jml(cos it ain't working)10:36
lifelessjml: its in the dist-cache10:36
lifelessjml: versions.cfg etc10:36
jmlahh, thanks.10:36
lifelessjml: works for me; check the fake librarian10:36
lifelessyou may have an old  branch?10:36
jmllifeless, I don't see *anything* that imports 'fixtures'10:37
jmllifeless, the branch is current as of Monday morning.10:38
lifelessline 2210:38
lifeless 16 files changed, 197 insertions(+), 122 deletions(-)10:38
jmllifeless, thanks. merging in latest stable gets it.10:39
lifelessjml: https://code.edge.launchpad.net/~lifeless/launchpad/bugtask+index/+merge/3611710:42
lifelessnoodles775: ping10:42
noodles775Hi lifeless10:42
lifelessso you've got a branch called 'expose distro diff' or thereabouts10:43
lifelessI wanted to check that you're happy for that to happen on lpnet, *now* (in principle)10:43
lifelessand that you're not basing it on 'this release cycle'10:43
noodles775erm, It's for db-devel?10:43
lifelessnoodles775: ah, I didn't see that :P10:43
noodles775I mighht not have set it...10:44
lifelesssafe enough then (though personally I'd still use a flag, in case you find a bug at the last minute or post release and want to pull the feature)10:44
wgrantIs the API flaggable?10:44
lifelessshould be10:44
lifelesssame request object IIRC10:45
lifelessyou can't change whats exported10:45
noodles775I've flagged the UI, but didn't see it as necessary to flag the API for it (since there will not be any data to access).10:45
lifelessbut you can control what it does10:45
lifelessI would't stress about the API10:45
wgrantI still don't see how the current derivation model can work.10:45
wgrantBut I guess we'll see.10:45
noodles775wgrant: in what way?10:45
wgrantnoodles775: We need multiple parents.10:46
lifelessjml: I got that branch exactly the right size it seems (800 lines)10:46
wgrantChanging parent_series after initialisation will break a few things and seems a bit ew.10:46
noodles775wgrant: ah, right. AFAIK multiple parents was discussed by bigjools-afk with others, but not included in the initial release. I don't know the details.10:46
lifelessjml: its getting late, so I'd like to know if you're going to look now, or later.10:50
jmllifeless, am looking now10:50
jmllifeless, is MilestoneTarget.has_milestone a property?10:52
lifelessif you mean has_milestones10:52
lifelessits about a brazillion times faster than the vocab10:53
jmllifeless, yeah.10:56
jmllifeless, I've approved modulo some clarity/style tweaks.10:56
jmllifeless, great patch. thanks.10:56
noodles775Has anyone seen "psycopg2.ProgrammingError: server closed the connection unexpectedly" when running `make schema`? http://pastebin.ubuntu.com/497604/11:03
lifelessjml: dumps(security_proxied_list) goes boom11:04
lifelessjml: thats why the comment; I've expanded it.11:04
jmllifeless, thanks.11:04
wgrantnoodles775: I haven't seen that. Does the postgres log say anything useful?11:04
lifelessjml: also I mean 'tp' query - 'tp = ' is 4 lines up.11:05
noodles775wgrant: yes - I should have checked there first. FATAL:  the database system is in recovery mode11:05
jmllifeless, heh. ok.11:05
lifelessnoodles775: nuke your db11:05
lifelessnoodles775: and start over; its the easiest way11:06
noodles775k, thanks lifeless, wgrant11:06
lifelessgmb: did you talk to the mozilla folk11:24
gmblifeless, Can you be more specific? I might have missed something whilst sifting through my mail pile yesterday11:26
gmblifeless, Ah11:27
gmbhttps://bugs.edge.launchpad.net/malone/+bug/642418 you mean?11:27
_mup_Bug #642418: Detect when remote Bugzilla bugs are duplicates and link to the duped-to bug instead <Launchpad Bugs:New> <https://launchpad.net/bugs/642418>11:28
gmblifeless, Exactly to whom should I be talking and what needs to be said?11:28
lifeless$folk and $mozilla that are $concerned11:29
lifelessI dunno11:29
gmblifeless, That's $unhelpful.11:29
lifelessI just got told you knew all about this stuff :)11:30
lifelessso I helpfully palmed the enquirers off with yourname and a glib assertion that you'd be helpful/11:30
lifelesspoolie: would appreciate any insight you have into https://bugs.edge.launchpad.net/launchpad-foundations/+bug/63434211:30
gmbHelpful doesn't start until this afternoon.11:30
_mup_Bug #634342: need a features 'scope' for page ids <qa-ok> <Launchpad Foundations:Fix Committed by lifeless> <https://launchpad.net/bugs/634342>11:30
gmblifeless, I know the code, at least. Let me dig further into this. I'll add it to my todo list for today.11:30
deryckMorning, all.12:00
lifelesshmm, clearly I'm up too late.12:18
spivlifeless: yes12:19
lifelessgary_poster: you're around now, I take it ?12:27
gary_posterlifeless, yes on call but what are you doing up?!12:27
lifelesswell, I had a itsy bitsy morning12:28
lifelessand then I got my teeth into BugTask:+index12:28
lifelessstill winding down from that ;)12:28
lifelessgary_poster: when do you think you'll be off the call?12:29
gary_poster10 min?12:29
lifelesshmmm, if you're up for a catch up post that, I might hang about a little more12:29
gary_posterlifeless, you are crazy, but I am off the call and available for ~20 min.  mumble, irc, skype...?12:34
lifelesslike a fox12:34
lifelesslets say skype12:34
wgrantWe really need to work out a better bug notification strategy.12:36
wgrantSince my top timeout is probably mostly that :(12:37
deryckwgrant, completely agree12:37
wgrantI wonder if it's as easy as removing BugNotificationRecipient and calculating it when we send them.12:37
wgrantOr if that will miss some things (like bugs being reassigned)12:37
deryckI think it could be more complex than that, but it's an easy check to remove it and run tests to see what breaks.12:39
wgrantYeah, it's not that simple, sadly.12:45
wgrantSince the recipients can change if the task target does, and the existing code will notify the old recipients as well.12:45
=== al-maisan is now known as almaisan-away
deryckwe will have to deal with this notification overhead sooner rather than later, though.12:50
deryckSo someone from the bugs team will look at this in the short term.  if you don't beat us to it, wgrant. ;)12:50
wgrantIt makes kernel bugs just about impossible to file.12:51
=== mrevell is now known as mrevell-lunch
* gmb lunches12:53
leonardrsinzui, when you come in, i'm getting an incomprehensible error message when i ec2 my branch, but not when i run the test locally:13:38
wgrantNot this again :(13:38
marswgrant, ?13:38
wgrantIt's the off-by-one OOPS ID thing.13:39
marsis there a bug for it?13:39
wgrantI forget.13:39
lifelesstry it with https://code.edge.launchpad.net/~lifeless/launchpad/lessGetLastOops/+merge/3599413:42
wgrantNight lifeless.13:42
marsI can't find a bug for it13:44
=== bigjools-afk is now known as bigjools
bigjoolsjml just munched on two faggots for the first time and loved it13:55
=== mrevell-lunch is now known as mrevell
StevenKmars: Hey, did you get my e-mail?14:09
gmbUrsinha, ping14:15
Ursinhagmb, hi14:16
gmbUrsinha, Howdy. So, I keep getting the "Please try again" page when I try to delete this bugwatch: https://bugs.edge.launchpad.net/bugs/642418/+watch/81562/+edit14:16
_mup_Bug #642418: Detect when remote Bugzilla bugs are duplicates and link to the duped-to bug instead <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/642418>14:16
Ursinhawhat my script did wrong14:16
gmbUrsinha, I don't know why it's not getting an OOPS raised, but do you know where I might be able to look to see if one's been raised but isn't showing up in the UI?14:16
gmbI'm kinda clutching at straws here, by the way :)14:17
Ursinhagmb, all oopses are on devpad, let me take a look14:17
gmbUrsinha, Thanks.14:17
gmbYou have *got* to be kidding me.14:18
gmbWe add the "bug watch foo has been deleted" notification before we delete the bugwatch.14:18
wgrantWhat gets really amusing is when transaction retries come into play.14:19
wgrantSo you end up with lots and lots of notifications.14:19
gmbwgrant, Indeed. As I just discovered.14:21
* gmb branches and fixes. That one's not hanging around.14:21
Ursinhagmb, I can't find oopses on devpad14:24
Ursinhathey're synced every 10 minutes, so I guess should be there already14:24
Ursinhabut I can't find anything grepping them14:24
gmbUrsinha, Thanks. I've found (possibly) a separate problem (see my rant above).14:25
Ursinhahehe, np14:26
gmbUrsinha, I'll grep later.14:26
=== almaisan-away is now known as al-maisan
rockstargmb, I need to reboot, but then I'd like to chat about the wizard widget.15:11
gmbrockstar, Sure. thing.15:18
gmbEr. Sure thing.15:18
gmbrockstar, I'm ready whenever you are.15:27
=== leonardr_ is now known as leonardr
nigelblifeless: I like the stump speech ;)16:22
marsleonardr, you missed this fix for your EC2 issue: https://code.edge.launchpad.net/~lifeless/launchpad/lessGetLastOops/+merge/3599416:34
marsStevenK, yes, I got it.  Have been working on other things, will get back to mainline build engineer work this week.16:35
leonardrmars: is it landed?16:36
marsleonardr, I don't think so.  Robert said to merge it into yours and then try again.16:37
leonardrmars: i see16:37
bachas anyone seen basic auth challenges popping up when running windmill, e.g. http://people.canonical.com/~bac/Screenshot-Authentication%20Required.png16:46
bacthey don't seem to interfere with the tests but are annoying16:47
marsbac, yes, a known problem since the summer.  Did this just start now?16:47
bacmars: yeah16:47
bacmars: just ignore it?16:48
marsbac, did you just upgrade to Maverick?16:48
bacmars: last week, yes16:48
marsbac, so it could be a problem with the test harness under Maverick, or it could be bug 60919016:50
_mup_Bug #609190: Login problem in windmill test <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/609190>16:50
bacmars: thanks16:51
marsbac, try '$ bin/test lp.translations.windmill.tests.test_import_queue', see if that fails16:51
marsif so, then it is systemic16:51
marsif not, then it is an issue introduced by new code16:52
henningeWhich style is the right style for headings in doctest? I for get ..16:58
bacmars the translations test did throw up a bunch of those auth windows but proceeded without error16:59
henninge== Heading ==16:59
bachenninge: latter16:59
henningecheers bac ;)16:59
bachenninge: some do16:59
bacwhich seems to be ok unless gmb is your reviewer.  :)16:59
bac(but only for the first)17:00
henningeI was about to ask that ...17:00
=== beuno is now known as beuno-lunch
marsbac, ok, noted.  I can check after I do the Maverick upgrade myself.  Was this a fresh system install?17:02
bacmars: yes17:02
danilosbigjools, hi, if I was to take on bug 644464, what would be the best approach?17:06
_mup_Bug #644464: Template generation jobs are not aggregated <Launchpad Translations:Triaged> <https://launchpad.net/bugs/644464>17:06
marshenninge, danilos, any idea what is up with the latest buildbot failure?  It died in  canonical.buildd.tests.test_translationtemplatesbuildmanager.TestTranslationTemplatesBuildManagerIteration.test_iterate_fail_GENERATE17:07
marshenninge, danilos, https://lpbuildbot.canonical.com/builders/lucid_lp/builds/171/steps/shell_7/logs/summary17:07
danilosmars, hum, no, let me take a look17:07
bigjoolsdanilos: you need to see if a job already exists for the same context; I don't know the best way of doing that for TTBJs17:07
marshenninge, danilos, nevermind, misread.  It failed in "Error in test lp.buildmaster.tests.test_builder.TestSlave.test_clean", not the translations stuff.17:08
danilosbigjools, well, I know that much :), I was asking more along the lines of where do you do that for other jobs in soyuz, and if it would be a good thing to do it at the same time?17:08
bigjoolsdanilos: for soyuz, we allow it, but we detect a more recent version and supersede the old build request instead of dispatching it17:08
bigjoolsdanilos: but I am worried that you had *so many* requests all queued up17:09
danilosbigjools, well, we don't have a limit per project yet, but we probably could add that too17:09
bigjoolsI think it's a good idea17:10
marsbigjools, when you are finished talking to Danilo, buildbot died with 'Error in test lp.buildmaster.tests.test_builder.TestSlave.test_clean' - are you the right person to handle that?17:10
danilosbigjools, anyway, dispatch time is probably the best time if we want to aim for minimum amount of repeated work17:10
bigjoolsmars: jml is working on it right now17:10
bigjoolsit's a test we added recently and then promptly broke17:10
marsbigjools, thanks.  jml, could you please send a message to the list about it?17:10
jmlmars, yes, of course17:11
marsthank you17:11
bigjoolsdanilos: I would address it on both fronts17:11
=== benji is now known as benji-lunch
danilosbigjools, well, if I address it on one (considering we are always getting tip of the branch, I can put that limit at 2 which would avoid all the concurrency issues)17:12
danilosbigjools, I don't see much need for addressing on the other... and vice versa17:12
danilosbigjools, btw, what was the number of builds that we had scheduled?17:13
bigjoolsdanilos: ok, I'm no so familar with the dynamics of how and when you generate the jobs so I trust whatever you say17:13
danilosbigjools, good :)17:13
danilosbigjools, or not so good, but anyway :)17:13
bigjoolsdanilos: I don't know how many were scheduled17:13
bigjoolsbut, one sec17:13
danilosbigjools, I see we did at least 360 for software-center17:15
danilosbigjools, in one day :(17:15
danilosbigjools, it definitely sounds as if something was wrong elsewhere as well17:16
bigjoolsI expect someone uploaded a bunch of stuff, somewhere17:17
danilosbigjools, because that particular branch only had 9 commits on 21st (when it did 360 template build jobs)17:17
bigjoolsdanilos: that's.... special17:18
danilosbigjools, yeah, so somewhere on the code side we generated a bunch of duplicated jobs needlessly17:18
danilosbigjools, so, I'd like to investigate this a bit, and will end up limiting the number of entries in the table per project to 2 anyway, but would like to check out why is this happening in the first place17:19
danilosbigjools, or perhaps we are retrying them for some reason? under what conditions does build-manager retry a build (just so I can check we are not causing it to do that)?17:20
bigjoolsdanilos: that's down to the behaviour class for templates17:21
bigjoolsthe b-m just picks the next item from the queue and dispatches it17:21
danilosbigjools, ok, since we haven't implemented that, we should not worry about it17:21
danilosbigjools, thanks, useful input, I'll go dive into code a bit17:22
bigjoolsdanilos: np17:23
bigjoolsdanilos: can you kill off all the unnecesary duplicated TTBJs in production?17:25
bigjoolsand unnecessary, even17:25
danilosabentley, btw, do you know of any way to check if branch-scanner has recently been detecting a single change more than once (I am looking at logs but I can't find current branch-scanner logs on devpad)17:25
danilosabentley, never mind, it's scan-branches.log17:25
danilosbigjools, sure, though it will take me a bit to rediscover all the tables that I need to purge from17:27
bigjoolsdanilos: see mawson:~/handy_sql/kill_a_build.sql for inspiration17:29
danilosbigjools, thanks17:29
=== salgado is now known as salgado-lunch
=== gary_poster is now known as gary-lunch
abentleyjtv, around?17:46
danilosabentley, he should be long gone17:51
abentleydanilos, mostly, but I have seen him around in the past.  I'm having problems with the FakeLibrarian.17:52
bigjoolsdanilos: hi17:52
danilosbigjools, hi :)17:52
danilosabentley, yeah, he does sometimes stick for much, much longer, but we are also trying to get that habit out of him :)17:53
bigjoolsthere's a rather large problem with builder histories being absent since about 6 hours ago, did you guys  change anything for your history?17:53
danilosbigjools, those changes only landed on devel a few days ago, are you talking about production?17:53
danilosbigjools, unless this was CPed in line with lifeless' new approach to rolling out stuff, then we haven't changed anything recently17:54
bigjoolsdanilos: that's what I am wondering17:54
bigjoolsit seems remarkably coinciding17:55
bigjoolswhat time did it get rolled out17:55
danilosbigjools, did it get rolled out?17:55
bigjoolsI;'ve no idea, I only just started looking at this17:56
=== beuno-lunch is now known as beuno
danilosbigjools, this is the last one: https://code.edge.launchpad.net/~lifeless/launchpad/cp/+merge/3596517:59
bigjoolsdanilos: what revno did the ttbj history land in?18:00
bigjoolsand was it on db-devel or devel?18:00
danilosbigjools, I am looking that up now18:00
=== al-maisan is now known as almaisan-away
danilosbigjools, fwiw, it's still marked as qa-needstesting for us, so I doubt it was CPed18:01
danilosbigjools, it's db-stable r979318:01
danilosbigjools, so, my guess is that it ain't that18:02
danilosbigjools, it's neither listed as part of any of the CPs18:02
* bigjools scratches head18:03
=== matsubara is now known as matsubara-lunch
=== Ursinha is now known as Ursinha-lunch
danilosbigjools, fwiw, I am struggling very much coming up with appropriate queries for the DB (I started using the new table jtv added and the query failed on production)18:12
danilosbigjools, that, at least, confirms that the code was not rolled out, but also makes my head hurt18:12
bigjoolsthe new model is a total headache :/18:12
danilosbigjools, two models are a bigger one when you rarely use either :)18:13
=== benji-lunch is now known as benji
=== deryck is now known as deryck[lunch]
mtaylorspiv: you awake yet?18:35
mtayloror really any losa - there is a chance that there may be a memory issue on maverick ppa builders ... we're seeing a drizzle test failure there that we're not seeing on our builders, and looking at the output it looks very similar to a weird failure we had on a machine a month ago that was also due to bad memory18:36
mtaylorI mean, it could also be a bug in our code - but I thought it might be worth someone looking at18:37
mtaylorhrm - perhaps nevermind ... similar but different is occuring in multiple places ...18:42
* rockstar lunches18:59
mtaylorquick followup - it is our bug, but for some reason it's only showing up on the launchpad ppa 32 builders and not our 32 build machines ... are you guys doing anything weird on your 32bit builders which would cause them to deal with memory differently? (asking to see if I can set up something similar to try to more easily reproduce)19:00
jmlmtaylor, there's an issue that we are investigating right now19:01
jmlmtaylor, but apparently that's a different problem19:02
brycehderyck[lunch], on 642418 is it LP or the bugzilla lp plugin that sets the See Also link?19:05
bigjoolsmtaylor: the PPA builders are virtual, which might be something, I dunno19:05
bigjoolsmtaylor: but it's best to check with lamont19:05
=== gary-lunch is now known as gary_poster
=== matsubara-lunch is now known as matsubara
=== Ursinha-lunch is now known as Ursinha
brycehderyck[lunch], lp:~bryceharrington/bugzilla-launchpad/no-dupe-link-update19:34
=== salgado-lunch is now known as salgado
* lifeless stretches19:37
=== deryck[lunch] is now known as deryck
lifelessnigelb: ?19:40
deryckbryceh, not sure if it's the plugin or lp.  If it's the plugin, it may not be worth worrying too much about, since people will need to install new, and newer bugzillas won't be affected.19:40
brycehderyck, since it sounds like it's a bugzilla policy whether to update links on dupe bugs, the plugin seems like the "right" place to fix it19:41
brycehderyck, as that'd prevent the incorrect behavior regardless of who was using the api19:42
brycehderyck, but for our case at least, i think we can conditionalize calling the set_link() api on it being a dupe... I'm checking19:42
bryceh(and it still leaves the more general issue of making us point to the proper bug, but I *think* that may be a more involved issue)19:43
* deryck is thinking....19:43
deryckbryceh, so the plugin is already configurable about whether or not to update links on dupe bugs?19:44
brycehderyck, no19:45
brycehheh that would make it especially easy19:45
bigjoolslifeless!  I'm surprised you don't have some way of accessing IRC when you're sleeping19:45
brycehbigjools, isn't that called a telephone?19:46
bigjoolsor is this your morning clone I am speaking to19:46
lifelessbigjools: I plug an ethernet cable into my ear.19:46
jmlwifi drains too much power in sleep mode19:46
deryckbryceh, ok, so I think we should do the right think on our end.19:47
deryckbryceh, if it's just a simple conditional to be quiet on upstream closed bugs, that's enough.19:47
brycehderyck, yes, it looks like I can slot in a check in linkLaunchpadBug()19:48
deryckbryceh, ok, excellent.19:48
jmlhello everyone19:49
brycehderyck, this will only prevent it from setting the bug link on the upstream bug though... are there other updates we want to prevent done to upstream dupes?19:49
jmlbigjools & I just finished busting up a bunch of doctests19:49
jmlbut we need to land the change, and I think someone ought to give it a cursory review first19:49
nigelblifeless: I just said your stump speech http://bit.ly/9mTfdf looks awesome :)19:49
bigjoolsit's death to .txt19:49
jmlwho would like to review https://code.edge.launchpad.net/~jml/launchpad/buildd-slavescanner-bustage/+merge/36187, which is 1k+ lines of doctest-removing fun?19:49
lifelessnigelb: ah right, I had no idea what you were referring to by 'stump speech'19:50
deryckbryceh, perhaps you and gmb should do a pre-imp on this tomorrow before he EODs.  I'm not sure about other updates honestly.19:50
lifelessnigelb: I'm glad you like it.19:50
nigelblifeless: the bzr team called it that19:50
jmlanyway, I'm off to the pub19:51
lifelessnigelb: its ok, I was just confused ;)19:51
jmlwhoever reviews that branch gets a complimentary compliment at the next Epic19:51
jmlmaybe a beer too19:51
nigelblifeless: considering that it must be something like 5 for you, totally understandable19:51
lifelessnigelb: 7 :P19:52
nigelblifeless: still way to early ;)19:52
deryckbryceh, the only other updates we would do would be comments, if supported, right?19:52
brycehderyck, dunno19:52
brycehderyck, yes19:53
deryckbryceh, I think so, but not 100% sure.  I can argue boths pros and cons here, so let's allow gmb the final say.19:53
bryceh           # Only sync comments and backlink if the local bug isn't a19:53
bryceh            # duplicate and the bug watch is associated with a bug task.19:53
bryceh            # This helps us to avoid spamming both upstream and19:53
bryceh    # ourselves.19:53
lifelessok, lets see how bad this bugtask index test breakage is19:53
brycehhaha, we avoid updating if the bug is a dupe on *our* end, but not *their* end :-P19:54
deryckright, so there's your answer.19:54
deryckbryceh, if there's is a dupe, do nothing.19:54
brycehderyck, ok, so don't bother with fixing the plugin, just fix locally?19:55
deryckbryceh, yes19:55
brycehok easy enough19:55
=== almaisan-away is now known as al-maisan
lifelesssigh, bug.getBugTask makes me sad.20:02
jamlifeless: I seem to be getting spurious failures when asking people to ec2land branches for me. I'm not really sure who to ask for help20:03
jambut I see you online :)20:03
lifelessjam: hi, what sort of failures?20:04
jamlifeless: I can send you the email if that would be easier20:05
jambut 2 in soyuz20:05
jamone in buildmaster20:05
bigjoolsgood night all20:07
jamlifeless: my change just updates the default location of 'codebrowse's config files after a clean 'make'20:07
jamlifeless: to elaborate, when starting with a clean machine, the build instructions didn't create the 'codebrowse.launchpad.dev' directory but they *do* create a 'bazaar.launchpad.dev' directory, which is also what is configured elsewhere in the code. I honestly wouldn't expect any tests to fail from this, as it is a setup thing.20:12
jamcertainly it doesn't touch soyuz code20:12
lifelessjam: do those tests pass locally ?20:13
lifelessjam: note that ec2 doesn't do a full setup, it starts partially setup AIUI20:13
lifelessjam: perhaps talk with mars20:13
jamlifeless: well, the last code I submitted was a comment-only change, and it still failed the ec2land test suite (for one person, for the other person it passed)20:14
jamas such, I don't have huge faith in ec2land...20:14
jamthough that time was a different test that failed20:14
lifelessjam: will be a minute20:21
cr3gary_poster: got a minute for a question about z3c.recipe.scripts? having problems importing lazr.restfulclient in /usr/lib/pymodules after importing the generated site.py file20:47
jambac: ^^20:51
bacoh, great20:51
jamso if the tests are already failing, it may be a particular version of devel that I merged, etc20:51
jamI suppose I should have merged something 'stable' ?20:51
bacjam: did you not start with the latest devel?20:51
baci mean the latest rev of devel20:52
jambac: I started the branch a while back, you asked me to update against devel so I did20:52
bacjam: right.20:53
jamI also may have started it as a db-devel branch, etc20:53
jamI would have thought by now the two would have converged, though, and the diff was the tiny one I expected20:53
bacjam: i started with a clean version of the latest devel and merged in your two-line change20:54
jamand you still got the failure?20:54
bacjam: and that is what i'm testing locally20:54
bacjam your branch on LP is what ec2 tested20:55
jam(missing shows that I am without about 48 devel patches)20:55
jambac: are you running the full suite? Or just the tests that failed?20:59
bacjam: just the failed20:59
jamgood to know they fail in isolation :)20:59
jam(I'm still trying to just get the test suite to run again, after updating to the latest lp/devel)21:02
bacjam: ec2 had 4 failures.  devel locally has 1.21:02
bacjam: will re-run your branch again and see what fails21:02
jambac: well one of the failures definitely looked to be an isolation issue, since it was checking for OOPS...6 and found OOPS...721:03
lifelessso this is the same uniquefileallocator thing21:03
lifelesstry with my branch?21:04
lifelessI'm waiting for *anyone* to tell me it works/doesn't work21:04
jamlifeless: what is the right way to update your branch? I tried "utilities/update-sourcecode" and now zc.buildout is failing at 'make' time21:05
bacjam:  your branch has 1 failure locally, not the OOPS one21:05
gary_postercr3: do you have include-site-packages = true in your .cfg file?  That will be the default in 1.5.2 but false is default before then21:05
jambac: is it the same one that 'devel locally' has ?21:05
bacjam:no.  devel has the OOPS failure.  your branch has ^^21:06
cr3gary_poster: yes21:06
jambac: well having it fail 3-different ways doesn't make me very trusting21:06
cr3gary_poster: I'm kinda confused why import can't find restfulclient under /usr/lib/pymodules/python2.6 when that path is actually in sys.path21:07
lifelessjam: you may need an apt-get update && apt-get upgrade in there21:07
gary_postercr3: can you give me instructions to dupe maybe?  want to help, but also have some other people to support right now21:08
cr3gary_poster: it seems that if I introduce /usr/lib/pymodules/python2.6 in the buildout_paths, in site.py, before the zope.exceptions egg path, then I can import lazr.restfulclient21:08
jamlifeless: did that, now I'm trying an update of the "download-cache" directory21:08
jam(bzr update)21:08
gary_posternot a good long-term solution cr3: setuptools won't work the right way like that21:09
jamwhich did pull in a bunch of stuff21:09
cr3gary_poster: I'll try to figure a way to easily dupe and get back to you21:09
gary_posterthanks cr321:09
cr3gary_poster: zc.recipe.egg seems to order paths correctly though21:09
jamI don't know why it didn't update with everything else...21:09
jamat this point, I'm ready to turn to a voodoo doll21:09
gary_postercr3 what definition of correc?21:09
cr3gary_poster: works :)21:09
gary_posterfair enough, cr3.  does not work in other circumstances though.21:10
jamlifeless: updating download-cache was enough to get 'make' to run21:10
cr3gary_poster: ugh, so if I use both zc.recipe.egg and z3c.recipe.scripts, I don't work in all circumstances :)21:11
gary_postercr3: correct.  z3c.recipe.scripts is supposed to be your one-stop-shop.21:11
gary_poster(the fact that it is not is a bug, obviously)21:11
gary_posterzc.recipe.egg will not change21:12
cr3gary_poster: I'll concentrate on porting my existing zc.recipe.egg stuff to z3c then21:12
gary_postercool, cr3.21:12
lifelessdon't you hate it when adding an assert makes things work ...21:13
marsleonardr, still around?  Did you get a chance to try lifeless's OOPS patch?21:20
leonardrmars: well, it's in ec2. i didn't get the problem locally21:20
lifelessUrsinha: could we have a 'latest' or 'current' symlink for the oops report html file ?21:20
Ursinhalifeless, it's always the deployment-stable one21:21
lifelessdifferent hting21:21
lifelesslike https://devpad.canonical.com/~lpqateam/oops-summaries/lpnet-oops-last-hour.txt21:21
lifelessbut for the day21:21
marsjam, any luck?  Normally to update LP you can run 'rocketfuel-get; cd mybranch; bzr merge ../devel'21:21
lifelessI thought there was one, but I can only find things for specific dates, not one I can put in by bookmarks.21:22
marsleonardr, the branch with the patch is in EC2?  Or just the plain old branch?21:22
leonardrthe branch with the patch is in ec221:23
marslifeless, ^21:23
jammars: well, things almost seem to work, but I still am trying to determine how to run a subset of the test suite based on failures from ec221:23
jamat least the test runner starts up and runs some tests and exits appropriately21:23
lifelessleonardr: cool, thanks21:23
lifelessjam: testr load < subunit-stream; testr run --failing21:24
marsjam, ok, in the branch: testr init; (save log.gz from ec2 to disk); zcat foo.log.gz | testr load; testr failing21:24
bacjam, lifeless: the failure i'm seeing is real and is going to put us in testfix mode very soon now.  https://lpbuildbot.canonical.com/builders/lucid_lp/builds/172/steps/shell_7/logs/stdio21:25
bacsearch for test_score_unusual_component21:25
Ursinhalifeless, I see what you want21:25
lifelessUrsinha: for that one html would be nice :)21:26
lifelessUrsinha: just needs to be a symlink to 2010-09-21.html, etc etc21:26
Ursinhalifeless, I got it21:26
jambac: so ec2land doesn't pre-merge the target branch and run the tests on the result?21:26
lifelessbac: was that landed without ec2land? do we know?21:26
bacjam: yes it does21:26
marsjam, ec2 land branches devel, then merges your branch, then runs the tests21:26
baclifeless: we have no way of knowing21:27
jamah, I was hoping so21:27
lifelessjml: yo21:27
bacsince it is soyuz i'd say not21:27
baci should rephrase21:27
lifelesshttps://lpbuildbot.canonical.com/builders/lucid_lp/builds/172/ is about to die and you're in the blame list. Should we rollback your commits ?21:27
bacbigjools does his testing locally.21:27
baclifeless: who was that directed to?21:28
marsjam, it was written by bzr folk - pre-merge is not natural, but all of you have learned to The Right Way To Do It21:28
lifelessmars: pre-merge?21:28
marslifeless, is that not what jam just called it?  "so ec2land doesn't pre-merge the target branch and run the tests on the result?"21:29
baclifeless: yes, it is certainly jml's r1158621:29
lifelessmars: 'EPARSE' on your last21:29
lifelessbac: land a backout of it21:29
lifelessbac: if you would21:29
=== al-maisan is now known as almaisan-away
lifelessand 11588 I guess, I'm assuming that that was an attempt to fix it that didn't21:29
baclifeless: so how would i do that?21:30
marsbac, please use "ec2 land --rollback 11586", or 'bzr lp-land --rollback 11586'21:30
marsbac, that way the QA flags will be correct21:31
bacmars: you rock!21:31
lifelessbac: bzr lp-land --rollback 1158621:31
lifelessbac: bzr lp-land --rollback 1158821:31
lifelessand then it should go its merry way21:31
lifelessmars: is that only commit message glue, or does it do the roll back merge too ?21:31
marsbac, by the way, you have to acutally run 'bzr merge 11586..11585'21:32
lifelessso, the full story then:21:32
lifelessbzr branch devel testfix21:32
lifelessin testfix21:32
mars(this is all in the not-yet-seen-but-does-exist user guide)21:32
lifelessbzr merge . -r 11586..1158521:32
lifelessbzr merge . -r 11588..11587 --force21:32
lifelessbzr commit -m "Rollback 11586 and 11588"21:33
lifelessbzr lp-land --rollback 11586 --rollback 1158821:33
lifelessmars: ^ sane?21:33
marslifeless, I hope so.  I don't know if the tagger or lp-land can take multiple --rollbacks21:33
bacseems right to me, though i would've forgotten the .21:33
lifelessmars: if it doesn't, it will need to ;)21:33
marsif it doesn't, we can roll each back on it's own maybe?21:34
marsbac, try it21:34
marstest flight!21:34
bacmars: trying21:34
bacstill generating me some wadl21:34
* mars hope's bac's branch doesn't end up in a smouldering crater21:34
marsI can watch the tagger logs21:35
marsah, maybe not.  There is a 4? hour lead time on landings21:36
marsputs is around 8:30 or 9pm here21:36
bacmars: what plugin provides lp-land?  mine doesn't have --rollback21:36
marsbac, bzr-pqm21:37
bacmars: i need to install from lp:pqm?21:37
marsbac, sorry, 'pqm' is the plugin, bzr-pqm is the ubuntu package21:37
bacmars: is there a ppa?21:37
marsbac, yes, it is in the LP ppa - what version do you have?21:37
marsapt-cache policy bzr-pqm21:38
bacii  bzr-pqm                 1.4.0~bzr69-1           bzr plugin to submit an email to a Patch Queue Manager21:38
marsout of date21:38
marswhy, it needs a maverick build?21:38
bacnot.  on.  lucid.21:38
bacno one should be on lucid21:38
bacwe all have to jump the broom together21:39
bacor something like that21:39
baci think i confused roots with thelma and louise21:39
bacmars would you mind trying the rollback since i'm crippled?21:40
marsok, having no series on the PPA page package list is a little annoying21:40
bacjam, please ping me tomorrow and i'll try again21:40
marswasn't there supposed to be JavaScript on this page?21:40
marsbac, ok, what is the branch, just devel?21:41
bacmars: yes, devel.21:41
marsbac, here is your problem: https://edge.launchpad.net/~launchpad/+archive/ppa?field.series_filter=maverick21:41
bactrying to undo changes 1 and 3 from https://lpbuildbot.canonical.com/builders/lucid_lp/builds/172/21:41
marscompare to: https://edge.launchpad.net/~launchpad/+archive/ppa?field.series_filter=lucid21:42
bacyes, many fewer21:42
marssome are legitimate, like python-debian21:42
marsothers are oversights21:42
marsadding that to my TODO list21:42
marsbac, running it now21:43
bacis celso really still uploading PPAs for us?21:44
bacre: geoip on maverick21:44
jambac: I'll say hi tomorrow, thanks for working with me on this21:48
bacjam: sorry we had so much fail for what should've been easy21:48
jamlifeless: so how in the 'testr' stuff do you run just the failing tests?21:50
jamtestr run looks like it runs everything21:50
lifelesstestr run --failing21:50
jamah, testr run --failing21:51
marsbac, waiting for the MP to show up, I'll need you or lifeless to approve it to appease lp-land21:52
bacmars: rs=me21:52
bacmars: oh, right, on the MP21:52
rockstarWow, Chromium consistently crashes X for me now.21:53
marsrockstar, eeew21:53
rockstarmars, what's worse is that it seems to happen whenever I'm REALLY focused on what I'm doing.21:54
marsX crashes are not fun.  I've had my share of those.21:54
marsat least an application freeze is humane.  An X crash is just brutal.21:54
jmllifeless, hello. what's up?21:55
jamlifeless: trying to run "testr run --failing" gives me: http://paste.ubuntu.com/49798121:55
jamwhich seems like a test system infrastructure issue21:55
marsbac, https://code.edge.launchpad.net/~mars/launchpad/rollback-11586-and-11588/+merge/3621221:56
jmljam, yeah. I've been meaning to chase down that bug.21:57
lifelessjml: bb found a flaw in your branch21:57
lifelessjml: we're backing it out21:57
jmllifeless, :(21:57
jmllifeless, can't I just fix it instead?21:57
jamjml: well, it *does* take 4 hours to land the reversal21:58
lifelessjml: it looks like you've tried once already; its an unknown amount of time before we find all the issues and we don't want to be in testfix for that long.21:58
jmllifeless, wrong.21:58
marsok, trying lp-land21:58
jmllifeless, different problem.21:58
lifelessmars: hang a sec21:58
mars(nice timing)21:58
lifelessjml: please expand;21:58
jmllifeless, the first test fix was for a different failing test with a completely different problem: not having a clean() method on build21:59
lifelessjml: [separately, meta] are you landing these things via ec2 land ?21:59
jmllifeless, the failure you see now is due to a different problem due to a change to the API of a factory method21:59
jmllifeless, no, I'm not.21:59
lifelessjml: please do.21:59
jmllifeless, I normally do.21:59
lifelessjml: well, please do what you normally do.21:59
jmllifeless, ok.21:59
lifelessjml: not doing it has spent about 1/2 an hour of engineer time22:00
lifelessjml: how confident are you that there are no other issues behind this one22:00
jmllifeless, sure. I know how tedious it is to fix other people's broken builds.22:00
jmllifeless, quite confident.22:01
lifelessok, I'm happy with 'fix it' - but if we head back into testfix from another issue in the same patch in 4 more hours, I'll roll it back and let you debug tomorrow, ok?22:01
jmllifeless, +122:01
marslifeless, jml, so if need be, we roll back both with https://code.edge.launchpad.net/~mars/launchpad/rollback-11586-and-11588/+merge/36212 ?22:02
marslifeless, or will you prepare a new patch?22:02
jmlmars, I'd rather you rollback only 1158822:02
lifelessjml: landing without ec2 land is ok from time to time - you know me and pragmatism; I guess I'm saying that the work you're doing is stuff I wouldn't have attempted to land without full tests :)22:02
lifelessmars: I'll leverage that one, if needed22:02
marslifeless, ok, I'll leave it to you and jml then22:02
lifelessmars: thanks22:03
lifelessmars: can you confirm that when bb *starts* a build pqm opens again ?22:03
bacyeah, thanks mars22:03
baclifeless: that is true22:03
jmllifeless, as I said, normally, I wouldn't either.22:03
marsbac, the 'pqm opens again' is true?22:04
marsI don't actually know22:04
bacwhen bb starts on at [testfix] it optimistically takes us out of testfix mode22:04
lifelessbac: thats a different question22:05
lifelessbac: what I asked is 'starts a build' - thats 'receives a testfix commit'22:05
lifelessbac: many testfixes are spurious22:05
jmllifeless, waiting eight hours before I can meaningfully integrate is particularly trying at a sprint. :\22:06
baclifeless: what do you mean many testfixes are spurious?22:06
lifelessbac: I mean our test suite is as flaky as 'Flake(tm)' chocolate22:07
baclifeless: i assumed you meant pqm was closed due to testfix mode22:07
lifelessyes, thats what I mean22:07
bacthen bb won't start again until it receives a testfix, right?22:07
lifelessa non trivial amount of the time just running the test suite again will work.22:07
lifelessbac: no, you can press 'force build'22:07
marsbut that does not open PQM again22:08
baclifeless: so you are asking if forcing a build while in testfix mode opens pqm.  that i do not know22:08
lifelessmars: it doesn't? Ok, can we make it do so?22:08
marslifeless, no idea.  That means the poller bit would have to know the difference between a forced build and a testfix build.  Would take some work.22:09
lifelessmars: why would it need to know the difference?22:09
bacmars: making it know the difference would argue that it currently would open pqm22:09
marsyep, you are right.  I was thinking 'if you only want to open PQM when a build is forced, then you need to do some footwork'22:10
jmllifeless, I object to your statement about our test suite being flaky. The tests themselves very, very rarely fail spuriously.22:10
marstunnel vision, not helpful, sorry22:10
lifelessjml: layers fail to come up; windmill on the lp builder is almost always dying, we have hung librarian processes a lot of the time22:11
jmllifeless, yes, but the tests themselves...22:12
jmllifeless, everything else never works, sure.22:12
lifelessjml: I referenced the 'test suite', which is the whole22:13
marslifeless, ?  I haven't seen any of those issues in a while.  Recently it has been the DB connections thing.  and XMLRPC code pulls fail.22:13
lifelessmars: yesterday.22:13
lifelessmars: lp build, lp_db, go boom.22:13
lifelessthis is bad:22:13
cr3gary_poster: it seems the problem might be related to bug #599332, now I'm getting: Error: Couldn't find a distribution for 'elementtree'. Still working on it...22:13
_mup_Bug #599332: python-lazr.enum makes python-lazr.restfulclient no longer be visible <lazr.restfulclient (Ubuntu):Fix Released> <https://launchpad.net/bugs/599332>22:13
lifelessDanger Zone.22:13
lifelessenter makeBugTask.22:13
lifeless233525392 1622:13
lifeless233525392 1622:13
lifeless233525392 1622:13
lifeless234934992 1622:13
lifelessenter makeBugTask.22:14
lifeless233525392 1622:14
lifelessthats id(bug), bug.id22:14
lifelessin one call to makeBugTask()22:14
lifelessnotice that we have two objects claiming to be the same bug, live at once.22:14
wallyworld___thumper: standup?22:14
marslifeless, if you are wondering about the hardy builders, we haven't been maintaining them22:15
marsyes, they blow up, because we aren't fixing the problems22:15
lifelessmars: which demonstrates my point about flakiness ;)22:15
gary_postercr3: :-) I'm sorry.  If you can just give me dupe instructions, I'll swim in themorass for you22:15
lifelessthe lp test environment is not a 'fire and forget' one.22:15
lifelessits forget, and you're on fire.22:16
gary_postercr3: IOW, I advise trying to keep from figuring out what's actually going on--but if you do, that's great22:16
cr3gary_poster: heh, that's really nice but I'm trying to spare you too much distraction22:16
cr3gary_poster: in that case, I'll try to push a branch which can reproduce the problem for you. one moment...22:17
gary_postercr3: great thanks.  You may have to give me apt package names that you have installed that you suspect are causing the problem.  It may also be lucid or maverick or something, so let me know what that is22:18
jmlone line diff: https://code.edge.launchpad.net/~jml/launchpad/fix-factory-change/+merge/3621522:18
marsjml, if you are asking 'does this pass our code standards', then yes! r=mars.  If you are asking if you missed anything critical in this patch, well, I have no idea :)22:19
jmlmars, good enough for me.22:20
marsjml, done22:21
jmlmars, ta.22:21
jmllanding now.22:22
lifelessI'd really quite like to delete most of our coding standards I think.22:22
lifelessthey seem, the more code I read, to be a poor surrogate for what we say we care about22:22
marslifeless, to be honest, I mostly review for readability now - a Smalltalk-influenced ideal I believe we should strive towards22:26
marsand all-around quality - more robust tests, using the right test type for the purpose and such22:26
lifelessmars: things like lines ending on 78/79/80, (,) vs (, ), what case to use for function names seem largely content free22:28
marsoh, and for codebase design improvements - "we have a helper for that", or "this would be great as a new helper"22:28
cr3gary_poster: email sent22:28
lifelessmars: thats sensible22:28
marslifeless, true, but standards are nice.  When one person codes to C, one to Java, and one to Smalltalk, things look... not as nice as they could be, to the other people22:29
marsand in Python as a whole, I have seen all three22:29
gary_postercr3, ack, investigating22:29
marsstyles, that is22:29
lifelessmars: if we're going to use a standard, we should use a standard.22:29
lifelessmars: not make our own22:29
lifelessmars: because we interoperate with the rest of the python world22:29
lifelessmars: I would simply say 'be roughly PEP8, make it readable, thanks.'22:30
gary_postercr3: maybe I don't have privs to see?22:30
jmlPEP 8 (alternatively, bzrlib's or Google's Python standard)22:30
cr3gary_poster: maybe I've been working too hard on this and I forgot to push that branch :)22:31
cr3gary_poster: it's done now, sorry about that22:32
gary_posterjml, lifeless, mars, changing coding standards in the middle of a codebase, particularly when you are talking about things that affect API, like camelcase versus underlines, doesn't seem like a great idea to me, but...eh, flyby :-) .22:32
jmlgary_poster, yeah, that's the other issue22:32
gary_postercr3: got it, thanks :-)22:32
jmlgary_poster, especially now that we've got an external API to preserve22:33
gary_posterwell, we can adjust that22:33
gary_posterbut still22:33
gary_posterunless we have a magic "change everything at once"22:33
gary_posterI'm -122:33
gary_posterWhich is funny in a way22:34
gary_posterBut I need to move on :-)22:34
cr3jml: I've I'm writing a brand spanking new API, is there anything I should consider from this conversation about coding style?22:34
jmlcr3, no.22:34
gary_posterPEP 8 :-)22:34
lifelesscr3: make it tasteful.22:34
lifelessgary_poster: changing public apis has consequences, yes.22:34
cr3lifeless: to me, it tastes just the same camelcase vs underlines22:34
cr3lifeless: ... as long as it's consistent with the context22:34
lifelessuse whitespace then, to add flavour22:34
* cr3 adds salt (random bits) to taste22:35
gary_posterdon't give me that branch, cr3 ;-)22:36
gary_postercr3: maybe you haven't checked in things to download-cache?22:36
gary_postermissing wsgi-interceptm for instance22:36
gary_postercan work around22:37
jmlanyway, my yak stack is pretty deep with things that actually bug me about LP development. coding standard is well low.22:38
cr3gary_poster: installed as package, I should add to setup.py22:38
gary_postercr3: ok.  got it as egg here22:38
* jml is hanging around until PQM says my branch has landed22:40
cr3gary_poster: ./bin/test, are you getting the same error as me?22:40
pooliehello all22:41
gary_postercr3, doing it a different way, but I think so :-)22:41
lifelesspoolie: hi22:41
pooliehi lifeless22:41
lifelessearly for you? Anyhow I could use a hand22:41
_mup_Bug #634342: need a features 'scope' for page ids <qa-ok> <Launchpad Foundations:Fix Committed by lifeless> <https://launchpad.net/bugs/634342>22:41
poolieslightly early22:41
poolieyes i saw22:41
pooliei would presume from that output that it is getting the value for memcache in that scope22:41
poolieunless you defined it at a lower prioirity than the default22:42
poolieclearly there's room for better docs and tests there22:42
lifelessthere was only one rule defined in the db22:42
cr3gary_poster: regarding wsgi.intercept, I'll probably be testing it differently once I get the bin/test script running22:42
lifelesspoolie: is 0 ultimate high or low pri ?22:42
pooliei will file a bug22:42
rockstarlifeless, https://code.edge.launchpad.net/~rockstar/launchpad/build-farm-job-constraint/+merge/3621922:42
rockstarlifeless, I'd be interested in your opinion here.22:42
lifelessrockstar: I need to pop out to the shops briefly, will look when I return22:42
poolieit uses the rule that is relevant and has the numerically highest priority22:43
poolieso 0 is ultimate middle :)22:43
poolieyou could go negative22:43
pooliebut i imagined we would use 0 for the defaults22:43
rockstarlifeless, ack.22:43
pooliehello gary_poster, rockstar22:43
gary_posterhey poolie :-)22:43
rockstarpoolie, hi.22:43
jml~30 mins :(22:50
jmlg'night all.22:50
pooliehi jml! sleep well22:50
poolielifeless: i'm trying to work out what the doc/usability bug should be22:51
leonardrlifeless: fyi, after merging your branch my branch ran ec2 with no test failures, but for some reason the ec2 run failed anyway22:52
leonardrTests failed (exit code 1)22:52
leonardrmake: *** [check] Error 122:52
pooliewell, https://bugs.edge.launchpad.net/launchpad-foundations/+bug/64475122:52
_mup_Bug #644751: should be easier to tell which flags are active <flags> <Launchpad Foundations:Confirmed for mbp> <https://launchpad.net/bugs/644751>22:52
pooliefeel free to add more22:52
marsrockstar, poolie, since you are on the topic of feature scopes: the docs don't say how to install a scope or feature for just one test.  Did either of you have an idea about how it could be done?22:55
marsrockstar, unping22:56
marslifeless, that last was meant for you22:56
pooliei'm just replying to noodles's mail about exactly that22:56
marshaven't looked today - on lp-dev?22:56
gary_postercr3: ah-ha!  You have encountered https://bugs.edge.launchpad.net/ubuntu/+source/python2.6/+bug/621348 !22:56
_mup_Bug #621348: .pth and .egg-info files in /usr/lib/pymodules/python2.6 are not being processed <python2.6 (Ubuntu):New> <python2.6 (Debian):Unknown> <https://launchpad.net/bugs/621348>22:56
gary_postercongratulations! ;-)22:56
marspoolie, ok.  The other thing I was wondering - why do I need a full database to fire up a feature?  Can we just do a simple dict-based controller for that?22:57
cr3gary_poster: reading and scrutinizing for workaround :)22:57
gary_postercr3: :-) workaround I would suggest is to use the packaged eggs.  Would that work for you?22:57
marspoolie, or some way to install a custom FeatureController that is just a dict :p.  Maybe a bit more with scopes if need be.22:58
gary_postersorry, "packaged eggs" was two mistakes in two words cr322:58
gary_posterI meant the .tgzs available from PyPI22:58
gary_posterOr launchpad in this case22:58
pooliemars istm that for most tests you shouldn't care about the rules and scopes22:58
cr3gary_poster: packaged eggs just for lazr.restfulclient?22:59
poolieyou just want to say "with active_features([blah, blah, blah])"22:59
cr3gary_poster: err, "tgz" just for lazr.restfulclient? :)22:59
gary_poster:-) yeah.  is that acceptable cr3?  I'd actually recommend all the lazr things be from Python source packages (tgzs)22:59
gary_posteryou could do that probably pretty easily23:00
marspoolie, "with active_features()" - is that documented?23:00
marspoolie, because I have yet to see it23:00
cr3gary_poster: I tried that earlier and got something about not being able to find distribution for elementtree, but that might've been because of the wsgi stuff. let me try again real quick23:00
* gary_poster needs to turn into pumpkin, cr323:00
pooliemars, that was science fiction23:01
pooliei'm saying that's what the api should be23:01
poolielike a famous announcement from our ex-government telco "no email should have been lost during this upgrade"23:01
poolieyeah, "shoudl"23:01
cr3gary_poster: darn, I'll try me best then and try to get some rest even if it doesn't work :)23:02
cr3Error: Couldn't find a distribution for 'elementtree'.23:02
cr3crap, no clue where that's coming from23:02
gary_postercr3: ack.  Send me an email (and a branch!) and I'll look at it tomorrow morning23:02
gary_posterlet me try23:03
marspoolie, so Michael's helper is a start.  It would be nice to break the database dependency, but that can be done later23:05
gary_postercr3, worked for me23:05
pooliei'll try to do that in my next slice on it23:05
pooliegot to do some bzr things today23:05
cr3gary_poster: 1. added lazr.restfulclient-0.10.0 to download-cache; 2. added lazr.restfulclient to eggs under [buildout]; 3. make build?23:06
gary_postercr3: http://pastebin.ubuntu.com/498023/23:07
reed|workbryceh: Why not just fix the link to go to the non-dupe bug?23:07
gary_poster8-9: hack to get packages23:07
reed|workas in, go up the chain23:07
gary_poster4-15 also23:07
gary_poster14-15 I mean23:07
reed|workbryceh: and update launchpad with the actual bug23:07
reed|workrather than the dupe23:07
gary_poster23 says :don't get my dependencies from site-packages23:07
gary_poster31 is a convenience to get an interpreter to play around with23:07
gary_poster41-43 should mean that you can reinstate allow-picked-versions = false23:08
brycehreed|work, right that's how I'd *like* to fix it23:08
gary_postercr3: argh import still fails23:09
gary_posteralso, it didn't get packages; looking23:09
gary_posterneed to go RSN23:09
reed|workbryceh: I was just reading your initial commit, which didn't seem like a very good fix ;)23:09
brycehreed|work, unfortunately launchpad appears to capture the fact that the watched bug has status DUPLICATE, but afaict does not record what the remote dupe bug was23:09
cr3gary_poster: at least I'm reassured this is not a trivial problem :)23:09
reed|workso, can that be fixed?23:09
gary_postercr3: well, the debian one isn't.  this is suposed to be23:10
brycehreed|work, aside from fixing a symptom rather than root cause why do you think it is a bad fix?23:10
pooliejohn has a big, useful patch stalled23:10
reed|workbecause launchpad won't be showing the *real* status of the upstream bug23:10
poolieif i read it, will someone from the lp team join me?23:10
reed|workwhich is what this is supposed to be providing23:10
reed|workit will just be showing the status of some dupe bug that nobody upstream is looking at23:11
jmlpoolie, I can try, but can't promise anything this week.23:11
brycehreed|work, I think you're jumping the gun a bit.  ;-)23:12
brycehreed|work, as I see it there are three possible ways to address the complaint23:12
brycehreed|work, two of the three are fairly trivial, and I did implementations of both of those23:12
brycehthe third, making the bugwatch auto-update, is not as easy as it seems23:12
brycehbecause while we capture the fact that the upstream bug is marked RESOLVED DUPLICATE, we do not capture the identity of the bug it was duped against23:13
gary_postercr3: your builout doesn't actually install your software :-P :-)23:13
brycehin any case, I have emailed gmb for advice on how to do that third approach, and shown him the other two23:13
cr3gary_poster: if I can't run the tests, it's a feature that it doesn't actually install :)23:14
bacmars, gary_poster: lucid_lp builder failed with "Cannot rebuild database. There are 1 open connections."  is that fixable by a forced rebuild?23:14
reed|workbryceh: where is the code that pulls from bugzilla?23:14
reed|worklike, pathwise or filewise23:14
* reed|work is pulling launchpad23:14
marsbac, yes, force rebuild23:14
marsbac, it could take a few tries to clear it23:15
brycehreed|work, recall the original complaint was that mozilla was feeling spammed by updates to dupe bugs; that complaint is easy to solve23:15
bacmars, gary_poster: thanks23:15
reed|workpeople are also whining about launchpad touching old bugs23:15
* gary_poster just watched, and started to reply23:15
reed|workthat haven't had activity forever23:15
reed|workbut I care less about that23:16
bacmars:  it just started and says "ETA of 2h 14m".  what is it smoking?23:16
brycehreed|work, well like I said that was probably just because we started tracking Importance now23:16
reed|workbryceh: sure23:16
reed|worksee those two bugs23:17
marsbac, I think that is close to normal now.  Might be the time from the last run?23:17
reed|workthose are complaints people have been having23:17
brycehreed|work, heh when my son whines we make a point not to give him what he's whining for so we don't reinforce the behavior ;-)23:17
marsbac, there is a lot of overhead in there besides running the suite23:17
gary_posterlol bryceh23:17
bacbryceh: do you squirt him with water?23:18
reed|workbryceh, that works, except when he's right.23:18
brycehbac, in the bathtub and he loves it ;-)23:18
brycehreed|work, anyway just stay tuned, I'm way ahead of you ;-)23:18
* bac just watched the big bang theory "positive reinforcement" episode...23:19
brycehbac, lol23:19
reed|workbryceh, you saw the complaint about launchpad re-adding the link when people remove it, right?23:20
reed|workespecially if it's wrong23:20
brycehreed|work, no didn't see that23:20
reed|worksee the two bugs I linked above ;)23:20
brycehreed|work, but having just looked at the update code I know why - if the field is blank it updates it23:21
brycehreed|work, obvious workaround would be to just put garbage in the field and it won't update.  "NOTHING" would do it23:21
reed|workwait, which field?23:21
reed|workyou can't put "NOTHING" in a see also field23:22
reed|workin bugzilla23:22
brycehwell, any value except blank would prevent launchpad from filling it in23:22
gary_postercr3: 21 minutes late, must go!  but this works.  http://pastebin.ubuntu.com/498032/ .  Builds a whole buncha buncha stuff.  Needs more cleanup.23:22
reed|workyeah, can't put anything but urls, and specific URLs at that23:22
gary_posterThere are some similar probs in your cfg23:22
reed|workso, that's not really a fix or workaround23:22
brycehreed|work, anyway, you'd need to raise this with deryck, I'm just hired gun here :-)23:22
gary_posterThat is, related to not actually building your package23:23
gary_posterCan look tomorrow cr323:23
cr3gary_poster: thanks, I'll catchup with you tomorrow!23:23
reed|workderyck around, or is he gone for the day?23:23
brycehreed|work, sounds like there'd need to be some way to record "not bug such-and-such" that would clue launchpad in23:23
brycehreed|work, he's probably gone for the day23:23
reed|worklet me think how that could work23:24
rockstarthumper, I'm going to go for walk.  I'll be back in a bit.23:26
lifelessmars: what was for me ?23:26
lifelessleonardr: was there any other data?23:26
marslifeless, leonardr's run of the patch23:27
marslifeless, oh, the other thing - the API suggesting.  Martin answered my question with artistic fiction - can't easily top that :)23:29
mkanatgmb: Do you have a Mozilla Bugzilla account?23:33
mkanatLooks like not.23:33
reed|workyes, he's graham@canon23:33
reed|workI cc'd him on the first bug23:34
reed|worknot the second one yet23:34
lifelesspoolie: so I'm still confused about the scopes bug thing; its not getting the value back.23:34
lifelessmars: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/63188423:35
_mup_Bug #631884: feature flags get out of sync easily <qa-untestable> <Launchpad Foundations:New for lifeless> <https://launchpad.net/bugs/631884>23:35
marslifeless, eh?23:35
wgrantjtv: You should probably warn people that setting a contact address will break all their bugmail.23:35
=== matsubara is now known as matsubara-afk
lifelessmars: something to be aware of if working up poolies sci fi23:37
marsright.  I think noodles' patch covers that23:37
marslifeless, would the helpers go in lp.testing, lp.testing.features, or lp.features.testing ?23:39
marsI need some help with the convention here23:39
marsor on lp.testing.TestCase as a helper method23:40
lifelesspoolie: the memcache code still runs, so I think that the 'disabled' is not propogating correctly.23:40
marsnah, that wouldn't work for views23:40
marslifeless, sorry to bug you with the questions, but you are the Test Engineering Ninja for this shift :)23:41
marsgreat, thanks23:41
lifelesslp/services/features/testing.py I mean, obviously ;)23:41
wgrantjml: Yay, you're killing buildd-slavescanner.txt23:41
jmlwgrant: a letting.23:43
jmlwgrant: a little, I mean.23:43
jmlbrain no longer connected to fingers. good night :)23:44
=== salgado is now known as salgado-afk

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!