/srv/irclogs.ubuntu.com/2011/09/05/#launchpad-dev.txt

* wgrant fires off a full test suite run in an oneiric container.00:23
wgrantHmm.00:25
wgrantmailman does not like the distutils installation :(00:26
wgrantAhh.00:26
* G looks to see who the OCR is today00:26
wgrantG: I'm not, but what needs reviewing?00:29
Goh, not online yet00:29
Gwgrant: I've got a branch for Bug #69715700:29
_mup_Bug #697157: Streamline "You have assigned this bug to yourself" message  <bugs> <trivial> <ui> <Launchpad itself:In Progress by dev-nigelj> < https://launchpad.net/bugs/697157 >00:29
Gexisting message for all cases except "You have assigned this bug to yourself" if you assigned it to yourself00:30
GI'm just going to propose it00:30
Gwgrant: https://code.launchpad.net/~dev-nigelj/launchpad/bug-697157/+merge/7403500:44
wgrantG: That seems like it will miss the case where it's assigned then unassigned quickly, once we move to queuing these (as we should). But you probably don't have enough information to detect that, so I guess it's reasonable.00:47
Gwgrant: hmmm I hadn't thought of that case00:48
Gwgrant: that said, wouldn't the method that e-mail is generated with change slightly if/when that happens00:49
GI didn't poke too much into how the scripted bugmail is generated, so I'd have to take another look at that but just thinking along that line00:50
=== mwhudson_ is now known as mwhudson
=== wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 254 - 0:[#######=]:256
StevenK /wrists01:20
wgrantAnd since we probably won't deploy for 48 hours or so, we should break the graph shortly :)01:21
Gouch01:47
StevenKwgrant: Declaring destroy-openidrp ready for a review. I will poke stub when he surfaces.01:49
wgrantStevenK: But it doesn't have DB changes, does it?01:52
StevenKwgrant: It does not, but stub is OCR. :_P01:54
GI'm heading out, can someone ask stub to have a look at my MR?01:55
StevenKwgrant: Do you think it's safe or unsafe to destroy the bazaar-experts team?01:57
wgrantStevenK: Safe. We've deployed everything except librarian1 since then, AFAIK.01:58
wgrantAnd if librarian1 needs bazaar-experts, then it should blow up, because something is seriously screwed.01:58
wgrantwallyworld_: Have you tried to run tests on oneiric yet? rabbitfixture doesn't support rabbitmq-server 2.5.0.03:01
wgrantI think I have a fix.03:01
wgrantNeed to test it on lucid too, though.03:01
wallyworld_wgrant: i've only run some yui tests and they went ok03:02
wallyworld_but nothong with rabbit03:02
wallyworld_wgrant: i'm pissed off ay qas. it keeps timing out for a simple operation :-(03:02
wgrant:(03:03
wallyworld_hard to know whether to qa-ok it. the sql is fine03:04
wgrantwallyworld_: Is this the bug privacy thing?03:09
wgrantThat's not a simple operation :)03:09
wgrantAha, got a rabbitfixture regex that works on both lucid and oneiric.03:09
wgrantBut is a bit evil.03:09
wgrantTest attempt #203:11
wallyworld_wgrant: yes, the bug privacy thing. the sql before and after the change is similar, except the fix has a couple of extra selects to update the subscription portal.03:16
wallyworld_wgrant: it works fine if i edit a bug that's on a smaller project than lp03:17
wallyworld_so i'm marking it as ok03:17
wallyworld_but, in testing, i found a bug that i confirmed is also on lp.net03:17
wallyworld_but it's not a result of any disclosure work so i'm not sure if we'll end up fixing it03:18
wgrantwallyworld_: What's the bug?03:36
wallyworld_wgrant: bug 84151103:36
_mup_Bug #841511: javascript error when changing a bug's security status <regression> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/841511 >03:36
wallyworld_wgrant: it's in the same area of code as i was qa'ing so i had to look at diffs and also try it out on lp.net and found that it's an unrelated problem03:37
wallyworld_i should just assign myself to the bug and fix it03:38
=== wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 255 - 0:[#######=]:256
StevenKSomeone promote a high to critical ...04:06
wgrantI'm just looking for one.04:07
nigelbMorning04:07
nigelb255? :(04:07
=== wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 257 - 0:[########** stack smashing detected : ./lp terminated
nigelbBAH!04:11
* ajmitch thought that number was meant to be going down04:12
nigelbActually, I'm not sure if 256 was right.04:12
nigelbBecause there were bugs fixed and newer bugs added.04:12
wgranthttp://webnumbr.com/launchpad-critical-bugs is what that number goes by. I just added two because I bumped two OOPSes that had been forgotten to critical.04:13
wgrantThe real number is significantly higher, as several are private.04:13
nigelbNo, I meant the 256 that acted as the target from which you startred04:14
wgrantAh/04:15
nigelbNow, it seems as though there was no progress.04:16
nigelbThat's bad.04:16
wgrantThere was no progress; that is bad :)04:16
nigelbHuh?04:17
nigelbBut Francis had this graph which said there were bugs closed as well as new ones opened.04:17
wgrantWell, yes, lots were closed. But lots more were found, and not all of them were new.04:17
nigelbI refuse to believe there was no progress. You guys aren't /that/ bad ;)04:17
pooliethere has been progress, things have improved, but they haven't progressed in the intended direction of reducing the number of criticals04:26
nigelbAh :(04:30
Gso what, focus urgently on Critical bugs eh?04:34
nigelbI spent two hours on this bug yesterday, 6634504:37
nigelbeventually I realized its probably a bug in that library.04:37
wgrantBug #6634504:37
_mup_Bug #66345: alt="" in dependency charts makes them poorly accessible <easy> <lp-blueprints> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/66345 >04:37
wgrantHmm.04:38
* StevenK looks for a critical, since the DSP vocab work is still blocked, and his two other branches are waiting for others04:38
* G spent ages trying to understand the API to fix a Medium bug, must try again now04:39
StevenKIsn't bug 669296 mostly fixed?04:41
_mup_Bug #669296: App servers die and hang <lp-foundations> <Launchpad itself:Triaged> < https://launchpad.net/bugs/669296 >04:41
wgrantStevenK: three died over the weekend.04:41
wgrantBut it is much better now that logrotation is fixed.04:41
wgrantFirst time it's happened in weeks, I think.04:41
wgrantRarrrgh buildbot is broken again.04:42
wgrantTempting to roll back the three bzr revisions from the last couple of weeks.04:42
wgrantHmm. Except that this looks like a dependency issue.04:42
nigelbIs this okay to fix? bug 9959304:43
_mup_Bug #99593: Package names in search results should be linked <bug-columns> <easy> <lp-bugs> <search> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/99593 >04:43
nigelbI'm not sure if I want it fixed :D04:43
wgrantnigelb: The next scheduled feature involves reworking bug listings.04:43
wgrantAh, and it's even tagged (bug-columns).04:44
wgrantSo I would avoid it.04:44
nigelbokay :)04:44
* nigelb is /very/glad he asked04:44
nigelbI can't reproduce bug 750384 in my setup.04:46
_mup_Bug #750384: After muting with Bug:+mute, you get redirected to +subscribe <story-better-bug-notification> <trivial> <Launchpad itself:Triaged> < https://launchpad.net/bugs/750384 >04:46
nigelb(devel)04:46
nigelbCould someone else confirm so I can close it.04:46
Gyeah I looked at that one04:46
Gcan't reproduce either04:46
nigelbExcellent.04:46
GI meant to note it, but couldn't help but think I was doing something majorly wrong :)04:47
StevenKpoolie: O hai -- you reviewed the Twisted patch for bug 703807, should we cherry-pick it, or wait for a upstream release?04:47
_mup_Bug #703807: launchpad sometimes serves download files as content-type text/html <regression> <Launchpad itself:Triaged> <pyOpenSSL:Invalid> <Twisted:Fix Released> <twisted-web (Ubuntu):Confirmed> < https://launchpad.net/bugs/703807 >04:47
pooliehi stevenk04:47
StevenKI have this feeling that there are multiple bugs about that issue04:47
pooliegood question04:47
pooliewell, it has certainly been duped several times04:48
poolieand perhaps caused some confusion or questions short of filing a bug04:48
poolieso python at the moment comes from an egg?04:48
pooliestevenk i guess the questions are04:50
poolie- how soon will we get an upstream release04:50
StevenKbug 174224 looks like a dupe04:50
_mup_Bug #174224: launchpadlibrarian sends wrong content-type header <librarian> <lp-foundations> <Launchpad itself:Triaged> < https://launchpad.net/bugs/174224 >04:50
poolie- how much do we care about getting it done soon vs later04:50
poolie- how much more work is it to cherry-pick?04:50
StevenKpossibly bug 64592404:50
_mup_Bug #645924: .tar.xz files are served with text/plain content-type by launchpadlibrarian <content-type> <launchpadlibrarian> <lp-foundations> <mime-type> <xz> <Launchpad itself:Triaged> < https://launchpad.net/bugs/645924 >04:50
wgrantStevenK: None of those are dupes.04:51
pooliein the thing i helpde fix in twisted, it would specifically give you text/html i think04:52
wgrantRight.04:52
pooliei suspect the first of them is just obsolete04:53
StevenKpoolie: So, I don't think we can answer how soon we get an upstream release; I'm not sure about making a cherry-pick release of Twisted ...04:54
StevenKIt is easier to just wait04:54
nigelbCan someone help me understand bug 294050?04:55
_mup_Bug #294050: Please add links for entries in the revision feed for a person <feeds> <lp-code> <trivial> <Launchpad itself:Triaged> < https://launchpad.net/bugs/294050 >04:55
nigelbI think it means the feeds in branches.04:55
nigelbBut I already see links to loggerhead04:55
pooliewhoa, feeds04:55
nigelbheh, that's not improving my sinking feeling :)04:56
poolieso, i can help you understand it04:56
nigelbExcellent.04:57
StevenKOH! I know what I can remove04:57
pooliewhich is that basically it's about inserting a hyperlink, i guess pointing to loggerhead, into the revision feed data04:57
StevenKDELAYED COPIES04:57
pooliehowever, i don't know if revision feeds are a very useful feature04:57
pooliei never look at them any more04:57
nigelbpoolie: I already see it everywhere I look.04:57
poolieoh, in the feed?04:57
nigelbyeah04:57
pooliethen maybe you should just close this04:58
nigelbso, is it something wrong in how I perceive the bug.04:58
nigelbFor example, in here I can see links http://feeds.launchpad.net/~launchpad-pqm/launchpad/devel/branch.atom04:59
nigelbAha, let me close the bug then :)04:59
pooliego you04:59
poolieStevenK, i can still reproduce bug 64592404:59
_mup_Bug #645924: .tar.xz files are served with text/plain content-type by launchpadlibrarian <content-type> <launchpadlibrarian> <lp-foundations> <mime-type> <xz> <Launchpad itself:Triaged> < https://launchpad.net/bugs/645924 >04:59
pooliei suspect it actually does have c-t text/plain in the librarian05:00
StevenKpoolie: What's the LFA ID?05:00
pooliei guess it's 56261240 from the url?05:00
StevenKSELECT mimetype from libraryfilealias where id = 56261240; => text/plain05:01
poolieyep05:01
wgrantSo the sniffer is (or was) bad.05:01
poolieso05:01
poolieif you add a new upload what does it get?05:01
wgrantI don't recall if we trust the browser or not.05:01
* poolie looks at other .tar.xz files05:03
StevenKHold on, I can do it faster05:03
pooliei'm looking through the staging db05:03
poolieso, only if you can type faster05:04
poolieor know the schema better05:04
poolieeither is possible05:04
StevenKI have a query running, so let's see :-)05:04
StevenKSELECT distinct(mimetype) from libraryfilealias where filename like '%.tar.xz';05:05
pooliealmost all are application/octet-stream, it seems.05:05
StevenKapplication/octet-stream and text/plain05:06
GI've been looking at https://bugs.launchpad.net/launchpad/+bug/674854 can't help notice that https://api.launchpad.dev/devel/ubuntu/hoary/architectures is a bit funny... total_size: 3, entries: []05:06
_mup_Bug #674854: [API] Add a way to get the list of "supported" architectures for each release <api> <lp-soyuz> <trivial> <Launchpad itself:Triaged> < https://launchpad.net/bugs/674854 >05:06
StevenKxz doesn't exist in /etc/mime.types05:06
StevenKapplication/octet-stream is debatable, but loads better than text/plain05:06
wgrantG: DistroArchSeries doesn't have a launchpad.View permission defined.05:06
Gwgrant: ahhh thanks05:07
wgrantG: Since they're all public, they need an adapter class deriving from AnonymousAuthorization (in canonical.launchpad.security)05:07
wgrantG: There's possibly already a bug for this.05:07
wgrantG: (they'll already appear if you're logged in, just not to anonymous users)05:07
poolieanyhow, it's not a dupe05:08
poolieif launchpad is sniffing the type, it seems to be mostly getting it right now05:09
Gwgrant: apparently not05:09
StevenKpoolie: So, we can probably correct that mimetype and close the bug, then05:10
Gwgrant: http://pastebin.ubuntu.com/682351/ - but I will look at existing DistroARchSeries bugs05:11
pooliei guess05:11
pooliejust fix it in the db?05:11
poolieironically that really may interact with the caches05:12
pooliebecause this is exactly the unusual cache of the ct changing without the content changing05:12
poolieso it will probably stay cached for a while05:12
pooliearguably there's a bug here if lp trusts the client's ct even when it shouldn't05:12
poolieStevenK: so how would we handle a cherrypick to something that currently comes from an egg?05:13
Gwgrant: oh I get you now05:14
Gwgrant: thanks for the pointer :)05:15
StevenKpoolie: We make a new egg and update the version in versions.cfg05:15
pooliecalled 0.12.0+0launchpad1 or something?05:15
poolieok05:15
wgrantStevenK: Aha, I know how to fix the Jenkins failure, and I'm just going to redisable the failing buildbot test (it was only reenabled a weekish ago)05:15
wgrantG: Great. If you run into any issues, ask :)05:16
Gwgrant: yeah, although I do like to have a big play w/ stuff before asking, because if it is a simple thing, I learn other stuffs/tidbits about the code by stuffing up :)05:17
StevenKpoolie: I hope not. :-)05:17
StevenKpoolie: Since the version of Twisted we have currently is 11.0.0 :-)05:18
poolieyou know what i mean05:18
StevenKWe have done cherry-picks of Twisted before, I just can't remember their format05:18
Gwgrant: so really, http://pastebin.ubuntu.com/682356/ is all that is needed if I understand it correctly05:22
wgrantjelmer: Ah, worked out that bzrdir import error05:23
wgrantjelmer: The daemon was old, from a previous test run.05:23
wgrantjelmer: So its tree had been deleted.05:23
wgrantG: You're missing a newline, but yup.05:23
Gmissing, don't you mean included by accident?05:24
poolieStevenK: so can i leave that with you?05:24
pooliei'm going to try to fix bug 643223 this afternoon05:24
_mup_Bug #643223: should accept dkim based on from address and signing address belonging to the same person <dkim> <lp-foundations> <mail> <Launchpad itself:Triaged by mbp> < https://launchpad.net/bugs/643223 >05:24
wgrantG: Two empty lines between classes.05:24
Gwgrant: oh good point, thanks :)05:25
GI am also missing something else...05:25
Ga test!05:25
wgrantMmm.05:25
wgrantHah. Old bug for test_import_bzrsvn's disablement was 541526. New one is 841556. A bit too similar...05:38
poolie4 out of 14 sampledata people still work at canonical05:39
poolienot counting foo bar05:39
nigelbis that good or bad? :)05:40
pooliei don't know05:40
poolieit's pretty old05:40
poolienot too bad05:40
pooliei know all but 2 of them05:40
nigelbsampledata = stuff that rf installs onto my machine right?05:41
poolieyes05:41
pooliedatabase/sampledata05:41
nigelbI only saw mark as a familiar name05:41
StevenKNo, the sampledata is stuff that make schema stuff into the development database05:41
StevenKrf installs the developer dependencies05:41
nigelbAh.05:42
pooliewell, strictly, make schema not rf, yes05:42
nigelbBut poolie got what I meant :)05:42
nigelbHrm, today's Google Doodle is pretty awesome.05:49
GI don't know, I haven't been impressed by Google's Doodle's for a while05:49
pooliei had no idea he was Parsi05:55
nigelbNeither did I :)05:56
Ghmmm think I may be able to put another bug in the 'fixed' basket06:17
pooliestevenk, wgrant, in a test, i need a user account with multiple email addresses in different domains06:23
pooliewhat's the tasteful way to get this?06:23
wgrantmakePerson()06:23
wgrantmakeEmailAddress()06:23
wgrantmakeEmailAddress()06:23
wgrantmakeEmailAddress()06:23
Gone at canonical.com one at ubuntu.com in makePerson?06:23
poolieon TestCaseWithFactory?06:23
Gpoolie: yeah, there is another test like that, that I was reading06:23
StevenKpoolie: self.factory.makePerson() ...06:24
pooliethanks06:24
poolieam i right that in general you prefer tests to always make the person (or whatever) rather than relying on the sample data?06:24
wgrantpoolie: Yes.06:26
wgrantpoolie: It is somewhat slow, but means we're not relying on fragile, rotten sampledata.06:26
wgrantWhich is largely invalid in great parts.06:26
StevenKI'd like to remove most sampledatsa06:30
StevenKs/sa/a/06:30
=== wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 258 - 0:[########*** stack smashing detected ***: ./lp terminated
wgrant258 :(06:31
StevenKWhat is the most recent?06:32
wgrantBug #84155606:33
_mup_Bug #841556: test_import_cvs, test_import_bzrsvn and test_import_subversion disabled <test-system> <Launchpad itself:Triaged> < https://launchpad.net/bugs/841556 >06:33
Gwgrant: I got it! http://bazaar.launchpad.net/~dev-nigelj/launchpad/bug-674854/revision/13863 allows me to do: http://pastebin.ubuntu.com/682386/06:36
Gwgrant: thanks massively for your help06:36
StevenKpoolie: This cherrypick is not so easy, since the diff from trac won't apply at all to 11.0.006:40
pooliemy patch for 721166 passed an ec2 test06:46
pooliebug 72116606:46
_mup_Bug #721166: Tests sometimes fail on EC2 due to _LockWarner garbage <build-infrastructure> <spurious-test-failure> <Launchpad itself:In Progress by mbp> < https://launchpad.net/bugs/721166 >06:46
poolieshould i land it?06:46
pooliei think the tests can now be reenabled since lp has bzr 2.406:47
StevenKDidn't wgrant just roll that back?06:47
poolierolled back bzr?06:47
wgrantI didn't roll back bzr.06:48
wgrantI redisabled the undisabled tests.06:48
pooliethe ones that gary changed?06:48
pooliethey might have been different06:48
wgrantNah, different ones.06:50
wgrantDid he actually end up disabling any?06:50
wgrantI think the bzrdir import issues were due to the three I just disabled.06:50
wgrantLeaving ancient processes around.06:50
poolieso my question is, really06:51
poolieif i land this, is there an unreasonable chance that either bzr will be downgraded or spurious failures will disrupt things?06:51
wgrant15 minutes ago the chance of a spurious failure disruption was probably around 0.95.06:51
wgrantNow it's hopefully lower.06:51
wgrantBut please avoid any even slightly disruptive changes until late in the week.06:52
wgrantWe've just had about 20 failures.06:52
poolieok, there's no rush06:52
poolieit's only a tidying-up fix06:52
wgrantWhat with the three invasive bzr infrastructure changes lately, it's a bit of a mess.06:52
poolieoh, what were the three?06:53
wgrantThere was the 2.4 upgrade, the puller SafeBranchOpener unification, and some other codeimport/puller change, the details of which I forget.06:53
pooliedid they break a lot of stuff?06:54
wgrantIt's not clear.06:54
wgrantBecause also around that time three tests were undisabled.06:54
wgrantThey spuriously failed in spectacular ways occasionally.06:54
wgrantBut then lots of other tests blew up in even more spectacular ways.06:54
wgrantbut I speculate that that is because of debris left by the original three broken tests.06:55
wgrantSo I've disabled them, will get buildbot cleaned up, and then hope that stuff stops failing dammit.06:55
wgrantOtherwise I'll just revert all the bzr changes in the last two weeks and work out wth is going on.06:55
poolieStevenK: you could just take my patch from http://twistedmatrix.com/trac/ticket/4156#comment:707:00
wgrantStevenK: How do I restart a Jenkins build? Just kill the executor and force?07:10
* wgrant tries.07:10
LPCIBotProject devel build #1,031: ABORTED in 21 min: https://lpci.wedontsleep.org/job/devel/1031/07:11
wgrantWe may have success in 4 hours.07:11
LPCIBotProject devel build #1,032: STILL FAILING in 6 min 34 sec: https://lpci.wedontsleep.org/job/devel/1032/07:17
wgrantLies.07:19
wgranturllib2.URLError: <urlopen error [Errno 2] No such file or directory: '/mnt/test/launchpad/workspace/devel/download-cache/dist/setuptools-0.6c11-py2.6.egg'>07:19
wgrantHow unpleasant.07:19
wgrantStevenK: You broke everything.07:20
wgrantSeems I can't destroy the executor.07:20
wgrant:(07:20
LPCIBotProject devel build #1,033: STILL FAILING in 39 sec: https://lpci.wedontsleep.org/job/devel/1033/07:21
wgrant:(07:22
Gouch, that doesn't sound good07:25
wgrantWell, the test suite has sped up by 360x. That can only be a good thing, right? :)07:26
rvbahehe07:26
Geek GPG tests just OOM'd in my VM07:27
wgrantHow much RAM is assigned?07:27
Gand that is 2GB RAM07:27
wgrantamd64?07:27
wgrantThere may be left over librarians and stuff eating a couple of hundred megabytes each.07:28
wgrantamd64 with 2GiB is doable, but pushing it.07:28
Gyeah, amd6407:29
GI might download lucid-server i386 and put on that VM tbh07:29
Glaptop is Core2Duo w/ 4GB07:30
wgrantMy laptop with the same specs runs it OK outside a VM, but if I want to do it in a VM i386 is better.07:33
nigelbIs bigjools going to be around today?07:36
nigelbThere's a bunch of bugs that I want to talk to him first before touching :)07:36
adeuringgood morning07:38
wgrantnigelb: I think he should be.07:38
wgrantlet me check.07:38
Gwgrant: yeah, don't blame you for using i38607:39
wgrantnigelb: Looks like he should be here today.07:41
rvbanigelb: Yes, he will be here today.07:42
nigelbwgrant / rvba: Thanks!07:42
LPCIBotProject devel build #1,034: STILL FAILING in 44 sec: https://lpci.wedontsleep.org/job/devel/1034/07:46
Gwgrant: fast but still failing?07:48
wgrantYeah, the slave is broken.07:50
wgrantNeed StevenK to fix.07:50
StevenKYou do?07:55
=== almaisan-away is now known as al-maisan
wgrantStevenK: I can't work out how to kill the executor.07:57
wgrantOnly the build.07:57
pooliewould it be tasteful to add a script to lp that just processes one mail message from stdin?07:58
poolieat least for the sake of testing07:58
StevenK Oh. Delete the slave.07:59
poolieit seems a lot easier than configuring a mailbox07:59
wgrantpoolie: It is distasteful to not have one.07:59
wgrantStevenK: warthogs doesn't seem to have permission to do that.07:59
poolieor, to add a parameter to process-mail to do this (though that seems a bit hard)07:59
wgrantOr at least I can't find a relevant button.07:59
poolie:)07:59
StevenK Navigate to the slave, hit the button on the left.08:01
wgranthttps://lpci.wedontsleep.org/computer/i-225ca042/ has only four links on the left.08:01
wgrantBack to List08:01
wgrantStatus08:01
wgrantBuild History08:01
wgrantLoad Statistics08:01
pooliewgrant, ok, will do08:05
pooliewe could potentially even get the mta/mda to feed messages directly in to this08:05
pooliethough startup time may make that suck08:05
wgrant"may"08:06
pooliewell08:06
pooliewill it be worse than the current latency from running from cron?08:06
poolieand, will the wasted cpu matter?08:06
pooliei don't know how busy the machine that currently does this is, and how much mail it gets08:07
wgrantIt's about 10s of CPU time per invocation.08:07
wgrantThat's not insignificant.08:07
poolietrue08:07
poolieespecially if we get mail more often than every 10s during peak time08:07
wgrantYes.08:07
poolieyes we do?08:08
wgrantAnd if someone sends an email to a few bugs, the world ends.08:08
pooliehmm08:08
poolieif the mail is exploded apart before it gets to us08:08
wgrantIt used to be frequent to email hundreds of bugs at once, but that is probably reduced by the API.08:08
pooliewhich it probably wouldn't be08:08
nigelbIs it known that loggerhead fails on first try to see launchpad code?08:08
pooliebut could be08:08
wgrantWouldn't it be?08:08
wgrantnigelb: Yes.08:08
poolieat any rate it's not optimized08:08
pooliei mean, it's not worth optimizing08:08
wgrantpoolie: Unless it's treating process-mail as a relay, it should be exploded.08:09
nigelbwgrant: Loggerhead suckage?08:09
poolieyes, just that08:09
poolieit needs to warm the cache08:09
wgrantnigelb: Yeah, plus LP is insanely huge.08:09
Gbigjools: hey, is https://code.launchpad.net/~dev-nigelj/launchpad/bug-674854 anything like what you were thinking of to fix bug 674854?08:09
_mup_Bug #674854: [API] Add a way to get the list of "supported" architectures for each release <api> <lp-soyuz> <trivial> <Launchpad itself:In Progress by dev-nigelj> < https://launchpad.net/bugs/674854 >08:09
nigelbAnd, looking the launchpad devel branch also timesout on first try.08:09
nigelbProbably related I guess.08:10
bigjoolsG: why did you need the security adapter change?08:17
bigjoolsdid anon access not work?08:17
Gbigjools: correct08:18
Git would reply w/ count: 3, entries: []08:18
wgrantbigjools: Remember that lazr.restful filters items out of collections unless launchpad.View is held.08:18
wgrantbigjools: And DAS has no launchpad.View.08:18
bigjoolsG: ah bugger.  Well the bug is essentially already fixed, you could note that and/or file a new one.08:19
wgrantSo it reverts to the default one, which is checkAuthenticated -> True, checkUnauthenticated -> False08:19
lifelesswow, 2 non-staff contributors. Excellent :)08:19
Gwgrant: it didn't even work when authenticated08:19
bigjoolslazr.restful is teh awsum08:19
nigelblifeless: both of us even have the same first name ;)08:20
wgrantEvening lifeless.08:20
lifeless'lo y'all :)08:20
nigelb(For added confusion)08:20
* bigjools disappears for calls and mountains of email08:20
Gbigjools: http://pastebin.ubuntu.com/682351/08:20
Glifeless: howdy08:20
lifelessI saw a mountain of cron spam this morning08:20
wgrantlifeless: Chameleon has that effect on people :(08:20
nigelblifeless: Glad to hear that everything is now fine :)08:20
wgrantThe broken eggs need fixing :(08:20
lifelesswgrant: these are setuptools non-dpkg eggs again ? -how-08:21
wgrantlifeless: Well, it changed from a directory to a symlink.08:21
wgrantAnd dpkg doesn't seem to always handle that nicely.08:21
wgrantBut sometimes it does.08:21
wgrantAnd blah.08:21
wgrantrm -rf, apt-get install --reinstall, problem solved.08:21
wgrantBut it is way down the queue :(08:22
pooliei have a process-one-mail.py08:23
pooliethat's so much better08:23
nigelbThis reminds me.08:24
StevenKwgrant: Do I still need to delete that slave?08:24
nigelbHow does one do QA of something email related on qastaging?08:24
wgrantStevenK: Please.08:24
StevenKnigelb: One panics.08:25
wgrantAlthough it seems to be gone.08:25
wgrantIf you haven't already killed it.08:25
nigelbStevenK: Test in production? :P08:25
StevenKwgrant: If the slave sits unused for 30 minutes, it gets killed08:25
nigelbWe tested summit in production /at/ UDS :D08:26
bigjoolsnigelb: ask someone with access to the staging inbox to look at it for you08:26
wgrantStevenK: Ah, that would do it. Thanks.08:26
nigelbbigjools: Great, that means anyone from LP team right?08:26
bigjoolsnigelb: not everyone IIRC, all squad leads and assorted others08:27
wgrantI think it's meant to be squad leads. But most people seem to know the password.08:27
nigelbAnyway, I need to get that branch merged.08:27
nigelbTestfix mode was blocking it.08:27
nigelbI don't know if that's done yet.08:28
wgrantI think I've fixed it all. But the current build is missing the last two revisions. If we are lucky we will be green in 3 hours. If we're not, it will be 8 hours.08:28
wgrantAnd I may of course be wrong, and we will still be red in 8 hours.08:28
nigelbOh good, so by the time I'm home.08:28
wgrantIn which case I will be sad.08:28
wgrantAnd hope that gary fixes it before I awake.08:28
nigelbBUt you won't sleep anyway.08:30
nigelb:D08:30
pooliewoo, locally accepted second-address signed mail08:36
pooliethat feels like a much better test08:36
lifelessbigjools: inbox is accessible to all canonical I think :)08:38
Gbigjools: yeah, just tested again, the <distro>.current_series.architectures method against devel definately returns no entries, when authenticated, but this change allows it to happen, but are you saying it should be a seperate bug?08:41
G(Just a tad confused, because this seems to be the last bit a puzzle (even if the majority of the work was done prior) to get what the description etc lays out08:42
StevenKstub: O hai!08:43
pooliewgrant, well, that script is added in https://code.launchpad.net/~mbp/launchpad/643223-dkim-aliases/+merge/74061 plus some other stuff08:48
pooliei'm happy to hold off landing it until things are safe enough08:48
StevenKstub: Can you please look at the SQL in bug 834384 and comment in the bug about it?08:48
_mup_Bug #834384: StaticDiff is unused and needs to be shot <Launchpad itself:Confirmed> < https://launchpad.net/bugs/834384 >08:48
stubStevenK: commented08:59
bigjoolsG: that is a separate bug because it seems the main one was fixed some time ago09:01
bigjoolsnot sure why it's not returning anything even when authenticated09:01
bigjoolsI'd hope there was a test for this somewhere!09:02
Gbigjools: couldn't find any, which is why my branch includes a test for this logic09:02
Gbigjools: so what, I close the current bug, open a new one tagged regression?09:02
bigjoolsG: yes, I think so.  I'd check to see if there's a passing test for DS.architectures as well.09:03
Gbigjools: yes, that is what the test, is testing09:03
bigjoolsapart from you new one I meant :)09:04
bigjoolsyour*09:04
Gthat the number of entries returned via the API is the same as in the DB09:04
Goh, I couldn't find any other tests that seemed to be testing it09:04
Gbigjools: oh funny. I tried again, and it worked for an authenticated launchpadlib setup this time09:14
Gso obviously it was just never enabled for anonymous09:14
bigjoolshah09:14
GI was going through everything again unauthed, then authed to make double sure, and for clean output for the bug09:15
Gso I guess less regression, more oversight/accident?09:15
bigjoolscould be an out of date wadl09:15
bigjoolscache09:15
Gthis dev environment has only existed for half a week09:16
bigjoolsoh you were testing against a .dev instance?09:17
pooliedoes anyone know how i could get a gpg key into a dev instance?09:17
Gbigjools: yeah09:17
pooliemaybe poking it into the db09:17
wgrantpoolie: utilities/start-dev-soyuz.sh, push key to keyserver.launchpad.dev, add key as normal.09:17
Gbigjools: maybe I originally took too long to validate the OAuth session or something the first time round :)09:18
pooliethansk09:18
poolie*thanks09:18
poolieand apparently i'm going to need to do a mail roundtrip to confirm it?09:18
wgrantOr just grab the token from logintoken;09:18
wgrantThen go to /token/$WHATEVER, and confirm.09:19
Gbigjools: alright, so I'll close 674854, and the new bug will be that it doesn't allow anonymous access :)09:19
bigjoolsG: great!#09:20
pooliejust manually testing i haven't broken that09:20
poolies// gpg signing09:20
Gbigjools: okay, done, I'll propose the branch for merging now under the new bug09:26
bigjoolstoday's Google Doodle is the best ever09:37
poolieis there going to be an ocr today?09:39
poolieo/ bigjools09:39
nigelbbigjools: I agree :)09:40
bigjoolso/ poolie09:41
bigjoolsnigelb: is your 'tache in honour of Fred? :)09:41
nigelbhaha, no09:41
nigelbhrm, but I can claim so now :P09:41
Gbigjools: I guess I'm too young to know much about Freddie Mercury the doodle is fancy, but yeah...09:42
nigelbAlso, pep8 and pyflakes are awesome. I just started using it on all my python code after getting used to make lint :D09:42
bigjoolsnigelb: use vim?09:42
Gany yeah, no OCR today?09:42
nigelbyeah, I'm guessing there's a plugin as well?09:43
bigjoolsG: let's just say he was the frontman for the biggest rock band of all time :)09:43
bigjoolsnigelb: there is and it's teh awsum09:43
nigelbOh good. let me find and isntall.09:43
Gbigjools: whats a good song of his band's then, I might look at it on YouTube or something09:44
StevenKG: Are you serious?!09:44
bigjools!09:44
bigjoolsG: you've heard of Queen, right?09:44
Gyeah... Queen09:44
nigelbG has successfully made everyone feel old I believe.09:45
GAnother One Bites the Dust, We Will, We Will, Rock You...09:45
StevenKBohemian Rhapsody?09:45
bigjoolsnigelb: you're a whippersnapper yourself09:45
Gyeah...09:45
StevenKI don't feel old, just surprised that someone doesn't recognise Freddie09:45
Gohhhh he was one of the main guys?09:45
Ghaha, never knew09:45
nigelbbigjools: Hey, but I've heard of Freddie.09:45
StevenKG: He was the vocalist until his death09:46
GI'd heard of Freddie, but never knew the Freddie<->Queen connection09:46
StevenKG: For David Bowie's 50th birthday concert, they had to get a women to sing Freddie's part of Under Pressure09:46
Ghmmm according to Wikipedia, I was born before he died09:46
Gso I guess that is something09:46
nigelbG: How old are you again?09:47
Glearn something new every day eh09:47
Gnigelb: > 2109:47
nigelbHrm, we may be around the same age :D09:48
Gheck, I'd never seen the Matrix films and the Back to the Future films until a few years ago09:49
nigelb....09:49
nigelbG: Star War / Star Trek?09:50
nigelb*Star Wars09:50
Gnigelb: I've seen a couple of Star Wars films, didn't really like them that much, and only really seen the latest Star Trek movie (which I thought was okay), but I'm not generally a big fan of the whole Sci-Fi genre09:51
StevenKG: Heathan.09:51
StevenKSigh. Heathen, even.09:51
GStevenK: it's just not my thing09:52
bigjoolsif it was two from episodes 1-3 them I am not surprised, I didn't like them either :)09:52
GThe Phantom Menace was one of the ones I know I've seen09:52
bigjoolsawful movie09:53
Gbut anyway, don't hold it against me :)09:53
bigjoolsgo and dig out the original 1977 Star Wars, it's great.09:54
nigelbAmen09:54
nigelbI watched the entire Star Wars series in one day from 8 am to 8pm (I was sick and had nothing else to do)09:55
Gbigjools: is http://www.fatso.co.nz/Movies/Star_Wars_-_Episode_IV_A_New_Hope_1977/5552 the one you were talking about?09:55
bigjoolsG: yes09:55
Gokay, it's in my queue, might even get it Wednesday09:56
bigjoolsrock on09:56
StevenKnigelb: More impressive if you do it with LOTR09:56
Gmeh, Demand: High :(09:56
bigjools"PG - Contains medium level violence" - heh09:56
nigelbStevenK: I didn't like LOTR for some reason.09:56
StevenKnigelb: *Out*09:56
Gbigjools: gotta love the NZ ratings :)09:56
nigelbI tried multiple times to get myself to watch it09:56
nigelbOr read the books09:57
GI'm a Kiwi, I watched all 3 LOTR movies, and didn't really like it, much prefer the books :)09:57
bigjoolsI faltered half way through the second LOTR book.  But then I was 13 when I tried to read it09:57
StevenKI managed all 3 when I was 1509:57
Gyeah, thats around when I managed to get them all ready09:58
G*read09:58
StevenKG: Out of interest, North or South?09:59
GStevenK: Auckland09:59
Gahh I see, you are Sydney10:00
bigjoolsjafa!10:01
Gyep, born & raised10:01
Galthough I spent a bit of time in Brisbane10:01
nigelbjafa?10:01
nigelbbigjools: Wait, your Australian as well?10:02
Gnigelb: Just Anotherl F'ing Aucklander10:02
nigelbhaha10:02
bigjoolsnigelb: no, Brit.10:02
GJaffa on the other hand, is a nice lolly10:02
nigelbbigjools: That's what I thought as well :)10:03
gmbbigjools: The comments on https://bugs.launchpad.net/launchpad-buildd/+bug/840055 seem to suggest it's not actually a bug but is user error of some sort. Can you cast your weather eye over it and tell me if that's the case please?10:05
_mup_Bug #840055: strange build loop with binutils 2.21.53 <Launchpad Auto Build System:New for dns> < https://launchpad.net/bugs/840055 >10:05
bigjoolsgmb: sure10:05
gmbTa10:05
bigjoolsgmb: it's a buildd-manager bug10:06
bigjoolsand a dupe10:06
gmbbigjools: Ah, thanks.10:06
* bigjools plays hunt the dupe10:07
bigjoolsgmb: I suspect it's bug 71796910:08
_mup_Bug #717969: storeBuildInfo is sometimes ineffective <boobytrap> <buildd-manager> <Launchpad itself:Triaged> < https://launchpad.net/bugs/717969 >10:08
gmbbigjools: Thanks, I'll mark it as such and include this commentary.10:09
bigjoolsta10:09
wgrantbigjools: A generic "audit" sounds a lot too much like "karma"10:23
StevenKgmb: O hai, we seem to be Henning-less today. Would you mind reviewing https://code.launchpad.net/~stevenk/launchpad/destroy-openidrp/+merge/74007 ?10:23
bigjoolscompletely different10:23
wgrantbigjools: Similarly FK-less and prone to weak keys.10:23
gmbStevenK: I can't look right now, but I'll try to take a look this pm.10:23
wgrantUnless treated very carefully.10:23
StevenKgmb: Just to sweeten the deal, the line count is +2 / -28310:24
Goh can someone comment on my suggestions for Bug 306569?10:25
_mup_Bug #306569: Link to https://help.launchpad.net/Code from project branch listing page <lp-code> <trivial> <ui> <Launchpad itself:In Progress by dev-nigelj> < https://launchpad.net/bugs/306569 >10:25
stubSomewhere buried in our wiki there will be a document documenting how to rollback revision and revert rollbacks. But I know not where.10:29
wgrantstub: You've fixed pgbouncer?10:32
stubThat will be part of this branch, yes10:32
wgrantstub: You're hacking PATH? :/10:33
stubNo, I'll install a new .egg10:33
wgrantOh.10:33
wgrantRight.10:33
wgrantSo, just do a cherrypick of the original revision, or an inverse cherrypick of my rollback.10:33
wgrantbzr merge -r1233..1234 .10:33
stubAny magic when it needs to go to pqm?10:35
stubWhy did we decide a hard coded path was best btw? That is making the module Ubuntu/Debian specific.10:37
lifelessjust add /usr/sbin to PATH in the subprocess call10:37
lifelesssubprocess has glue for that10:37
lifelessand it won't make it so specific10:38
stubk10:38
nigelbgmb: Thanks! (re:spamming user)11:00
gmbnigelb: No worries. I enjoy taking the loving mallet of correction (as John Scalzi would call it) to spammers.11:01
* gmb -> lunch11:01
nigelbhehe11:01
wgrantjtv: Hm, that's upsetting.12:00
jtvwgrant: what is?12:00
wgrantjtv: That the rabbitfixture fix didn't fix/12:00
jtvAh that.  Yes.12:00
wgrantCan you pastebin 'sudo rabbitmqctl status'?12:00
jtvwgrant: http://paste.ubuntu.com/682518/12:06
wgrantBaaaah.12:07
wgrantIt's like postgres.12:07
wgrantIt only quotes when necessary.12:07
wgrantRip the quotes out of the first line of the regex. Should work then.12:07
wgrantOr ? them, as desired.12:07
jtvwgrant: quotes?  I only see one quote on that line.  And a few more in other lines.12:09
jtvMaking what look to be the opening and closing quotes optional…12:10
jtv—and that does the trick.  Thanks wgrant!12:10
wgrantThe first line of the original uncommented version of the regex that I never pastebinned, come on, read my mind.12:10
jtvNot on Mondays, you know that.12:11
jtvReview needed!  Any takers?  https://code.launchpad.net/~jtv/launchpad/pre-832647/+merge/7408712:12
Ggood point, I've got two MPs for review when there is an OCR around :)12:14
StevenKI have one as well12:17
StevenKG: I bet my MP removes more code than yours. :-P12:17
GStevenK: you can be sure of it :)12:18
Ghold on. is this a "My MP is bigger than yours" thing? :)12:18
jtvStevenK: how about we trade reviews and then take one of G's each?12:18
nigelbbigjools: I think there's something ajax that changes the links from blue to green.12:19
wgrantnigelb: You can't actually see the links he's talking about.12:20
wgrantnigelb: They are indeed blue and un-spinnered.12:20
nigelbwgrant: Why can't I see those?12:20
nigelb :(12:20
wgrantThey were mostly hacked up during the thunderdome, and are enabled only for ~launchpad right now AFAIK.12:20
nigelbAlso GAH for these things.12:20
nigelbCan I see it on my dev instance?12:20
wgrantSure, if you push up a branch, go to https://launchpad.net/+feature-rules, add 'code.ajax_revision_diffs.enabled default 0 on', and view the MP page.12:21
wgrants/net/dev/12:21
nigelbokay :)12:21
wgranthttps://launchpad.dev/+feature-info describes all the current feature flags.12:22
nigelbQAing this will be fun.12:24
jtvStevenK: I reviewed yours.  Will you review mine?12:27
wgrantStevenK: Looks like Jenkins will succeed in 10 minutes.12:27
wgrantWith a little luck buildbot will follow.12:28
=== al-maisan is now known as almaisan-away
cjwatsonRT#47040 fixed - does that mean I can have another go at landing multiarch-translations?12:41
_mup_Bug #47040: while booting initial kernel, progress bar looks like Win98 <gfxboot-theme-ubuntu (Ubuntu):Confirmed> < https://launchpad.net/bugs/47040 >12:41
cjwatsonor do I still need to wait for bug 809123 to actually go fix-released?12:41
_mup_Bug #809123: we cannot deploy DB schema changes live <fastdowntime> <qa-ok> <Launchpad itself:Fix Committed by stub> < https://launchpad.net/bugs/809123 >12:41
wgrantAs of a few hours ago we are, roughly, fastdowntime-capable. But we haven't actually done one yet.12:42
cjwatsonI suppose that multiarch-translations-schema (landed on db-devel a while back) is the thing that needs to be fastdowntimed12:43
wgrantThere is an as-yet undeployed Storm reconnection fix that we probably want to wait for, but if all goes to plan (as it hasn't for the last week) we'll be deploying that tomorrow.12:43
wgrantRight. Let's see what the fastdowntime queue looks like.12:44
wgrantI guess we could do more frequent episodes to clear it out.12:44
wgrantBut the first run will hopefully be on Wednesday, and it will be a no-op.12:44
wgrantstub: Anything to wait for but the Storm fix?12:45
cjwatsonjust bounce all the connections to see if anything falls over?12:45
wgrantBasically.12:45
wgrantThe full update script asks the DB cluster to sync, kills all connections, works out patches to apply, applies any patches, reconfigures security, syncs the cluster again, turns connections back on.12:45
wgrantAll the clients just see no DB for a while. Which could have amusing effects.12:46
jtvG: I reviewed one of yours in hopes that StevenK would perform the other half of the deal I proposed, but he seems to have bugged out.12:55
Gjtv: hold on, let me load it up on LP itself12:57
Gjtv: I'm not too sure on transaction.commit() I'll test it now12:58
stubwgrant: Nothing required but the storm fix. The branch-revision fix would be nice, but that is an understood problem we can work around by bouncing that service immediately after the update.13:04
stubSame for the storm fix really, but I'd like to minimize the amount of bouncing that needs to happen.13:05
wgrantstub: We should be able to deploy all the way tomorrow, unless buildbot still wants me dead.13:05
stubYes, agreed.13:05
stubmthaddon: Can we do updates tomorrow?13:06
LPCIBotYippie, build fixed!13:06
LPCIBotProject devel build #1,035: FIXED in 4 hr 36 min: https://lpci.wedontsleep.org/job/devel/1035/13:06
wgrant!!!!13:06
* stub looks into which particular db patch cjwatson needs13:06
mthaddonstub: what kind of updates?13:06
stubmthaddon: fastdowntime db update13:06
mthaddonstub: have we done a test run?13:06
stubmthaddon: This is the test run. Staging has been doing it for a while.13:07
mthaddonstub: okey doke13:07
wgrantWe're starting with a no-op, right? :)13:07
cjwatsonstub: https://code.launchpad.net/~cjwatson/launchpad/multiarch-translations-schema13:07
stubwgrant: I can argue it either way, but will go with lifeless' suggestion of a no-op first up.13:08
wgrantstub: We've not had wondrous success syncing the cluster recently, so it is tempting to eliminate unnecessary risk.13:09
wgrantEven though we're not dropping a slave, so it should all be fine.13:09
wgrantI still don't trust SSO to not end the world.13:09
wgrantShould we provisionally announce the 0800 window for tomorrow?13:09
stubcjwatson: I'll make sure getting that patch up is a priority, even if we can't get all the patches applied13:09
Goh is LP.net getting updated13:09
cjwatsongreat, thanks very much13:09
wgrantG: We normally update launchpad.net roughly daily.13:09
wgrantG: But database patches were previously applied only monthly, as they required 60-90 minutes of downtime.13:10
wgrantG: But the new process allows us to it in hopefully less than a minute.13:10
stubAnnouncements are not necessary for 5 minute outages apparently, so we just need to announce to identi.ca for the window.13:10
wgrantThe preparations are finally complete.13:10
Gwgrant: ahhh right13:10
wgrantstub: Once we have them down below a couple of minutes, a couple of hours is probably enough. But we might want to give a bit more this time, as it's the first and not exactly unrisky.13:11
stubWe can make a blog post if we want I guess since it is the trial by fire of the new process.13:11
wgrantHmm, no mrevell.13:11
wgrantidenti.ca two hours before, then :)13:11
wgrantHeh. Only 1.5 months after the blog post announcing the start of fastdowntime.13:12
stubNot really much risk apart from OOPSes and more extended outages for the bits that need to be manually bounced.13:12
wgrantstub: So we say until the cluster falls apart.13:12
wgrantAnd takes an hour to recover.13:12
stubwgrant: No risk of that with a no-op update13:13
wgrantHahaha.13:13
wgrantAlthough I guess the last few bad rollouts have been because we didn't want to abort.13:13
wgrantBecause the window was huge and expensive.13:13
wgrantIf stuff breaks with fastdowntime we can just abort.13:13
stubyup. everything that has caused failures in the past is checked for up front and the process doesn't kick off.13:14
wgrantAnd even if it does break we don't have to recover into a deployable state immediately.13:14
wgrantBut hopefully SSO won't desync any more.13:14
stubAnd even if there is some internal slony failure, we can switch to master-only fairly quickly with pgbouncer. We have covered everything we can, and at some point we need to push the button.13:16
wgrantYup.13:16
wgrantOK, where is the fastdowntime process page these days? The LEP is unhelpful.13:16
stubmthaddon: I'm going to need a fresh db-stable tree on wildcherry13:17
mthaddonstub: fresh means which revno?13:18
stubmthaddon: Not sure if we thought about that with your tree pushing scripts13:18
mthaddonstub: we pull from wildcherry, but yep, we have thought of it :)13:18
stubmthaddon: current head of launchpad/db-stable13:18
mthaddoner, db-stable?13:18
stubmthaddon: I will need to trim out unwanted patches for the no-op run13:18
wgrantstub: Can't we just merge the new scripts to devel?13:18
mthaddonyeah, my thoughts exactly13:19
wgrantMost of that should come across harmlessly.13:19
stubIf we want to test something other than the scripts we have been testing...13:19
wgrantI was actually looking at that this morning, when I cherrypicked some other stuff across.13:19
wgrantstub: Hm? Can't we just merge all non-DB-patch-depending changes into devel?13:19
stubYes, and hope we get it right and we didn't miss any dependencies13:20
wgrantTrue.13:20
stubOr we can just use db-stable13:20
stubI just noticed as I was going to check what db patches need to be pruned for the no-op run and if we need to do multiple real runs or a single 'real' update.13:20
wgrantHm. Did we get timings from staging yesterday?13:20
wgrantI'm guessing not.13:20
wgrantBecause it likes to break just a day or two before the restore.13:21
wgrantIt broke again.13:21
stubI think we have all the timings we need from all the previous runs.13:22
wgrantLots of permission denied (publickey) :(13:22
stub2011-09-02 14:44:00 INFO    2208-78-1 applied 2011-08-28 in 9.5 seconds13:22
stub2011-09-02 14:44:00 INFO    2208-79-0 applied 2011-08-28 in 1.3 seconds13:22
stub2011-09-02 14:44:00 INFO    2208-79-1 applied 2011-08-28 in 0.5 seconds13:22
stub2011-09-02 14:44:00 INFO    2208-80-1 applied 2011-08-28 in 0.1 seconds13:22
stub2011-09-02 14:44:00 INFO    2208-81-1 applied 2011-08-28 in 0.0 seconds13:22
wgrantAh, it was hwdb stuff breaking the update. The SSH errors were unrelated, apparently :/13:23
wgrantstub: Do a no-op run tomorrow, and if all goes well do another one also tomorrow with everything?13:24
stubyup, or even later in the same day or same 5 minute window ;)13:25
stubI guess same 5 minute window is a bit ambitious given we need to confirm everything survived :)13:25
wgrantRight, I meant later in the same day, as in the same 5 minute window is a little adventurous.13:26
stubBut the db patches I'm looking at all look really, really boring.13:26
wgrantThey are pretty bad, yeah.13:26
wgrantA good start.13:26
wgrantApart from the 10s one.13:26
stubIts still boring.13:26
wgrantIt is.13:26
wgrantBut slow boring.13:26
stub10s is not slow. We have had patches that took hours in the past ;)13:26
wgrantOh yes, I recall.13:27
wgrantThe good old days of 12-hour downtime to open a new Ubuntu release, and then the final 10-hour downtime to migrate to the new translations schema.13:27
stubyer, and get that down to 10s and people still are not happy. I wonder why I bother sometimes :-D13:30
=== henninge changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: henninge | Critical bugs: 258 - 0:[########*** stack smashing detected ***: ./lp terminated
stubLooks like just those 5 patches.13:31
wgrantstub: What about 76-3 and 76-4?13:31
wgrantOh. Those are bugsummaryrollupbreakage, aren't they.13:31
stubThey are both live13:31
wgrantHow could I forget.13:31
stubhttp://paste.ubuntu.com/682567/ will have to do since there is no report yet13:32
wgrantIndeed, just those 5.13:32
stubAnd upgrade.py --dry-run if I'm feeling hungover tomorrow.13:33
wgrantDo you happen to feel like checking BugMessage.owner's NULLness, just in case?13:33
wgrantThat's the main reason i wanted the restore.13:33
stubStill 013:34
wgrantThe appservers aren't going to choke on 79-0? I can't remember if that restriction was relaxed.13:34
stubI can't see why the appservers would complain.13:35
wgrantA month ago they would refuse to start if there was a -0 patch applied but not in the tree.13:35
stubThat code has been turned off completely.13:35
stubGuess we should check that revision is live13:36
wgrantIt seems to have been deleted.13:36
wgrantI can't find it anywhere nor remember the rev it was deleted in, so it's deployed everywhere.13:36
wgrantrevno: 13638 [merge]13:36
wgranttimestamp: Tue 2011-08-09 15:28:37 +000013:36
wgrant  [r=gmb][bug=809636,13:37
wgrant        815706] Remove runtime checks of LaunchpadDatabaseRevision13:37
stubta13:37
wgrantlibrarian1 would have been the last holdout.13:37
wgrantSo we're clean now.13:38
stubonly the appservers checked anyway13:38
wgrantYeah.13:38
Ghenninge: as OCR, can you at some point take a look at https://code.launchpad.net/~dev-nigelj/launchpad/bug-674854/+merge/74064 please13:39
henningeG: sure but otp right now13:39
StevenKOMG, devel built on Jenkins13:41
wgrantStevenK: Yup. jelmer fixed the four main failures on db-devel a week ago. I cherrypicked it into devel tonight, and disabled the other failing tests, which cascaded into everything else.13:41
wgrantSo we should be green for a while now.13:41
stubLong enough to take buildbot out the back and shoot it?13:42
wgrantThe new bzr seems to have made everything more reliable.13:42
bigjoolssigh, is the poppy bug STILL not fixed? :/13:43
wgrantThe dozens of varied failures over the last week or two can all be explained by fallout from the three tests.13:43
StevenKstub: Is that a +1 for Jenkins from you, then? :-)13:43
wgrantbigjools: Nope.13:43
wgrant:)13:43
wgrantSee /topc.13:43
wgrant /topic13:43
* bigjools hunts in vain13:43
stubStevenK: I'm for whatever has the least maintenance burden, and whatever I don't have to be involved with.13:44
StevenKHeh13:44
jmlprobably jenkins13:44
StevenKDeath to buildbot and buildbot-poller13:44
Gmust say, as an observer, jenkins looks kinda neat13:45
stubI would like a bot I can tell 'Go test lp:~stub/launchpad/foo and give me the results', but we need the basics first.13:45
StevenKstub: bin/ec2 test? :-)13:45
jmlthat's easy to do w/ jenkins, but breaks polling ime13:45
StevenKjml: Do it as a seperate job13:45
StevenKBut yes, fairly easy13:46
wgrantIdeally we do that with Jenkins in front of a couple of machines that run a couple of 4-way test suites or so.13:46
stubStevenK: I hate the 5 minutes it takes and the risk of leaving hung instances chewing through cash.13:46
stubStevenK: I also hate having too many environments that tests have to pass on. We currently need everything to pass on 'local dev environment', ec2 image, buildbot image, staging environment, production environment. With the wonders of modern VM tech, there is no reason we can't get that down to one environment that is identical everywhere.13:47
StevenKstub: +1,000,000,00013:48
Gyeah, I was kind of surprised when I kept seeing EC2 been mentioned13:52
StevenKG: Whyfor?13:52
stubBecause while the test suite is solid, there are so many dependencies involved the environment ends up fragile. Time consuming to put together an alternative that is as automated and reliable.13:54
GStevenK: great for testing, but it was a "I thought they'd be using an inhouse cloud or something using UEC"13:54
sorenI'm curious.. How many instances are spawned to run the tests?13:54
GStevenK: put it this was, I was surprised, but understood13:54
StevenKsoren: One per branch13:55
sorenStevenK: Oh, just one?13:55
=== almaisan-away is now known as al-maisan
sorenHm.13:55
sorenOk.13:55
StevenKsoren: Yup.13:55
sorenHow long does a test run take?13:55
stubPeople are working on parallel testing, but it is still science fixtion13:55
StevenKsoren: 4 and a bit hours.13:55
sorenOk.13:56
wgrantstub: LXC-based parallel testing is pretty much doable now.13:56
wgrantIt has been done.13:56
StevenKsoren: It's not okay, it's awful. :-P13:56
wgrantIt's not completely reliable yet.13:56
stubYer, just the second 95% to do now.13:56
sorenStevenK: Well, yes :)13:57
Ghmmm so 4 & a bit hours at $0.68/hr?13:57
stubYer, the coffee you drink while waiting costs more.13:58
GI guess it could be much worse13:59
stubBeing able to rent out servers by the hour for < $1 is pretty awesome.14:00
stubSomeone is going to notice this Cloud thing one day.14:00
Gstub: you mean they haven't?14:01
stubDunno. I stopped watching TV when I moved into exile.14:02
StevenKwgrant: Are you happy for me to toss destroy-openidrp at ec2?14:02
wgrantStevenK: Let me glance over it first.14:03
wgrantStevenK: Hm, you now always warn?14:04
wgrantI guess that's OK.14:05
StevenKIt's still dangerous14:05
=== jkakar_ is now known as jkakar
wgrantStevenK: To ec2 with it!14:06
StevenKwgrant: Happy with no-qa, or shall I file a bug just so we can try it out?14:07
wgrantBug would be nice.14:07
bacn14:08
StevenKwgrant: You and I name too many branches beginning with 'destroy-'14:10
jmls/too many/not enough/14:12
jmlfunctionality is an asset. code is a liability.14:12
nigelbStevenK: Start using nuke- then ;)14:13
StevenKicbm-14:13
nigelb+114:14
Gdiediedie-14:14
nigelbG wins.14:14
Goh actually, why not go all James Bond... 007- :)14:14
StevenKI think both wgrant and I have used 'diediedie' in a branch name before14:14
nigelblol14:14
Gor from the movie... RED- (Retired & Extremely Dangerous) :)14:14
Gtbh, I like 007- better14:17
nigelbI'll go with diediedie14:17
StevenKI don't like the idea of 007-. If he likes the look of the code, he'll seduce it and sleep with it14:18
GStevenK: in that case, there is far bigger issues to worry about than the bug you are trying to fix :)14:19
bigjoolsStevenK: the code is already fucked though14:20
nigelbbigjools++14:21
=== jkakar is now known as jkakar_
=== jkakar_ is now known as jkakar
=== dpm_ is now known as dpm
wgrantjelmer: Hi. Can you please QA the bzr upgrade? It's getting reasonably urgent.14:56
wgrantaaaergwejfwf14:57
wgranttest_import_bzr failed this time.14:57
* wgrant kills them all.14:57
jelmerwgrant: I've QA'd it as much as I can so happy to mark it qa-ok14:58
wgrantjelmer: Great, thanks.14:58
jelmerwgrant: that's test_import_bzr for the workermonitor?15:00
wgrantjelmer: Yes.15:01
wgrantjelmer: I've already disabled the three that you reenabled a couple of weeks back.15:01
wgrantBut now test_import_bzr is failing the same way.15:01
wgrantI think script startup is just too slow now.15:01
wgrantSo it hits the 20s timeout.15:01
wgrantBug #84155615:01
_mup_Bug #841556: test_import_bzr, test_import_cvs, test_import_bzrsvn and test_import_subversion disabled <test-system> <Launchpad itself:Triaged> < https://launchpad.net/bugs/841556 >15:01
=== salgado is now known as salgado-lunch
* wgrant sleeps.15:04
jelmerblergh15:04
jelmerwgrant: thanks for handling that15:04
henningeG: reviewed. Sorry for taking so long. Please have a look.15:09
Ghenninge: yeah, I couldn't find any tests that related to the distroarchseries webservice15:12
Ghenninge: oh, so you are saying I should replace the whole setUp w/ a _makeDistroandArch() type thing?15:13
henningeG: yup15:14
Gin lib/tests/factory (or the correct path) or in the new file?15:14
henningeG: Much more obvious when you read it15:14
henningeG: no, directly in your test case.15:14
Goh right15:14
GI'll be back in a few then :)15:15
Ghenninge: everything else good okay though?15:15
henningeyes, looks fine15:16
Ghenninge: that said, as multiple tests can benefit from the one-time setup for the data, isn't that what setUp is for?15:17
G(just curious)15:17
G(i.e. if I was to do an authenticated test as well, surely the one time setup, instead of multiple setups, would be faster) or am I on the wrong thought track?15:19
henningeG: No, it is not faster15:20
henningeG: setUp is executed before each test.15:20
Goh it is, okay then15:20
=== salgado-lunch is now known as salgado
=== Ursinha is now known as Ursinha-lunch
GOkay, giving this a whirl15:33
nigelbG: Don't you ever sleep? :)15:36
nigelbI see you *all* the time :D15:36
nigelbg3215:36
Gnigelb: I have a weird sleep pattern15:36
bigjoolsI might also say that about nigelb!15:36
nigelbbigjools: :D15:37
nigelbKettle, pot, black. I know!15:37
nigelbI need to pull an all-nighter. I'll throw that frustration away by writing some patches.15:37
Gnigelb: I do some of my best thinking late at night15:37
nigelbSomeone needs to write "The therapeutic effects of coding"15:37
G(not just coding, but thinking in general)15:38
nigelbG: Join the club.15:38
nigelbI sleep at about 2 am local time.15:38
GTue Sep  6 03:38:30 NZST 201115:38
Galthough, most python applications would have be believe it's currently +1300 for some reason :)15:38
nigelbIs your tzdata package up-to-date?15:39
Gyep15:39
Gbut I don't know when the last time NZ has been +1300 this early, ever15:40
nigelbTry using pytz to figure out the date. That should work correctly.15:42
nigelbIf it doesn't, you can always crib to its author right here ;)15:43
nigelbs/date/date\/time/g15:43
nigelbPyCharm people just need to hire bigjools as their community manager :P15:45
bigjoolsthey need to write it in Python first :)15:45
nigelbPyCharm isn't Python/15:45
nigelb?15:45
bigjoolsJava15:45
* bigjools throws up a little15:45
nigelbEwww.15:45
Ga Python IDE written in Java, heh15:46
bigjoolsand it uses Swing15:46
* G vomits a little15:46
bigjoolsbut I forgive it so far15:46
nigelbIs it as slow as Eclipse?15:46
bigjoolspretty snappy for the most part15:46
bigjoolsbut then I have 4 cores15:46
nigelbjml: ping15:51
jmlnigelb: yes?15:51
nigelbjml: Could you join #ubuntu-classroom, #ubuntu-classroom-backstage, and #ubuntu-classroom-chat?15:51
jmlnigelb: done15:51
nigelbThe bot's complaining that you aren't around :)15:51
Ghenninge: I think I might want to take another look at this again tomorrow15:56
=== beuno is now known as beuno-lunch
GI think my anonymous test, may not be logging into the API so anonymously afterall16:05
=== jkakar_ is now known as jkakar
nigelbI'm liking jono's session :)16:08
henningeG: np at all, get some sleep first ... ;)16:09
henningeHow do I update sourcedeps.cache?16:09
jelmerhenninge: run ./utilities/update-sourcecode16:10
henningejeblair: next question: when would I do that?16:10
* henninge never had to do it.16:10
henningejelmer: ^16:10
henningesorry jeblair16:10
nigelbI think either rf-branch or make run already did that16:11
jelmerhenninge: After you change utilities/sourcedeps.conf16:11
henningeaha16:11
jelmerhenninge: (or after a merge changes it)16:11
henningeok16:11
henningejelmer: thanks16:11
jelmerhenninge: it would indeed be neat to add some deterministic ordering to it to prevent spurious conflicts and the like16:12
henningejelmer: I already found the place where to do that, I just wanted to find out how to try/test it.16:12
jelmerhenninge: cool16:16
Ghenninge: actually, I worked out the problem, should be an updated branch coming to you in a moment :)16:19
henningeG: cool16:19
Gtwo last bin/test's then I might be able to sleep easier :)16:19
=== al-maisan is now known as almaisan-away
Gokay, it works :)16:21
Ghenninge: okay, I pushed the updated fix to my branch, if it's okay please feel free to land it, if there are other issues, I'll tackle them tomorrow :)16:29
henningeG: I'll have a look in a minute16:30
Gyep, no hurry, I'm off to bed :)16:30
=== beuno-lunch is now known as beuno
=== henninge changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 258 - 0:[########*** stack smashing detected ***: ./lp terminated
=== Ursinha-lunch is now known as Ursinha
=== almaisan-away is now known as al-maisan
nigelbsigh, buildbot20:25
=== matsubara is now known as matsubara-afk
=== al-maisan is now known as almaisan-away
=== salgado is now known as salgado-afk
* wgrant crushes buildbot into the ground.22:34
wallyworld_wgrant: i forced a buildbot build but the compile failed - looks like a simple timeout. you seen this sort of thing before?22:52
wgrantYes, we are fucked.22:54
wgrantI'm trying to track down when this failure first showed up.22:54
wgrantBut buildbot doesn't actually keep logs for more than a week or so.22:54
wgrantAlthough I guess they should be in emails...22:54
wgrantOr not.22:55
wgrantIt just emails links.22:55
wgrantSigh.22:55
wallyworld_wgrant: so, it's not an actual compile error right?22:57
wgrantThere is something real there.22:57
wgrantOh.22:57
wgrantThat last failure is not real, no. That was hloeung killing the slave.22:57
wgrantThought you were talking about the previous ne.22:57
wgrantBut you were hopefully too asleep to force that.22:58
wallyworld_yes, just saw that one issue when i checked earlier and thought i'd try a force22:58
wallyworld_since it appeared to be a timeout22:58
wgrantYeah, we've forced that failure about 20 times in the last two weeks.22:59
wallyworld_hopefully one of the maintenace squads is looking22:59
wgrantlol22:59
wgrantUS and Canada were away yesterday.22:59
wallyworld_ah, labour day22:59
=== wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wgrant | Critical bugs: 259 - 0:[########*** stack smashing detected ***: ./lp terminated
wallyworld_wgrant: you tried to install nautilus elementary in oneiric? i really need to be able to customise the toolbars23:00
wallyworld_natty packages don't work :-(23:00
wgrantI've not. I hardly use nautilus, but when I do the new one is nice and clean.23:04
wallyworld_clean yes, but you can't do simple things like add a "up to Parent" button to the toolbar. very annoying when you need to navigate up one directory23:05
wgrantAh.23:06
wallyworld_nautilus elementary also has a bunch of other useful features. it really should be what we ship imho23:10
jelmerhas anybody seen this error on oneiric: "psycopg2.OperationalError: fe_sendauth: no password supplied" ?23:10
wgrantjelmer: Running the test suite locally?23:14
jelmerwgrant: yeah - it worked fine a couple of days ago23:14
wgrantjelmer: We connect over TCP now (as of the weekend, then it was reverted, and now as of a couple of hours ago)... perhaps your postgres config doesn't permit ident/hba over TCP?23:14
jelmerwgrant: that might be it, I'll have a look at that23:15
wgranthost    all         all         127.0.0.1/32      trust23:15
wgrantThat line should be in your pg_hba.conf23:15
wgrantOdd that it's not /823:15
wgrantPerhaps your localhost resolves to something else in 127/8 instead?23:15
jelmermaybe it's ipv623:16
wgrantAh, or that.23:17
wgrantBut that should be ip6-localhost, normally.23:17
wgrantMost of my home network is IPv6, and it all still works for me.23:17
jelmerI can't find it right now and it's getting quite late, I'll have a look tomorrow.23:20
wgrantNight!23:21
=== wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wgrant | Critical bugs: 260 - 0:[########*** stack smashing detected ***: ./lp terminated
nigelb:\23:30
nigelbAre the critical bugs breeding :(23:30
wgrantYup!23:31
wgrantwallyworld_: Are you running LP on oneiric still, or back in LXC?23:31
wallyworld_wgrant: still on oneiric23:31
wgrantwallyworld_: You've just been dealing with JS, then?23:32
wgrantSince you haven't wanted my rabbitfixture fix.23:32
wallyworld_wgrant: yes. but i'm in the middle of a python branch so that will change23:33
wgrantwallyworld_: See the list when you run into RabbitMQLayer startup trouble. I have a properish fix now.23:33
wallyworld_wgrant: awesome. thanks23:33
wgrantpygtk seems somewhat broken on oneiric, so the YUI testrunner doesn't work.23:34
wgrantBut there are only 8 other oneiric failures.23:34
pooliewgrant, if you have a chance can you read my dkim patch?23:41
pooliei see one nit in it at least ,though23:41
wallyworld_wgrant: i can run yui tests ok23:44
wgrantwallyworld_: Odd. Maybe you upgraded so things work better?23:48
wallyworld_wgrant: i did upgrade from natty but it hosed my system. so i installed oneiric from scratch23:48
wallyworld_just to clarify, bin/test --layer=YUITest works, as does running individual test cases from the browser by loading the html page. what error do you get?23:49
wgrantwallyworld_: No module named gtk. Then I installed python-gtk2, and now it says it can't initialise Gtk or something.23:50
wgrant  File "/usr/lib/python2.6/dist-packages/gi/overrides/Gtk.py", line 1406, in <module>23:50
wgrant    raise RuntimeError("Gtk couldn't be initialized")23:50
wallyworld_hmmm. i never saw anything like that23:50
wgrantRuntimeError: Gtk couldn't be initialized23:50
wallyworld_i don't think i installed everything gtk though23:51
wgrantThere's probably some dep I'm missing, since this is in a minimal LXC, while you're presumably running with a full desktop environment.23:51
wgrantAh, umask changes are breaking some of the other tests :(23:52
wgrant002 ftl :(23:52

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!