[00:32] <thumper> where are the soupmatchers?
[00:33]  * thumper pulls again
[00:33] <lifeless> lp:
[00:33] <lifeless> ?
[00:37] <wgrant> lifeless: Did your parallel-supporting DB fixture ever land?
[00:37] <lifeless> no
[00:37] <wgrant> :(
[00:37] <lifeless> feel free to hack on it
[00:41] <wgrant> lifeless: Hm, that's a bit of an odd one.
[00:41] <wgrant> Thanks.
[00:41] <lifeless> wgrant: oh, its easy
[00:41] <lifeless> wgrant: canonical.lp.dbhost
[00:41] <lifeless> etc
[00:41] <wgrant> Probably, yes.
[00:41] <lifeless> are redudant fugly shit
[00:41] <wgrant> It's just not what I would have expected.
[00:41] <lifeless> the script entry point isn't resetting it quite right
[00:42] <lifeless> with the result that fti.py attempts to work on launchpad_dev
[00:42] <lifeless> rather than the supplied -d launchpad_test_template or whatever
[00:42] <wgrant> Yep.
[00:59] <thumper> gah
[00:59] <thumper> I have a test that failed on ec2, but passes locally
[00:59] <thumper> WTF?
[00:59] <thumper> I've done a make schema
[00:59] <thumper> but still weird shit ...
[01:00] <lifeless> isolation error?
[01:00] <lifeless> what test
[01:01] <thumper> p.code.browser.tests.test_sourcepackagerecipe.TestSourcePackageRecipeView.test_request_builds_page
[01:01] <thumper> wit an l at the start
[01:03]  * thumper fires up make run
[01:04] <poolie> thumper: hi?
[01:04] <thumper> poolie: hey
[01:07] <poolie> shall we have a chat if you're back?
[01:08] <thumper> sure
[01:10]  * thumper shoves it back through ec2 again
[01:38] <ccxCZ> too tired to write the mail now, here is link if anybody wants to play with it http://webprojekty.cz/ccx/trac/wiki/lpbot
[01:49] <thumper> ccxCZ: a link to what?
[01:53] <ccxCZ> irc notification bot for launchpad projects
[01:53] <ccxCZ> anyway gn, I have to go
[01:54] <lifeless> wgrant: databasefixture and librarian branches are updated and pushed, if you're interested
[02:16] <lifeless> [02:16] <lifeless>     Hard / Soft  Page ID
[02:16] <lifeless>      134 / 5092  Archive:+index
[02:16] <lifeless>       70 /  240  BugTask:+index
[02:16] <lifeless>       24 /  249  Distribution:+bugs
[02:16] <lifeless>       20 /  118  ProjectGroupSet:CollectionResource:#project_groups
[02:16] <lifeless>       11 /  265  POFile:+translate
[02:16] <lifeless>       11 /   13  DistroArchSeries:+index
[02:16] <lifeless>       11 /    1  ProjectGroup:+milestones
[02:16] <lifeless>        7 /   35  Milestone:+index
[02:16] <lifeless>        6 /  306  Distribution:+bugtarget-portlet-bugfilters-stats
[02:17] <lifeless>        5 /    9  Person:+bugs
[02:17] <wgrant> Ow :(
[02:17] <wgrant> I must get back to Archive:+index.
[02:29] <lifeless> wgrant: that would be the bomb
[03:48] <wallyworld> thumper: ping
[04:05] <jtv> poolie: a thought… would it save EC2 startup time (or server load) if we could use a virtual-machine image that had some fixed version of our source tree or repo pre-installed, so that it only needs to download the updates?
[04:07] <lifeless> jtv: thats what we have, isn't it ?
[04:07] <jtv> oh :)
[04:41] <thumper> wallyworld: hey
[04:41] <thumper> wallyworld: it is time for pizza and movies :)
[04:41] <wallyworld> just wanted to confirm that you want me to look at bug 670452 next
[04:42] <_mup_> Bug #670452: Hard to find related branches when composing recipe <recipe> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/670452>
[04:42] <thumper> wallyworld: aye
[04:43] <wallyworld> cool. i'll take a look once i finish seeing if i can improve that oops reporting test
[04:43] <wallyworld> haveagoodweekend
[04:50]  * wgrant theorises that HTML forms were designed as a cruel joke.
[04:56] <jtv> wgrant: careful—that line of reasoning leads down a dangerous path to paranoia, misanthropy and most probably, truth.
[04:57] <jtv> General observation about software stuff, not a comment about HTML forms specifically.
[04:58] <wgrant> Heh.
[04:58] <jtv> Think Mahabharata-style stripping of illusions.
[04:59] <jtv> I could say something about pills in The Matrix if I could bear the thought of that crappy film.
[04:59] <jtv> Besides, I'm regularly informed through emails from helpful strangers that given the choice, there is no good reason to spurn the blue pill.
[05:03] <jtv> …
[05:03] <jtv> You're still reading the Wikipedia entry on Mahabharata, aren't you?
[05:03] <wgrant> I was debating whether or not to look it up.
[05:04] <jtv> If you were debating it on your own, you probably want prozac, not viagra.  My bad.
[05:04] <StevenK> jtv: Hey, the first one was awesome. It was just the other two that made me cringe
[05:05] <jtv> StevenK: the first pill, the first book of the Mahabharata, or the first wgrant personality in the debate?
[05:07] <StevenK> jtv: I'm sorry, the first Matrix movie
[05:08] <jtv> Ah that one.  Did one thing for me: realize how precious my time was.  Only time I ever walked out of a cinema shortly after the film started.
[05:08] <jtv> I've met other people who did that.
[05:08] <wgrant> I only saw them a few months ago. I quite liked the first one.
[05:10] <spm> jtv: I had a similar experience with my wife, sister and sisters friend at a production of Les Mis. Yay. Play is over. We can go home now. "Um Steve? no. That's just the first half, this is the intermission" AAAAAAAAAAAAAAAAAAAAA
[05:11] <jtv> spm: hang on, still laughing
[05:11] <spm> :-)
[05:11] <jtv> Hah.  Whew.  That's better.
[05:11] <spm> So I went home, ladies, enjoy the play. Ring me when you want to be picked up. :-D
[05:12] <spm> Bored out of my tiny little mind. Some lady was whing about doing it tough. Some dudes with guns were whatever. Bored.
[05:12] <spm> Bored against the backdrop of the French Revolution. But still bored.
[05:13] <jtv> Saw one cartoon about this kind of thing… spectator #1: "man but this is boring.  I should have brought a good book!"
[05:13] <jtv> Spectator #2: "that's why I always read the reviews first."  Goes back to reading Tolstoy.
[05:13] <spm> bwhahahaa
[05:18] <poolie> jtv, hi, it could
[05:22] <jtv> poolie: lifeless noted that we already seem to do this.
[05:22] <poolie> could be good to know what fraction of total time is spent pulling that source
[05:22] <poolie> i suspect it's not very high
[06:02] <henninge> One of the current test failures in devel is strange - it reports a "test with error" but the test does not output a traceback or anything ...
[06:03] <henninge> looks like this: henning@dustpuppy:$ bin/test -vvcm lp.services.job.tests.test_runner -t TestJobCronScript.test_configures_oops_handler
[06:03] <henninge> Running tests at level 1
[06:03] <henninge> Running canonical.testing.layers.LaunchpadZopelessLayer tests:
[06:03] <henninge>   Set up canonical.testing.layers.BaseLayer in 0.004 seconds.
[06:03] <henninge>   Set up canonical.testing.layers.ZopelessLayer in 9.778 seconds.
[06:03] <henninge>   Set up canonical.testing.layers.DatabaseLayer in 0.482 seconds.
[06:03] <henninge>   Set up canonical.testing.layers.LibrarianLayer in 10.697 seconds.
[06:03] <henninge>   Set up canonical.testing.layers.MemcachedLayer in 0.112 seconds.
[06:03] <henninge>   Set up canonical.testing.layers.LaunchpadLayer in 0.000 seconds.
[06:03] <henninge>   Set up canonical.testing.layers.LaunchpadScriptLayer in 0.005 seconds.
[06:03] <henninge>   Set up canonical.testing.layers.LaunchpadZopelessLayer in 0.000 seconds.
[06:03] <henninge>   Running:
[06:03] <henninge>  lp.services.job.tests.test_runner.TestJobCronScript.test_configures_oops_handler
[06:03] <henninge>   Ran 1 tests with 0 failures and 1 errors in 0.848 seconds.
[06:03] <henninge> Tearing down left over layers:
[06:03] <henninge>   Tear down canonical.testing.layers.LaunchpadZopelessLayer in 0.000 seconds.
[06:03] <henninge>   Tear down canonical.testing.layers.LaunchpadScriptLayer in 0.002 seconds.
[06:03] <henninge>   Tear down canonical.testing.layers.ZopelessLayer ... not supported
[06:03] <henninge>   Tear down canonical.testing.layers.LaunchpadLayer in 0.000 seconds.
[06:03] <henninge>   Tear down canonical.testing.layers.LibrarianLayer in 0.003 seconds.
[06:03] <henninge>   Tear down canonical.testing.layers.MemcachedLayer in 0.101 seconds.
[06:03] <henninge>   Tear down canonical.testing.layers.DatabaseLayer in 0.236 seconds.
[06:03] <henninge>   Tear down canonical.testing.layers.BaseLayer in 0.001 seconds.
[06:03] <henninge> Tests with errors:
[06:03] <StevenK> henninge: Pastebin?
[06:03] <henninge>    lp.services.job.tests.test_runner.TestJobCronScript.test_configures_oops_handler (subunit.RemotedTestCase)
[06:03] <lifeless> is that test a twisted test?
[06:03] <henninge> oops
[06:04] <lifeless> henninge: does it have run_tests_with = ?
[06:04] <henninge> sorry, meant to paste http://paste.ubuntu.com/536572/
[06:04] <henninge> lifeless: I don't konw what that is but that string does not appear in the file .
[06:05] <henninge> looks like this http://paste.ubuntu.com/536573/
[06:06] <lifeless> henninge: what about ZopeTestInSubProcess
[06:07] <henninge> actually, I just realized that that I pasted the wrong output but it's similar.
[06:08] <lifeless> ah, interesting
[06:09] <henninge> lifeless: no, that neither
[06:09] <lifeless> this might be allenaps change to use unicode in subunit
[06:10] <henninge> aha
[06:12]  * henninge tries with that reverted
[06:14] <henninge> nope,same result :(
[06:19] <lifeless> wgrant: we're turning the publicrestrictedlibrarian back on
[06:20] <lifeless> because it seems to work in tests
[06:20] <wgrant> lifeless: Yay.
[06:20] <wgrant> But why? :/
[06:20] <wgrant> Was Apache fixed?
[06:20] <lifeless> wgrant: can you give it a thorough spin
[06:20] <lifeless> wgrant: To quote spm, FIIK.
[06:20] <wgrant> Heh.
[06:20] <wgrant> Sure.
[06:20] <wgrant> Say when.
[06:21] <henninge> so, r11980 seems to be the culprit for the current test failure
[06:21] <henninge> at least for two of them ...
[06:21] <henninge> the ones that don't produce output
[06:23] <spm> wgrant: no, no changes in apache. I did a moderate amount of local faffing around to determine that the proposed changes shouldn't make a difference (apache ignores those bits), ie it should "Just Work" as is. So we switched it on, tested, off. and it worked. ie FIIK.
[06:23] <lifeless> wgrant: oh, it was live at '19:21 < lifeless> wgrant: can you give it a thorough spin
[06:23] <lifeless> '
[06:24] <wgrant> lifeless: Still broken.
[06:25] <wgrant> Can you see https://launchpad.net/~soyuz-team/+archive/ppa/+build/1334368?
[06:25] <wgrant> The build log link there.
[06:25] <wgrant> And lots of other similar ones.
[06:25] <StevenK> The page you are trying to view cannot be shown because it uses an invalid or unsupported form of compression.
[06:26] <lifeless> I'm not in soyuz-team
[06:26] <lifeless> wgrant: gimme a url
[06:26] <wgrant> lifeless: Neither am I, but I have a sub. I thought you might be a commercial admin.
[06:26] <wgrant> https://i35262589.restricted.launchpadlibrarian.net/35262589/buildlog_ubuntu-karmic-i386.openproj_1.4-2%2Bpx1_FAILEDTOBUILD.txt.gz?token=b8e6f2f070469191af32ce99bb92fae8
[06:27] <wgrant> StevenK: Right, it's the usual uncompressed 404, served with gzip headers.
[06:27] <wgrant> So there are two levels of WTFery.
[06:27] <StevenK> Haha
[06:28] <StevenK> HTTP request sent, awaiting response... 404 Not Found
[06:28] <StevenK> Ah yes
[06:30] <lifeless> spm: please turn it off again
[06:31] <spm> done
[06:31] <spm> so how did it work for me/us? but not here?
[06:31] <spm> Oh. its the 404?
[06:31] <wgrant> It works for some files.
[06:35] <lifeless> spm: ok, so it *is* the 404s
[06:35] <spm> not the files themselves, the 404 response?
[06:35] <lifeless> wgrant: does that same build log show properly now?
[06:36] <StevenK> Repeating the same wget still gives me a 404
[06:36] <lifeless> of course
[06:36] <lifeless> you need the source url
[06:36] <lifeless> on the appservers
[06:36] <lifeless> thats what redirects or proxies
[06:36] <StevenK> Yes, the build log now works
[06:36] <wgrant> lifeless: Proxying works fine.
[06:37] <lifeless> wgrant: whats the original url ?
[06:38] <StevenK> https://launchpad.net/~soyuz-team/+archive/ppa/+build/1334368/+files/buildlog_ubuntu-karmic-i386.openproj_1.4-2%2Bpx1_FAILEDTOBUILD.txt.gz ?
[06:39] <lifeless> I suspect url mangling
[06:39] <lifeless> I need to chat to spm briefly
[06:39]  * spm questions the brief part
[06:39] <StevenK> Bwahaha
[06:46] <lifeless> so I think its the %
[06:51] <lifeless> wgrant: whats the debian version string for that, unmangled
[06:51] <StevenK> 1.4-2+px1
[07:02] <lifeless> thanks
[07:03] <lifeless> so, % urls are mangled by something in the request path
[07:03] <lifeless> they are decoded
[07:03] <lifeless> and then passed on decoded?!
[07:03]  * lifeless bets on apache
[07:03] <lifeless> wouldn't know a standard if it bit it
[07:03] <StevenK> lifeless: I was thinking that
[07:13] <lifeless> https://bugs.launchpad.net/launchpad-foundations/+bug/681675
[07:13] <_mup_> Bug #681675: content coding for build logs is done in apache rather than the librarian <Launchpad Foundations:New> <https://launchpad.net/bugs/681675>
[07:13] <lifeless> updated https://bugs.launchpad.net/launchpad-foundations/+bug/677270
[07:13] <_mup_> Bug #677270: apache/squid  breaks restricted librarian on urls with percent encoded characters. <Launchpad Foundations:In Progress by canonical-losas> <https://launchpad.net/bugs/677270>
[07:14] <henninge> Anybody have a clue for me? This test is failing *even* when I revert to the revision of the last successful buildbot run.
[07:15] <henninge> http://paste.ubuntu.com/536585/
[07:15] <henninge> It start two jobs and both seem to fail on _pythonpath_. Is that a local error for me?
[07:16] <henninge> I am getting this on devel r11979
[07:16] <StevenK> So does buildbot, it seems
[07:16] <henninge> which is said "last known good" revision
[07:16] <StevenK> henninge: If changing devel doesn't fix it, I'd blame an egg change?
[07:16] <henninge> hm
[07:17] <lifeless> henninge: 11980 is the twisted runner change, I was thinking it was to blame :)
[07:17] <henninge> but doesn't version.cfg control which version of an egg a revision is using?
[07:17] <lifeless> jml: ^
[07:17] <wgrant> henninge: You need to remake.
[07:17] <lifeless> henninge: you have to run bin/buildout again
[07:17] <henninge> lifeless: it is to blame for the other two failing test
[07:17] <henninge> wgrant: make clean ; make build? Did that ...
[07:18] <wgrant> henninge: Hmm.
[07:18]  * henninge tries bin/buildout
[07:18] <lifeless> there's probably a missing dep on versions.cfg
[07:19] <henninge> no change
[07:19] <lifeless> buh
[07:21]  * StevenK stares at Hudson: "took 5 min 5 sec"
[07:21] <StevenK> However, r11980 is devel #250 on Hudson, and that didn't fail any tests
[07:23]  * henninge tries r11980 locally again
[07:28]  * henninge notices that db-devel passed revision 10000
[07:28] <StevenK> org.apache.commons.jelly.JellyTagException: jar:file:/var/run/hudson/war/WEB-INF/lib/hudson-core-1.377.jar!/hudson/model/AbstractBuild/sidepanel.jelly:34:42:  java.lang.OutOfMemoryError: Java heap space
[07:28] <StevenK> Wheee!
[07:28] <lifeless> \//
[07:28] <StevenK> Java, you suck, and I hate you
[07:29] <StevenK> I think Hudson is now unhappy, since it now gives 503s
[07:31] <StevenK> And even /etc/init.d/hudson stop ... doesn't
[07:33] <StevenK> Heh
[07:33] <StevenK> Right, it does, it just takes 2 minutes? Srsly?
[07:34] <lifeless> you were in swap
[07:34] <StevenK> Grah, it is 50 MiB into it
[07:34] <StevenK> How is 1.5GiB not enough for Java?!
[07:35] <lifeless> how much memory did you tell java it can use?
[07:35] <StevenK> Whatever the default is?
[07:35] <lifeless> you might have been swapped out even if you weren't exceeding the machine memory in one proc
[07:38] <StevenK> It's still 48MiB into swap with apache and java dead, so it isn't that
[07:38] <StevenK> This machine still has a full Launchpad development environment on it, so that can't be helping
[07:41] <StevenK> And restarting Hudson means it has lost track of the build it was in the middle of
[07:41]  * StevenK shoots the buildslave in the head
[07:47] <LPCIBot> Project parallel-test build (15): STILL FAILING in 34 sec: https://hudson.wedontsleep.org/job/parallel-test/15/
[07:49] <StevenK> lifeless: Do you want me a queue another parallel-test build?
[07:49] <lifeless> StevenK: ask me on the 3rd
[07:49] <lifeless> StevenK: of january
[07:50] <lifeless> sorry, the 4th
[07:50] <StevenK> By which it would have landed?
[07:51] <lifeless> no, by which I'll be back at work
[08:00] <spm> lifeless: "by which I'll be back at work"? you haven't heard? we're replacing you with a perl script. I thought you knew. ??
[08:00] <lifeless> spm: never a win
[08:00] <StevenK> #!/usr/bin/perl -w
[08:00] <lifeless> see
[08:00] <StevenK> while (1) { nag(); }
[08:00] <lifeless> minus win
[08:00] <spm> nah. -w is for wussies
[08:01] <StevenK> Perl coder for six years at previous job. Old habits die hard
[08:16] <StevenK> And there goes X in a screaming heap
[08:37] <adeuring> good morning
[09:06] <jml> hello
[09:07] <jml> who's fixing the build?
[09:17] <jml> henninge: hello
[09:18] <jml> henninge: have you submitted anything to fix the build?
[09:18] <wgrant> I'm confused.
[09:18] <wgrant> r10005 doesn't look like a testfix to me...
[09:18] <henninge> jml: I hoped to find out what to do about the third test still failing
[09:19] <henninge> not much point in fixing two out of three tests
[09:19] <jml> wgrant: what about it? (I haven't looked, trying to deal with the current failure)
[09:19] <henninge> wgrant: no but then db-devel wasn't broken
[09:19] <jml> henninge: agreed. just wanted to check
[09:20] <wgrant> henninge: It's labelled as testfix.
[09:20] <jml> hey look
[09:20] <wgrant> But doesn't seem to have a test fix.
[09:20] <jml> it's ZopeTestInSubProcess again
[09:20] <jml> ffs
[09:21] <henninge> wgrant: sorry, jtv is just telling me off about it, too ...
[09:21] <jtv> "now now henninge, that's not nice"
[09:21] <jtv> there, see?
[09:21] <wgrant> I think it might confuse the QA bot a bit, but I guess we'll see.
[09:21] <henninge> we'll see ...
[09:22] <jml> I wonder what happened re upgrading subunit
[09:23] <jml> I see we're at the "someone should do something about this" phase of discussion
[09:23] <jml> ok.
[09:23] <wgrant> jml: That's surely the best phase.
[09:24] <jml> wgrant: I don't know, I'm a "someone else just did something about it" kind of guy.
[09:30] <jml> henninge: test_timeout fails for you locally, even if you revert to r11979?
[09:35] <jml> we should change these to be testtools RunTest thingies anyway
[09:37] <lifeless> presumably they are in sub processes for a good reason.
[09:38] <henninge> jml: yes, it does.
[09:38] <jml> lifeless: I mean, there should be a testtools subprocess RunTest thingy
[09:38] <jml> lifeless: rather than using a mixin with a run() method
[09:38] <lifeless> +1
[09:39] <jml> henninge: ok. so something is very weird.
[09:39] <jml> because a 'known good' build is breaking
[09:40] <jml> I'm going to try to get more info on what the child process is doing
[09:40] <jml> I *really* doubt r11980 has anything to do with these failures, fwiw.
[09:41] <lifeless> jml: Its plausible
[09:41] <lifeless> jml: if the reactor is in a surprising state
[09:41] <lifeless> it forks, it doesn't exec
[09:44] <jml> how can I get at the stdout & stderr of the child process?
[09:45] <bigjools> hello
[09:46]  * jml cheats
[09:48] <jml> lifeless: oh wait, it's not plausible at all
[09:48] <lifeless> jml: ok
[09:48] <jml> lifeless: because the test fails with r11979 too
[09:50] <jml> also fails when not run in a subprocess
[09:55] <henninge> jml: to be clear: I could verify that the other two failing tests (*not* the timeout test) pass on 11979 but fail on 11980. The timeout test failure does not seem to be connected to that revision.
[09:56] <henninge> s/verify/verify locally/
[10:01] <jml> henninge: do you have a plausible reason as to why?
[10:02] <henninge> no, I don't ... :(
[10:02] <henninge> actually ...
[10:03] <jml> allenap: I don't think it's related to the unicode thing
[10:03] <jml> henninge: I suspect bad local state, actually
[10:04] <StevenK> jml: As well as inside the buildbot slaves?
[10:04] <jml> StevenK: yes.
[10:04] <henninge> jml: I don't deny that. Did you try it on your machine?
[10:04] <henninge> jml: the only clue is this: http://paste.ubuntu.com/536585/
[10:04] <StevenK> I find that a little hard to believe. Can haz proof?
[10:04] <henninge> The test starts two jobs wich both fail on a missing _pythonpath_
[10:05] <jml> StevenK: working on it.
[10:05] <henninge> But I don't know how that can happen just for those two and not for everthing else.
[10:05] <jml> the clue is that reverting to about 20 revisions ago still has the failure
[10:05] <henninge> good point
[10:06] <StevenK> jml: Hudson might actually be your proof
[10:06] <StevenK> jml: Now that I think about it
[10:07] <jml> also, remember, the Makefile has been changed four times recently (11937, 11957, 11958 and 11979)
[10:08] <jml> StevenK: in that it works?
[10:08] <StevenK> jml: Indeed
[10:08] <StevenK> jml: But the Hudson slaves are different to the buildbot slaves
[10:08] <jml> naturally
[10:10] <jml> how do we clean out the build on buildbot?
[10:10] <StevenK> I suspect we run make clean only
[10:11] <jml> how do we move from suspicion to certainty?
[10:12] <StevenK> We channel mars, or someone who actually does more with buildbot than try and replace it
[10:16]  * jml has to go
[11:06] <bigjools> rf-get and update-sourcecode doesn't want to fix the lack of a AsynchronousDeferredRunTestForBrokenTwisted module....
[11:09] <allenap> jml: I did find an interesting problem. The read side of the pipe in ZTISP was being opened rU. Locally that has caused tests to error, but with no output at all. Changing it to rb means that they succeed. I don't really know why, but perhaps this is related to the unicode thing too. I'm landing a branch now to revert the getreader() thing, and to open both ends of the pipe as binary.
[12:31] <jml> bigjools: that sucks. will try to look into that after the build is fixed
[12:31] <jml> allenap: cool. that fixes all three failing tests?
[12:32] <bigjools> jml: a make clean fixored it
[12:32] <jml> bigjools: ahh good
[12:32] <bigjools> not sure why I needed that though :(
[12:36] <jml> bigjools: me neither. maybe I did something a bit stupid as I developing it.
[12:36] <jml> it's not clear how to evolve Launchpad & an upstream in parallel
[12:38] <jml> hmm
[12:38] <jml> I wonder how I can make a script that downloads an openid-protected page
[12:40] <allenap> jml: I think it should fix them, but then I haven't been able to replicate the exact failures. I'm hoping :)
[12:41] <bigjools> jml: see utilities/pastebin
[12:41] <bigjools> well, the old one
[12:41] <allenap> jml: from faustian_pact import convenient_openid_wrapper
[12:41] <jml> hmm. I don't think I have that one.
[12:41] <bigjools> sure you do, you have bzr repo? :)
[12:42] <jml> bigjools: faustian_pact is in Launchpad's history?
[12:42] <bigjools> maybe not that one
[12:42] <bigjools> I can't see you facing allenap on IRC when you talk
[12:43] <jml> yeah, they probably haven't ported that to kde yet
[12:44] <jml> db_lp broken again
[12:45] <jml> any volunteers?
[12:48]  * bigjools 's kde sarcasm detector app goes off
[12:48] <jml> allenap: test_timeout still fails for me with your branch, but the other two are fine.
[13:01] <jml> https://bugs.launchpad.net/launchpad-foundations/+bug/681770
[13:01] <_mup_> Bug #681770: Failure in lp.services.job.tests.test_runner.TestTwistedJobRunner.test_timeout <Launchpad Foundations:New> <https://launchpad.net/bugs/681770>
[13:34] <jml> jkakar_: hi
[13:36] <jkakar_> Hiya jml!
[13:37] <jml> jkakar: how's things?
[13:39] <jkakar> jml: Going really well, thanks.  I'm sprinting in Barcelona, so it's been an exciting week.  Just got back from lunch.
[13:39] <jkakar> jml: How about you?
[13:40] <jml> jkakar: well. bit of a cold this week but otherwise fine. slowly mastering the art of the automobile
[13:41] <jml> jkakar: you coming to London any time soon?
[13:44] <jml> (also, yay sprinting)
[13:47] <jkakar> jml: Driving, oooh, fun times. :)
[13:47] <jkakar> jml: Will be in London in a couple weeks, yep.
[13:48] <jml> all confirmed?
[13:57] <bigjools> bloody windmill
[13:58] <mars> bigjools, ?
[13:58] <bigjools> it hangs when I run it locally, which means my test run never completes
[13:59] <mars> ah, not fun.  That is new, I don't think anyone has mentioned that yet
[13:59] <mars> did Deryck kill the windmill suite?
[13:59] <mars> (yet?)
[13:59] <bigjools> never mind Deryck, I want to kill it :)
[13:59] <mars> heh
[13:59] <mars> bigjools, I should ask: is this problem new?
[13:59] <jml> speaking of which...
[13:59] <mars> jml, your @skiptest decorator is the big win for us there
[14:00] <jml> the build is still broken, failing with a windmill test. I suspect a fix like http://pastebin.ubuntu.com/533852/ might be needed
[14:00] <bigjools> mars: I'm not sure :(
[14:00] <bigjools> it only happens on one of my boxes
[14:00] <jml> mars: wasn't mine,  I don't think
[14:01]  * jml doesn't actually know anything about windmill, that paste is just what deryck did last time
[14:01] <mars> jml, that fix didn't land?
[14:02] <jml> mars: it landed. a different test is failing.
[14:05] <mars> jml, if the needed fix is similar to what you posted, then it will have to be applied by inspecting the test for races caused by missing timeouts
[14:06] <jml> mars: make sure you pass that on to whoever sets about actually fixing it.
[14:06] <mars> sure, I can do that
[14:07] <mars> Deryck's windmill fix has not landed
[14:08] <jml> well, I guess we'll look at the bug filed for that particular breakage and find out why
[14:46] <flacoste> jml: is it only you and me today on The Strategerie?
[14:56] <jml> flacoste: https://bugs.launchpad.net/launchpad-foundations/+bug/681770
[14:56] <_mup_> Bug #681770: Failure in lp.services.job.tests.test_runner.TestTwistedJobRunner.test_timeout <Launchpad Foundations:New> <https://launchpad.net/bugs/681770>
[14:57] <flacoste> jml: lost connection?
[15:26] <bigjools> would it be evil to use a storm validator to auto-update a different field?
[15:45] <bigjools> oh dear, it's not good when the same class name appears twice in a test file is it ...
[16:02] <jml> flymake! pyflakes! happiness!
[16:02] <bigjools> pyflakes didn't detect it
[16:03] <bigjools> I only noticed because on of my new tests wasn't getting run
[16:03] <bigjools> one*
[16:03] <jml> really?
[16:03] <jelmer> it gives a warning here for one of the two declarations
[16:03] <bigjools> hmm
[16:03] <jml> bigjools: if so, that's a bug I'm going to fix right now.
[16:04] <bigjools> I wonder if I've got an old vim pyflakes
[16:06] <bigjools> can someone please look at this, I've got a test failure that I cannot fathom.  It's totally unrelated to my changes yet reverting them makes the test pass.   http://pastebin.ubuntu.com/536736/
[16:08] <jelmer> bigjools: It must trigger a database flush or something..
[16:09] <jelmer> bigjools: my bet is that the fact you've changed builderok into a property is the culprit
[16:09] <bigjools> jelmer: indeed - but how is that affecting whether something else is unicode?
[16:10] <bigjools> oh, I wonder if it hits the DB and gets converted
[16:10] <jelmer> bigjools: Yeah, that's what's probably happening.
[16:10] <bigjools> that's nasty
[16:10] <bigjools> thanks jelmer
[16:11] <bigjools> jml: bin/lint.sh does not show the duplicated clas
[16:11] <bigjools> s
[16:11] <bigjools> see lib/lp/buildmaster/tests/test_builder.py
[16:11] <jml> bigjools: I don't use or care about lint.sh
[16:11] <bigjools> TestBuilder is declared twice
[16:12] <bigjools> yes but vimflakes doesn't see it either - what do you care about?
[16:12] <jml> pyflakes
[16:12] <bigjools> that does not see it either
[16:13] <bigjools> pyflakes lib/lp/buildmaster/tests/test_builder.py
[16:13] <bigjools> nowt
[16:13] <jml> I see.
[16:13]  * jml fixes
[16:16] <bigjools> jelmer: heh - we both missed the point of the test - the values should be unequal.  The unicode was a red herring.
[16:17] <jelmer> argh, I see
[16:17] <jelmer> bigjools: it was a database flush that was causing the issues though?
[16:18] <bigjools> I don't know
[16:18] <bigjools> fg
[16:27] <jml> /home/jml/src/launchpad/devel/lib/lp/buildmaster/tests/test_builder.py:127: redefinition of class 'TestBuilder' from line 89
[16:27] <jml> \o/
[16:35] <jml> lp:~jml/pyflakes/duplicate-class-defs if anyone wants to use it.
[16:35] <jml> http://divmod.org/trac/ticket/3034 if you care about upstreaming
[16:37]  * bigjools claps jml
[16:38] <jelmer> jml: that was quick!
[16:39] <jml> jelmer: pyflakes is really easy to patch
[16:39] <jml> particularly after the first patch :)
[16:40] <jelmer>  jml: I tried to merge the latest trunk into the bzr lazy_import branch at one point and gave up.
[16:40] <jml> oh yeah. they changed it quite significantly recently to use ast or something
[16:41] <jml> still structured the same but quite different textually
[16:46] <bigjools> jelmer: I found the problem
[16:46] <bigjools> Builder.selectBy(builderok=True)
[16:47] <jelmer> ah, a select on a non-storm property?
[16:47] <bigjools> yeah
[16:48] <jelmer> confusing error then..
[16:48] <bigjools> the selectBy doesn't break!
[16:48] <bigjools> it just returns nothing
[17:12] <jml> who wants a Friday afternoon vaguely on-topic distraction?
[17:12] <jml> I have pretty pictures
[17:12] <jml> http://people.canonical.com/~jml/lpstats/good-and-bad-by-time-bar.png
[17:12] <jml> http://people.canonical.com/~jml/lpstats/how-many-tests-are-slow.png
[17:13]  * bigjools needs one of those
[17:13] <bigjools> not a metric? :)
[17:13] <jml> right
[17:14] <jml> it *could* be a metric, but the numbers don't quite add up and I haven't figured out why
[17:14] <bigjools> kidding - slightly.  What's the colours/numbers measuring?
[17:14] <jml> the colours are ... bands of test run time
[17:15] <jml> e.g. tests that take under a second to run, but over 0.1s are green in both graphs
[17:15] <bigjools> right, so it's a distribution
[17:15] <jml> http://people.canonical.com/~jml/lpstats/good-and-bad-by-time.png and http://people.canonical.com/~jml/lpstats/good-and-bad.png are different versions of the same respective data
[17:15] <jml> bigjools: yes
[17:15] <bigjools> is it easy to change to a line graph?
[17:15] <jml> bigjools: one distribution is by number of tests within that band
[17:15] <bigjools> might be easier to interpret
[17:16] <jml> bigjools: the other is by time taken by tests in that band
[17:16] <jml> bigjools: I think the bar graphs are close to what you mean by line graphs
[17:16] <jml> unless you mean something like http://people.canonical.com/~jml/Tests-finished-each-minute.png
[17:17] <bigjools> I've only ever seen distribution graphs done as a curvy line, usually looking like a bell :)
[17:17] <jml> they are frequently done as bars, particularly for logarithmic distributions
[17:18] <jml> (i.e. http://people.canonical.com/~jml/lpstats/good-and-bad-by-time.png and http://people.canonical.com/~jml/lpstats/good-and-bad.png)
[17:18] <jml> or maybe I meant e.g.
[17:18] <bigjools> but then it's Friday afternoon, I've had no sleep from watching too much cricket, and I need a beer
[17:18] <jml> hah
[17:19] <bigjools> apart from that I feel quite lucid
[17:19] <jml> anyway, the rough conclusion is that the tests that take over a second make up a small fraction of the tests by number
[17:19] <jml> (~2000 / 12000)
[17:20] <bigjools> I bet we could make those 1 second or under tests faster
[17:20] <jml> but make up the majority by time
[17:20] <jml> (~8000 / 10000)
[17:20] <jml> bigjools: we could, and that would be great, but the graphs basically say that if we _only_ did that, it wouldn't matter much.
[17:22] <bigjools> maybe
[17:22] <jml> if we took the test suite as it is now, and ran the fastest tests first, those taking under a second would finish in the first half hour, the remaining time would be spent in the ~15% of the tests that take over a second
[17:22] <bigjools> let's just delete them all
[17:23] <bigjools> btw how do doctests fit in this?
[17:23] <bigjools> are they one test or lots of tests?
[17:23] <jml> each doctest is counted as one test.
[17:24] <bigjools> I bet those 15% are largely doctests :)
[17:24] <bigjools> (including pagetests)
[17:24] <bigjools> in fact the latter are slower than an Aussie outfielder
[17:24] <jml> data is gathered from: ./bin/test --subunit | subunit-ls --times --no-passthrough | sed -e 's/ /,/' > test-runtime-data.csv
[17:24] <jml> hah
[17:26] <bigjools> nice analysis though
[17:27] <bigjools> ditching pagetests gets more weight! :)
[17:32] <jml> bigjools: fwiw, of the ~1500 tests that take over a second, 574 are doctests
[17:33] <bigjools> oh, wow
[17:33] <jml> 315 of those are pagetests (rough count)
[17:33] <bigjools> I expect manty of the others are soyuz uploader tests
[17:34] <jml> I did this: gzip -dc /home/jml/Downloads/testtools-experiment-r11753.subunit.gz | subunit-ls --times --no-passthrough | awk '{ print $2 " " $1 }' | sort -n > test-run-times
[17:34] <jml> then used emacs to chop out all of the sub 1s tests
[17:34] <jml> then wc -l test-run-times, grep -c _txt test-run-times and grep -c ' [0-9x][0-9x]' test-run-times
[17:35] <jml> bigjools: slow unit tests, ordered by time: http://pastebin.ubuntu.com/536769/
[17:36] <bigjools> some obvious suspects
[17:37] <jml> bigjools: by name: http://pastebin.ubuntu.com/536771/
[17:37] <jml> heh
[17:38] <bigjools> test_all_scripts makes me weep
[17:38] <jml> of course, subunit encodes layer start up as a test :)
[17:38] <bigjools> it's duplicating tests done elsewhere
[17:38] <jml> 82.739 gina_txt
[17:38] <jml> 86.680 soyuz-upload_txt
[17:38] <bigjools> the shame
[18:07] <salgado> abentley, any idea what's with the list of 'properties changed' messages in the diff at https://code.launchpad.net/~salgado/launchpad/expose-blueprints/+merge/41898 ?
[18:08] <jml> g'night all.
[18:08] <jml> have a good weekend
[18:09] <salgado> abentley, looks like a bug in the incremental diff thing?
[18:09] <abentley> salgado: No, they're new to me.
[18:09] <abentley> salgado: Yes, definitely that.
[18:13] <salgado> abentley, can you look into that or do you want me to report a bug and/or complain on the ML?
[18:14] <abentley> salgado: please file a bug.
[18:14] <abentley> salgado: I will investigate, but it needs tracking.
[18:16] <salgado> abentley, https://bugs.launchpad.net/launchpad-code/+bug/681885
[18:16] <_mup_> Bug #681885: Spurious 'properties changed' line on incremental diffs <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/681885>
[18:17] <salgado> thanks for investigating
[18:27] <abentley> salgado: So I think there's a bug in bzrlib that makes the execute bit of directories and symlinks handled differently for RevisionTrees and PreviewTrees.  For RevisionTrees, it's False and for PreviewTrees, it's None.
[18:29] <abentley> salgado: This bug will only be exercised if you merge from trunk or your prerequisite branch, but we certainly want you to be able to do that.
[18:30] <salgado> oh, right, that makes sense.  the busted diff is the one which includes a merge from trunk
[18:30] <abentley> salgado: I'm going to see if the same thing happens with merge --preview.
[18:44] <abentley> flacoste: I'm confused about the recent changes to the release process, and lifeless's proposal and how they would affect the PQM closure window.
[18:45] <flacoste> abentley: let's have a call about it in 30 minutes?
[18:45] <abentley> flacoste: great!
[19:23] <flacoste> abentley: i'm on mumble
[19:39] <cr3> in launchpad, is there a preference in naming methods when getting latest items, like IPersonPublic.latestKarma compared to IPersonPublic.getLatestMaintainedPackages?
[19:40] <cr3> the former could've been named getLatestKarma or the latter named latestMaintainedPackages, so I'm wondering if there's a general preference... and a motivation would also be nice :)
[19:40] <salgado> cr3, the former is not an appropriate method name in Launchpad -- methods must be named after verbs
[19:42] <cr3> salgado: awesome, I'm pleased so hear that methods should start with active verbs
[19:43] <salgado> in general we have good coding/naming standards in Launchpad.  the only problem is that we don't always follow them
[19:45] <cr3> salgado: no worries, this innevitably happens in projects when they become very large even with the best of intentions
[19:46] <cr3> salgado: for example, I can imagine having to define my own oauth classes to be used by the python oauth module which has different naming conventions, and then having those same conventions leak into the rest of my code.
[19:47] <cr3> in those cases, I usually duplicate method or attribute names at the end of my class definition with a comment like: # Compatibility with oauth module
[19:49]  * lifeless thinks that this is noise and overspecified in launchpad
[19:51] <cr3> lifeless: what's noise and overspecified exactly? naming conventions or duplicating names for compatibility purposes?
[19:51] <lifeless> both
[19:52] <cr3> lifeless: I'm not sure I follow, what's noisy about naming conventions?
[19:52] <lifeless> the former is more to remember to meet 'ok to land' which adds little if any actual value; the latter is extra code which makes backtraces longer, will confuse folk that actually use the real module when analysing your code
[19:58] <cr3> lifeless: I'm not sure I agree but I'll admit that the importance of naming conventions is mostly a matter of taste, so open to pointless debates like vim vs emacs, but I happen to find value in following conventions for consistency purposes which gives me pleasure when using interface that are predictable
[19:59] <lifeless> I think there are several reasons I don't find that [very] valuable
[20:00] <lifeless> firstly, its an all or nothing: either you can completely confidently guess at the right thing, or you need to do a search & read docs.
[20:01] <lifeless> even in small programs there is often enough nuance needed that the obvious thing isn't necessarily the right thing.
[20:01] <lifeless> secondly, unless you completely ban all external code (which is a bad idea for so many reasons), you have to work on multiple code bases when fixing and refactoring.
[20:02] <lifeless> Its extremely rare to be able to consume a component and *never* find a bug or issue in it that needs fixing.
[20:05] <lifeless> I do find value in contracts - e.g. in the context manager contract
[20:05] <lifeless> that can be argued as a consistency thing, but I think its more reuse.
[20:06] <lifeless> Its not 'same style', its interface substitutable
[20:07] <cr3> lifeless: contract definately has value
[20:12] <flacoste> mars: is there a reason that we didn't disable domain hashing in our google analytics tracker code?
[20:13] <flacoste> mars: setAllowHash
[20:14] <flacoste> mars: if i read http://code.google.com/apis/analytics/docs/tracking/gaTrackingSite.html#domainSubDomains correctly we should be using it
[20:14] <flacoste> otherwise, it means that we are overestimating visitors as each visitor to a different subdomain is considered different
[20:44] <mars> flacoste, I wasn't aware of that option
[20:44] <flacoste> mars: ok, i'll land the fix
[20:46] <mars> flacoste, thanks.  I haven't seen that option before - IIRC I read the help docs, but not the tracking setup docs
[20:54] <lifeless> flacoste: did you see the apache pain ? :)
[20:55] <flacoste> lifeless: yes, it was a deeper issue than i thought, but i'm glad that we have a fix through mod_rewrite
[20:55] <lifeless> flacoste: we do?
[20:55] <lifeless> flacoste: we left it unfixed..
[20:55] <flacoste> lifeless: [B]
[20:55] <flacoste> it seems that will work, no?
[20:55] <lifeless> flacoste: not in our testing
[20:55] <flacoste> [P,B]
[20:55] <lifeless> it mangled the /
[20:56] <flacoste> !
[20:56] <flacoste> wow
[20:56] <flacoste> that sucks big time
[20:56] <lifeless> GET /57533044/foo+bar.txt?token=48
[20:56] <lifeless> wrong due to +
[20:56] <flacoste> would setting the header in the librarian, bypass this?
[20:56] <flacoste> (as we wouldn't have to use <Location)?
[20:56] <lifeless> GET /57533044%252ffoo%252bbar%252etxt?token=488
[20:57] <lifeless> flacoste: the content encoding header is orthogonal
[20:57] <lifeless> flacoste: it corrupts 404s, it doesn't cause them.
[20:57] <lifeless> AFAICT
[20:57] <flacoste> why is it working in the other setup?
[20:57] <lifeless> flacoste: its not working anywhere is it ?
[20:58] <lifeless> flacoste: define 'it' :)
[20:58] <flacoste> well, it works when we disable the feature flag
[20:58] <lifeless> the librarian isn't used then
[20:58] <flacoste> we are able to retrieve these files now
[20:58] <flacoste> doh, of course
[20:58] <lifeless> with the flag off, the appserver serves it using the internal librarian port to obtain the content
[20:58] <flacoste> it works because we are serving this content from the app server
[20:58] <flacoste> right
[20:58] <flacoste> missed that
[20:59] <flacoste> so the only fix is either fix apache or change front-end for the restricted librarian?
[20:59] <lifeless> of course, with the flag off kees has a script that gets 503 timeouts so often it takes hours to download security debs to check :(
[20:59] <flacoste> lifeless: we could turn the feature flag on for kees :-)
[20:59] <lifeless> flacoste: we could try to reverse engineer which things will be decoded by apache and match those rules in the librarian.
[21:00] <lifeless> but that seems spectacularly fugly.
[21:00] <flacoste> yeah
[21:00] <lifeless> flacoste: folk are still commenting on this cluster of bugs in apache this year.
[21:00] <lifeless> even though they are marked variously 'wontfix' and 'fix released'
[21:00] <flacoste> how did tom feel with switching front-end?
[21:00] <lifeless> there may be some dark art to make it work.
[21:00] <lifeless> we didn't talk in detail about it
[21:01] <lifeless> apache is a known quantity, and there is a feeling that it adds some security vs running (whatever) backend service live on the interwebs
[21:01] <elmo> I'm not having whatever backend service terminal SSL :-P
[21:01] <elmo> s/terminal/terminate/
[21:01] <lifeless> elmo: I was thinking squid
[21:02] <elmo> well
[21:02] <elmo> I don't believe this is intractable
[21:02] <lifeless> elmo: I wouldn't want the complexity of SSL in the librarian code - shudder.
[21:02] <elmo> U1 ran into similar rewrite escaping issues and were able to work around it
[21:02] <lifeless> elmo: I thought they did, but I couldn't find the resolution
[21:02] <elmo> I'd like to explore fixing or working around apache and at least filing bugs about it's broken behaviour further before we seriously consider abandoning it
[21:03] <lifeless> elmo: certainly
[21:03] <lifeless> elmo: could you find what U1 did to address this?
[21:03] <elmo> lifeless: right now?  no, but I'll have a look
[21:03] <lifeless> thanks
[21:04] <lifeless> elmo: in terms of fixing apache
[21:04] <lifeless> I linked to one insightful comment from apache upstream
[21:04] <lifeless> apparently it does a escape decode very early in its processing pipeline
[21:04] <lifeless> before mod rewrite or mod proxy get close to the request
[21:05] <lifeless> I guess adding a request variable at that point to save the original value and pass it down might help.
[21:27] <henninge> Um ...
[21:30] <henninge> I think it is getting too late for me ...
[22:40] <wgrant> The DB config is far more duplicated and evil than I could ever have imagined :/
[22:42] <lifeless> wgrant: looking at my branch ?
[22:42] <wgrant> lifeless: Yeah, I've fixed it.
[22:42] <lifeless> wgrant: I'd like to nuke the canonical.lp.dbhost etc
[22:42] <lifeless> wgrant: I just didn't want to make the branch bigger
[22:42] <wgrant> lifeless: Yes...
[22:42] <lifeless> wgrant: what did I have wrong?
[22:43] <wgrant> lifeless: The db_options option parser thing used to set canonical.lp.dbname. But you removed that when you turned it into a function.
[22:43] <wgrant> Also, you missed two references to canonical.lp.dbname.
[22:44] <wgrant> lifeless: I've made db_options set lp.dbname_override, which lp.get_dbname() now consults. Any less horrific suggestions?
[22:49] <lifeless> seems fine
[23:32] <lifeless> wgrant: want me to land it ?
[23:37] <wgrant> lifeless: Please do.
[23:37] <wgrant> make check runs OK now, until it hangs as it does on devel.
[23:37] <wgrant> So ec2 should be happy.
[23:37] <lifeless> wheres your fix? I'll pull into my branch
[23:37] <wgrant> lifeless: lp:~wgrant/launchpad/databasefixture
[23:45] <lifeless> how do you get other people to get emails on ec2land ?
[23:46] <wgrant> ec2 land, I don't know.
[23:46] <wgrant> ec2 test takes a -e option.
[23:46] <lifeless> ah
[23:46] <lifeless> well, I've ec2landed it
[23:46] <lifeless> will forward you the epic failure, should there be one, tonight
[23:46] <wgrant> Hopefully it won't be necessary!
[23:46] <wgrant> Thanks.
[23:49] <lifeless> also \o/ the site message is gone
[23:49] <lifeless> hopefully we'll see less noddy bugs on lp itself.
[23:49] <wgrant> Yep.
[23:50] <lifeless> i have a reasonable query implementation in my head now
[23:50] <lifeless> need to write it up
[23:54] <lifeless> its away