[00:32] where are the soupmatchers? [00:33] * thumper pulls again [00:33] lp: [00:33] ? [00:37] lifeless: Did your parallel-supporting DB fixture ever land? [00:37] no [00:37] :( [00:37] feel free to hack on it [00:41] lifeless: Hm, that's a bit of an odd one. [00:41] Thanks. [00:41] wgrant: oh, its easy [00:41] wgrant: canonical.lp.dbhost [00:41] etc [00:41] Probably, yes. [00:41] are redudant fugly shit [00:41] It's just not what I would have expected. [00:41] the script entry point isn't resetting it quite right [00:42] with the result that fti.py attempts to work on launchpad_dev [00:42] rather than the supplied -d launchpad_test_template or whatever [00:42] Yep. [00:59] gah [00:59] I have a test that failed on ec2, but passes locally [00:59] WTF? [00:59] I've done a make schema [00:59] but still weird shit ... [01:00] isolation error? [01:00] what test [01:01] p.code.browser.tests.test_sourcepackagerecipe.TestSourcePackageRecipeView.test_request_builds_page [01:01] wit an l at the start [01:03] * thumper fires up make run [01:04] thumper: hi? [01:04] poolie: hey [01:07] shall we have a chat if you're back? [01:08] sure [01:10] * thumper shoves it back through ec2 again [01:38] too tired to write the mail now, here is link if anybody wants to play with it http://webprojekty.cz/ccx/trac/wiki/lpbot [01:49] ccxCZ: a link to what? [01:53] irc notification bot for launchpad projects [01:53] anyway gn, I have to go [01:54] wgrant: databasefixture and librarian branches are updated and pushed, if you're interested === ccxCZ- is now known as ccxCZ [02:16] === Top 10 Time Out Counts by Page ID === [02:16] Hard / Soft Page ID [02:16] 134 / 5092 Archive:+index [02:16] 70 / 240 BugTask:+index [02:16] 24 / 249 Distribution:+bugs [02:16] 20 / 118 ProjectGroupSet:CollectionResource:#project_groups [02:16] 11 / 265 POFile:+translate [02:16] 11 / 13 DistroArchSeries:+index [02:16] 11 / 1 ProjectGroup:+milestones [02:16] 7 / 35 Milestone:+index [02:16] 6 / 306 Distribution:+bugtarget-portlet-bugfilters-stats [02:17] 5 / 9 Person:+bugs [02:17] Ow :( [02:17] I must get back to Archive:+index. [02:29] wgrant: that would be the bomb [03:48] thumper: ping [04:05] poolie: a thought… would it save EC2 startup time (or server load) if we could use a virtual-machine image that had some fixed version of our source tree or repo pre-installed, so that it only needs to download the updates? [04:07] jtv: thats what we have, isn't it ? [04:07] oh :) [04:41] wallyworld: hey [04:41] wallyworld: it is time for pizza and movies :) [04:41] just wanted to confirm that you want me to look at bug 670452 next [04:42] <_mup_> Bug #670452: Hard to find related branches when composing recipe [04:42] wallyworld: aye [04:43] cool. i'll take a look once i finish seeing if i can improve that oops reporting test [04:43] haveagoodweekend [04:50] * wgrant theorises that HTML forms were designed as a cruel joke. [04:56] wgrant: careful—that line of reasoning leads down a dangerous path to paranoia, misanthropy and most probably, truth. [04:57] General observation about software stuff, not a comment about HTML forms specifically. [04:58] Heh. [04:58] Think Mahabharata-style stripping of illusions. [04:59] I could say something about pills in The Matrix if I could bear the thought of that crappy film. [04:59] Besides, I'm regularly informed through emails from helpful strangers that given the choice, there is no good reason to spurn the blue pill. [05:03] … [05:03] You're still reading the Wikipedia entry on Mahabharata, aren't you? [05:03] I was debating whether or not to look it up. [05:04] If you were debating it on your own, you probably want prozac, not viagra. My bad. [05:04] jtv: Hey, the first one was awesome. It was just the other two that made me cringe [05:05] StevenK: the first pill, the first book of the Mahabharata, or the first wgrant personality in the debate? [05:07] jtv: I'm sorry, the first Matrix movie [05:08] Ah that one. Did one thing for me: realize how precious my time was. Only time I ever walked out of a cinema shortly after the film started. [05:08] I've met other people who did that. [05:08] I only saw them a few months ago. I quite liked the first one. [05:10] jtv: I had a similar experience with my wife, sister and sisters friend at a production of Les Mis. Yay. Play is over. We can go home now. "Um Steve? no. That's just the first half, this is the intermission" AAAAAAAAAAAAAAAAAAAAA [05:11] spm: hang on, still laughing [05:11] :-) [05:11] Hah. Whew. That's better. [05:11] So I went home, ladies, enjoy the play. Ring me when you want to be picked up. :-D [05:12] Bored out of my tiny little mind. Some lady was whing about doing it tough. Some dudes with guns were whatever. Bored. [05:12] Bored against the backdrop of the French Revolution. But still bored. [05:13] Saw one cartoon about this kind of thing… spectator #1: "man but this is boring. I should have brought a good book!" [05:13] Spectator #2: "that's why I always read the reviews first." Goes back to reading Tolstoy. [05:13] bwhahahaa [05:18] jtv, hi, it could [05:22] poolie: lifeless noted that we already seem to do this. [05:22] could be good to know what fraction of total time is spent pulling that source [05:22] i suspect it's not very high [06:02] One of the current test failures in devel is strange - it reports a "test with error" but the test does not output a traceback or anything ... [06:03] looks like this: henning@dustpuppy:$ bin/test -vvcm lp.services.job.tests.test_runner -t TestJobCronScript.test_configures_oops_handler [06:03] Running tests at level 1 [06:03] Running canonical.testing.layers.LaunchpadZopelessLayer tests: [06:03] Set up canonical.testing.layers.BaseLayer in 0.004 seconds. [06:03] Set up canonical.testing.layers.ZopelessLayer in 9.778 seconds. [06:03] Set up canonical.testing.layers.DatabaseLayer in 0.482 seconds. [06:03] Set up canonical.testing.layers.LibrarianLayer in 10.697 seconds. [06:03] Set up canonical.testing.layers.MemcachedLayer in 0.112 seconds. [06:03] Set up canonical.testing.layers.LaunchpadLayer in 0.000 seconds. [06:03] Set up canonical.testing.layers.LaunchpadScriptLayer in 0.005 seconds. [06:03] Set up canonical.testing.layers.LaunchpadZopelessLayer in 0.000 seconds. [06:03] Running: [06:03] lp.services.job.tests.test_runner.TestJobCronScript.test_configures_oops_handler [06:03] Ran 1 tests with 0 failures and 1 errors in 0.848 seconds. [06:03] Tearing down left over layers: [06:03] Tear down canonical.testing.layers.LaunchpadZopelessLayer in 0.000 seconds. [06:03] Tear down canonical.testing.layers.LaunchpadScriptLayer in 0.002 seconds. [06:03] Tear down canonical.testing.layers.ZopelessLayer ... not supported [06:03] Tear down canonical.testing.layers.LaunchpadLayer in 0.000 seconds. [06:03] Tear down canonical.testing.layers.LibrarianLayer in 0.003 seconds. [06:03] Tear down canonical.testing.layers.MemcachedLayer in 0.101 seconds. [06:03] Tear down canonical.testing.layers.DatabaseLayer in 0.236 seconds. [06:03] Tear down canonical.testing.layers.BaseLayer in 0.001 seconds. [06:03] Tests with errors: [06:03] henninge: Pastebin? [06:03] lp.services.job.tests.test_runner.TestJobCronScript.test_configures_oops_handler (subunit.RemotedTestCase) [06:03] is that test a twisted test? [06:03] oops [06:04] henninge: does it have run_tests_with = ? [06:04] sorry, meant to paste http://paste.ubuntu.com/536572/ [06:04] lifeless: I don't konw what that is but that string does not appear in the file . [06:05] looks like this http://paste.ubuntu.com/536573/ [06:06] henninge: what about ZopeTestInSubProcess [06:07] actually, I just realized that that I pasted the wrong output but it's similar. [06:08] ah, interesting [06:09] lifeless: no, that neither [06:09] this might be allenaps change to use unicode in subunit [06:10] aha [06:12] * henninge tries with that reverted [06:14] nope,same result :( [06:19] wgrant: we're turning the publicrestrictedlibrarian back on [06:20] because it seems to work in tests [06:20] lifeless: Yay. [06:20] But why? :/ [06:20] Was Apache fixed? [06:20] wgrant: can you give it a thorough spin [06:20] wgrant: To quote spm, FIIK. [06:20] Heh. [06:20] Sure. [06:20] Say when. [06:21] so, r11980 seems to be the culprit for the current test failure [06:21] at least for two of them ... [06:21] the ones that don't produce output [06:23] wgrant: no, no changes in apache. I did a moderate amount of local faffing around to determine that the proposed changes shouldn't make a difference (apache ignores those bits), ie it should "Just Work" as is. So we switched it on, tested, off. and it worked. ie FIIK. [06:23] wgrant: oh, it was live at '19:21 < lifeless> wgrant: can you give it a thorough spin [06:23] ' [06:24] lifeless: Still broken. [06:25] Can you see https://launchpad.net/~soyuz-team/+archive/ppa/+build/1334368? [06:25] The build log link there. [06:25] And lots of other similar ones. [06:25] The page you are trying to view cannot be shown because it uses an invalid or unsupported form of compression. [06:26] I'm not in soyuz-team [06:26] wgrant: gimme a url [06:26] lifeless: Neither am I, but I have a sub. I thought you might be a commercial admin. [06:26] https://i35262589.restricted.launchpadlibrarian.net/35262589/buildlog_ubuntu-karmic-i386.openproj_1.4-2%2Bpx1_FAILEDTOBUILD.txt.gz?token=b8e6f2f070469191af32ce99bb92fae8 [06:27] StevenK: Right, it's the usual uncompressed 404, served with gzip headers. [06:27] So there are two levels of WTFery. [06:27] Haha [06:28] HTTP request sent, awaiting response... 404 Not Found [06:28] Ah yes [06:30] spm: please turn it off again [06:31] done [06:31] so how did it work for me/us? but not here? [06:31] Oh. its the 404? [06:31] It works for some files. [06:35] spm: ok, so it *is* the 404s [06:35] not the files themselves, the 404 response? [06:35] wgrant: does that same build log show properly now? [06:36] Repeating the same wget still gives me a 404 [06:36] of course [06:36] you need the source url [06:36] on the appservers [06:36] thats what redirects or proxies [06:36] Yes, the build log now works [06:36] lifeless: Proxying works fine. [06:37] wgrant: whats the original url ? [06:38] https://launchpad.net/~soyuz-team/+archive/ppa/+build/1334368/+files/buildlog_ubuntu-karmic-i386.openproj_1.4-2%2Bpx1_FAILEDTOBUILD.txt.gz ? [06:39] I suspect url mangling [06:39] I need to chat to spm briefly [06:39] * spm questions the brief part [06:39] Bwahaha [06:46] so I think its the % [06:51] wgrant: whats the debian version string for that, unmangled [06:51] 1.4-2+px1 [07:02] thanks [07:03] so, % urls are mangled by something in the request path [07:03] they are decoded [07:03] and then passed on decoded?! [07:03] * lifeless bets on apache [07:03] wouldn't know a standard if it bit it [07:03] lifeless: I was thinking that [07:13] https://bugs.launchpad.net/launchpad-foundations/+bug/681675 [07:13] <_mup_> Bug #681675: content coding for build logs is done in apache rather than the librarian [07:13] updated https://bugs.launchpad.net/launchpad-foundations/+bug/677270 [07:13] <_mup_> Bug #677270: apache/squid breaks restricted librarian on urls with percent encoded characters. [07:14] Anybody have a clue for me? This test is failing *even* when I revert to the revision of the last successful buildbot run. [07:15] http://paste.ubuntu.com/536585/ [07:15] It start two jobs and both seem to fail on _pythonpath_. Is that a local error for me? [07:16] I am getting this on devel r11979 [07:16] So does buildbot, it seems [07:16] which is said "last known good" revision [07:16] henninge: If changing devel doesn't fix it, I'd blame an egg change? [07:16] hm [07:17] henninge: 11980 is the twisted runner change, I was thinking it was to blame :) [07:17] but doesn't version.cfg control which version of an egg a revision is using? [07:17] jml: ^ [07:17] henninge: You need to remake. [07:17] henninge: you have to run bin/buildout again [07:17] lifeless: it is to blame for the other two failing test [07:17] wgrant: make clean ; make build? Did that ... [07:18] henninge: Hmm. [07:18] * henninge tries bin/buildout [07:18] there's probably a missing dep on versions.cfg [07:19] no change [07:19] buh [07:21] * StevenK stares at Hudson: "took 5 min 5 sec" [07:21] However, r11980 is devel #250 on Hudson, and that didn't fail any tests [07:23] * henninge tries r11980 locally again [07:28] * henninge notices that db-devel passed revision 10000 [07:28] org.apache.commons.jelly.JellyTagException: jar:file:/var/run/hudson/war/WEB-INF/lib/hudson-core-1.377.jar!/hudson/model/AbstractBuild/sidepanel.jelly:34:42: java.lang.OutOfMemoryError: Java heap space [07:28] Wheee! [07:28] \// [07:28] Java, you suck, and I hate you [07:29] I think Hudson is now unhappy, since it now gives 503s [07:31] And even /etc/init.d/hudson stop ... doesn't [07:33] Heh [07:33] Right, it does, it just takes 2 minutes? Srsly? [07:34] you were in swap [07:34] Grah, it is 50 MiB into it [07:34] How is 1.5GiB not enough for Java?! [07:35] how much memory did you tell java it can use? [07:35] Whatever the default is? [07:35] you might have been swapped out even if you weren't exceeding the machine memory in one proc [07:38] It's still 48MiB into swap with apache and java dead, so it isn't that [07:38] This machine still has a full Launchpad development environment on it, so that can't be helping [07:41] And restarting Hudson means it has lost track of the build it was in the middle of [07:41] * StevenK shoots the buildslave in the head [07:47] Project parallel-test build (15): STILL FAILING in 34 sec: https://hudson.wedontsleep.org/job/parallel-test/15/ [07:49] lifeless: Do you want me a queue another parallel-test build? [07:49] StevenK: ask me on the 3rd [07:49] StevenK: of january [07:50] sorry, the 4th [07:50] By which it would have landed? [07:51] no, by which I'll be back at work [08:00] lifeless: "by which I'll be back at work"? you haven't heard? we're replacing you with a perl script. I thought you knew. ?? [08:00] spm: never a win [08:00] #!/usr/bin/perl -w [08:00] see [08:00] while (1) { nag(); } [08:00] minus win [08:00] nah. -w is for wussies [08:01] Perl coder for six years at previous job. Old habits die hard === almaisan-away is now known as al-maisan [08:16] And there goes X in a screaming heap [08:37] good morning [09:06] hello [09:07] who's fixing the build? [09:17] henninge: hello [09:18] henninge: have you submitted anything to fix the build? [09:18] I'm confused. [09:18] r10005 doesn't look like a testfix to me... [09:18] jml: I hoped to find out what to do about the third test still failing [09:19] not much point in fixing two out of three tests [09:19] wgrant: what about it? (I haven't looked, trying to deal with the current failure) [09:19] wgrant: no but then db-devel wasn't broken [09:19] henninge: agreed. just wanted to check [09:20] henninge: It's labelled as testfix. [09:20] hey look [09:20] But doesn't seem to have a test fix. [09:20] it's ZopeTestInSubProcess again [09:20] ffs [09:21] wgrant: sorry, jtv is just telling me off about it, too ... [09:21] "now now henninge, that's not nice" [09:21] there, see? [09:21] I think it might confuse the QA bot a bit, but I guess we'll see. [09:21] we'll see ... [09:22] I wonder what happened re upgrading subunit [09:23] I see we're at the "someone should do something about this" phase of discussion [09:23] ok. [09:23] jml: That's surely the best phase. [09:24] wgrant: I don't know, I'm a "someone else just did something about it" kind of guy. [09:30] henninge: test_timeout fails for you locally, even if you revert to r11979? [09:35] we should change these to be testtools RunTest thingies anyway [09:37] presumably they are in sub processes for a good reason. [09:38] jml: yes, it does. [09:38] lifeless: I mean, there should be a testtools subprocess RunTest thingy [09:38] lifeless: rather than using a mixin with a run() method [09:38] +1 [09:39] henninge: ok. so something is very weird. [09:39] because a 'known good' build is breaking [09:40] I'm going to try to get more info on what the child process is doing [09:40] I *really* doubt r11980 has anything to do with these failures, fwiw. [09:41] jml: Its plausible [09:41] jml: if the reactor is in a surprising state [09:41] it forks, it doesn't exec [09:44] how can I get at the stdout & stderr of the child process? [09:45] hello [09:46] * jml cheats [09:48] lifeless: oh wait, it's not plausible at all [09:48] jml: ok [09:48] lifeless: because the test fails with r11979 too [09:50] also fails when not run in a subprocess [09:55] jml: to be clear: I could verify that the other two failing tests (*not* the timeout test) pass on 11979 but fail on 11980. The timeout test failure does not seem to be connected to that revision. [09:56] s/verify/verify locally/ [10:01] henninge: do you have a plausible reason as to why? [10:02] no, I don't ... :( [10:02] actually ... [10:03] allenap: I don't think it's related to the unicode thing [10:03] henninge: I suspect bad local state, actually [10:04] jml: As well as inside the buildbot slaves? [10:04] StevenK: yes. [10:04] jml: I don't deny that. Did you try it on your machine? [10:04] jml: the only clue is this: http://paste.ubuntu.com/536585/ [10:04] I find that a little hard to believe. Can haz proof? === jkakar_ is now known as jkakar [10:04] The test starts two jobs wich both fail on a missing _pythonpath_ [10:05] StevenK: working on it. [10:05] But I don't know how that can happen just for those two and not for everthing else. [10:05] the clue is that reverting to about 20 revisions ago still has the failure [10:05] good point [10:06] jml: Hudson might actually be your proof [10:06] jml: Now that I think about it [10:07] also, remember, the Makefile has been changed four times recently (11937, 11957, 11958 and 11979) [10:08] StevenK: in that it works? [10:08] jml: Indeed [10:08] jml: But the Hudson slaves are different to the buildbot slaves [10:08] naturally [10:10] how do we clean out the build on buildbot? [10:10] I suspect we run make clean only [10:11] how do we move from suspicion to certainty? [10:12] We channel mars, or someone who actually does more with buildbot than try and replace it [10:16] * jml has to go [11:06] rf-get and update-sourcecode doesn't want to fix the lack of a AsynchronousDeferredRunTestForBrokenTwisted module.... [11:09] jml: I did find an interesting problem. The read side of the pipe in ZTISP was being opened rU. Locally that has caused tests to error, but with no output at all. Changing it to rb means that they succeed. I don't really know why, but perhaps this is related to the unicode thing too. I'm landing a branch now to revert the getreader() thing, and to open both ends of the pipe as binary. === matsubara-afk is now known as matsubara === al-maisan is now known as almaisan-away [12:31] bigjools: that sucks. will try to look into that after the build is fixed [12:31] allenap: cool. that fixes all three failing tests? [12:32] jml: a make clean fixored it [12:32] bigjools: ahh good [12:32] not sure why I needed that though :( [12:36] bigjools: me neither. maybe I did something a bit stupid as I developing it. [12:36] it's not clear how to evolve Launchpad & an upstream in parallel [12:38] hmm [12:38] I wonder how I can make a script that downloads an openid-protected page [12:40] jml: I think it should fix them, but then I haven't been able to replicate the exact failures. I'm hoping :) [12:41] jml: see utilities/pastebin [12:41] well, the old one [12:41] jml: from faustian_pact import convenient_openid_wrapper [12:41] hmm. I don't think I have that one. [12:41] sure you do, you have bzr repo? :) [12:42] bigjools: faustian_pact is in Launchpad's history? [12:42] maybe not that one [12:42] I can't see you facing allenap on IRC when you talk [12:43] yeah, they probably haven't ported that to kde yet [12:44] db_lp broken again [12:45] any volunteers? [12:48] * bigjools 's kde sarcasm detector app goes off [12:48] allenap: test_timeout still fails for me with your branch, but the other two are fine. [13:01] https://bugs.launchpad.net/launchpad-foundations/+bug/681770 [13:01] <_mup_> Bug #681770: Failure in lp.services.job.tests.test_runner.TestTwistedJobRunner.test_timeout === jml changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 2 of 10.12 | PQM is open | firefighting: db-devel broken | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting [13:34] jkakar_: hi [13:36] Hiya jml! === jkakar_ is now known as jkakar [13:37] jkakar: how's things? [13:39] jml: Going really well, thanks. I'm sprinting in Barcelona, so it's been an exciting week. Just got back from lunch. [13:39] jml: How about you? [13:40] jkakar: well. bit of a cold this week but otherwise fine. slowly mastering the art of the automobile [13:41] jkakar: you coming to London any time soon? [13:44] (also, yay sprinting) [13:47] jml: Driving, oooh, fun times. :) [13:47] jml: Will be in London in a couple weeks, yep. [13:48] all confirmed? [13:57] bloody windmill [13:58] bigjools, ? [13:58] it hangs when I run it locally, which means my test run never completes [13:59] ah, not fun. That is new, I don't think anyone has mentioned that yet [13:59] did Deryck kill the windmill suite? [13:59] (yet?) [13:59] never mind Deryck, I want to kill it :) [13:59] heh [13:59] bigjools, I should ask: is this problem new? [13:59] speaking of which... [13:59] jml, your @skiptest decorator is the big win for us there [14:00] the build is still broken, failing with a windmill test. I suspect a fix like http://pastebin.ubuntu.com/533852/ might be needed [14:00] mars: I'm not sure :( [14:00] it only happens on one of my boxes [14:00] mars: wasn't mine, I don't think [14:01] * jml doesn't actually know anything about windmill, that paste is just what deryck did last time [14:01] jml, that fix didn't land? [14:02] mars: it landed. a different test is failing. [14:05] jml, if the needed fix is similar to what you posted, then it will have to be applied by inspecting the test for races caused by missing timeouts [14:06] mars: make sure you pass that on to whoever sets about actually fixing it. [14:06] sure, I can do that [14:07] Deryck's windmill fix has not landed [14:08] well, I guess we'll look at the bug filed for that particular breakage and find out why === matsubara is now known as matsubara-lunch === almaisan-away is now known as al-maisan [14:46] jml: is it only you and me today on The Strategerie? === jelmer_ is now known as jelmer [14:56] flacoste: https://bugs.launchpad.net/launchpad-foundations/+bug/681770 [14:56] <_mup_> Bug #681770: Failure in lp.services.job.tests.test_runner.TestTwistedJobRunner.test_timeout [14:57] jml: lost connection? === matsubara-lunch is now known as matsubara [15:26] would it be evil to use a storm validator to auto-update a different field? === salgado is now known as salgado-lunch [15:45] oh dear, it's not good when the same class name appears twice in a test file is it ... === beuno is now known as beuno-lunch [16:02] flymake! pyflakes! happiness! [16:02] pyflakes didn't detect it [16:03] I only noticed because on of my new tests wasn't getting run [16:03] one* [16:03] really? [16:03]