[00:02] thumper: is there a bug report for the fact that i can be requested to review a branch i can't see? [00:03] mwhudson: probably [00:03] mwhudson: I don't remember one [00:04] it rings a vague bell [00:06] yeah, me too [00:10] thumper: can't you actually fix https://bugs.edge.launchpad.net/launchpad-code/+bug/624009 now? [00:10] <_mup_> Bug #624009: Email should not be sent for new WIP merge proposals [00:10] yeah... [00:10] it is still showing on our kanban board :) [00:10] and my bug is there https://bugs.edge.launchpad.net/launchpad-code/+bug/330290 [00:10] <_mup_> Bug #330290: Can request a review from someone who can't see the merge proposal

[00:31] $ time ./bin/py -c pass [00:31] real 0m0.610s [00:31] that's quite slow... [00:33] $ strace ./bin/py -c pass 2>&1 | grep -c stat [00:33] 19347 [00:33] that's quite a lot of stats [00:33] * mwhudson wonders if this has regressed again [00:37] buildout ftw [00:37] ? [00:37] yeah, to some extent certainly [00:37] (by which I mean pth files, egg lookups etc) [00:37] stat()ing /home/mwh/canonical/checkouts/trunk/eggs/RestrictedPython-3.5.1-py2.6.egg three times seems a bit silly [00:37] that stack needs to be tuned and optimised for everyones sske [00:39] there is this pattern of stats for every egg afaics: http://pastebin.ubuntu.com/515317/ [00:44] * thumper has errands in town, back soonish [01:00] lifeless: can you please look at https://pastebin.canonical.com/38714/ [01:00] do you agree with the approach? [01:00] mmm [01:01] a few concerns [01:02] 'config' is establish as an idiom meaning 'the current config' [01:02] so overloading that will confuse people [01:02] lets not do that [01:02] so i just don't use the alias? [01:02] secondly, testrunner-appserver is hopefully deletable [01:02] see the current thread on the list [01:02] even if isn't deletable, it won't be usable because its still going to be static [01:03] that said, I'd expect that individual tests *already* are running in that config, aren't they ? [01:03] not sure. i can check. [01:03] (if not, how are they managing to connect at all ?) [01:04] one of the ones i'm looking at to start is lib/canonical/launchpad/doc/launchpadlib.txt [01:04] if the answer is 'a subprocess is run in that config' then the approach of instantiating a specific config to let us refer to the config the external process is using makes sense, but lets not make it a global [01:04] for tw oreasons [01:04] two reasons [01:05] firstly, I'm this || close to sending in a patch to dynamically generate configs in the test run, so any global like that will be stale [01:05] secondly we have way too many globals, lets reduce them. [01:05] sure, i was trying to fit in with how stuff had been done previously [01:06] I appreciate that :) [01:06] and I'm glad -very glad- you're looking at this [01:06] i can just create a config instance on the fly: config = CanonicalConfig('testrunner-appserver') [01:06] something like [01:06] appserver_config = BaseLayer.getAppserverConfig() [01:06] is probably whats needed [01:07] the hard coded string there would be a problem [01:07] ok. my exposure to this stuff is still very limited :-) [01:07] mine too [01:08] I started looking at the config system code on sunday :) [01:08] so the BaseLayer.getAppServerConfig() should return me the correct config instance? [01:08] i started 30 minutes ago :-) [01:08] I think that that will work [01:08] ok. thanks. i'll suck it and see [01:09] there's quite a few hard coded urls to fix. that's why i wanted to ask first :-) [01:10] wallyworld: want a chat? [01:10] wallyworld: yeah [01:10] wallyworld: what I'm suggesting will give us one place to change if we need to change [01:11] \o/ databasefixture has landed [01:11] thumper: ok [01:12] lifeless: that's what i was looking for too. i'll let you know how i get on. thanks for the input [01:33] lifeless: compile failed? [01:34] thumper: did it? [01:34] lifeless: aye [01:35] lifeless: can you parse the error? [01:36] Could not locate /srv/buildbot/slaves/launchpad/lucid-devel/build/orig_sourcecode/eggs/setuptools-0.6c11-py2.6.egg/setuptools/package_index.py:155: UserWarning: Unbuilt egg for ClientCookie [unknown version] (/usr/lib/python2.6/dist-packages) [01:36] ops issue i think, machine was being upgraded or some such [01:37] hmm? [01:40] lifeless: i presume the plan after edge and lpnet are running the same code is to remove the edge.launchpad.net domain? [01:40] well, not remove, deprecate & redirect from [01:40] wallyworld: https://dev.launchpad.net/Code/BranchRevisions [01:43] mwhudson: yes [01:44] spm: was any maintenance being done on that machine [01:44] spm: that could cause that crazy output [01:46] the lucid-devel slave? [02:06] spm: yes [02:07] wallyworld: I'm exposing the config for you as BaseLayer.appserver_config_name [02:08] lifeless: ok. i'm just coding everything now. i can make my branch dependent on yours? [02:08] sure [02:08] for now i'll use a string literal in the getAppServerConfig, just to get stuff working [02:08] sure [02:10] How often will qastaging's DB be clobbered? [02:14] Project db-devel build (78): STILL FAILING in 4 hr 4 min: https://hudson.wedontsleep.org/job/db-devel/78/ [02:25] lifeless: is that a moderately recent error? because there hasn't been any work on the slaves in ~ 4 days ish. afaict/find [02:25] spm: today [02:25] wgrant: same time stagings is [02:27] lifeless: http://www.mail-archive.com/launchpad-dev@lists.launchpad.net/msg02726.html ?? [02:28] ie. lmgtfy ;-) [02:28] spm: huh?! *today* [02:28] same error [02:29] we've seen this beofre. it's funky crap inside the test suite [02:29] so you're thinkinging dirt in the lp buildbot build area? [02:29] no. I'm thinking funky crap inside the test suite(s) [02:30] spm: the only change was mine, and it passed ec2. [02:33] lifeless: not sure. All I can say is that we've seen that error at least once before, back in March. it was weirdness inside the lp code/tests/includes/whatevers. I'd focus there. [02:33] lifeless: what are we doing about the thread garbage in ec2 tests? [02:39] The current policy seems to be "lalalala I'm going to throw it at the wall until it sticks"... [02:40] :( [02:40] Although Hudson's incessant whinging has almost convinced me to dig into it. [02:42] thumper: I'm going to talk with stub again [02:42] ack [02:42] I was advocating disabling the check last week [02:43] I think it has to be downgraded to a warning for now, at most. [02:43] but I need data [02:43] The remaining failures are just Windmill, aren't they? [02:48] I think so [02:57] poolie, hi. Do feature flags also work with the API? [02:58] what do you mean? [02:58] can code called by the api consult flags? [02:58] i think it will work, but you may be the first to do it [03:08] poolie, I mean the web api. If I land my work, it's going to show up in the web api, isn't it. [03:10] rockstar: yes, I think so [03:10] thumper, yeah, that's what I figured. [03:11] I'm not sure whether or not I care enough to try and fix it. I mean, merge queues have been available through the API for a while. :) [03:11] I guess I'll just need to be quick. [03:13] rockstar: i think it should just work but it depends on to what extent the web api uses the standard publication stuff [03:14] i would really appreciate hearing that either works, crashes, or silently fails [03:14] rockstar: just be quick :) [03:14] ... by way of a bug in the latter two cases [03:15] poolie, how do you specify in an interface that a property is only accessibly through a feature flag? [03:15] You can't. [03:16] You have to check the flag inside the method. [03:16] i guess you can make a method error in some meaningful way if a flag is not in effect [03:16] a property... dunno [03:21] (I believe the DistroSeriesDifference API stuff is currently protected by flags) [03:22] right [03:22] lifeless: do you think calls like https://lp-oops.canonical.com/oops.py/?oopsid=1751XMLP32 could block the xmlrpc server for the other calls? [03:22] rockstar: i mean presumably you would get... an AttributeError... if it didn't exist at all? [03:22] you could raise that? or Unauthorized, perhaps? [03:25] rockstar: you'd need to change the wadl processing to use a flag in that fashion [03:25] rockstar: as wgrant says, its not designed for static things like that, because it depends on the db [03:25] I'd just do your things, merge queues are classified as unsupported by thumpers fiat ;) [03:25] lifeless, yeah. [03:26] as long as what you land is working at any point, its ok [03:26] lifeless, yeah, that's the plan. [03:29] lifeless: did you boot buildbot? [03:30] yes [03:30] hoping its nonsense [03:30] which it looks like it was [03:50] good morning rockstar + antipodes [03:50] Afternoon. [03:54] wgrant: did you get all three of your branches landed, after the annoying ec2 faliures? [03:55] hi bac [03:57] bac: And other three after that, yeah. [03:57] bac, are you in Japan? [03:57] After a couple of runs. [03:57] rockstar: while i am big in japan i'm actually in vietnam [04:30] wallyworld: rev 11741 in lp:~lifeless/launchpad/uniqueconfig is pushing now [04:30] wallyworld: I'm not done, but the bits you need are done. [04:31] lifeless: thanks. only 100+ more tests to fix :-) [04:31] lifeless: i'm just assuming your rev with include the app config name - i've got methods for all the rest i need [04:32] s/with/will [04:32] wallyworld, did you get your windmill issues fixed? [04:32] rockstar: no :-( but i talked to adam from the windmill devs and he suggested a windmill proxy issue may be the cause [04:32] i'm not 100% sure exactly what that means just yet [04:33] all i know is YUI and windmill don't play nicely together [04:33] wallyworld, yeah. Here's what I've found. The same tests fail (relatively) consistently on my machine, but ec2 has failed 6 different runs with different tests failing each time with no changes from me. [04:34] wallyworld, I don't see how YUI and windmill wouldn't play well together. windmill doesn't know YUI from Adam. [04:34] rockstar: the same general failure mode? ie YUI breaks somehow? [04:34] from what adam said, there's some proxying the windmill does [04:35] s/the/that [04:37] rockstar: adam talked about needing to blacklist a url to prevent windmill proxying it and gave this reference [04:37] http://github.com/windmill/windmill/wiki/Preferences-File [04:37] wallyworld, hm, I was under the impression that the windmill proxy was important. [04:38] i'm not sure - i haven't devoted enough brainspace to understanding the windmill architecure at this point [04:39] i was hoping it would make more immediate sense to you :-) [04:41] wallyworld, it doesn't make much sense to me now, but it's also a bit late right now. Maybe I can make some sense of it tomorrow. [04:41] rockstar: ok. we'll talk then. have a good evening [04:47] wallyworld: it generates unique testrunner-appserver configs in BaseLayer.setUp [04:47] wallyworld: it doesn't allocate unique ports in the generated config [yet] [04:47] wallyworld: and really the whole config thing is backwards here, we want to start the server and readback the port as I described on the list [04:47] EFUTURE [04:49] lifeless: np. i'll look at the branch to see what's there. atm, i'm fixing the test logic but the expected results still contain the hardcoded 8085. if the ports are generated, i'll have to come back and deal with that too [04:50] wallyworld: what do you mean by 'expected results' ? [04:50] wallyworld: the goal is to use different ports so we can run tests in parallel [04:50] lifeless: yeah, i realise that. here's a snippet from on e of the doc tests [04:51] >>> browser.url [04:51] 'http://launchpad.dev:8085/~stimpy' [04:51] see how the expected url value is hard coded [04:51] ok [04:52] i can deal with that. my first aim is to get the test logic sorted out, then come back around and fix this other stuff [04:52] ok [04:52] nuking the darn doctests would be one way ;) [04:52] well, the unit tests do the same thing :-) [04:52] wallyworld: except assertions are easier there [04:52] +1 [04:53] perhaps browser could grow some support [04:53] like [04:53] vhost : launchpad.dev / bugs.launchpad.dev etc [04:53] urlpath : /~stimpy [04:54] the windmill layer setup does this sort of thing [04:55] if browser had that [04:55] then you could write the above as [04:55] >>> browser.vhost [04:55] launchpad.dev [04:55] >>> browser.urlpath [04:55] /~stimpy [04:55] which would probably be easier than converting to unittest in the short term [04:56] or even have a normalise_url, or something. [04:56] yes. doc tests are here for a while yet given how much is in them [04:58] what i am playing with now is I have a method called getRootLaunchpadUrl(sitename='mainsite') which delgates to BaseLayer.xxx [04:58] so with one line of doc in doc tests i can get the url i needed [04:58] but i could look at adding that to the browser as you suggest [04:58] the main thing for doctests is that you need to be able to write the literal expected value [04:59] if you can do that, awesome [04:59] I'm just riffing on the approach is all [04:59] np. it's always very good to discuss different ideas [05:06] https://code.edge.launchpad.net/~lifeless/launchpad/uniqueconfig/+merge/38689 \o/ [05:06] jml: testtools upgrade broke bizarrely [05:20] Project devel build (124): STILL FAILING in 4 hr 9 min: https://hudson.wedontsleep.org/job/devel/124/ [05:22] is there a config serialiser class ? for writing to the configs ? [06:25] spm: Around? [06:25] wgrant: ish... [06:25] spm: The reporter of https://bugs.edge.launchpad.net/ubuntu/+source/beryl-core/+bug/109550 needs your magical banhammer. [06:25] <_mup_> Bug #109550: Black/White/unrendered windows in Beryl [06:26] ahh. a special person eh? [06:26] Spam of some variety. [06:26] Only to that bug, that I've seen. [06:26] awesome. ta. [06:33] StevenK: Ohloh seems to be less glacial now... might be done tomorrow. [06:33] Colour me suprised. [06:34] http://paste.ubuntu.com/515453/ :-( [06:35] StevenK: That can mean a missing export_as_webservice_entry() [06:36] I think. [06:36] What's your diff? [06:37] It can also happen if your thing doesn't get imported into canonical.launchpad.interfaces.* [06:37] But your diff will tell me everything. [06:38] https://code.edge.launchpad.net/~stevenk/launchpad/db-add-derivedistroseries-api/+merge/38189 [06:38] wgrant: It's messy due to the MP being targetted at db-devel, but now I'm free to land on devel, so read with caution [06:44] StevenK: What's the real diff? [06:45] wgrant: The test changes and lib/lp/registry/{interfaces,model}/distroseries.py [06:45] Addition of deriveDistroSeries() [06:49] Ah. [06:50] StevenK: You probably need to patch your 'distribution' argument's interface. [06:50] Since IDistroSeries.distribution is an Interface for circular import reasons. [06:50] wgrant: fyi. account zotted; old comments zotted as well [06:51] You know about the _schema_circular_imports fun, right? [06:51] spm: Thanks. [06:57] wgrant: Oh, right, so I can copy_field, as long as I also deal with circular_imports? [07:01] I think so. [07:01] Anyway, I need to run. [07:28] \o/ [07:28] dynamic db names working. [07:28] in tests as in. [07:29] lets see if it works for the librarian too [07:32] seems too, but something remade the ftest db. Hmm. [07:53] O M G [07:53] lp test start reads from dev/random [07:55] lifeless: Does it support sharding? [07:59] jpds: it must. Its web scale. [07:59] :!bin/test -vvt test_pgsql --parallel [07:59] Tests running... [07:59] Ran 15 tests in 27.225s [07:59] ^- woo [08:00] StevenK: hey, what instance size are you using in hudson ? [08:31] good morning [08:32] lifeless: The largest [08:32] lifeless: If you're running out of space on /, fiddle things to use /mnt [08:33] StevenK: well, its just that I'm curious how many cores it will get [08:34] lifeless: Oh, right. Lemme check [08:34] StevenK: largest is ill defined [08:34] there are ;argest-cpu, largest-memory etc [08:35] m1.xlarge ? [08:35] lifeless: Picking on one of the running instances, it has 2GB of RAM and 4 QEMU cores. [08:35] c1.xlarge would be interesting to try [08:35] 8 cores [08:36] That is c1.xlarge [08:36] surely not [08:36] Yes [08:36] http://aws.amazon.com/ec2/instance-types/ [08:36] It isn't using ec2 [08:37] oh [08:37] thats right [08:37] it would be nicer if they matched up [08:37] File a ticket? :-) [08:37] But I daresay, it's a problem of resource management [08:37] StevenK: btw [08:37] https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html [08:38] blames you for needing to QA :) [08:38] lifeless: Yes, I'm blocked on dogfood, I'm waiting for Julian and Tom. [08:38] ok [08:38] please file a bug / RT ticket explaining what we need to change to let this get qa'd on qastaging. [08:38] we /have/ to fix this pipeline [08:39] * wgrant can hear people crying already. [08:39] lifeless: Shell access [08:39] StevenK: surely you can come up with a better answer [08:39] StevenK: like 'this is a shell script, so I can write a test driver for it that the losas can run on request' [08:40] lifeless: No, I can't. The bits are infrastructure and I need to reach in and rummage around. [08:40] StevenK: why? [08:40] lifeless: Because I'd like to confirm things look good at a database level before I kick off a script, etc, etc. [08:40] I'd like to be perfectly clear the right things are happening [08:41] StevenK: this sounds automatable [08:41] (also, most of the archive admin tools still require shell access) [08:41] but even if its not, we need to be able to qa on qastaging [08:41] this is a fundamental tenant of being able to do continuous deployment. [08:41] (wgrant: most of them can be run by waldo though, by someone else if given the arguments) [08:42] it needs to be written down [08:42] handed off to a losa [08:42] -or something- [08:42] lifeless: In other news, can you poke at http://people.canonical.com/~stevenk/windmill-thread-debug-r11734.subunit.gz [08:43] forbidden [08:43] mail it to me or something [08:43] * StevenK curses [08:43] lifeless: Fixed, please retry [08:44] so what do you want me to see? [08:45] lifeless: I added some more debugging to the windmill teardown, there's an oddness with a Timer thread there [08:46] I'd say thats a threading subclass [08:46] lifeless: And I also got 'test process died with exit code 2, but no tests failed', which I don't believe. [08:46] used by windmill [08:47] I think that test process thing is damage in zope.testing's subunit support, which I put a patch up for months ago [08:47] benji landed it in the weekend I think [08:47] Oh, damn it [08:47] That revision is qa-ok, but Ursinha's script munged it [08:48] lifeless: When does that page get re-generated? [08:50] StevenK: 10 minutes or so [08:50] Ursinha-afk: can you run it any more frequently now? [08:50] lifeless: As in, 'in ten minutes' or 'every ten minutes' ? [08:50] StevenK: its in cron, I think its a 10 minute frequency [08:53] grr wtf OperationalError: FATAL: database "launchpad_ftest_template" does not exist [08:55] Right, now that page doesn't blame me, excellent. [08:56] Does it blame me instead? [08:56] No, sinzui [08:56] sinzui [08:56] Ah. [08:56] however, anyone can qa [08:56] in principle [08:56] wgrant: We can change it so it always blames you [08:56] Bug 46581 is marked as not-ok. [08:56] <_mup_> Bug #46581: Change a poll type URL manually crashes

=== almaisan-away is now known as al-maisan [09:01] I gotta say, wtf is dropping the template db [09:03] Crash dump was written to: erl_crash.dump [09:03] Kernel pid terminated (application_controller) ({application_start_failure,kernel,{shutdown,{kernel,start,[normal,[]]}}}) [09:03] Wheee, yay for readable logs, rabbitmq [09:04] -rw-r----- 1 rabbitmq rabbitmq 243K 2010-10-18 08:02 /var/lib/rabbitmq/erl_crash.dump [09:04] * StevenK sobs [09:04] Project devel build (125): STILL FAILING in 3 hr 44 min: https://hudson.wedontsleep.org/job/devel/125/ [09:04] * Launchpad Patch Queue Manager: [r=henninge][ui=henninge, [09:04] sinzui][bug=639703] Correctly report bug tracking status for project [09:04] groups. [09:04] * Launchpad Patch Queue Manager: [r=mwhudson][ui=none][no-qa] Cleanup some partially-migrated-from\n\ttest support code. [09:05] * Launchpad Patch Queue Manager: [r=mwhudson][ui=none][bug=660843][no-qa] Move the garbo scripts into [09:05] lp.scripts [09:05] Hm, I think I need to turn that off. [09:09] https://bugs.edge.launchpad.net/launchpad-foundations/+bug/662519 made me cry [09:09] <_mup_> Bug #662519: lp test startup exhausts dev/random [09:21] test run should be finished soon. [09:21] \o/ [09:24] lifeless: I can kill that parallel-test build if you wish [09:24] nono [09:24] its nearly done [09:25] Even with the no _template db errors? [09:25] need to get the details and debug them [09:25] most tests are passing [09:25] its probably the tests that sampledata loading/dumping works, or some such [09:26] hmm https://hudson.wedontsleep.org/job/parallel-test/4/ took 2 hours [09:26] so it may have an hour to go [09:33] wgrant: how much testing have you done on bug 655690? [09:33] <_mup_> Bug #655690: release_files_needed shouldn't live on FTPArchiveHandler

[09:36] Announcing Amazon SNS Management Console looks interesting [09:42] bigjools: I've tested both a-f and NMAF fairly thoroughly. [09:42] Plus the tests are a bit better now. [09:44] want to see a blast from the past? https://devpad.canonical.com/~jml/mirror-failure.png [09:45] meep [09:49] jml: wow [09:49] lifeless: fwiw, something else weird is going on with your branch. You should definitely not get an email like that for test failures. [09:51] jml: the long spew of info? i filed a bug, its having a root logging handler installed that uses std* [09:51] lifeless: no, not the root logger [09:52] lifeless: although that's another problem [09:52] lifeless: I changed the email so that it should start with a list of failing tests, and then have only the error info for those tests [09:52] lifeless: it sends out the full horrible dump only when something surprisingly bad happens. [09:53] but a quick scan of the full log doesn't reveal anything [09:55] jml: what full horrible dump ? [09:56] lifeless: this is what a failing branch email is supposed to look like: Tests started at approximately 2010-10-16 10:04:52.243410 [09:56] Source: bzr+ssh://bazaar.launchpad.net/~wgrant/launchpad/better-publisher-index-tests r9886 [09:56] Target: bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/ r11722 [09:56] 10565 tests run in 3:58:20.128578, 0 failures, 1 errors [09:56] Tests with errors [09:56] ----------------- [09:56] lp.code.windmill.tests.test_branchmergeproposal_commitmessage.TestQueueStatus.test_inline_queue_status_setting [09:56] ====================================================================== [09:56] ERROR: lp.code.windmill.tests.test_branchmergeproposal_commitmessage.TestQueueStatus.test_inline_queue_status_setting (subunit.RemotedTestCase) [09:56]