[00:00] Night. [00:01] lifeless: Could you please send both the branches on https://code.edge.launchpad.net/~wgrant/launchpad/+activereviews off to ec2? [00:03] night jml [00:03] wgrant: url me up [00:04] lifeless: https://code.edge.launchpad.net/~wgrant/launchpad/bug-661109-buildable-architectures/+merge/38529 and https://code.edge.launchpad.net/~wgrant/launchpad/publisher-release-cleanup/+merge/38530 [00:07] any special flags? [00:08] lifeless: The latter is incremental but has a bug of its own, so probably not. [00:08] Oh, hm, the bug might not be linked. [00:09] It is. [00:10] first one is spinning up now [00:11] ‽ [00:11] Traceback (most recent call last): [00:11] File "bin/test", line 73, in [00:11] assert os.environ['STORM_CEXTENSIONS'] == '1' [00:11] File "/usr/lib/python2.6/UserDict.py", line 22, in __getitem__ [00:11] raise KeyError(key) [00:11] KeyError: 'STORM_CEXTENSIONS' [00:12] funky [00:12] 9fixed) [00:14] -ugh- [00:14] File "/home/robertc/launchpad/lp-branches/working/lib/canonical/ftests/pgsql.py", line 263, in tearDown [00:14] self.dropDb() [00:14] File "/home/robertc/launchpad/lp-branches/working/lib/canonical/ftests/pgsql.py", line 321, in dropDb [00:14] cur.execute('VACUUM pg_catalog.pg_shdepend') [00:14] File "/home/robertc/launchpad/lp-branches/working/lib/canonical/ftests/pgsql.py", line 108, in execute [00:14] return self.real_cursor.execute(*args, **kwargs) [00:15] psycopg2.InternalError: right sibling's left-link doesn't match: block 260 links to 259 instead of expected 314 in index "pg_shdepend_depender_index" [00:15] ... pardon? [00:15] That's a new one. [00:15] data corruption in pg [00:16] Looks like it. [00:18] lifeless: Both branches are running reasonably happily now? [00:27] just started second one [00:28] Thanks. [00:31] and my pg is repaired. [00:32] By recreating everything? [00:32] no [00:32] you reindex pg_catalog.pg_shdepend in single user mode [00:33] Ah. [00:33] I had to lookup how to do single user mode in 8.4 [00:33] command line changed :) [00:40] * wgrant cleans up the next 6 branches. [01:10] Project devel build (120): STILL FAILING in 1 hr 33 min: https://hudson.wedontsleep.org/job/devel/120/ [02:02] wgrant: hey [02:03] http://dev.launchpad.net/Contributions?action=diff&rev1=306&rev2=307 [02:03] lists jcsackett as outside the lp team [02:03] this is bogus ;) [02:03] Project db-devel build (74): STILL FAILING in 4 hr 10 min: https://hudson.wedontsleep.org/job/db-devel/74/ [02:03] likewise danilo, ian booth, and benji [02:04] lifeless: Yeah, and a few others. [02:04] * wgrant fixes. [02:06] thanks [02:10] wgrant: is the script in the lp tree, that does that ? [02:10] lifeless: utilities/community-contributions.py [02:11] (kfogel used to run it on a cron job, but that obviously doesn't happen any more.) [02:16] oh, we should do that [02:17] should be easy to put it in a service account on dosium [02:24] lifeless: All looks reasonably sane now. [02:27] Missed one. [02:27] Curses. [02:33] :> [02:46] the sweet sound of deleted code vanishing [02:47] What are you eliminating? [02:47] ftests.harness [02:47] Aha. [02:47] I didn't mean to [02:48] but I needed to make LaunchpadTestSetup be used as an instance not a magical stateless beasite [02:50] and in the course of that I found only one user each of its subclasses [02:50] with those gone it was trivial to do stub's XXX and move LTS to layers.py [02:52] Heh. [02:54] lifeless: Thanks. Will you lp-land it? [02:59] already spinning up [03:00] my databasefixture branch (pushing now) has the cleanup I mentioned [03:01] pushed [03:02] 10 files changed, 106 insertions(+), 296 deletions(-) [04:28] Yay, both succeeded. [04:28] lifeless: Thanks. [04:30] You mean something actually passed ec2? [04:31] Dear ohloh, import faster. [04:31] It's doing 10k every 12 hours or so. [04:32] StevenK: It was sitting in a queue for most of yesterday. [04:32] It took more than a month the first time, but I think their bzr importer is better now. [04:32] Step 2 of 3: Importing source code into database (Running 31872/90554) [04:32] It has another 4 days, by my hand-wavy estimate. [04:32] deryck is still beating me by a few revs :( [04:33] And then I have to defeat henning to make it onto page 3. [04:33] I've lost count of mine [04:33] It's going to be hard to rise above jml, I fear. [04:34] Yes, but jml has been inactive for a while. [04:35] are both of these ignorable ? [04:35] Ah, I will pull ahead of deryck in about 10 minutes :P [04:35] lp.soyuz.windmill.tests.test_archive_packages.TestArchivePackagesSourcesExtra.test_sources_extra_available [04:35] and [04:35] lp.translations.windmill.tests.test_documentation_links.DocumentationLinksTest.test_documentation_links [04:35] lifeless: The latter I haven't seen before. [04:35] [] [04:35] [] [04:35] But the former has been happening for ages. [04:36] is it 'retoss at ec2' [04:36] That's the same symptom as the leak I fixed. [04:36] or direct to pqm time ? [04:36] lifeless: Which branch? [04:36] bzr+ssh://bazaar.launchpad.net/~lifeless/launchpad/paralleltest [04:36] bzr+ssh://bazaar.launchpad.net/~lifeless/launchpad/paralleltests [04:36] The first one is ignorable. I'd also be ignoring the second one, depending on what you changed. [04:36] But I'm bloody stumped if I want to print thread counts during every windmill test. [04:36] Mmm. Not sure. [04:36] I added a file [04:37] which nothing calls [04:37] and has a few unit tests of its own [04:37] and I bumped the fixtures dep [04:39] I'd run it again for safety, personally. [04:39] lifeless: Keep in mind that Hudson {db-,}devel tests fail with similar errors. [04:39] * StevenK needs to pester mars again. [04:39] but I haven't seen the test_documentation_links one before. [04:40] wgrant: Go back through Hudson's history -- they keep changing. [04:40] test_worker was the only constant for a while. [04:41] Come on PQM... work faster. [04:42] How does it take half an hour to merge? [04:42] I've been wondering that myself. [04:42] bzr branch [04:42] Why doesn't it just merge directly from the remote branch? [04:42] its how long its taking to do a full clone [04:43] StevenK: because it runs the precommit checks in a chroot, assuming maximum hostility from the submitter [04:43] Yes, but we're only hostile towards PQM, not trunk. [04:44] I can think of a few ways to address it, including 'allow merges to fuck the repository' [04:44] but you asked why. thats why. [04:45] if the precommit checks didn't call bzr at all (which they certainly used to) we could use a lightweight checkout [04:46] lifeless: Out of curosity, what are the pre-commit checks doing? Just 'make'? [04:46] I presume it's no longer test_on_merge.py. [04:47] I don't have access to the pqm both these days (haven't for years) [04:48] so I can't answer that [04:48] lifeless: Okay, so what did it used to run? [04:48] Idle curosity is all, so I'm not after a definite answer. [04:49] Yay, PQM loves me. [04:49] well it used to do check-on-merge, uhm [04:49] it ran the tests obviously [04:49] but also checked lint for the commit, which needed a diff, which needs bzr history [04:50] another way we might be able to structure it is a http heavyweight checkout of the target branch with/stacked local branch stacked on the target. [06:09] lifeless: :( you added code to canonical.* [06:15] wgrant: and? [06:15] wgrant: I know that there's this big goal to nuke it [06:15] but I have very limited coding time. [06:15] Should I a): [06:15] make the test suite 16 times faster [06:15] or b) [06:15] move some code around to no practice difference [06:16] wgrant: or are you referring to testing.parallel? I guess that could be in lp.services.testing or wherever [06:17] lifeless: testing.parallel, yeah. [06:17] Altering existing code in there is fine IMO. But adding a new file there is probably bad. [06:20] shrug [06:20] can move it later [06:20] ideally that wouldn't exist at all, but I haven't had time to work on testrepository recently, nor am I sure that everyone uses testr [06:29] \o/ unique db names [06:32] Nice! [06:32] I will be very interested to see how this performs. [06:34] now, I need to glue the unique db names up to the config system [06:34] in process and out of process [06:37] Oh urrgh. [06:37] Out of process... hm... [06:38] in process should be easy once I figure out what layer the idiotic configs are read in [06:46] hmm [06:47] whups [06:47] launchpad_ftest | robertc | UTF8 | C | C | [06:47] launchpad_ftest_1099 | robertc | UTF8 | C | C | [06:47] launchpad_ftest_1470 | robertc | UTF8 | C | C | [06:47] launchpad_ftest_1579 | robertc | UTF8 | C | C | [06:47] launchpad_ftest_929 | robertc | UTF8 | C | C | [06:47] launchpad_ftest_978 | robertc | UTF8 | C | C | [06:48] wgrant: do you happen to know the dead chicken for increasing a vm disk image size? [06:51] lifeless: libvirt? [06:57] http://wiki.libvirt.org/page/Tips#Increasing_the_disk_size_of_a_virtual_machine [06:58] If it's LVM it's easy. qcow2 I don't really know... I normally just create a new image and copy the data across. [06:59] which is odd, I've have expected to be able to just loopback it [06:59] Hm? [07:00] I loopback the new and old devices then mount them both. [07:02] loopback as a disk, then ext2resize [07:02] well, fdisk, ext2resize [07:03] Oh, I just resize the virtual block device then do the rest from within the VM. [07:03] well, thats what they say on that wiki page [07:03] it just seems a bit redundant to me [07:11] lifeless: Can you manually submit that c-c.py branch? [07:11] It failed with that LayerIsolationFailure. [07:11] Despite only changing an untested script. [07:21] what should the mangled commit message be [07:21] [r=lifeless][ui=none][no-qa] Update community-contributions.py with new name mappings and statuses. [07:22] and the branch url [07:23] wgrant: ^ [07:24] lifeless: lp:~wgrant/launchpad/community-contributions-updates [07:24] sent [07:24] Thanks. [07:24] wow, we have some leaky shit here [07:26] oh, -ha- [07:26] layers blows up very badly when testSetup goes boom [07:27] wgrant: if you're interested, rev 11738 pushing now [07:29] or 739 which has devel for the parallel tests. [08:00] jml: https://code.edge.launchpad.net/~lifeless/launchpad/databasefixture/+merge/38643 I'm hoping you'll review that today, so I can get some momentum up. [08:08] lifeless: But it's a Sunday? [08:09] StevenK: yes [08:09] and? You can review it if you like; its very mechanical [08:09] I would love it if you did review it :) [08:11] I would, but I'm about to run out the door [09:01] Project devel build (121): STILL FAILING in 4 hr 5 min: https://hudson.wedontsleep.org/job/devel/121/ [10:07] lifeless: What did you do about that STORM_CEXTENSIONS thing? [10:10] make [10:11] Ah. [10:11] Odd. [10:11] Thanks. [10:28] hi ho [10:32] jml: hi [10:32] lifeless: I'm just getting to your branch :) [10:34] awesome [10:34] Project db-devel build (75): STILL FAILING in 4 hr 8 min: https://hudson.wedontsleep.org/job/db-devel/75/ [10:34] I'm polishing the next one - setting a uniqe env variable in BaseLayer [10:35] after that it will be using a custom ftesting.zcml + config dir, though I need to really understand WTF the overrides we write are [10:35] oh [10:35] I think I have an answer to your layer question [10:35] * jml checks [10:35] then, in theory, we're down to fine tuning, fatsam etc. [10:37] actually, I don't [10:37] my guess was going to be that it had something to do with the way zope goes up the inheritance tree for you [10:38] but since there aren't setUp or tearDown methods on LayerProcessController, I have no idea [10:38] I think its because DatabaseLayer has multiple tasks [10:38] but for external tests environments like windmill, we don't need them all [10:38] or something [10:47] yeah, maybe [10:47] lifeless: working through your branch now [10:47] jml: thank you [10:52] jml: lp:~lifeless/launchpad/uniqueconfig is where my next thrust is going, if you're interested. It is selfcontained but not useful until I do config mangling which is why I am not planning on putting it up for review just yet. [10:52] lifeless: thanks [10:52] this is a big branch [10:53] I mention it because it alters the branch you are looking at, a little. [10:53] jml: its nearly all mechanical. [10:53] uhm [10:53] yeah, so I'm seeing :) [10:53] in fact I think its reasonable to say that its all mechanical [10:53] static -> instance [10:53] unused stuff deleted [10:54] and the one tiny patch, adding dbname dynamic allocation support [10:54] ooh, I think I've got to the meat :) [10:55] > - DatabaseLayer._reset_sequences_sql = None [10:55] lifeless: why'd you delete that line? [10:55] because its now held by the _db_fixture [10:57] the fixture has two uses, calculating the reset sequences logic, and doing test db initialisation and restoration (which uses the sequences *from the template db* - but the choice to do that is held by the layer [10:57] not by the fixture [10:57] thats something we can clean up further in future I think. [10:58] lifeless: > + self.dbname = self.__class__.dbname + "_" + str(os.getpid()) [10:58] lifeless: why self.__class__.dbname and not self.dbname? [10:59] because self.dbname would be wrong [10:59] mmm [10:59] rephrase [10:59] self.dbname would be more magical [10:59] I want to be explicit here. [11:00] cool. [11:00] just wanted to make sure it was deliberate [11:01] sure [11:02] DoesNotStartWith sure looks like it could be a simply curry of Not(StartsWith) [11:03] I haven't seen it, but probably yes [11:05] lifeless: also, I've found a case for which Matchers are not a natural fit. [11:05] lifeless: "assert that this Deferred will fail with this exception type" [11:05] well, matching would match the exception [11:06] I have one of those I need to port to testtools, its in testrepository atm [11:06] 'assert that this deferred will raise something matching ' [11:06] return d.addCallback(self.assertThat, Matches(FooError)) [11:06] fsvo Matches [11:06] yeah, I guess [11:06] I spelt it, in regular code as [11:07] err = self.assertRaises(Exception, thing) [11:07] self.assertThat(err, MatchesException(...)) [11:07] right [11:07] this doesn't seem unnatural to me [11:07] well, for deferreds it's a bit awkard [11:08] I guess I could add a convenience function like lambda d, *exc_types: return d.addCallback(self.assertThat, FailureMatches(*exc_types)) [11:09] actually, ugh [11:09] it would need to be more complex [11:09] because you actually want to addErrback instead, but also add a callback that will fail if it gets called [11:10] yes [11:10] but it can be done [11:10] but this glue, isn't it directly equivalent to assertThat + assertRaises ? [11:10] e.g., its the adaption to let the concerns be separated [11:10] well... [11:10] the issue is that assertRaises doesn't fit Deferreds [11:11] and the tricky thing is where to put the deferred glue if not on the base class [11:11] thats getting interesting ;) [11:11] assertThat lives on the base class [11:11] hmm [11:12] this needs some consideration [11:12] hey, do you want to review me moving StartsWith and DoesNotStartWith to testtools? [11:12] so what I meant is that matchers as a solution for adding extra assertion types without doing inheritance magic aren't a good fit for this particular problem, even if they might be a part of the solution [11:12] sure [11:13] jml: ok, I get that now, and completely agree; matchers are always going to be just a component [11:13] jml: https://code.edge.launchpad.net/~lifeless/testtools/matchers/+merge/38645 [11:15] jml: I'd like to roll that into an updated testtools snapshot for lp [11:15] lifeless: sure. [11:15] jml: or perhaps even a release, though its late and I don't want to do a release at this time of night (but if you did I could update LP in the morning) [11:15] lifeless: yesterday, I was looking into the snapshot bug you filed on testtools [11:16] lifeless: I don't know how to fix it [11:17] lifeless: https://bugs.edge.launchpad.net/testtools/+bug/613734 [11:17] oh, I remember [11:17] <_mup_> Bug #613734: Hard to make snapshots for use with buildout [11:17] I mean, I can think of ways to fix it [11:17] but surely there's already a bog standard way of dealing with this [11:18] presumably [11:18] or we could go to commits are releases. [11:18] if we mad it a little more automated [11:22] jml: so, a testtools release, or a snapshot? I'm easy. [11:23] lifeless: release sounds good [11:23] jml: do you perhaps feel like doing one? [11:23] lifeless: not right at this minute, but I could do one today [11:23] that would be awesome [11:24] I'll update testtools in lp tomorrow, rip out the now duplicated code. [11:24] cool [11:24] lifeless: I'm also thinking of making a web page summarizing a bunch of testing stuff that we've done [11:25] lifeless: fixtures, testscenarios, testr, subunit, testtools, tribunal – anything else? [11:25] testresources [11:25] ahh, of course [11:25] bzrlib.tests [11:25] python's unittest addCleanup, load_tests - we provided the protocol and discussion, call it 'moral creation' [11:26] or something catchy ;) [11:26] lifeless: good point [11:27] wgrant: I almost wasn't going to review the branch you just posted, and then I read the description [11:27] jml: Hm? [11:27] more-a-f-cleanup, re pockets [11:28] The code is old and revolting :( [11:28] And I have another 7 or so branches in this series :( [11:31] wgrant: that's a good thing :) [11:31] wgrant: I added getSuite when I was doing the package branches work. I'm glad it's now actually helping more broadly. [11:32] jml: Ahhh, I wondered why nobody was using it. [11:32] I didn't realise it was brand new. [11:32] "brand new" meaning early 2009 [11:33] jml: This code is largely unmodified from 2005... [11:33] if I did it again now, I might have even added a Suite class. [11:33] point. [11:35] What is the policy on sampledata changes in devel these days? [11:36] sodomy non sapiens [11:37] Every few times I make a change somebody tries to kill me. I guess it's worth a try. [11:37] wgrant: I say edit boldly [11:39] wgrant: ec2 landing that branch [11:39] jml: Thanks. [11:47] wgrant: depends on what you mean by sampledata change [11:47] wgrant: if you mean 'deleteing a tonne' DOIT [11:47] if you mean 'adding a bunch'. Uhm, lets talk. [11:50] lifeless: I'm removing DistroSeries.lucilleconfig in favour of ComponentSelection. But one of the sampledata series is broken, having no ComponentSelections. [11:50] So I am adding four rows to ComponentSelection rather than rewriting hundreds of tests. [11:51] doit [11:51] no question that that is the right thing to do. [11:51] Thanks. [11:58] The sampledata regeneration bloats the diff by 700 lines :( [12:00] wgrant: why, its only four rows you said? [12:01] wgrant: put them in by hand :) [12:01] lifeless: The sampledata hasn't been regenerated for a while. [12:01] wgrant: thats not entirely true [12:01] what pg version have you used? [12:01] I can see columns being added. [12:01] The order is fine. [12:01] hmm [12:01] https://code.edge.launchpad.net/~wgrant/launchpad/destroy-distroseries-lucilleconfig/+merge/38648 [12:02] wgrant: its midnight, I'm pumpkining [12:02] but I smell fishy [12:02] Some of it is order, you're right. [12:02] But there is also new stuff. [12:02] I'm running whatever pg maverick has. [12:03] 8.4.5 [12:03] The order in my branch is right. [12:03] It's alphabetical. [12:03] The old one wasn't. [12:03] I wonder how that happened. [12:03] so someone else had a sampledata patch [12:03] last week [12:04] and it didn't have columns [12:04] db-devel has been merged since. [12:04] hmm [12:04] gnight ;) [12:04] Night! [12:04] wgrant: I have one thought [12:05] if you care to do it. [12:05] Sure. [12:05] wgrant: write a test that the sampledata is in some reference order; pin that to the pg version on ec2 at the moment (so it skips if it can't run) [12:05] * lifeless laughs maniacally [12:05] * lifeless goes to bed [12:07] lifeless: g'night. [12:25] lifeless: When you wake up it'd be nice if you could review https://code.launchpad.net/~niemeyer/storm/bug-620615/+merge/38459 [12:25] lifeless: It's a blocker for releasing 0.18 which will fix at least two problems you're experiencing with Storm in Launchpad. [12:43] Project devel build (122): STILL FAILING in 3 hr 42 min: https://hudson.wedontsleep.org/job/devel/122/ [12:43] * Launchpad Patch Queue Manager: [r=lifeless][ui=none][no-qa] Update community-contributions.py with new name mappings and statuses. [12:43] * Launchpad Patch Queue Manager: [r=jml][ui=none][no-qa] parallel testing facility that works with subunit. [12:43] * Launchpad Patch Queue Manager: [r=adeuring][ui=none][bug=655690] Replace the messy [12:43] FTPArchiveHandler.release_files_needed dict with a simple [12:43] Publisher.release_files_needed set of suites. [14:23] Project db-devel build (76): STILL FAILING in 3 hr 49 min: https://hudson.wedontsleep.org/job/db-devel/76/ [14:23] Launchpad Patch Queue Manager: [rs=buildbot-poller] automatic merge from stable. Revisions: 11729 [14:23] included. [18:47] jkakar: hi [18:47] lifeless: Hi! [18:47] jkakar: sure, for future ref I'm be happy with storm folk reviewing such things [18:48] lifeless: Yeah, I figured. At the time, I was the only reviewer, was mentioning it so you could take second if no one stepped up, but therve has now with some interesting questions about it. [18:51] jkakar: that layer of code is pretty opaque, I have to reread it three times each time I approach it :( [18:51] lifeless: Me too. :) [18:59] that makes me a little sad [19:01] jml: I'm not sure why you're using a branch to do the release. The setup.py changes look trivially fine. [19:02] jkakar: also thanks for reading the guide/presentation [19:02]