[00:09] hi all [00:17] hiya poolie [00:17] poolie: re: bug #838469, should improvements to mini-tutorial be submitted for released versions as well, or will they only be accepted for bzr.dev? [00:17] Launchpad bug 838469 in Bazaar "Mini tutorial doesn't explain repository directories" [Low,Confirmed] https://launchpad.net/bugs/838469 [00:19] hi mathrick [00:20] we can take doc updates for released versions [00:22] poolie: great, should I then make a patch against the 2.4 braching point, or should I rebase it, or should it rather be cherry-picked for release branches? [00:23] just do a revision based off lp:bzr/2.4 tip [00:23] OK [06:19] hi all ! Happy Friday ! [06:20] yay [06:54] morning all === vila is now known as vila_at_dentist [07:28] hi, is there already a bug open for bzr-builddeb's lack of support for .tar.xz? that's going to be especially annoying since GNOME 3.3 won't ship other tarballs [07:35] morning all [07:58] jam, hi - you're feisty today! :) === vila_at_dentist is now known as vila [08:25] mgz: I have some ideas about bug #869366, are you looking at it, or just reporting it for posterity? [08:25] Launchpad bug 869366 in Bazaar "TestTCPServerInAThread.test_server_crash_while_responding random failure" [Medium,Confirmed] https://launchpad.net/bugs/869366 [08:25] just reporting, as I ran into it EOD yesterday [08:25] I seem to be getting lucky with random failures on PQM [08:29] looking at the test, it seems possible for the subthread to not do any futher work after the sync [08:32] mgz: yes, that's the race I was mentioning the other day, the same one exists for the test just above, it triggers very rarely and it took quite a while to put my finger on it [08:33] mgz: the main thread really want to wait for both the connection thread to raise the exception *and* to the server thread to re-raise it [08:33] I'm not even sure what tools python gives us to resolve such problems [08:34] I guess in this case try:/raise/finally:/sync2 would work? [08:35] because the exception won't propogate until the test looks for it? [08:35] mgz: the test looks for the exception to be swallowed :) [08:35] mgz: for that particular test, you could call 'server.stop_server()" instead of "server.pending_exception()" [08:35] mgz: the previous test (test_server_crash_while_responding) looks for the exception [08:35] stop can be called twice [08:35] and that way it will have joined its threads [08:36] which is not what the test is about: test_exception_swallowed_while_serving [08:37] this a test about the server not about the client [08:39] we don't want the server to shut down, we want it to keep serving (swallowed_while_serving) [08:39] vila: that is the other test [08:39] "test_server_crash_while_responding" wants it to raise the exception [08:39] we can just change pending_exception to stop_server and it will sync with the clients [08:39] client threads [08:40] maybe connection threads is a better term [08:40] though I guess that doesn't make the tests symmetric except for the exception being ignored. [08:42] with 'that' == test_server_crash_while_responding, this test is about crashing the server, not shutting it down [08:42] pending_exception is what is used to implement shutting down [08:43] which illustrate why I disagreed with your modifications which were about client behavior and as such replacing these tests instead of adding your owns [08:44] bug mgz encounters the issue in 2.4 so your modifications are out of scope here [08:45] mgz: by the way, thanks for filing that bug, I thought I did as this failure has been seen on babune on several occasions (rare enough to not trigger a fix) [08:45] hm yes, I might not have said that in big enough letters in the bug, but this was a landing on the 2.4 branch [08:46] mgz: ... and *never* triggered on pqm AFAIK... before you claimed the unlucky badge :) [08:49] two different random failures in about four landings on 2.4 PQM is pretty good going [08:49] mgz: yup, don't let them go ! :) [08:50] oh hey, looking at babune, lucid is red [08:50] and guess what the failing test is :) [08:50] mgz: but can we assume the cause is the same now that the test is not ? [08:51] ...one thing going for testtools is seeing both tracebacks makes this miles clearer [08:51] vila: if there's still just the one sync before the raise, then yes I think [08:54] mgz: I was kidding ;) test_server_fails_while_serving_or_stopping shouldn't suffer from the same race because it explicitly call stop_server I think... [08:54] hi mgz, vila [08:54] helli pooloe [08:54] meh [08:54] hello poolie ;) [08:55] hellipolis? [09:02] mgz: so the ' Wait for the exception to propagate' is a lie, it waits for the exception to be 'about to be raised' and just doing 'sync.wait()' is not enough to guarantee that. That's where the race is. A possible fix will be to connect with another client and get served properly (i.e. yes, the server is still up and running) [09:02] mgz: trying to be more precise would require hooking somewhere the server really swallows [09:12] mgz: hehe, but that's in fact required as this test server cannot server properly (by test design :) [09:37] is there a way to force bzr to use its built in ssh rather then another ssh that is on the path? [09:38] (installed git, and of course it has an ssh.exe that causes bzr to just hang for some reason) [09:39] AuroraBorealis: "export BZR_SSH=paramiko" [09:39] where do i put that if i'm on windows? [09:39] if you're on Windows it is spelled [09:39] set BZR_SSH=paramiko [09:39] but you can set it in global env vars [09:39] right click My Computer, Properties, Advanced System Settings, Environment Variables [09:40] ah kk [09:40] user or system variable, does it matter? [09:42] AuroraBorealis: doesn't matter for you [09:42] depends if you have more than one user on the machine [09:42] but python getenv can access both ? [09:43] (guess just a python question now =P ) [09:45] bug 641330 is the shelve issue [09:45] Launchpad bug 641330 in Bazaar "unable to unshelve shelved root-id change" [Medium,Confirmed] https://launchpad.net/bugs/641330 [09:45] bug 82555 looks like the root cause [09:45] Launchpad bug 82555 in Bazaar "Merging to an empty branch doesn't work" [High,Confirmed] https://launchpad.net/bugs/82555 [09:46] bug 242175 is the mitigation for that Riddell added for the merge case [09:46] Launchpad bug 242175 in Bazaar "ValueError: WorkingTree.set_root_id with fileid=None when merging into empty branch" [High,Fix released] https://launchpad.net/bugs/242175 [09:48] AuroraBorealis: windows merges them before starting a process as your user [09:48] k [10:15] Do bzr-builddeb's `import-upstream` and bzrtools's `import` use the same change detection / file-id (ie, file checksum, perhaps) algorithms? [10:16] (anyone know?) [10:16] not me :> [10:17] AfC: hi [10:17] AfC: change against what? [10:17] jelmer: hi [10:18] jelmer: so, in both cases, there's considerable work done to detect (for example) renames, right? [10:18] [so I infer] [10:18] AfC: no, no renames are inferred in either case as far as I know [10:18] oh [10:19] I could be wrong, I don't have much experience with either [10:19] actually, "bzr import" could help [10:19] since it doesn't commit [10:19] so you can do "bzr import ..; bzr mv --auto; bzr ci -m 'import x'" [10:19] jelmer: ok, well, it's doing *something* for the minutes of CPU time its sitting there burning away [10:19] mv --auto, yeah, that might be interesting [10:20] AfC: which one is burning CPU? import or import-upstream? [10:20] [both] [10:20] I just now did an import; import-upstream was taking ~9 minutes each. [10:20] AfC: what are you importing? [10:20] :) [10:21] that large well known project prominently using someone else's version control system that I've been blundering along with this p [10:21] past week [10:22] if its git, we are already seeing that fast-import or whatever is taking huge amounts of time / memory. [10:23] AuroraBorealis: so I'm not working with bzr-git in this case; I've instead used bzr-builddeb's facilities against the upstream project's release tarballs. [10:23] ah [10:23] AfC: I don't think either import or import-upstream have been optimized for kernel-sized trees.. [10:24] AfC: any particular reason you're not using the launchpad imports, or are you just seeing how far you can get? [10:24] jelmer: that's ok, I'm not bitching about import performance per se [10:24] jelmer: so, having established the deterministic behaviour, I tried the lp imports branch [10:25] jelmer: but I kept getting swap thrash. I'm sure I'll be able to make use of it at some point, though [10:26] AfC: you're getting swapping trash even just running "bzr branch lp:linux linux" ? [10:28] yeah [10:28] [anyway, that's done, I let it run overnight or whatever] [10:28] AfC: what version of bzr are you running? [10:28] jelmer: daily [10:28] is lp:linux already a bazaar repo? [10:28] yes [10:29] AuroraBorealis: ^ [10:30] weird that its thrashing when its just branching. [10:30] it just took a long time; it was quite a few days ago; I've had bigger problems since then doing other stuff, but as I said, it's here now should I need it. [10:31] bzr tags [10:31] might be related to the uhh [10:31] bug i'm having when importing the git linux repo [10:32] where its using so much memory (therefore thrashing if you don't have enough ram) cause its keeping the entire history in memory [10:35] AuroraBorealis: where are you getting that? using bzr-git, bzr-fastimport, regular clone? [10:36] none of them should really be keeping all history in RAM [10:37] https://bugs.launchpad.net/bugs/864217 [10:37] Ubuntu bug 864217 in Bazaar "MemoryError when repacking repository with large numbers of revisions " [Medium,Confirmed] [10:37] doing fast-import of the git linux kernel uses about 2 gigs of memory , and it wasn't even done [10:37] jelmer: well it is attempting to keeping 3.0 GB of something in memory. This doesn't go down well :/ [10:39] and mgz said something about it was keeping history in memory [10:39] jelmer: (this was even with bzr branch -r 1 URL; I never could get it to finish with -r > 100) [10:40] (I imagine looping one revision at a time over n days would have worked out, but having established it was deterministic and didn't have missing revision relative to the lp import, I knew I could trust that) [10:46] AfC: did you see the bug I filed earlier about -r1 not actually doing that? [10:48] jelmer: from a (partial) memdump AuroraBorealis had from fast-import, there was a _KnownGraph taking up a bg chunk of memory [10:48] that may well be a bug, there's a _known_heads cache dict that seems essentially unbounded [10:49] it only ever gets cleared on a couple of error paths [10:49] have you worked on meliae not actually dumping all the objects? [10:49] so we can see more of whats in the dumps? [10:49] nope, but I've got that down as a project for this weekend [10:50] kk [10:51] i should be around, although you seem to be on the other side of the world from me so hours will be interesting =P [10:52] on Monday I saw a reproducable segfault, some type errors, and the issue with missing objects causing the dump to not load [10:53] fixing all those should get us somewhere, but the incomplete dump may still be a win64 size/alignment issue which will be a pain to work out from here [10:53] oh gee ;< [10:53] do you have access to a win64 computer? [10:53] no, that's the fun bit :) [10:54] the code has the wrong size of all the pointer->int casts though, so there's some hope I can just fix things till it magically works [10:57] i'll mail you my crappy 500 dollar laptop with windows 7 64 bit on it thats collecting dust behind me === Vorpal_ is now known as Vorpal [11:00] jelmer: no, I didn't; I'll look [11:00] where on the other side of the world are you AuroraBorealis? :) [11:00] arizona, usa [11:01] I would have thought you would have to go a bit further north than that to see the aurora borealis. Unless we just got spanked by a giant gamma ray burst... [11:02] Plate tectonics. Just takes a little longer. [11:02] ah, yes, of course :) [11:03] yes [11:03] ('course, it does make you wonder what kinda nutcase would be in AZ, awake, and on IRC at this hour :p) [11:03] [I only ever saw it once. I was rather far up, as I recall. Was pretty] [11:03] the nutcase who was procostinating on his small prolog assignment [11:03] * mgz thinks fullermd isn't one to talk [11:03] Saying 'prolog' isn't helping your case! [11:04] haha [11:04] Hey, I'm a solid hour ahead of him. That makes me totally sane. [11:07] haha [11:11] anyway, bed time for me. school in a few hours [11:11] chat with you this weekend i guess mgz === medberry is now known as med_out === beuno is now known as beuno-lunch [16:05] james_w: would you mind merging my branch? [16:10] sure thing [16:28] jml, james_w : sry about that, I planned to do it but get interrupted longer than expected [16:29] vila: np === med_out is now known as medberry === beuno-lunch is now known as beuno === deryck is now known as deryck[lunch] === cody-somerville_ is now known as cody-somerville === yofel_ is now known as yofel === deryck[lunch] is now known as deryck [19:22] is there a way to do a sparse/bare checkout in bzr? in svn i'd do --depth immediates and then svn up --set-depth infinity on the folders i wanted, looking for something similar [20:19] when merging someone else's branch, should I commit, then do fix ups, or can I merge, fix + commit result and still be able to usefully differieniate the two sets of changes? [20:36] elmo: difftastic/difftacular - I never remember the name - can untangle them for you [20:37] so sure, do whatever ;) [21:45] lifeless: can I pester you for your thoughts on bug 866107 at some point? [21:45] Launchpad bug 866107 in Bazaar "Use testresources for bzr selftest" [Medium,Confirmed] https://launchpad.net/bugs/866107 [21:48] sure, not right now sadly [21:48] I shall ping again over the weekend at some point. [21:53] and in other news, I have once again made PQM fail in some weird way [21:53] this time... all the tests pass but... [21:54] star-merge succeeded at Fri Oct 7 16:26:04 2011 (0:24:02.142211) [21:54] 'Exception processing merge: [Errno 2] No such file or directory' [21:55] O.o [21:56] stderr has the slightly suspicious: [21:56] Fri Oct 7 16:02:43 UTC 2011 : selftest starts [21:56] failed to open trace file: [Errno 13] Permission denied: '/you-should-use-TestCaseInTempDir-if-you-need-a-log-file' [21:56] Fri Oct 7 16:26:01 UTC 2011 : selftest ends [21:56] but there's no mention of a test with that issue in the test suite output [21:58] is there a cli command to get the latest revid out of a repo... ie. analogue of revno? [21:58] red herring, it's a leak somewhere, has been there for ages [21:58] wgz: ^ [21:58] okay, so it's the merge failing somehow that's the issue [21:58] but... after all the tests run, which is just weird [21:59] wgz: more precisely some TestCase should be a TestCaseInTemp... err, yeah, that's what the path says ;) [21:59] hazmat: `bzr revision-info`? [21:59] wgz: could be related to the issue with the lack of mails to bazaar-commits [22:00] I'll post to the mp so MvG knows I'm not just holding out on him. [22:03] wgz: anyhow, testresources is good; I plan to supercede its model with testfixtures once I or someone else implements a fixture graph with reasonable introspection semantics (such that you can do the testresources planning on unsetup fixtures) [22:12] lifeless: do you have any particular opinions about what bzr needs on this front? [22:12] wgz: I'd write your stuff as fixtures and use a shallow fixture->resource-manager shim [22:13] as I don't expect you to need deep route planning, merely aggregation of all the tests using a given external server [22:14] one part of your fixtures readme really confused me [22:15] resources had a clear part about dependent resources [22:15] wgz, perfect thanks.. didn't show up on a bzr help commands | grep rev [22:15] whereas fixtures had a long example about how you couldn't share a tempdir if there were also a db and a webserver [22:16] which... I didn't get at all [22:17] fixtures is a simpler model [22:18] but that seems simpler to the point of not being useful [22:18] fixtures has seen more use in less time [22:18] this is one of the paradoxes of design :) [22:18] if I can't share the tempdir created by TestCaseInTempDir (with some added carefulness in reset), I can't build any other fixtures of use on top of it [22:19] of course you can, if you want to [22:19] nearly everything thats expensive enough to want to share between tests first derives from TestCaseInTempDir [22:20] does testresources basically get around this by running all webserver tests together, then all db tests, and if it can't do that tearing down the lot between? [22:22] well back up a step [22:22] I think you have some conflation [22:22] this is likely. [22:22] TestCaseInTempDir using unique temp dirs for each test case [22:22] it shares a ROOT_DIR [22:22] this is a bad thing, because it's global state [22:22] yes [22:22] its something we were working to nuke [22:23] this is something I'd like to nuke with one of your nice packages :) [22:23] anyhow, fixtures describes an extended context manager basically [22:23] which happens to be very useful for tests because it exposes things like getDetails [22:23] and a convenient addCleanup for the code within it. [22:24] testresources describes an optimisation framework for expensive things; each of which could be a fixture (but needs a graph which fixtures doesn't expose) [22:25] the fixtures README talks about using just fixtures in similar situations to what testresources is targeted at - and in that world its kludgy, because of the lack of an introspectable graph [22:25] (you can't tell that the webserver and db are going to be using the same temp dir, and you can [22:26] 't (if they use addFixture) cleanup just the webserver or just the db, because that will implicitly cleanup the tempdir, hurting the other user [22:26] so [22:26] this is why I was saying to use testresources, but write the resources as a manager shimming across to a fixture [22:26] now, as far as test dir reuse, I think you want an external server to have the following properties: [22:28] - totally decoupled from the test process [22:28] - but managed by it [22:29] so I don't think you want testcaseintempdir to have anything to do with the server [22:29] just have the server make its own temp dir, for its logs etc, and each test case can tell the server where the test case is rooted etc [22:29] well, often the tests that need a server have their own local branch for the client [22:29] which then chats to the server [22:29] so, it would have a server resource, and some other resources for the actual test [22:30] definitely a server resource, but unless you intend to share the other resources, they don't need to be exposed to testresources [22:30] I think it would be worth sharing the safe-contain-in-filesystem infrastructure [22:31] and can be just fixtures (which bzr already uses though without using the library) [22:31] the branch and trying to escape test isolation stuff [22:31] which is basically just some state on disk, that doesn't change unless something goes wrong (and could then be reset) [22:31] sure [22:32] fixtures does have the advantage of not needing the actual package [22:32] and bzr may not really need the fancy graph stuff [22:32] I think bzr should use the packages :) [22:32] the fixtures package has very useful helpers [22:32] the existing test arrangements may be enough without whole test suite reordering [22:32] as much as I like your graph code :) [22:32] and the testresources graph stuff will be needed if you want a single reused server for each of the server types [22:33] well, we really only need it shared 'enough' [22:33] bzr tests are currently totally isolated except for the safety mechanisms [22:33] wgz: I -really- wouldn't do 'enough' - do it right, don't NIH [22:33] starting a new one per module rather than per test method is still a win. [22:34] wgz: with the in-test servers, thats not at all clear when compared to an out of test server [22:34] wgz: starting in-test servers is pretty darn cheap [22:35] yeah, I feel very fearful of the NIH, but I'm also wondering if I'm going to need as much plumbing as doing something dumb like that would be anyway [22:38] I think I actually like the testresources api better (except for that ugly hack you have at the bottom to sniff out a TestResult), but if leaning towards fixtures I guess I need to work out how they could actually be used instead [22:39] ...I don't really like context managers with unittest, doesn't fit with the seperate methods for setup/teardown style [22:40] +you [22:40] in one of those malformed sentences somewhere [22:42] okay, thanks for your time lifeless, I'll try some things out and see where I get [22:44] wgz: so my point was you need two things, you should use both of them [22:45] testresources takes care of the ordering and external bringup-takedown aspects [22:45] fixtures takes care of easy code for encapsulated services [22:45] wgz: tearDown should be deleted, can't be for compat, but don't use it.