[00:09] <poolie> hi all
[00:17] <mathrick> hiya poolie
[00:17] <mathrick> poolie: <mathrick> re: bug #838469, should improvements to mini-tutorial be submitted for released versions as well, or will they only be accepted for bzr.dev?
[00:19] <poolie> hi mathrick
[00:20] <poolie> we can take doc updates for released versions
[00:22] <mathrick> poolie: great, should I then make a patch against the 2.4 braching point, or should I rebase it, or should it rather be cherry-picked for release branches?
[00:23] <poolie> just do a revision based off lp:bzr/2.4 tip
[00:23] <mathrick> OK
[06:19] <vila> hi all ! Happy Friday !
[06:20] <AuroraBorealis> yay
[06:54] <jam> morning all
[07:28] <jbicha> hi, is there already a bug open for bzr-builddeb's lack of support for .tar.xz? that's going to be especially annoying since GNOME 3.3 won't ship other tarballs
[07:35] <mgz> morning all
[07:58] <poolie> jam, hi - you're feisty today! :)
[08:25] <jam> mgz: I have some ideas about bug #869366, are you looking at it, or just reporting it for posterity?
[08:25] <ubot5`> Launchpad bug 869366 in Bazaar "TestTCPServerInAThread.test_server_crash_while_responding random failure" [Medium,Confirmed] https://launchpad.net/bugs/869366
[08:25] <mgz> just reporting, as I ran into it EOD yesterday
[08:25] <mgz> I seem to be getting lucky with random failures on PQM
[08:29] <mgz> looking at the test, it seems possible for the subthread to not do any futher work after the sync
[08:32] <vila> mgz: yes, that's the race I was mentioning the other day, the same one exists for the test just above, it triggers very rarely and it took quite a while to put my finger on it
[08:33] <vila> mgz: the main thread really want to wait for both the connection thread to raise the exception *and* to the server thread to re-raise it
[08:33] <mgz> I'm not even sure what tools python gives us to resolve such problems
[08:34] <mgz> I guess in this case try:/raise/finally:/sync2 would work?
[08:35] <mgz> because the exception won't propogate until the test looks for it?
[08:35] <vila> mgz: the test looks for the exception to be swallowed :)
[08:35] <jam> mgz: for that particular test, you could call 'server.stop_server()" instead of "server.pending_exception()"
[08:35] <vila> mgz:  the previous test (test_server_crash_while_responding) looks for the exception
[08:35] <jam> stop can be called twice
[08:35] <jam> and that way it will have joined its threads
[08:36] <vila> which is not what the test is about: test_exception_swallowed_while_serving
[08:37] <vila> this a test about the server not about the client
[08:39] <vila> we don't want the server to shut down, we want it to keep serving (swallowed_while_serving)
[08:39] <jam> vila: that is the other test
[08:39] <jam> "test_server_crash_while_responding" wants it to raise the exception
[08:39] <jam> we can just change pending_exception to stop_server and it will sync with the clients
[08:39] <jam> client threads
[08:40] <jam> maybe connection threads is a better term
[08:40] <jam> though I guess that doesn't make the tests symmetric except for the exception being ignored.
[08:42] <vila> with 'that' == test_server_crash_while_responding, this test is about crashing the server, not shutting it down
[08:42] <vila> pending_exception is what is used to implement shutting down
[08:43] <vila> which illustrate why I disagreed with your modifications which were about client behavior and as such replacing these tests instead of adding your owns
[08:44] <vila> bug mgz encounters the issue in 2.4 so your modifications are out of scope here
[08:45] <vila> mgz: by the way, thanks for filing that bug, I thought I did as this failure has been seen on babune on several occasions (rare enough to not trigger a fix)
[08:45] <mgz> hm yes, I might not have said that in big enough letters in the bug, but this was a landing on the 2.4 branch
[08:46] <vila> mgz: ... and *never* triggered on pqm AFAIK... before you claimed the unlucky badge :)
[08:49] <mgz> two different random failures in about four landings on 2.4 PQM is pretty good going
[08:49] <vila> mgz: yup, don't let them go ! :)
[08:50] <mgz> oh hey, looking at babune, lucid is red
[08:50] <mgz> and guess what the failing test is :)
[08:50] <vila> mgz: but can we assume the cause is the same now that the test is not ?
[08:51] <mgz> ...one thing going for testtools is seeing both tracebacks makes this miles clearer
[08:51] <mgz> vila: if there's still just the one sync before the raise, then yes I think
[08:54] <vila> mgz: I was kidding ;) test_server_fails_while_serving_or_stopping shouldn't suffer from the same race because it explicitly call stop_server I think...
[08:54] <poolie> hi mgz, vila
[08:54] <vila> helli pooloe
[08:54] <vila> meh
[08:54] <vila> hello poolie ;)
[08:55] <mgz> hellipolis?
[09:02] <vila> mgz: so the ' Wait for the exception to propagate' is a lie, it waits for the exception to be 'about to be raised' and just doing 'sync.wait()' is not enough to guarantee that. That's where the race is. A possible fix will be to connect with another client and get served properly (i.e. yes, the server is still up and running)
[09:02] <vila> mgz: trying to be more precise would require hooking somewhere the server really swallows
[09:12] <vila> mgz: hehe, but that's in fact required as this test server cannot server properly (by test design :)
[09:37] <AuroraBorealis> is there a way to force bzr to use its built in ssh rather then another ssh that is on the path?
[09:38] <AuroraBorealis> (installed git, and of course it has an ssh.exe that causes bzr to just hang for some reason)
[09:39] <jam> AuroraBorealis: "export BZR_SSH=paramiko"
[09:39] <AuroraBorealis> where do i put that if i'm on windows?
[09:39] <jam> if you're on Windows it is spelled
[09:39] <jam> set BZR_SSH=paramiko
[09:39] <jam> but you can set it in global env vars
[09:39] <jam> right click My Computer, Properties, Advanced System Settings, Environment Variables
[09:40] <AuroraBorealis> ah kk
[09:40] <AuroraBorealis> user or system variable, does it matter?
[09:42] <jam> AuroraBorealis: doesn't matter for you
[09:42] <jam> depends if you have more than one user on the machine
[09:42] <AuroraBorealis> but python getenv can access both ?
[09:43] <AuroraBorealis> (guess just a python question now =P )
[09:45] <mgz> bug 641330 is the shelve issue
[09:45] <ubot5`> Launchpad bug 641330 in Bazaar "unable to unshelve shelved root-id change" [Medium,Confirmed] https://launchpad.net/bugs/641330
[09:45] <mgz> bug 82555 looks like the root cause
[09:45] <ubot5`> Launchpad bug 82555 in Bazaar "Merging to an empty branch doesn't work" [High,Confirmed] https://launchpad.net/bugs/82555
[09:46] <mgz> bug 242175 is the mitigation for that Riddell added for the merge case
[09:46] <ubot5`> Launchpad bug 242175 in Bazaar "ValueError: WorkingTree.set_root_id with fileid=None when merging into empty branch" [High,Fix released] https://launchpad.net/bugs/242175
[09:48] <mgz> AuroraBorealis: windows merges them before starting a process as your user
[09:48] <AuroraBorealis> k
[10:15] <AfC> Do bzr-builddeb's `import-upstream` and bzrtools's `import` use the same change detection / file-id (ie, file checksum, perhaps) algorithms?
[10:16] <AfC> (anyone know?)
[10:16] <AuroraBorealis> not me :>
[10:17] <jelmer> AfC: hi
[10:17] <jelmer> AfC: change against what?
[10:17] <AfC> jelmer: hi
[10:18] <AfC> jelmer: so, in both cases, there's considerable work done to detect (for example) renames, right?
[10:18] <AfC> [so I infer]
[10:18] <jelmer> AfC: no, no renames are inferred in either case as far as I know
[10:18] <AfC> oh
[10:19] <jelmer> I could be wrong, I don't have much experience with either
[10:19] <jelmer> actually, "bzr import" could help
[10:19] <jelmer> since it doesn't commit
[10:19] <jelmer> so you can do "bzr import ..; bzr mv --auto; bzr ci -m 'import x'"
[10:19] <AfC> jelmer: ok, well, it's doing *something* for the minutes of CPU time its sitting there burning away
[10:19] <AfC> mv --auto, yeah, that might be interesting
[10:20] <jelmer> AfC: which one is burning CPU? import or import-upstream?
[10:20] <AfC> [both]
[10:20] <AfC> I just now did an import; import-upstream was taking ~9 minutes each.
[10:20] <jelmer> AfC: what are you importing?
[10:20] <AfC> :)
[10:21] <AfC> that large well known project prominently using someone else's version control system that I've been blundering along with this p
[10:21] <AfC> past week
[10:22] <AuroraBorealis> if its git, we are already seeing that fast-import or whatever is taking huge amounts of time / memory.
[10:23] <AfC> AuroraBorealis: so I'm not working with bzr-git in this case; I've instead used bzr-builddeb's facilities against the upstream project's release tarballs.
[10:23] <AuroraBorealis> ah
[10:23] <jelmer> AfC: I don't think either import or import-upstream have been optimized for kernel-sized trees..
[10:24] <jelmer> AfC: any particular reason you're not using the launchpad imports, or are you just seeing how far you can get?
[10:24] <AfC> jelmer: that's ok, I'm not bitching about import performance per se
[10:24] <AfC> jelmer: so, having established the deterministic behaviour, I tried the lp imports branch
[10:25] <AfC> jelmer: but I kept getting swap thrash. I'm sure I'll be able to make use of it at some point, though
[10:26] <jelmer> AfC: you're getting swapping trash even just running "bzr branch lp:linux linux" ?
[10:28] <AfC> yeah
[10:28] <AfC> [anyway, that's done, I let it run overnight or whatever]
[10:28] <jelmer> AfC: what version of bzr are you running?
[10:28] <AfC> jelmer: daily
[10:28] <AuroraBorealis> is lp:linux already a bazaar repo?
[10:28] <AfC> yes
[10:29] <AfC> AuroraBorealis: ^
[10:30] <AuroraBorealis> weird that its thrashing when its just branching.
[10:30] <AfC> it just took a long time; it was quite a few days ago; I've had bigger problems since then doing other stuff, but as I said, it's here now should I need it.
[10:31] <AfC> bzr tags
[10:31] <AuroraBorealis> might be related to the uhh
[10:31] <AuroraBorealis> bug i'm having when importing the git linux repo
[10:32] <AuroraBorealis> where its using so much memory (therefore thrashing if you don't have enough ram) cause its keeping the entire history in memory
[10:35] <jelmer> AuroraBorealis: where are you getting that? using bzr-git, bzr-fastimport, regular clone?
[10:36] <jelmer> none of them should really be  keeping all history in RAM
[10:37] <AuroraBorealis> https://bugs.launchpad.net/bugs/864217
[10:37] <ubot5`> Ubuntu bug 864217 in Bazaar "MemoryError when repacking repository with large numbers of revisions " [Medium,Confirmed]
[10:37] <AuroraBorealis> doing fast-import of the git linux kernel uses about 2 gigs of memory , and it wasn't even done
[10:37] <AfC> jelmer: well it is attempting to keeping 3.0 GB of something in memory. This doesn't go down well :/
[10:39] <AuroraBorealis> and mgz said something about it was keeping history in memory
[10:39] <AfC> jelmer: (this was even with bzr branch -r 1 URL; I never could get it to finish with -r > 100)
[10:40] <AfC> (I imagine looping one revision at a time over n days would have worked out, but having established it was deterministic and didn't have missing revision relative to the lp import, I knew I could trust that)
[10:46] <jelmer> AfC: did you see the bug I filed earlier about -r1 not actually doing that?
[10:48] <mgz> jelmer: from a (partial) memdump AuroraBorealis had from fast-import, there was a _KnownGraph taking up a bg chunk of memory
[10:48] <mgz> that may well be a bug, there's a _known_heads cache dict that seems essentially unbounded
[10:49] <mgz> it only ever gets cleared on a couple of error paths
[10:49] <AuroraBorealis> have you worked on meliae not actually dumping all the objects?
[10:49] <AuroraBorealis> so we can see more of whats in the dumps?
[10:49] <mgz> nope, but I've got that down as a project for this weekend
[10:50] <AuroraBorealis> kk
[10:51] <AuroraBorealis> i should be around, although you seem to be on the other side of the world from me so hours will be interesting =P
[10:52] <mgz> on Monday I saw a reproducable segfault, some type errors, and the issue with missing objects causing the dump to not load
[10:53] <mgz> fixing all those should get us somewhere, but the incomplete dump may still be a win64 size/alignment issue which will be a pain to work out from here
[10:53] <AuroraBorealis> oh gee ;<
[10:53] <AuroraBorealis> do you have access to a win64 computer?
[10:53] <mgz> no, that's the fun bit :)
[10:54] <mgz> the code has the wrong size of all the pointer->int casts though, so there's some hope I can just fix things till it magically works
[10:57] <AuroraBorealis> i'll mail you my crappy 500 dollar laptop with windows 7 64 bit on it thats collecting dust behind me
[11:00] <AfC> jelmer: no, I didn't; I'll look
[11:00] <mgz> where on the other side of the world are you AuroraBorealis? :)
[11:00] <AuroraBorealis> arizona, usa
[11:01] <AfC> I would have thought you would have to go a bit further north than that to see the aurora borealis. Unless we just got spanked by a giant gamma ray burst...
[11:02] <fullermd> Plate tectonics.  Just takes a little longer.
[11:02] <AfC> ah, yes, of course :)
[11:03] <AuroraBorealis> yes
[11:03] <fullermd> ('course, it does make you wonder what kinda nutcase would be in AZ, awake, and on IRC at this hour   :p)
[11:03] <AfC> [I only ever saw it once. I was rather far up, as I recall. Was pretty]
[11:03] <AuroraBorealis> the nutcase who was procostinating on his small prolog assignment
[11:03]  * mgz thinks fullermd isn't one to talk
[11:03] <fullermd> Saying 'prolog' isn't helping your case!
[11:04] <AuroraBorealis> haha
[11:04] <fullermd> Hey, I'm a solid hour ahead of him.  That makes me totally sane.
[11:07] <AuroraBorealis> haha
[11:11] <AuroraBorealis> anyway, bed time for me. school in a few hours
[11:11] <AuroraBorealis> chat with you this weekend i guess mgz
[16:05] <jml> james_w: would you mind merging my branch?
[16:10] <james_w> sure thing
[16:28] <vila> jml, james_w : sry about that, I planned to do it but get interrupted longer than expected
[16:29] <jml> vila: np
[19:22] <sponge-> is there a way to do a sparse/bare checkout in bzr? in svn i'd do --depth immediates and then svn up --set-depth infinity on the folders i wanted, looking for something similar
[20:19] <elmo> when merging someone else's branch, should I commit, then do fix ups, or can I merge, fix + commit result and still be able to usefully differieniate the two sets of changes?
[20:36] <lifeless> elmo: difftastic/difftacular - I never remember the name - can untangle them for you
[20:37] <lifeless> so sure, do whatever ;)
[21:45] <wgz> lifeless: can I pester you for your thoughts on bug 866107 at some point?
[21:48] <lifeless> sure, not right now sadly
[21:48] <wgz> I shall ping again over the weekend at some point.
[21:53] <wgz> and in other news, I have once again made PQM fail in some weird way
[21:53] <wgz> this time... all the tests pass but...
[21:54] <wgz> star-merge succeeded at Fri Oct  7 16:26:04 2011 (0:24:02.142211)
[21:54] <wgz> 'Exception processing merge: [Errno 2] No such file or directory'
[21:55] <vila> O.o
[21:56] <wgz> stderr has the slightly suspicious:
[21:56] <wgz> Fri Oct 7 16:02:43 UTC 2011 : selftest starts
[21:56] <wgz> failed to open trace file: [Errno 13] Permission denied: '/you-should-use-TestCaseInTempDir-if-you-need-a-log-file'
[21:56] <wgz> Fri Oct 7 16:26:01 UTC 2011 : selftest ends
[21:56] <wgz> but there's no mention of a test with that issue in the test suite output
[21:58] <hazmat> is there a cli command to get the latest revid out of a repo... ie. analogue of revno?
[21:58] <vila> red herring, it's a leak somewhere, has been there for ages
[21:58] <vila> wgz: ^
[21:58] <wgz> okay, so it's the merge failing somehow that's the issue
[21:58] <wgz> but... after all the tests run, which is just weird
[21:59] <vila> wgz: more precisely some TestCase should be a TestCaseInTemp... err, yeah, that's what the path says ;)
[21:59] <wgz> hazmat: `bzr revision-info`?
[21:59] <vila> wgz: could be related to the issue with the lack of mails to bazaar-commits
[22:00] <wgz> I'll post to the mp so MvG knows I'm not just holding out on him.
[22:03] <lifeless> wgz: anyhow, testresources is good; I plan to supercede its model with testfixtures once I or someone else implements a fixture graph with reasonable introspection semantics (such that you can do the testresources planning on unsetup fixtures)
[22:12] <wgz> lifeless: do you have any particular opinions about what bzr needs on this front?
[22:12] <lifeless> wgz: I'd write your stuff as fixtures and use a shallow fixture->resource-manager shim
[22:13] <lifeless> as I don't expect you to need deep route planning, merely aggregation of all the tests using a given external server
[22:14] <wgz> one part of your fixtures readme really confused me
[22:15] <wgz> resources had a clear part about dependent resources
[22:15] <hazmat> wgz, perfect thanks.. didn't show up on a bzr help commands | grep rev
[22:15] <wgz> whereas fixtures had a long example about how you couldn't share a tempdir if there were also a db and a webserver
[22:16] <wgz> which... I didn't get at all
[22:17] <lifeless> fixtures is a simpler model
[22:18] <wgz> but that seems simpler to the point of not being useful
[22:18] <lifeless> fixtures has seen more use in less time
[22:18] <lifeless> this is one of the paradoxes of design :)
[22:18] <wgz> if I can't share the tempdir created by TestCaseInTempDir (with some added carefulness in reset), I can't build any other fixtures of use on top of it
[22:19] <lifeless> of course you can, if you want to
[22:19] <wgz> nearly everything thats expensive enough to want to share between tests first derives from TestCaseInTempDir
[22:20] <wgz> does testresources basically get around this by running all webserver tests together, then all db tests, and if it can't do that tearing down the lot between?
[22:22] <lifeless> well back up a step
[22:22] <lifeless> I think you have some conflation
[22:22] <wgz> this is likely.
[22:22] <lifeless> TestCaseInTempDir using unique temp dirs for each test case
[22:22] <wgz> it shares a ROOT_DIR
[22:22] <wgz> this is a bad thing, because it's global state
[22:22] <lifeless> yes
[22:22] <lifeless> its something we were working to nuke
[22:23] <wgz> this is something I'd like to nuke with one of your nice packages :)
[22:23] <lifeless> anyhow, fixtures describes an extended context manager basically
[22:23] <lifeless> which happens to be very useful for tests because it exposes things like getDetails
[22:23] <lifeless> and a convenient addCleanup for the code within it.
[22:24] <lifeless> testresources describes an optimisation framework for expensive things; each of which could be a fixture (but needs a graph which fixtures doesn't expose)
[22:25] <lifeless> the fixtures README talks about using just fixtures in similar situations to what testresources is targeted at - and in that world its kludgy, because of the lack of an introspectable graph
[22:25] <lifeless> (you can't tell that the webserver and db are going to be using the same temp dir, and you can
[22:26] <lifeless> 't (if they use addFixture) cleanup just the webserver or just the db, because that will implicitly cleanup the tempdir, hurting the other user
[22:26] <lifeless> so
[22:26] <lifeless> this is why I was saying to use testresources, but write the resources as a manager shimming across to a fixture
[22:26] <lifeless> now, as far as test dir reuse, I think you want an external server to have the following properties:
[22:28] <lifeless>  - totally decoupled from the test process
[22:28] <lifeless>  - but managed by it
[22:29] <lifeless> so I don't think you want testcaseintempdir to have anything to do with the server
[22:29] <lifeless> just have the server make its own temp dir, for its logs etc, and each test case can tell the server where the test case is rooted etc
[22:29] <wgz> well, often the tests that need a server have their own local branch for the client
[22:29] <wgz> which then chats to the server
[22:29] <wgz> so, it would have a server resource, and some other resources for the actual test
[22:30] <lifeless> definitely a server resource, but unless you intend to share the other resources, they don't need to be exposed to testresources
[22:30] <wgz> I think it would be worth sharing the safe-contain-in-filesystem infrastructure
[22:31] <lifeless> and can be just fixtures (which bzr already uses though without using the library)
[22:31] <wgz> the branch and trying to escape test isolation stuff
[22:31] <wgz> which is basically just some state on disk, that doesn't change unless something goes wrong (and could then be reset)
[22:31] <lifeless> sure
[22:32] <wgz> fixtures does have the advantage of not needing the actual package
[22:32] <wgz> and bzr may not really need the fancy graph stuff
[22:32] <lifeless> I think bzr should use the packages :)
[22:32] <lifeless> the fixtures package has very useful helpers
[22:32] <wgz> the existing test arrangements may be enough without whole test suite reordering
[22:32] <wgz> as much as I like your graph code :)
[22:32] <lifeless> and the testresources graph stuff will be needed if you want a single reused server for each of the server types
[22:33] <wgz> well, we really only need it shared 'enough'
[22:33] <lifeless> bzr tests are currently totally isolated except for the safety mechanisms
[22:33] <lifeless> wgz: I -really- wouldn't do 'enough' - do it right, don't NIH
[22:33] <wgz> starting a new one per module rather than per test method is still a win.
[22:34] <lifeless> wgz: with the in-test servers, thats not at all clear when compared to an out of test server
[22:34] <lifeless> wgz: starting in-test servers is pretty darn cheap
[22:35] <wgz> yeah, I feel very fearful of the NIH, but I'm also wondering if I'm going to need as much plumbing as doing something dumb like that would be anyway
[22:38] <wgz> I think I actually like the testresources api better (except for that ugly hack you have at the bottom to sniff out a TestResult), but if leaning towards fixtures I guess I need to work out how they could actually be used instead
[22:39] <wgz> ...I don't really like context managers with unittest, doesn't fit with the seperate methods for setup/teardown style
[22:40] <wgz> +you
[22:40] <wgz> in one of those malformed sentences somewhere
[22:42] <wgz> okay, thanks for your time lifeless, I'll try some things out and see where I get
[22:44] <lifeless> wgz: so my point was you need two things, you should use both of them
[22:45] <lifeless> testresources takes care of the ordering and external bringup-takedown aspects
[22:45] <lifeless> fixtures takes care of easy code for encapsulated services
[22:45] <lifeless> wgz: tearDown should be deleted, can't be for compat, but don't use it.