/srv/irclogs.ubuntu.com/2011/10/07/#bzr.txt

pooliehi all00:09
mathrickhiya poolie00:17
mathrickpoolie: <mathrick> re: bug #838469, should improvements to mini-tutorial be submitted for released versions as well, or will they only be accepted for bzr.dev?00:17
ubot5Launchpad bug 838469 in Bazaar "Mini tutorial doesn't explain repository directories" [Low,Confirmed] https://launchpad.net/bugs/83846900:17
pooliehi mathrick00:19
pooliewe can take doc updates for released versions00:20
mathrickpoolie: great, should I then make a patch against the 2.4 braching point, or should I rebase it, or should it rather be cherry-picked for release branches?00:22
pooliejust do a revision based off lp:bzr/2.4 tip00:23
mathrickOK00:23
vilahi all ! Happy Friday !06:19
AuroraBorealisyay06:20
jammorning all06:54
=== vila is now known as vila_at_dentist
jbichahi, is there already a bug open for bzr-builddeb's lack of support for .tar.xz? that's going to be especially annoying since GNOME 3.3 won't ship other tarballs07:28
mgzmorning all07:35
pooliejam, hi - you're feisty today! :)07:58
=== vila_at_dentist is now known as vila
jammgz: I have some ideas about bug #869366, are you looking at it, or just reporting it for posterity?08:25
ubot5`Launchpad bug 869366 in Bazaar "TestTCPServerInAThread.test_server_crash_while_responding random failure" [Medium,Confirmed] https://launchpad.net/bugs/86936608:25
mgzjust reporting, as I ran into it EOD yesterday08:25
mgzI seem to be getting lucky with random failures on PQM08:25
mgzlooking at the test, it seems possible for the subthread to not do any futher work after the sync08:29
vilamgz: yes, that's the race I was mentioning the other day, the same one exists for the test just above, it triggers very rarely and it took quite a while to put my finger on it08:32
vilamgz: the main thread really want to wait for both the connection thread to raise the exception *and* to the server thread to re-raise it08:33
mgzI'm not even sure what tools python gives us to resolve such problems08:33
mgzI guess in this case try:/raise/finally:/sync2 would work?08:34
mgzbecause the exception won't propogate until the test looks for it?08:35
vilamgz: the test looks for the exception to be swallowed :)08:35
jammgz: for that particular test, you could call 'server.stop_server()" instead of "server.pending_exception()"08:35
vilamgz:  the previous test (test_server_crash_while_responding) looks for the exception08:35
jamstop can be called twice08:35
jamand that way it will have joined its threads08:35
vilawhich is not what the test is about: test_exception_swallowed_while_serving08:36
vilathis a test about the server not about the client08:37
vilawe don't want the server to shut down, we want it to keep serving (swallowed_while_serving)08:39
jamvila: that is the other test08:39
jam"test_server_crash_while_responding" wants it to raise the exception08:39
jamwe can just change pending_exception to stop_server and it will sync with the clients08:39
jamclient threads08:39
jammaybe connection threads is a better term08:40
jamthough I guess that doesn't make the tests symmetric except for the exception being ignored.08:40
vilawith 'that' == test_server_crash_while_responding, this test is about crashing the server, not shutting it down08:42
vilapending_exception is what is used to implement shutting down08:42
vilawhich illustrate why I disagreed with your modifications which were about client behavior and as such replacing these tests instead of adding your owns08:43
vilabug mgz encounters the issue in 2.4 so your modifications are out of scope here08:44
vilamgz: by the way, thanks for filing that bug, I thought I did as this failure has been seen on babune on several occasions (rare enough to not trigger a fix)08:45
mgzhm yes, I might not have said that in big enough letters in the bug, but this was a landing on the 2.4 branch08:45
vilamgz: ... and *never* triggered on pqm AFAIK... before you claimed the unlucky badge :)08:46
mgztwo different random failures in about four landings on 2.4 PQM is pretty good going08:49
vilamgz: yup, don't let them go ! :)08:49
mgzoh hey, looking at babune, lucid is red08:50
mgzand guess what the failing test is :)08:50
vilamgz: but can we assume the cause is the same now that the test is not ?08:50
mgz...one thing going for testtools is seeing both tracebacks makes this miles clearer08:51
mgzvila: if there's still just the one sync before the raise, then yes I think08:51
vilamgz: I was kidding ;) test_server_fails_while_serving_or_stopping shouldn't suffer from the same race because it explicitly call stop_server I think...08:54
pooliehi mgz, vila08:54
vilahelli pooloe08:54
vilameh08:54
vilahello poolie ;)08:54
mgzhellipolis?08:55
vilamgz: so the ' Wait for the exception to propagate' is a lie, it waits for the exception to be 'about to be raised' and just doing 'sync.wait()' is not enough to guarantee that. That's where the race is. A possible fix will be to connect with another client and get served properly (i.e. yes, the server is still up and running)09:02
vilamgz: trying to be more precise would require hooking somewhere the server really swallows09:02
vilamgz: hehe, but that's in fact required as this test server cannot server properly (by test design :)09:12
AuroraBorealisis there a way to force bzr to use its built in ssh rather then another ssh that is on the path?09:37
AuroraBorealis(installed git, and of course it has an ssh.exe that causes bzr to just hang for some reason)09:38
jamAuroraBorealis: "export BZR_SSH=paramiko"09:39
AuroraBorealiswhere do i put that if i'm on windows?09:39
jamif you're on Windows it is spelled09:39
jamset BZR_SSH=paramiko09:39
jambut you can set it in global env vars09:39
jamright click My Computer, Properties, Advanced System Settings, Environment Variables09:39
AuroraBorealisah kk09:40
AuroraBorealisuser or system variable, does it matter?09:40
jamAuroraBorealis: doesn't matter for you09:42
jamdepends if you have more than one user on the machine09:42
AuroraBorealisbut python getenv can access both ?09:42
AuroraBorealis(guess just a python question now =P )09:43
mgzbug 641330 is the shelve issue09:45
ubot5`Launchpad bug 641330 in Bazaar "unable to unshelve shelved root-id change" [Medium,Confirmed] https://launchpad.net/bugs/64133009:45
mgzbug 82555 looks like the root cause09:45
ubot5`Launchpad bug 82555 in Bazaar "Merging to an empty branch doesn't work" [High,Confirmed] https://launchpad.net/bugs/8255509:45
mgzbug 242175 is the mitigation for that Riddell added for the merge case09:46
ubot5`Launchpad bug 242175 in Bazaar "ValueError: WorkingTree.set_root_id with fileid=None when merging into empty branch" [High,Fix released] https://launchpad.net/bugs/24217509:46
mgzAuroraBorealis: windows merges them before starting a process as your user09:48
AuroraBorealisk09:48
AfCDo bzr-builddeb's `import-upstream` and bzrtools's `import` use the same change detection / file-id (ie, file checksum, perhaps) algorithms?10:15
AfC(anyone know?)10:16
AuroraBorealisnot me :>10:16
jelmerAfC: hi10:17
jelmerAfC: change against what?10:17
AfCjelmer: hi10:17
AfCjelmer: so, in both cases, there's considerable work done to detect (for example) renames, right?10:18
AfC[so I infer]10:18
jelmerAfC: no, no renames are inferred in either case as far as I know10:18
AfCoh10:18
jelmerI could be wrong, I don't have much experience with either10:19
jelmeractually, "bzr import" could help10:19
jelmersince it doesn't commit10:19
jelmerso you can do "bzr import ..; bzr mv --auto; bzr ci -m 'import x'"10:19
AfCjelmer: ok, well, it's doing *something* for the minutes of CPU time its sitting there burning away10:19
AfCmv --auto, yeah, that might be interesting10:19
jelmerAfC: which one is burning CPU? import or import-upstream?10:20
AfC[both]10:20
AfCI just now did an import; import-upstream was taking ~9 minutes each.10:20
jelmerAfC: what are you importing?10:20
AfC:)10:20
AfCthat large well known project prominently using someone else's version control system that I've been blundering along with this p10:21
AfCpast week10:21
AuroraBorealisif its git, we are already seeing that fast-import or whatever is taking huge amounts of time / memory.10:22
AfCAuroraBorealis: so I'm not working with bzr-git in this case; I've instead used bzr-builddeb's facilities against the upstream project's release tarballs.10:23
AuroraBorealisah10:23
jelmerAfC: I don't think either import or import-upstream have been optimized for kernel-sized trees..10:23
jelmerAfC: any particular reason you're not using the launchpad imports, or are you just seeing how far you can get?10:24
AfCjelmer: that's ok, I'm not bitching about import performance per se10:24
AfCjelmer: so, having established the deterministic behaviour, I tried the lp imports branch10:24
AfCjelmer: but I kept getting swap thrash. I'm sure I'll be able to make use of it at some point, though10:25
jelmerAfC: you're getting swapping trash even just running "bzr branch lp:linux linux" ?10:26
AfCyeah10:28
AfC[anyway, that's done, I let it run overnight or whatever]10:28
jelmerAfC: what version of bzr are you running?10:28
AfCjelmer: daily10:28
AuroraBorealisis lp:linux already a bazaar repo?10:28
AfCyes10:28
AfCAuroraBorealis: ^10:29
AuroraBorealisweird that its thrashing when its just branching.10:30
AfCit just took a long time; it was quite a few days ago; I've had bigger problems since then doing other stuff, but as I said, it's here now should I need it.10:30
AfCbzr tags10:31
AuroraBorealismight be related to the uhh10:31
AuroraBorealisbug i'm having when importing the git linux repo10:31
AuroraBorealiswhere its using so much memory (therefore thrashing if you don't have enough ram) cause its keeping the entire history in memory10:32
jelmerAuroraBorealis: where are you getting that? using bzr-git, bzr-fastimport, regular clone?10:35
jelmernone of them should really be  keeping all history in RAM10:36
AuroraBorealishttps://bugs.launchpad.net/bugs/86421710:37
ubot5`Ubuntu bug 864217 in Bazaar "MemoryError when repacking repository with large numbers of revisions " [Medium,Confirmed]10:37
AuroraBorealisdoing fast-import of the git linux kernel uses about 2 gigs of memory , and it wasn't even done10:37
AfCjelmer: well it is attempting to keeping 3.0 GB of something in memory. This doesn't go down well :/10:37
AuroraBorealisand mgz said something about it was keeping history in memory10:39
AfCjelmer: (this was even with bzr branch -r 1 URL; I never could get it to finish with -r > 100)10:39
AfC(I imagine looping one revision at a time over n days would have worked out, but having established it was deterministic and didn't have missing revision relative to the lp import, I knew I could trust that)10:40
jelmerAfC: did you see the bug I filed earlier about -r1 not actually doing that?10:46
mgzjelmer: from a (partial) memdump AuroraBorealis had from fast-import, there was a _KnownGraph taking up a bg chunk of memory10:48
mgzthat may well be a bug, there's a _known_heads cache dict that seems essentially unbounded10:48
mgzit only ever gets cleared on a couple of error paths10:49
AuroraBorealishave you worked on meliae not actually dumping all the objects?10:49
AuroraBorealisso we can see more of whats in the dumps?10:49
mgznope, but I've got that down as a project for this weekend10:49
AuroraBorealiskk10:50
AuroraBorealisi should be around, although you seem to be on the other side of the world from me so hours will be interesting =P10:51
mgzon Monday I saw a reproducable segfault, some type errors, and the issue with missing objects causing the dump to not load10:52
mgzfixing all those should get us somewhere, but the incomplete dump may still be a win64 size/alignment issue which will be a pain to work out from here10:53
AuroraBorealisoh gee ;<10:53
AuroraBorealisdo you have access to a win64 computer?10:53
mgzno, that's the fun bit :)10:53
mgzthe code has the wrong size of all the pointer->int casts though, so there's some hope I can just fix things till it magically works10:54
AuroraBorealisi'll mail you my crappy 500 dollar laptop with windows 7 64 bit on it thats collecting dust behind me10:57
=== Vorpal_ is now known as Vorpal
AfCjelmer: no, I didn't; I'll look11:00
mgzwhere on the other side of the world are you AuroraBorealis? :)11:00
AuroraBorealisarizona, usa11:00
AfCI would have thought you would have to go a bit further north than that to see the aurora borealis. Unless we just got spanked by a giant gamma ray burst...11:01
fullermdPlate tectonics.  Just takes a little longer.11:02
AfCah, yes, of course :)11:02
AuroraBorealisyes11:03
fullermd('course, it does make you wonder what kinda nutcase would be in AZ, awake, and on IRC at this hour   :p)11:03
AfC[I only ever saw it once. I was rather far up, as I recall. Was pretty]11:03
AuroraBorealisthe nutcase who was procostinating on his small prolog assignment11:03
* mgz thinks fullermd isn't one to talk11:03
fullermdSaying 'prolog' isn't helping your case!11:03
AuroraBorealishaha11:04
fullermdHey, I'm a solid hour ahead of him.  That makes me totally sane.11:04
AuroraBorealishaha11:07
AuroraBorealisanyway, bed time for me. school in a few hours11:11
AuroraBorealischat with you this weekend i guess mgz11:11
=== medberry is now known as med_out
=== beuno is now known as beuno-lunch
jmljames_w: would you mind merging my branch?16:05
james_wsure thing16:10
vilajml, james_w : sry about that, I planned to do it but get interrupted longer than expected16:28
jmlvila: np16:29
=== med_out is now known as medberry
=== beuno-lunch is now known as beuno
=== deryck is now known as deryck[lunch]
=== cody-somerville_ is now known as cody-somerville
=== yofel_ is now known as yofel
=== deryck[lunch] is now known as deryck
sponge-is there a way to do a sparse/bare checkout in bzr? in svn i'd do --depth immediates and then svn up --set-depth infinity on the folders i wanted, looking for something similar19:22
elmowhen merging someone else's branch, should I commit, then do fix ups, or can I merge, fix + commit result and still be able to usefully differieniate the two sets of changes?20:19
lifelesselmo: difftastic/difftacular - I never remember the name - can untangle them for you20:36
lifelessso sure, do whatever ;)20:37
wgzlifeless: can I pester you for your thoughts on bug 866107 at some point?21:45
ubot5Launchpad bug 866107 in Bazaar "Use testresources for bzr selftest" [Medium,Confirmed] https://launchpad.net/bugs/86610721:45
lifelesssure, not right now sadly21:48
wgzI shall ping again over the weekend at some point.21:48
wgzand in other news, I have once again made PQM fail in some weird way21:53
wgzthis time... all the tests pass but...21:53
wgzstar-merge succeeded at Fri Oct  7 16:26:04 2011 (0:24:02.142211)21:54
wgz'Exception processing merge: [Errno 2] No such file or directory'21:54
vilaO.o21:55
wgzstderr has the slightly suspicious:21:56
wgzFri Oct 7 16:02:43 UTC 2011 : selftest starts21:56
wgzfailed to open trace file: [Errno 13] Permission denied: '/you-should-use-TestCaseInTempDir-if-you-need-a-log-file'21:56
wgzFri Oct 7 16:26:01 UTC 2011 : selftest ends21:56
wgzbut there's no mention of a test with that issue in the test suite output21:56
hazmatis there a cli command to get the latest revid out of a repo... ie. analogue of revno?21:58
vilared herring, it's a leak somewhere, has been there for ages21:58
vilawgz: ^21:58
wgzokay, so it's the merge failing somehow that's the issue21:58
wgzbut... after all the tests run, which is just weird21:58
vilawgz: more precisely some TestCase should be a TestCaseInTemp... err, yeah, that's what the path says ;)21:59
wgzhazmat: `bzr revision-info`?21:59
vilawgz: could be related to the issue with the lack of mails to bazaar-commits21:59
wgzI'll post to the mp so MvG knows I'm not just holding out on him.22:00
lifelesswgz: anyhow, testresources is good; I plan to supercede its model with testfixtures once I or someone else implements a fixture graph with reasonable introspection semantics (such that you can do the testresources planning on unsetup fixtures)22:03
wgzlifeless: do you have any particular opinions about what bzr needs on this front?22:12
lifelesswgz: I'd write your stuff as fixtures and use a shallow fixture->resource-manager shim22:12
lifelessas I don't expect you to need deep route planning, merely aggregation of all the tests using a given external server22:13
wgzone part of your fixtures readme really confused me22:14
wgzresources had a clear part about dependent resources22:15
hazmatwgz, perfect thanks.. didn't show up on a bzr help commands | grep rev22:15
wgzwhereas fixtures had a long example about how you couldn't share a tempdir if there were also a db and a webserver22:15
wgzwhich... I didn't get at all22:16
lifelessfixtures is a simpler model22:17
wgzbut that seems simpler to the point of not being useful22:18
lifelessfixtures has seen more use in less time22:18
lifelessthis is one of the paradoxes of design :)22:18
wgzif I can't share the tempdir created by TestCaseInTempDir (with some added carefulness in reset), I can't build any other fixtures of use on top of it22:18
lifelessof course you can, if you want to22:19
wgznearly everything thats expensive enough to want to share between tests first derives from TestCaseInTempDir22:19
wgzdoes testresources basically get around this by running all webserver tests together, then all db tests, and if it can't do that tearing down the lot between?22:20
lifelesswell back up a step22:22
lifelessI think you have some conflation22:22
wgzthis is likely.22:22
lifelessTestCaseInTempDir using unique temp dirs for each test case22:22
wgzit shares a ROOT_DIR22:22
wgzthis is a bad thing, because it's global state22:22
lifelessyes22:22
lifelessits something we were working to nuke22:22
wgzthis is something I'd like to nuke with one of your nice packages :)22:23
lifelessanyhow, fixtures describes an extended context manager basically22:23
lifelesswhich happens to be very useful for tests because it exposes things like getDetails22:23
lifelessand a convenient addCleanup for the code within it.22:23
lifelesstestresources describes an optimisation framework for expensive things; each of which could be a fixture (but needs a graph which fixtures doesn't expose)22:24
lifelessthe fixtures README talks about using just fixtures in similar situations to what testresources is targeted at - and in that world its kludgy, because of the lack of an introspectable graph22:25
lifeless(you can't tell that the webserver and db are going to be using the same temp dir, and you can22:25
lifeless't (if they use addFixture) cleanup just the webserver or just the db, because that will implicitly cleanup the tempdir, hurting the other user22:26
lifelessso22:26
lifelessthis is why I was saying to use testresources, but write the resources as a manager shimming across to a fixture22:26
lifelessnow, as far as test dir reuse, I think you want an external server to have the following properties:22:26
lifeless - totally decoupled from the test process22:28
lifeless - but managed by it22:28
lifelessso I don't think you want testcaseintempdir to have anything to do with the server22:29
lifelessjust have the server make its own temp dir, for its logs etc, and each test case can tell the server where the test case is rooted etc22:29
wgzwell, often the tests that need a server have their own local branch for the client22:29
wgzwhich then chats to the server22:29
wgzso, it would have a server resource, and some other resources for the actual test22:29
lifelessdefinitely a server resource, but unless you intend to share the other resources, they don't need to be exposed to testresources22:30
wgzI think it would be worth sharing the safe-contain-in-filesystem infrastructure22:30
lifelessand can be just fixtures (which bzr already uses though without using the library)22:31
wgzthe branch and trying to escape test isolation stuff22:31
wgzwhich is basically just some state on disk, that doesn't change unless something goes wrong (and could then be reset)22:31
lifelesssure22:31
wgzfixtures does have the advantage of not needing the actual package22:32
wgzand bzr may not really need the fancy graph stuff22:32
lifelessI think bzr should use the packages :)22:32
lifelessthe fixtures package has very useful helpers22:32
wgzthe existing test arrangements may be enough without whole test suite reordering22:32
wgzas much as I like your graph code :)22:32
lifelessand the testresources graph stuff will be needed if you want a single reused server for each of the server types22:32
wgzwell, we really only need it shared 'enough'22:33
lifelessbzr tests are currently totally isolated except for the safety mechanisms22:33
lifelesswgz: I -really- wouldn't do 'enough' - do it right, don't NIH22:33
wgzstarting a new one per module rather than per test method is still a win.22:33
lifelesswgz: with the in-test servers, thats not at all clear when compared to an out of test server22:34
lifelesswgz: starting in-test servers is pretty darn cheap22:34
wgzyeah, I feel very fearful of the NIH, but I'm also wondering if I'm going to need as much plumbing as doing something dumb like that would be anyway22:35
wgzI think I actually like the testresources api better (except for that ugly hack you have at the bottom to sniff out a TestResult), but if leaning towards fixtures I guess I need to work out how they could actually be used instead22:38
wgz...I don't really like context managers with unittest, doesn't fit with the seperate methods for setup/teardown style22:39
wgz+you22:40
wgzin one of those malformed sentences somewhere22:40
wgzokay, thanks for your time lifeless, I'll try some things out and see where I get22:42
lifelesswgz: so my point was you need two things, you should use both of them22:44
lifelesstestresources takes care of the ordering and external bringup-takedown aspects22:45
lifelessfixtures takes care of easy code for encapsulated services22:45
lifelesswgz: tearDown should be deleted, can't be for compat, but don't use it.22:45

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!