/srv/irclogs.ubuntu.com/2010/12/04/#bzr.txt

maxbsvz90: Easiest thing to do would be to configure it in your OpenSSH .ssh/config file00:02
svz90maxb: Thanks.00:04
patterni've just done a "bzr add foo", but before i committed the change i changed my mind and i no longer want to add "foo"05:32
patternhow can i tell bzr i don't want to add "foo" anymore?05:32
bob2rm07:31
bob2--keep or whatever07:31
renatosilvaenglish question: in a commit message like "reverted changeset xyz", reverted is past participle, not simple past, right? that is, changeset was reverted, not I reverted it.14:06
MethsAs a commit message it would make more sense to say "Reverts changeset xyz" as the message should say what that changeset does.  But in the above either works really, not sure which way it would be interpreted by a grammer perfectionist though.14:09
fullermdI'd go with "revert changeset" myself.  Imperative FTW.14:24
renatosilvacan it be read, is it usually read, as past participle?14:28
renatosilvais it "undone commit 123.", "commit 123 undone.", or "undid commit 123."?14:29
Meths3rd IMO14:37
renatosilvathere's one guy saying 1st could be read as "the commit has undone commit 123"14:51
renatosilvaand other guys saying "commit 123 undone" ("commit 123 has been undone") is ok, even though 3rd one is more usual14:52
fullermdvila: Around?15:35
vilafullermd: briefly, may I help ?16:01
fullermdI wondered if you've ever seen oddities of selftest on your FreeBSD vm with it locking itself up waiting on a semaphore.16:02
vilathis doesn't ring any bell, FreeBSD (the babune slave) is on of the most stable (if the not *the* most)16:03
vilafullermd: is it something you've seen with an explicit output or only a weird behavior16:03
vila?16:03
fullermdWell, for instance, http://babune.ladeuil.net:24842/job/selftest-freebsd/lastCompletedBuild/testReport/bzrlib.tests.blackbox.test_branch/TestSmartServerBranching/test_branch_from_trivial_branch_streaming_acceptance/?16:04
fullermdThat one.16:04
vilawhile being stable it's also quite slow (compared to the others), but I don't have a good explanation for this16:04
fullermdIf I run it via 'bzr selftest branch_streaming_acceptance', it runs fine.16:04
fullermdIf I just like 'bzr selftest' hit it, it just stops dead.  python had 'usem' as a wchan in ps/top.  I've let it sit for 4 or 5 minutes, and gets nowhere.16:05
vilahaaa interesting16:05
fullermd(runs in <3 seconds standalone)16:05
fullermdI've tried running a full selftest a half dozen times, and I keep finding new places further on that it locks down.16:05
fullermdExcept last try around, where I foudn another place _early_ on that it started locking.  Gruuh.16:05
vilaso, IIRC, that's a test which runs both a client and a server in the same process16:06
* fullermd nods.16:06
fullermdAll the ones that lock are of that ilk AFAICS.16:06
vilaI've been suspecting a weird interaction between python and BSDs (including OSX) but other OSes too (but quite differently) but I've never been able to diagnose precisely enough16:07
vilaweird interaction around sockets16:07
vilasome double handling of socket state (especially when both sides of the socket are used in the same process)16:08
vilabut that's only a theory so far... and pretty weak :-/16:08
fullermdMmm.  And you're using --parallel and somehow don't it it?16:09
fullermd(hit it)16:09
vilaI do use --parallel yes16:09
vilawell babune that is16:09
fullermdAnnoyingly it seems like sometimes it sneaks past it fine.  Weird timing stuff; always works when it's part of a short list of tests being run.16:09
vilaha, that rings a bell16:10
fullermdWould make it easy enough to work around by running a couple times, if it didn't happen 5 or 10 minutes into a freakin' test run...16:10
* fullermd is now trying `./bzr selftest -x branch_streaming_acceptance -x test_create_clone_on_transport_use_existing_dir -x RemoteBranch -x RemoteBzr`. Furrfu.16:10
vilamy suspicion began when there was a bunch of sockets waiting to die which seem to slow the overall run16:10
vilawhich led me to suspect some select() call16:11
vilaa bit like if python was relying on the OS to sort things out while.... somehow checking until it was happy with sleeps() intermixed... as you can see nothing very concrete16:12
* vila baktracks16:13
vilafullermd: you encounter hangs ?16:13
fullermdExcept for me, python ends up waiting on a semaphore of some sort, so it never checks again.16:13
fullermdWell...  halting problem.  It COULD just be being very very slow, for 4 or 5 minutes, using no CPU and never waking up.16:13
fullermdBut it sure smells like...  oh, look.16:13
* fullermd tries a-fscking-gain...16:14
* vila holds his breath...16:14
vilalook what ?16:14
vila:D16:14
fullermdOh, it locked itself again a thousand or so tests in.16:15
vilawhat's the flag... -Dthreads -Ethreads ?16:15
vila-Ethreads16:16
vilaIt was quite useful when debugging the leaks, but I think it was a but too intrusive and led to some failures (changing the output that some tests are checking), so be careful while encountering failures16:17
fullermdWell, we're 1600-some in to this attempt...16:18
fullermdAaaand, it does.16:18
vilaoh yes, lsof !16:19
vilalsof was also useful combined with -Ethreads whose output gives the socket references by tests16:20
vilaboth client and server side16:20
fullermd% fstat -p59868 | grep -c tcp16:20
fullermd21216:20
fullermdThat's a couple TCP sockets open...16:20
vilathat's bad16:20
vilaso, the other factor here is paramiko which leaves pending sockets16:21
fullermdNot a factor here.  I've had paramiko uninstalled for, like, a year and a half.16:21
vilabut they can't be easily collected as they are internal to pa...16:21
viladamn16:21
vilaone more theory dead16:21
* fullermd , slayer of theories!16:22
fullermdWhen you say -Ethreads, does that mean some sort of python internal threads, or does it use OS-level threads?16:24
vilapython threads16:25
vila-Ethreads outputs debug statements at various "interesting" points16:25
fullermdThere are 3 threads in the hung process.  One seems to be sitting in the 'accept' wchan, which I WAG may mean it's sitting in accept(2)...16:26
* vila hates vbox a bit more every day especially when killing one VM sometimes kill *all* the running VMs16:26
vilaWAG ?16:26
fullermdWild-Add Guess16:26
vila:)16:26
vilaone thread for the accept(), one thread for the serving, one thread for the client (usually the main one)16:27
vilaand by the way, the selftest in running with BZR_CONCURRENCY=4 and the VM has 2 processors configured16:29
vilas/in/is/16:29
fullermdWell, I'm not doing any --parallel'ing.16:32
vilaI got that, just mentioning16:32
* fullermd nods.16:32
vilaI'm running a straight selftest right now16:33
fullermdOh, hey, it locked up only 197 in this time.16:34
vilawith -Ethreads ?16:35
fullermdOnly 16 TCP sockets open this time.16:35
fullermdTrying that out now.16:35
vilathis... is surprising, I was pretty sure we collect almost all sockets now, except for the paramiko ones (which aren't relevant here)16:36
vilayou're running bzr.dev there right ?16:36
fullermdWell, locked up, but I guess I should use -v too so's to have some idea where it is...16:36
fullermdWell, technically, this is bzr.dev+someotherchanges.  But nothing that would mess with network.16:37
fullermdWell, it dumps a pile of information, but I don't see how it's much useful...16:38
vila2700 tests and running and fstat reports only 'internet stream tcp' lines, that's sockets right ?16:38
fullermdYah.16:38
fullermd'sockstat | grep python' should tell you what they are (well, unless you've got a lot of other stuff python running; then grep for the pid)16:38
vilait helps making sure the sockets are for past tests not the current ones and it also helps in finding which tests are leaking16:38
* fullermd frowns.16:40
fullermdHm.16:40
fullermdI'm trying t selftest --no-plugins now.  That seems to clean up after itself pretty fast; I'm in the 500's now and nothing hanging around...16:40
vilaright, ~4000 here and still no socket leaks (only 3 displayed)16:41
fullermdOh, well, that was a good theory while it lasted.  Locked up.16:41
vilaout of curiosity, what changes do you keep there ? Are you sure they aren't some gems worth proposing ? ;D16:42
fullermdWell, I WANT to propose them.  That's why I want selftest to run   :p16:42
fullermdHere's an interesting thing:16:42
fullermdUSER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS16:42
vilaha great :)16:42
fullermdfullermd python     60430 4  tcp4   127.0.0.1:39758       *:*16:42
fullermdfullermd python     60430 5  tcp4   127.0.0.1:39758       127.0.0.1:1648316:42
fullermdThat's presumably the socket of the current test.  One of the threads is waiting in accept.  The socket is listening.  But there's also a connection made to it.16:43
fullermdThat's on a test_pull_smart_stacked_[something]16:44
vilayes, could be the client and the accept() or the serving one, -Ethreads should help there no ?16:44
vilaalso, you should certainly see the mirrored one most of the time but if you happened to list while they are killed ?16:44
fullermdOK...16:45
fullermd...nching.test_branch_from_trivial_branch_streaming_acceptanceServer thread ('127.0.0.1', 31415) started16:45
fullermdClient thread ('127.0.0.1', 10740) -> ('127.0.0.1', 31415) started16:45
fullermdfullermd python     60607 4  tcp4   127.0.0.1:31415       *:*16:45
fullermdfullermd python     60607 5  tcp4   127.0.0.1:31415       127.0.0.1:1074016:45
fullermdfullermd python     60607 6  tcp4   127.0.0.1:10740       127.0.0.1:3141516:45
vila7000 and fstat is still clean as a baby, your 212 number above is still very weird16:46
fullermdOnly tcp sockets open for the process.16:46
fullermdSo it's got both sides of the connection open.  But one thread is still sitting in accept.16:46
vilasounds fine16:46
fullermdAnd it's dead there.16:47
vilathere is one thread waiting in the accept, spawning a thread for serving each connection and accept'ing again16:47
viladead and staying dead right now ?16:47
vilakeep it alive !!!!16:47
fullermdWell, it's been a couple minutes...16:48
vilaseriously,16:48
vilathis is the dirty bit when shutting down a test server16:48
vilawhat's the test name ?16:48
fullermd...nching.test_branch_from_trivial_branch_streaming_acceptance16:49
vilalet me see16:49
fullermdIn this case.  I don't think it matters though; it's semi-random.16:49
* fullermd just fired off a selftest in a virgin bzr.dev, just for kicks.16:49
vilacould be, but I want to look at what kind of test server is involved and show you the relevant code so you may have your own take on it16:50
vila(not asking to debug it, but look at the code in a context where it seem to be failing )16:50
fullermdMan, the relevant code is python.  My take will be "Hey, I can just rewrite that in perl..."   :p16:50
vilahehe, no I mean as socket code, whatever the language, we're almost doing C there :)16:51
fullermdRemember, any test it's halted on yet, it works just peachy if I run it alone.16:52
vilaok, so that's a TestCaseWithTransport so it should be an http server16:52
fullermdOr in a small group.  It's only when I run a big enough group (e.g., a full selftest) that it semi-arbitrarily picks a time to lock itself up.16:52
vilaI've seen very very very weird failures when chasing the leaks16:52
vilaexactly16:52
fullermdOh, there went virgin bzr.dev.16:52
fullermd| [1516/24890 in 3m27s, 1 failed] per_branch.test_branch.TestBranch.test_comm..16:52
vilabah, showing you the code is a bad idea, too many classes involved16:54
* fullermd has no class.16:54
vilaha, no, have a look at bt.test_server.TestingTCPServerInAThread.stop_server16:55
vilathe dirty bit I was referring to is defined there: the server is blocked in an accept call, so we 1) tell him to stop acception connections (after getting out of it's current call), 2) give him a dummy connection16:56
vilathere is also a if debug_threads():... call that you could copy/adapt/add in various points16:57
vilaI suspect the case you encounter should be in this area but exhibit an unexpected behavior16:57
vila14000 test and still clean16:58
vilahmm, IPV6 ?16:59
fullermdAll those connections are over 127.0.0.117:00
vilanaah, you're running py2.6 right and anyway, I'm pretty sure I've used a variation of the right python code17:00
vilawe've seen weird things even when using only ipv4 for us, as long as the host is configured for ipv6, (paramiko at least as a bug for this kind of config)17:00
vilamost of the time we force ipv4 by using 127.0.0.1 but some tests may still use localhost and went unnoticed17:01
vilawe have some fugitives like that here and there ;D17:02
fullermdJust switch it to force ipv6 by using ::1 instead.  If their system doesn't support v6 yet, set it in fire; better for everyone that way   :p17:02
vilahehe17:04
vila18000 and still clean, babune:~ :) $ fstat -p1004 |wc -l17:05
vila      1217:05
vilabut if you encounter the problem with bzr.dev and --no-plugins that should rule your proposal out :)17:06
vila20000 and raising briefly at ~40 (paramiko), back at 1217:08
fullermdIt doesn't seem to [generally] do any notable accumulation of stuff over time.  Most of the locks have no sockets except the current ones.17:08
vilabut when it blocks it's on a socket right ?17:09
vilaand one in the accept() state ? (with no foreign address ?)17:09
fullermdYah.17:10
vilaor are you unsure about that ?17:10
fullermdWell, I'm not _sure_ insofar as I've dug into the stack.  But that's what everything looks like.17:10
vilaso, I encounter weird things when trying the kill-the-socket-with-shutdown approach IIRC17:11
vilaI tried various ways with mixed results until I ended up giving it a dummy connection and even there I think I tried various tricks before settling on a simple close()17:12
vilathat's why I suggested adding some sys.stderr.write in this area but if you're blocked on a case where there is no foreign address...17:13
vilait kind of means that this last_conn didn't succeed ?17:13
vilaor did you even not reach reach the point where 'Server thread %s will be joined' ? Or is this message not flushed ? <shudder>17:14
fullermdI don't recall ever seeing joins on the locks.17:16
vilaoook17:16
fullermdSee the one I pasted ~half hour ago.  The "Client thread started" is the last thing that shows up.17:16
fullermdAs if the server thread sat in accept() waiting for the client thread (which the OS sees as connected), and never got out of accept().17:17
vilathat part is expected, the server thread has spawned another thread17:17
vilawhat isn't expected is that this spawned thread can't finish (or its related client thread)17:18
* fullermd sighs.17:18
vilathe client thread being the main thread17:18
fullermdIt sucks trying to figure out why changes were made when you can't ask the maker   :|17:18
vilaso, this kind of hang is the hardest17:18
vilawhich changes ?17:18
fullermdThe ones I'm working on cleaning up, which led me to trying selftest in the first place.17:19
vilawhich changes ? :D17:19
fullermdupgrade enhancements.17:20
vilaha :-/17:20
vilaerr, may I misinterpreted here, you mean in upgrade.py ?17:21
fullermdYah.17:21
vilaqblame mentions only living people, so you should be able to reach them no ?17:22
fullermdYes, that would be for the existing merged code, not the outstanding unmerged.17:22
fullermdWhich is igc   :(17:23
vilaha, I didn't misinterpreted :-/17:23
vilayou're digging an old mp ?17:23
fullermdhttps://code.launchpad.net/~bzr/bzr/smooth-upgrades/+merge/892117:23
* vila reading17:26
vilaby the way, the selftest succeeded here with only: bzrlib.tests.test_smart_transport.TestServerHooks.test_server_started_hook_memory is leaking threads among 2 leaking tests.17:27
vila1 non-main threads were left active in the end.17:27
vilahmm, lots of stuff there ;-/ Nice to see you working on it, I should leave now, but I'll be pp next week and will look at anything you'll submit, so feel free to begin with high level questions or even re-start the discussion on the ML17:32
vila--pack should be useless, --cleanup I think has been implemented elsewhere, so it's worth having a look at that while it still makes sense to propose it there (I went with a bunch of upgrades lately and could have used it)17:33
fullermdYou're thinking of pack --clean-obsolete-packs I think.17:36
jelmerfullermd: that's an interesting mp17:42
fullermdYeah, it should be a real nice addition to 2.0.0   :|17:44
jelmer:-P17:50
fullermd's one of the things whose continued absense pisses me off.  I don't really have time or inclination to work on it, but I have even less of both to keep not having it.  Sigh.17:56
fullermdvila: Just dropped what I could do on it.  Enjoy your piloting   :)19:23
nonix4if revision x in branch a adds file c, and revision y in branch b adds file d, where d is actually descendant of c... is there a simple way to tell that to bzr after both commits have been made? (both mentioned revisions containing other unrelated changes as well)19:26
fullermdI'm not sure what you mean by "descendant".  But the answer will be "no".  Once you've made a revision in the current World Order, it's too late to go back and tell bzr more about it.19:27
nonix4except for uncommit?19:30
fullermdWell, semantics.  Even uncommit doesn't change a revision, it just pops things off the top of the history and throws them away.19:31
fullermdThere is occasional discussion about ways to _annotate_ information after the fact (specifically to this case, things like equivalences between separate files).  But it's all speculative and far-future, which doesn't help anything now.19:34
nonix4well... how about cherry-picking creation of one file from a distant branch?19:47
fullermdYou could do that.  Being a cherry-pick, it wouldn't record any of the revision info.  But it would use the same file-id.19:47
mgzah, was going to ask about the 'local oddities' fullermd ran into with the test suite, but I see it's in the history so I'll just read up.21:38
mgzI'd be interested if lp:~gz/bzr/cleanup_testcases_by_collection_613247 helped though, I recall freebsd has som lower default resource limits than other nixes so leaks could hurt21:41
lifelesshi mgz21:45
lifelessmgz: if you're interested in testrepository, I would love to know if trunk still works on win3221:46
mgzhm, looks more like 's just theclient-server tests being dodgy still from the log21:49
mgzlifeless: I'll pull and test.21:49
lifelessthanks!21:51
lifelessmgz: its specifically 'testr run' I'm hacking on21:51
lifelessmgz: I've added a SIGPIPE fixup for unix21:52
lifelessbut that may bork win32 subprocess invocations21:52
mgzhm, some kind of import issue21:52
mgzI wish the framework didn't hide that stuff21:52
lifeless?21:52
mgz...  File "C:\Python24\lib\unittest.py", line 532, in loadTestsFromName21:52
mgz    parent, obj = obj, getattr(obj, part)21:52
mgzAttributeError: 'module' object has no attribute 'tests'21:52
mgznot useful.21:52
lifelessah yes21:52
lifelessI filed a bug on python21:52
lifelessand there is one on testtools too21:52
lifelessworkaround21:52
lifelesspython21:53
lifelessimport testrepository21:53
mgzyup.21:53
lifelessimport testrepository.tests21:53
lifeless...21:53
* lifeless wags on testrepository.ui.cli21:54
mgzwhat are the dependencies explictly?21:54
mgzI'm failing to find them documented anywhere.21:55
lifelesscat INSTALL.txt21:55
mgzdoh.21:55
mgzokay, I'll fix up my python path and go again.21:55
lifelessthe autotools have kindof devalued that file21:55
lifelesswhich is a bit sad21:55
mgz...22:01
mgz  File "C:\bzr\testscenarios\lib\testscenarios\scenarios.py", line 26, in ?22:01
mgz    from itertools import (22:01
mgzImportError: cannot import name product22:01
mgzwill fix that and see what else I hit.22:01
mgzso, your sigpipe fixup also wasn't valid 2.4 syntax, but that and a few other things look like easy fixes.22:18
mgz17 failures, a bunch look quoting related.22:18
lifelessheh22:19
mgzsome rename-over-existing-file ones.22:19
lifelessfeel like filing bugs?22:19
mgzonly if I can't fix 'em.22:19
lifelesscool22:19
lifelesswell I'm mid rewrite of run to do parallel test invocation22:20
mgzhm, so, could either fix the quoting in commands.run, or make it use a subprocess pipe rather than a shell pipe. you got any strong feelings there?22:24
lifelessthats one of the things I'm mid rewrite on22:25
mgzokay, will leave that one for the mo.22:25
mgza test with a tempdir that needs quoting in would be worth adding though.22:25
lifelessthe shell=True is for shell expansion22:25
lifelessmgz: please do22:26
fullermdmgz: Yeah, it's not a resource limit.  More something squirrel with the sockets.  Race or sumfin'.22:32
mgzwow python-dev has had a lot of long pointless threads of late.23:38
lifelessyeah23:39
lifelesswin23:39
mgzlifeless: I hate DocTestMatches, it's awful. Down to six failures though, four of which are quoting related.23:39
lifelessmgz: :(23:40
lifelessmgz: I'd welcome a more powerful full text matcher23:40
mgzfullermd: I'd be interested if you give that branch a run anyway, though it probably won't help. I'll do yours in turn to check you've not regresses anything.23:40
mgzlifeless:23:41
mgzFAIL: testrepository.tests.test_testr.TestExecuted.test_runs_and_returns_run_argv_no_args23:41
mgzAssertionError: Match failed. Matchee: "True True True23:41
mgz['b:\\temp\\tmprbtwer\\testr']23:41
mgz"23:41
mgzMatcher: DocTestMatches('True True True\n[...]\n', flags=8)23:41
mgzDifference: Expected:23:41
mgzI don't see why that's a mismatch, and the output sucks.23:42
mgzit possibly has an extra newline off the end? who knows.23:42
mgzthe test_bdist was even worse because it prints pages of text every time.23:43
mgzI don't think the Matcher scheme works well with more than one line of text, the top thingy needs to know about every leaf object to not spam you.23:45
lifelessmgz: it should have printed a difflike thing23:51
mgzbut the matchee is always printed anyway.23:53
mgz`assertThat(some_giant_thing, WhateverMatcher)` brings pain.23:54
lifelessmgz: that can be changed23:54
lifelessmgz: OTOH its good to be able to see definitional errors23:54
mgzit's good to be able to see more than the tail of the last failing test's output.23:55

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!