/srv/irclogs.ubuntu.com/2011/10/03/#bzr.txt

=== vila changed the topic of #bzr to: Bazaar version control <http://bazaar.canonical.com> | try https://answers.launchpad.net/bzr for more help | http://irclogs.ubuntu.com/ | Patch pilot: vila
vilahi all !06:13
jammorning all06:54
mgzmorning all07:52
Riddellhello08:16
mgzhey Riddell08:16
jamvila: https://code.launchpad.net/~jameinel/bzr/2.5-no-hanging-teardown/+merge/77870 should handle the hanging tests issue08:18
jamboth some fixes to the testing infrastructure to avoid future hangs, and fix the actual trigger08:18
jelmervila: did you see https://code.launchpad.net/~bryan-tongminh/bzr-webdav/ctype-fix/+merge/77824 ?09:45
vilajelmer: yup, on my TODO list09:51
jelmervila: great, just checkin' :)10:00
jelmerwe should really get some of those inactive projects out of the "bazaar" project on Launchpad10:00
vilayeah, this one have been rumored to be merged into core but the maintainer is slacking ;)10:01
jelmerhehe10:04
jelmerRiddell: just noticed, the changelog entry still says a "get-tar" command was added rather than get-orig-source11:33
jelmersorry for not catching that during review11:33
Riddelljelmer: oh aye, fixing11:34
=== medberry is now known as med_out
rom1hi all12:15
rom1I have a functional question about branches merges : is it possible to do a "merge revision" without merging the files, but selecting the OTHER code ?12:17
rom1Or in other words, is it possible to do a pull/push --overwrite with the creation of a revision that bundkles all the pulled/pushed revisions ?12:18
jelmerrom1: I'm not sure I follow entirely - is this to e.g. land a branch on trunk?12:21
rom1jelmer : not sure what land means... In fact, i have some developers requesting a workflow a la gitflow.12:22
rom1in this workflow, i can have hotfixes in the master12:22
jelmerrom1: Can you give a concrete example ?12:23
jelmerrom1: There is no one git work flow afaik :)12:23
rom1but to merge the develop revisions to the master, i want to forget about the hotfixes changes : either they have been merged into the develo branch, and no problem, either not, and it means that it has to be replaced by the develop branch12:23
rom1jelmer : i talk about this one : http://nvie.com/posts/a-successful-git-branching-model/12:24
jelmerrom1: I'm still not sure if I follow but I think you mean "bzr merge OTHER && bzr ci -m 'Merge other'."12:26
poolieo/ jelmer12:26
poolie(not really here, should go to bed)12:26
jelmerhey Martin12:26
jelmerAh, right.. labor day in .au ?12:27
jelmerOr I guess that would be labour day?12:27
nigelblol12:27
pooliein NSW12:27
jelmerah, that explains why wgrant and wallyworld were still around. I figured they just forgot ;-)12:28
jelmerhey mrevell, back on the interwebz?12:28
mrevellja!12:28
mrevell:)12:28
rom1jelmer : in fact, i do not want  to merge the files, i want to get exactly the code from OTHER (just like the ush/pull --overwrite) but in a single revision...12:29
nigelbjelmer: forgetting is entirely possible :)12:30
jelmerrom1: so you want to discard the changes in the current branch, but have history indicate they're present?12:30
jelmernigelb: you're talking to somebody who has once accidentally worked on a holiday :)12:31
nigelbjelmer: hahaha12:31
nigelbI guess that happens when you work from home :)12:31
rom1jelmer : well, discussing about that, i notice that it isn't very clear even for me.12:33
rom1:p12:33
jelmerrom1: is there a specific git command that does the same thing that you're looking for?12:34
rom1jelmer : when we create a hotfix on a production branch, it may be a dirty quick patch to quickly resolve the issue. When we release a new version containing a quick fix of the issue, i do not want to merge the dirty patch and the clean fix, but only take the clean one.12:35
jamrom1: bzr merge $OTHER; bzr revert -r -1:$OTHER; bzr commit -m "Merge and reset the tree state to $OTHER" ?12:35
jamOr just simply:12:36
rom1Yep ! i didn't think about revert !12:36
jambzr revert -r -1:$OTHER; bzr commit -m "Set the tree state to exactly OTHER but don't mark it as merged"12:36
jamI don't really think you want to set it to other without merging, but you could, if you really want to throw away all of OTHER's history.12:36
rom1jam : i understand. I haven't validated so far this workflow. I wanted first to see if it was feasible with bzr and our release management. I understand that a "merge without merge" is somehow surprising...12:39
rom1sorry, in my post to jelmer, i was meaning : "When we release a new version containing a CLEAN fix of the issue[...]"12:40
jamrom1: the issue is just what exactly you mean by "pull --overwrite" with only a single revision12:42
jamdoing a merge, and revert, will create a single new mainline revision12:43
jamhowever12:43
jamthis also assumes that you don't have any state in mainline that you want to keep12:43
jamspecifically, say that you emergency fix X, then you do a normal fix of Y, then you finally finish the real fix for X12:43
jamdoing the revert will throw away the updates to Y.12:43
jamWhich is why I would suggest just doing "bzr merge && bzr commit"12:44
jam*but* you know your process better than I do12:44
jamAssuming that the dev branch always supersedes the production branch sounds a bit risky, but if that is your process, you can stick to it.12:44
rom1jam : you're right, a temporary hotfix hasn't to be released in a production branch. Just branching it in a dead end branch, and keeping my proudtcion branch with merges only.12:47
rom1Thx jam and jelmer12:47
systemclientis bzr 0.18.0 usable with current repos at all?12:49
pooliesystemclient, it won't be able to read the default format created by recent bzr releases12:50
pooliepre-1.0 is pretty old12:50
systemclientpoolie: isn't pre 2.0  old already?12:51
poolieyeah, therefore 0.18 is really quite old12:54
poolie2.1 and later are still in support12:55
jampoolie: I know you aren't really here, but if maybe your ghost is around, I have an initial prototype up for https://bugs.launchpad.net/bzr/+bug/81960413:08
ubot5Ubuntu bug 819604 in Bazaar 2.1 "when an idle ssh transport is interrupted, bzrlib errors; should reconnect instead" [High,In progress]13:08
jamAnd it would be nice to get some feedback about where it is going.13:08
mgzokay, lunch before caches confuse me any more13:28
jelmerI never go caching before lunch either.13:29
jelmervila: thanks!13:53
jamvila: I replied to your review. I tried to run the babune jobs, but it just told me the servers are unavailable, and it looks like you deleted the requests. (Perhaps I entered them wrong?)14:28
vilaha, whic requests did you enter on which jobs ?14:29
jamvila: freebsd, natty, lucid14:29
jamselftest-subset-*14:29
vilajam: I' jusr recovering from a babune crash and *I* was typing text in a firefox, unlikely to crash...14:29
vilajam: for which tests ?14:30
jamvila: I was running all of them, since the failing tests tend to be randomly distributed. I know which ones to suspect if you don't want a full test run.14:30
vilaI'd prefer that, yes, especially if something is crashing14:31
vilajam: the freebsd slave is running a fsck, leave it some time to recover14:32
jamvila: So, after the changes, nothing should crash or hang :). The issue is that the test was hanging once it hits 4.0s of runtime. So you need a bit of load to slow the test suite down enough. Here they normally finish in about 2.5s14:35
jam(Which is why it seemed so random, a given test has to get some sort of hiccup and go over the 4.0s mark.)14:35
vilano test takes 4s to run on babune AFAIK14:35
jamvila: I'm pretty sure it did, though not consistently14:35
viladid you find reports to back that up ?14:37
jamvila: I can trigger the test suite hang by making the test take longer than 4.0s14:40
jamis that good enough for you?14:40
vilameh, of course not, I mean, this is certainly a bug but it doesn't mean it explain the ones we encountered on babune14:41
vilaa test taking 4 seconds is already a bug and we've never seen such huge variation without a good reason14:41
jamvila: aka, this diff: http://paste.ubuntu.com/701706/14:42
mgzload vila?14:42
jamvila: its already at 2.5s here on my reasonably fast laptop14:42
jamit isn't *that* far from 4.0s14:42
mgzhow careful are you to only be running one test suite on a box at once?14:42
vilamgz: I rely on jenkins for that (don't remember the details)14:43
jamvila: didn't you have to implement locks to reduce the load spike at midnight?14:45
jamI was pretty sure you schedule all jobs to run daily, and then you restrict it via some sort of inter-locking to something like 2 concurrent runs.14:45
jamalso, you are running --parallel, right?14:45
jamif you had per-test timing (which I really had hoped you would), we might have been able to see something like that14:45
jamI realize it doesn't get exposed via our junit xml adapter14:46
vilainter-locking is on slaves14:46
jamthough again, it may not strictly matter, since once a test hit 4.2s it would hang, and we wouldn't see it. But we could see that in the past, some test happened to spike higher than 4.0s.14:46
vilajam: look at the Test results, the timings are there for all tests (consolidated by prefix)14:46
jamvila: ah, just only for successful runs, right?14:47
vilayup, but it would very weird that a spike *never* occurred14:47
jamvila: http://babune.ladeuil.net:24842/job/selftest-chroot-lucid/lastSuccessfulBuild/testReport/bzrlib.tests.per_interrepository.test_fetch/TestInterRepository/14:48
jamhas a test that takes 2s14:48
jamvila: note that it only started failing if you have a 4.0s spike with the ConnectionTimeout patch14:48
jamso it certainly could have been spiking in the past, and just didn't cause a failure/hang14:48
jamvila: http://babune.ladeuil.net:24842/view/FreeBSD/job/selftest-freebsd/lastStableBuild/testReport/bzrlib.tests.per_interrepository.test_fetch/TestInterRepository/ has a test that takes 5.5s14:49
jamtest_fetch_from_stacked_smart_old(InterDifferingSerializer,RepositoryFormat2a,RepositoryFormatKnitPack6RichRoot) 5.5 secPassed14:49
jamtest_fetch_parent_inventories_at_stacking_boundary_smart(InterDifferingSerializer+get_known_graph_ancestry,RepositoryFormatKnitPack1,RepositoryFormatKnitPack6RichRoot)  took 5.4s14:50
vilacan you paste the precise URL instead of letting me find it in ~100 line pages ?14:50
jamand the ones after it are taking 6.x *14:50
jamvila: I was trying to show you the overview14:50
jamhttp://babune.ladeuil.net:24842/view/FreeBSD/job/selftest-freebsd/lastStableBuild/testReport/bzrlib.tests.per_interrepository.test_fetch/TestInterRepository/test_fetch_parent_inventories_at_stacking_boundary_smart_InterDifferingSerializer_RepositoryFormat2a_RepositoryFormatKnitPack6RichRoot_/?14:50
jamthere is a direct test14:50
jam'took 6.7s'14:50
jamwhich also explains why the freebsd was more likely to hang14:50
vilaha, ok14:54
vilabut why spuriously then ?14:54
jamvila: you have to have 4.0s of idle time on a given connection, and then get another connection after that 4.0s. So it isn't strictly a 'test takes >4.0s'.14:55
vilajam: if you look at this same test in the previous builds, it always above 4.0s14:55
jamSay you get the last connection at 3.9s, and then spend 2.7s working with the last connection.14:55
jamso, vila, even if I'm slightly wrong with my analysis (though I've spent the last 3 days on it), I'm sure that this change makes behavior more friendly, and fixes the "time.sleep(4.0)" problem.15:03
jamit certainly should change behavior so rather than hanging, we at least get a failure/exception when appropriate.15:03
vilajam: we can spend days discussing, if you know exactly how to trigger the hand, you should be able to make the test fail in a simple way to demonstrate it, you don't need to change all test servers for that leaving the doubt about whether you shake the code enough to make the hang go somewhere else15:04
jamanyway, EOD, here, I'll see if we can start talking about it earlier tomorrow.15:04
jamvila: I *did* change the tests to prove it15:04
vilaindeed, and focus on smart server only if that's where the issue is15:04
jamclient.read() was hanging15:04
vilabut you din't use this15:04
jamvila: the test didn't do it15:04
jamin fact, you had the test set a client timeout15:04
jam*because* it was hanging15:04
jamI was able to fix the test to just read and get a closed connection15:05
vilain test_test_server only, that's not used anywhere else !15:05
jamvila: test_server is the base implementation of SmartTCPServer_for_testing which is used in every test that calls make_smart_server()15:05
vilano  !15:05
vilatest_test_server not test_server15:05
vilayour change is in the former not the later15:06
jamvila: TestingTCPServerMixin is the class that needed updating as it was the part that implemented the code that SmartTCPServer_for_testing uses15:06
jamthe tests for that class are in "test_test_server"15:06
jamthere aren't any tests in "test_server"15:06
jamit is the implementation of the "TestServer"15:07
jamvila: if you go to "bzrlib.tests.__init__" you can see that we add "bzrlib.tests.test_test_server" but not "bzrlib.test_server" to the test suite.15:07
jamanyway, really, I need to go pick up my son. I'm not sure why you don't believe me15:08
vilathat's what I'm saying, I know how these files are named, I created them15:08
mgzokay, I think this has just made my morning worthwhile.15:08
mgz>>> om[3062558728]15:09
mgzstr(3062558728 4194212B 1par 'f\xe7_chknode:\n65536\n1\n1382\n\n\x00\x00sha1:6d13c15b49497a74b59b064e0f1bb074dd05b3be\n\x01\x00sha1:ce73daef8871866fd78')15:09
mgz>>> om[3043770376]15:09
mgzstr(3043770376 4194212B 1par 'f\xe7_chknode:\n65536\n1\n1382\n\n\x00\x00sha1:6d13c15b49497a74b59b064e0f1bb074dd05b3be\n\x01\x00sha1:ce73daef8871866fd78')15:09
jamvila: I fixed "test_server.py" to close connections on an exception, or when validate_request() returns false. I updated the tests in test_test_server to test those cases. If you just run the updated tests without the fixes, the test suite hangs.15:10
jamI'm not sure what else you want.15:10
jammgz: do you need some help understanding those?15:10
mgzof the 25 LRUSizeCache objects over repository packs, two have duplicates15:10
jamThat is a groupcompress record contain CHK nodes.15:10
jamIf you open a repository twice, you'll get duplicates15:11
vilajam: I want your fix to be specific to the smart test server not invading other servers15:11
jamif you open a source and a target, they both might have a copy15:11
jamyou can check the parents to see who is referencing them.15:11
mgzjam, as I understand that readout, there are two different GroupCompressBlock objects with the same content15:11
jamvila: I think the other servers are poorly behaved, because they will also cause clients to hang when the server gives up on talking to them15:11
jammgz: also certainly possible15:11
jamvila: the tests you have today actually were hanging, but you forced the client to use a socket.timeout to avoid it15:12
vilajam: you can't say that without evidence, I told you last friday this is don't happen *except* for the smart server which uses daemons threads15:12
jammgz: you can ping me tomorrow after standup and I'll poke around with you if you want.15:12
jamvila: I can15:12
vilajam: don't generalize from a single test specifically designed15:12
jamI have evidence15:12
jamvila: if you take out the socket.timeout ... the test hangs15:12
jamreliably15:12
jamthe comment is that the "server doesn't get cycles" is false15:13
jamit is because the server "doesn't close the connection" until teardown15:13
jamsorry "socket.settimeout"15:13
mgzjam, thanks. I'll plug on for a bit longer now and see where I get.15:13
jamvila: http://bazaar.launchpad.net/~bzr-pqm/bzr/bzr.dev/view/head:/bzrlib/tests/test_test_server.py#L16615:14
jambut for now, I'm gone15:14
jamsee you all tomorrow15:14
jamhave a good night15:14
mgzbye!15:14
abentleyjelmer: some time I'd like to get up to speed on co-located branches and their implications for pipelines.15:15
vilajam: the comment says "whether our reads or writes may hang" this test *requires* a timeout15:15
jelmerabentley: the colocated branch format hasn't landed yet, so it might still be a bit too early. As far as I can tell pipelines just use the regular bzr APIs, in which case I think15:17
jelmerpipelines will just work out of the box with colocated branches.15:17
abentleyjelmer: pipelines can create and use bzr-colo-style branches using "reconfigure-pipeline".15:20
jelmerabentley: colocated branches in core are different from bzr-colo15:21
abentleyjelmer: Once colocated branches in core are usable, I think reconfigure-pipeline should switch to them.  And "add-pipe" will need to support them too, I imagine.15:22
mgzlooks very hopeful for some duplicate elimination: <http://pastebin.ubuntu.com/701737/>15:24
jelmerabentley: It should be able to support both, if you're ok with having the extra code to do so.15:24
abentleyjelmer: I'm okay with that.15:24
=== deryck is now known as deryck[lunch]
jelmervila: hi, still there?15:56
vilajelmer: oh sorry, yes16:26
=== deryck[lunch] is now known as deryck
AuroraBorealishiya mgz/wgz whatever you are today16:41
AuroraBorealisi dunno how hard it is to fix whatever was going wrong with the meliae dumps but if one could figure that out then maybe we could fianlly get somewhere xD16:45
mgzafter you went to sleep I succeeded in loading the dump by getting meliae to ignore ids that are not present,16:54
mgzwhich there's a TODO over but I'm not sure of the neatest way of doing16:54
mgz...and was on the other box16:54
AuroraBorealislol16:55
mgzbut if you add something similar the dump will at least load16:55
AuroraBorealisremember what file i should look in?16:56
mgzI've also been looking at where memory is used this morning, so have a generally better idea of what I'm doing16:56
mgzsec, nearly done for the day here, will transfer down below16:56
AuroraBorealisah ok16:56
wgzokay.17:01
wgzworkaround: <http://paste.ubuntu.com/701810/>17:02
AuroraBorealisthat works17:02
wgzcan still fall over later, but gets the thing loaded17:02
wgzso, can you also do `om.summarize()` now? if so, we can progress.17:04
AuroraBorealisi shall work on that now17:06
AuroraBorealisfinally got i t17:23
AuroraBorealisit17:23
AuroraBorealisit wasn't repacking during this, but doing the fast-import again (2 gigs of memory)17:23
AuroraBorealishttp://paste.ubuntu.com/701828/17:24
wgzomg.17:24
wgzwell, that's only finding 440MB usage at that point17:26
wgzbut 120MB in frozenset is pretty crazy17:26
wgzdo `om.get_all("frozenset")[1]`17:27
wgz`om.get_all("frozenset")[0]` even.17:27
AuroraBorealiseven though the memory usage was 2 gb in the process the dump file was only 1 gb17:28
AuroraBorealiswhich was weird17:28
wgzit's possible fast-import has 1.5GB of unfindable allocations17:28
AuroraBorealisfrozenset(37820232 2272B 31refs 1par)17:28
AuroraBorealisand frozenset(1340577832 736B 15refs 1par)17:29
=== beuno is now known as beuno-lunch
AuroraBorealisfor [0] and [1]17:29
wgzhm, that's not big, it's just lots of teeny ones adding up to pain then.17:30
wgzuse _.c to see what's in one.17:30
AuroraBorealisafter the get_all()[1] call?17:31
wgzin the python terminal _ just refers to the last object17:31
AuroraBorealisoh17:31
wgzyou can bind one to a name instead of you want17:31
AuroraBorealisgetting "address not present" again17:31
wgztry some other indexes, see if they're all missing contents17:32
wgzmight be were some of the extra mem usage is to be found17:32
AuroraBorealis[0] worked17:32
AuroraBorealishttp://paste.ubuntu.com/701837/17:33
AuroraBorealisjust look like strings o.o17:33
wgzheh, yeah, [0] is likely not typical17:33
wgztry some other numbers nearer the middle17:33
AuroraBorealis2-7 all return KeyError >.>17:34
wgzthere are 562876 frozenset objects, so pick some bigger indexes17:35
AuroraBorealisoh17:35
AuroraBorealislol17:35
wgzif lots of them have the same problem, it's likely there's our meliae bug to fix17:36
AuroraBorealisyeah17:36
AuroraBorealis200,000 does it, 300,000, 400,00017:36
AuroraBorealisall keyerror17:36
wgztry in the other direction, use .p rather than .c17:37
wgzand find out what's holding on to them17:37
wgzkeep going up with _.p[0].p ..etc as needed17:38
wgz(.c is 'children' - the list of object this object references, and .p is 'parents' - the list of objects that reference this object)17:39
AuroraBorealishttp://paste.ubuntu.com/701839/17:39
AuroraBorealistried some numbers17:39
AuroraBorealisoh17:39
AuroraBorealisseems to be the same dictionary tho17:40
wgzyeah, dict is too generic, go up again till you find a class or something more signpost-y17:40
wgzit's all the same dict at least17:40
wgzhm... actually, I think I may know where in the code this is17:40
AuroraBorealis[bzrlib._known_graph_pyx.KnownGraph(170691824 72B 2refs 1par)]17:41
AuroraBorealisthey all say that17:41
wgzokay, so that's the entire history in memory.17:45
AuroraBorealisthat seems bad17:45
wgzwell, fastest way provided it fits, probably17:45
AuroraBorealisfast importing the linux kernel does seem like an extreme case17:46
wgzAuroraBorealis: get that KnownGraph object and look at its children17:50
wgzand also maybe summarize it (with `om.summarize(kg)`)17:51
AuroraBorealisi think this is right17:52
AuroraBorealishttp://paste.ubuntu.com/701847/17:52
wgzthink there was one to many .c to get kg, but gives the right idea17:55
AuroraBorealisyeah without the .c it still shows the same thing17:56
AuroraBorealisactually i lied17:57
wgzwe've lost the giant dictionary with the frozenset objects somehow in that output17:57
AuroraBorealishttp://paste.ubuntu.com/701848/17:57
AuroraBorealisthat?17:57
wgzthere he is.17:58
wgzso, that's big, but not huge. however, we seem to have no content for any of those containers, which is apparently where the dump went wrong17:59
AuroraBorealisis that something wrong with meliae?17:59
AuroraBorealisthat it didn't dump everything17:59
wgzyup, I'm guessing, will try and repo so it can be fixed.18:01
AuroraBorealisok18:03
AuroraBorealisi should be around18:03
AuroraBorealiswell i have school, and i am probably going to be at school late cause some company is coming and i want to sit in on their talk18:03
AuroraBorealisyou can email me with stuff to do at markgrandi@gmail.com though :D18:03
=== beuno-lunch is now known as beuno
=== med_out is now known as med
=== med is now known as medberry
=== medberry is now known as med_out
pooliejam, hi, i see your mail23:12
pooliehi all23:13
jelmer_hi poolie23:22

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!