/srv/irclogs.ubuntu.com/2010/05/22/#bzr.txt

LorWhat would be the most efficient way to make a gazillion pulls from a particular host, with the assumption that most of those pulls will do nothing (the local branch will be up-to-date)?00:05
LorWould it make sense to open multiple transports in parallel?00:06
exarkunIf I want my shared repo to be more general, can I just move the .bzr directory up a directory level or two?00:09
* exarkun tries it to see what happens anyway00:10
LorI think it ought to work.00:14
exarkunHey cool it's working I guess00:28
Kamping_Kaiserhow can bzr use 33013KB to repack 850kb of text? ;s01:07
Kamping_Kaiser[################|   ]  33319KB    48KB/s | Fetching revisions:Inserting stream:repacking texts:texts 9184/1636001:07
* Kamping_Kaiser kills it01:07
=== joerg is now known as Guest53356
exarkunHow did I make this happen?  And how do I fix it?01:51
exarkunbzr: ERROR: Repository KnitPackRepository('file:///var/lib/buildbot/.bzr/repository/') is not compatible with repository SvnRepository('svn://svn.twistedmatrix.com/svn/Twisted')01:51
hnoHmm.. shouldn't bzr automatically figure out if my local shared repository already have a copy of a referenced remote branch?07:28
hnodid a bzr diff -r ancestor:. --new lp:...   and the shared repository of the current branch also includes a copy of the lp branch.07:30
hnoand yet it fetches some 10MB of data from the lp branch.07:32
=== wgrant_ is now known as wgrant
spivhno: it should, yes08:18
spivhno: mind adding -Dhpss to that command line and filing a bug?08:18
spivhno: (attach your ~/.bzr.log)08:19
hnospiv: Will do.09:28
hnoHPSS calls: 193 (188 vfs) SmartSSHClientMedium(bzr+ssh://henriknordstrom@bazaar.launchpad.net/)09:31
hnohi lifeless09:31
hnoHmm.. I must be stupid but how do I file a bug report. Can only figure out how to search for them...09:36
lifelesshi hno09:36
lifelesson ?09:36
hnobzr09:37
lifelessbugs.launchpad.net/bzr09:37
lifelesstop right 'report a bug'09:37
hnoah, the big red one...09:37
lifelessvila: haihai09:58
vilalifeless: hey ;-)10:05
lifelessI wish lp apis were easier to test with.10:11
lifelessdo we have a supported mocking tool ?10:12
vilanot that I know of :-(10:12
vilaI think I read people using staging for that lately10:12
vilayes, I know you asked for a mockup10:13
lifelesso/~ shave that yak10:19
lifelessyakkity yak10:20
hnobug filed.10:35
hnowonder why I always seem to run into these cases of excess bandwidth usage by bzr.. can't imagine my use of bzr is that strange compared to what most others are doing.10:37
lifelesswe have two critical bugs open at the moment about just that10:38
lifelessmaybe you filed a dupe10:38
hnobzr diff remotebranch fails to find that the remote revisions is already in the local repository.10:39
hnohttps://bugs.launchpad.net/bzr/+bug/58414810:40
ubot5Launchpad bug 584148 in Bazaar "bzr diff remotebranch makes poor use of revisions on local repository (affected: 1, heat: 0)" [Undecided,New]10:40
lifelessah10:40
lifelessthat isn't what we've got open; and it is a little unusal10:41
lifelesswhile it works - and thats important - most folk mirror things locally to play with them10:41
hnolifeless: I don't think the case based on bzr switch is that unusual...10:43
lifelesshno: thats orthogonal really10:43
hno?10:44
hnoto the bug yes.10:44
lifelessI use switch based workflow myself10:45
lifelessfor most everything10:45
hnoso it will bite you as well.10:45
hnoexcept that you diff by using merge...10:46
exarkunUh so15:06
exarkunhttp://buildbot.twistedmatrix.com/builders/hardy32-py2.5-glib2/builds/24/steps/bzr/logs/stdio15:06
exarkunAnybody?15:30
exarkunI see there's a ticket filed about this.15:35
exarkunAnyone know of any docs about bzr-svn plus tdb?15:40
exarkunThe last comment on https://bugs.launchpad.net/null/+bug/185200 suggests that just installing python-tdb will address the issue, but I've had it installed since I started setting this up, and it doesn't seem to have been used.15:41
ubot5Launchpad bug 185200 in Bazaar Subversion Plugin ""database is locked" bzr internal error (affected: 2, heat: 0)" [High,Fix released]15:41
jmljelmer_, hello?16:15
jelmer_jml: hey16:16
jmljelmer_, I was wondering if I could tempt you into helping out exarkun16:17
* jelmer_ looks at the backlog16:17
jelmer_exarkun: hi16:18
exarkunHeya16:18
jelmer_exarkun: you may have to remove the existing cache files since they will be in sqlite16:18
exarkunIn ~/.bazaar/svn-cache/ ?16:18
jelmer_exarkun: yep, or ~/.cache/bazaar/svn16:18
exarkunOkay, I'll try that.16:19
jelmer_we should be able to leave this whole thing behind us when the cache format becomes pack-file-based16:20
exarkunThe new cache file is a "SQLite 3.x database"16:20
jelmer_exarkun: what version of bzr-svn and python-tdb do you have?16:21
exarkunBazaar (bzr) 2.1.116:22
jelmer_exarkun: and bzr-svn and python-tdb?16:22
exarkunUh, "svn 1.0.2"?16:23
exarkunAnd python-tdb 1.1.1~svn26294-1 (ie, hardy package)16:23
jelmer_exarkun: the hardy package of python-tdb is probably too old16:24
jelmer_anyway, gtg - back in about an hour16:24
exarkunI suppose I'll fix this some other way then16:29
strkis 'bzr clone' different from 'bzr branch' ?17:03
exarkunno17:04
exarkunSee the "Aliases" section of "bzr branch --help"17:04
exarkunUhm17:13
exarkunHow about this?17:14
exarkunbzr: ERROR: Pack '51d53eb11548019c7208e76646979ac4' already exists in <bzrlib.repofmt.groupcompress_repo.GCRepositoryPackCollection object at 0x1cddc50>17:14
exarkunlifeless: I think next time someone asks if concurrent access is safe, you should say no.17:14
jelmer_exarkun: that's a known bug in older versions of bzr iirc17:38
jelmer_exarkun: it doesn't have anything to do with bzr-svn at least17:38
marioxcc`hi all18:30
marioxcc`i have a question:18:30
marioxcc`how are the branch in bazaar relative to branch in git?18:30
marioxcc`¿do bazaar and git use the same model of heads or what are the diferences?18:31
exarkunjelmer_: Seems maybe to have gone away after upgrading from 2.0 to 2.1, thanks18:35
ricqlesHi all !19:40
exarkunCan I have hooks run when 'bzr svn-update' updates a bzr branch from an svn repository?19:46
mkanatexarkun: post_branch_tip_change would work, wouldn't it?19:48
=== Guest53356 is now known as joerg
exarkunmkanat: I have no idea, I've never heard of that thing before19:49
exarkunGoogle returns no hits for it19:49
mkanatexarkun: "bzr hooks" will get you a list of hooks.19:49
exarkunNeat19:49
exarkunOops19:49
exarkunI get a traceback19:49
mkanatexarkun: It's pre_change_branch_tip -- I tyoped it slightly. :-)19:50
mkanatexarkun: Or post_change_branch_tip19:50
effstopshey guys, has anyone had trouble with the OS X installer?19:50
effstopsas it's installing sip I get the error "The Installer could not install some files in "/".19:50
exarkunmkanat: Thanks.19:51
exarkunHm, I guess that's not really my top priority now though20:03
exarkunI still need to figure out how to fix this SQLite3 "database is locked" issue.20:03
exarkunI bet I can set $HOME to get bzr-svn to use a different svn-cache20:03
exarkunI wonder if there's a better way, though (hopefully a way that won't accidentally affect a ton of other stuff too)20:04
exarkunDoes 'bzr checkout svn://...' make a lot of connections to the svn server?20:16
marioxcc`exarkun: use netstart to check20:17
exarkunnetstart?20:17
mkanatexarkun: netstat20:19
exarkunAh.  That's kind of tricky, since I'd have to be sure to run it in a super fast loop and then filter the results to see how many connections it made.  I was hoping someone would just know.20:20
jelmer_exarkun: one or two20:20
exarkunI can use strace, though.20:20
exarkunjelmer_: Aha, thanks.20:20
exarkunI'm now seeing my inetd decide to disable svnserve because too many connections are coming in20:20
jelmer_exarkun: it will reuse connections it opens, and it fetches revisions serially20:20
exarkunbut all I've done is switch things from 'svn co ...' to 'bzr co ...'20:20
exarkunI suppose if it was already near the limit and sometimes bzr uses two, that might have pushed it over.20:21
exarkunjelmer_: Any hints on a good way to get my different... actors... to use different svn-cache directories, aside from setting $HOME differently for each of them?20:22
jelmer_exarkun: why would you want to?20:25
exarkunjelmer_: Because I can't find a python-tdb package for Hardy that's new enough for bzr-svn and so most of my 'bzr checkout' commands are failing with SQLite3 errors20:27
exarkunIf you have other ideas about how to fix that, I'm open to suggestions20:28
jelmer_exarkun: You could avoid using the cache altogether but that will impact performance20:29
exarkunYea... that's like an extra 30 - 45 seconds per checkout?20:30
exarkunOr rather, it's whatever the regular cache initialization overhead is, but every time, because there's no cache?20:31
jelmer_exarkun: no, it's less than that I think20:37
exarkunMaybe I can try it and see.  How would I do it?20:38
jelmer_exarkun: set use-cache = False in subversion.conf20:47
exarkunam I overlooking some docs for bzr-svn?  I read what I could find with google, but it wasn't much, and didn't cover this.20:49
exarkunWhat section does use-cache go in?20:57
exarkunProbably not bbbe8e31-12d6-0310-92fd-ac37d47ddeeb20:57
jelmer_the applicable repository21:01
exarkunreally, really, really slow :/21:08
exarkunminutes in, still running21:08
exarkundone... about three and a half minutes21:09
exarkunso, sort of defeats the point in this case21:09
LorIs there some conceptual difference between push and pull? Or are they the same operation, just from different perspectives?21:27
jelmer_Lor: they don't have any significant conceptual differences21:32
LorThey don't seem to share much code, though.21:33
LorAll right, Branch.update_revisions seems to be the highest-level operation that both call.21:35
Lor(Not very high, it would seem)21:35
jelmer_Lor: it's pretty high21:36
LorWell yes, but ideally, "bzr push -d foo bar" would just be an alias for "bzr pull -d bar foo", whereas now there are huge amounts of push- or pull-specific code.21:41
jelmer_Lor: I think huge is exaggerating21:43
jelmer_Lor: There's also a difference between conceptual difference and implementation21:44
LorAll right, granted.21:44
jelmer_Lor: I think the implementation is optimized so that you usually push from a local repo and pull into a local repo21:44
jelmer_rather than pushing from a remote repo into a local repo, etc.21:44
LorSurely either operation could check which of the branches is local and choose a suitable implementation accordingly at runtime?21:45
jelmer_well, s/implemented/intended to be implemented/21:45
jelmer_Lor: sorry, s/local/quick/, s/remote/slow/21:45
LorWhich would be faster if it is to be expected that both branches are in sync already?21:46
jelmer_about the same speed I guess21:47
lifelessLor: cd into one of the branches, run push or pull appropriately21:47
exarkunjelmer_: Will python-tdb 1.1.5 work with libdb1 1.1.1?21:47
=== radoe_ is now known as radoe
Lorlifeless, I don't understand. How is the working directory relevant?21:48
jelmer_exarkun: I'm not sure I follow - libdb1 is unrelated21:50
exarkunPardon me21:50
exarkunlibtdb121:50
jelmer_I think so but I would recommend just upgrading libtdb1 as well while you're at it21:51
jelmer_they're both from the sam upstream source package21:51
exarkunThat seems to work22:01
lifelessLor: push [largely] ignores the working tree, pull updates the local working tree22:05
lifelesspush updating the target working tree in a local push is a bit of an anachronism22:06
lifelessLor: are you preparing a patch for push and pull?22:06
LorNo, I'm just wondering what's the best way to do a two-way sync between two repositories that have a large number of branches, only a few of which have probably changed.22:07
lifelessthere is a multi push / multi pull plugin aruond22:08
LorYes, but this is a bit different.22:08
lifelessthe largest overhead there is that the same repo gets serially reopened and probed every time22:08
lifelessideally you'd gather the revs needed from all branches + all tags, fetch those at the repo layer, then update the branches22:09
lifelessits somewhere on my list to make openning a branch with an existing repo object easier22:09
LorActually, I have a directory hierarchy that contains a number of projects, each of which has its own repository, and one or more branches.22:10
LorI read somewhere that it's not a good idea to store unrelated projects in a single shared repository.22:10
LorBut if both the local and remote branch are at the same revision already, then is the repository opened at all?22:13
lifelessyes22:14
lifelessopening a branch opens the repo22:14
lifelessthats a single RPC to do both if you're using bzr+ssh22:15
lifelesswe don't care either way if you use a single repo or several; b+tree indices mean there is only marginal overhead until you get to loads and loads of revisions22:15
Lorhttps://lists.ubuntu.com/archives/bazaar/2008q1/038715.html22:19
LorSo this is outdated?22:19
lifelessno its accurate22:19
lifelessjohn describes the constraints well22:20
LorAll right.22:20
lifelessour scaling goal, when building the indices we use, was a repository with 1000000 revisions, and a total of 1000000 files.22:20
=== retupmoca` is now known as retupmoca
LorShouldn't a repository be able to segregate different projects into distinct stores internally?22:21
lifelessno22:21
LorAfter all, related branches always have the same tree root id nowadays, right?22:21
lifelesssame root id doesn't mean same project, because of old trees22:21
lifelessalso it would be a bunch of cpu overhead to do such an analysis22:22
lifelessif you want separate stores, it should be really easy to use different repos22:22
LorYes, except the directory layout is constrained by the fact that a repository must be in a parent directory.22:23
LorI can't do a branch/project layout, I have to do project/branch22:24
lifelessregardless; we support having unrelated projects in one project.22:24
lifelessthey should perform ok22:24
LorAll right.22:24
LorI'm a bit wary of that, though, since I remember that at one time I stumbled upon a bug that gave some collision alerts when I tried to store multiple unrelated projects in a shared repo.22:25
lifelessif they don't, and you're under the 10^6x10^6 scaling factor, its likely a shallow bug. If you're scaling above that we may have deeper work to do.22:25
LorI know it's been fixed, but nowadays I'd rather keep each repository small just to limit possible damage if a repository gets broken somehow.22:25
LorYes, I don't think I'm going to have any performance problems regarding the repositories themselves.22:26
LorI'm more worried about latency when doing a large number of pulls and pushes.22:26
LorBut this is premature, I will soon see how well thing work in practice.22:27
lifelessthe potato programming around branches will be a signficant slowdown22:27
lifelessWe should help you with that and make it better for all. \o/.22:27
LorDoes push record the pushed revision-id so that if another push to the same location is attempted before there are new local revisions, no transport is even opened?22:30
lifelessno22:30
LorAll right, I will probably implement something like that myself, then.22:31
lifelessthat would be a bug in any case, because the remote repo might have regressed22:31
lifelesss/repo/branch22:31
LorAh.22:31
lifelessthe possible_transports parameter will be of use to you, I suspect.22:32
LorI'm not sure. I'm probably arranging things so that I open a transport only once and then do everything with that transport.22:33
LorI was also toying with the idea of opening multiple transports to the same location.22:34
lifelessas bzr is synchronous that won't help much unless you also use threads22:34
LorThat was implicit. :)22:35
lifelessif you use threads be sure to instantiate all bzr objects within that thread - we're thread safe as a library [modulo bugs] but don't support using a single object concurrently in different threads ;)22:35
lifelesswil you be using bzr+ssh ?22:36
LorProbably, yes.22:36
lifelessok22:36
lifelessI have a suggestion22:36
lifelessthe server is pluggable22:36
lifelessyou can add methods to it22:36
lifelesscalling them is a little....ugly, but doable.22:36
lifelesswrite two server side methods22:37
lifelessone to take a root dir22:37
lifelessopen all the branches22:37
lifelessit would return two things22:38
lifelessfirstly, for each branch: the (branch lock token) and the (last_revision) for each branch22:38
lifelesssecondly, a set of heads obtained by checking the tags and last_revisions for each branch and using the local (its on the server remember) repository to remove ancestors - just call heads basically.22:39
LorI looked at the bzrtools implementation of heads and it seemed surprisingly nonstraightforward.22:40
lifelessthe second server side method would take a list of branches  with each getting a branch lock token, a new tip, a new tags dict, and would apply it to each branch, unlocking them afterwards22:40
lifelessLor: repository.get_graph.heads()22:40
* lifeless handwaves22:41
lifelessits trivial to just do the core22:41
lifelessanyhow, you'd then do this:22:41
lifelessnew_method_1()22:41
lifelessopen the remote repo (or pethaps new-method-1 returns it too22:41
lifelessdo a missing-graph-chatter [ its in remote.py's fetch codepath, I think. I can help you, just not today, on this bit]22:42
lifelessremote_repo.fetch(local_repo, search_result=result-of-missing-graph-chatter)22:42
lifelessnew_method_2()22:42
lifelessso, 5 or 6 round trips22:42
LorSo basically a protocol extension to support repo-push directly?22:43
lifelessyes22:43
lifelesswe designed to permit extensions [like this] - though this is one we do want the core to have eventually anyway22:44
LorI get the idea, but compatibility with current versions is important to me (I don't want to install a cutting-edge bzr.dev on every remote machine).22:45
LorAlso, I just remembered, I try to use encrypted repositories when possible, and until bzr supports it directly, the best solution I've found is to use encfs+sshfs for the remote repo.22:46
lifelessøshouldn't impact this22:46
LorWell it does, since then the remote repository will look like a local one, and the protocol is not used.22:47
LorThe point of this all (if you're wondering) is to use bzr to back up everything in my home directory, using a possibly untrusted remote backup machine.22:48
LorAnd if you don't want to trust the remote machine at all, I don't think that the remote server could do anything very sensible with the encrypted data in the repository, other than send it to the client.22:50
lifelessoh22:50
LorAnd that's what sshfs does already.22:50
lifelessso if you're writing over vfs basically then it doesn't really matter; its going to slow no matter what ;P22:50
LorWell yeah.22:50
LorI also intend to use this to synchronize the home directory between trusted machines, so there the bzr protocol is relevant.22:51
exarkunAnyone interested in this?22:59
exarkunbzr: ERROR: Unprintable exception PermissionDenied: dict={'path': u'/srv/d_ubuntu-gandi/buildbot/.bzr', '_preformatted_string': None, 'extra': ": [Errno 13] Permission non accord\xc3\xa9e: '/srv/d_ubuntu-gandi/buildbot/.bzr'"}, fmt='Permission denied: "%(path)s"%(extra)s', error=UnicodeDecodeError('ascii', ": [Errno 13] Permission non accord\xc3\xa9e: '/srv/d_ubuntu-gandi/buildbot/.bzr'", 34, 35, 'ordinal not in range(128)')22:59
lifelesssure23:01
lifelesswhat bzr version ?23:01
lifelesswe've been fixing a bunch of issues like that in bzr.dev this last couple of weeks23:01
exarkun2.1.1, from the ppa23:01
lifelesspossibly a dupe23:01
lifelessmgz: ^ ?23:01
exarkunI don't think I'm in a position to test a newer dev version on this machine, unfortunately23:02
lifelessthats ok23:03
lifelessleave it with us23:03
lifelessthe underlying error is clear enough to you though ?23:03
exarkunYep23:03
exarkunHrm.23:29
exarkunSame bzr version, 2.1.1, from the ppa, still seeing this error:23:30
exarkunbzr: ERROR: Pack 'a60d0a0175e881a665d9d991452f2b79' already exists in GCRepositoryPackCollection(CHKInventoryRepository('file:///var/lib/buildbot/bigdog24-twisted/.bzr/repository/'))23:30
lifelessthats interesting23:31
lifelessare you pulling from two bzr-svn branches concurrently ?23:31
exarkunThere is definitely concurrency23:31
lifelesscan you do bzr dump-btree file:///var/lib/buildbot/bigdog24-twisted/.bzr/repository/pack-names23:31
exarkunBut it's probably from the same bzr-svn branch multiple times23:31
lifelesshas any of your tests reached a full checkout yet ?23:32
exarkundump-btree fails23:33
exarkungenerates a crash report23:33
exarkunI dunno, hopefully.  It's kind of hard to tell though.23:34
lifelessthe crash report will have a backtrace in it23:34
exarkunShould I pastebin it, or file a bug report?23:34
exarkunmaybe both, anyway http://pastebin.com/aJV30gXW23:35
lifelessplease file a bug23:35
lifelessthats unexpected, at best23:36
exarkunThis looks maybe related?  https://bugs.launchpad.net/bzr/+bug/48860723:40
ubot5Launchpad bug 488607 in Bazaar "bzr dump-btree .cix fails: tuple index out of range (affected: 1, heat: 6)" [Low,Fix released]23:40
exarkunI'm not sure how to see which release the fix has been included in, if any, though.23:41
lifelessdifferent index file23:41
exarkunOkay23:41
lifelesshowever, it would be in 2.1.2 I think23:41
lifelesslets see23:41
lifelessno, 2.2.023:42
exarkunk, filed https://bugs.launchpad.net/bzr/+bug/58436223:45
ubot5Launchpad bug 584362 in Bazaar "Unhandled IndexError from 'bzr dump-btree .../repository/pack-names' (affected: 1, heat: 0)" [Undecided,New]23:45
lifelessthanks23:45
lifelessoff to see more houses; ciao23:45
exarkunHave fun23:46
exarkun(but I don't really care about `dump-btree`, I want `bzr checkout ...` to work :( :( :( :( :( :( :()23:46

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!