[00:05] <Lor> What would be the most efficient way to make a gazillion pulls from a particular host, with the assumption that most of those pulls will do nothing (the local branch will be up-to-date)?
[00:06] <Lor> Would it make sense to open multiple transports in parallel?
[00:09] <exarkun> If I want my shared repo to be more general, can I just move the .bzr directory up a directory level or two?
[00:10]  * exarkun tries it to see what happens anyway
[00:14] <Lor> I think it ought to work.
[00:28] <exarkun> Hey cool it's working I guess
[01:07] <Kamping_Kaiser> how can bzr use 33013KB to repack 850kb of text? ;s
[01:07] <Kamping_Kaiser> [################|   ]  33319KB    48KB/s | Fetching revisions:Inserting stream:repacking texts:texts 9184/16360
[01:07]  * Kamping_Kaiser kills it
[01:51] <exarkun> How did I make this happen?  And how do I fix it?
[01:51] <exarkun> bzr: ERROR: Repository KnitPackRepository('file:///var/lib/buildbot/.bzr/repository/') is not compatible with repository SvnRepository('svn://svn.twistedmatrix.com/svn/Twisted')
[07:28] <hno> Hmm.. shouldn't bzr automatically figure out if my local shared repository already have a copy of a referenced remote branch?
[07:30] <hno> did a bzr diff -r ancestor:. --new lp:...   and the shared repository of the current branch also includes a copy of the lp branch.
[07:32] <hno> and yet it fetches some 10MB of data from the lp branch.
[08:18] <spiv> hno: it should, yes
[08:18] <spiv> hno: mind adding -Dhpss to that command line and filing a bug?
[08:19] <spiv> hno: (attach your ~/.bzr.log)
[09:28] <hno> spiv: Will do.
[09:31] <hno> HPSS calls: 193 (188 vfs) SmartSSHClientMedium(bzr+ssh://henriknordstrom@bazaar.launchpad.net/)
[09:31] <hno> hi lifeless
[09:36] <hno> Hmm.. I must be stupid but how do I file a bug report. Can only figure out how to search for them...
[09:36] <lifeless> hi hno
[09:36] <lifeless> on ?
[09:37] <hno> bzr
[09:37] <lifeless> bugs.launchpad.net/bzr
[09:37] <lifeless> top right 'report a bug'
[09:37] <hno> ah, the big red one...
[09:58] <lifeless> vila: haihai
[10:05] <vila> lifeless: hey ;-)
[10:11] <lifeless> I wish lp apis were easier to test with.
[10:12] <lifeless> do we have a supported mocking tool ?
[10:12] <vila> not that I know of :-(
[10:12] <vila> I think I read people using staging for that lately
[10:13] <vila> yes, I know you asked for a mockup
[10:19] <lifeless> o/~ shave that yak
[10:20] <lifeless> yakkity yak
[10:35] <hno> bug filed.
[10:37] <hno> wonder why I always seem to run into these cases of excess bandwidth usage by bzr.. can't imagine my use of bzr is that strange compared to what most others are doing.
[10:38] <lifeless> we have two critical bugs open at the moment about just that
[10:38] <lifeless> maybe you filed a dupe
[10:39] <hno> bzr diff remotebranch fails to find that the remote revisions is already in the local repository.
[10:40] <hno> https://bugs.launchpad.net/bzr/+bug/584148
[10:40] <lifeless> ah
[10:41] <lifeless> that isn't what we've got open; and it is a little unusal
[10:41] <lifeless> while it works - and thats important - most folk mirror things locally to play with them
[10:43] <hno> lifeless: I don't think the case based on bzr switch is that unusual...
[10:43] <lifeless> hno: thats orthogonal really
[10:44] <hno> ?
[10:44] <hno> to the bug yes.
[10:45] <lifeless> I use switch based workflow myself
[10:45] <lifeless> for most everything
[10:45] <hno> so it will bite you as well.
[10:46] <hno> except that you diff by using merge...
[15:06] <exarkun> Uh so
[15:06] <exarkun> http://buildbot.twistedmatrix.com/builders/hardy32-py2.5-glib2/builds/24/steps/bzr/logs/stdio
[15:30] <exarkun> Anybody?
[15:35] <exarkun> I see there's a ticket filed about this.
[15:40] <exarkun> Anyone know of any docs about bzr-svn plus tdb?
[15:41] <exarkun> The last comment on https://bugs.launchpad.net/null/+bug/185200 suggests that just installing python-tdb will address the issue, but I've had it installed since I started setting this up, and it doesn't seem to have been used.
[16:15] <jml> jelmer_, hello?
[16:16] <jelmer_> jml: hey
[16:17] <jml> jelmer_, I was wondering if I could tempt you into helping out exarkun
[16:17]  * jelmer_ looks at the backlog
[16:18] <jelmer_> exarkun: hi
[16:18] <exarkun> Heya
[16:18] <jelmer_> exarkun: you may have to remove the existing cache files since they will be in sqlite
[16:18] <exarkun> In ~/.bazaar/svn-cache/ ?
[16:18] <jelmer_> exarkun: yep, or ~/.cache/bazaar/svn
[16:19] <exarkun> Okay, I'll try that.
[16:20] <jelmer_> we should be able to leave this whole thing behind us when the cache format becomes pack-file-based
[16:20] <exarkun> The new cache file is a "SQLite 3.x database"
[16:21] <jelmer_> exarkun: what version of bzr-svn and python-tdb do you have?
[16:22] <exarkun> Bazaar (bzr) 2.1.1
[16:22] <jelmer_> exarkun: and bzr-svn and python-tdb?
[16:23] <exarkun> Uh, "svn 1.0.2"?
[16:23] <exarkun> And python-tdb 1.1.1~svn26294-1 (ie, hardy package)
[16:24] <jelmer_> exarkun: the hardy package of python-tdb is probably too old
[16:24] <jelmer_> anyway, gtg - back in about an hour
[16:29] <exarkun> I suppose I'll fix this some other way then
[17:03] <strk> is 'bzr clone' different from 'bzr branch' ?
[17:04] <exarkun> no
[17:04] <exarkun> See the "Aliases" section of "bzr branch --help"
[17:13] <exarkun> Uhm
[17:14] <exarkun> How about this?
[17:14] <exarkun> bzr: ERROR: Pack '51d53eb11548019c7208e76646979ac4' already exists in <bzrlib.repofmt.groupcompress_repo.GCRepositoryPackCollection object at 0x1cddc50>
[17:14] <exarkun> lifeless: I think next time someone asks if concurrent access is safe, you should say no.
[17:38] <jelmer_> exarkun: that's a known bug in older versions of bzr iirc
[17:38] <jelmer_> exarkun: it doesn't have anything to do with bzr-svn at least
[18:30] <marioxcc`> hi all
[18:30] <marioxcc`> i have a question:
[18:30] <marioxcc`> how are the branch in bazaar relative to branch in git?
[18:31] <marioxcc`> ¿do bazaar and git use the same model of heads or what are the diferences?
[18:35] <exarkun> jelmer_: Seems maybe to have gone away after upgrading from 2.0 to 2.1, thanks
[19:40] <ricqles> Hi all !
[19:46] <exarkun> Can I have hooks run when 'bzr svn-update' updates a bzr branch from an svn repository?
[19:48] <mkanat> exarkun: post_branch_tip_change would work, wouldn't it?
[19:49] <exarkun> mkanat: I have no idea, I've never heard of that thing before
[19:49] <exarkun> Google returns no hits for it
[19:49] <mkanat> exarkun: "bzr hooks" will get you a list of hooks.
[19:49] <exarkun> Neat
[19:49] <exarkun> Oops
[19:49] <exarkun> I get a traceback
[19:50] <mkanat> exarkun: It's pre_change_branch_tip -- I tyoped it slightly. :-)
[19:50] <mkanat> exarkun: Or post_change_branch_tip
[19:50] <effstops> hey guys, has anyone had trouble with the OS X installer?
[19:50] <effstops> as it's installing sip I get the error "The Installer could not install some files in "/".
[19:51] <exarkun> mkanat: Thanks.
[20:03] <exarkun> Hm, I guess that's not really my top priority now though
[20:03] <exarkun> I still need to figure out how to fix this SQLite3 "database is locked" issue.
[20:03] <exarkun> I bet I can set $HOME to get bzr-svn to use a different svn-cache
[20:04] <exarkun> I wonder if there's a better way, though (hopefully a way that won't accidentally affect a ton of other stuff too)
[20:16] <exarkun> Does 'bzr checkout svn://...' make a lot of connections to the svn server?
[20:17] <marioxcc`> exarkun: use netstart to check
[20:17] <exarkun> netstart?
[20:19] <mkanat> exarkun: netstat
[20:20] <exarkun> Ah.  That's kind of tricky, since I'd have to be sure to run it in a super fast loop and then filter the results to see how many connections it made.  I was hoping someone would just know.
[20:20] <jelmer_> exarkun: one or two
[20:20] <exarkun> I can use strace, though.
[20:20] <exarkun> jelmer_: Aha, thanks.
[20:20] <exarkun> I'm now seeing my inetd decide to disable svnserve because too many connections are coming in
[20:20] <jelmer_> exarkun: it will reuse connections it opens, and it fetches revisions serially
[20:20] <exarkun> but all I've done is switch things from 'svn co ...' to 'bzr co ...'
[20:21] <exarkun> I suppose if it was already near the limit and sometimes bzr uses two, that might have pushed it over.
[20:22] <exarkun> jelmer_: Any hints on a good way to get my different... actors... to use different svn-cache directories, aside from setting $HOME differently for each of them?
[20:25] <jelmer_> exarkun: why would you want to?
[20:27] <exarkun> jelmer_: Because I can't find a python-tdb package for Hardy that's new enough for bzr-svn and so most of my 'bzr checkout' commands are failing with SQLite3 errors
[20:28] <exarkun> If you have other ideas about how to fix that, I'm open to suggestions
[20:29] <jelmer_> exarkun: You could avoid using the cache altogether but that will impact performance
[20:30] <exarkun> Yea... that's like an extra 30 - 45 seconds per checkout?
[20:31] <exarkun> Or rather, it's whatever the regular cache initialization overhead is, but every time, because there's no cache?
[20:37] <jelmer_> exarkun: no, it's less than that I think
[20:38] <exarkun> Maybe I can try it and see.  How would I do it?
[20:47] <jelmer_> exarkun: set use-cache = False in subversion.conf
[20:49] <exarkun> am I overlooking some docs for bzr-svn?  I read what I could find with google, but it wasn't much, and didn't cover this.
[20:57] <exarkun> What section does use-cache go in?
[20:57] <exarkun> Probably not bbbe8e31-12d6-0310-92fd-ac37d47ddeeb
[21:01] <jelmer_> the applicable repository
[21:08] <exarkun> really, really, really slow :/
[21:08] <exarkun> minutes in, still running
[21:09] <exarkun> done... about three and a half minutes
[21:09] <exarkun> so, sort of defeats the point in this case
[21:27] <Lor> Is there some conceptual difference between push and pull? Or are they the same operation, just from different perspectives?
[21:32] <jelmer_> Lor: they don't have any significant conceptual differences
[21:33] <Lor> They don't seem to share much code, though.
[21:35] <Lor> All right, Branch.update_revisions seems to be the highest-level operation that both call.
[21:35] <Lor> (Not very high, it would seem)
[21:36] <jelmer_> Lor: it's pretty high
[21:41] <Lor> Well yes, but ideally, "bzr push -d foo bar" would just be an alias for "bzr pull -d bar foo", whereas now there are huge amounts of push- or pull-specific code.
[21:43] <jelmer_> Lor: I think huge is exaggerating
[21:44] <jelmer_> Lor: There's also a difference between conceptual difference and implementation
[21:44] <Lor> All right, granted.
[21:44] <jelmer_> Lor: I think the implementation is optimized so that you usually push from a local repo and pull into a local repo
[21:44] <jelmer_> rather than pushing from a remote repo into a local repo, etc.
[21:45] <Lor> Surely either operation could check which of the branches is local and choose a suitable implementation accordingly at runtime?
[21:45] <jelmer_> well, s/implemented/intended to be implemented/
[21:45] <jelmer_> Lor: sorry, s/local/quick/, s/remote/slow/
[21:46] <Lor> Which would be faster if it is to be expected that both branches are in sync already?
[21:47] <jelmer_> about the same speed I guess
[21:47] <lifeless> Lor: cd into one of the branches, run push or pull appropriately
[21:47] <exarkun> jelmer_: Will python-tdb 1.1.5 work with libdb1 1.1.1?
[21:48] <Lor> lifeless, I don't understand. How is the working directory relevant?
[21:50] <jelmer_> exarkun: I'm not sure I follow - libdb1 is unrelated
[21:50] <exarkun> Pardon me
[21:50] <exarkun> libtdb1
[21:51] <jelmer_> I think so but I would recommend just upgrading libtdb1 as well while you're at it
[21:51] <jelmer_> they're both from the sam upstream source package
[22:01] <exarkun> That seems to work
[22:05] <lifeless> Lor: push [largely] ignores the working tree, pull updates the local working tree
[22:06] <lifeless> push updating the target working tree in a local push is a bit of an anachronism
[22:06] <lifeless> Lor: are you preparing a patch for push and pull?
[22:07] <Lor> No, I'm just wondering what's the best way to do a two-way sync between two repositories that have a large number of branches, only a few of which have probably changed.
[22:08] <lifeless> there is a multi push / multi pull plugin aruond
[22:08] <Lor> Yes, but this is a bit different.
[22:08] <lifeless> the largest overhead there is that the same repo gets serially reopened and probed every time
[22:09] <lifeless> ideally you'd gather the revs needed from all branches + all tags, fetch those at the repo layer, then update the branches
[22:09] <lifeless> its somewhere on my list to make openning a branch with an existing repo object easier
[22:10] <Lor> Actually, I have a directory hierarchy that contains a number of projects, each of which has its own repository, and one or more branches.
[22:10] <Lor> I read somewhere that it's not a good idea to store unrelated projects in a single shared repository.
[22:13] <Lor> But if both the local and remote branch are at the same revision already, then is the repository opened at all?
[22:14] <lifeless> yes
[22:14] <lifeless> opening a branch opens the repo
[22:15] <lifeless> thats a single RPC to do both if you're using bzr+ssh
[22:15] <lifeless> we don't care either way if you use a single repo or several; b+tree indices mean there is only marginal overhead until you get to loads and loads of revisions
[22:19] <Lor> https://lists.ubuntu.com/archives/bazaar/2008q1/038715.html
[22:19] <Lor> So this is outdated?
[22:19] <lifeless> no its accurate
[22:20] <lifeless> john describes the constraints well
[22:20] <Lor> All right.
[22:20] <lifeless> our scaling goal, when building the indices we use, was a repository with 1000000 revisions, and a total of 1000000 files.
[22:21] <Lor> Shouldn't a repository be able to segregate different projects into distinct stores internally?
[22:21] <lifeless> no
[22:21] <Lor> After all, related branches always have the same tree root id nowadays, right?
[22:21] <lifeless> same root id doesn't mean same project, because of old trees
[22:22] <lifeless> also it would be a bunch of cpu overhead to do such an analysis
[22:22] <lifeless> if you want separate stores, it should be really easy to use different repos
[22:23] <Lor> Yes, except the directory layout is constrained by the fact that a repository must be in a parent directory.
[22:24] <Lor> I can't do a branch/project layout, I have to do project/branch
[22:24] <lifeless> regardless; we support having unrelated projects in one project.
[22:24] <lifeless> they should perform ok
[22:24] <Lor> All right.
[22:25] <Lor> I'm a bit wary of that, though, since I remember that at one time I stumbled upon a bug that gave some collision alerts when I tried to store multiple unrelated projects in a shared repo.
[22:25] <lifeless> if they don't, and you're under the 10^6x10^6 scaling factor, its likely a shallow bug. If you're scaling above that we may have deeper work to do.
[22:25] <Lor> I know it's been fixed, but nowadays I'd rather keep each repository small just to limit possible damage if a repository gets broken somehow.
[22:26] <Lor> Yes, I don't think I'm going to have any performance problems regarding the repositories themselves.
[22:26] <Lor> I'm more worried about latency when doing a large number of pulls and pushes.
[22:27] <Lor> But this is premature, I will soon see how well thing work in practice.
[22:27] <lifeless> the potato programming around branches will be a signficant slowdown
[22:27] <lifeless> We should help you with that and make it better for all. \o/.
[22:30] <Lor> Does push record the pushed revision-id so that if another push to the same location is attempted before there are new local revisions, no transport is even opened?
[22:30] <lifeless> no
[22:31] <Lor> All right, I will probably implement something like that myself, then.
[22:31] <lifeless> that would be a bug in any case, because the remote repo might have regressed
[22:31] <lifeless> s/repo/branch
[22:31] <Lor> Ah.
[22:32] <lifeless> the possible_transports parameter will be of use to you, I suspect.
[22:33] <Lor> I'm not sure. I'm probably arranging things so that I open a transport only once and then do everything with that transport.
[22:34] <Lor> I was also toying with the idea of opening multiple transports to the same location.
[22:34] <lifeless> as bzr is synchronous that won't help much unless you also use threads
[22:35] <Lor> That was implicit. :)
[22:35] <lifeless> if you use threads be sure to instantiate all bzr objects within that thread - we're thread safe as a library [modulo bugs] but don't support using a single object concurrently in different threads ;)
[22:36] <lifeless> wil you be using bzr+ssh ?
[22:36] <Lor> Probably, yes.
[22:36] <lifeless> ok
[22:36] <lifeless> I have a suggestion
[22:36] <lifeless> the server is pluggable
[22:36] <lifeless> you can add methods to it
[22:36] <lifeless> calling them is a little....ugly, but doable.
[22:37] <lifeless> write two server side methods
[22:37] <lifeless> one to take a root dir
[22:37] <lifeless> open all the branches
[22:38] <lifeless> it would return two things
[22:38] <lifeless> firstly, for each branch: the (branch lock token) and the (last_revision) for each branch
[22:39] <lifeless> secondly, a set of heads obtained by checking the tags and last_revisions for each branch and using the local (its on the server remember) repository to remove ancestors - just call heads basically.
[22:40] <Lor> I looked at the bzrtools implementation of heads and it seemed surprisingly nonstraightforward.
[22:40] <lifeless> the second server side method would take a list of branches  with each getting a branch lock token, a new tip, a new tags dict, and would apply it to each branch, unlocking them afterwards
[22:40] <lifeless> Lor: repository.get_graph.heads()
[22:41]  * lifeless handwaves
[22:41] <lifeless> its trivial to just do the core
[22:41] <lifeless> anyhow, you'd then do this:
[22:41] <lifeless> new_method_1()
[22:41] <lifeless> open the remote repo (or pethaps new-method-1 returns it too
[22:42] <lifeless> do a missing-graph-chatter [ its in remote.py's fetch codepath, I think. I can help you, just not today, on this bit]
[22:42] <lifeless> remote_repo.fetch(local_repo, search_result=result-of-missing-graph-chatter)
[22:42] <lifeless> new_method_2()
[22:42] <lifeless> so, 5 or 6 round trips
[22:43] <Lor> So basically a protocol extension to support repo-push directly?
[22:43] <lifeless> yes
[22:44] <lifeless> we designed to permit extensions [like this] - though this is one we do want the core to have eventually anyway
[22:45] <Lor> I get the idea, but compatibility with current versions is important to me (I don't want to install a cutting-edge bzr.dev on every remote machine).
[22:46] <Lor> Also, I just remembered, I try to use encrypted repositories when possible, and until bzr supports it directly, the best solution I've found is to use encfs+sshfs for the remote repo.
[22:46] <lifeless> øshouldn't impact this
[22:47] <Lor> Well it does, since then the remote repository will look like a local one, and the protocol is not used.
[22:48] <Lor> The point of this all (if you're wondering) is to use bzr to back up everything in my home directory, using a possibly untrusted remote backup machine.
[22:50] <Lor> And if you don't want to trust the remote machine at all, I don't think that the remote server could do anything very sensible with the encrypted data in the repository, other than send it to the client.
[22:50] <lifeless> oh
[22:50] <Lor> And that's what sshfs does already.
[22:50] <lifeless> so if you're writing over vfs basically then it doesn't really matter; its going to slow no matter what ;P
[22:50] <Lor> Well yeah.
[22:51] <Lor> I also intend to use this to synchronize the home directory between trusted machines, so there the bzr protocol is relevant.
[22:59] <exarkun> Anyone interested in this?
[22:59] <exarkun> bzr: ERROR: Unprintable exception PermissionDenied: dict={'path': u'/srv/d_ubuntu-gandi/buildbot/.bzr', '_preformatted_string': None, 'extra': ": [Errno 13] Permission non accord\xc3\xa9e: '/srv/d_ubuntu-gandi/buildbot/.bzr'"}, fmt='Permission denied: "%(path)s"%(extra)s', error=UnicodeDecodeError('ascii', ": [Errno 13] Permission non accord\xc3\xa9e: '/srv/d_ubuntu-gandi/buildbot/.bzr'", 34, 35, 'ordinal not in range(128)')
[23:01] <lifeless> sure
[23:01] <lifeless> what bzr version ?
[23:01] <lifeless> we've been fixing a bunch of issues like that in bzr.dev this last couple of weeks
[23:01] <exarkun> 2.1.1, from the ppa
[23:01] <lifeless> possibly a dupe
[23:01] <lifeless> mgz: ^ ?
[23:02] <exarkun> I don't think I'm in a position to test a newer dev version on this machine, unfortunately
[23:03] <lifeless> thats ok
[23:03] <lifeless> leave it with us
[23:03] <lifeless> the underlying error is clear enough to you though ?
[23:03] <exarkun> Yep
[23:29] <exarkun> Hrm.
[23:30] <exarkun> Same bzr version, 2.1.1, from the ppa, still seeing this error:
[23:30] <exarkun> bzr: ERROR: Pack 'a60d0a0175e881a665d9d991452f2b79' already exists in GCRepositoryPackCollection(CHKInventoryRepository('file:///var/lib/buildbot/bigdog24-twisted/.bzr/repository/'))
[23:31] <lifeless> thats interesting
[23:31] <lifeless> are you pulling from two bzr-svn branches concurrently ?
[23:31] <exarkun> There is definitely concurrency
[23:31] <lifeless> can you do bzr dump-btree file:///var/lib/buildbot/bigdog24-twisted/.bzr/repository/pack-names
[23:31] <exarkun> But it's probably from the same bzr-svn branch multiple times
[23:32] <lifeless> has any of your tests reached a full checkout yet ?
[23:33] <exarkun> dump-btree fails
[23:33] <exarkun> generates a crash report
[23:34] <exarkun> I dunno, hopefully.  It's kind of hard to tell though.
[23:34] <lifeless> the crash report will have a backtrace in it
[23:34] <exarkun> Should I pastebin it, or file a bug report?
[23:35] <exarkun> maybe both, anyway http://pastebin.com/aJV30gXW
[23:35] <lifeless> please file a bug
[23:36] <lifeless> thats unexpected, at best
[23:40] <exarkun> This looks maybe related?  https://bugs.launchpad.net/bzr/+bug/488607
[23:41] <exarkun> I'm not sure how to see which release the fix has been included in, if any, though.
[23:41] <lifeless> different index file
[23:41] <exarkun> Okay
[23:41] <lifeless> however, it would be in 2.1.2 I think
[23:41] <lifeless> lets see
[23:42] <lifeless> no, 2.2.0
[23:45] <exarkun> k, filed https://bugs.launchpad.net/bzr/+bug/584362
[23:45] <lifeless> thanks
[23:45] <lifeless> off to see more houses; ciao
[23:46] <exarkun> Have fun
[23:46] <exarkun> (but I don't really care about `dump-btree`, I want `bzr checkout ...` to work :( :( :( :( :( :( :()