[00:05] What would be the most efficient way to make a gazillion pulls from a particular host, with the assumption that most of those pulls will do nothing (the local branch will be up-to-date)? [00:06] Would it make sense to open multiple transports in parallel? [00:09] If I want my shared repo to be more general, can I just move the .bzr directory up a directory level or two? [00:10] * exarkun tries it to see what happens anyway [00:14] I think it ought to work. [00:28] Hey cool it's working I guess [01:07] how can bzr use 33013KB to repack 850kb of text? ;s [01:07] [################| ] 33319KB 48KB/s | Fetching revisions:Inserting stream:repacking texts:texts 9184/16360 [01:07] * Kamping_Kaiser kills it === joerg is now known as Guest53356 [01:51] How did I make this happen? And how do I fix it? [01:51] bzr: ERROR: Repository KnitPackRepository('file:///var/lib/buildbot/.bzr/repository/') is not compatible with repository SvnRepository('svn://svn.twistedmatrix.com/svn/Twisted') [07:28] Hmm.. shouldn't bzr automatically figure out if my local shared repository already have a copy of a referenced remote branch? [07:30] did a bzr diff -r ancestor:. --new lp:... and the shared repository of the current branch also includes a copy of the lp branch. [07:32] and yet it fetches some 10MB of data from the lp branch. === wgrant_ is now known as wgrant [08:18] hno: it should, yes [08:18] hno: mind adding -Dhpss to that command line and filing a bug? [08:19] hno: (attach your ~/.bzr.log) [09:28] spiv: Will do. [09:31] HPSS calls: 193 (188 vfs) SmartSSHClientMedium(bzr+ssh://henriknordstrom@bazaar.launchpad.net/) [09:31] hi lifeless [09:36] Hmm.. I must be stupid but how do I file a bug report. Can only figure out how to search for them... [09:36] hi hno [09:36] on ? [09:37] bzr [09:37] bugs.launchpad.net/bzr [09:37] top right 'report a bug' [09:37] ah, the big red one... [09:58] vila: haihai [10:05] lifeless: hey ;-) [10:11] I wish lp apis were easier to test with. [10:12] do we have a supported mocking tool ? [10:12] not that I know of :-( [10:12] I think I read people using staging for that lately [10:13] yes, I know you asked for a mockup [10:19] o/~ shave that yak [10:20] yakkity yak [10:35] bug filed. [10:37] wonder why I always seem to run into these cases of excess bandwidth usage by bzr.. can't imagine my use of bzr is that strange compared to what most others are doing. [10:38] we have two critical bugs open at the moment about just that [10:38] maybe you filed a dupe [10:39] bzr diff remotebranch fails to find that the remote revisions is already in the local repository. [10:40] https://bugs.launchpad.net/bzr/+bug/584148 [10:40] Launchpad bug 584148 in Bazaar "bzr diff remotebranch makes poor use of revisions on local repository (affected: 1, heat: 0)" [Undecided,New] [10:40] ah [10:41] that isn't what we've got open; and it is a little unusal [10:41] while it works - and thats important - most folk mirror things locally to play with them [10:43] lifeless: I don't think the case based on bzr switch is that unusual... [10:43] hno: thats orthogonal really [10:44] ? [10:44] to the bug yes. [10:45] I use switch based workflow myself [10:45] for most everything [10:45] so it will bite you as well. [10:46] except that you diff by using merge... [15:06] Uh so [15:06] http://buildbot.twistedmatrix.com/builders/hardy32-py2.5-glib2/builds/24/steps/bzr/logs/stdio [15:30] Anybody? [15:35] I see there's a ticket filed about this. [15:40] Anyone know of any docs about bzr-svn plus tdb? [15:41] The last comment on https://bugs.launchpad.net/null/+bug/185200 suggests that just installing python-tdb will address the issue, but I've had it installed since I started setting this up, and it doesn't seem to have been used. [15:41] Launchpad bug 185200 in Bazaar Subversion Plugin ""database is locked" bzr internal error (affected: 2, heat: 0)" [High,Fix released] [16:15] jelmer_, hello? [16:16] jml: hey [16:17] jelmer_, I was wondering if I could tempt you into helping out exarkun [16:17] * jelmer_ looks at the backlog [16:18] exarkun: hi [16:18] Heya [16:18] exarkun: you may have to remove the existing cache files since they will be in sqlite [16:18] In ~/.bazaar/svn-cache/ ? [16:18] exarkun: yep, or ~/.cache/bazaar/svn [16:19] Okay, I'll try that. [16:20] we should be able to leave this whole thing behind us when the cache format becomes pack-file-based [16:20] The new cache file is a "SQLite 3.x database" [16:21] exarkun: what version of bzr-svn and python-tdb do you have? [16:22] Bazaar (bzr) 2.1.1 [16:22] exarkun: and bzr-svn and python-tdb? [16:23] Uh, "svn 1.0.2"? [16:23] And python-tdb 1.1.1~svn26294-1 (ie, hardy package) [16:24] exarkun: the hardy package of python-tdb is probably too old [16:24] anyway, gtg - back in about an hour [16:29] I suppose I'll fix this some other way then [17:03] is 'bzr clone' different from 'bzr branch' ? [17:04] no [17:04] See the "Aliases" section of "bzr branch --help" [17:13] Uhm [17:14] How about this? [17:14] bzr: ERROR: Pack '51d53eb11548019c7208e76646979ac4' already exists in [17:14] lifeless: I think next time someone asks if concurrent access is safe, you should say no. [17:38] exarkun: that's a known bug in older versions of bzr iirc [17:38] exarkun: it doesn't have anything to do with bzr-svn at least [18:30] hi all [18:30] i have a question: [18:30] how are the branch in bazaar relative to branch in git? [18:31] ¿do bazaar and git use the same model of heads or what are the diferences? [18:35] jelmer_: Seems maybe to have gone away after upgrading from 2.0 to 2.1, thanks [19:40] Hi all ! [19:46] Can I have hooks run when 'bzr svn-update' updates a bzr branch from an svn repository? [19:48] exarkun: post_branch_tip_change would work, wouldn't it? === Guest53356 is now known as joerg [19:49] mkanat: I have no idea, I've never heard of that thing before [19:49] Google returns no hits for it [19:49] exarkun: "bzr hooks" will get you a list of hooks. [19:49] Neat [19:49] Oops [19:49] I get a traceback [19:50] exarkun: It's pre_change_branch_tip -- I tyoped it slightly. :-) [19:50] exarkun: Or post_change_branch_tip [19:50] hey guys, has anyone had trouble with the OS X installer? [19:50] as it's installing sip I get the error "The Installer could not install some files in "/". [19:51] mkanat: Thanks. [20:03] Hm, I guess that's not really my top priority now though [20:03] I still need to figure out how to fix this SQLite3 "database is locked" issue. [20:03] I bet I can set $HOME to get bzr-svn to use a different svn-cache [20:04] I wonder if there's a better way, though (hopefully a way that won't accidentally affect a ton of other stuff too) [20:16] Does 'bzr checkout svn://...' make a lot of connections to the svn server? [20:17] exarkun: use netstart to check [20:17] netstart? [20:19] exarkun: netstat [20:20] Ah. That's kind of tricky, since I'd have to be sure to run it in a super fast loop and then filter the results to see how many connections it made. I was hoping someone would just know. [20:20] exarkun: one or two [20:20] I can use strace, though. [20:20] jelmer_: Aha, thanks. [20:20] I'm now seeing my inetd decide to disable svnserve because too many connections are coming in [20:20] exarkun: it will reuse connections it opens, and it fetches revisions serially [20:20] but all I've done is switch things from 'svn co ...' to 'bzr co ...' [20:21] I suppose if it was already near the limit and sometimes bzr uses two, that might have pushed it over. [20:22] jelmer_: Any hints on a good way to get my different... actors... to use different svn-cache directories, aside from setting $HOME differently for each of them? [20:25] exarkun: why would you want to? [20:27] jelmer_: Because I can't find a python-tdb package for Hardy that's new enough for bzr-svn and so most of my 'bzr checkout' commands are failing with SQLite3 errors [20:28] If you have other ideas about how to fix that, I'm open to suggestions [20:29] exarkun: You could avoid using the cache altogether but that will impact performance [20:30] Yea... that's like an extra 30 - 45 seconds per checkout? [20:31] Or rather, it's whatever the regular cache initialization overhead is, but every time, because there's no cache? [20:37] exarkun: no, it's less than that I think [20:38] Maybe I can try it and see. How would I do it? [20:47] exarkun: set use-cache = False in subversion.conf [20:49] am I overlooking some docs for bzr-svn? I read what I could find with google, but it wasn't much, and didn't cover this. [20:57] What section does use-cache go in? [20:57] Probably not bbbe8e31-12d6-0310-92fd-ac37d47ddeeb [21:01] the applicable repository [21:08] really, really, really slow :/ [21:08] minutes in, still running [21:09] done... about three and a half minutes [21:09] so, sort of defeats the point in this case [21:27] Is there some conceptual difference between push and pull? Or are they the same operation, just from different perspectives? [21:32] Lor: they don't have any significant conceptual differences [21:33] They don't seem to share much code, though. [21:35] All right, Branch.update_revisions seems to be the highest-level operation that both call. [21:35] (Not very high, it would seem) [21:36] Lor: it's pretty high [21:41] Well yes, but ideally, "bzr push -d foo bar" would just be an alias for "bzr pull -d bar foo", whereas now there are huge amounts of push- or pull-specific code. [21:43] Lor: I think huge is exaggerating [21:44] Lor: There's also a difference between conceptual difference and implementation [21:44] All right, granted. [21:44] Lor: I think the implementation is optimized so that you usually push from a local repo and pull into a local repo [21:44] rather than pushing from a remote repo into a local repo, etc. [21:45] Surely either operation could check which of the branches is local and choose a suitable implementation accordingly at runtime? [21:45] well, s/implemented/intended to be implemented/ [21:45] Lor: sorry, s/local/quick/, s/remote/slow/ [21:46] Which would be faster if it is to be expected that both branches are in sync already? [21:47] about the same speed I guess [21:47] Lor: cd into one of the branches, run push or pull appropriately [21:47] jelmer_: Will python-tdb 1.1.5 work with libdb1 1.1.1? === radoe_ is now known as radoe [21:48] lifeless, I don't understand. How is the working directory relevant? [21:50] exarkun: I'm not sure I follow - libdb1 is unrelated [21:50] Pardon me [21:50] libtdb1 [21:51] I think so but I would recommend just upgrading libtdb1 as well while you're at it [21:51] they're both from the sam upstream source package [22:01] That seems to work [22:05] Lor: push [largely] ignores the working tree, pull updates the local working tree [22:06] push updating the target working tree in a local push is a bit of an anachronism [22:06] Lor: are you preparing a patch for push and pull? [22:07] No, I'm just wondering what's the best way to do a two-way sync between two repositories that have a large number of branches, only a few of which have probably changed. [22:08] there is a multi push / multi pull plugin aruond [22:08] Yes, but this is a bit different. [22:08] the largest overhead there is that the same repo gets serially reopened and probed every time [22:09] ideally you'd gather the revs needed from all branches + all tags, fetch those at the repo layer, then update the branches [22:09] its somewhere on my list to make openning a branch with an existing repo object easier [22:10] Actually, I have a directory hierarchy that contains a number of projects, each of which has its own repository, and one or more branches. [22:10] I read somewhere that it's not a good idea to store unrelated projects in a single shared repository. [22:13] But if both the local and remote branch are at the same revision already, then is the repository opened at all? [22:14] yes [22:14] opening a branch opens the repo [22:15] thats a single RPC to do both if you're using bzr+ssh [22:15] we don't care either way if you use a single repo or several; b+tree indices mean there is only marginal overhead until you get to loads and loads of revisions [22:19] https://lists.ubuntu.com/archives/bazaar/2008q1/038715.html [22:19] So this is outdated? [22:19] no its accurate [22:20] john describes the constraints well [22:20] All right. [22:20] our scaling goal, when building the indices we use, was a repository with 1000000 revisions, and a total of 1000000 files. === retupmoca` is now known as retupmoca [22:21] Shouldn't a repository be able to segregate different projects into distinct stores internally? [22:21] no [22:21] After all, related branches always have the same tree root id nowadays, right? [22:21] same root id doesn't mean same project, because of old trees [22:22] also it would be a bunch of cpu overhead to do such an analysis [22:22] if you want separate stores, it should be really easy to use different repos [22:23] Yes, except the directory layout is constrained by the fact that a repository must be in a parent directory. [22:24] I can't do a branch/project layout, I have to do project/branch [22:24] regardless; we support having unrelated projects in one project. [22:24] they should perform ok [22:24] All right. [22:25] I'm a bit wary of that, though, since I remember that at one time I stumbled upon a bug that gave some collision alerts when I tried to store multiple unrelated projects in a shared repo. [22:25] if they don't, and you're under the 10^6x10^6 scaling factor, its likely a shallow bug. If you're scaling above that we may have deeper work to do. [22:25] I know it's been fixed, but nowadays I'd rather keep each repository small just to limit possible damage if a repository gets broken somehow. [22:26] Yes, I don't think I'm going to have any performance problems regarding the repositories themselves. [22:26] I'm more worried about latency when doing a large number of pulls and pushes. [22:27] But this is premature, I will soon see how well thing work in practice. [22:27] the potato programming around branches will be a signficant slowdown [22:27] We should help you with that and make it better for all. \o/. [22:30] Does push record the pushed revision-id so that if another push to the same location is attempted before there are new local revisions, no transport is even opened? [22:30] no [22:31] All right, I will probably implement something like that myself, then. [22:31] that would be a bug in any case, because the remote repo might have regressed [22:31] s/repo/branch [22:31] Ah. [22:32] the possible_transports parameter will be of use to you, I suspect. [22:33] I'm not sure. I'm probably arranging things so that I open a transport only once and then do everything with that transport. [22:34] I was also toying with the idea of opening multiple transports to the same location. [22:34] as bzr is synchronous that won't help much unless you also use threads [22:35] That was implicit. :) [22:35] if you use threads be sure to instantiate all bzr objects within that thread - we're thread safe as a library [modulo bugs] but don't support using a single object concurrently in different threads ;) [22:36] wil you be using bzr+ssh ? [22:36] Probably, yes. [22:36] ok [22:36] I have a suggestion [22:36] the server is pluggable [22:36] you can add methods to it [22:36] calling them is a little....ugly, but doable. [22:37] write two server side methods [22:37] one to take a root dir [22:37] open all the branches [22:38] it would return two things [22:38] firstly, for each branch: the (branch lock token) and the (last_revision) for each branch [22:39] secondly, a set of heads obtained by checking the tags and last_revisions for each branch and using the local (its on the server remember) repository to remove ancestors - just call heads basically. [22:40] I looked at the bzrtools implementation of heads and it seemed surprisingly nonstraightforward. [22:40] the second server side method would take a list of branches with each getting a branch lock token, a new tip, a new tags dict, and would apply it to each branch, unlocking them afterwards [22:40] Lor: repository.get_graph.heads() [22:41] * lifeless handwaves [22:41] its trivial to just do the core [22:41] anyhow, you'd then do this: [22:41] new_method_1() [22:41] open the remote repo (or pethaps new-method-1 returns it too [22:42] do a missing-graph-chatter [ its in remote.py's fetch codepath, I think. I can help you, just not today, on this bit] [22:42] remote_repo.fetch(local_repo, search_result=result-of-missing-graph-chatter) [22:42] new_method_2() [22:42] so, 5 or 6 round trips [22:43] So basically a protocol extension to support repo-push directly? [22:43] yes [22:44] we designed to permit extensions [like this] - though this is one we do want the core to have eventually anyway [22:45] I get the idea, but compatibility with current versions is important to me (I don't want to install a cutting-edge bzr.dev on every remote machine). [22:46] Also, I just remembered, I try to use encrypted repositories when possible, and until bzr supports it directly, the best solution I've found is to use encfs+sshfs for the remote repo. [22:46] øshouldn't impact this [22:47] Well it does, since then the remote repository will look like a local one, and the protocol is not used. [22:48] The point of this all (if you're wondering) is to use bzr to back up everything in my home directory, using a possibly untrusted remote backup machine. [22:50] And if you don't want to trust the remote machine at all, I don't think that the remote server could do anything very sensible with the encrypted data in the repository, other than send it to the client. [22:50] oh [22:50] And that's what sshfs does already. [22:50] so if you're writing over vfs basically then it doesn't really matter; its going to slow no matter what ;P [22:50] Well yeah. [22:51] I also intend to use this to synchronize the home directory between trusted machines, so there the bzr protocol is relevant. [22:59] Anyone interested in this? [22:59] bzr: ERROR: Unprintable exception PermissionDenied: dict={'path': u'/srv/d_ubuntu-gandi/buildbot/.bzr', '_preformatted_string': None, 'extra': ": [Errno 13] Permission non accord\xc3\xa9e: '/srv/d_ubuntu-gandi/buildbot/.bzr'"}, fmt='Permission denied: "%(path)s"%(extra)s', error=UnicodeDecodeError('ascii', ": [Errno 13] Permission non accord\xc3\xa9e: '/srv/d_ubuntu-gandi/buildbot/.bzr'", 34, 35, 'ordinal not in range(128)') [23:01] sure [23:01] what bzr version ? [23:01] we've been fixing a bunch of issues like that in bzr.dev this last couple of weeks [23:01] 2.1.1, from the ppa [23:01] possibly a dupe [23:01] mgz: ^ ? [23:02] I don't think I'm in a position to test a newer dev version on this machine, unfortunately [23:03] thats ok [23:03] leave it with us [23:03] the underlying error is clear enough to you though ? [23:03] Yep [23:29] Hrm. [23:30] Same bzr version, 2.1.1, from the ppa, still seeing this error: [23:30] bzr: ERROR: Pack 'a60d0a0175e881a665d9d991452f2b79' already exists in GCRepositoryPackCollection(CHKInventoryRepository('file:///var/lib/buildbot/bigdog24-twisted/.bzr/repository/')) [23:31] thats interesting [23:31] are you pulling from two bzr-svn branches concurrently ? [23:31] There is definitely concurrency [23:31] can you do bzr dump-btree file:///var/lib/buildbot/bigdog24-twisted/.bzr/repository/pack-names [23:31] But it's probably from the same bzr-svn branch multiple times [23:32] has any of your tests reached a full checkout yet ? [23:33] dump-btree fails [23:33] generates a crash report [23:34] I dunno, hopefully. It's kind of hard to tell though. [23:34] the crash report will have a backtrace in it [23:34] Should I pastebin it, or file a bug report? [23:35] maybe both, anyway http://pastebin.com/aJV30gXW [23:35] please file a bug [23:36] thats unexpected, at best [23:40] This looks maybe related? https://bugs.launchpad.net/bzr/+bug/488607 [23:40] Launchpad bug 488607 in Bazaar "bzr dump-btree .cix fails: tuple index out of range (affected: 1, heat: 6)" [Low,Fix released] [23:41] I'm not sure how to see which release the fix has been included in, if any, though. [23:41] different index file [23:41] Okay [23:41] however, it would be in 2.1.2 I think [23:41] lets see [23:42] no, 2.2.0 [23:45] k, filed https://bugs.launchpad.net/bzr/+bug/584362 [23:45] Launchpad bug 584362 in Bazaar "Unhandled IndexError from 'bzr dump-btree .../repository/pack-names' (affected: 1, heat: 0)" [Undecided,New] [23:45] thanks [23:45] off to see more houses; ciao [23:46] Have fun [23:46] (but I don't really care about `dump-btree`, I want `bzr checkout ...` to work :( :( :( :( :( :( :()