[00:47] <spiv> poolie: if you're doing reviews, a review of http://bundlebuggy.aaronbentley.com/project/bzr/request/%3C20081027005419.GA20736%40steerpike.home.puzzling.org%3E would be good.
[00:47] <poolie> spiv, ok
[00:47] <poolie> am doing mail atm but will try to get to it
[01:58]  * spiv fires some branches to the mailing list and goes to lunch
[02:10] <pdf23ds> extmerge doesn't like me. :(
[02:10] <pdf23ds> Unable to load plugin u'extmerge' from u'C:/Program Files/Bazaar/plugins'
[02:12] <spiv> pdf23ds: try "bzr -Derror ...", or look in your bzr.log (see "bzr --version" for its location) to get a traceback that may explain why.
[02:12] <spiv> pdf23ds: if nothing else, the traceback will be useful in a bug report against extmerge :)
[02:13] <pdf23ds> OK, I think it might have something to do with abently's commit yesterday.
[02:13]  * spiv really goes to lunch now
[02:13] <pdf23ds> AttributeError: 'dict' object has no attribute 'register_lazy'
[02:14] <pdf23ds> But thanks for the tip.
[02:14] <pdf23ds> How to change the tree to an old rev?
[02:14] <pdf23ds> Just my luck to have a commit yesterday break me on a project that hadn't seen any commits since March.
[02:15] <pdf23ds> bzr up -r x?
[02:18] <pdf23ds> bzr revert -r x
[02:19] <pdf23ds> crazy australians. going to lunch at 9 PM.
[02:59] <pdf23ds> Anybuddy know how to resume when you get conflicts with the replay command?
[02:59] <pdf23ds> rebase-continue?
[03:07] <lifeless> pdf23ds: does the help tell you?
[03:07] <pdf23ds> bzr replay help is empty.
[03:07] <pdf23ds> and bzr rebase help doesn't say explicitly.
[03:07] <pdf23ds> I filed a documentation bug. :)
[03:08] <pdf23ds> In the meantime I guess I'll try and see what happens.
[03:13] <pdf23ds> It sort of acts like continue might do the right thing, tho -abort and -todo complain that you're not rebasing.
[03:14] <pdf23ds> Hmm. I didn't mess as much with Mercurial, but Bazaar seems much rougher-around-the-edges than Mercurial. But I still like it more.
[03:14] <pdf23ds> And OTOH, I just ran into a Subversion bug today with the latest version.
[03:15] <pdf23ds> (Couldn't merge a path with spaces over https protocol, works over svn protocol. Weird.)
[03:17] <lifeless> meep
[03:18] <lifeless> please file a bug
[03:18] <pdf23ds> About replay you mean? Where should I file that?
[03:19] <pdf23ds> https://bugs.launchpad.net/bzr-rebase/ ?
[03:20] <pdf23ds> (The merge thing was a SVN bug I just got hit by today.)
[03:20] <lifeless> about the https file with spaces thing
[03:21] <pdf23ds> yeah, that's svn.
[03:21] <pdf23ds> I had the same reaction.
[03:22] <lifeless> a bug on bzr-svn would be good for that; it may be something jelmer has msised
[03:23] <pdf23ds> It wasn't related to bzr, I mean.
[03:23] <pdf23ds> I work with SVN during the day, I'm converting my home projects to bzr.
[03:24] <pdf23ds> I won't be able to get my job using bzr until TortoiseBZR is mature, and until I convince my coworkers of the joys of branching.
[03:24] <jonoxer> Hi party ppl, I seem to be having some problems with interop between bzr v1.3.1 and v1.6.1 but only on some branches.
[03:25] <jonoxer> Most of our dev machines (and the repo) are on Hardy, with bzr 1.3.1. But on a certain big repo when I try a checkout with v1.6.1 (on Intrepid) it first complains about the server not understanding net protocol 3, then falls back to an earlier protocol, then sits for a minute or two, then bombs with a traceback like this:
[03:25] <jonoxer> File "/usr/bin/bzr", line 119, in <module>
[03:25] <jonoxer> sys.stdout.flush()
[03:25] <jonoxer> ValueError: I/O operation on closed file
[03:25] <jonoxer> ...which looks like a permissions problem, but I don't think it is
[03:25] <jonoxer> because checkouts work fine from Hardy machines
[03:26] <jonoxer> Any suggestions where to look? I tried with -Dhpss and it didn't provide any more detail
[03:28] <pdf23ds> Hmm. Maybe I don't need replay after all. Rebase might do the job.
[03:29] <lifeless> jonoxer: I don't think its interop
[03:30] <lifeless> jonoxer: 1.6 has some memory issues; try 1.8
[03:30] <jonoxer> lifeless: so maybe it's a problem with this particular client
[03:30] <jonoxer> lifeless: OK, I'll give it a shot
[03:32] <pdf23ds> Nope, rebase requires a common ancestor too.
[03:32] <pdf23ds> Revision 0 isn't considered a common ancestor, right?
[03:32] <pdf23ds> bzr branch A B -r 0 is the same as bzr init B?
[03:46] <lifeless> pdf23ds: nearly the same; branch will preserve the specific format whereas init will use the current default
[03:47] <lifeless> this only matters if either the source branch uses a very new feature, or you're interoperating with clients much older than yours
[03:53] <pdf23ds> lifeless: Well, it matters if you want to merge those branches later and find out it can't be done.
[04:01] <mkanat> Wow, bzr doesn't like it when my local commits are identical to upstream commits, and I do a "bzr up".
[04:01] <mkanat> Almost everything showed up as a conflict.
[04:02] <lifeless> mkanat: it shouldn't conflict there; unless you added files
[04:02] <jonoxer> mkanat: when you say "identical", do you mean the files actually are identical? As in the same checksums?
[04:02] <mkanat> jonoxer: No, I have local uncommitted changes in addition to the local commits.
[04:03] <mkanat> jonoxer: Also, the upstream files may have different permissions than my local files.
[04:03] <lifeless> permissions shouldn't be an issue there; its likely the local changes
[04:03] <lifeless> mkanat: but there are some things update does that make conflicts worse than they should be
[04:03] <mkanat> Yeah, that's probably what I'm running into.
[04:03] <mkanat> I was just surprised, didn't know if you guys knew it behaved that way.
[04:04] <mkanat> It's not that bad, I have my local changes as a patch elsewhere and I can just revert and re-apply them.
[04:13] <jonoxer> lifeless: I've just upgraded the problematic machine to bzr 1.8.1 (package from ppa) and it still bombs out on a checkout from the same repo
[04:13] <jonoxer> The error message is identical to with 1.6.1
[04:15] <jonoxer> The repo is shown as "Shared repository with trees (format: pack-0.92)" fwiw
[04:16] <jonoxer> The branch I'm checking out takes about 7.8G on disk, in case that's relevant
[04:19] <lifeless> jonoxer: hmm, perhaps we haven't landed the patch yet
[04:19] <jonoxer> lifeless: think it's worth trying 1.3.1 on that machine? That's what we're running on every other box, and it works fine
[04:20] <lifeless> jonoxer: I suspect it will work, yes
[04:21] <Rolly> I'm surprised bzr-loom isn't listed in http://bazaar-vcs.org/BzrPlugins
[04:22] <jonoxer> lifeless: it seems to be working (1.3.1 on Intrepid)
[04:25] <lifeless> jonoxer: the memory patch may not have landed; or it may be something else
[04:25] <lifeless> jonoxer: please file a bug
[04:39] <jonoxer> No problem, done (Bug #290558)
[05:42] <poolie> spiv, i agree with your comments on the hooks doc patch, so i'll apply them and merge it
[05:48] <spiv> poolie: great
[05:49] <poolie> i still find your nickname kind of bizarre :)
[05:50] <spiv> I should trawl a dictionary for more flattering four letter words that are unlikely to already be registered on popular services...
[05:50] <poolie> http://newmatilda.com/2008/01/17/attack-spivs
[05:50] <poolie> excuse the distraction
[05:50] <poolie> "They no longer sport pencil-line moustaches and porkpie hats, but their tell-tale signs give them away - their faux Italian pointy-toe shoes and the hairstyles sculpted into those absurd little tufts. "
[05:50] <poolie> indeed :)
[05:50] <spiv> Oh, I don't mind being distracted.
[05:51] <spiv> That's the problem ;)
[05:51] <poolie> (:
[05:53] <lifeless> night all
[05:55] <poolie> cheerio
[06:07] <poolie> spiv, can you look at my reply there quickly?
[06:07] <poolie> by quickly i mean 'it should not take long' not 'urgently'
[06:13] <spiv> poolie: the reply to the hook docs patch?  Looks good to me.
[06:13] <poolie> thanks
[06:13] <poolie> i'll send it in
[06:25]  * spiv -> yoga
[06:25] <spiv> (and hacking on the train)
[06:27] <poolie> i'm going to stop in a bit
[07:07] <vila> hi all
[08:15] <matkor> Hi ! Is there bzr-gtk release 0.96 ?  I see in maillist: Status: Fix Committed => Fix Released  Target: None => 0.96.0 but nothing about 0.96 on http://bazaar-vcs.org/bzr-gtk ..
[08:38] <vila> matkor: 0.96.0 is where the commits are made to *prepare* 0.96 :)
[08:39] <vila> matkor: i.e. the trunk, a specific branch will be made once 0.96 is out I think
[08:43] <matkor> vila: Ok.. so status (Fix Committed => Fix Released) change is misleading ? shoudl be left on Fix Committed ?
[08:44] <vila> matkor: committed means the fix is available somewhere, released means it's merged on trunk
[08:46] <matkor> ah ... thanks a lot for explanation, vila
[08:47] <vila> you're welcome and not the first one needing the explanation which indicates something should be changed though...
[09:28] <matkor> vila: Perhaps: Fix Commited ->  Fix merged mainline (or Accepted to next release) -> Fix Released (in official version) could  be more self-explenatory ...
[09:32] <vila> The last step means that releasing a version requires an additional step whereas the actual usage doesn't... Most of the users will update only when a new release is announced and are unaware of the subtle difference. I know there is work in progress to somehow automate that step but I forgot the details
[09:57] <Odd_Bloke> Presumably launchpadlib will allow it to be done (if it doesn't already).
[10:11] <thumper> Odd_Bloke: what was that?
[10:13] <Odd_Bloke> thumper: Marking bugs as Fix Released when an actual release happens.
[10:13] <thumper> Odd_Bloke: ah
[10:13] <Odd_Bloke> thumper: A better fix might be to fix the bug statuses though. ;)
[10:13] <thumper> heh
[10:14] <thumper> I'd say that a fix is "in progress" until it is landed on trunk
[10:14] <thumper> but hey, that's just me
[10:15] <thumper> I did manage to get an hg->bzr convert last night
[10:15] <thumper> just with "versioned directories" and "bzr shelve"
[10:15] <Odd_Bloke> Well, 'we' (i.e. bzr) do (In Progress) --submitted for review--> (Fix Committed) --merged--> (Fix Released).
[10:16]  * thumper nods
[10:18] <james_w> congratulations jelmer
[10:22] <thumper> lifeless: ping
[10:24] <jelmer> james_w, thanks, but with what ? :-)
[10:24] <james_w> heh :-)
[10:33] <Odd_Bloke> jelmer: I'm guessing for your AM report...
[10:33] <jelmer> ah, I wasn't aware that was announced someplace
[10:37] <Odd_Bloke> jelmer: debian-newmaint list.
[10:37] <LarstiQ> jelmer: you know https://nm.debian.org/nmstatus.php I guess
[10:37] <jelmer> Odd_Bloke, ah, thanks
[10:38] <jelmer> LarstiQ, I don't expect (hope?) James to check that on a daily basis :-)
[10:38] <LarstiQ> jelmer: James has resigned from DAM work afaik
[10:38] <LarstiQ> oh, james_W
[10:38] <LarstiQ> sorry :)
[10:38] <LarstiQ> jelmer: yeah, debian-newmaint
[10:39]  * LarstiQ takes this as a clear sign he should do something about breakfast
[10:48] <jelmer> LarstiQ, :-)
[10:49] <Odd_Bloke> jelmer: Also on a Debian related note, is there any documentation about how to use the pkg-bazaar repository?  I haven't been able to work it out, and I have an ITP for bzr-xmloutput sitting around.
[11:01] <jelmer> Odd_Bloke, basically, we have one repository on bzr.debian.org per package
[11:01] <jelmer> and then one branch per debian distribution (unstable, experimental)
[11:02] <jelmer> the branches are bzr builddeb branches, and should ideally contain the upstream source plus debian/ directory and any necessary changes
[11:34] <Odd_Bloke> jelmer: Bug 290664 is the one I promised to file a couple of days ago.
[11:41] <jelmer> Odd_Bloke, thanks, I hope I can have a look at it during the weekend
[11:42] <Odd_Bloke> jelmer: I have commit access, so if there's a way I can fix the repository using that I'd be happy to do so.
[11:46] <jelmer> Odd_Bloke, it should be possible to add a fix for this to bzr-svn, I just need some time to analyse what's going wrong
[11:47] <Odd_Bloke> jelmer: Cool, I really meant if a repository fix would be possible sooner, then I'd like to do that. :)
[13:03] <jelmer> beuno, loggerhead made it into experimental btw
[13:03] <beuno> jelmer, I saw!
[13:03] <beuno> I've been spying
[13:03] <beuno> thanks  :)
[13:03] <jelmer> :-)
[13:04] <jelmer> you're welcome
[13:04] <jelmer> I hope we can now help get it running on bzr.debian.org
[13:05] <beuno> oh, I didn't know they needed help
[13:05] <beuno> let me know if I can do anything at all
[13:07] <jelmer> will do
[13:07] <jelmer> thanks
[13:24] <davi_> what is the easiest way to search for a revision that removed a couple of lines?
[13:34] <EarthLion> hey can i revert a single file to a specify version using bzr revert -r 46 foo.py ?
[13:35] <pdf23ds> jelmer: abently's latest commit to extmerge breaks it on 1.8. I had to figure that out myself after branching.
[13:51] <jelmer> pdf23ds, ?
[14:53] <jam> vila: ping
[14:53] <vila> jam: pong
[14:53] <jam> hey, want to chat about the inventory stuff?
[14:53] <vila> sure
[14:53] <jam> I haven't heard any background to know if robert's idea is corerct
[15:28] <jam> hi asmodehn
[15:29] <asmodehn> hey hey jam ;-
[15:29] <asmodehn> ;)
[15:31] <jam> I'm a bit surprised you sent the .bzr.log to the list, but I suppose filenames aren't particularly private
[15:32] <jam> it does give me a little bit of insight into your game, though :)
[15:32] <asmodehn> oh did I ^^
[15:32] <jam> you certainly have a very *wide* tree (lots of files)
[15:33] <jam> approx 30k
[15:33] <asmodehn> nah I only sent it to you I think ;-)
[15:33] <asmodehn> dont scare me like that :-p hehe
[15:34] <asmodehn> anyway as you can see I dont use bazaar to store the source code, but rather distributing different version of it in differnt locations
[15:34] <asmodehn> which explains the size and the latency I am dealing with..
[15:34] <jam> asmodehn: you're right, I wasn't looking at the folder I was in
[15:35] <asmodehn> but yes lots of big files
[15:35] <asmodehn> the repository with working trees is now 24GB
[15:35] <asmodehn> so whenever I doa  branch it takes some time :-p
[15:36] <asmodehn> especially when it s remotely
[15:36] <asmodehn> I also have scripted the beginning of an automated framework on top of it, to keep different location in sync, with web interface, etc.
[15:36] <asmodehn> just a beginning :-p
[15:38] <asmodehn> also because of the size of it my disk is pretty full now, not sure I can do a lot of tests ... I am lacking backup space...
[15:39] <jam> well, if you want to try my "convert_to_dev2" script, you can actually just use a hardlink of the old repository
[15:39] <jam> and the only new data will be the indices
[15:39] <jam> which is small meg
[15:39] <jam> rather than 24GB
[15:40] <asmodehn> mmm ok.. no need to upgrade before at all right ?
[15:41] <asmodehn> ha yeah ok this is doing the upgrade... ok I ll do that then and let you know...
[15:42] <jam> so I should say maybe 10MB or so, but that is pretty trivial next to the 24GB. But I would only do it on a copy
[15:42] <jam> for now
[15:43] <jam> then again
[15:43] <jam> I could just write you the reverse script
[15:43] <jam> to convert them back
[15:43] <jam> not to mention my convert_to_dev2 script leaves them lying around, IIRC
[15:43] <asmodehn> thats good enough then, if I can put tehm back somehow ;-)
[15:43] <asmodehn> I have remote backups if needed
[15:44] <jam> yeah
[15:44] <jam> the script creates new indices
[15:44] <jam> and just moves the old one out of the way
[15:44] <jam> without deleting it
[15:44] <jam> I just verified that
[15:47] <asmodehn> "python convert_to_dev2.py" and thats it correct ?
[15:47]  * asmodehn just a bit scared :-p hehe
[15:49] <asmodehn> jam : ok running...
[15:49] <jam> you have to have "bzrlib" in your python install, but it sounds like you do
[15:49] <jam> I'm looking at it now
[15:49] <jam> just for your comfort
[15:49] <asmodehn> thanks ;-)
[15:50] <asmodehn> jam : ok done ^^ pretty fast
[15:51] <jam> yeah, that is all you needed to do
[15:51] <jam> asmodehn: again, rewriting 20MB isn't bad versus rewriting 24GB
[15:52] <asmodehn> ok but I changed only the repository I am branching towards, not the repository I am branching from..
[15:52] <asmodehn> do I need to do that too ?
[15:52] <asmodehn> yep :-)
[15:52] <jam> the one you are branching *from* is the important one
[15:52] <asmodehn> yeah I would have guessed so....
[15:52] <asmodehn> mmm hang on...
[15:52] <jam> I'm also not 100% sure if you need my prefetch code
[15:53] <jam> Are you comfortable running bzr from source instead of from an install?
[15:53] <jam> (it isn't very hard)
[15:53] <asmodehn> mmm it should be ok, but I really want to avoid to mess up my repositories too much
[15:53] <asmodehn> so I guess I ll try one thing at a time ;-)
[15:54] <asmodehn> but yeah I could do that...
[15:54] <jam> it doesn't change the disk formats at all
[15:54] <jam> *just* how we access it
[15:54] <jam> but start with this
[15:54] <jam> see what happens
[15:54] <asmodehn> k
[15:54] <jam> and then we'll try that as the next step
[15:55] <jam> also, can you run a quick command for me
[15:55] <asmodehn> sure
[15:55] <jam> find .bzr -name "*.?ix" -print0 | xargs -0 du -kshc
[15:55] <jam> I think that is right
[15:55] <jam> let me run it here
[15:56] <jam> yeah, that works
[15:56] <asmodehn> ok on which repo ?
[15:57] <asmodehn> the source one right ?
[15:57] <jam> the one you just upgraded
[15:57] <asmodehn> ah ok
[15:57] <jam> you could do the source as well
[15:57] <jam> I mostly just want to see how big your indexes were
[15:57] <jam> versus how big they are in btree format
[15:57] <asmodehn> oh ok
[15:58] <asmodehn> I l send hte result in email ot you ;-)
[16:01] <asmodehn> done
[16:02] <asmodehn> ah I have enough space on the disk of the source repo to back everything up :-) good news ;-)
[16:04] <asmodehn> backup everything ^^ sorry me english bad :-p
[16:04] <asmodehn> I m doing that now then I convert to dev2 as well so I can try the branch again...
[16:08] <dstanek> how do i revert a commit? is it the same way that you do it in subversion? merge the changes between new and old back into the working copy?
[16:08] <asmodehn> bzr uncommit is your friend ;-)
[16:09] <asmodehn> if you really want to lose your commit
[16:09] <jam> asmodehn: so from what I can see, your 31MB text index becomes 9.1MB in btree form
[16:09] <jam> and your 14MB => 4Mb
[16:09] <jam> MB
[16:09] <jam> which is good
[16:10] <asmodehn> yep wouldnt hurt ;-)
[16:11] <asmodehn> if we want to add log to the data transfer, I would need to run bzr from source and modify it right ?
[16:11] <dstanek> asmodehn: great thanks, i'll take a look
[16:11] <asmodehn> cause at the moment the log doesnt display nothing while the process seems to be stuck...
[16:11] <asmodehn> dstanek : no worries ;-)
[16:12] <jam> dstanek: you could also do "bzr merge -r 10..9" if you want to revert something old
[16:12] <jam> but yeah, 'bzr uncommit' is probably what you want
[16:12] <jam> if it was recent
[16:12] <jam> (as in the last commit)
[16:13] <jam> asmodehn: so the index seems to go from almost 70MB to down to 30MB, which is about what I would expect.
[16:13] <jam> Also, it does look like this repository could stand to have "bzr pack" run
[16:14] <jam> Though I'll mention that running it after the conversion means we don't have a simple way of bringing the indexes back
[16:14] <dstanek> it look like uncommit can also take a -r argument - will it work if you specify a revision that is not the last?
[16:14] <jam> dstanek: it will uncommit everything back to that revision
[16:14] <jam> not just the changes for that rev
[16:14] <jam> for something old
[16:14] <dstanek> ah i see
[16:14] <jam> you want "bzr merge -r REV..REV-1"
[16:14] <jam> sorry
[16:14] <jam> you want "bzr merge . -r REV..REV-1"
[16:14] <asmodehn> yeah I was thinking about that too... but I wonder if that might make one big pack, and therefore we might lose some details in debugging what is wrong in the data transfer
[16:14] <jam> the '.' is important
[16:15] <jam> asmodehn: it will create 1 big pack
[16:15] <jam> so we can look at that more later if you prefer
[16:15] <dstanek> jam: uncommit works for my immediate need
[16:16] <asmodehn> jam : also having one huge pack might not be good for me if there is no way to track the progress of the download...
[16:16] <asmodehn> I prefer downloading slowly but know that it s working ;-)
[16:17] <asmodehn> btw, same thing, is there an incremental pull / push ?
[16:17] <asmodehn> revision by revision ?
[16:17] <jam> asmodehn: 'bzr push -r X", "bzr push -r X+1"
[16:18] <asmodehn> yes but when I have 123 revision to push one by one because they are big and they might fail, I dont want to type that 123 times ;)
[16:18] <asmodehn> was thinking of a "bzr ipush -r REV..REV " that will push revision one by one
[16:19] <asmodehn> and if it breaks in the middle, the last push that worked is still there...
[16:19] <asmodehn> but maybe that belongs more in a script around bzr than bzr itself
[16:19] <asmodehn> jam : ... my backup copy on the source repository is still going...
[16:20] <jam> asmodehn: for i in `seq REV..REV`; do bzr push -r $i; done
[16:20] <jam> sorry
[16:20] <jam> for in `seq REV REV`; do echo $i;  bzr push -r $i; done
[16:20] <asmodehn> yeah thats what I meant ;-)
[16:21] <asmodehn> but you want to check for error and stuff, so I ll gues I ll write a script for that...
[16:21] <asmodehn> eventually
[16:21] <asmodehn> at the moment I am pulling revision 10 by 10 to know a bit where I am at...
[16:25] <jam> well, I'll mention that some of the data transferred is forgotted by the next copy
[16:25] <jam> so it will be slower
[16:27] <asmodehn> yep I start doing that when using 1.5, because for long transfer it used to error out
[16:28] <asmodehn> where few by few it was working
[16:28] <asmodehn> and since I upgraded to 1.8 it doesnt error as much which is good ;-)
[16:28] <asmodehn> but it seems to get stuck longer for smoem reason :-s
[16:32] <jam> 1.8 will switch to grabbing whole indexes from time to time, which might be what you are seeing
[16:32] <jam> 1.5 would only grab incrementally
[16:32] <jam> but could end up grabbing 2x the inventory under certain circumstances
[16:33] <jam> the dev2 format solves both of those issues
[16:33] <asmodehn> cool ;) good to know
[16:33] <asmodehn> backup copy almost finished...
[16:55] <jam> asmodehn: how's it going?
[16:55] <asmodehn> jam : branching now...
[16:55] <asmodehn> conversion went without problems
[16:55] <jam> you're using -Dhpss for me, right?
[16:56] <asmodehn> yep
[16:57] <asmodehn> but looking at the log... the copy of the index is fine...
[16:58] <asmodehn> looks like the copy of the pack doesnt generate a lot of transfer on teh network though.. I am a bit confused...
[16:59] <jam> copying the pack should be saturating your network
[16:59] <asmodehn> my pick is 1.2Kbps now ...
[16:59] <jam> are you sure about that?
[16:59] <asmodehn> looking at iftop onthe source network now..
[17:00] <asmodehn> source server I mean
[17:00] <asmodehn> it was high up during indexes
[17:00] <jam> my only other guess is that you are running into swap issues on the source
[17:00] <asmodehn> oh^
[17:00] <asmodehn> mmm I ll check my RAM..
[17:00] <jam> as we are trying to copy 20GB across
[17:01] <asmodehn> nope everything looks fine
[17:01] <asmodehn> source :
[17:01] <asmodehn> Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
[17:01] <asmodehn> Mem:   3366488k total,  3221756k used,   144732k free,     1672k buffers
[17:01] <asmodehn> Swap:  2650684k total,   702536k used,  1948148k free,   558160k cached
[17:01] <asmodehn> dest :
[17:01] <asmodehn> Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
[17:01] <asmodehn> Mem:   6110456k total,  5198228k used,   912228k free,    78484k buffers
[17:01] <asmodehn> Swap:  1951888k total,       72k used,  1951816k free,  4550312k cached
[17:02] <asmodehn> and I am still branching -r1 from a repo on source_dev2
[17:02] <asmodehn> \ [[17:03] <jam> 700MB in swap seems like a lot to me
[17:03] <jam> but I don't know what load you run on that server
[17:03] <asmodehn> well these server are running a bunch of other things ^^
[17:04] <jam> so you are still seeing very little data being copied, right?
[17:04] <asmodehn> but that swap used is not increasing right now anyway
[17:04] <asmodehn> no transfer ont eh network
[17:04] <asmodehn> well very little
[17:04] <asmodehn> and with -Dhpss
[17:04] <jam> you might try top of the running process as well
[17:04] <jam> it is also possible that your disks are being heavily loaded ATM
[17:04] <asmodehn> log stuck at 85.410     result:   ('readv',)
[17:04] <asmodehn> 105.947                1099613 body bytes read
[17:04] <asmodehn> 108.023  hpss call w/readv: 'readv', '/home/autobzr/deployBZR_dev2/.bzr/repository/packs/f65de233ce31cbdace7370611f49937c.pack'
[17:04] <asmodehn> 108.024                11675 bytes in readv request
[17:05] <jam> asmodehn: can you send  the 'ls' results for the remote repository?
[17:06] <jam> I would assume that is a big file, but it isn't in the listing you sent
[17:06] <jam> (since that is the local repo)
[17:08] <asmodehn> this pack is 1.1GB
[17:10] <asmodehn> source :
[17:10] <asmodehn>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
[17:10] <asmodehn> 12072 autobzr   20   0 1925m 1.9g 2404 S    0 58.4   0:03.46 bzr
[17:10] <asmodehn> dest:
[17:10] <asmodehn>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
[17:10] <asmodehn> 25889 autobzr   21   0  228m 159m 3956 S    0  2.7   0:06.94 bzr
[17:11] <jam> so it is consuming 1.9GB, which may not be terrible, but isn't a great sign
[17:11] <asmodehn> yep
[17:11] <jam> mostly for when you hit the 20GB files
[17:11] <jam> It is possible that is taken up by other info
[17:11] <jam> but I'm expecting it is the actual data that will be transmitted
[17:12] <jam> asmodehn: I think it is going to have serious problems soon. I'm seeing:
[17:12] <jam>         backing_bytes = ''.join(bytes for offset, bytes in
[17:12] <jam>             self._backing_transport.readv(self._relpath, offsets))
[17:13] <jam> which means... "read the requested bytes, and buffer it into a single string"
[17:13] <jam> and return that
[17:13] <asmodehn> ah...
[17:13] <jam> aka, buffer 20GB in RAM before sending it over the wire
[17:13] <asmodehn> that will not work :-p
[17:15] <jam> so... the specific issue is that since 1.6 we are able to request the streams for multiple files simultaneously
[17:15] <jam> in 1.5 we could only request one file at a time
[17:15] <jam> which makes things way faster in most cases
[17:15] <jam> but in your case
[17:15] <jam> it means we end up requesting 90% of the pack file in one go
[17:15] <jam> and you end up dying
[17:15] <asmodehn> argh
[17:16] <asmodehn> yep...
[17:16] <jam> we could probably work around this a little bit in the client
[17:16] <jam> by capping the size of a readv request
[17:16] <asmodehn> but that also means there no easy perfect solution right ?
[17:16] <asmodehn> ie, how much oyu are capping will depend on the RAM size, the badwith, etc.
[17:17] <jam> yeah
[17:17] <jam> but something like 500MB is generally a safe bet
[17:17] <jam> also, could you try this over sftp
[17:17] <jam> ?
[17:17] <jam> I realize some bits will be slower
[17:17] <asmodehn> yep
[17:17] <jam> but I think it the SFTP spec requires that we transfer in 32kB chunks
[17:17] <jam> so it won't actually hit this
[17:17] <asmodehn> oh ^^
[17:17] <jam> You don't have to
[17:17] <asmodehn> worht a shot ;-)
[17:18] <asmodehn> it s ok ;-)
[17:18] <jam> but it would be worth having the data point
[17:18] <asmodehn> yep
[17:18] <asmodehn> and I am pretty desperate atm to be honest... if it doesnt work I ll go back to 1.5 and transfer revision by revision to avoid hitting errors too often...
[17:18] <jam> I was in this code recently, and I'm pretty sure the client still spools up the whole request as well
[17:20] <asmodehn> here we go sftp...
[17:26] <jam> I may have a quick fix on the readv stuff, I'll let you know
[17:26] <asmodehn> k
[17:26] <asmodehn> sftp still copying inventory text...
[17:27] <jam> asmodehn: I'll just mention that bzr hasn't really been designed around handling 10GB trees
[17:27] <jam> We'll try to accommodate as we can
[17:27] <asmodehn> yeah I would guess so ;)
[17:27] <asmodehn> no worries
[17:27] <jam> but it isn't a base design criteria
[17:28] <asmodehn> I am just using it because it s better than copying 4GB zip files around evrytime there is just a little difference...
[17:28] <jam> have you considered unison/rsync?
[17:28] <asmodehn> ie easy way to get just patches transparently
[17:28] <asmodehn> mmm dont know about unison...
[17:28] <asmodehn> let me google that ;-)
[17:35] <asmodehn> mmm looks interesting... however I need more than what it does though, such as a proper SCM who let me go back in revision history ...
[17:35] <asmodehn> I feel like I need a BZR multisite :-p
[17:35] <asmodehn> lol
[17:38] <asmodehn> mmm sftp doesnt load the source server like bzr+ssh did...
[17:38] <asmodehn> and now there is still traffic on the pipe... 200 - 300 Kbps
[17:38] <jam> yeah
[17:38] <jam> again, requests are made at 32kB per hunk
[17:38] <jam> and the sftp code was updated to stream that back to the next layer as well
[17:39] <jam> while the bzr+ssh code does a lot of "finish reading the response, then return"
[17:39] <asmodehn> k
[17:40] <asmodehn> so whats the plan from now .. ? I guess I ll use sftp from now on, until the bzr+ssh code is improved to handle big packs ?
[17:42] <asmodehn> and I keep using my old Knits repo until the dev2 format is finalized, tested, and announced ?
[17:44] <jam> well, I have a patch for it to be released in 1.9, which may happen by this Friday
[17:44] <jam> so you won't exactly be waiting long
[17:44] <jam> can you send me the bzr logs with the dev2 format?
[17:44] <jam> I'm just curious how much of an improvement we see
[17:44] <jam> (if any)
[17:45] <asmodehn> sure ;-)
[17:45] <asmodehn> just the destination though... the source is a bit loaded by other bzr processes running :-s
[17:45] <asmodehn> ok ?
[17:45] <asmodehn> would htat be enough for you ?
[17:46] <asmodehn> mmm hang on maybe I can get the source one too if you need actually, the process was stuck since two days ago :-p
[17:47] <jam> no, just the local one is fine
[17:47] <jam> I don't think you were running -Dhpss on the server anyway
[17:48] <jam> I'm not specifically interested in the remote one either
[17:48] <asmodehn> yeha true, no Dhpss..
[17:48] <asmodehn> but but btu ...
[17:48] <asmodehn> my .bzr.log is now only 28K :-s
[17:48] <jam> .bzr.log.old
[17:48] <asmodehn> haaa :-)
[17:49] <jam> I believe we keep 1 level of .bzr.log
[17:49] <jam> just to prevent it from consuming lots of disk space
[17:49] <jam> at somewhere around 4MB we copy it to .old
[17:49] <jam> and start fresh
[17:50] <asmodehn> k I am bziping both now... I ll send that in your email
[17:50] <asmodehn> thanks a lot for the help by the way ;-)
[17:50] <jam> np
[17:51] <asmodehn> I ll get 1.9 on my etch next week I guess then ;-)
[17:51] <asmodehn> let me know if you want me to test something ;-)
[17:52] <jam> asmodehn: well, I'll likely have some readv code that I might have you play with
[17:52] <jam> I'm still putting together a "summary of what is happening" to the mailing list
[17:52] <asmodehn> cool ;-) thanks ;-)
[18:02] <jam> asmodehn: can you grab a branch of bzr.dev to start with?
[18:05] <asmodehn> yep
[18:05] <asmodehn> url ?
[18:06] <asmodehn> got it
[18:07] <jam> 'bzr branch lp:bzr' should be fine
[18:07] <asmodehn> branching from bazaar-vsc.org/bzr/bzr.dev atm...
[18:08] <jam> same thing
[18:09] <jam> lp:bzr is a mirror of that url
[18:09] <jam> just a lot shorter to type
[18:09] <jam> :)
[18:09] <asmodehn> hehe cool, I ve never used lp before though :-p I more of a SVN / BSD guy :-p
[18:09] <asmodehn> dev C++ too not much python
[18:10] <asmodehn> but i ve heard it s quite siple, never had time to get my head around it though..
[18:10] <jam> yeah, I came from C++ => python
[18:11] <jam> all depends on what you are doing
[18:11] <asmodehn> yep
[18:12] <jam> one weird thing, the last .bzr.log seems to have a different bandwidth
[18:12] <jam> like 100kB/s rather than 200kB/s
[18:12] <jam> is that real?
[18:12] <asmodehn> huh ?
[18:12]  * asmodehn is trying to think of what happened 2 minutes ago...
[18:13] <asmodehn> any clue about where in the log that changes ?
[18:14] <jam> http://dpaste.com/87602/
[18:15] <asmodehn> weird...
[18:15] <jam> though I also see:
[18:15] <jam> 43.321     result:   ('ok',)
[18:15] <jam> 391.397                22190509 body bytes read
[18:15] <jam> which I believe is about 60kB/s
[18:16] <jam> so your bandwidth seems rather variable
[18:16] <asmodehn> the locations are quite far away from each other, and a lot of other thing might happen on the network, so that wouldnt surprise me
[18:18] <asmodehn> I cant recall in which order I stopped / restarte the differnet bazaar processes on hte machines in the last hour to know if it affected something or not...
[18:18] <jam> I was just trying to figure if the new format was saving you anything, but it looked slower at first glance
[18:18] <jam> but if your bandwith can vary 2x
[18:18] <asmodehn> I see
[18:18] <jam> I can say it is copying less data
[18:19] <jam> do you have bzr.dev yet?
[18:19] <asmodehn> Branched 3806 revision(s).
[18:19] <jam> asmodehn: ok, try doing "bzr pull --ovewrite http://bzr.arbash-meinel.com/branches/bzr/1.9-dev/remote_readv_sections"
[18:19] <asmodehn> ok ;-)
[18:20] <jam> and then run
[18:20] <jam> make
[18:20] <jam> and then you should be able to run "path/to/bzr branch -r1 -Dhpss bzr+ssh://..." for me
[18:21] <asmodehn> not a branch... maybe  url rpbolem ??
[18:21] <jam> oh, and use "-Dindex" if you would
[18:21] <asmodehn> k
[18:21] <jam> ah just a sec
[18:21] <jam> I just forgot to publish it :)
[18:21] <asmodehn> hehe :-p
[18:21] <jam> done
[18:21] <jam> try again
[18:21] <asmodehn> pulling...
[18:22] <asmodehn>  M  NEWS
[18:22] <asmodehn>  M  bzrlib/help_topics/en/hooks.txt
[18:22] <asmodehn>  M  bzrlib/transport/__init__.py
[18:22] <asmodehn>  M  bzrlib/transport/remote.py
[18:22] <asmodehn>  M  doc/en/user-guide/hooks.txt
[18:22] <asmodehn> All changes applied successfully.
[18:22] <asmodehn> Now on revision 3806.
[18:24] <jam> asmodehn: sounds right
[18:24] <asmodehn> huh ? 3806 -> pull -> 3806 ?
[18:24] <jam> different tips
[18:24] <asmodehn> different one then ;-)
[18:24] <asmodehn> k
[18:24] <asmodehn>  Change the default readv() packing rules.
[18:24] <asmodehn> ah make... what do I need to run that ?
[18:25] <jam> gmake, gcc, python-pyrex probably
[18:25] <asmodehn> k
[18:25] <jam> I could give you the .c files if you don't want to install python-pyrex
[18:25] <asmodehn> nah it s ok I can
[18:26] <jam> you don't *have* to, because bzr can fall back to pure python
[18:26] <jam> but certain ops are slower
[18:26] <asmodehn> k
[18:26] <asmodehn> I ll go package hunting ;-)
[18:26] <asmodehn> brb
[18:26] <jam> though not at the scale of copying 20GB across a remote network
[18:26] <jam> asmodehn: so if it is difficult, you can skip it
[18:35] <asmodehn> k bunch of dev packages + make + bunch of warnings, but it seems fine ...
[18:40] <jam> another small advantage of the patch
[18:40] <jam> if it works, you should get a little bit more progress
[18:40] <jam> well, the spinner should spin more often
[18:40] <asmodehn> k
[18:40] <jam> I don't think it will log much more
[18:40] <asmodehn> just the Dindex I guess...
[18:41] <asmodehn> well hte start was pretty fast
[18:42] <asmodehn> I guess dev2 indexing improvements ?
[18:42] <jam> probably
[18:42] <asmodehn> now it seem stuck as usual, but my network usage is around 1.2Mbps :) I feel better
[18:42] <jam> :)
[18:42] <jam> can you get to the remote server to check the mem usage?
[18:43] <jam> It should certainly be << 1G
[18:43] <asmodehn> yep
[18:44] <asmodehn>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
[18:44] <asmodehn>  6942 autobzr   15   0  124m 120m 2404 S    0  3.7   0:00.39 bzr
[18:44] <jam> 120M sounds a lot better than 1.9G
[18:45] <asmodehn> yep it does ;-)
[18:47] <asmodehn> the data transfer rate oscllate between 900Kbps and 1.2Mbps ;-) nice nice...
[18:48] <asmodehn> note the spinning bar doesnt seem to turn a lot in between different readv though...
[18:49] <asmodehn> 91.631     result:   ('readv',)
[18:49] <asmodehn> 534.986                52094902 body bytes read
[18:49] <asmodehn> in between the bar didnt move
[18:50] <jam> I would expect the readv to request data, then when it comes back it will spin, then it will request more, back and forth
[18:51] <asmodehn> mmm
[18:51] <jam> the actual transfer code still buffers stuff
[18:51] <asmodehn> I dont see the bar moving while it s transferring data through network...
[18:51] <jam> it is just that inbetween we are smarter about it
[18:51] <asmodehn> yeah probably
[18:51] <asmodehn> k
[18:51] <jam> well, we make smaller requests
[18:51] <asmodehn> but anyway much better now ;)
[18:51] <jam> can I get the .bzr.log for it?
[18:52] <asmodehn> sure ... I ll  put that in a mail ;-)
[18:52] <jam> also, I'm going to have 1 more version for you to test in a sec
[18:52] <asmodehn> sure
[18:52] <asmodehn> I ll stop that now then
[18:52] <asmodehn> and send hte mail k ?
[18:53] <jam> yeah
[18:53] <jam> ok, do another "pull --overwrite ..."
[18:53] <jam> (you can use --remember if you don't want to keep typing the URL)
[18:57] <asmodehn> it s ok I like my shell history ;-)
[18:59] <asmodehn> building extension modules.
[18:59] <asmodehn> python setup.py build_ext -i
[18:59] <asmodehn> Cannot build extension "bzrlib._dirstate_helpers_c" using
[18:59] <asmodehn> your version of pyrex "0.9.4.1". Please upgrade your pyrex
[18:59] <asmodehn> install. For now, the non-compiled (python) version will
[18:59] <asmodehn> be used instead.
[18:59] <asmodehn> running build_ext
[18:59] <asmodehn> huh ?
[19:00] <asmodehn> ah but you just changed the transport in bzr lib, so that pyrex stuff shouldnt matter I guess
[19:00] <jam> you shouldn't actually have anything to do there, but whatever
[19:00] <jam> asmodehn: right pyrex 0.9.4.1 has a known bug
[19:01] <jam> so we don't compile the extension
[19:01] <jam> because when we do it segfaults
[19:01] <asmodehn>  ah ok ^^ just hte version in my etch packages :-s
[19:01] <asmodehn> anyway I ll branch -r -Dhpss -Dindex again...
[19:01] <jam> I really like ~line 871 in bzr.log:
[19:01] <jam> 25.841  expanding: 5d3c775ea20da5c234f29fb643c5d8df.tix	offsets: [13, 14, 15 ...
[19:01] <jam> 25.841    not expanding large request (1424 >= 16)
[19:01] <jam> with 1424 offsets in there :)
[19:02] <asmodehn> heh :p
[19:02] <jam> interestingly for this specific branch
[19:02] <jam> the prefetch doesn't help much
[19:02] <jam> we are either just reading a tiny bit
[19:02] <jam> or reading a huge amount at once anyway
[19:03] <asmodehn> oh added some debug message did you ;)
[19:03] <jam> yeah
[19:03] <jam> -Dindex is that part
[19:03] <jam> but I don't think it is in 1.8
[19:03] <jam> well, I *know* it isn't in 1.8
[19:04] <jam> as it just landed in bzr.dev 3805
[19:04] <asmodehn> mmm error
[19:05] <asmodehn> bzr: ERROR: exceptions.KeyError: 1566
[19:05] <asmodehn> any idea ?
[19:06] <jam> quick guess is that it doesn't like the partial readv I worked on, let me look a bit
[19:06] <asmodehn>   File "/home/autobzr/bzr.dev/bzrlib/btree_index.py", line 1046, in iter_entries
[19:06] <asmodehn>     node = nodes[node_index]
[19:06] <jam> it is possible that a page used to be cached, so we didn't request it
[19:06] <jam> but when we wanted to use it
[19:06] <jam> it was gone
[19:07] <jam> asmodehn: can you send that log?
[19:07] <jam> Looking at the code it should always have those keys
[19:07] <jam> but I might have a small bug in the new code
[19:07] <asmodehn> yep I ll do that
[19:07] <jam> key 1566 means we are requesting really far down in the index
[19:08] <jam> but that still shouldn't be failing
[19:08] <jam> before the last pull, the cap was at 50MB, so it probably never hit it with your indexes
[19:08] <jam> but I dropped it to 5MB
[19:08] <jam> and that may have triggered something
[19:08] <jam> since that index is 5.8MB in size
[19:11] <asmodehn> sent ;-)
[19:11] <jam> you're clogging my inbox :)
[19:11] <asmodehn> hehe sorry :-p
[19:13] <jam> is that one very big?
[19:14] <asmodehn> no last one was 15Ko
[19:14] <asmodehn> KB
[19:14] <jam> weird, usually your mail gets here faster
[19:15] <asmodehn> my gmail said sent 4 minutes ago
[19:15] <asmodehn> mail is not that big
[19:16] <asmodehn> starts with "Logs with second patch :"
[19:16] <asmodehn> maybe some of your client / server thinks I am a spammer :-p
[19:16] <jam> well all your others came through fine
[19:16] <jam> I can check my spam folder
[19:17] <jam> there it is
[19:17] <jam> just took a bit
[19:17] <asmodehn> ;-)
[19:20] <jam> ah, I see the problem
[19:20] <jam> just a sec
[19:20] <jam> (it was popping a node off 2x)(
[19:20] <jam> so it would skip that at a boundary point
[19:21] <asmodehn> he ;) no rush ;)
[19:24] <synic> is there any interest in something like gitosis for bzr?
[19:24] <jam> asmodehn: ok, pull up and try the new version
[19:25] <jam> synic: there is contrib/bzr_access
[19:25] <synic> ok
[19:25] <jam> it is probably not as complete as gitosis, but it describes how to set up multiple ssh keys to access one account, etc.
[19:26] <jam> asmodehn:  3811 refactor for clarity
[19:26] <jam> though 3810 is the important change
[19:27] <asmodehn> mmm
[19:27] <asmodehn> sorry :p
[19:27] <asmodehn>   File "/home/autobzr/bzr.dev/bzrlib/transport/remote.py", line 346, in _readv
[19:27] <asmodehn>     next_offset = [cur_offset_and_size.next()]
[19:27] <asmodehn> NameError: global name 'cur_offset_and_size' is not defined
[19:27] <jam> stupid bugs
[19:27] <asmodehn> hehe ;)
[19:28] <jam> asmodehn: this time I double checked and ran the test suite, 3812
[19:28] <jam> I really should put the new code under test
[19:28] <jam> but I wanted to make sure it was worth the effort
[19:28] <asmodehn> k ;) no worries let me know when I should pull and test ;)
[19:28] <jam> now
[19:29] <asmodehn> roger ;-)
[19:30] <asmodehn> branching...
[19:31] <jam> asmodehn: so if I follow the logs correctly, the old repository was taking 900s to get to the point where it could start text.pack data. The last test showed you hit that at 90s
[19:31] <jam> that seems.... way to good to be true
[19:31] <asmodehn> mmm
[19:31] <asmodehn> k error again...
[19:32] <jam> what is the error now?
[19:32] <asmodehn>   File "/home/autobzr/bzr.dev/bzrlib/btree_index.py", line 1050, in iter_entries
[19:32] <asmodehn>     node = nodes[node_index]
[19:32] <asmodehn> KeyError: 1566
[19:32] <asmodehn> 81.151  return code 4
[19:32] <jam> hm... same error
[19:32] <asmodehn> yeh...
[19:32] <asmodehn> and about the time to arrive to text.pack yes it s been really fast...
[19:32] <jam> can you send that log again
[19:33] <jam> I want to check something
[19:35] <jam> asmodehn: bah, I know the problem
[19:36] <jam> asmodehn: 3813 ready for you to test
[19:37] <asmodehn> log sent anyway ^^
[19:37] <asmodehn> I ll pull that ;)
[19:39] <asmodehn> branching...
[19:40] <jam> asmodehn: just as a point of comparison, could you try to branch the original repo again as well?
[19:40] <asmodehn> ?
[19:40] <jam> just to see if the 900s => 90s is real
[19:40] <jam> use my code
[19:40] <asmodehn> ah the one not dev2 correct ?
[19:41] <jam> right
[19:41] <jam> I would like you to do both
[19:41] <jam> so I can compare
[19:41] <jam> the 900s may have been a fluke
[19:41] <asmodehn> mmm I still have hte remote knit and the remote dev2
[19:41] <jam> or may be real
[19:41] <asmodehn> but the local ( destination ) is only dev2 now
[19:41] <jam> asmodehn: that is fine
[19:41] <jam> I'm more concerned about remote
[19:41] <asmodehn> maybe 900 is when I lost patiance and stopped it because it was sending nothing ?
[19:41] <asmodehn> k then ;-)
[19:42] <asmodehn> I can compare both , and hten send you the log...
[19:42] <asmodehn> just branching the same stuff from both knit and dev2 repo...
[19:43] <asmodehn> dev2 -> dev2 is now at 303 seconds...
[19:43] <asmodehn> is that enough to compare ?
[19:43] <asmodehn> should I stop that one and do the old one just after ?
[19:44] <asmodehn> hte old knit repo I mean
[19:44] <asmodehn> knit -> dev2
[19:45] <jelmer> Odd_Bloke, ping
[19:47] <asmodehn> jam : ok I saw the line you re talking about around 90s... I ll cancel my branching now from dev2 and start to branch from knit again
[19:48] <asmodehn> jam: argh IncompatibleRepositories: different rich-root support
[19:48] <jam> thats strange
[19:49] <asmodehn> one weird thing : KnitPackRepository [...]/.bzr/repository/
[19:50] <asmodehn> RemoteRepository [...]/.bzr/
[19:51] <asmodehn>   File "/home/autobzr/bzr.dev/bzrlib/repository.py", line 2506, in _assert_same_model
[19:51] <asmodehn>     "different rich-root support")
[19:51] <jam> yeah, I understand the code path. I just don't know why it would think that.
[19:51] <jam> just a sec
[19:51] <asmodehn> sure :)
[19:51] <jam> can you get to the remote machine?
[19:51] <asmodehn> yep
[19:51] <jam> what does "bzr info" say
[19:52] <asmodehn> Shared repository with trees (format: rich-root-pack)
[19:52] <jam> oj
[19:52] <jam> ok
[19:52] <asmodehn> local says : Shared repository with trees (format: development2)
[19:53] <jam> so my upgrade hack turned it into something closer to "pack-0.92" without rich-root
[19:53] <jam> which doesn't matter for testing, except for when you go between formats
[19:53] <jam> no big deal
[19:53] <jam> I'll get you something you can use
[19:54] <asmodehn> mmm what abotu reverting my local repo to the old knitpack now that we have a answer for the slowliness problem ?
[19:54] <asmodehn> I can wait a bit before using the new dev2 repo format...
[19:55] <jam> sure
[19:55] <jam> if you go to the repo
[19:55] <jam> you can see files/directories that end in -gi
[19:55] <jam> just move the existing file/dir to -bi
[19:55] <jam> and then mv -gi => normal
[19:55] <jam> asmodehn: does that make sense?
[19:56] <asmodehn> yep
[19:56] <asmodehn> format and indexes
[19:56] <asmodehn> and packnames
[19:56] <jam> right
[19:57] <asmodehn> btw is there any doc around about making the repo smaller... some kind of good practice ?
[19:57] <asmodehn> I am wondering if it gets rid of the packs that are not useful anymore, and that kind of stuff..
[19:59] <asmodehn> ok reverting to rich-root-pack done
[19:59] <asmodehn> now branching again from knitrrp to knitrrp to see how long it takes before we hit the .pack transfer...
[20:00] <jam> sure
[20:00] <jam> I think you'll still need to use my code
[20:00] <jam> since otherwise you'll hit the wall again
[20:00] <asmodehn> yep I m using your version of bzr
[20:00] <asmodehn> with the debug and all the stuff ;-)
[20:04] <dobey> is there any way to see the status of a submitted merge request for the bazaar project?
[20:05] <jam> dobey: http://bundlebuggy.aaronbentley.com ?
[20:05] <jam> possibly under Pending, or under Merged
[20:06] <asmodehn> jam : first readv on pack at 296 secs...
[20:06] <dobey> ah
[20:06] <jam> Or if you are on the mailing list, there will be a link to the merge request
[20:06] <jam> which is a stable URL
[20:06] <dobey> yeah, i'm not on the list
[20:06] <jam> asmodehn: so that is 296 => about 90?
[20:06] <jam> log files please :)
[20:06] <asmodehn> I remember I saw a 92
[20:06] <asmodehn> yep they re coming ;-)
[20:07] <jam> asmodehn: oh and the output of: du -ksh .bzr/repository/indices ; du -ksh .bzr/repository/indices-bi
[20:07] <asmodehn> k
[20:07] <jam> actually 'du -kshc' if you don't mind
[20:07] <asmodehn> sure ;)
[20:08] <jam> so I get the impression your "latency" bound is actually
[20:09] <jam> 1) Bandwidth limited for indexes
[20:09] <jam> followed by
[20:09] <jam> 2) latency bound while the server queues up GB of data to return
[20:09] <jam> though not really network latency
[20:09] <jam> more, "takes too long and uses all of swap"
[20:11] <asmodehn> yep
[20:11] <asmodehn> the uses all of swap appeard only when I upgraded from 1.5
[20:11] <asmodehn> before it used to transfer then error
[20:11] <asmodehn> so I was transferring few revision at a time and that went fine...
[20:11] <asmodehn> but recently I think the data became too huge
[20:12] <jam> note that my change isn't going to make it work if you have a single file that is too big
[20:12] <jam> like a single 600MB file is likely to clog things up
[20:12] <asmodehn> yep and thats fine ;-)
[20:12] <jam> but it looks more like you have *lots* of moderately sized files
[20:12] <asmodehn> now that it doesnt error anymore on a long transfer ;-)
[20:13] <asmodehn> yep
[20:13] <jam> asmodehn: well, you haven't tested a transfer to completion yet, have you?
[20:14] <jam> also, you say "first readv on pack" but is that after reading the text indexes?
[20:14] <jam> or is that reading the inventory
[20:14] <jam> there are a few different stages
[20:14] <jam> judging by your times, it is after reading .tix
[20:14] <jam> I just wanted to make sure
[20:15] <asmodehn> hehe true havnt tested a transfer till the end yet..
[20:15] <asmodehn> about the stage before pack I am not sure... you got the log in your email now ;-)
[20:16] <jam> asmodehn: why that I do :)
[20:16] <asmodehn> hehe :-p but I think it s after hte text index yes ;-) remembering what the progress bar was telling me
[20:20] <asmodehn> jam: so I guess we re done for today are we ? I ll keep using sftp waiting for the awesome new version and new repository format ;)
[20:21] <asmodehn> I am getting hungry hehe
[20:21] <jam> I think we're done
[20:21] <jam> enjoy your meal
[20:21] <asmodehn> thanks ;-)
[20:22] <asmodehn> and thanks a lot for the help ;) ill go eat and then I clean all remains of my experiments ;)
[21:04] <jam> dobey: your patch is here, by the way: http://bundlebuggy.aaronbentley.com/project/bzr/request/%3C1225157843.25420.7.camel%40lunatari%3E
[21:05] <dobey> yeah, i found it
[21:05] <dobey> thanks
[21:20] <ktenney> Howdy, I'm trying to retrieve the commit message for a revision
[21:20] <ktenney> it looks like lf = log.LineLogFormatter(file_handle)
[21:20] <ktenney> m = lf.message(rev)
[21:20] <ktenney> should work, but if so, I haven't discovered what rev should be
[21:24] <ktenney> rev from revno, rev = b.last_revision_info() doesn't work
[21:25] <james_w> ktenney: you need a Revision, not a revision id
[21:25] <james_w> repository.get_revision(rev_id) might be it
[21:25] <james_w> revno, rev_id = b.last_revision_info()
[21:26] <james_w> if you don't need the revno then just just b.last_revision()
[21:38] <ktenney> james_w: so I need to look at repository.Revision(self, _format, a_bzrdir, control_files, _revision_store, control_store, text_store).get_revision(rev_id) ?
[21:39] <james_w> ktenney: no, I don't think so
[21:39] <ktenney> james_w: revision commit messages aren't fetchable from a working tree?
[21:39] <ktenney> I haven't found any shortcut
[21:40] <james_w> wt.branch.repository.get_revision(rev_id)
[21:42] <lifeless> ktenney:
[21:42] <lifeless> t = bzrlib.workingtree.WorkingTree.open('.'); t.lock_read(); rev = t.branch.repository.get_revision(t.last_revision())
[21:43] <james_w> msg = rev.message as well I think
[21:43] <james_w> morning lifeless
[21:43] <lifeless> ktenney: we use factory methods a lot in bzr, because the library supports working directly with multiple VCS systems
[21:43] <lifeless> hi james_w
[21:43] <lifeless> james_w: how goes $stuff
[21:44] <james_w> lifeless: good thanks. you?
[21:44] <james_w> lifeless: I met with the code team and others yesterday, and we got somewhere on how the LP stuff should look.
[21:48] <ktenney> james_w: lifeless: got it, thanks
[21:50] <lifeless> james_w: great
[22:16] <poolie> jam, i was planning today to do more reviews
[22:16] <poolie> and i have a meeting
[22:16] <poolie> basically just that
[22:16] <jam> do you ever not have a meeting ?
[22:16] <jam> :)
[22:17] <poolie> :/
[22:17] <jam> Today I worked on 3 things.
[22:17] <poolie> the timezones make it more disruptive
[22:17] <jam> 1) Working with vila to talk about the inventory corruption he saw, and possibly what it needs to fix it
[22:18] <spiv> Today I've just sent the autopack RPC patch for review (so if poolie is doing reviews... *hint*), and am in the process of landing other approved patches from this cycle.
[22:18] <poolie> oh, yay
[22:18] <poolie> well done
[22:18] <poolie> i'll do it first
[22:18] <jam> 2) I worked with someone who has a 16GB repository
[22:18] <jam> and was having a hard time doing "bzr branch"
[22:18] <jam> it turns out that our ability to discuss multiple file histories
[22:18] <spiv> I also want to look at jam's remote readv merge/rfc (first impression is that we should approve it, and start work on a streaming readv a jam suggests)
[22:19] <jam> was requesting a readv() that was about 10GB long
[22:19] <jam> which would be buffered on the server
[22:19] <jam> before sending
[22:19] <jam> and then buffered on the client
[22:19] <jam> before processing
[22:19] <jam> which... doesn't work so well
[22:19] <jam> and sent that to the list as spiv mentions
[22:19] <jam> 3) We also played with btree indexes
[22:20] <jam> which show up nicely for him
[22:20] <jam> so I'm encouraged to help push a --format=1.9 in
[22:20] <jam> And for future work, still trying to get the retry code working for fetch()
[22:21] <jam> (end of text :)
[22:22] <lifeless> I've a LeafNode I'm reasonably happy with
[22:22] <poolie> ok, nice
[22:22] <lifeless> working on handling too big content and spilling into a InternalNode with LeafNode children
[22:22] <jam> lifeless: did you see my email about that? I think fullpath is a better route than parent_id,basename
[22:22] <lifeless> considering sidetracking to fix ISO storage
[22:22] <lifeless> jam: haven't seen it yet, no
[22:23] <jam> lifeless: np, I sent it about 15 min ago
[22:28] <jam> spiv: Is there a reason you need to special case everything, rather than just defining "autopack(self)" on Repository and using it instead of target._pack_collection.autopack()?
[22:29] <jam> It at least seems simpler than special casing the InterPackToRemotePack code
[22:29] <spiv> jam: which special casing are you referring to?
[22:29] <jam> InterPackToRemotePack._pack
[22:29] <spiv> Ah, I see.
[22:29] <jam> it seems like you duplicate most of the def _pack code
[22:30] <spiv> If you uncommit the last revision, you'll see why ;)
[22:30] <jam> just to get the autopack() called
[22:30] <spiv> It's mainly because there was a half-done check_references RPC as well that I have set aside for now.
[22:30] <jam> ah, I saw that you moved that
[22:30] <jam> and I wondered why
[22:31] <jam> also, your "_get_target_revision_index"
[22:31] <spiv> Hmm, and I guess I've pushed some of the other _pack differences into RemoteRepo.autopack
[22:31] <jam> blindly uses "self.target._real_repository"
[22:31] <jam> without calling _ensure_real()
[22:31] <jam> are we sure that is safe?
[22:31] <spiv> Yes, because is_compatible will have called _ensure_real
[22:31] <jam> ah, ok
[22:32] <jam> Anyway, I need to get going, but I saw those things to at least think about
[22:32] <spiv> (sadly!  There's no better way to get the remote format atm, though)
[22:33] <spiv> jam: thanks
[22:34] <jam> anyway, there is a lot of duplication in the _pack code, that if you could just define .autopack() seems like it would go away
[22:35] <jam> Why doesn't InterOtherToRemote take prescendence over InterPackRepo ?
[22:35] <jam> anyway, gotta go
[22:35] <jam> have a good day
[22:49] <thumper> using bzrlib, how do I get the files added/removed in a commit, and the diff / diffstat?
[22:50] <poolie> tree.iter_changes(other)...
[22:50] <thumper> poolie: and now for a non bzr hacker?
[22:50] <thumper> poolie: I have a branch that I want to compare mainline revisions of
[22:51] <poolie> b = Branch.open('thing')
[22:52] <poolie> b.lock_read()
[22:53] <poolie> tree1 = b.repository.revision_tree(revid_1)
[22:53] <poolie> tree2 = b.repository.revision_tree(revid_2)
[22:53] <poolie> changes = tree2.iter_changes(tree1)
[22:53] <poolie> changes=list(changes)
[22:54] <poolie> for c in changes: print c
[22:54]  * thumper goes to a python shell
[22:54] <poolie> show_diff_trees(tree1, tree2)
[22:54] <poolie> season to taste
[22:55] <thumper> poolie: thanks
[22:59] <lifeless> thumper: 'bzr st -r x..y' ;)
[23:06] <LeoNerd> Anyone here using bzrweb, the HTTP frontend thing? I've got:  ImportError: cannot import name LocalBranch
[23:08] <lifeless> LeoNerd: I tend to use loggerhead
[23:08] <LeoNerd> Hrm...
[23:08] <LeoNerd> I'll let you into a secret - I want to embed it in my existing website generation system
[23:09] <LeoNerd> Which is perl. Having looked over the bzrweb code, it looks quite easy to drop that in by Inline::Python, and do a small bit of extra work
[23:20] <poolie> LeoNerd: i think that error means it's out of date with bzr trunk
[23:21] <LeoNerd> Ya.. that seems likely
[23:21] <LeoNerd> Tobehonest I may just take the ideas in the code here and reimplement them myslef
[23:22] <LeoNerd> Write some small bits of connector logic in Inline::Python and do the rest from perl
[23:23] <poolie> is there something about bzrweb that makes it easier to do this than with loggerhead?
[23:23] <LeoNerd> Not really.
[23:23] <LeoNerd> It's just bzrweb was a small simple chunk of code I could easily read and work out what it does
[23:24] <LeoNerd> I'm sure I could do it all myself, it'd just mean me spending a while flicking about in the bzrlib docs to find stuff
[23:24] <LeoNerd> And possibly reinventing tricks this code already has
[23:26] <jisakiel> greetings... I was just asking in launchpad: is there a (reasonable) way to have a server with fine-grained permissions to different projects inside a repository?
[23:26] <jisakiel> they pointed me to bzr_access but I see that is incomplete, only supporting [/] repo
[23:26] <jisakiel> I'm just wondering if there might be something similar to the subversion authentication available
[23:27] <poolie> jisakiel: why not have different repositories for each project?
[23:27] <jisakiel> hmmmm
[23:28] <jisakiel> bzr_access does not seem to support that?
[23:29] <jisakiel> I really like the launchpad model
[23:30] <jisakiel> being a personal server I don't want to give +w to everybody who is able to push via bzr (as I track different projects there)
[23:30] <jisakiel> and I'd rather not have shell accounts accesible
[23:30] <poolie> jisakiel: launchpad has a repository per branch
[23:31] <jisakiel> but how do they authenticate then
[23:31] <jisakiel> they just told me they use a custom ssh client
[23:31] <jisakiel> er... ssh server  ^^
[23:32] <spiv> jisakiel: hmm, I thought it was possible to configure bzr_access differently
[23:33] <jisakiel> as far as I understand
[23:33] <spiv> jisakiel: ah, there's also bzr_ssh_path_limiter in the contrib/ directory
[23:33] <jisakiel> in 1.8??
[23:33] <spiv> Yes, and probably earlier too
[23:34] <jisakiel> huh, don't see it when I do apt-get source bzr
[23:34] <spiv> Oh, huh.
[23:34] <spiv> It's not in 1.8
[23:35] <spiv> It is in bzr.dev
[23:35] <jisakiel> d'oh
[23:35] <spiv> It had originally been posted on the mailing list for quite a while before that, I guess that's why I was confused.
[23:37] <spiv> jisakiel: you can always have different SSH keys for different repositories.
[23:37] <jisakiel> point is
[23:37] <jisakiel> users would be using their own ssh key
[23:37] <jisakiel> just like in launchpad
[23:38] <jisakiel> and I, somehow, would be able to give them + or - W
[23:38] <jisakiel> just like editing svnauth
[23:38] <Peng_> Argh, Loggerhead is unhappy.
[23:39] <jisakiel> I guess the only "reasonable" way is to explode the number of groups in the server
[23:39] <jisakiel> one per project
[23:39] <jisakiel> and use filesystem permissions
[23:40] <jisakiel> but *urgh*
[23:41] <spiv> jisakiel: there's another possibility -- if you're just interested in restricting writes (not reads), then you could use PQM to manage the branches you want to restrict write access to.
[23:41] <spiv> jisakiel: it's a different workflow, so it might not be suitable for you.
[23:42] <spiv> jisakiel: it's how the bzr trunk is managed, for example; bzr developers keep their own branches of bzr wherever they like, but to commit to the trunk you need to send a request to the PQM bot.
[23:43] <jisakiel> heh, I guess it might be easier to extend the bzr_access script than to deploy that
[23:43] <jisakiel> ;)
[23:43] <spiv> jisakiel: PQM checks that the request is authorised (via PGP), does the requested merge, runs the test suite, and then only if the tests pass does it commit and publish the merge.
[23:44] <spiv> It solves a slightly different problem, but maybe it's useful to you.
[23:44] <jisakiel> seems too complicated but txs anyway
[23:44] <spiv> Fair enough.
[23:44] <jisakiel> ( and test suites are still utopia in the CS courses ;) )
[23:45] <spiv> The problem with extending bzr_access is that the bzr client always asks for --directory=/ when it connects over bzr+ssh.
[23:45] <spiv> (Ah! ;) )
[23:46] <spiv> (and because the bzr client can't know which directories it will need in advance, it can't really do anything else)
[23:47] <jisakiel> I see
[23:47] <jisakiel> yikes
[23:47] <jisakiel> :D
[23:47] <spiv> You could make it spawn something other than plain bzr serve, i.e. write a bzr plugin to provide a virtual filesystem with the restrictions you need.
[23:48] <jisakiel> --directory=(?P<dir>\S+)\s+
[23:48] <spiv> There are already the building blocks for that in bzrlib (there's psuedo-chrooting logic and also force-readonly logic already there in bzrlib.transport.decorators)
[23:49] <spiv> It wouldn't be trivial to write, though.  Certainly not as easy as editing a config file!
[23:49] <jisakiel> still seems like a not-trivial ammount of work
[23:49] <jisakiel> that's it
[23:50] <spiv> jisakiel: right, that regex is always going to just see /.
[23:50]  * spiv nods
[23:50] <jisakiel> then get_directory?
[23:50] <spiv> If you can bear it, using filesystem groups sounds the most practical for you.
[23:50] <spiv> get_directory will always get / :(
[23:51] <jisakiel> and bzr+ssh sends / because...?
[23:51] <spiv> The author of bzr_access was a bit optimistic about what the bzr client would do.
[23:51] <jisakiel> lol :D
[23:51] <jisakiel> just like me
[23:51] <spiv> Because the client can't know what directories it will need to look at before connecting.
[23:52] <spiv> Opening a branch may require opening a shared repository in an arbitrary parent directory, or resolving a branch reference at an arbitrary location, etc.
[23:52] <jisakiel> aaahm
[23:53] <jisakiel> I was just scratching my head, as bzr+ssh://host/whatever/path seems complete
[23:54] <spiv> e.g, "bzr log bzr+ssh://host/whatever/path" can't spawn a remote bzr serve with --directory=/whatever/path, because the actual revision data might be at /whatever.
[23:54] <spiv> Or even /!
[23:54] <jisakiel> but nevertheless
[23:54] <jisakiel> ah
[23:54] <jisakiel> bzr serve is, of course, restricted in doing ".."
[23:55] <jisakiel> because what would be the point in having --directory
[23:55] <jisakiel> otherwise while browsing the log one could just go "up" like I guess bzr log /whatever/path does
[23:56] <spiv> --directory is there so that "bzr serve" over TCP can serve just part of your filesystem, and also so that bzr_ssh_path_limiter can be written.
[23:56] <spiv> Right.
[23:57] <jisakiel> (which btw I can't find in bzr-dev, at least not in http://bazaar-vcs.org/bzr/bzr.dev/contrib/ )
[23:57] <spiv> Hmm, someone really ought to write that hypothetical plugin I was referring to and add it to contrib/
[23:57] <jisakiel> lol
[23:58] <jisakiel> seems overwhelming to me right now, sorry :D
[23:58] <spiv> (that someone is probably me)
[23:59] <jisakiel> I guess this excludes some "business" uses of bzr more-or-less-centralized workflows
[23:59] <spiv> Huh, it's not in bzr.dev.  Oh man, I'm an idiot.
[23:59] <spiv> It's in my local checkout, not committed!
[23:59] <jisakiel> heh, bzr add perhaps? :P
[23:59] <jisakiel> it happens xd
[23:59] <spiv> jisakiel: http://rafb.net/p/P1TEcP10.html