[00:11] <speakman> it was caused by not setting whoami on the SERVER side
[00:14] <lifeless> jelmer: ping
[00:17] <jelmer> lifeless: pong
[00:18] <lifeless> two questions re your 440K patch:
[00:19] <lifeless> - I think you had a submit branch of bzr.dev, so its kinda hard to review :P (and note that [split-inventories][MERGE] is *much* ebetter, otherwise BB will pick it up
[00:20] <lifeless> - why do y ou need create_by_apply_delta on regular inventories? (The Repository is the appropriate interface for making a new inventory normally
[00:21] <lifeless> bbiab, car time
[00:21] <jelmer> lifeless, bzr-svn's fetch uses the lhs parent inventory
[00:45] <awilkins> jelmer: I'm trying bzr-svn 0.4 against svn 1.5.4 and while it builds, I'm having trouble running it. I'm getting "DLL load failed: The specified module could not be found."
[00:46] <jelmer> awilkins, sorry, I'm not the right person for windows stuff
[00:46] <awilkins> Hmm, appears to be DLL names
[00:47] <awilkins> It's looking for libapr.dll when it's called libapr-1.dll
[00:49] <jelmer> I've heard that problem being mentioned before
[00:49] <awilkins> It seems to be the client.pyf
[00:50] <lifeless> jelmer: I don't follow
[00:50] <jelmer> lifeless, bzr-svn uses the lhs parent inventory when interpreting changes the server sends us
[00:50] <lifeless> jelmer: yes... ?
[00:50] <awilkins> The real libapr-1.dll is also being loaded
[00:50] <jelmer> lifelss: fetch itself builds up an inventory delta
[00:50] <lifeless> jelmer: this seems unrelated to my point
[00:51] <jelmer> lifeless, argh, sorry, I mixed up two patches
[00:51] <jelmer> lifeless, that first patch is moot if the second is accepted
[00:52] <lifeless> 	Make Repository.add_inventory_delta() return the resulting inventory.
[00:52] <lifeless> ?
[00:52] <jelmer> yes, that's the second one
[00:52] <lifeless> jelmer: +1
[00:52] <lifeless> jelmer: just land it
[00:53] <jelmer> lifeless, thanks
[01:17] <awilkins> jelmer: The reason is that the APR distribution in the win32 dev pack I'm using is not the correct one
[01:17] <awilkins> Grr, squish those SVN devs
[01:19] <awilkins> jelmer: They are shipping 1.3 libs with 1.5.4 but the headers are the 0.9.6 versions
[01:24] <lifeless> LOL
[01:26] <awilkins> markh: You around?
[01:26] <markh> awilkins: I am - hi
[01:26] <awilkins> Do you build bzr-svn at all (and use it?_
[01:27] <markh> I build it - I don't use it much as the svn repos I use have svn:externals and svn:eol-style :(
[01:27] <markh> setup.py should be fairly accurate with build instructions
[01:28] <awilkins> Heh, see above about not being able to use 1.5 version with windows (if you take the distributed library pack)
[01:28] <markh> the "python based" installer?
[01:28] <awilkins> They are shipping the 1.x series APR but the library pack has the 0.9 libs and headers
[01:28] <awilkins> The python installer doesn't even build the pyd files
[01:28] <awilkins> I'm building my own
[01:28] <markh> yeah
[01:29] <awilkins> If I want to access a 1.5 repo over ra_local I'm going to have to build APR in the morning.
[01:29] <markh> awilkins: so where are you getting the apr binaries from?
[01:30] <awilkins> The SVN guys host a "dev pack" for windows with libs and headers in
[01:30] <markh> right - and you are building bzr-svn yourself?
[01:30] <awilkins> The binaries in the 1.5.4 win32 distro are the 1.3x versions
[01:30] <awilkins> the dev pack is 0.9
[01:31] <awilkins> Yes, building bzr-svn extensions using VC++ 2003 toolkit and it's libs and headers
[01:31] <awilkins> Which corresponds to the compiler used to build python 2.5
[01:31] <markh> so - if you follow setup.py, one of those .zip files should have exactly matching apr binaries - or I'm still a little confused :)
[01:31] <awilkins> The zip files do not have the right APR bits
[01:31] <markh> don't try and use any other binaries with it
[01:32] <markh> hrm
[01:32] <markh> they do for me
[01:32] <awilkins> They don't match the ones in the win32 distribution I downloaded
[01:32] <awilkins> They match the 1.4.6 binaries
[01:32] <markh> the bzr win32 distro?
[01:32] <awilkins> But the 1.5.4 distro uses APR 1.3.x not 0.9.6
[01:33] <awilkins> Alas, the 1.5.4 dev pack has the 0.9.6 libs and headers
[01:34] <markh> OK - I last built using 1.5.2 - that package should be OK.  Note the "distro" of the version should not be relevant - you don't need anything other than the binaries bzr-svn (tries to) provide.
[01:34] <awilkins> markh: I'm falling back on the path to load the DLLs from my installed SVN
[01:35] <markh> right - which is the root of the problem
[01:35] <awilkins> markh: And since the libs don't match the DLLs I have it's irrelevant anyway
[01:35] <awilkins> Even if I copied them into the same folder
[01:35] <markh> well - binaries build from the 1.5.2 dev pack appear to work fine and be internally consistent
[01:35] <awilkins> Jolly good
[01:35] <awilkins> I shall grab that one
[01:35] <markh> so can you build with that?
[01:35] <markh> cool
[01:36] <awilkins> Oh hold on
[01:36]  * awilkins slaps forehead
[01:36] <awilkins> Maybe if I take the Apache 2.0 version
[01:37]  * awilkins checks dev pack is not different on each page
[01:37] <markh> awilkins: its quite possible 1.9 and later bzr binary builds have been made with the 1.4 svn devpacks
[01:37] <markh> setup.py references them - even though it works with 1.5 - and that is quite possibly exactly what jam did when setting up the binary builds.
[01:38]  * markh tries to look
[01:38] <awilkins> If we can get 1.5 working consistently we should use it - well, the reason I am trying is so I can get rid of the per-file properties when pushing over ra_local
[01:39] <awilkins> I am an idiot
[01:39] <awilkins> I've been using the apache 2.2 distro with the 2.0 devpack
[01:39] <awilkins> It would be helpful if they named the archives more extensively
[01:40] <markh> cool :)  it turns out 1.9 should have 1.5.4 anyway
[01:40] <awilkins> Right, now I know, I'm turning in, I shall have another go tomorrow.
[01:41]  * awilkins mumbles curses at management types who insist on SVN even though the rest of the team in question is using Bazaar
[03:43] <jam> markh, poolie: Kerguelen seems to be up, but seems to not have http access for me to download any data for packaging a 1.10rc1 installer.
[03:44] <jam> Even browsing to "http://www.google.com" fails
[03:44] <markh> jam: its not just IE broken?
[03:44] <markh> IIRc that was one of the updates :S
[04:00] <linrunix> hi
[04:00] <markh> jam: any luck?
[04:06] <hunmonk> hi guys.  does anybody know if the EPEL repo is going to get an updated version of bzr anytime soon?  looks to me like it still has only 1.3
[04:12] <linrunix> markh: i just upload my project to launchpad
[04:12] <linrunix> the Initial branch
[04:13] <linrunix> if somebody wanna change something to a certain file what does he has to do?
[04:13] <linrunix> let's say he took example.py and changed something... how does he commit these changes?
[04:14] <linrunix> sorry i'm kind of confused
[04:14] <markh> linrunix: they would create a local branch or checkout of your project, than commit those changes to that local branch or checkout.  He would then use 'bzr send' to send the changes
[04:14] <markh> or if he had permissions, he could push or checkin directly to launchpad
[04:14] <linrunix> yes he has
[04:14] <linrunix> but what does he do?
[04:14] <linrunix> he download the branch to his computer
[04:14] <linrunix> change the file
[04:14] <linrunix> and commit?
[04:14] <linrunix> the whole file
[04:14] <markh> he probably should just do a "bzr cp lp:your_branch"
[04:15] <markh> oops
[04:15] <markh> "bzr co ..."
[04:15] <markh> then make changes, then "bzr ci" - that is the easiest (but probably not the most flexible)
[04:16] <markh> linrunix: check out http://doc.bazaar-vcs.org/bzr.dev/en/tutorials/tutorial.html
[04:17] <markh> linrunix: or even better, http://bazaar-vcs.org/Workflows
[04:17] <linrunix> very thanks
[04:50] <linrunix> markh: sorry i was reading but it looks like I dont understand it very well
[04:51] <linrunix> i'm just trying to learn how to use it so.. we can start coding from there
[04:51] <markh> it could be argued there are too many options :)
[04:51] <markh> options for ways to work
[04:51] <linrunix> i got my friend to download the branch
[04:51] <markh> how exactly?
[04:51] <linrunix> let me give u how
[04:52] <linrunix> bzr branch http://bazaar.launchpad.net/~linrunix/pyseleccion
[04:53] <linrunix> now, does he have to upload his own branch to his user?
[04:53] <markh> ok - so your friend can then make changes and commit as often as he likes.  When he is ready to submit the changes, he should do a "bzr merge" (in case others have changed the branch since he pulled it), then "bzr push"
[04:53] <linrunix> bzr push that's it
[04:53] <markh> then your branch will be updated to have his changes
[04:54] <markh> anyone doing "bzr pull" etc after that will get them
[04:54] <linrunix> he doesnt have to have the branch in his user
[04:54] <markh> he could push it somewhere else if he wants
[04:55] <linrunix> but to update the actual branch
[04:55] <linrunix> just a bzr push
[04:55] <markh> if he can't push directly, or wants you to comments first, for example, he may well push to a private place.
[04:55] <linrunix> ernie@intrepid:~/Desktop/trunk$ bzr push
[04:55] <linrunix> bzr: ERROR: No push location known or specified.
[04:55] <markh> yeah - first time you must say where you want to push to
[04:55] <markh> bzr doesn't assume you want to push back to the same place
[04:56] <linrunix> so taht will be  http://bazaar.launchpad.net/~linrunix/pyseleccion right?
[04:56] <markh> often you don't - you may well want to push somewhere private
[04:56] <markh> the same url used to pull it would be fine
[04:56] <markh> it will remember then - so "bzr push" will push back to the same spot thereafter
[04:56] <markh> but you can obviously specify a new location any time too
[04:57] <linrunix> like that bzr push bzr://bazaar.launchpad.net/~linrunix/pyseleccion
[04:58] <markh> that should be fine - but the "lp:blah" version should work too
[04:58]  * markh can't recall exactly how they all expand...
[05:15] <linrunix> bzr: ERROR: Cannot lock LockDir(lp-44825360:///~linrunix/pyseleccion/trunk/.bzr/branchlock): Transport operation not possible: readonly transport
[05:15] <linrunix> any help?
[05:16] <linrunix> too_short: how do I set the branch so another user can upload too?
[05:19] <spiv> linrunix: for the error, "bzr lp-login"
[05:19] <spiv> linrunix: to let other LP users commit to your branch, change it's owner to be a team rather than your user
[05:20] <linrunix> ok
[05:20] <linrunix> thanks
[05:32] <linrunix> spiv: after changing a file i need to commit b4 pushing right?
[05:32] <linrunix> -m "bla bla bla" right?
[07:38] <vila> hi all
[09:43] <VSpike> http://pastebin.com/m6516f466 that looks a bit odd.  How did I get into that state? :)
[10:02] <awilkins> jelmer: I resolved my bzr-svn with 1.5 svn issue ; I was using the Apache 2.2 binaries and the 2.0 devpack.
[10:02] <awilkins> D'oh.
[12:22] <awilkins> jelmer: Why does subvertpy have to go in bzrlib?
[12:26] <awilkins> jelmer: Or somewhere other than plugins\svn .. I'm not sure I understand this so far
[12:31] <sven>  sven
[12:54] <awilkins> This is odd, the exception handler isn't working
[13:00] <markh> every time I've seen that it's been due to multiple copies of the same module, meaning multiple exception classes that *should* be the same...
[13:00] <markh> bugger - past midnight - I'm officially late for bed :)
[13:01] <awilkins> markh: It doesn't even throw
[13:01] <awilkins> markh: If you feed it a wildcard handler it never triggers
[13:01] <awilkins> It just spews an error to the console
[13:01] <awilkins> It works on a console, just not in the bzr-svn code
[13:01] <awilkins> grr
[13:02] <awilkins> gnight anyway
[13:02] <awilkins> Having hacked my way around it, I'm running into other issues
[13:02] <markh> well - now I'm officially late, a little later wont hurt ;)
[13:02] <awilkins> 0.4 is working well, this is 0.5 :-)
[13:02] <markh> oh - still svn woes :(
[13:03] <awilkins> Heh, the one I've run into looks more like a jelmer issue
[13:03] <awilkins> 0.5 is supposed to cope with disorganised repos with evil folder histories better
[13:04] <awilkins> I have a very large repo to test it on.
[13:04] <awilkins> If it successfully manages to map all the branches in it ( to the point where pulling each one results in a small revision and not over 100MB), I'll be impressed
[13:05] <awilkins> On the flipside I have a manager demanding I use SVN for an external repository (sob), so the highly-functional 0.4 support is very welcome
[13:06] <awilkins> I have to migrate this huge backup folder of archived copies and turn it into a repository
[13:06] <awilkins> Consultants are bad enough, but OLD consultants with no experience of VCS systems.....
[13:08] <awilkins> Plus all the files are renamed with version numbers... and they all include each other. Grr.
[13:08] <awilkins> Got that out of the way, mostly, I just need to bend my brain a bit further around which order this next set are in.
[13:11] <awilkins> I'd go to bed
[13:11] <awilkins> I'll pester jelmer when he's awake
[14:18] <jelmer> awilkins, hi
[14:18] <jelmer> awilkins, subvertpy is under bzrlib.plugins.svn because it's included with the plugin
[14:40] <awilkins> jelmer: I think I'm having a problem with the import code in svn\__init__.py
[14:41] <awilkins> jelmer: For some reason it doesn't seem to be catching the first ImportError and trying the code in the except: block
[14:43] <awilkins> jelmer: Once I hack my way around that, I run into a SubversionException for the case I'm trying.
[14:43] <jelmer> what's the error exactly?
[14:44] <awilkins> jelmer: THe import just reports "No module named subvertpy" at the console, the error doesn't percolate to the top of the stack (presumably the plugin loader is catching it)
[14:47] <jelmer> awilkins, is subvertpy installed in the right location?
[14:47] <awilkins> jelmer: It's nested in the svn plugin
[14:47] <jelmer> I never actually install bzr-svn myself, just run it from ~/.bazaar/plugins
[14:47] <awilkins> Ah, that may be why
[14:47] <jelmer> bzrlib/plugins/svn/subvertpy/ ?
[14:48] <awilkins> The "build" command of setup.py makes a different tree to the source
[14:48] <awilkins> in the source, subvertpy is inside another subvertpy folder
[14:49] <awilkins> I had to remove the path near line 87 to make that bit work
[14:49] <awilkins> The join of the path of  __file__ to subvertpy
[14:50] <awilkins> In the source tree it's svn/subvertpy/subvertpy
[14:50] <jelmer> I'll fix the imports, one sec
[14:51] <awilkins> What really puzzled me was that the first import (before it extends the path) fails (as expected) but the exception isn't trapped ; it never gets to the bit where the path is extended.
[14:51] <awilkins> Debugged with primitive "print" statements
[14:53] <awilkins> http://paste.ubuntu.com/80356/  # this doesn't work and never prints "Caught the importerror"
[14:54] <awilkins> But if you remove the exception handling and just go straight for extending the path it works fine.
[14:54] <jelmer> that import should work fine in your case
[14:54] <jelmer> because it's a relative import
[14:55] <jelmer> it doesn't work in a couple of other cases (when importing subvertpy from bzrlib.plugins.svn.mapping3.scheme, for example)
[14:56] <awilkins> Well, it should work, and printing __file__ confirms that it's the right file being run, but it doesn't
[15:04] <awilkins> jelmer: Maybe I should look at stack traces more often, the problem is in ra.py
[15:05] <awilkins> jelmer: Adding the path in svn\__init__.py  just masks it
[15:15] <awilkins> jelmer: It seems that a number of imports in subvertpy are using the subvertpy namespace to import things that are inside the subvertpy namespace - but it can't find it because it's not on the path.
[15:15] <awilkins> Once you fix these up to be relative to subvertpy it works fine  ( is doing ' from __init__ import <stuff> ' acceptable ?)
[15:15] <jelmer> awilkins, that's what the sys.path.insert call is supposed to fix
[15:16] <awilkins> jelmer: That path call only happens if the initial import fails
[15:16] <jelmer> it's not correct in your case though
[15:16] <jelmer> awilkins, relative imports don't work, since it means you can't import any subvertpy stuff from python modules that are not in the top-level code of bzr-svn
[15:16] <jelmer> as there is no ".." in imports
[15:17] <awilkins> Ok, so it needs to always add subvertpy to the path regardless then?
[15:18] <jelmer> or use bzrlib.plugins.svn.subvertpy everywhere
[15:18] <jelmer> awilkins, to work around it, you should be able to move subvertpy to the top-level for now
[15:23] <jam> markh: No luck on kerguelen. IE is busted with "No Module Found", but bzr with http:// urls is giving Couldn't resolve host 'bazaar-vcs.org'.
[15:23] <jam> Could be a DNS issue
[15:26] <awilkins> jelmer: Moving that package also seems to fix my SubversionException
[15:37] <jelmer> awilkins, so everything works now?
[15:37] <awilkins> jelmer: It would appear to.
[16:07] <awilkins> jelmer: Hmm, that's disapointing ; something in 0.4 is a real memory hog
[16:07] <awilkins> I tried pushing a branch to a fresh svn repo and on reaching the third revisions it eats 1.3GB of memory
[16:07] <awilkins> Now, it is a big revision, but the entire bzr repo is less than 65MB
[16:07] <jelmer> What about 0.5 ?
[16:08] <awilkins> I think 0.5 was doing it to (but I didn't know whether to write that off as running against a Bazaar with no extensions built)
[16:09] <awilkins> It rapidly pushes the machine into swap at which point it just crawls
[16:09] <awilkins> Last I tried it, pulling big repos wasn't a problem
[16:09] <awilkins> But maybe pushing is different
[16:10] <awilkins> I mean, pulling still eats a few 100MB
[16:10] <awilkins> I'm using 0.4 with 1.9 and 0.5 with bzr.dev (no extensions built)
[16:10] <jelmer> the code that handles some of this stuff is a lot different between 0.4 and 0.5
[16:10] <jelmer> although 0.4 shouldn't be leaking that much either
[16:11] <awilkins> Does 1.10 rc1 have "foreign" ?
[16:11] <jelmer> yes
[16:11] <awilkins> I suppose it's marked as compatble in the code, that answers my question
[16:11] <awilkins> Bah, no installer anyway :-)
[16:12] <jelmer> Importing bzr-svn or bzr.dev into svn works fine here, btw, no heavy memory usage
[16:12] <jelmer> I wonder what's making it problematic for you
[16:12] <awilkins> It's a lot more paths and size than bzr.dev
[16:13] <lifeless> yo
[16:13] <awilkins> bzr.dev is 1013 files and ~15MB
[16:13] <jelmer> hey lifeless
[16:14] <jelmer> lifeless, travelling in some strange part of the world or just awake at night?
[16:14] <awilkins> This is 6237 paths and 62.5 MB
[16:14] <lifeless> uds
[16:14] <awilkins> And most of that happens in a single revision (which it seems to be having trouble with)
[16:14] <jelmer> ahh, of course
[16:15] <awilkins> I think that one revision has about 3600 paths in it and most of that data
[16:15] <jelmer> awilkins, ah, that may be slow indeed
[16:15] <awilkins> Pulling similar sizes FROM svn repos  (and larger) doesn't seem to cause the same explosion of memory
[16:20] <jelmer> all texts for the commit have to be kept in memory during push
[16:20] <jelmer> all changed texts, that is
[16:20] <awilkins> Ow
[16:20] <awilkins> Hmm, that's still only 65 MB though?
[16:21] <awilkins> I realise that overhead is important but... 1.25 GB of it?
[16:22] <lifeless> jelmer: all at once? Can't you callback on each separately?
[16:22] <jelmer> awilkins: I suspect there's more going on
[16:22] <awilkins> As do I :-)
[16:22] <jelmer> lifeless, it can be done a bit more lazily
[16:23] <awilkins> It consumes the memory very quickly on reaching that third revision
[16:23] <jelmer> lifeless, but I doubt that's really the problem here
[16:26] <awilkins> jelmer: There are quite a few merges in either direction on this branch, would that affect it?
[16:26] <awilkins> Well, not by the third revision (duh)
[16:27] <awilkins> Right, time for a shower and a quick forage for wife-food
[16:27] <awilkins> I shall come back later
[16:29] <jelmer> awilkins_away, should't matter
[16:29] <jelmer> python really sucks when it comes to debugging memory usage :-/
[16:30]  * Tak s/debugging //, get kicked from channel
[16:33] <lifeless> Tak: it is higher on memory usage than some lower level languages, due to various choices; but it should be ok for nearly everything, as long as you don't leak :)
[16:33] <lifeless> (where a leak could be contained to a single function but still be very large)
[16:33] <Tak> I know, I'm only kidding
[16:34] <Tak> besides, I'm a ruby fan, I don't have any room to complain
[16:34] <lifeless> heh!
[16:42] <LarstiQ> jelmer: iirc I've been happy with http://guppy-pe.sourceforge.net/ in the past
[16:50] <evarlast> can I get a log for a folder in a branch? am I stupid for wanting to do so?
[16:52] <NfNitLoop> evarlast: I was wanting to do that just the other day.
[16:52] <NfNitLoop> I didn't find a way to do so, though.
[16:53] <Tak> hmm - it works for a branch from a svn repo
[16:54] <NfNitLoop> right, you can bzr log <branch>
[16:54] <NfNitLoop> but not bzr log <subdir>
[16:54] <NfNitLoop> or, if you do, you just get a single revision for when the dir was created.
[16:54] <NfNitLoop> not when all files within that dir were last modified.
[16:55] <Tak> I mean, it works at the directory level for a branch from an svn repo
[16:56] <jam> lifeless: any chance you could look at: http://bundlebuggy.aaronbentley.com/project/bzr/request/%3C4935F649.2010706%40arbash-meinel.com%3E
[16:56] <jam> I'd like to base the rest of my work on it
[16:56] <jam> since it makes the map/unmap "stable"
[16:57] <lifeless> jam: sure
[16:57] <jam> thx
[16:57] <lifeless> upgrading to intrepid right now, and I'm on leave today
[16:58] <jam> sure
[16:58] <jam> It isn't something you have to do
[16:58] <lifeless> if you think its ok, I suggest you build on iti anyhow
[16:58] <jam> but since martin, you, and spiv are all afk...
[16:58] <lifeless> or even land it, and we review it later
[17:00] <jam> I guess I'll go for that, then.
[17:08] <kjcole> Any instructions on removing a damaged knit file w/o consequence?  I have determined that the contents of the file are something I removed from the repository long ago.
[17:08] <kjcole> I'm just looking for a way to cascade it's removal without screwing up bzr.
[17:09] <lifeless> kjcole: so, if the file is used by *any* revision still in the repository, bzr will be unhappy whena ccessing that revision
[17:09] <lifeless> the easiest way to have a completely clean repo is to make a new one and branch everything into it
[17:10] <kjcole> lifeless: I tried branching.  No luck.
[17:10] <lifeless> kjcole: it errored?
[17:10] <kjcole> lifeless: Yep.
[17:11] <lifeless> kjcole: ok. do you have any idea of how the damage occured?
[17:11] <kjcole> lifeless: "bzr check" reveals a zlib.error "invalid distance too far back"
[17:12] <kjcole> lifeless: No idea how it occurred, but it occcured a long way back: The file in question was removed in revision 3, and the repository's up to revision 80.
[17:13] <lifeless> kjcole: ok. do you have another copy of it anywhere?
[17:13] <kjcole> lifeless: I only learned of it while trying to move the repository from an old dapper machine w/ bzr 0.8.2 to a hardy machine w/ a more current bzr.
[17:13] <kjcole> lifeless: Of the repository? No.
[17:13] <lifeless> ok
[17:14] <jelmer> LarstiQ, I couldn't get guppy to work and the project appears to've been dead since 2005
[17:14] <lifeless> so you have a damaged data file for revision's 1 and 2
[17:15] <lifeless> and the rest is fine.
[17:15] <lifeless> (as far as you know)
[17:15] <kjcole> lifeless: I guess.  Since only a single knit file gives an error, and there are two other knit files with the same starting name...
[17:16] <kjcole> lifeless: (I'm gonna check the dates on those files.  BRB)
[17:19] <kjcole> lifeless: all created on the same day.  (It's been a while, but what I believe I did was bzr init, add everything in an existing directory tree, commit, and then weed out the few things I didn't want.
[17:19] <kjcole> lifeless: (all on the same day).
[17:20] <lifeless> ok
[17:20] <lifeless> so I suspect a disk bit error, as its data that bzr has had no reason to touch for a very long time
[17:21] <kjcole> lifeless: The damaged file isn't something I would have changed -- it was part of the Python Library Reference docs that was inadvertently sitting in the tree.
[17:21] <lifeless> kjcole: the easiest thing to do would be to trim revision 1 and 2 - by rebasing the rest of the branch
[17:22] <lifeless> kjcole: exactly, its data that was probably written fine, but not read from for several years
[17:22] <kjcole> lifeless: This sounds promising.  Thanks.
[17:23] <lifeless> I believe someone posted to the list a recipe to do this, they didn't have a damaged repo,but rather had text they couldn't show people and needed to eliminate; same principle though
[17:23] <LarstiQ> jelmer: hmm
[17:23] <LarstiQ> jelmer: I'm reasonably sure it was guppy that I used a couple of months ago to debug memory issues.
[17:23] <LarstiQ> jelmer: but I'll look it up after dinner.
[17:25] <kjcole> lifeless: Oook.  If worse comes to worse, since I don't intend to move backwards on this, I'll try to generate a history report (log plus list of files affected) and just "rm -rf .bzr" and start again.
[17:25] <kjcole> lifeless: But I wanted to salvage what I could first.
[17:25] <lifeless> kjcole: you should be able to salvage everything but the damaged revs.
[17:26] <beuno> lifeless, hey hey. Are you in SFO yet?
[17:26] <kjcole> lifeless: to the list archives, then -- I hope there was a decent subject line. ;-)
[17:26] <lifeless> beuno: yes
[17:26] <beuno> lifeless, cool. We're heading towards the UDS hotel today
[17:27] <kjcole> lifeless: Thanks again.  Later.
[17:27] <lifeless> kjcole: anytime
[17:28] <lifeless> beuno: cool
[17:32] <vila> jelmer: regarding bug #303959, I'm done with the fixes I could make to bzr. Have you seen my last comment ? Does that sounds right ?
[17:33] <vila> lifeless: it turns out it was a bit more work than anticipated but I think the result was worth it (IMHO)
[17:33] <lifeless> vila: the key question for me is 'can jc2k pull'
[17:34] <vila> if jc2k == bug reporter: return True
[17:35] <lifeless> John Carr
[17:35] <vila> See the last bug comment for a more elaborate answer
[17:35] <lifeless> also, 'return jc2k == bug_reporter'
[17:35] <vila> hehe
[17:37] <lifeless> I've read it now
[17:37] <lifeless> yes, I concur, bzr-svn should handle foo->foo/ as 'cannot be a svn repo'
[17:38] <vila> also, 'ireturn jc2k == bug_reporter' doesn't cover the implicit else: return None :)
[17:39] <lifeless> it returns False istead
[17:39] <vila> which isn't the answer I wanted to give :)
[17:41] <Jc2k> o_O
[17:41] <Jc2k> bug_reporter.state == 'amused'
[17:41] <vila> Jc2k: did you try my patch ?
[17:41] <Jc2k> vila: not yet. just got home, in a bit of a rush..
[17:41] <lifeless> jelmer: ping
[17:42] <jelmer> lifeless, pong
[17:42] <vila>  bug_reporter.state == 'checking_later_I_promise' :-)
[17:42] <Jc2k> vila: is it in bzr.dev?
[17:42] <Jc2k> otherwise me needs somewhere to pull or grab patch from
[17:42] <vila> not yet, but it is in the branch associated with th bug: lp:~vila/bzr/303959-redirection
[17:44] <Jc2k> ooh shiny
[17:44] <vila> Jc2k: feeback very welcome since I can't reproduce the bug with my actual setup
[17:44] <Jc2k> will spin in next 45 minutes if i can, otherwise it will be on my todo for asap
[17:45] <lifeless> vila: am I right that in theory this bug is fixable just via bzr-svn?
[17:45] <lifeless> [the actual reported fault, I mean]
[17:45] <jelmer> lifeless, yes
[17:46] <lifeless> jelmer: hi
[17:46] <vila> lifeless: totally
[17:46] <lifeless> jelmer: so pinging re this bug; also how did split-inv work for you;
[17:46] <jelmer> lifeless, split inv seems to work fine, especially now those two changes are in
[17:46] <jelmer> I also noticed that CHKInventory.has_id() is quick while CHKInventory.__contains__() isn't
[17:46] <lifeless> faster/slower/can't-tell-yet?
[17:47] <jelmer> In percentages, it's definitely faster (smaller percent of work is now in updating the inventory)
[17:48] <lifeless> jelmer: __contains__ isn't defined on CHKInventory :P
[17:48] <jelmer> lifeless, I was seeing references to __iter__
[17:48] <lifeless> oh right
[17:48] <lifeless> probably an implicit behaviour
[17:48] <lifeless> I'll move the __contains__ definition
[17:49] <jelmer> lifeless, the remaining culprits are:
[17:49] <jelmer> - Repository.add_inventory_delta() (22%)
[17:51] <jelmer> - VersionedFile.add_lines() (15.59%)
[17:51] <jelmer> - VersionedFile.get_record_stream() (13%)
[17:53] <lifeless> 22% is still quite high
[17:53] <lifeless> fix for __contains__ pushed
[17:54] <lifeless> what does that 22% break down into?
[17:55] <jelmer> let me just run it again locally
[18:31] <rockstar> lifeless, holy crap, why are you awake!?
[18:31] <lifeless> I'm in mountain view
[18:32] <rockstar> lifeless, okay, that makes more sense.  :)
[18:32] <lifeless> awake is not the right term though
[18:32] <rockstar> lifeless, about the memory middleware, I can land it, but it's a SERIOUS hack.
[18:32] <lifeless> rockstar: is it in its own module?
[18:33] <rockstar> Basically, it reads it's own memory usage from proc
[18:33] <rockstar> lifeless, yes, loggerhead requires a flag to use it.
[18:33] <lifeless> rockstar: bzr has a function to do that btw
[18:33] <jelmer> lifeless, new results:
[18:33] <rockstar> lifeless, bzr has a serious lack of documentation.  :)
[18:33] <jelmer> add_inventory_delta(): 13%
[18:34] <jelmer> CHKInventory.children: 36.12%
[18:34] <jelmer> VF.add_lines(): 21%
[18:34] <rockstar> lifeless, one of these days, I'm going to start doing things on my TODO list, and writing docs for bzr is on there.
[18:34] <jelmer> VF.get_record_stream(): 14.46%
[18:35] <lifeless> rockstar: we have docs :> problem is knowing what to look for :)
[18:35] <jelmer> the rest is (obviously) bzr-svn overhead
[18:35] <lifeless> rockstar: bzrlib.trace.debug_memory
[18:35] <rockstar> lifeless, I think that Launchpad Code team kinda has that problem too.  Those guys are REAL slackers.  :)
[18:35] <lifeless> rockstar: probably wants patching to accept a callback function
[18:36] <rockstar> lifeless, great, I'll take a look at it.
[18:36] <lifeless> anyhow
[18:36] <lifeless> I woud land it if its ugliness is isolated
[18:37] <lifeless> jelmer: are those percent-of-total?
[18:38] <lifeless> and do you mean CHKInventoryDirectory.children ?
[18:55] <jelmer> lifeless, yes
[18:55] <jelmer> lifeless, they are percentages of total
[18:56] <lifeless> what are the callers of .children?
[18:57] <jelmer> bzr-svn itself uses it to recursively remove entries, and to find the file id a file had in the lhs parent inventory
[18:57] <lifeless> for the latter, path2id is better
[18:58] <lifeless> for the former, can you enlarge? I thought you used an inventory delta now?
[18:58] <jelmer> except I have to find the id based on the parent file id and the current file name
[18:58] <jelmer> lifeless, yes, I do use an inventory delta, but inventory deltas require all entries that are removed to be mentioned explicitly
[18:58] <lifeless> jelmer: ok
[18:59] <lifeless> so back to finding the id , you have old_parent_id, new_file_name ?
[18:59] <jelmer> actually, the other way around
[18:59] <jelmer> new_parent_id, old_filename
[19:00] <lifeless> this is in the case of a reparenting?
[19:01] <lifeless> and is old-filename the basename or path_from_tree_root
[19:04] <jelmer> old-filename is the basename
[19:05] <jelmer> I can work around it and only check children in case there is a reparenting
[19:05] <jelmer> but it surprises me a bit .children is so slow, even though it could've cached that data
[19:06] <jelmer> (since this is a from-scratch import, and all the inventory entries have been added in this session)
[19:06] <lifeless> jelmer: entry.children is dynamically populated
[19:07] <lifeless> apply_by_create starts with a fresh cache
[19:08] <lifeless> jelmer: so, you have new_parent_id, old_basename - and do you know what the old_parent_id was?
[19:08] <lifeless> (i'm trying to understand whats really going on here)
[19:08] <lifeless> we have a (parent_id, basename) -> file_id index in development4
[19:09] <lifeless> its not really exposed directly, but if it fits your needs we can make sure it is
[19:09] <jelmer> that is *exactly* what I would need here
[19:09] <lifeless> bbs I hope
[19:09] <lifeless> jelmer: do an isintance and have a poke at it then
[19:09] <lifeless> see CHKInventoryDirectory.children for example code
[19:10] <jelmer> Thanks
[19:27] <lifeless> jelmer: I wasn't sure it would help though because you have a new,old pair rather than new,new or old,old
[19:44] <jelmer> lifeless, what's the easiest way to query parent_id_basename_to_file_id for this sort of information?
[19:45] <jelmer> I see a iteritems(), but I'd rather avoid that if possible..
[19:54] <jelmer> lifeless, in a lot of cases, it will actually be old,old
[20:23] <jam> lifeless: I also noticed create_by_apply_delta being a bit slow for the mysql conversion, (about 50% of total time), and I'm guessing it is because we start with a fresh cache each time.
[20:24] <jam> Well, CHKRepo.apply_inventory_delta() reads the basis revision_tree from scratch
[20:24] <jam> which means it has to page in everything
[20:24] <jam> even if it just paged in that one
[20:25] <lifeless> jelmer: iteritems([(id, name)])
[20:25] <lifeless> jam: it needs a page cache not an entry cache I think, for this particular use case
[20:26] <jam> lifeless: I think we need *something* :)
[20:26] <lifeless> jam: sure
[20:26] <jam> There are several bits that may effect this
[20:27] <jam> 1) hash tries shrink the tree depth, so we probably won't have as many pages to bring in
[20:27] <lifeless> jam: what mysql conversion are you referencing in this conversation btw
[20:27] <jam> converting my mysql repo to --dev4
[20:27] <lifeless> kk
[20:27] <jam> The current tries are about 9 nodes deep
[20:27] <jam> w/ something like 18 node changes per commit
[20:27] <jam> let me double check
[20:28] <jam> average of 18.8 inv changes per revision
[20:29] <jam> average depth of 6 for the file_id=>entry
[20:29] <jam> and depth of 9 for pid,name=>file_id
[20:29] <jam> max depth 17, 19 respectively
[20:30] <lifeless> jam: lets hook in it and then perf test; we can change our minds :)
[20:30] <jam> with an average of 3.8 texts per commit
[20:30] <jam> 2) a page cache would make accessing the pages cheaper
[20:30] <jam> the problem is figuring out what the lifetime is
[20:31] <lifeless> jelmer: iteritems with a filter should be very efficient
[20:31] <jam> since it seems like it should be shared between all the CHKInventories at least
[20:31] <lifeless> jelmer: so you can actually query all your changes for a commit at once
[20:32] <jelmer> lifeless, thanks, iteritems() seems to work
[20:32]  * jelmer does another lsprof run
[20:33] <jelmer> unscientifically speaking, it seems to be faster
[20:34] <lifeless> :P
[20:38] <beuno> so...
[20:39] <beuno> anyone know what could cause this: http://paste.ubuntu.com/80499/
[20:39] <beuno> new error for me
[20:40] <lifeless> jelmer: jelmer for the 'recursive file id gathering'
[20:40] <beuno> ah, bug 303856
[20:40] <jelmer> lifeless, the recursive file id gathering doesn't affect performance I think
[20:41] <beuno> mwhudson, ping
[20:41] <jelmer> not significantly, I mean
[20:41] <jelmer> lifeless, It only gets called in the rare situation that you remove an entire subtree
[20:41] <jelmer> lifeless, the other code (parent_id,basename->file_id) gets called at least once for each changed file
[20:43] <lifeless> jelmer: yeah, you can write a recursive query just on that index though
[20:43] <lifeless> jelmer: that will avoid processing all the entry data for the removed files
[20:44] <jelmer> lifeless, ah, cool
[20:51] <lifeless> jelmer: so did you get an lsprof result with the updated code?
[20:51] <jelmer> almost done, just 300 more revisions left
[20:52]  * jelmer uses the vala svn repository for testing these days
[20:52] <lifeless> jelmer: what are vala's dimensios
[20:53] <jelmer> 2400 revisions, 817 files in the last revision
[20:54] <lifeless> so small :)
[20:54] <lifeless> beuno: update your bzr, you will be fine
[20:54] <jelmer> lifeless, well, this used to take a few hours with bzr-svn :-)
[20:55] <lifeless> jelmer: ah
[20:55] <lifeless> jelmer: I think you will like CommitBuilder.record_iter_changes
[20:55] <jelmer> anyway, results are in:
[20:56] <jelmer> seems this just isn't a signifcant factor anymore..
[20:56] <beuno> lifeless, doing that now...
[20:57] <beuno> crappy hotel wireless
[20:57] <jelmer> VF.get_record_stream(): 25%
[20:57] <jelmer> Repository.add_inventory_delta(): 18%
[20:57] <jelmer> VF.add_lines(): 30%
[20:59] <jelmer> svn file parsing, etc; 12%
[20:59] <jelmer> the rest is overhead from bzr-svn itself
[21:00] <jelmer> lifeless: in other words: please can we have Inventory.parent_basename2id() ?
[21:03] <lifeless> jem er, lets make it generalish
[21:05] <lifeless> e.g. Inventory.iter_child_ids([(parentid, None_or_asename...])
[21:06] <lifeless> the same interface as iteritems on the dict
[21:06] <lifeless> that can be easily implemented on Inventory, and on CHKInventory is a trivial thunk
[21:06] <jelmer> sure
[21:06] <lifeless> it (return self.parent_id_basename_to_childid.iteritems(query))
[21:08] <lifeless> jelmer: if you wanted to doa patch for bzr.dev adding that for Inventory, I'd be delighted to review it.
[21:09] <jelmer> Yeah, I'll have a look at doing that
[21:11] <jelmer> lifeless: create_by_apply_delta() is slow mainly because of CHKMap.apply_delta() (11%) and CHKInventory.__getitem__ (5.8%)
[21:13] <lifeless> so, the 11% is the delta application
[21:13] <lifeless> make sure you're running up to date brisbane-core
[21:13] <lifeless> uhm
[21:13] <lifeless> can you drill into those a little further? or post a callgrind file ?
[21:14] <jelmer> CHKMap.map() takes 7.6%
[21:15] <lifeless> within the 11%?
[21:15] <jelmer> No, out of the total
[21:15] <jelmer> so ~ 70% of CHKmap.apply_delta()
[21:16] <jelmer> http://samba.org/~jelmer/bzr/vala.callgrind
[21:16] <lifeless> yes, but is all that from apply_delta I mean
[21:16] <jelmer> yes, it's all from apply_delta
[21:16] <jelmer> unless I'm interpreting the view kcachegrind gives me wrong
[21:18] <lifeless> oh nice, kcachegrind got a facelift
[21:22] <lifeless> so 2000 delta calls becomes 14K map calls
[21:23] <lifeless> and 17K map calls
[21:23] <lifeless> sorry 14K __getitem__ calls before
[21:25] <lifeless> and that 17K becomes 58K with recursive handoffs
[21:25] <lifeless> but _iter_nodes is 70%
[21:25] <lifeless> (relative)
[21:25] <lifeless> and get_record_stream is 63% of that
[21:26] <lifeless> so if you wanted to do an experiment
[21:26] <lifeless> add a global cache (using the LRUCache) in chk_map.py
[21:27] <lifeless> whenever a page is read
[21:27] <lifeless> add the page under the CHK
[21:27] <lifeless> e.g. sha1:123456789012347890 -> page_bytes
[21:27] <lifeless> and in _iter_nodes lookup pages there first
[21:28] <lifeless> make it a decent size, say 2M == 512 items
[21:29] <jelmer> lifeless: It's going to take me a while to do that, given that I'm not familiar with that code. Do you perhaps have those changes ready ?
[21:29] <lifeless> let me take a quick toilet break and I'll whip it up
[21:29] <jelmer> or can you point out what exactly to insert, etc?
[21:29] <jelmer> thanks
[21:33] <`mousey> does anyone know if you can diff a single file when using tortoisebzr? everytime I try to diff revisions it shows a diff of every single modified file
[21:33] <beuno> http://tech.slashdot.org/article.pl?sid=08/12/04/1625226
[21:34] <beuno> gittorrent?
[21:41] <lifeless> `mousey: right mouse on the file perhaps?
[21:44] <lifeless> jam: ping
[21:44] <lifeless> jam: you use get_record_stream directly quite a bit; I'd like to avoid that if we could, helper function on $object - like the _read_bytes one perhaps
[21:45] <lifeless> jam: also, why do you use an adapter rather than just asking for get_bytes_as('fulltext') ? You have asked for them to be fulltextable always
[21:55] <lifeless> jelmer: pushing a read record cache now, writes-into-cache in a second
[22:00] <lifeless> jelmer: give it a spin
[22:09] <lifeless> jelmer: - still here?
[22:13] <arrbee> /topic
[22:21] <`mousey> lifeless: yeah, I've tried the right mouse, and it shows the revisions where the file was modifier, however doing a diff on 2 revisions will show every file that was changed between the 2 revisions
[22:22] <`mousey> i'm not sure if it's a bug or an intended feature
[22:22] <lifeless> if you're entering the historic diff from a single-file, I'd call it a bug, but if you're entering from a directory, its usualy to show the recursive diff
[22:23] <lifeless> jelmer: ping
[22:23] <markh> jam: did you have any luck with that box?
[22:23] <`mousey> yeah, it sounds like a bug. I'll see if I can fix it and supply a patch
[22:24] <`mousey> oh, also, is it possible to kick off an external merge program when diffing a single file from tortoisebzr?
[22:24] <lifeless> `mousey: I don't know
[22:36] <Rolly> you can with the command line
[22:37] <Rolly> maybe you could alias the diff command
[22:38] <lifeless> jelmer: ping
[22:42] <markh> jam: apparently not :)  it appears there is no dns
[22:43] <jelmer> lifeless, re
[22:44] <jelmer> lifeless, are you still there as well ? :-)
[22:44] <jelmer> VF.get_record_stream(): 25%
[22:44] <lifeless> jelmer: yes
[22:44] <jelmer> VF.add_lines(): 37%
[22:44] <jelmer> Repo.add_inventory_delta(): 10.78%
[22:44] <jelmer> svn overhead: 12%
[22:44] <jelmer> that's with your first revision
[22:44] <lifeless> jelmer: is that a significant improvement?
[22:44] <jelmer> lifeless, add_inventory_delta() is down 7%
[22:44] <lifeless> good
[22:44] <lifeless> ok, use the second rev, should be better still
[22:45] <mwhudson> beuno: hi
[22:46] <jelmer> lifeless, running
[22:50] <mwhudson> beuno: pong pong
[22:53] <lifeless> is it :cvar: to epydoc a class variable?
[22:54] <mwhudson> lifeless: yes
[23:05] <beuno> mwhudson, hi hi!
[23:05] <beuno> what can you tell me about bug 303856
[23:06] <beuno> I upgraded to the latest bzr nightly
[23:06] <beuno> but it still broke
[23:06] <beuno> I had to delete and push again
[23:06] <mwhudson> beuno: it was caused by an interaction between the new autopack code and some caching
[23:07] <beuno> mwhudson, and it
[23:07] <mwhudson> it was disabled in the 1.10 branch at least
[23:07] <beuno> it's suppose to be fixed in trunk?
[23:07] <jelmer> lifeless, last change doesn't seem to've helped much
[23:07] <mwhudson> dunno, bzr log --short | less and read i guess :)
[23:07] <lifeless> jelmer: interesting; are you sure the first run only had the first patch ? :)
[23:07] <jelmer> get_record_stream() -> 25%, add_inventory_delta() -> 11%, add_lines() -> 36%
[23:08] <jelmer> lifeless, perhaps the first run had the second patch as well
[23:08] <beuno> mwhudson, heh, ok
[23:08] <lifeless> jelmer: ok
[23:08] <beuno> mwhudson, did you happen to peak at my email about LH?
[23:08] <mwhudson> beuno: only peek
[23:09] <lifeless> so 25% reading from disk, 36% writing to disk (will incur a read as well, for every write), and 11% in metadata
[23:09] <beuno> right, so I haven't tricked anyone into fixing it yet
[23:09] <lifeless> jelmer: is < 50% of the time in bzrlib ?
[23:09] <jelmer> lifeless: basically, yep
[23:10] <mwhudson> beuno: i'll read it more thoroughly next week i guess
[23:11] <beuno> mwhudson, I hope to make time to fix it by then. We'll see
[23:11] <beuno> it would be good to rollout the latest version on LP soon-ish
[23:12] <lifeless> jelmer: well, I'm happy with bzrlib being less than half the time
[23:12] <lifeless> lower is better of course
[23:13] <jelmer> lifeless, well, 25%+36%+11% > 50%
[23:14] <lifeless> jelmer: add_lines uses get_record_stream
[23:14] <lifeless> jelmer: I'm not sure those figures are summable; can you post the callgrind?
[23:14] <jelmer> lifeless, according to callgrind it uses get_content_maps but not get_record_stream()
[23:14] <jelmer> http://samba.org/~jelmer/bzr/vala.callgrind
[23:15] <jelmer> back in ~30
[23:16] <lifeless> kk
[23:19] <lifeless> jelmer: there is definitely still overlap
[23:19] <lifeless> apply_delta -> get_record stream
[23:19] <lifeless> and -> add_lines
[23:20] <lifeless> why does fetch call get_record_stream?
[23:38] <lifeless> jelmer: I will be back, but afk for a bit (heading to hotel)
[23:57] <jelmer> lifeless, fetch calls get_record_stream() to fetch the data so it can apply the svn delta to it
[23:57] <jelmer> at the moment, I only have an implementation of the svn delta algorithm on streams
[23:58] <jelmer> bytestreams, that is, not line-streams
[23:59] <jelmer> if there's some easy way around that, I'm interested :-)