[00:11] it was caused by not setting whoami on the SERVER side [00:14] jelmer: ping [00:17] lifeless: pong [00:18] two questions re your 440K patch: [00:19] - I think you had a submit branch of bzr.dev, so its kinda hard to review :P (and note that [split-inventories][MERGE] is *much* ebetter, otherwise BB will pick it up [00:20] - why do y ou need create_by_apply_delta on regular inventories? (The Repository is the appropriate interface for making a new inventory normally [00:21] bbiab, car time [00:21] lifeless, bzr-svn's fetch uses the lhs parent inventory [00:45] jelmer: I'm trying bzr-svn 0.4 against svn 1.5.4 and while it builds, I'm having trouble running it. I'm getting "DLL load failed: The specified module could not be found." [00:46] awilkins, sorry, I'm not the right person for windows stuff [00:46] Hmm, appears to be DLL names [00:47] It's looking for libapr.dll when it's called libapr-1.dll [00:49] I've heard that problem being mentioned before [00:49] It seems to be the client.pyf [00:50] jelmer: I don't follow [00:50] lifeless, bzr-svn uses the lhs parent inventory when interpreting changes the server sends us [00:50] jelmer: yes... ? [00:50] The real libapr-1.dll is also being loaded [00:50] lifelss: fetch itself builds up an inventory delta [00:50] jelmer: this seems unrelated to my point [00:51] lifeless, argh, sorry, I mixed up two patches [00:51] lifeless, that first patch is moot if the second is accepted [00:52] Make Repository.add_inventory_delta() return the resulting inventory. [00:52] ? [00:52] yes, that's the second one [00:52] jelmer: +1 [00:52] jelmer: just land it [00:53] lifeless, thanks [01:17] jelmer: The reason is that the APR distribution in the win32 dev pack I'm using is not the correct one [01:17] Grr, squish those SVN devs [01:19] jelmer: They are shipping 1.3 libs with 1.5.4 but the headers are the 0.9.6 versions [01:24] LOL [01:26] markh: You around? [01:26] awilkins: I am - hi [01:26] Do you build bzr-svn at all (and use it?_ [01:27] I build it - I don't use it much as the svn repos I use have svn:externals and svn:eol-style :( [01:27] setup.py should be fairly accurate with build instructions [01:28] Heh, see above about not being able to use 1.5 version with windows (if you take the distributed library pack) [01:28] the "python based" installer? [01:28] They are shipping the 1.x series APR but the library pack has the 0.9 libs and headers [01:28] The python installer doesn't even build the pyd files [01:28] I'm building my own [01:28] yeah [01:29] If I want to access a 1.5 repo over ra_local I'm going to have to build APR in the morning. [01:29] awilkins: so where are you getting the apr binaries from? [01:30] The SVN guys host a "dev pack" for windows with libs and headers in [01:30] right - and you are building bzr-svn yourself? [01:30] The binaries in the 1.5.4 win32 distro are the 1.3x versions [01:30] the dev pack is 0.9 [01:31] Yes, building bzr-svn extensions using VC++ 2003 toolkit and it's libs and headers [01:31] Which corresponds to the compiler used to build python 2.5 [01:31] so - if you follow setup.py, one of those .zip files should have exactly matching apr binaries - or I'm still a little confused :) [01:31] The zip files do not have the right APR bits [01:31] don't try and use any other binaries with it [01:32] hrm [01:32] they do for me [01:32] They don't match the ones in the win32 distribution I downloaded [01:32] They match the 1.4.6 binaries [01:32] the bzr win32 distro? [01:32] But the 1.5.4 distro uses APR 1.3.x not 0.9.6 [01:33] Alas, the 1.5.4 dev pack has the 0.9.6 libs and headers [01:34] OK - I last built using 1.5.2 - that package should be OK. Note the "distro" of the version should not be relevant - you don't need anything other than the binaries bzr-svn (tries to) provide. [01:34] markh: I'm falling back on the path to load the DLLs from my installed SVN [01:35] right - which is the root of the problem [01:35] markh: And since the libs don't match the DLLs I have it's irrelevant anyway [01:35] Even if I copied them into the same folder [01:35] well - binaries build from the 1.5.2 dev pack appear to work fine and be internally consistent [01:35] Jolly good [01:35] I shall grab that one [01:35] so can you build with that? [01:35] cool [01:36] Oh hold on [01:36] * awilkins slaps forehead [01:36] Maybe if I take the Apache 2.0 version [01:37] * awilkins checks dev pack is not different on each page [01:37] awilkins: its quite possible 1.9 and later bzr binary builds have been made with the 1.4 svn devpacks [01:37] setup.py references them - even though it works with 1.5 - and that is quite possibly exactly what jam did when setting up the binary builds. [01:38] * markh tries to look [01:38] If we can get 1.5 working consistently we should use it - well, the reason I am trying is so I can get rid of the per-file properties when pushing over ra_local [01:39] I am an idiot [01:39] I've been using the apache 2.2 distro with the 2.0 devpack [01:39] It would be helpful if they named the archives more extensively [01:40] cool :) it turns out 1.9 should have 1.5.4 anyway [01:40] Right, now I know, I'm turning in, I shall have another go tomorrow. [01:41] * awilkins mumbles curses at management types who insist on SVN even though the rest of the team in question is using Bazaar === kiko is now known as kiko-zzz [03:43] markh, poolie: Kerguelen seems to be up, but seems to not have http access for me to download any data for packaging a 1.10rc1 installer. [03:44] Even browsing to "http://www.google.com" fails [03:44] jam: its not just IE broken? [03:44] IIRc that was one of the updates :S [04:00] hi [04:00] jam: any luck? [04:06] hi guys. does anybody know if the EPEL repo is going to get an updated version of bzr anytime soon? looks to me like it still has only 1.3 [04:12] markh: i just upload my project to launchpad [04:12] the Initial branch [04:13] if somebody wanna change something to a certain file what does he has to do? [04:13] let's say he took example.py and changed something... how does he commit these changes? [04:14] sorry i'm kind of confused [04:14] linrunix: they would create a local branch or checkout of your project, than commit those changes to that local branch or checkout. He would then use 'bzr send' to send the changes [04:14] or if he had permissions, he could push or checkin directly to launchpad [04:14] yes he has [04:14] but what does he do? [04:14] he download the branch to his computer [04:14] change the file [04:14] and commit? [04:14] the whole file [04:14] he probably should just do a "bzr cp lp:your_branch" [04:15] oops [04:15] "bzr co ..." [04:15] then make changes, then "bzr ci" - that is the easiest (but probably not the most flexible) [04:16] linrunix: check out http://doc.bazaar-vcs.org/bzr.dev/en/tutorials/tutorial.html [04:17] linrunix: or even better, http://bazaar-vcs.org/Workflows [04:17] very thanks [04:50] markh: sorry i was reading but it looks like I dont understand it very well [04:51] i'm just trying to learn how to use it so.. we can start coding from there [04:51] it could be argued there are too many options :) [04:51] options for ways to work [04:51] i got my friend to download the branch [04:51] how exactly? [04:51] let me give u how [04:52] bzr branch http://bazaar.launchpad.net/~linrunix/pyseleccion [04:53] now, does he have to upload his own branch to his user? [04:53] ok - so your friend can then make changes and commit as often as he likes. When he is ready to submit the changes, he should do a "bzr merge" (in case others have changed the branch since he pulled it), then "bzr push" [04:53] bzr push that's it [04:53] then your branch will be updated to have his changes [04:54] anyone doing "bzr pull" etc after that will get them [04:54] he doesnt have to have the branch in his user [04:54] he could push it somewhere else if he wants [04:55] but to update the actual branch [04:55] just a bzr push [04:55] if he can't push directly, or wants you to comments first, for example, he may well push to a private place. [04:55] ernie@intrepid:~/Desktop/trunk$ bzr push [04:55] bzr: ERROR: No push location known or specified. [04:55] yeah - first time you must say where you want to push to [04:55] bzr doesn't assume you want to push back to the same place [04:56] so taht will be http://bazaar.launchpad.net/~linrunix/pyseleccion right? [04:56] often you don't - you may well want to push somewhere private [04:56] the same url used to pull it would be fine [04:56] it will remember then - so "bzr push" will push back to the same spot thereafter [04:56] but you can obviously specify a new location any time too [04:57] like that bzr push bzr://bazaar.launchpad.net/~linrunix/pyseleccion [04:58] that should be fine - but the "lp:blah" version should work too [04:58] * markh can't recall exactly how they all expand... [05:15] bzr: ERROR: Cannot lock LockDir(lp-44825360:///~linrunix/pyseleccion/trunk/.bzr/branchlock): Transport operation not possible: readonly transport [05:15] any help? [05:16] too_short: how do I set the branch so another user can upload too? [05:19] linrunix: for the error, "bzr lp-login" [05:19] linrunix: to let other LP users commit to your branch, change it's owner to be a team rather than your user [05:20] ok [05:20] thanks [05:32] spiv: after changing a file i need to commit b4 pushing right? [05:32] -m "bla bla bla" right? [07:38] hi all [09:43] http://pastebin.com/m6516f466 that looks a bit odd. How did I get into that state? :) [10:02] jelmer: I resolved my bzr-svn with 1.5 svn issue ; I was using the Apache 2.2 binaries and the 2.0 devpack. [10:02] D'oh. === doko_ is now known as doko === mark1 is now known as markh [12:22] jelmer: Why does subvertpy have to go in bzrlib? [12:26] jelmer: Or somewhere other than plugins\svn .. I'm not sure I understand this so far [12:31] sven [12:54] This is odd, the exception handler isn't working [13:00] every time I've seen that it's been due to multiple copies of the same module, meaning multiple exception classes that *should* be the same... [13:00] bugger - past midnight - I'm officially late for bed :) [13:01] markh: It doesn't even throw [13:01] markh: If you feed it a wildcard handler it never triggers [13:01] It just spews an error to the console [13:01] It works on a console, just not in the bzr-svn code [13:01] grr [13:02] gnight anyway [13:02] Having hacked my way around it, I'm running into other issues [13:02] well - now I'm officially late, a little later wont hurt ;) [13:02] 0.4 is working well, this is 0.5 :-) [13:02] oh - still svn woes :( [13:03] Heh, the one I've run into looks more like a jelmer issue [13:03] 0.5 is supposed to cope with disorganised repos with evil folder histories better [13:04] I have a very large repo to test it on. [13:04] If it successfully manages to map all the branches in it ( to the point where pulling each one results in a small revision and not over 100MB), I'll be impressed [13:05] On the flipside I have a manager demanding I use SVN for an external repository (sob), so the highly-functional 0.4 support is very welcome [13:06] I have to migrate this huge backup folder of archived copies and turn it into a repository [13:06] Consultants are bad enough, but OLD consultants with no experience of VCS systems..... [13:08] Plus all the files are renamed with version numbers... and they all include each other. Grr. [13:08] Got that out of the way, mostly, I just need to bend my brain a bit further around which order this next set are in. [13:11] I'd go to bed [13:11] I'll pester jelmer when he's awake === sdboyer is now known as sdboyer|vurk [14:18] awilkins, hi [14:18] awilkins, subvertpy is under bzrlib.plugins.svn because it's included with the plugin [14:40] jelmer: I think I'm having a problem with the import code in svn\__init__.py [14:41] jelmer: For some reason it doesn't seem to be catching the first ImportError and trying the code in the except: block [14:43] jelmer: Once I hack my way around that, I run into a SubversionException for the case I'm trying. [14:43] what's the error exactly? [14:44] jelmer: THe import just reports "No module named subvertpy" at the console, the error doesn't percolate to the top of the stack (presumably the plugin loader is catching it) [14:47] awilkins, is subvertpy installed in the right location? [14:47] jelmer: It's nested in the svn plugin [14:47] I never actually install bzr-svn myself, just run it from ~/.bazaar/plugins [14:47] Ah, that may be why [14:47] bzrlib/plugins/svn/subvertpy/ ? [14:48] The "build" command of setup.py makes a different tree to the source [14:48] in the source, subvertpy is inside another subvertpy folder [14:49] I had to remove the path near line 87 to make that bit work [14:49] The join of the path of __file__ to subvertpy [14:50] In the source tree it's svn/subvertpy/subvertpy [14:50] I'll fix the imports, one sec [14:51] What really puzzled me was that the first import (before it extends the path) fails (as expected) but the exception isn't trapped ; it never gets to the bit where the path is extended. [14:51] Debugged with primitive "print" statements [14:53] http://paste.ubuntu.com/80356/ # this doesn't work and never prints "Caught the importerror" [14:54] But if you remove the exception handling and just go straight for extending the path it works fine. [14:54] that import should work fine in your case [14:54] because it's a relative import [14:55] it doesn't work in a couple of other cases (when importing subvertpy from bzrlib.plugins.svn.mapping3.scheme, for example) [14:56] Well, it should work, and printing __file__ confirms that it's the right file being run, but it doesn't [15:04] jelmer: Maybe I should look at stack traces more often, the problem is in ra.py [15:05] jelmer: Adding the path in svn\__init__.py just masks it [15:15] jelmer: It seems that a number of imports in subvertpy are using the subvertpy namespace to import things that are inside the subvertpy namespace - but it can't find it because it's not on the path. [15:15] Once you fix these up to be relative to subvertpy it works fine ( is doing ' from __init__ import ' acceptable ?) [15:15] awilkins, that's what the sys.path.insert call is supposed to fix [15:16] jelmer: That path call only happens if the initial import fails [15:16] it's not correct in your case though [15:16] awilkins, relative imports don't work, since it means you can't import any subvertpy stuff from python modules that are not in the top-level code of bzr-svn [15:16] as there is no ".." in imports [15:17] Ok, so it needs to always add subvertpy to the path regardless then? [15:18] or use bzrlib.plugins.svn.subvertpy everywhere [15:18] awilkins, to work around it, you should be able to move subvertpy to the top-level for now [15:23] markh: No luck on kerguelen. IE is busted with "No Module Found", but bzr with http:// urls is giving Couldn't resolve host 'bazaar-vcs.org'. [15:23] Could be a DNS issue [15:26] jelmer: Moving that package also seems to fix my SubversionException [15:37] awilkins, so everything works now? [15:37] jelmer: It would appear to. [16:07] jelmer: Hmm, that's disapointing ; something in 0.4 is a real memory hog [16:07] I tried pushing a branch to a fresh svn repo and on reaching the third revisions it eats 1.3GB of memory [16:07] Now, it is a big revision, but the entire bzr repo is less than 65MB [16:07] What about 0.5 ? [16:08] I think 0.5 was doing it to (but I didn't know whether to write that off as running against a Bazaar with no extensions built) [16:09] It rapidly pushes the machine into swap at which point it just crawls [16:09] Last I tried it, pulling big repos wasn't a problem [16:09] But maybe pushing is different [16:10] I mean, pulling still eats a few 100MB [16:10] I'm using 0.4 with 1.9 and 0.5 with bzr.dev (no extensions built) [16:10] the code that handles some of this stuff is a lot different between 0.4 and 0.5 [16:10] although 0.4 shouldn't be leaking that much either [16:11] Does 1.10 rc1 have "foreign" ? [16:11] yes [16:11] I suppose it's marked as compatble in the code, that answers my question [16:11] Bah, no installer anyway :-) [16:12] Importing bzr-svn or bzr.dev into svn works fine here, btw, no heavy memory usage [16:12] I wonder what's making it problematic for you [16:12] It's a lot more paths and size than bzr.dev [16:13] yo [16:13] bzr.dev is 1013 files and ~15MB [16:13] hey lifeless [16:14] lifeless, travelling in some strange part of the world or just awake at night? [16:14] This is 6237 paths and 62.5 MB [16:14] uds [16:14] And most of that happens in a single revision (which it seems to be having trouble with) [16:14] ahh, of course [16:15] I think that one revision has about 3600 paths in it and most of that data [16:15] awilkins, ah, that may be slow indeed [16:15] Pulling similar sizes FROM svn repos (and larger) doesn't seem to cause the same explosion of memory [16:20] all texts for the commit have to be kept in memory during push [16:20] all changed texts, that is [16:20] Ow [16:20] Hmm, that's still only 65 MB though? [16:21] I realise that overhead is important but... 1.25 GB of it? [16:22] jelmer: all at once? Can't you callback on each separately? [16:22] awilkins: I suspect there's more going on [16:22] As do I :-) [16:22] lifeless, it can be done a bit more lazily [16:23] It consumes the memory very quickly on reaching that third revision [16:23] lifeless, but I doubt that's really the problem here [16:26] jelmer: There are quite a few merges in either direction on this branch, would that affect it? [16:26] Well, not by the third revision (duh) [16:27] Right, time for a shower and a quick forage for wife-food [16:27] I shall come back later === awilkins is now known as awilkins_away [16:29] awilkins_away, should't matter [16:29] python really sucks when it comes to debugging memory usage :-/ [16:30] * Tak s/debugging //, get kicked from channel [16:33] Tak: it is higher on memory usage than some lower level languages, due to various choices; but it should be ok for nearly everything, as long as you don't leak :) [16:33] (where a leak could be contained to a single function but still be very large) [16:33] I know, I'm only kidding [16:34] besides, I'm a ruby fan, I don't have any room to complain [16:34] heh! [16:42] jelmer: iirc I've been happy with http://guppy-pe.sourceforge.net/ in the past [16:50] can I get a log for a folder in a branch? am I stupid for wanting to do so? [16:52] evarlast: I was wanting to do that just the other day. [16:52] I didn't find a way to do so, though. [16:53] hmm - it works for a branch from a svn repo [16:54] right, you can bzr log [16:54] but not bzr log [16:54] or, if you do, you just get a single revision for when the dir was created. [16:54] not when all files within that dir were last modified. [16:55] I mean, it works at the directory level for a branch from an svn repo [16:56] lifeless: any chance you could look at: http://bundlebuggy.aaronbentley.com/project/bzr/request/%3C4935F649.2010706%40arbash-meinel.com%3E [16:56] I'd like to base the rest of my work on it [16:56] since it makes the map/unmap "stable" [16:57] jam: sure [16:57] thx [16:57] upgrading to intrepid right now, and I'm on leave today [16:58] sure [16:58] It isn't something you have to do [16:58] if you think its ok, I suggest you build on iti anyhow [16:58] but since martin, you, and spiv are all afk... [16:58] or even land it, and we review it later [17:00] I guess I'll go for that, then. [17:08] Any instructions on removing a damaged knit file w/o consequence? I have determined that the contents of the file are something I removed from the repository long ago. [17:08] I'm just looking for a way to cascade it's removal without screwing up bzr. [17:09] kjcole: so, if the file is used by *any* revision still in the repository, bzr will be unhappy whena ccessing that revision [17:09] the easiest way to have a completely clean repo is to make a new one and branch everything into it [17:10] lifeless: I tried branching. No luck. [17:10] kjcole: it errored? [17:10] lifeless: Yep. [17:11] kjcole: ok. do you have any idea of how the damage occured? [17:11] lifeless: "bzr check" reveals a zlib.error "invalid distance too far back" [17:12] lifeless: No idea how it occurred, but it occcured a long way back: The file in question was removed in revision 3, and the repository's up to revision 80. [17:13] kjcole: ok. do you have another copy of it anywhere? [17:13] lifeless: I only learned of it while trying to move the repository from an old dapper machine w/ bzr 0.8.2 to a hardy machine w/ a more current bzr. [17:13] lifeless: Of the repository? No. [17:13] ok [17:14] LarstiQ, I couldn't get guppy to work and the project appears to've been dead since 2005 [17:14] so you have a damaged data file for revision's 1 and 2 [17:15] and the rest is fine. [17:15] (as far as you know) [17:15] lifeless: I guess. Since only a single knit file gives an error, and there are two other knit files with the same starting name... [17:16] lifeless: (I'm gonna check the dates on those files. BRB) [17:19] lifeless: all created on the same day. (It's been a while, but what I believe I did was bzr init, add everything in an existing directory tree, commit, and then weed out the few things I didn't want. [17:19] lifeless: (all on the same day). [17:20] ok [17:20] so I suspect a disk bit error, as its data that bzr has had no reason to touch for a very long time [17:21] lifeless: The damaged file isn't something I would have changed -- it was part of the Python Library Reference docs that was inadvertently sitting in the tree. [17:21] kjcole: the easiest thing to do would be to trim revision 1 and 2 - by rebasing the rest of the branch [17:22] kjcole: exactly, its data that was probably written fine, but not read from for several years [17:22] lifeless: This sounds promising. Thanks. [17:23] I believe someone posted to the list a recipe to do this, they didn't have a damaged repo,but rather had text they couldn't show people and needed to eliminate; same principle though [17:23] jelmer: hmm [17:23] jelmer: I'm reasonably sure it was guppy that I used a couple of months ago to debug memory issues. [17:23] jelmer: but I'll look it up after dinner. [17:25] lifeless: Oook. If worse comes to worse, since I don't intend to move backwards on this, I'll try to generate a history report (log plus list of files affected) and just "rm -rf .bzr" and start again. [17:25] lifeless: But I wanted to salvage what I could first. [17:25] kjcole: you should be able to salvage everything but the damaged revs. [17:26] lifeless, hey hey. Are you in SFO yet? [17:26] lifeless: to the list archives, then -- I hope there was a decent subject line. ;-) [17:26] beuno: yes [17:26] lifeless, cool. We're heading towards the UDS hotel today [17:27] lifeless: Thanks again. Later. [17:27] kjcole: anytime [17:28] beuno: cool === mw_ is now known as mw [17:32] jelmer: regarding bug #303959, I'm done with the fixes I could make to bzr. Have you seen my last comment ? Does that sounds right ? [17:32] Launchpad bug 303959 in bzr "missing redirect handler: no repository found when pulling a branch from bzr-playground" [High,Fix committed] https://launchpad.net/bugs/303959 [17:33] lifeless: it turns out it was a bit more work than anticipated but I think the result was worth it (IMHO) [17:33] vila: the key question for me is 'can jc2k pull' [17:34] if jc2k == bug reporter: return True [17:35] John Carr [17:35] See the last bug comment for a more elaborate answer [17:35] also, 'return jc2k == bug_reporter' [17:35] hehe [17:37] I've read it now [17:37] yes, I concur, bzr-svn should handle foo->foo/ as 'cannot be a svn repo' [17:38] also, 'ireturn jc2k == bug_reporter' doesn't cover the implicit else: return None :) [17:39] it returns False istead [17:39] which isn't the answer I wanted to give :) [17:41] o_O [17:41] bug_reporter.state == 'amused' [17:41] Jc2k: did you try my patch ? [17:41] vila: not yet. just got home, in a bit of a rush.. [17:41] jelmer: ping [17:42] lifeless, pong [17:42] bug_reporter.state == 'checking_later_I_promise' :-) [17:42] vila: is it in bzr.dev? [17:42] otherwise me needs somewhere to pull or grab patch from [17:42] not yet, but it is in the branch associated with th bug: lp:~vila/bzr/303959-redirection [17:44] ooh shiny [17:44] Jc2k: feeback very welcome since I can't reproduce the bug with my actual setup [17:44] will spin in next 45 minutes if i can, otherwise it will be on my todo for asap [17:45] vila: am I right that in theory this bug is fixable just via bzr-svn? [17:45] [the actual reported fault, I mean] [17:45] lifeless, yes [17:46] jelmer: hi [17:46] lifeless: totally [17:46] jelmer: so pinging re this bug; also how did split-inv work for you; [17:46] lifeless, split inv seems to work fine, especially now those two changes are in [17:46] I also noticed that CHKInventory.has_id() is quick while CHKInventory.__contains__() isn't [17:46] faster/slower/can't-tell-yet? [17:47] In percentages, it's definitely faster (smaller percent of work is now in updating the inventory) [17:48] jelmer: __contains__ isn't defined on CHKInventory :P [17:48] lifeless, I was seeing references to __iter__ [17:48] oh right [17:48] probably an implicit behaviour [17:48] I'll move the __contains__ definition [17:49] lifeless, the remaining culprits are: [17:49] - Repository.add_inventory_delta() (22%) [17:51] - VersionedFile.add_lines() (15.59%) [17:51] - VersionedFile.get_record_stream() (13%) [17:53] 22% is still quite high [17:53] fix for __contains__ pushed [17:54] what does that 22% break down into? [17:55] let me just run it again locally [18:31] lifeless, holy crap, why are you awake!? [18:31] I'm in mountain view [18:32] lifeless, okay, that makes more sense. :) [18:32] awake is not the right term though [18:32] lifeless, about the memory middleware, I can land it, but it's a SERIOUS hack. [18:32] rockstar: is it in its own module? [18:33] Basically, it reads it's own memory usage from proc [18:33] lifeless, yes, loggerhead requires a flag to use it. [18:33] rockstar: bzr has a function to do that btw [18:33] lifeless, new results: [18:33] lifeless, bzr has a serious lack of documentation. :) [18:33] add_inventory_delta(): 13% [18:34] CHKInventory.children: 36.12% [18:34] VF.add_lines(): 21% [18:34] lifeless, one of these days, I'm going to start doing things on my TODO list, and writing docs for bzr is on there. [18:34] VF.get_record_stream(): 14.46% [18:35] rockstar: we have docs :> problem is knowing what to look for :) [18:35] the rest is (obviously) bzr-svn overhead [18:35] rockstar: bzrlib.trace.debug_memory [18:35] lifeless, I think that Launchpad Code team kinda has that problem too. Those guys are REAL slackers. :) [18:35] rockstar: probably wants patching to accept a callback function [18:36] lifeless, great, I'll take a look at it. [18:36] anyhow [18:36] I woud land it if its ugliness is isolated [18:37] jelmer: are those percent-of-total? [18:38] and do you mean CHKInventoryDirectory.children ? === Mario_ is now known as pygi [18:55] lifeless, yes [18:55] lifeless, they are percentages of total [18:56] what are the callers of .children? [18:57] bzr-svn itself uses it to recursively remove entries, and to find the file id a file had in the lhs parent inventory [18:57] for the latter, path2id is better [18:58] for the former, can you enlarge? I thought you used an inventory delta now? [18:58] except I have to find the id based on the parent file id and the current file name [18:58] lifeless, yes, I do use an inventory delta, but inventory deltas require all entries that are removed to be mentioned explicitly [18:58] jelmer: ok [18:59] so back to finding the id , you have old_parent_id, new_file_name ? [18:59] actually, the other way around [18:59] new_parent_id, old_filename [19:00] this is in the case of a reparenting? [19:01] and is old-filename the basename or path_from_tree_root [19:04] old-filename is the basename [19:05] I can work around it and only check children in case there is a reparenting [19:05] but it surprises me a bit .children is so slow, even though it could've cached that data [19:06] (since this is a from-scratch import, and all the inventory entries have been added in this session) [19:06] jelmer: entry.children is dynamically populated [19:07] apply_by_create starts with a fresh cache [19:08] jelmer: so, you have new_parent_id, old_basename - and do you know what the old_parent_id was? [19:08] (i'm trying to understand whats really going on here) [19:08] we have a (parent_id, basename) -> file_id index in development4 [19:09] its not really exposed directly, but if it fits your needs we can make sure it is [19:09] that is *exactly* what I would need here [19:09] bbs I hope [19:09] jelmer: do an isintance and have a poke at it then [19:09] see CHKInventoryDirectory.children for example code [19:10] Thanks [19:27] jelmer: I wasn't sure it would help though because you have a new,old pair rather than new,new or old,old [19:44] lifeless, what's the easiest way to query parent_id_basename_to_file_id for this sort of information? [19:45] I see a iteritems(), but I'd rather avoid that if possible.. [19:54] lifeless, in a lot of cases, it will actually be old,old [20:23] lifeless: I also noticed create_by_apply_delta being a bit slow for the mysql conversion, (about 50% of total time), and I'm guessing it is because we start with a fresh cache each time. [20:24] Well, CHKRepo.apply_inventory_delta() reads the basis revision_tree from scratch [20:24] which means it has to page in everything [20:24] even if it just paged in that one [20:25] jelmer: iteritems([(id, name)]) [20:25] jam: it needs a page cache not an entry cache I think, for this particular use case [20:26] lifeless: I think we need *something* :) [20:26] jam: sure [20:26] There are several bits that may effect this [20:27] 1) hash tries shrink the tree depth, so we probably won't have as many pages to bring in [20:27] jam: what mysql conversion are you referencing in this conversation btw [20:27] converting my mysql repo to --dev4 [20:27] kk [20:27] The current tries are about 9 nodes deep [20:27] w/ something like 18 node changes per commit [20:27] let me double check [20:28] average of 18.8 inv changes per revision [20:29] average depth of 6 for the file_id=>entry [20:29] and depth of 9 for pid,name=>file_id [20:29] max depth 17, 19 respectively [20:30] jam: lets hook in it and then perf test; we can change our minds :) [20:30] with an average of 3.8 texts per commit [20:30] 2) a page cache would make accessing the pages cheaper [20:30] the problem is figuring out what the lifetime is [20:31] jelmer: iteritems with a filter should be very efficient [20:31] since it seems like it should be shared between all the CHKInventories at least [20:31] jelmer: so you can actually query all your changes for a commit at once [20:32] lifeless, thanks, iteritems() seems to work [20:32] * jelmer does another lsprof run [20:33] unscientifically speaking, it seems to be faster [20:34] :P [20:38] so... [20:39] anyone know what could cause this: http://paste.ubuntu.com/80499/ [20:39] new error for me [20:40] jelmer: jelmer for the 'recursive file id gathering' [20:40] ah, bug 303856 [20:40] Launchpad bug 303856 in bzr "caching writes to pack repository causes ShortReadvError on pushing stacked branch" [High,Fix committed] https://launchpad.net/bugs/303856 === thumper_laptop is now known as thumper [20:40] lifeless, the recursive file id gathering doesn't affect performance I think [20:41] mwhudson, ping [20:41] not significantly, I mean [20:41] lifeless, It only gets called in the rare situation that you remove an entire subtree [20:41] lifeless, the other code (parent_id,basename->file_id) gets called at least once for each changed file [20:43] jelmer: yeah, you can write a recursive query just on that index though [20:43] jelmer: that will avoid processing all the entry data for the removed files [20:44] lifeless, ah, cool [20:51] jelmer: so did you get an lsprof result with the updated code? [20:51] almost done, just 300 more revisions left [20:52] * jelmer uses the vala svn repository for testing these days [20:52] jelmer: what are vala's dimensios [20:53] 2400 revisions, 817 files in the last revision [20:54] so small :) [20:54] beuno: update your bzr, you will be fine [20:54] lifeless, well, this used to take a few hours with bzr-svn :-) [20:55] jelmer: ah [20:55] jelmer: I think you will like CommitBuilder.record_iter_changes [20:55] anyway, results are in: [20:56] seems this just isn't a signifcant factor anymore.. [20:56] lifeless, doing that now... [20:57] crappy hotel wireless [20:57] VF.get_record_stream(): 25% [20:57] Repository.add_inventory_delta(): 18% [20:57] VF.add_lines(): 30% [20:59] svn file parsing, etc; 12% [20:59] the rest is overhead from bzr-svn itself [21:00] lifeless: in other words: please can we have Inventory.parent_basename2id() ? [21:03] jem er, lets make it generalish === bac is now known as bac_afk [21:05] e.g. Inventory.iter_child_ids([(parentid, None_or_asename...]) [21:06] the same interface as iteritems on the dict [21:06] that can be easily implemented on Inventory, and on CHKInventory is a trivial thunk [21:06] sure [21:06] it (return self.parent_id_basename_to_childid.iteritems(query)) [21:08] jelmer: if you wanted to doa patch for bzr.dev adding that for Inventory, I'd be delighted to review it. [21:09] Yeah, I'll have a look at doing that [21:11] lifeless: create_by_apply_delta() is slow mainly because of CHKMap.apply_delta() (11%) and CHKInventory.__getitem__ (5.8%) [21:13] so, the 11% is the delta application [21:13] make sure you're running up to date brisbane-core [21:13] uhm [21:13] can you drill into those a little further? or post a callgrind file ? [21:14] CHKMap.map() takes 7.6% [21:15] within the 11%? [21:15] No, out of the total [21:15] so ~ 70% of CHKmap.apply_delta() [21:16] http://samba.org/~jelmer/bzr/vala.callgrind [21:16] yes, but is all that from apply_delta I mean [21:16] yes, it's all from apply_delta [21:16] unless I'm interpreting the view kcachegrind gives me wrong [21:18] oh nice, kcachegrind got a facelift [21:22] so 2000 delta calls becomes 14K map calls [21:23] and 17K map calls [21:23] sorry 14K __getitem__ calls before [21:25] and that 17K becomes 58K with recursive handoffs [21:25] but _iter_nodes is 70% [21:25] (relative) [21:25] and get_record_stream is 63% of that [21:26] so if you wanted to do an experiment [21:26] add a global cache (using the LRUCache) in chk_map.py [21:27] whenever a page is read [21:27] add the page under the CHK [21:27] e.g. sha1:123456789012347890 -> page_bytes === bac_afk is now known as bac [21:27] and in _iter_nodes lookup pages there first [21:28] make it a decent size, say 2M == 512 items [21:29] lifeless: It's going to take me a while to do that, given that I'm not familiar with that code. Do you perhaps have those changes ready ? [21:29] let me take a quick toilet break and I'll whip it up [21:29] or can you point out what exactly to insert, etc? [21:29] thanks [21:33] <`mousey> does anyone know if you can diff a single file when using tortoisebzr? everytime I try to diff revisions it shows a diff of every single modified file [21:33] http://tech.slashdot.org/article.pl?sid=08/12/04/1625226 [21:34] gittorrent? [21:41] `mousey: right mouse on the file perhaps? [21:44] jam: ping [21:44] jam: you use get_record_stream directly quite a bit; I'd like to avoid that if we could, helper function on $object - like the _read_bytes one perhaps [21:45] jam: also, why do you use an adapter rather than just asking for get_bytes_as('fulltext') ? You have asked for them to be fulltextable always [21:55] jelmer: pushing a read record cache now, writes-into-cache in a second [22:00] jelmer: give it a spin [22:09] jelmer: - still here? [22:13] /topic [22:21] <`mousey> lifeless: yeah, I've tried the right mouse, and it shows the revisions where the file was modifier, however doing a diff on 2 revisions will show every file that was changed between the 2 revisions [22:22] <`mousey> i'm not sure if it's a bug or an intended feature [22:22] if you're entering the historic diff from a single-file, I'd call it a bug, but if you're entering from a directory, its usualy to show the recursive diff [22:23] jelmer: ping [22:23] jam: did you have any luck with that box? [22:23] <`mousey> yeah, it sounds like a bug. I'll see if I can fix it and supply a patch [22:24] <`mousey> oh, also, is it possible to kick off an external merge program when diffing a single file from tortoisebzr? [22:24] `mousey: I don't know [22:36] you can with the command line [22:37] maybe you could alias the diff command [22:38] jelmer: ping [22:42] jam: apparently not :) it appears there is no dns [22:43] lifeless, re [22:44] lifeless, are you still there as well ? :-) [22:44] VF.get_record_stream(): 25% [22:44] jelmer: yes [22:44] VF.add_lines(): 37% [22:44] Repo.add_inventory_delta(): 10.78% [22:44] svn overhead: 12% [22:44] that's with your first revision [22:44] jelmer: is that a significant improvement? [22:44] lifeless, add_inventory_delta() is down 7% [22:44] good [22:44] ok, use the second rev, should be better still [22:45] beuno: hi [22:46] lifeless, running [22:50] beuno: pong pong [22:53] is it :cvar: to epydoc a class variable? [22:54] lifeless: yes [23:05] mwhudson, hi hi! [23:05] what can you tell me about bug 303856 [23:05] Launchpad bug 303856 in bzr "caching writes to pack repository causes ShortReadvError on pushing stacked branch" [High,Fix committed] https://launchpad.net/bugs/303856 [23:06] I upgraded to the latest bzr nightly [23:06] but it still broke [23:06] I had to delete and push again [23:06] beuno: it was caused by an interaction between the new autopack code and some caching [23:07] mwhudson, and it [23:07] it was disabled in the 1.10 branch at least [23:07] it's suppose to be fixed in trunk? [23:07] lifeless, last change doesn't seem to've helped much [23:07] dunno, bzr log --short | less and read i guess :) [23:07] jelmer: interesting; are you sure the first run only had the first patch ? :) [23:07] get_record_stream() -> 25%, add_inventory_delta() -> 11%, add_lines() -> 36% [23:08] lifeless, perhaps the first run had the second patch as well [23:08] mwhudson, heh, ok [23:08] jelmer: ok [23:08] mwhudson, did you happen to peak at my email about LH? [23:08] beuno: only peek [23:09] so 25% reading from disk, 36% writing to disk (will incur a read as well, for every write), and 11% in metadata [23:09] right, so I haven't tricked anyone into fixing it yet [23:09] jelmer: is < 50% of the time in bzrlib ? [23:09] lifeless: basically, yep [23:10] beuno: i'll read it more thoroughly next week i guess [23:11] mwhudson, I hope to make time to fix it by then. We'll see [23:11] it would be good to rollout the latest version on LP soon-ish [23:12] jelmer: well, I'm happy with bzrlib being less than half the time [23:12] lower is better of course [23:13] lifeless, well, 25%+36%+11% > 50% [23:14] jelmer: add_lines uses get_record_stream [23:14] jelmer: I'm not sure those figures are summable; can you post the callgrind? [23:14] lifeless, according to callgrind it uses get_content_maps but not get_record_stream() [23:14] http://samba.org/~jelmer/bzr/vala.callgrind [23:15] back in ~30 [23:16] kk [23:19] jelmer: there is definitely still overlap [23:19] apply_delta -> get_record stream [23:19] and -> add_lines [23:20] why does fetch call get_record_stream? [23:38] jelmer: I will be back, but afk for a bit (heading to hotel) === TheMuso_ is now known as TheMuso [23:57] lifeless, fetch calls get_record_stream() to fetch the data so it can apply the svn delta to it [23:57] at the moment, I only have an implementation of the svn delta algorithm on streams [23:58] bytestreams, that is, not line-streams [23:59] if there's some easy way around that, I'm interested :-)