[00:16] jelmer: I wouldn't use chk_map unless you need updatable values [00:16] jelmer: if you just need key:value and they are immutable use BTreeIndex [00:17] lifeless: I need a big path -> tuple map that's different per revision but should have relatively small (line-based) differences between revisions [00:17] lifeless: very much like an inventory [00:17] s/line-based/path-based/ [00:17] ok for that you want chkmap [00:18] you may want to look at chk inventories parent_id,basename->child_id map [00:18] as a related use of chk_map [00:18] we use two maps [00:18] lifeless: What about the Pack container bits from bzr-search, where should i be looking? [00:18] one for paths [00:18] one for the data [00:18] b.p.search.index [00:19] basically to work with chkmap you need a VersionedFiles object that supports hash naming [00:19] it might be possible to convince knits to do this [00:19] I'm not sure it would be a good idea ;) [00:19] so the layering is: [00:19] chkmap->VF [00:20] (KnitVersionedFiles|GroupCompressVersionedFiles)->(collection of btreeindex + pack files) [00:20] and its that last bit that bzr-search has a smallish implementation of [00:21] thanks [00:21] this is all a bit new to me, VersionedFiles are still very much an abstract thing to me [00:26] ok [00:27] I think you should learn about the internals here [00:27] its hampering bug fixing [00:27] also our layers could be better [00:27] got 20 minutes? [00:29] jelmer: ^ [00:31] lifeless: if you can give a quick introduction that might help [00:31] thats my plan [00:31] so [00:31] one sec [00:36] jelmer: ok [00:36] jelmer: so lets start with VersionedFiles [00:37] lifeless: ok [00:37] lifeless: fwiw I'm familiar with the API [00:37] lifeless: just not how it works underneath [00:38] sure [00:38] so [00:38] Weaves work by having a thunk class, not very interesting. [00:39] more interesting is how Knits and Packs work for the KnitVersionedFiles clas [00:39] bring up knit.py [00:40] look for make_file_factory [00:40] right, I think that's what I'm using right now for the file id map [00:40] hi people [00:40] so this creates the two key objects KVF needs to work [00:41] hello all [00:41] an index, which is responsible for locating bytes and storing the graph [00:41] is there documentation on using the merge sorted graph? [00:41] and an access object, which is responsible for taking index descriptions of content and reading them back, or for writing and returning index descriptions [00:41] hi thumper poolie no [00:42] hi thumper, poolie [00:42] jelmer: so _KndxIndex is the implemenetation of an index based on ,kndx files, and _KnitGraphIndex is the implementation of same building on the 'GraphIndex' interface [00:43] likewise _KnitKeyAccess and _DirectPackAccess [00:43] * jelmer nods [00:44] so, to use chk_map on .kndx and .knit files, we'd create a KVF much like make_file_factory does [00:44] now if you look at KnitVersionedFiles itself [00:44] this is the common logic that actually cares about knit deltas, their constraints and needs and so on [00:45] thumper: i can probably tell you more than you want to know :) [00:45] so it knows about delta chains (we'd want 0), annotations (0 again) [00:46] it also knows about fallbacks to other VF objects, which is much of how stacking works on the client [00:46] what is the easiest way to get the revision_id for any mainline (non-dotted) revno? [00:46] branch.revno2revid [00:47] lifeless: no attribute called ... [00:47] jelmer: ok, I'm not sure where to drill down into; we can look at how _DirectPackAccess and _KnitGraphIndex work [00:47] thumper: its roughly that [00:47] thumper: pydoc and / for the win [00:47] lifeless: there are revision_id_to_revno [00:48] jelmer: or we can look at actually wiring up chk_map to .knit files for you [00:48] lifeless: but not the other way around [00:48] lifeless: let's look at the latter first [00:48] mwhudson: where is your pydoc stuff for bzr? [00:48] lifeless: actually working with this code should help [00:48] dotted_revno_to_revision_id and get_rev_id and [00:49] jelmer: ok [00:49] so [00:49] lets pop over to chkmap [00:49] CHKMap wants a store, root_key and a search key func [00:49] the search key func is for hashed keys [00:49] we can ignore that for now === abentley1 is now known as abentley [00:50] root_key is something you need to store elsewhere [00:50] its the 'id' of a given version of a dict stored in a chk_map [00:50] so you'll need someplace to put that. perhaps another knit, or whatever. [00:51] to bootstrap, when you first start, you use from_dict [00:51] so - [00:51] my_map = CHKMap.from_dict(store, {}) is the minimal call [00:52] that will get a non-splitting map with single-element tuples as keys [00:52] what you probably want is maximum_size=4096 [00:53] so to wire it up [00:53] store = get_a_versioned_files() [00:53] empty_map = CHKMap.from_dict(store, {}, maximum_size=4096) [00:53] * jelmer nods [00:53] then you can stash stuff in it. [00:54] you can individually map() and unmap() [00:54] however its much better to use apply_delta [00:54] because map and unmap write on each change [00:54] apply_delta applies all the changes and then writes [00:55] the get_a_versioned_files call should return a VF whose key width is 1 [00:55] and no graph [00:55] so lets pop back to knit.py [00:57] access = _KnitKeyAccess(transport, mapper); mapper=ConstantMapper('myknit'); index = _KndxIndex(transport, mapper, lambda:None, lambda:True, lambda:True) [00:57] You want ConstantMapper because you want the same .knit and .kndx file for all keys [00:58] ok [00:58] the lambdas are for handling lock/unlock callbacks [00:58] you should wire them up more carefully [00:58] kudos to the bzr developers for making an api even I find easy! [00:59] before using with concurrent processes [00:59] then its just [00:59] store = KnitVersionedFiles(index, access, annotated=False, max_delta_chain=0) [01:00] lifeless: ok, let me give that a try [01:02] mwhudson: btw, I've "fixed" the bzr-git importing strange modes issue [01:02] thumper: http://starship.python.net/crew/mwh/bzrlibapi/ [01:02] jelmer: cool [01:02] mwhudson: it just means it won't be able to regenerate the sha map correctly if you remove it [01:02] mwhudson: it'll also warn if it hits strange modes [01:03] mwhudson: same thing for the XML-invalid characters that are escaped [01:03] jelmer: i'm not sure i understand [01:03] jelmer: remove what? [01:03] mwhudson: git.[t]db [01:04] mwhudson: which contains the information of which bzr revid/fileid maps to which git sha [01:04] jelmer: ok, but i still don't understand what that has to do with strange modes [01:04] mwhudson: git works with shas everywhere [01:04] mwhudson: so when we talk to it we need to tell it which shas we have [01:05] jelmer: currently, the git.db does not persist between runs [01:05] mwhudson: we generate those shas from the bzr data we have by converting to git objects and shaing those [01:05] mwhudson: if we can't regenerate the exact same git object we can't find a particular sha [01:06] jelmer: ahh [01:06] jelmer: so "persisting the sha data between runs" has become a bit more important? [01:07] mwhudson: yeah, a bit. previously we would just flat out refuse to import anything with strange file modes, now we just warn [01:07] mwhudson: we could store the file modes in revision properties at some point [01:07] mwhudson: in which case we could regenerate the sha map correctly again [01:08] mwhudson: s/file modes/unusual file modes/ (they're really rare) [01:10] jelmer: yes but the kernel is a pretty obvious target, i think we need to support that [01:10] (ultimately, not necessarily in the next release) [01:10] mwhudson: sure [01:11] mwhudson: Storing in revision properties should fix this, and keeping the sha map around works around the problem [01:11] there's only 3 files in the kernel with unusual modes [01:11] jelmer: so how did that work? [01:15] lifeless: still working on it [01:15] ok [01:15] apply_delta is real simple [01:16] I'm going to start working on these presentations [01:16] gimme a shout anytime [01:17] jam: wow bvg is very, uhm, brief [01:20] lifeless: Hmm, CHKMap.from_dict() returns the new root key [01:20] lifeless: so I just use CHKMap(store, root_key) to get it out again? [01:21] yes [01:21] apply_delta also returns a replacement key [01:21] (from_dict is a shim around apply_delta) [01:22] ok, that's working now [01:22] lifeless: in the case of inventories, what do you use the key tuples for ? [01:23] and is there any reason why I might want e.g. the individual directory names in the tuple instead of just a 1-tuple with a path? [01:24] jelmer: size [01:24] we store inventories as two dicts [01:24] (parentid, child_name)->child_id [01:24] (fileid,) -> serialized_entry [01:25] the former gives us 'ls /foo/bar' [01:25] the latter gives us tree.inventory['fooid'] [01:26] jelmer: what are you storing [01:31] lifeless: I'm storing unicode_path -> (file_id, revision_id, created_foreign_revid) [01:32] and on only ever lookup on path ? [01:32] lifeless: no, lookup by file id is also possible [01:32] s/on/you/ [01:32] ok [01:32] so you'll need two maps [01:33] if you want fast lookup on fileid [01:33] lifeless: but lookup by file id is very rare ("bzr check" uses it) [01:33] check is getting overhauled ;) [01:33] this is for svn ? [01:33] lifeless: You're overhauling bzr-svn's check too ? ;-) [01:33] no, but the api calls bzr makes are being overhauled [01:33] lifeless: yeah, bzr-svn does check too (it checks the consistency of the bzr metadata) [01:34] so the access pattern is going to radically change [01:34] alrighty, we cn ignore it at your risk:) [01:34] so [01:34] do you need to iterate paths ? [01:35] or is it 'give me data for X' [01:35] amd do you need to do case-insensitive matches [01:37] I mostly need to just look at particular items [01:37] by path [01:37] so if you store every path you'll have a lot of duplicate data [01:37] the trie will reduce some/much of o that [01:38] but not all [01:38] duplicate in the sense that duplicate revids are used? [01:38] in particular leaf nodes have the full key of each item to allow finding the thing you asked for [01:38] no duplicate in the sense that foo/bar and foo/gam both have foo/ in the key [01:38] ah, right [01:39] so you should zcat your knit and ahve a look, we had some optimisation plans and I don't recall exactly what items jam got to [01:39] and what we deferred/didn't do. [01:39] anyhow [01:40] that's not worse than what we have now though [01:41] you could hash the keys (via the key func), you could use two dicts (parent,child)->filed, fileed->(revid, foreignid) [01:41] or you could say 'I'm happy as it stands' [01:41] a more important thing though [01:41] 'knits' load the full index always [01:41] this sucks for doing little operations [01:42] so when you hve this working we should look at backing it onto packs+btree indices [01:43] ok === luke-jr_ is now known as luke-jr__ [01:47] man, I so want to code [01:48] what's stopping you? === luke-jr__ is now known as luke-jr_ [01:48] jelmer: thinking about it, i think relying on the sha map persisting is pretty smelly :) [01:49] mwhudson: yes, it is. that's why we're warning [01:49] jelmer: presentations to write [01:49] lifeless: ah. uds? [01:49] yes [01:50] and bzr and git are hot topics [01:50] likely ot have a hundred folk, I suspect [02:24] lifeless: uhm, how do I get my data out of a CHKMap? [02:24] other than iteritems()? [02:24] any way you want ? :) [02:24] iteritems [02:24] ok [02:24] and iter_changes [02:25] iteritems(['path1']) [02:25] is == __getitem__('path1'), if we had it [02:25] we don't because its not cheap and the api should reflect the reality [02:26] ValueError: Prefix has too many nulls versus width ? [02:27] ('path1',) [02:29] thanks [02:33] morning (at last) [02:36] igc: just :) [02:36] lifeless: :-) [02:36] hi [02:36] thanks for the review [02:36] no probs [02:37] I think its not much work to do your approach at the right layer [02:37] And I agree we can defer really deeply integrating it. [02:37] I do think its important to have tests, and that they are really best written against the right layer. [02:37] me too [02:38] I hope I suggested a reasonable way to do that [02:38] lifeless: I did [02:38] lifeless: I would really like to leverage the specfiic files masking at the layer below [02:38] while perhaps keeping the parent yielding ... [02:39] and exclude masking a layer above [02:39] lifeless: but I got a weird error from record_iter_changes when I tried [02:39] is that fixed by parent yielding do you know? [02:40] paste the error ? [02:40] Git pull of dulwich: 2.43 MiB in 0m53.437s [02:40] Bzr pull of dulwich: 360KB in 0m36.602s [02:40] woo [02:40] lifeless: I'll need to change the code again and re-run. Will do later [02:40] jml: I think jam and I have to rename our talk [02:41] 'when bzr became faster than git' [02:41] lifeless: that's a very nice result. [02:41] igc: so, without seeing the error I can't do much more than guess [02:42] lifeless: are those numbers representitive of normal usage? [02:42] thumper: 'bzr branch lp:~lifeless/dulwich/trunk-bbc [02:42] thumper: vs 'git init; git pull git://gitorious.org/dulwich/mainline.git' [02:42] lifeless: but were it me, I'd leave the talk with the same title and demonstrate its obsolescence with a big bar graph. [02:42] thumper: so, I'd say yes [02:43] jml: yes, I was jesting about the rename [02:43] the title is nice and contentious [02:43] lifeless: :D [02:43] very good result though. [02:43] I wanted to paste the result though [02:43] * jml is keen to see the results for the kernel [02:43] I was thinking, 'lets get some numbers'. [02:43] and boom. [02:43] thumper: big histories may scale differently. I don't know precisely. [02:43] lifeless: very cool [02:45] of course, over http it took 6 minutes to get dulwich bzr [02:45] don't mention the war [02:45] lifeless: but git isn't using http is it? [02:46] no [02:46] git is using its native protocol [02:46] lifeless: so we are still comparing oranges with tangerines rather than oranges to fork lifts [02:46] is lp especially aggressive about packing things? [02:46] bob2: no [02:47] thumper: yes, both protocols are fruity [02:48] lifeless: nice! [02:49] lifeless: do I explicitly have to remove entries from the chk map or is overwriting them fine as well? [02:57] thumper: want to catch up about nested trees sometime? [02:57] poolie: yes [02:57] now? [02:58] sure [03:01] jelmer: use apply delta [03:01] thumper: call me? [03:02] lifeless: you about? [03:02] hello johnf [03:03] johnf: sure am [03:03] poolie: howdy [03:03] lifeless: re bug https://bugs.launchpad.net/bzr/+bug/243391 [03:03] Ubuntu bug 243391 in bzr "TooManyConcurrentRequests during commit" [High,Fix released] [03:03] johnf: there's this theory I'm paid to do this sort of thing :) [03:03] igc: one small chore for you next week: please moderate the mailing list [03:03] it gets a ridiculous amount of spam but enough real posts are caught you do need to check it [03:03] poolie: sure [03:04] I'm getting a similar error when an upstream router returns admin proibited. Would that be fixed or should I file another bug? [03:04] johnf: bzr version ? [03:04] client 1.14.1 server 1.13 [03:04] johnf: new bug, that was 1.8 [03:05] johnf: I take it you've fixed your router though ? :> [03:05] lifeless: no it's normal. Happens if the VPN is down [03:06] johnf: so, I suspect this is a variant of 'ssh tells us shit', unless you're committing over http [03:06] lifeless: this is using bzr:// [03:07] johnf: ok. please be sure to include that detail in your new bug report ;) [03:07] johnf: root cause will be that we try to perform an operation on the protocol object while its still thinking its busy [03:08] lifeless: I think I'll add to this bug as it seems to be the same problem https://bugs.launchpad.net/bzr/+bug/323456 [03:08] Ubuntu bug 323456 in bzr "TooManyConcurrentRequests when committing and the smart server is offline" [Undecided,New] [03:08] gimme a sec to read it [03:10] yah, do it [03:10] or just metoo the bug [03:11] jelmer: can you suggest a midsize project with history in git and bzr - same history, as dulwich has [03:11] jelmer: I'd like to do a slightly less 'new project' test [03:12] lifeless: I'm using apply delta, but may be overwriting keys without removing the old data at the same key - is that a problem? [03:12] lifeless: hmm [03:12] jelmer: its expected [03:13] lifeless: cool, thanks [03:13] jelmer: you can't have a key repeated in a chkmap [03:13] lifeless: Not sure about other projects. You could use one from gnome that's available in both svn and git [03:13] jam: hi :) [03:13] jam: gathering data for our talk, I found: [03:13] 11:35 < lifeless> Git pull of dulwich: 2.43 MiB in 0m53.437s [03:13] 11:35 < lifeless> Bzr pull of dulwich: 360KB in 0m36.602s [03:17] jelmer: I might ask a favour, if you're willing, as you have gitorious etc accounts [03:17] jelmer: put together a similar sample case, but something that is say 10x the size of dulwich [03:21] * lifeless binds nose to ooo [03:24] lifeless: is pqm broken? I just tried resubmitting a patch and it fails pretty well instantly grumbling about 'subunit' I think [03:25] igc: forward to me please [03:25] it shouldn't be, as subunit is optional. [03:25] lifeless: I don't get an email either re success or failure as best I can tell [03:25] igc: and to land a patch making it mandatory the patch would have to pass [03:27] lifeless: I'll run the test suite on my integration branch and see if it's my fault [03:27] bug in pqm [03:27] it may be fixed i nwhich case we can just say 'spm please upgrade pqm' [03:27] lets start with that [03:27] spm: please upgrade pqm [03:29] igc, abentley, i read through the updated design doc and i have some comments [03:30] i know some of my questions on the user doc were answered in the design doc, but i wasn't always asking them for the sake of the answer [03:30] more to say they should be described here [03:31] poolie: I'm planning to update the userdoc given the feedback from you and others [03:31] * lifeless taps the mic [03:31] spm: ping [03:31] igc: I'm sorry you haven't had feedback from me yet [03:31] igc: feeling the crunch with allhands and 1.15 [03:32] igc: your branch is probably faulty btw, I think its a bug in pqm that you don't get told, not a bug in pqm causing it to fail [03:32] poolie: but I don't think the userdoc is the critical path given abentley is working from the design doc [03:33] poolie: except for how the userdoc impacts the design doc [03:34] lifeless: sorry was on the phone. couple of minor family hassles atm. so upgrade pqm on balleny? I assume a bzr pull equiv would do the trick? [03:35] spm: yeah, cd ~/source/pqm; bzr pull [03:35] spm: then igc can try again [03:35] and we see what breaks. [03:35] oki. one sec or more... [03:43] lifeless: igc: upgraded. go for it. [03:50] Is "1.9-rich-root" is the still the current public rich root format, or is there a newer one I should tell someone to use? [03:52] AfC: its what --default-rich-root will create. [03:52] k [03:52] AfC: if you need working tree filters you can use 1.14-rich-root, but only windows users are likely to be desperate for that [03:52] nah [03:53] and if you want to help beta test the bbc format you can use --development-rich-root [03:54] igc - thats what you hit in pqm https://bugs.edge.launchpad.net/pqm/+bug/340361 [03:54] Ubuntu bug 340361 in pqm "bad format usage" [High,Confirmed] [03:57] lifeless: http://rafb.net/p/dMFRc592.html [03:57] lifeless: anything obviously wrong? [03:58] I trying to use a new integration branch fwiw [03:58] one on lp as poolie has requested [03:58] igc: your branch url is probably the problem [03:58] igc: two things [03:59] firstly, ~ianc/bzr/integration would be a better url [03:59] no point having your personal integration branch a team branch [03:59] secondly, I don't know if balleny can talk to lp [03:59] well, except lp fails whether I try to push to ~ian-clatworthy/bzr/ianc-integration :-( [03:59] spm: can you check that pqm user on ballent, outside chroot, can do 'bzr info bzr+ssh://bazaar.launchpad.net/%7Ebzr/bzr/ianc-integration/' [03:59] igc: it does, how ? [04:00] igc: or rather, in what way [04:00] grumbles about a read-only transport [04:00] and breaklock doesn't fix it [04:00] pastebin please [04:01] lifeless: it can yes. [04:02] lifeless: http://rafb.net/p/gnKmyI33.html [04:02] lifeless: looks like ~ian-clatworthy/bzr/integration has worked though [04:03] much nicer url :) [04:03] lifeless: well, except for the vfs noise: http://rafb.net/p/CMcA4E84.html [04:04] igc: yes, bzr is running 1.14 [04:04] 1.15 will remove that for a simple push [04:04] dunno why you get that lock error [04:04] readonly transport means that lp thinks *you* can't write to it [04:05] file a question on launchpad-code [04:05] I suspect a bug [04:05] lifeless: ok, I'll try a http url to that new branch [04:05] igc: bzr+ssh should be ok, spm has confirmed it works [04:06] igc: what commit message were you giving pqm [04:06] ? [04:07] lifeless: see http://rafb.net/p/dMFRc592.html [04:07] maybe the & is causing an issue? [04:09] I would suspect so [04:20] lifeless: so neither changing to a http URL, nor changing & => 'and' fixed the problem [04:21] I added 'public_branch:policy' to branch.conf but that didn't help either [04:21] the full test suite passes locally [04:22] I don't have push_location:policy set to norecurse. Is that likely to cause an issue? [04:22] jml - whats up with https://code.edge.launchpad.net/~jelmer/pqm/merge [04:23] jml: I can't access it as 'lp:~jelmer/pqm/merge' [04:23] lifeless: that is because there is nothing there [04:24] https://code.edge.launchpad.net/~jelmer/pqm/merge/+merge/6122 seemed to find something at some point [04:24] lifeless: or broken [04:24] jelmer: ^^^^ [04:24] lifeless: probably created with bzr send [04:25] isn't that meant to create a working branch? [04:32] * igc lunch [04:42] no matter how many times I branch dulwich, git takes twice as long [04:42] did I recall seing that bbc branches stack in 1.15dev? [04:44] yes [04:44] bob2: :) [04:44] lifeless: cool [04:49] or at least, jam has it mostly going and we're aiming for 1.15 [04:50] spm: please pull bzr from people.ubuntu.com/~robertc/pqm/trunk [04:54]