[00:03] lifeless: I'm just doing bzr -r1 [00:03] lifeless: pardon - bzr diff -r 1 [00:14] furicle: ok, as a workaround you can use filterdiff -x '*.pdf' or something like that, to remove the pdfs in your diff [00:14] please file a bug, this is something we can clearly do better on [00:14] Jc2k: avg 0.00275434756196 [00:14] avg 0.0118949759048 [00:16] lifeless: did gc just kick some arse or do i need an eye test.. [00:16] arse kicking [00:17] as usual the story is more complex [00:17] ah boo :[ [00:17] let me paste results [00:17] http://paste.ubuntu.com/113016/ [00:18] lifeless: thanks for the info - I can pipe it out to vim and hack it back out, just wondered if I was doing something wrong. Thanks for the info [00:19] furicle: are you going to file a bug? If not, I will - but I don't have the test data to reproduce if its not obvious [00:20] Jc2k: in particular look at the max time [00:20] 0.17 is because we read one big zlib hunk [00:20] but after that the entire inventory for all of the repo is in one compressed record [00:23] so your saying the zlib reads are gonna kill us anyway? [00:23] I'm saying its going to depend a lot [00:23] >< [00:23] like [00:23] annotate will kick arse [00:24] but 'read the most recent commit message only' will be slow [00:24] read all commits will be fast [00:24] etc [00:25] we can trade time and space here [00:25] if we cap the groups smaller [00:25] then the zlib read will be less [00:25] but we'll have more groups [00:27] its not beyond hope [00:27] we can still look at short reads and decompresses for random access [00:27] we could keep recent texts at the front of groups [00:28] so in comparison to git: git looking at the most recent commit can generally look at the first object in the pack (or near) and it will generally be a full text rather than a delta so it will be (seek a few bytes in) (hot zlib action) done. [00:28] yah [00:28] what does gc have to do? [00:29] * Jc2k missed your intro to gc as rob led you away at guadec :( [00:29] can I say it depends and wave my hands a lot ? [00:29] sure :D [00:29] it depends, wave wave wave [00:29] ok [00:30] have you read the README in bzr-groupcompress? [00:30] nope, *scurries off to look* [00:31] when you've read that you will understand the tuning I'm talking about I hope [00:31] though I can and will enlarge from there [00:32] GPL GPL GPL GPL, lets make things smaller. read DESIGN. k thx end. :p [00:32] oh yeah DESIGN. [00:32] me [00:32] h [00:32] I knew which document I meant [00:34] pet peeve, -d isn't global [00:36] ok. generic process parsed. [00:36] wow, think I found a knit bug >< [00:36] so [00:36] two layers [00:36] stream-of-build-recipes [00:37] efficient *fast* compressor on top [00:37] the build recipe stream should be 80% of the compress, or something like that [00:38] cost to get a single text out is clearly: [00:39] cost to decomopress outer enveloper up to the end of the text build recipe + cost to parse and execute the build recipe [00:39] best case cost would be no outer envelope and the first text in the stream [00:40] because we could just read that text and ignore the rest [00:40] lifeless: I will file a bug. [00:41] furicle: thanks [00:41] Jc2k: did you catch my cheating ? [00:44] lifeless: so.. to clarify "no outer envelope", no envelope compression at all? no zlib of the recipes? [00:44] right [00:44] we can get rid of the decomp cost [00:44] in which case, you just read a full text and there is no decompression or assembly to do at all [00:44] so its uber fast [00:44] just read as much of the stream as needed [00:45] but larger [00:45] say 100% larger [00:45] which of course for network IO isn't such a great thing [00:46] but its an option [00:46] but we can fix that with smart reads, right? [00:46] depends on the use case [00:46] like, smart server, the server could zlib on the fly [00:46] sftp, you will be reading whats on disk directly [00:46] but group compress + smart server could use delta bases that it knows are on the client for example [00:46] ah [00:47] look closely at a recipe [00:47] its not a delta as suc [00:47] this is a recipe: [00:47] ['label: label\n', [00:48] 'sha1: xxxxxxxxxxxxxxxxxx\n' [00:48] 'i,3\n' [00:48] 'strange\n', [00:48] 'common\n', [00:48] '\n', [00:48] ] [00:48] that recipe will create the text ['strange\n', 'common\n'] [00:48] this recipe: [00:48] 'label: newlabel\n', [00:49] 'sha1: xxxxxxxxxxxxxxxxxxxxxxxxxxxx\n' [00:49] 'c,72,7\n', [00:49] 'i,1\n', [00:49] 'different\n', [00:49] 'i,1\n', [00:49] '\n' [00:49] ] [00:49] copies 7 bytes from byte 72 into its output [00:50] so recipes allow you to select ranges from the base text and insert next text? [00:51] from the stream so far [00:51] ah [00:51] where as git patch opcodes only let you reference the base text [00:52] this is why it compresses so well without threaded computation of best-parent [00:52] because it uses every single text compressed so far as a parent [00:52] gotcha [00:53] so the least cpu work to get the first text out is no zlib wrapper, and read in only the bytes for the first text [00:54] least average work is probably zlib, because less IO does matter [00:54] theres a typical U graph trading off space and time in different scenarios ere [00:55] now, the current decompression code, for the first text, reads the entire zlib envelope, decompresses the entire thing, puts that in a lru cache, then grabs the recipe it needed and generates the output [00:56] changing this to read less is complex in two ways: how much of the zlib record has to be read to decompress the recipe for the first text, and how much will eventually be wanted [00:56] the greedier we are, the more upfront work, but the less round trips [00:58] hey all. I'm having trouble merging a task branch back to trunk; bzr is telling me there are no changes to commit after I do the merge, despite showing me a +N line during the merge. What am I missing? See http://pastebin.com/d646d281 [00:59] spats: its because you are trying at revision 0 [00:59] spats: do your test case again, but after initing 'main', do a commit [00:59] even a commit with no content [00:59] aha! okay, thanks :) [01:00] revision 0 can't be merged with things, its a blackhole to bootstrap from :) [01:00] so your merge is in effect a pull [01:00] its a little wart, but one most folk can't ever encounter :P [01:01] lifeless: I was trying to follow the "never check in to the mainline" theory a bit too literally - make a branch for initial code checkin. I might just add an empty checkin to the process. === mlh_ is now known as mlh [01:06] lifeless: tough problem :) [01:07] Jc2k: does git do delta combining? [01:07] Jc2k: that is, to recreate something at the end of a chain of 200 texts [01:07] does it get(base); patch(base->2); patch(2->3);; etc [01:08] or does it get(base), combine_patches(2,3,4,5,...); patch(base->200) [01:08] it can chain, and there is a limit on the length of chains [01:08] its the 1st [01:08] ok [01:08] base + patch = object 2, object 2 + patch = object 3 [01:09] and the limit is much lower than 200 from what i recall [01:09] folk on forums say --200 --200 [01:09] and benchmarkers are using that [01:09] default is 50 [01:12] i think it was just 10 in the beginning [01:15] anyhow [01:15] so to be faster than git [01:16] we have to be faster at some key operations [01:16] and also faster in things like fetch that inspect every object [01:16] I think gc is there now, modulo optimising [01:16] but the recent-history case still concerns me [01:17] I think that that perhaps some hinting to the engine [01:17] can we be a bit bigger and faster locally, and smaller and a bit slower when doing stuff remote? [01:17] 'I only want one thing, kthanks' [01:17] yes, thats possible assuming decompress and recompress during push [01:17] we could even just change the zlib layer for that === timchen119 is now known as nasloc__ [01:31] ok [01:31] so a rough figure [01:31] first 1000 revs of bzr [01:31] knits->knits 5 seconds [01:31] gc->gc 19 seconds [01:31] note that we have much better integrity checking in the latter case, so its not quite apples-apples [01:32] and the gc repo is 50% of the size [01:32] $ du -sh gc/.bzr [01:32] 1.3M gc/.bzr [01:32] $ du -sh plain/.bzr [01:32] 2.4M plain/.bzr === asac_ is now known as asac [01:33] slightly faster at some ops too: [01:34] $ time bzr log -v plain/t > /dev/null [01:34] real 0m11.920s [01:34] $ time bzr log -v gc/t > /dev/null [01:34] real 0m10.775s [01:36] tree building too [01:36] gc/t$ bzr remove-tree && bzr checkout . && bzr remove-tree && time bzr checkout . [01:36] [01:36] real 0m0.914s [01:36] plain/t$ bzr remove-tree && bzr checkout . && bzr remove-tree && time bzr checkout . [01:36] [01:36] real 0m1.069s [01:40] sleep time for me (its 1.30 already!) [01:40] good luck lifeless [01:40] gnight! [01:45] >< thumper - bzr push bzr+ssh://bazaar.launchpad.net/~lifeless/bzr/fix-chunked [01:45] ... [01:46] bzr: ERROR: exceptions.KeyError: 'pqm@pqm.ubuntu.com-20090131231933-8o4phfvmuuizyyn6' [01:46] Traceback (most recent call last): [01:47] lifeless: hi [01:49] lifeless: is the branch packs5/branch6? [01:49] mwhudson: think so [01:50] was probably initially pushed when that combo made a broken branch [01:50] $ bzr push bzr+ssh://bazaar.launchpad.net/~lifeless/bzr/fix-chunked [01:50] Using default stacking branch /~bzr/bzr/trunk at bzr+ssh://bazaar.launchpad.net/%7Elifeless/bzr/ [01:50] Source format does not support stacking, using format: '1.6' [01:50] Packs 5 (adds stacking support, requires bzr 1.6) [01:50] thumper: new branch today [01:50] arse [01:51] igc: new groupcompress pushed, FYI [01:51] spiv: ^ [01:51] lifeless: i suppose you're not using bzr 1.9 still are you? :p [01:51] lifeless: shiny [01:51] mwhudson: bzr.dev [01:51] lifeless: For the record - bug#320783 - already existed - I just added my two cents - might be any pdf file [01:51] furicle: thanks! [01:52] I'm going to get gc able to pull all of bzr.dev [01:52] lifeless: it seems that the branch has repo format 1.9 and branch format 1.6 [01:53] urgh [01:53] so it stacked, but didn't upgrade the branch format on the fly [01:53] and is not stacked [01:53] oh [01:53] it is a stackable format branch [01:53] just not stacked [01:53] http://bazaar.launchpad.net/~lifeless/bzr/fix-chunked/.bzr/repository/format [01:53] says 1.9 [01:53] thumper: it should be 1.9, I've upgraded [01:54] branch.conf is empty [01:54] http://bazaar.launchpad.net/~lifeless/bzr/fix-chunked/.bzr/branch/format [01:54] but the branch I pushed from is branch6 or so [01:54] says Branch Format 7 [01:54] Bazaar Branch Format 6 (bzr 0.15) [01:54] right [01:54] thats the autoupgrade stuff kicking in to support stacking [01:54] k [01:55] mwhudson: where is the stacked branch kept? [01:55] so it converts the formatting appropriately, but doesn't actually set the stacking [01:55] which is a bit odd for folk that don't want their project to suddenly grow a dependency on latest bzr; or is it per-project enabled in lp ? [01:55] thumper: .bzr/branch/branch.conf [01:57] lifeless: i'm pretty sure the auto-upgrading only happens when you have a mixed format like a new repo but old branch [01:58] mwhudson: ok, anyhow lets not get distracted [01:58] yeah [01:58] theory is the autoupgrade half-worked [02:00] well i'm already distracted :) [02:00] but yes [02:00] there should be tests for this stuff, maybe they only check the formats, not that the new branch is stacked? [02:00] none in blackbox.test_push [02:01] nor branch_implementations.test_stacking it seems [02:01] spiv: hey you, you fixed this bug [02:01] def test_push_doesnt_create_broken_branch [02:01] is close [02:02] that sounds like something i wrote :) [02:02] hm, maybe not [02:03] test_push_with_default_stacking_does_not_create_broken_branch [02:03] in branch_implementations.test_push [02:04] spiv: want to hear a fun bug [02:08] lifeless: sure [02:08] lifeless: so long as I don't have to fix it ;) [02:09] you know how we check file content graph <-> revision graph correspondence [02:09] we greate graphs like that with '' as the content, for directories. [02:09] guess what 'bzr check' and 'bzr reconcile' do not check [02:10] They don't check directories? [02:10] RevisionNotPresent: Revision {('selftest-20050621060616-bb8b5b36e3c950c8', 'robertc@robertcollins.net-20050919060519-f582f62146b0b458')} not present in "". [02:10] the source repo checks cleanly [02:11] thats the directory 'bzrlib/tests' [02:11] its being triggered by inserting a child of that [02:11] Oops. [02:11] so, I think the child's parent pointer is wrong :P [02:13] Yeah, using a directory's "file content" as a parent for anything does seem rather weird. [02:15] lifeless: new groupcompress in brisbane-core or it's own branch? [02:18] igc: its still seperate [02:18] spiv: well, its another dir [02:19] spiv: this is _very early_ commits [02:19] spiv: so its appropriate, its just not aligned with the graph [02:19] early weave bug I think [02:19] * igc lunch [02:20] igc: but mutual dependencies [02:56] so lifeless, to benchmark your latest groupcompress, I need to grab the latest plugin code [02:56] igc: yes [02:57] lifeless: and bzr.dev or brisbane-core or doesn't matter? [02:57] I assume brisbane core is more interesting to you? [02:57] or not? [02:57] if you are testing bbc node compression, then you need the format in there to use gc [02:58] right now I'm isolating the parameters and performance for the compression layer, which is [mostly] orthogonal to bbc [02:59] igc: have you seen compressbench [03:00] lifeless: I know of it but I haven't tried it [03:01] well, I'm mainly working with that for micro and then use case tests, for the --gc-plain format, right at the moment [03:01] still trying to rule-out-or-in [03:01] sure [03:02] uhm, if you want to put the gc-plain format through usertest, that would be nice [03:04] lifeless: shall do. I have chemo Wed & Fri this week so something mindless like running benchmarks will fit in well :-) [03:05] :) [03:05] look, its legal drugs, can't complain [03:06] :-) [03:06] Speaking of which... coffee. [03:07] * spiv -> afk [03:39] spiv: and for entertainment value, this bug is in out mainline repo [03:42] jelmer: ping? [03:46] hi [03:46] I just did a bzr merge -r.. /path/to/whatever [03:47] well actually, a bunch of them [03:47] is there a way to list what is about to be merged? [03:47] bzr missing [03:47] lifeless, no I mean what I am about to commit [03:47] I thought bzr st -v listed the relevant data but apparently not [03:48] the merged revisions? [03:48] or the file changes? [03:48] oh sorry [03:48] you have cherry picked [03:49] cherry picks are special and do not record the revision [03:49] orly? [03:49] they are essentially just a normal diff+patch operation [03:49] why? [03:50] because we haven't implemented a non-transitive graph relationship that is scalable in the cases we know people do cherrypicks in [03:50] sorry if that answer went straight off into technobabble :P [03:50] we have plans to get this fixed, but its not at the top of anyones queue as far as I know [03:50] so if I do bzr merge /path/to/whatever it will try to merge again? [03:51] yes, but if you are doing that, why were you cherrypicking in the first place :) [03:51] well because for now I just want stuff that i know works [03:51] and then later the rest of it [03:51] fair enough [03:52] it shouldn't cause many spurious conflicts when it does merge again [03:52] unless you have changed the lines further [03:52] oki, thanks [03:53] igc: around? [03:54] is it possible to say "ignore these revisions when merging for now"? [03:55] sohail: no, because merging acts on the whole files, and there is no copy of the files you are merging from that is missing those lines *and* has any new lines added [which might supercede those very lines you wanted to eliminate] [03:56] lifeless: ping [03:56] igc: I was thinking, for usertest [03:56] I understood your other babble better than the latest one :-) [03:56] it would be nice to have both time-to-clone-natively and time-to-clone-from-pack-0.92, which is the only one we have today, isn't it ? [03:57] sohail: heh. So when we merge, you could imagine taking a series of patches and applying them. [03:57] lifeless: it depends ... [03:57] sohail: or, taking the file contents of the source, and the target, and a common version they both shared, then finding what changes have occured at that level [03:58] the source repo can be whatever format you like [03:58] I see.. I think :-) [03:58] sohail: in the former approach you could 'skip a patch', but in the latter approach, you don't know which lines are from a given patch without doing rather more work [03:58] the test is currently "bzr branch mirror feature-branch" [03:59] for gc-plain testing right now, .... [03:59] sohail: and even you did, you then need to detect dependencies in later changes on the patch you are skipping [03:59] I add an extra step though as I assmue that the source isn't in that format [03:59] sohail: 'darcs' does something in this area, and we'd like to too, but it makes most folks minds hurt ;) [04:00] igc: righto; I'm saying both steps are useful data [04:00] igc: 'how slow to get to this format' and 'how slow to clone in the format' [04:00] lifeless, well I'd rather just say something like: "When you are merging from this branch, never ever merge revisions a..b" [04:00] lifeless: understood [04:01] sohail: the way you say that today is: merge up to b (but don't commit yet), then merge -r b..a --force, and then commit [04:01] hmm [04:01] let me try that [04:01] sohail: that commits a revert of the a..b change in your branch [04:02] if the other branch merges from you, they will have that revert applied to them [04:02] oh [04:02] remember that the point of merging is to make two branches content be combined [04:02] so that isn't what I want either... [04:02] so given two branches A and B [04:02] in A, merge B+commit [04:02] then in B, merge A + commit [04:02] is defined as giving you the same content [04:03] or maybe I can make a third integration branch? [04:04] I don't know what your scenario is sorry :P [04:05] that's ok, you've been helpful [04:05] I'm happy to discuss things further if you like [04:05] I'm ok if you're ok :-) [04:05] we're all ok then [04:06] I have two branches, master & next. master is meant to be stable... The next branch got some stuff checked in that wasn't quite ready [04:06] but I need to make the master use the latest good stuff [04:06] I want bzr to keep track of it all for me :-) [04:06] fair enough [04:06] I use looms for that [04:06] but you don't need to [04:07] * sohail is waiting for The Bazaar Book :-) [04:07] the key thing to know is that I cherrypick when needed, and I have a branch for each conceptual change I'm making [04:07] yeah I think that's the thing [04:07] if a branch B depends on branch A [04:07] next time I gotta make a branch for each change [04:07] I merge A into B [04:07] now, if A has bad stuff in it, but the conceptual change in B could be merged to trunk anyway [04:07] what I do is a cherrypick [04:08] -r branch:A..branch:B [04:09] what does that mean.. [04:09] -r branch:A..branch:B [04:09] well [04:09] difference between those two branches? [04:09] like -r x..y [04:09] except that x is the tip of branch A [04:09] and y is the tip of branch B [04:09] so yeah, the difference between the branches [04:10] but how does that apply here? [04:11] it cherry picks the unique work in B without the stuff in A I was unhappy with [04:11] it may not apply to you, I was just sharing how I use bzr to track things for me [04:11] oh I see [04:11] this works if you have a branch per conceptual change [04:12] yes [04:13] so do you have any suggestion for my situation? [04:13] you have two branches, master and next, next has some good stuff, and after it some bad stuff, and then some more good stuff? [04:14] and you want all the good stuff in master? [04:14] yep [04:14] and I want bzr to track this [04:14] ok, firstly I'd merge all the good stuff to master that is before anything bad [04:14] bzr merge -r last_good_before_bad; commit [04:15] then, I'd : bzr merge next; bzr merge -r last_bad..first_bad --force next; bzr commit -m "Pull in all ready code from next"; [04:15] I would then go immediately to next and [04:16] bzr merge master; "bzr merge -r first_bad..last_bad ."; bzr commit -m "Master excluded $topic_that_is_bad but merged everything else." [04:16] which will merge back from master but keep the bad content (by reinstating it during the merge) [04:17] at this point, master has all the good stuff; next has everything including bad; the next merge from next to master will carry over the bad stuff [04:17] I would then rename 'next' to 'bad-feature' [04:17] and make a new next :) [04:19] spiv: any idea how to get pdb to handle BZR_PDB=1 when inside a generator? [04:20] lifeless: Hmm, I'd expect it to Just Work, aside from the usual trick with generators which is they don't really start until the first iteration. [04:21] spiv: bt shows me the current call stack up to the generator only [04:21] spiv: I can tell you how to reproduce easily if you like [04:21] Weird, but I'm not totally surprised either. pdb is a bit stupid. [04:21] Sure. [04:21] lifeless, OK, gonna try now [04:21] spiv: install bzr-groupcompress to start with [04:21] I have encountered wierdnesses with pdb + generators before. [04:21] sohail: cool [04:21] spiv: I know which is why I am mentioning [04:21] lifeless, ok so bzr merge -r last_good -> bzr is not tracking this merge [04:22] i.e., it is a normal diff+patch [04:22] sohail: it should track it. Its '-r X..Y' that it won't track [04:22] sohail: '-r X' will get tracked [04:22]