/srv/irclogs.ubuntu.com/2007/11/28/#bzr.txt

lifelessjam-laptop: ok, reviewed.00:00
lifelessbasically I think what you have done is the wrong approach, though the code is reasonable.00:00
lifelessjam-laptop: and I've suggested an alternative approach00:01
jam-laptoplifeless: I think that might end up returning data out of order00:05
jam-laptopcoalesce will re-order to get better packing00:06
lifelessjam-laptop: so some options.00:06
lifelesswe could coaelsece knowing that we'll submit 200 at a time00:07
jam-laptopI suppose you could just not collapse them?00:07
lifelessso we have a window of 200 and we pack linearly out-of order up to 200, then a new window etc00:07
lifelessanother option is to do what I suggest but use that to accumulate the result rather than to return it00:08
lifelessthat is, coalesce optimally, then do all the reads, then start yielding in-order.00:08
lifeless(also not that we have an out-of-order-is-ok option for readv)00:08
lifelesss/not/note/00:08
abentleyCan't we just join the gaps between ranges until we have a suitable number of ranges?00:09
jam-laptopabentley: lifeless is saying that it ends up downloading too much00:09
lifelessabentley: that results in reading vast amounts of data00:09
jam-laptopthat was my approach00:09
lifelessits exactly the bug that I filed00:09
lifelessnow it turned out that the case I filed was trivially solved, but when it gets more complex we will still trigger bad behaviour00:10
lifelessthe cost of a new round trip, for nearly anyone on the planet is < 2 seconds00:10
abentleyRight.00:10
lifelessa 512kbps line - thats 1Mbit, or 128K00:11
jam-laptopI *do* think that multiple GETs is the right approach00:11
jam-laptopthis was mostly expedient00:11
jam-laptop(Default fudge is 128 bytes, by the way)00:11
lifelessso any expansion over 128K is a net loss in this scenario.00:11
abentleyWell, I'm going over to the dark side to see if I can't get the case-insensitivity problems fixed.00:11
lifelessfaster links expand the amount we can tolerate, but also tend to reduce latency, shrinking it.00:11
lifelessabentley: may the force be with you00:12
abentleytx.  laters.00:12
jam-laptopabentley: good luck, later00:12
PengHmm, anything happen since Thursday?00:25
lifelesscode00:32
PengWow, pulling bzr.dev has been going for almost 5 minutes now.00:32
fullermdWell, it took me a minute and a half, and I pulled yesterday, so...00:32
PengWhat did you do, update to Python 3k?00:33
PengRewrite it in Perl? :D00:34
PengJust finished. 7 minutes.00:34
PengIt only downloaded r3011-3038.00:34
lifelessyou're probably experiencing the http readv coalescing bug00:35
lifelessthat we fixed:)00:35
PengWere 'bzr cat' and bzr+ssh fixed for packs?00:37
lifelessyes00:38
PengYay.00:38
Peng'Course, I'd have to use bzr.dev..00:38
lifelessthough mixed packs and knits over bzr*:// may do bad things00:38
Penglifeless: Mixed?00:39
lifelessits being investigated right now00:39
PengLike having a pack repo locally and pushing to a knit repo?00:39
lifelessbzr pull bzr+ssh://someknitrepo local-pack00:39
lifelesspushing will be fine00:39
PengOkay, good.00:39
lifelesspull may insert incorrectly formatted data.00:39
lifelesslook at bugs tagged packs in launchpad00:39
PengUm. How bad is that?00:39
PengIs that like rm-the-repo bad or reconcile-while-whine-some-more bad?00:40
lifelessPeng: *if* it doesn't handle it correctly, you'll get raw knit hunks that are annotated in a pack repo; or vice verca.00:40
lifelessif the error checking is there (I think it is), it will just error.00:41
lifelessPeng: and this *only* affects pull.00:41
PengOh.00:41
PengDoes it affect clone?00:41
lifelesscloning is pulling00:41
PengOr merge?00:41
PengOkay.00:41
lifelessmerging does a pull00:41
lifelessbasically, if you have a pack repo today, don't pull/merge/whatever from a knit repository over bzr+ssh using bzr.dev until https://bugs.edge.launchpad.net/bzr/+bug/165304 has some comments on it00:42
ubotuLaunchpad bug 165304 in bzr "smart server data streams not used across repositoryrepresentations" [Undecided,New]00:42
Pengwill*00:44
Penglifeless: Like I said, is it rm-the-repo bad or reconcile-will-whine-some-more bad?00:45
lifelessrm the repo00:45
fullermdIWBNI(tm) the SS data were totally divorced from the repo formats.  That way the client wouldn't even have to be able to read or know anything about the server format (or vice versa), as long as it could store the requisite data.00:45
fullermdI'd also like a pony.00:45
lifelessfullermd: it is00:45
lifelessfullermd: the question is if the client code DTRT00:46
fullermdI'm pretty sure it isn't, as broadly as I mean it.00:46
PengHmm.00:47
PengThis affects 0.92 too, right?00:47
lifelessPeng: 0.92 is safe because bzr+ssh and packs will just die00:47
PengHaha. That's right.00:47
PengI think I'll stick with using sftp just to be safe.00:48
fullermdMy bzr 1.0 client Should(tm) be perfectly able to read over bzr*:// a branch9/pack5 branch, as long as my local data format can store all the same info as the remote.00:48
PengEven though I won't pull much.00:48
PengWell, I don't expect to pull over bzr+ssh at all, but just to be safe...00:48
lifelessfullermd: yes, for the current smart server verbs, thats exactly what can be done00:48
lifelessfullermd: we haven't made all operations smart server verbs, so it won't actually /work/ but that's being picky :)00:49
PengHey, wait a minute. Isn't this a data loss bug? Or are they okay in bzr.dev?00:49
lifelessPeng: bzr.dev is fine00:49
lifelessPeng: sorry, I don't get what you mean quite;00:49
fullermdWell, in much the same way that we're totally faster than git; it's just incompletely implemented   ;p00:50
PengThe homepage claims you've never had a data loss bug or something.00:50
lifelessPeng: in a release00:50
PengHeh.00:50
fullermdRiding dev head on anything is wild and wooly.00:50
PengHow many data loss bugs have snuck into bzr.dev? :X00:50
lifelessright now, there is this *potential* one. And its not necessarily a bug: we're checking.00:51
fullermd(says the guy running bzr head under his window manager head+ on his OS head...)00:51
lifelessits 'There might be a problem, please dont do X until we have checked'00:51
lifelessnot 'There is a problem, dont do X'00:51
PengBoth mean "don't do X".00:53
lifelessright.00:53
lifelessand in a few hours we'll know if this is an actual bug or not00:53
lifelessI suspect its not, it will just error. But safety first.00:54
lifelesshi jelmer01:00
PengMan, my cat sat down on my lap literally 30 seconds before I was planning to get up.01:00
fullermdThat's pretty bad.  Most cats have much better timing than that; 30 seconds is long enough that you might think it a coincidence.01:01
PengHeh.01:02
PengI've been away for a few days too.01:02
PengAnyway, bye.01:02
lifelessciao01:02
PengTo help decide which VCS to use, are you guys dog or cat people? :)01:03
* Peng wanders over to #mercurial to ask, and then leaves.01:03
fullermdProbably merits a FAQ entry...01:03
lifelesscat person01:04
lifelessis what I am01:05
poolielooking at bug 16444301:33
ubotuLaunchpad bug 164443 in bzr "can't branch bzr.dev from codehost over bzr+ssh" [Critical,Confirmed] https://launchpad.net/bugs/16444301:33
lifelesspoolie: I'm waiting for pqm to quiesce to reconcile it for you01:34
pooliethe bzr.dev branch01:34
poolieyou can kill my 'post-review cleanups' branch if that would be easier01:34
lifelesshmm,  I might do that.01:36
lifelessreconcile started01:52
poolieok01:58
pooliedo i need to resubmit that, or will you reenqueue it when it's done?01:59
lifelessyou will need to resubmit01:59
lifelessit's in the replay set already01:59
jelmerthe FAQ mentions tailor, svn2bzr and bzr-svn for migrating from svn (in that order)02:00
jelmerI think tailor shouldn't be listed first because it's going to give the worst results of the three02:01
lifelessagreed02:01
lifelesssurely bzr-svn is the best02:01
fullermdThat's so that by the time the user gets to bzr-svn, they really appreciate it   ;)02:01
pooliewhat is a replay set?02:02
lifelesspoolie: the hash from the gpg signature is noted02:07
lifelesspoolie: if we get the same again it is rejected02:08
lifelessstops replay attacks02:08
lifelessPeng: bzr+ssh is safe02:15
lifelessPeng: it will just error02:15
abentleyPhew!  Made it back with my soul intact.  Turns out the last version of bzr I had on Windows was 0.802:17
lifelessrofl02:17
lifelesspoolie: we're safe to change the default02:18
abentleybialix isn't kidding when he says it's messy.  I fixed four test cases for transform while I was there.02:18
lifelesshttps://bugs.edge.launchpad.net/bzr/+bug/16530402:18
ubotuLaunchpad bug 165304 in bzr "smart server data streams not used across repositoryrepresentations" [Undecided,New]02:18
lifelessabentley: cool02:18
abentleylifeless: I know that WorkingTree4 has a much stricter locking policy, but I wonder if we should have the same policy for is_executable on *nix and win32.02:19
lifelessabentley: I'm not sure what you mean02:19
lifelesscould you phrase it as a change02:19
abentley*nix doesn't need a lock and win32 needs a lock.02:19
lifelessI said in a review to bialix that I thought requiring a lock on unix was fine02:20
abentleyWe could wrap is_executable in a lock decorator, or we could make it require a lock even on *nix.02:20
lifelesshe had some massive cruft in his 'make things be known failures' patch02:20
abentleyOkay, maybe that's a saner way forward.  Otherwise, we'll just introduce new lock failures without noticing.02:21
lifelessI think most things should require a lock; because it will help track adhoc api abuse02:22
lifelessthat can cause errors and performance bugs02:22
abentleyI miss the days when locks were almost implicit.02:24
lifelessit was easy02:27
lifelessbut it performed terribly02:27
ubotuNew bug: #172483 in bzr "progress bars on http push behave badly" [Undecided,New] https://launchpad.net/bugs/17248302:30
Penglifeless: Great.02:32
abentleylifeless: One thing I noted on win32 is that our config directory is still *\Bazaar\2.0\02:33
abentleyI don't know if we *can* fix it, but it seems like it would be good.02:33
lifelesswell, we'll reach 2.0 eventually :)02:34
dewdbtw, I'm updating to trunk latest to give pack a new try with bzr+ssh push. :P hopefully it's going to work. hehe. exciting.02:36
lifelessdewd: okies02:46
lifelessdewd: it should packs->packs02:46
dewdi thought it was the default format already. looks like dirstate-tags is the one still02:48
dewdsomething wrong? I created a new packs repo on the remote server, and when branching it on the local machine with bzr+ssh, the local format says "dirstate-tags".02:52
abentleydewd: No, that's a bug.02:55
lifelessI'm just looking it up in fact02:56
lifelessbug 16462802:56
ubotuLaunchpad bug 164628 in bzr "branching from hpss doesn't preserve repository format" [High,Confirmed] https://launchpad.net/bugs/16462802:56
abentleyYou can work around it by creating a branch in pack fomat, then pulling into it.  Or by not using the smart server for the branch command.02:57
dewdok. thx.02:58
spivHello everyone.03:10
spivI'm back from honeymooning.03:10
lifelesspoolie: check passed; pushed to a packs branch, then pushed that to a new one gave the same size pack, now running check in the resulting branch03:10
spivlifeless: good afternoon03:10
lifelessyo spiv03:10
pooliehello spiv03:10
lifelessyou working or slumming?03:10
spivpoolie: hello.03:10
spivlifeless: working a half day, starting now.03:10
pooliei should really go and see "Keating!"03:10
spivpoolie: I nearly saw that, but the night I had tickets for got cancelled due to a performer having a cold :(03:11
poolielifeless, is pqm running? the web ui is stuck on the output from my killed job03:11
poolieit's still running03:11
pooliehe's not very kind to spivs though :)03:11
lifelesspoolie: yes, pqm is running, dunno why the UI shows that03:11
lifelesspoolie: oh, I know03:11
spivHeh.03:11
poolielifeless, you're right, using a reconciled branch in the same way works fine03:12
lifelesspoolie: its got jobs,s o the status file looks like the end of the run I canned03:12
poolie?03:12
pooliespiv, quick call?03:17
spivpoolie: jml just got in first03:18
spivpoolie: I'll ping you when we're done?03:18
pooliejust call me03:18
poolielifeless, you were right it just needed reconcile03:18
spivpoolie: will do03:19
PengWow, almost 1 MB pack from just the last 6 days.03:20
Peng(... of changes to bzr.dev)03:21
dewdwow. packs rocks03:22
abentleylifeless: if you've got a branch where packs are the default, I'd be interested in running bzrtools against it.03:22
lifelessabentley: I posted a patch to the list03:22
lifelessjust change the format name to pack-0.92 and it will work03:23
lifelessdewd: glad you like them03:23
pooliespiv, the streaming hpss changes don't seem to be mentioned in NEWS; can you please add something appropriate?03:24
spivpoolie: ack03:24
thumperhi spiv03:24
abentleyspiv: welcome back.03:24
thumperspiv: welcome back03:24
thumperabentley: snap03:24
abentleyheh.03:25
dewdlifeless: packs seems much more network friendly for me now in my sample tests. congrats. :-)03:26
lifelessdewd: we have quite a few future enhancements planned03:26
lifelessthat will help even more03:26
spivabentley, thumper: thanks!03:27
dewdit's a winner in my book. nice decision to go for it sooner. much more pleasant03:27
pooliegreat03:29
lifelesspoolie: ok, it passed, migrating to escuderi03:33
PengISTM that if one is going to use packs, one should use bzr.dev.03:40
lifelessPeng: seems fair to me.03:48
lifelessin 0.92 they were explicitly tagged experimental03:48
PengTrue.03:48
PengI didn't expect that "experimental" would mean "half-broken", though.03:49
PengHonestly, I thought that they were marked experimental because they weren't fully optimized and because you're planning to change the format a couple times in the future. I didn't know that there was more work to be done for them to work fully.03:51
PengHmm. "bzr diff -c X.Y" seems to be a lot slower than "bzr diff -c Z".04:01
PengWoah, you made bzr.dev packs already?04:10
lifelessPeng: we'll issue more versions of packs as we change it.04:16
PengMm-hmm.04:17
PengThat's whast I meant.04:17
Pengwhat*04:17
lifelessok.04:17
lifelessPeng: diff -c x..y will have to extract two full inventories vs 1, so if thats slow (many files) then yes.04:18
Penglifeless: Not two dots, just one. The crazy revnos from branching and merging.04:19
lifelessoh, hmm..04:19
PengIt's not like I did real scientific testing. But when reading through the log of new things in bzr.dev and diffing some, it felt slower.04:20
jameshPeng: do the two revisions change similar numbers of files?04:21
Pengjamesh: Didn't pay attention.04:21
jameshi.e. could it be that the time taken is proportional to the size of the change rather than whether it is on mainline04:21
PengSince I'm using packs, would you recommend using bzr.dev? I mean, I'm afraid of data loss and stuff, so I don't even always use the release candidates.04:22
jameshthe other check would be to see whether "bzr diff -c revid:whatever" differs in speed to the dotted lookup04:22
lifelessabentley: please resubmit your patch04:23
abentleyOkay.04:23
lifelesspqm is missing some lock correctness stuff itself04:23
poolie_Peng, if you're concerned about data loss i'd generally recommend using releases04:23
lifelessright now bzr.dev looks solid to me04:23
lifelessPeng: we called it experimental cause it was; it was supported in 0.92, so if you had encountered corruption or something we'd have happily spent time fixing your data04:24
lifelessPeng: and we needed more feedback on it; I wouldn't call it half-broken. It was literally 4 days work to fix all the glitches we encountered making packs be able to be default04:25
Pengjamesh: Argh, not sure.04:25
lifelesspacks took way more than 8 days to make :)04:25
lifelesspoolie_: you too, please resubmit.04:25
lifelessI'll monitor the first one to go through04:25
poolie_again?! :)04:25
lifelessyup04:26
Pengjamesh: Um, yes. Using the revid changes 3009.2.29 from taking 8.5 seconds to 0.5.04:26
lifelesstree.lock_write() FTW04:26
poolie_done04:26
jameshPeng: I guess you've found the reason then04:26
jameshthe dotted revnos are the sort of thing that could probably be cached in the branch though04:27
jameshso it is something that could be fixed04:27
Penglifeless: <kinda quiet voice>Well if it only took 4 days, you could've delayed the release.</>04:27
Peng(</> is valid SGML, right?)04:28
lifelessheh04:28
lifeless' needs &amp; though04:28
jameshPeng: depends on your sgml definition04:28
jameshor whatever they call it04:29
Peng:D04:29
lifelesswell, the release hasn't happened yet has it? And at 0.92 time that was a month ago, there was UDS and allhands, and time based releases04:29
i386lifeless: ping04:29
lifelesswe didn't know at 0.92's timeframe how long the piece of string was, or what issues would arise04:29
lifelessi386: yo04:29
lifelessi386: you in SF?04:29
i386yo04:29
i386Yeah I am04:29
Penglifeless: Well, maybe not half-broken, but 'bzr cat' and pushing over bzr+ssh didn't work, and there seems to have been a *lot* of work in bzr.dev to fix things.04:30
jameshthe SGML declaration is what I was thinking of04:30
PengWill bzr.dev being packs make pulling a lot faster?04:31
lifelessPeng: try it ;), using bzr.dev because 0.92 has the http range combining bug04:31
PengI was just about to try it, then I discovered I have two extra copies of bzr.dev from November 2, at r2954. I wonder why?04:33
PengBoth are dirstate-tags.04:33
PengOne is dirstate.04:33
* Peng shrugs.04:33
PengUsing packs over bzr+ssh, is there any reason to upgrade the server to bzr.dev?04:34
PengOr is 0.92 okay?04:34
lifelessyou'll need bzr.dev, there were a few broken methods04:35
PengOoh.04:35
PengThat sounds unappealing.04:35
PengEspecially since I'll never remember to keep it up-to-date.04:35
lifelessjust the one update04:35
lifeless then at our next release you can switch back to releases04:35
PengShould I run check and reconcile now? Or will it still report millions of non-error errors and take gigs of RAM?04:36
lifelesscheck and reconcile in bzr.dev are a-ok04:37
PengLast time I tried a couple weeks ago it used 1.6 GB of RAM.04:37
lifelessthey will check but not reconcile packs.04:37
lifelessthey will check and reconcile knits04:37
PengBlaah.04:37
PengNever mind, then.04:37
lifelesspack reconcile is on my laptop and my top priority tomorrow04:37
Peng:)04:38
lifelessit got sidelined with this 'get packs ready for default'04:39
PengWow.04:39
PengErr. Not wow at that.04:40
PengWith 0.92 and http://bazaar-vcs.org/bzr/bzr.dev/ being dirstate, pulling from 3010-3038 took some 7 minutes. With bzr.dev and the remote being packs, 3010-3040 took 1m15s. Nice.04:40
PengWait. The 1m15s test was also without a working tree.04:42
PengBut that couldn't take more than about 5 seconds.04:42
lifelesshmm, 1 minute is still slow04:42
lifelessI bet you its in graph traversal04:43
lifelesswhat speed adsl do you have?04:43
lifeless1M is 16 seconds on 51204:43
Peng1.5 down.04:43
lifelessso, 6 seconds + figuring out what to pull04:48
lifeless4 round trips there minimum, should be able to do this in say 12 seconds total04:48
lifelesse.g we still suck04:48
PengYikes.04:49
* jml takes his inner pedant out for a coffee04:49
Penglifeless: Are you going to abandon the knit version of your repository branch now?04:49
lifelessPeng: I'll probably make it a packs version, so I can hack on the next experimental format04:51
Pengrepository will be the new format, repository.knits will be the current format. Okies.04:52
lifelessPeng: well, I'll rename it too, for clarity :)04:54
PengRight.04:55
lifelessok, pqm is happy now05:00
lifelessso I'm off05:00
PengPushing packs over sftp, is bzr.dev any faster than 0.92?05:03
Peng(Creating a new branch.)05:03
PengWait, never mind. I am using bzr.dev.05:04
Peng(I was asking because if it was faster, I might want to kill a push I'm running and start over with bzr.dev. But since I am using bzr.dev, doesn't matter. Still might be interesting to know though.)05:05
lifelessshould be the same05:07
lifelessif its a shared repo it should be really fast05:07
lifelessif its not really fast and is a shared repo, please file a bug, tagged packs.05:08
jmlwhy does addCleanup enforce uniqueness of cleanup callables?05:22
poolie_jml, no good reason05:23
jmlok05:23
poolie_if you're motivated to do it, send a separate patch removing that check05:24
jmlpoolie_: the motivation is low.05:24
ubotuNew bug: #172499 in bzr "Bazaar knit corruption "Not a gzipped file" " [Undecided,New] https://launchpad.net/bugs/17249905:31
jamesh^^ sounds like the web server helpfully adding Content-Encoding or Transfer-Encoding headers05:36
Penglifeless: I dunno. It took 25 minutes to push into an empty shared repo, but it was a large branch (bzr.dev), so that might not be bad.05:48
lifelessPeng: oh, new content has to upload; It's 50MB05:48
lifelessPeng: whats your upload rate05:48
Penglifeless: I'm not sure. I thought it was 256k.05:50
lifelessso, thats 32 seconds/MB05:50
lifelessor 2MB/minutes - so 25 minutes05:50
PengOkay.05:50
lifelessPeng: sounds like we got wire rate total time05:50
PengAnd that's with sftp's overhead.05:50
lifelessnot to shabby05:51
lifelesswhen we halve the size of storage it will get faster again05:51
PengHalve? How?05:51
lifelessnew storage level diff algorithms05:52
PengNew delta algorithm?05:52
lifelessyah05:53
lifelessviao05:53
lifelessciao05:53
PengWhich one?05:54
poolie_lifeless, can you please chmod the .knits branch?05:56
ubotuNew bug: #172506 in bzr "allow users to force an empty commit message" [Low,Confirmed] https://launchpad.net/bugs/17250606:21
lifelesswb06:34
lifelesspoolie_: sure; will do shortly06:34
lifelesspoolie_: done06:49
poolie_dude go home!06:49
lifelessI am :)06:50
lifelessjust got dinner06:50
spivlifeless: the memory consumption of reconcile is much better now; thanks!07:14
* spiv stops work for the day07:14
PengDefinitely.07:18
PengWait, I only ran check.07:18
Peng0.92's new "bzr help formats" is nice. :)09:05
Odd_BlokeHave I missed an email regarding a new release schedule (as bugs are now being targeted for alpha releases I've never heard of :p)?09:20
PengHuh, Ctrl+Cing pushing from packs to dirstate tracebacked.09:21
lifelessPeng: file a bug please09:21
mwhudsonwhen did 'bzr viz' become so much more awesome?10:08
mwhudsonsix months ago if i fired it up on launchpad it produced a window about 5000 pixels wide10:08
PengUh-oh. Using a copy of bzr not installed in the system means the plugins aren't available.10:30
OlberdIs there some way to find out what revision I had before a bzr pull10:42
OlberdA log or something10:42
PengNo.10:43
=== Ng_ is now known as Ng
=== cprov-out is now known as cprov
Odd_BlokeWhat can cause 'bzr shelve' to fail to remove changes from the working tree?11:11
=== kiko-fud is now known as kiko
=== mrevell is now known as mrevell-lunch
ubotuNew bug: #172567 in bzr "Per-file log is *very* slow with packs" [Undecided,New] https://launchpad.net/bugs/17256713:01
Penglifeless: Seems my upload speed is 512k (1/3 of the downstream? huh). That would mean bzr over sftp is half the wire speed.14:38
Odd_BlokePeng: I assume you're using ADSL, in which the A stands for asymmetric (that is to say, they use more lines for down than up as most people use more down than up).14:47
PengOdd_Bloke: I know that. I thought it was lower than 1/3, though.14:55
vilajam-laptop: ping14:56
=== jam-laptop is now known as jam
jamvila: <dodge>14:57
vilalol14:57
jamwhat's up?14:57
vilalooking at doing multiple get for readv, will be simpler if we don't have to respect the provided order for the offets14:57
jamvila: you can look at the sftp readv code14:58
jamit does out-of-order fetching14:58
jambut only buffers what it cannot return14:58
jamall we really need is to change how we parse the values14:58
vilahooo14:58
jamI've done all that work14:58
jamI just haven't had time to rewrite the http response parser14:58
vilaI think we can avoid that rewriting for now, but I have to look at sftp first14:59
jamwell, I think all the actual parsing is fine15:00
jamit is just written to parse a complete response15:00
jamrather than parsing as you go15:00
vilahmmm, that will buffer quite a lot of data... well price to pay to respect the order while minimizing network use, I suppose you did the analysis at the time15:05
vilaDo you think it's still valid with packs handling megs of data ?15:05
jamvila: in my testing the amount of out-of-order data was all very small15:15
vilaso reading as you go bufferd only small amounts of data, ok, let's say that ;-)15:16
jamvila: also if adjust_for_latency = True15:16
jamwe can return things out of order15:16
jamBut I think that may be taken care of in the base readv() implementation15:17
jam(it sorts them so the individual implementations don't know it is out of order)15:17
jamvila: And I believe most pack operations use that15:18
vilasoooo, I don't have to worry15:18
vilaphone15:18
=== kiko is now known as kiko-fud
radixIs there a way to get "bzr shelve" to be "tighter" in what it considers a single hunk?15:46
radixAnd by "tighter hunks", I mean what "bzr diff --diff-options=-U0" might give.15:47
elmoGGGGGGGGGGGGGGGGGGAR16:02
elmobzr add precious; bzr rm precious16:02
elmooh look, 'precious' is no more16:02
* elmo stabs himself repeatedly16:02
fullermdLooooost!  Precious is lost!16:03
* fullermd hugs his rm --keep alias.16:04
luksit doesn't actually remove anything that isn't in the branch history16:05
elmoluks: except in the example I just posted16:06
elmoI just did this IRL and lost like 70 files as a result16:07
lukswhat version of bzr?16:07
luksmine says "bzr: ERROR: Can't safely remove modified or unknown files:"16:07
elmoluks: 0.9216:07
elmoah, it's a bit more involved16:10
elmobzr init; bzr add precious; bzr ignore precious; bzr rm precious16:10
luksouch16:12
luksthat sounds like a pretty nasty bug16:12
luksit probably doesn't check the modified status for ignored files16:13
=== kiko-fud is now known as kiko
warrenWill bzr upstream ever reconsider the decision to not track all permission bits in bzr?16:15
warren(not just execute)16:16
warrenbzr is *really* close to being suitable to track system config files, for example.16:16
elmowarren: AFAIK, it's not that they don't want to do that, they just haven't decided quite how to16:16
warrenelmo, oh16:16
warrenelmo, yeah, I guess a solid plan would be necessary.16:16
warrenthere are other permission bits than the chmod XXXX16:17
warrenthere are also filesystem acl's and xattrs now16:17
elmowarren: lifeless mailed me just the other day, asking for details of how we use bzr for managing /etc and how we'd like permission management to work16:17
jamelmo: with the 'bzr ignore' involved I do see the file being deleted16:19
elmojam: yeah, I filed it as https://bugs.launchpad.net/bzr/+bug/17259816:19
ubotuLaunchpad bug 172598 in bzr "bzr rm <file> loses data under some circumstances" [Undecided,New]16:19
jamotherwise I see the "cannot safely remove modified files"16:19
warrenelmo, ok, looking forward to a future of true permissions management =)16:20
fullermdI'd think the most likely near-term solution would be plugins using a per-file properties system (as might come about for line-ending and such support)16:23
ubotuNew bug: #172598 in bzr "bzr rm <file> loses data under some circumstances" [Undecided,New] https://launchpad.net/bugs/17259816:25
ddaaWhat is the shellyscript way to find that a bzr tree has uncommitted changes?16:46
LeoNerdSomething around bzr di   maybe?16:47
ddaalooks like it would work...16:48
Pengbzr st?16:48
Pengbzr modified16:48
Peng?16:48
ddaaI was looked at bzr st, but it does not have non-zero exit status if uncommitted changes are found16:48
* Peng shrugs.16:48
ddaabzr di has16:48
Peng(modified would only show files that had changed, not other things like added files, of course.)16:49
PengSomething with python -c "import bzrlib..." ! :D16:49
ddaa"if ! bzr diff > /dev/null" it is16:49
jamddaa: bzr status -q ?16:50
jamno, it isn't that quite16:50
ddaajam: has zero exit status in all cases16:50
jamquiet16:50
jamdiff seems to be the way to go16:51
jamI thought we had something different16:51
ddaayup16:51
ddaaa bit wasteful, but not biggy16:51
ddaajam: maybe we can have "bzr diff --no-diffs"? :)16:51
fullermdMaybe something with version-info --check-clean?  Think that triggers on unknowns, though...16:52
ddaaif I upgrade to packs, will "commit after a big crazy merge" stop being slow?16:54
ddaaIt used to be fast, but he regressed quite badly in recent releases.16:54
mwhudsonhmmmmmm16:54
mwhudsondoes pull with packs have a progress bar yet?16:54
ddaas/he/it/16:54
ddaamwhudson: ask linus, you do not need progress bars when you're fast ;)16:55
ddaaabout "commit after big merge", it's a real issue for me16:55
Pengddaa: Committing merges with packs can be/is slow. There's been recent work to make it much faster, but it didn't solve what made it slower than with knits in the first place.16:56
Pengddaa: You could try it.16:56
ddaaPeng: not really16:56
ddaaupgrading the launchpad repo to packs requires something like 20GB RAM16:56
ddaawaiting for that to be fixed16:57
mwhudsonddaa: eh, no16:57
mwhudsonddaa: i pulled launchpad/devel into a packs repo and it was perfectly smooth16:57
ddaaam I confuzzled?16:57
mwhudsonit's reconcile that takes/took megagigabytes of ram16:58
ddaawasn't there this discussion about "bzr upgrade" eating crazy memory for the launchpad repo?16:58
ddaaah right16:58
ddaabut upgrade won't work if it's not reconciled, right?16:58
* mwhudson is confused too now16:58
* ddaa holsters his confuzzlerizer16:59
PengSome repos need to be reconciled, some don't.17:00
ddaamh17:00
ddaamaybe some launchpad repos have this problem and others don't17:01
ddaaso maybe it's worth for me to let bzr check run on mine17:01
keiris there any way to do something like bzr push --uncomitted? this is not a good description; what i want to do is push uncommitted changes in my working tree to the branch i'm bound to17:18
keiruse case: i suddenly have to leave while working on my laptop. i'm 80% done a feature, but it doesn't even compile at the moment. i want to push my working tree to the branch i'm bound to, without actually committing.17:19
keirthis happens to me a fair amount17:20
keirwhat i do now, is have many commits of 'junk state'17:20
keirbut i'm trying to move to a more sanitized mainline17:20
mwhudsonwell, there's merge --uncommitted . other-location isn't there?17:20
mwhudsonah, merge --uncommitted -d other-location .17:21
keiroh cool17:21
keirbut this won't work with a bound branch; so i'd have to unbind first?17:21
mwhudsonthough i don't know if that'll update a working tree over sftp or whatever17:22
mwhudsonhm not sure, i don't use bound branches17:22
mwhudsonkeir: but hang on a moment17:22
mwhudsonkeir: there's another answer here: don't commit to mainline17:22
keiri didn't either until recently; they are pretty handy for my laptop/desktop combo!17:22
keirmwhudson, yes, i understand that option; i still don't like it because e.g. bisect becomes much less useful17:22
fullermdA third option would be to uncommit away the dirty rev if it bothers you that much.17:23
keirbut then it'd be uncomitted on my desktop when i get back, but still committed on my laptop17:23
fullermdWell, you have to 'update' on the laptop anyway, so what's the difference?17:23
keirunless i made a script commit, unbind, uncommit, ssh into desktop, uncommit17:23
keiri suppose what i'm looking for is bzr synctree17:24
fullermdNo, you just update.17:24
keirif i 'update' on my laptop after uncommitting on the desktop, does that uncommit the changes on the laptop?17:25
fullermdPretty sure I've done it more'n once.17:25
fullermdupdate's pretty much like pull after all; move the head to the new rev, merge any outstanding WT changes.17:26
fullermdJust because the new head isn't a descendant of the old one doesn't change anything material there.17:26
Odd_Blokekeir: You might have to 'pull --overwrite', but I don't use bound branches much so I don't know.17:26
keirinteresting17:27
keiri recently turned one of the machine learning researchers here at U of Toronto onto bzr; he generally likes it17:28
keirbut had some problems: http://article.gmane.org/gmane.comp.version-control.bazaar-ng.general/3352517:28
jamddaa, Peng: The commit after merge has been fixed for both knits and packs17:28
jamThe problem was searching the global revision graph17:28
jamversus the per-file ones17:28
jamwhen finding common ancestor, etc.17:28
keiri didn't have a good answer to his issue, so i asked if he could post to the list, but no one has responded.17:28
jamBoth are *valid*, just the global graph has more nodes to search17:28
Pengjam: Oh, cool.17:28
jamThere is more work to be done (to make searching the graph faster)17:28
ddaajam: thanks, I'll wait for my lpdev update, then :)17:29
jamddaa: yeah, I don't recommend you converting your personal one to packs until the upstream is packs.17:29
jamOh, and the reconcile memory consumption has also been fixed17:30
jamI'm not sure what it would be for LP now17:30
ddaacool17:30
mwhudsonis b.control_files._lock.peek()['user'] really the easiest way to find out who has locked a branch?17:30
jambut for bzr.dev it dropped quite a bit17:30
ddaathat means we'll probably switch devpad to packs shortly after the next bzr release.17:30
ddaaI appreciate how keeping memory use in control is difficult in Python17:31
keiruncommit after a commit from a bound branch is not putting the changes back into the working copy17:31
ddaait's much easier when you can use custom allocators.17:31
ddaathat's probably the only thing I think really sucks with Python17:31
jamddaa: well, having an algorithm that reads all inventories and buffers them in memory17:33
jamis just a poor algorithm17:33
jamIt stored a slightly more efficient form17:33
jambut still had a record for every file in every version17:33
ddaaSure, but you'd have found out earlier if you had bounded cache mechanism with a custom allocator, so you could have tested it with different memory limits.17:34
ddaaIt's a recurring problem in bzr.17:34
ddaa"If we do not cache, it's slow, if we cache it eats tons of memory."17:35
PengLRUCache?17:35
fullermdWell, Ubuntu used to send out those free CD's.  Can't we send out free DIMM's to bzr users?   ;)17:35
ddaa(I realize this is a blanket statement and that they are many bad reasons for things to be slow or eat tons of memory, I am just trying to make a point about memory management in general)17:36
Pengfullermd: Sign me up!17:36
ddaaif we do that, people will complain that those DIMMs do not have free VHDL.17:36
ddaaSo we balanced the PR implications, and decided not to ;)17:37
fullermd"Lightsabres aren't open source!"17:37
ubotuNew bug: #172612 in bzr "new commit is overly verbose" [Undecided,New] https://launchpad.net/bugs/17261217:41
PengWow, bzr 0.12 is only 1 year old. You've been busy.18:43
lifelessvila: packs make an effort to request in-order for .pacll18:44
lifelessvila: and same for indices - but with indices they set the flag saying out-of-order-ok18:44
lifelesselmo: and are you going to mail me back :)18:45
lifelessddaa: yes packs commit merge is faster18:45
lifelessddaa: reconcile is fast now, in .dev18:46
PengYou were just adding dotted revnos and bzr+http.18:47
Peng(I wonder if bzr+http worked then. Grumble grumble.)18:48
lifelesshi jam, how is the missing-compression-parent patch coming ?19:03
lifelessddaa: actually, we knew it was bad on memory in 0.9219:04
lifelessddaa: I don't think the actual developer realised how bad bad memory issues become in the sort of scaling environments we encounter19:04
lifelesstap tap tap is this thing on?19:08
keirlifeless, i think your tapping knocked ddaa's connection out19:14
lifeless:)19:15
ganderHi. Does anyone here know anything about these bzr irc logs? <http://bzr.arbash-meinel.com/irc_log/bzr/2007/>19:26
=== kiko is now known as kiko-zzz
james_wgander: that would be jam19:28
ganderWhy would he do that? It's a spammers delight19:29
james_wwhy?19:30
jelmergander: why's that? hostmask accidently matches email address?19:30
ganderBecause anyone can trawl for email addresses of it. Cant they?19:31
james_wyes, but the percentage of working email addresses you would get would be tiny.19:33
fullermdIt'd be pretty poor trawling, I'd think.  The density of addresses is miniscule.19:33
lifelessmost users email and hostmask don't match the reverse lookup of the machine they IRC from.19:33
lifelessthat said, santising them should not be hard. jam: ^19:33
fullermdHey, the more nonexistent hosts spammers try, the less time they'll spend rapping on my servers   :p19:34
ganderLooks like I need to mask mine then19:34
lifelessyou'll likely find other channels you are in are also logged19:35
PengIf you participate in FOSS development, your email address is screwed anyway. Bug trackers, VCS author information, mailing lists ...19:36
ganderPah! :(19:36
lifelessPeng: yes, but gander may be just using bzr :)19:36
PengTrue.19:36
Pengs/True/Right/19:36
lifelessluks: ping19:36
lukspong19:36
lifelesshi19:36
lifelessnice performance regression19:37
lifelesswell, not nice, but nice catch noticing it19:37
luksthe file log?19:37
lifelessyes19:37
lifelessI've replied with an analysis of the callgrind19:37
lifelesswe can probably trivially drop it to 7 minutes by removing the duplicate call to get_revision_graph19:37
luks7 seconds, I hope :)19:38
lifelessthe other two will take a little more work; if you're interested in hacking on this code ...19:38
lukswell, I have generally very mixed feeling about packs, so I'm probably not motivated enough :)19:39
luksmaybe it's just me, but it feels a lot slower on many local operations, which is what I'm primarily interested in19:40
lifelessluks: file bugs, lots of bugs.19:40
luksit's all caused by parsing the big indexes19:40
lifelessluks: I know for absolute fact that its faster and many local operations; but we're likely not looking at the same data set, nor the same operations.19:40
lifelesss/and/on/19:41
luksyep, simple things like diff or st are faster19:41
lukshistorical diff and st, I mean19:41
luksbut for example just genering a bundle for 'send' is slower19:41
lifelessluks: this is why is was experimental in 0.92; and I was expecting a little more time to find performance regressions (where 'all history' become more expensive, but 'partial history' became possible.) and fix then19:42
lifelesss/then/them/19:42
lifelessbug 165309 will help with the index performance19:42
ubotuLaunchpad bug 165309 in bzr "pack index has no topological locality of reference" [Medium,Confirmed] https://launchpad.net/bugs/16530919:42
lifelessbut changing all-history API usage to partial-history API usage is the key thing needed to improve performance19:42
lifelessknits are incapable of partial-history operations19:43
lifelessbut they were tuned to do all-history very fast19:43
lifelessit turned out that this was not a good enough solution19:43
luksknits with better indexes would be, IMO19:43
luksthe current text index for packs probably needs some kind of subindex to make it fast19:44
lifelessluks: the current text index for packs is /way/ faster than knit indices at extracting say 1% of keys19:45
lifelessanyhow, bug 165309 will deliver a subindex of sorts19:46
ubotuLaunchpad bug 165309 in bzr "pack index has no topological locality of reference" [Medium,Confirmed] https://launchpad.net/bugs/16530919:46
PengI think uncommit is slower with packs.19:46
lukslifeless, which is why diff -c X and st -c X are fast19:46
lifelessluks: anyhow, here's what I'd *love*. File a bug anytime a pack operation is slower than a knit operation.19:46
luksok19:47
lifelessthey will nearly all involve commands accessing too much history19:47
fullermdThat's interesting.  I wouldn't expect uncommit to need to do any repository actions at all.19:47
lifelessPeng: please file a bug.19:47
lifelessI'll mail the list right now.19:47
fullermdI guess to find the parent, maybe.19:47
lifelessluks: can I quote you in my mail? I'd like to give people context about what I'm asking19:48
lukssure :)19:48
PengShrug. Uncommit takes a couple seconds with packs, and I don't remember it being non-instant before.19:49
PengI'll confirm it in a few minutes.19:49
PengBut now to brush my teeth!19:49
buxyHi, I'd like to convert an existing CVS repository to bzr19:52
buxyI tried using tailor but it fails: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=43912419:53
ubotuDebian bug 439124 in tailor "tailor: Fails to convert debian-admin CVS repository to bzr" [Important,Open]19:53
Penglifeless: Hey, what's hte status of pack reconcile?19:53
Pengthe*19:53
* Peng goes back to tooth-brushing.19:53
fullermdbuxy: Have you tried using the cvsps-import plugin?  IME it's a lot more reliable than tailor, as well as _much_ faster.19:53
buxyfullermd: no, do you have a pointer?19:54
ksmith99Hi folks, I have a share-repository I have been checking  out of but the dns name got changed... what do I do to use the new name? thxs19:54
buxyof found at http://bazaar-vcs.org/CVSPSImport19:54
buxyI'll look into it, thanks19:55
marianomksmith99: I know you can use --remember with "pull" to make it change the default location19:57
marianomdo you guys know when bzr-gtk 0.92.x will be in the repository http://bazaar-vcs.org/releases/debs ?19:58
ksmith99marianom: thanks, looks like a good start19:58
marianomksmith99: this is not as clean (and maybe it's a bad advice) but you can change the parent branch address accessing .bzr/branch/parent19:59
lifelessPeng: I'm hacking on it as we speak20:00
lifelessmarianom: we're not currently building bzr-gtk; next monday I should get time to do repository stuff20:01
marianomthanks lifeless20:02
ksmith99marianom: thanks, I might try that if remember doesn't work20:06
PengIs there a command to make a repository default to having working trees or not?20:10
fullermdThe bzr install hacks your system to add those capabilities to the 'rm' and 'touch' commands   ;)20:11
* Peng wanders off.20:12
lucasv1hi20:12
lifelesshi20:13
lifelessPeng: you can rm .bzr/repository/no-working-trees; I don't think we have a UI for it yet20:13
lucasv1I have moved a bzr repository to a different server, now it give me the following error: http://pastebin.org/977320:13
lifelessPeng: there is a python API to change it20:14
lifelessluks: use a modern bzr20:14
lifelessmeh20:14
lifelesslucasv1: use a modern bzr20:14
lifeless0.11 is very old20:14
lifelesswe're at 0.92 today20:14
buxySimilar question, I have a git repository I'd like to convert to bzr, I found bzr-git but there's no doc on how to use it20:15
lifelessbzr-git is incomplete20:15
lifelessjam: was most recently working on it20:16
lucasv1lifeless: it's debian stable  I think20:17
lucasv1that's what my hoster gives me20:17
lifelesslucasv1: you've been using a more recent bzr20:17
lucasv1possible, I have an ubuntu workstation20:17
lifelesslucasv1: the one you have switched to cannot handle kind changes properly20:17
lucasv1ah ok20:17
lifelessso, you need to get them to upgrade; newer bzr's are in debian backports20:18
lukslifeless, damn, now you got me interested in packs again :)20:19
lifelessluks: my email ?20:19
luksI'm looking on the code right now, but is there some limit when a pack is considered "final" and won't be repacked?20:19
fullermdaleph-null?   ;)20:19
luksyes + the irc discussion20:19
luks_max_pack_count seems kind of random20:19
lifelessluks: no; we could consider that though. Some things to consider - each item is moved log10(total commits) times.20:19
lifelessluks: its arbitrary yes, but not random :)20:20
lukswell, for 100000 revisions you get only 1 pack, for 99 revision you get 18 packs20:20
luksnot very good distribution, IMO20:20
lifelessat 100000 revisions, the things committed in the first commit will have been copied 5 times.20:20
lifelessluks: and you'll need to do 900000 commits to get them copied again20:21
lukswell, let's say I have 999 revisions, so I have 27 packs20:21
lifelessright20:22
luksnow I do a commit and all of them get repacked20:22
lifelessat that point you're paying quite a high latency price20:22
lukswhich means I have to reupload the whole repository20:22
lifelessso we correct that by dropping it to 120:22
lifelessluks: with the smart server, once streaming push is implemented, you won't upload the repository again; instead the server itself will do the repack.20:22
luksI think lucene-like repacking would work better20:22
lukswell, but dumb servers are a big advantage of bzr :)20:23
lifelessdo you have a link for that ?20:23
luksumm, no I don't think so20:23
lifelessif you can find one I'd be delighted to read up on it.20:23
luksbit the point is that you have ~ logaritmic distribution of index sizes20:23
luksand you combine smaller indexes into bigger ones20:24
lifelessluks: so far this is exactly what we do20:24
luksbut almost never touch the "top" indexes20:24
lukswell, 27 -> 1 is not exactly that :)20:24
lifelessluks: thats precisely that. You're sampling only the times that we do touch the top indices.20:24
lukslucene would take for example the lower 5 ones and pack those20:24
lifelessluks: again, thats exactly what we do.20:24
luksnope20:25
lifelesssay you have 899 revisions, with 26 packs20:25
luksit always maintains a certain number of indexes20:25
luksit will never drop to 120:25
luksunless you manually merge them20:25
lifelesswell, we could do that easily; just change the distribution logic20:26
lifelessyou could do that with a plugin in fact :). The code that takes a distribution already chooses to only move around the smallest amount of revisions possible20:26
lifelessthere are two problems that it has though, that seem to me to need serious consideration20:27
lifelessone is scaling20:27
lifelessif you have a really big repository its reasonable to pay more latency20:27
lifelessso a constant number of indices is really bad for small repositories (too much latency), or really bad for big repositories (too much repacking)20:28
luksthe number is not constant, it depends on the whole index size in lucene20:29
lifelessok20:29
lifelessthe other thing is that if you are going to touch two indices its cheapest to just combine them together rather than in some other form20:29
lifelessthis isn't quite represented correctly in the autopack.20:29
lifelessits something I realised after doing the initial code and before coming back to it.20:30
* lifeless files a bug20:30
jelmerlifeless: is there some way to tell the code how often to repack?20:32
jelmerlifeless: I'm thinking of mass-imports that could potentially be speeded up if the number of repacks could be reduced20:33
=== cprov is now known as cprov-out
lifelessjelmer: well, you'll trade index latency off against repacking.20:34
lifelessjelmer: if mass imports will be improved, improve the autopack logic.20:35
lifelessjelmer: (but I do understand the desire to control this through code; I'm entirely open to a disable_autopack() api call.20:35
lifelessluks: so I've filed a bug about that. if we're going to read N packs, only ever bother writing 1 new pack.20:36
lifelessI don't know if lucene does this.20:36
lifelessbut basically its silly to aim for a constant number. Because - to keep it constant, you have to shuffle data *between* your indices[packs]20:37
lifelessand as each pack is write-once.20:37
lifelessthis means reading N packs and writing M packs, where N=M+1 once you hit your constant number20:38
lukswell, the point is that you shuffle only the small ones or in small incremental steps20:38
lifeless(each new commit adds one pack, then you put it into your constant-sized collection)20:38
lifelessluks: but thats what we do? We only shuffle the smallest ones20:38
lifelessluks: I feel like we're talking past each other here.20:39
luksyes, but you drop the number of packs radically20:40
luksthat's the only thing I don't like -- I don't want to reupload most of my repository if I really don't have to20:40
lifelessits at power of 10 intervals20:40
ubotuNew bug: #172644 in bzr "autopacking should just combine all packs it touches" [Undecided,New] https://launchpad.net/bugs/17264420:40
lifelessto reupload most of your repo occurs extremely rarely.20:41
lifelessand mathematically I can't see that happens any more often than lucene would do it20:41
lifelessfor instance, bzr is at 15K commits now after 3ish years.20:42
luksin the terms on total used bandwidth probably not20:42
lifelessour next full-upload is going to be at 100K commits20:42
luksbut in the case of lucene, it would be done in small steps20:42
lukshere it's a "all or nothing" situation20:42
lifelessluks: then you have described lucence to be badly; because logarthimic expansion of size will behave identically20:43
lifelessat some point lucence will decide to touch a top index20:43
lifelessand make a bigger20:43
luksyes, and it would be more often than in case of bzr20:43
lifelessthat bigger one will have 10 times what the previous top had in it (what whatever logN they are using)20:43
luksbut the distribution of pack sizes would be different20:44
luksso you will be never in situation you have to reupload/redownload the *whole* repository20:44
lifelessI think you are wrong about the impact and frequency; rather than debating - point me at code/docs.20:44
PengWith bzr.dev working on a fresh copy of bzr.dev, "bzr uncommit --dry-run" uses 0.41 user seconds in dirstate, 0.86 in dirstate-tags, and 3.84 in packs.20:44
lifelessPeng: please file a bug as per my email20:44
lukscan't find anything except API docs, which doesn't explaing it at all20:45
PengI haven't seen your email, but I will.20:45
lifelessPeng: thanks!20:45
lifelessPeng: my email contains a request for a callgrind file20:45
lifelessPeng: just of the packs version20:45
PengJust of the packs version? Okay.20:46
lifelessyah20:46
PengWill the time spent waiting for me to hit enter to exit it hurt callgrind?20:46
lifelessrarely need a knit version, its usually obvious where packs are slow20:46
lifelessPeng: not at all20:46
lifelessPeng: but you can probably enter in advance20:46
lifelessluks: so for clarity; I'm open to improving autopacks user experience.20:47
lifelessluks: It will need serious sum of operations and frequency of size impact analysis.20:47
PengI'll file a bug in a second.20:47
lifelessluks: I did this for the current code, and I'm very happy to help with the analysis of a proposed replacement algorithm.20:48
jamPeng: you can also use "bzr uncommit --force" to avoid the prompt.20:49
lifelessluks: for instance, here's a replacement algorithm: pack the smallest packs we want to get rid of into a new pack bigger than any current pack.20:49
lukshmm20:49
lifelessluks: this would guarantee that we always leave at least the current top in place; and only do this when the sum of the smaller packs is bigger than the current top, so autopack would be about choosing a 'top' to use20:49
Pengjam: Oh, good. That even works with --dry-run.20:50
=== mw is now known as mw|food
lifeless(so if we are leaving (say) the 10K packs alone, we might choose a 'top' to spin around of a 1K pack. Then all the < 1K packs become a new ~10K pack.20:50
lifelessthis of course has other problems. We currently have the situation that most data most operations want is in the smallest packs.20:50
lifelessbecause we strictly shuffle small->next size up.20:51
jamlifeless: don't forget that pull just creates 1 new pack of all streamed data, right?20:51
lifelessand that lets us optimise the order in which we attempt to retrieve index entries as well as providing high locality of data20:51
lifelessjam: right20:51
jamdo we have 'push' doing the same thing?20:52
lifelessjam: it still tends to preserve this property.20:52
luksoh, even pull over http?20:52
lifelessluks: yes, pull over http -> one new pack.20:52
lifelessjam: yes push does the same thing20:52
luksso it autopacks everything into a single pack on the client side?20:52
lifelessluks: single *new* pack.20:52
jamluks: basically it just figures out what it will need, starts streaming the data to the pack, and then finishes once it got everything.20:52
PengWhat's the pack bzr+ssh thing that corrupts the repo?20:52
jam(into a new pack)20:52
lifelessluks: then after the transaction it checks to see if it needs to autopack the whole repo.20:52
lifelessPeng: there isn't one.20:52
lifelessPeng: it was a concern that was all.20:53
Penglifeless: Okay, right. Just an error.20:53
lukser, I actually didn't realize you can "pick" only some revisions from pack even over http20:54
lifelessluks: so here are the parameters: We want adjacent texts to combine into the same pack as we combine packs, so that they can be efficiently delta compressed. We want to provide a natural cap on latency. We always have to replace an entire pack - we can't delete from a pack, nor add to one. We never want a user to have to worry about the internals.20:54
lifelessluks: over http we: perform bisection in 64K chunks on the .*ix indices. Then we plan a readv for each pack, and readv the data we want precisely out of the pack file.20:55
lifelessluks: we do one readv per pack for the revisions we need. Then one readv per pack for the inventories we need. Another for the text deltas. And finally one for digital signatures.20:56
lifelessluks: each readv is done in ascending order to make range combining as efficient as possible.20:56
luksyep, that's makes my concerns about redownloading completely invalid20:56
lifelesswe're very strong about downloading; your concerns about re*uploading* are valid.20:57
luksI thought people will have to download whole packs if I repack something on the server20:57
lifelessbut a reupload does not screw readers over :)20:57
lifelessluks: ah! any suggestion about where you got that impression?20:57
luksfrom the fact that packs are write-once and then only for reading20:58
luksso I though bzr will treat it as one big blob20:58
lifelessluks: ah; you didn't join the 'better at partial operations' thing to that20:58
lifelessluks: patches for doco appreciated :)20:58
lukslifeless, I did, but only for indexes, not for packs theirselfs :)20:58
lifelessbrb20:59
ubotuNew bug: #172649 in bzr "uncommit is slow with packs" [Undecided,New] https://launchpad.net/bugs/17264921:10
Peng:D21:11
lifelessthanks pen21:16
lifeless*Peng*21:17
PengOnce someone called me Feng a bunch of times.21:17
PengI should change my nick.21:17
lifelessheh21:17
lifelessto Shui?21:17
PengHeh.21:18
PengThat might work.21:18
PengAww, already taken.21:18
lifelessjam: ping21:18
PengActually, registered 2 years 33 weeks ago and last seen 2 years 33 weeks ago. The opers might delete it if I ask.21:18
jamlifeless: pong21:19
lifelessjam: how is the missing-compression-parent patch coming21:19
jamok, I'm still at a bit of a loss for how to force corruption into the repository21:19
jamthat, and I got side-tracked reporting a performance impact for "bzr status" after a merge21:19
lifelessthere's a BrokenRepository test base class somewhere21:19
jamget_ancestry(revision_id) is a lot slower now21:19
lifelesswhat I do with such things is note them and move on21:19
lifelessyah, thats a typical size_history vs partial_history tradeoff21:20
jamwell, you had asked for a callgrind, and I have trouble not looking at it myself21:20
lifelessjam: :)21:20
lifelessI was just saying what I do :)21:20
jamall the time is spent in _lookup_keys_bylocation21:20
lifelessyup21:20
lifelessthe next index layer will remove bisection21:20
lifelessso I'm not that interested in tuning this21:21
lifelesss/.*/am not interested in/tuning this21:21
jamis there any way to upload a file as part of the bug submission?21:22
lifelessif you use the web ui21:22
jamyeah, but I wanted to "just send an email"21:22
jamso I don't have to click 10 times to set the status and the tags21:22
jambut then I don't get a bug number for a long time21:22
jamto upload the callgrind too21:23
lifelesshttps://bugs.launchpad.net/malone/+bug/3022521:23
ubotuLaunchpad bug 30225 in malone "Attach files via email" [High,Confirmed]21:23
jamto21:23
=== mw|food is now known as mw
ubotuNew bug: #172657 in bzr "bzr status after merge slower with packs" [Undecided,Triaged] https://launchpad.net/bugs/17265721:30
lifelessjam: ping; can I ask for an insta-review?21:44
lifelessjam: my trivial progress bar fix - I'd like to flush that21:44
jamI'll give it a shot21:44
lifelessits 3 lines or so :)21:45
jamI don't see it in BB or in my emali21:45
jamemail21:45
jamlifeless: you can paste-bin if you want21:46
lifelessgrah21:46
lifelesshttp://rafb.net/p/ArhA5197.html21:46
jamlifeless: you are changing that function into a generator21:48
jamwhich with python2.421:48
jamwill fail21:48
jamtry/finally doesn't work with a yield there21:48
jambb:reject21:48
jamsorry21:48
lifelessoh foo21:48
jampython2.5 only21:48
lifelessfuckity21:48
jamSince people have troubles even getting 2.4, I don't think we want to depend on 2.5 just yet :)21:48
jamI know we discussed it a long time ago, but it really seems like at least RHEL is taking *forever* to get past 2.321:49
Peng2.6 is close to alpha, right? What about it?21:49
lifelesstry: except ?21:49
lifelessI don't even have 2.4 to test on now21:49
jamlifeless: try/except works (I believe), but I'm not sure what you would catch.21:49
lifelessException: and raise21:49
jamand then a plain "pb.finished()" as well?21:50
jam(outside the exception)21:50
jamwe can go for it21:50
lifelesstry:except Exception: finished();raise() else: finished()21:50
jamlifeless: I don't really like it, but BB:approve anyway21:50
jamnot much else we can do, unless we want to allocate the pb in a non-generator21:50
lifeless        try:21:51
lifeless            for result in self._do_copy_nodes_graph(nodes, index_map, writer,21:51
lifeless                write_index, output_lines, pb):21:51
lifeless                yield result21:51
lifeless        except Exception:21:51
lifeless            pb.finished()21:51
lifeless            raise21:51
lifeless        else:21:51
lifeless            pb.finished()21:51
lifelessaka what try:finally: stands for.21:51
lifelessbastards.21:51
lifeless^ ok with that ?21:51
jamlifeless: except it isn't guaranteed to finish21:51
jambecause someone in 2.4 may not consume the whole generator21:51
lifelessright21:51
jam2.5 actually guarantees it will finish21:51
lifelessthats ok with me21:52
lifelesswe control all the consumers21:52
jamanyway, BB:approve, though the "correct" fix is to move pb = ui.ui_factory.nested_progress_bar() up a level21:52
lifelessit gets fugly fast if we do that21:52
lifelessthere are multiple callers of this21:52
lifelessand that sort of cruft is what we want pb's to avoid21:53
jamlifeless: go ahead21:53
jamlifeless: maybe with a comment so people understand the bit of cruftiness21:53
lifelessdone21:58
warrenbzr diff has -r, but is there an equivalent for a tag instead of revision number?22:19
fullermd-rtag:FOO22:19
fullermd(see 'bzr help revisionspec' for the full list of possibilities)22:20
lifelessbbs22:21
warrenfullermd, thx =)22:22
pooliehello22:53
lifelesspoolie: please save me from elevator music22:59
pooliejam, lifeless, igc, spiv : ping22:59
jam-laptopconference time, right poolie ?23:00
poolieyep23:00
pooliei'm in23:00
lifelessjam-laptop: to setup a repo where a fetch will find a missing text parent, here is an easy way23:11
lifelessjam-laptop: repo a: do two commits, A and B that add and alter a file foo, with id foo-id23:11
lifelessrepo b: do one commit, A, that does not add the file foo23:11
lifelessthen fetching into b from a will only copy the deltas for B, and the text parent for foo-id will be absent.23:12
lifelesslong term this test has some flaws, but it will ensure that the sanity checking we need today is in place; and defers the other work until we have code that will need it, which is a good thing.23:12
jam-laptoplifeless: why wouldn't fetching into 'repo b' copy both A and B23:13
jam-laptopand thus fetch the basis23:13
lifelessjam-laptop: because B already has an 'A'23:13
jam-laptopor are you giving them the same revision id23:13
jam-laptopok23:13
lifelessalternatively23:14
lifelesscommit A and B23:14
lifelessthen23:14
lifelessrepo_b.add_inventory(repo_a.get_inventory(A))23:14
lifelessrepo_b.add_revision(repo_a.get_revision(A))23:14
lifelessrepo_b.fetch(repo_a, 'B')23:14
lifeless*boom*23:14
jam-laptopadding the inv and revision without adding the texts....23:14
jam-laptopok23:14
jam-laptopthankns23:15
lifelessnp :)23:17
lifelesspoolie: yes pqm is sending mails AFAIK23:23
poolielifeless, yes, it just seemed to be delayed somewhere23:25
lifelessabentley: diff is snappy now; thanks23:25
lifelessabentley: in particular no-op diff is 1/2 sec23:25
lifelesspoolie: I want time before going on leave to fix bzr-email to be fast23:32
lifelesspoolie: its by far the slowest part of commit23:32
lifelessabentley: ping; are you on packs yet? :)23:35
jam-laptopwell, I've proven that Knit1 repositories *don't* fail under those conditions23:43
jam-laptopbecause they always copy all missing revisions in the ancestry23:43
jam-laptopwe probably knew that already23:43
lifelessgood;23:44
lifelessknit->pack will today do the same thing; a test that it stays like that would be good23:44
lifelesspack->pack will explode23:44
lifelesswell23:44
lifelessfail to explode23:44
jam-laptopright23:44
jam-laptop:)23:44
lifelesswhere's the ka-boom.23:44
lifelessthere's meant to be an earth shattering KA-BOOM23:45
poolielifeless, well, performance of plugins is less important than performance of the core23:45
lifelesspoolie: depends on the plugin though; if you were evaluating a VCS, would you setup email on commits ?23:47
poolieprobably not in my initial evaluation23:48
poolie(tbh)23:48
lifelesspoolie: I think the core has to support high performing plugins; this is likely a case where core changes are needed23:48
poolieit's important, just imo not as important as the stability and performance of the core tool23:49
lifeless10:48 < lifeless> poolie: I think the core has to support high performing plugins; this is likely a case where core changes are needed23:49
lifelessbecause deployments will eventually expand out to need the whole ecosystem for a given project to be fast23:49
lifelessanyhow, after friday23:49

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!