/srv/irclogs.ubuntu.com/2014/02/10/#bzr.txt

killermoehrehi. at $COMPANY, we use bazaar for our internal development. now the problem appeared, that some repository needs over 500 seconds to download, while the cli says: "Inserting stream:Estimate 751/2255" with no progress. AFAIK any other repository runs well. any advice?09:11
fullermdWhat's it doing during that time?  Churning the disk, slamming the network, chewing the CPU?09:13
killermoehrefullermd: nothing. it takes it RAM share at the repo server (the checkout is via the ssh transport), but the disk and network is idling09:14
killermoehreno cpu usage, too09:14
killermoehrefullermd: you could think someone stopped the job (no, nobody does this)09:15
fullermdCheck both server and client side?09:16
killermoehrefullermd: both server and client side09:16
killermoehrefullermd: would you mind to prefix posts for me with my username, so I get a higlight? this would be really nice :)09:17
killermoehrefullermd: at the server side, the .bzr.log with some debug flags says "Adding the key (<bzrlib.btree_index.BTreeGraphIndex object at 0x22e7f10>, 23942126, 8836427) to an LRUSizeCache failed. value 62761550 is too big to fit in a the cache with size 41943040 52428800", client says "inserting substream: texts", after that both went idling09:22
fullermdkillermoehre: What, to sort it out from all the other conversations happening here?   :p09:24
fullermdkillermoehre: I'm pretty sure that's more of a diagnostic message than an error, so probably ignorable.09:25
fullermdkillermoehre: Still, I have a hard time imagining how both sides could be sitting around doing nothing.  Something's waiting on something.2, and that's either because something.2 is feasting on some resource, something is broken, or things are deadlocked.09:26
fullermdIf (3), then it'll never get anywhere, and points to a ginormous bug.  I doubt it.  (2) is always possible, but is also ginormous bug I haven't seen.  So I'm guessing (1) is the case somewhere.09:26
killermoehrefullermd: I don't see the numbers you're referring ^^09:27
fullermdkillermoehre: Just the 3 things I mentioned in the prior line.09:28
killermoehrefullermd: well, I don't see any kind of resource usage, but maybe you can hint me something. the server is a debian7 with bzr 2.6.0dev2, the clients are very different, but features all the same problem, so I think it's a problem with the server.09:32
killermoehreand now I see with a test at the server, that a local checkout hangs, too09:33
fullermdtop for CPU, iostat for disk, netstat for network.  The more other stuff is being done by the system, of course, the harder to sort out what's happening with bzr itself...09:34
fullermdIf something's memory constrained, there's always the chance it's falling into swap.  That would completely tank performance, but you'd probably notice the pagein/out activity pretty easily.09:34
killermoehrefullermd: yeah, 100% cpu! success, I found something09:35
killermoehreand constant growing RAM usage09:35
killermoehreor not09:35
killermoehrebut enough RAM usage09:35
zygakillermoehre: what is the version of the repository you are branching from?09:35
zygakillermoehre: maybe you can upgrade / pack it?09:35
killermoehreto be fair, this is a VM with 476MB RAM09:36
fullermdWell, 2000-some texts isn't particularly large, so it's probably not a pure history size filling stuff up.  Big files in the history?09:36
killermoehrefullermd: how can I check? I'm only the admin, I don't use bzr very often09:36
fullermdThat _is_ a bit tight RAMwise by modern standards.  'course, it's enough if it's not paging in and out of swap, but it probably can still kill your page cache etc...09:36
fullermdkillermoehre: Well, by just knowing what's in the code/whatever is being versioned.  Images, binaries, ISO's, core dumps, etc?09:37
killermoehrefullermd: should be just php code09:37
fullermd2000 texts of program source code is tiny, but if it's multi-meg images, that's a whole 'nother matter.09:37
fullermdAfter doing a branch, you should have a repo with a single pack in it.  How big is that pack?09:39
killermoehrefullermd: pack?09:39
fullermdll .bzr/repository/packs/09:40
killermoehrefullermd: 32MB09:41
fullermdOK, even a 2002-era memory size shouldn't have too much trouble with _that_.09:42
fullermdSo let's look at zyga's first suggestion; format mismatch.  Look at 'info' on the source repo, see what format it says it's in.09:42
killermoehrefullermd, zyga: format says 2a09:43
fullermdWell, that should be OK, unless you're branching it in _to_ an old format repo, which is pretty unlikely.09:45
fullermd(would probably take some conscious effort to squirrel things up)09:45
killermoehrefullermd: I don't think so I do this with a new checkout09:45
=== alexlist` is now known as alexlist
fullermdkillermoehre: Well, that's all my particularly good ideas.  It's not a tiny repo, but it's not huge either, so I wouldn't expect massive waits on it putting stuff together.09:54
fullermdOf course, VM does mean all kinds of constrained environment, so "all the CPU available for a long time" might not really be all that much in larger terms...09:54
killermoehrefullermd: as I said, I don't see anything in vmstat, too09:55
killermoehrezyga: do you have another idea?09:56
zygakillermoehre: not really09:56
zygakillermoehre: try bzr pack on the remote end09:57
zygakillermoehre: see if that helps09:57
fullermdVM as in virtual machine; the CPU churnings in top.09:57
zygakillermoehre: use a shared repo locally to avoid the traffic in the first place09:57
killermoehrezyga: well, we have multiple developers and they want to checkout the repo first to have it local ^^09:57
fullermdFor comparison, here's a local repo I have of a project full of PHP and SQL files, but with a bunch of images thrown in too.  The history is about 37 megs.  pack takes about 30 seconds.09:58
killermoehrefullermd: I can't see the cpu utilization if I do a remote checkout09:58
zygakillermoehre: doing a checkout or a branch?09:59
zygakillermoehre: checkouts are pretty crap IMHO09:59
killermoehrezyga: checkout09:59
killermoehrefunfact: a lightwight checkout runs fast10:00
zygakillermoehre: checkouts are horrible, don't use them unless you really have to10:00
zygakillermoehre: everything about them is wrong IMHO10:00
killermoehrezyga: it's the way the devs work here10:00
killermoehreI can't change this10:00
killermoehreI'm just the admin10:00
fullermdCheckouts are awesome.  Lightweight checkouts, now, of non-local branches, are nutso.10:00
fullermdThe CPU burning may be on the server side, not the client.10:01
killermoehrefullermd: the bzr pack takes 100% cpu at the server10:01
killermoehrefunny10:01
zygakillermoehre: it runs once and then stops10:01
fullermdWell, yeah; pack is CPU bound, unless you have _really_ slow disks.10:01
zygakillermoehre: recommend your developers not to use checkouts, they probably don't need them and are just used to "svn"10:01
fullermdAnd most VM hosts don't have ESDI controllers, so yours probably aren't that slow   ;p10:01
fullermdA heavy checkout ain't gonna be any slower than a branch.10:02
zygafullermd: the point with checkouts is that they are against the spirit of distribution, just do a local branch, use a shared repository to get local, low-cost branching, push branches back, that's the most efficient bzr workflow10:03
zygacentralized checkouts are all evil, no way to CI, single point of contention, I cannot understand why people even use that10:03
killermoehrezyga: right now the workflow is "checkout → developing → merging with the local checkout → after developing is done, merging with the repo"10:03
zygakillermoehre: branch, hack, merge || rebase, push, have CI merge it10:04
zygakillermoehre: (push to new branch on the far end)10:04
fullermdPfui.  Checkouts are an _excellent_ mechanism to coordinate work on shared branches.  The only rational alternatives are just manually implementing the checkout workflow, and that's silly.10:04
zygakillermoehre: anyway10:04
zygafullermd: I have yet to see a case where a checkout is worth anything over CI gatekeeper10:04
zygafullermd: but perhasps in some specal cases that makes sense10:04
killermoehrezyga: CI?10:04
fullermd"not being nutso"   :)10:04
zygafullermd: for me checkouts are slow and crappy10:04
zygakillermoehre: http://en.wikipedia.org/wiki/Continuous_integration10:05
killermoehrezyga: ah, ok10:05
killermoehrezyga: I don't think this is appliable right now here10:05
zygakillermoehre: well, you have what you have then10:06
zygakillermoehre: slow stuff10:06
killermoehrefullermd: the pack now hangs at "\ repacking texts:texts 147/650"10:06
killermoehrecan I see which file this is?10:06
killermoehreor what bzr does there?10:07
fullermdIt's not hung, it's just digesting.10:07
killermoehrefullermd: hanging := no visable output of success or computing10:08
zygakillermoehre: it's not hang, it's just compressing, wait10:08
killermoehrezyga: any way to see what it's compressing?10:08
killermoehrehmm … lsof10:08
fullermdWon't tell you anything useful.10:09
killermoehreright -__10:09
killermoehreright -__10:09
killermoehreright -_-10:09
killermoehreargh!10:09
fullermdIt's extracting the texts out of the existing packs, and re-compressing them into new.  Maybe verbose logging can tell you text ids, but without a lot of digging that still won't tell you anything too useful.10:10
killermoehrefullermd: so this is an internal action and probably the problem can only be solved by throwing more resources at the machine?10:11
fullermdProbably.  How long did the pack take?10:12
killermoehrefullermd: if I read the ~/.bzr.log right, it took  810 seconds10:14
killermoehrewell, I have multiple debug flags enabled, so this value could be a little to high for production use10:15
fullermd810 seconds?  For a 36 meg repo?  So 24 times as long as on my workstation here?10:16
fullermdI guess it's possible the contents could be pathological somehow.  But yeah, I'd guess the more likely answer is "time to retire that Pentium II".10:17
killermoehrefullermd: again, it's a VM. two cores at 3GHz10:17
killermoehrebrb, coffee wants to come out10:18
fullermdWell, pack isn't multithreaded, so 2 cores wouldn't be much different from 1.10:19
fullermdBut either it's _really_ busy with something else, or that's a 3 GHz single-issue in-order core with no cache, 'cuz I'm only on a not particularly awe-inspiring 3.7.10:20
fullermdOr, as above, possibly an ultra-pathological set of contents.  But I'd be pretty surprised at that sort of disparity.10:20
fullermdI do love VM's, but the species does generally describe an animal that's oversubscribed to heck and back, and gets really cranky when subjected to much load.  And, absent aforementioned magic, that repo really shouldn't be much load.10:26
fullermdA lot of possible pathologies could be come up with to explain things running like molasses, but they're a pretty far reach.  Running out of suds is endemic to the VM world, and accounts for it.10:28
killermoehrefullermd: well, the hosting server has 8 cores and 8 GB RAM, summed up the machines uses 10 cores and 7.5 GB RAM.10:32
zygakillermoehre: re-do that operation on your $laptop10:35
zygakillermoehre: (just before the pack)10:35
zygakillermoehre: if you wonder about virtualization impact10:35
killermoehrezyga: the "bzr pack" at the checkout I did?10:36
zygakillermoehre: not at the checkout10:38
zygakillermoehre: you need the original branch, if you still have it10:38
fullermdThe checkout will do fine.10:38
fullermd(assuming it's not a light one, of course.  But that's silly anyway)10:38
killermoehrefullermd, zyga: started a bzr pack at my $laptop in the checkout I did. seems to "hang", too10:39
killermoehreI will see how long it takes10:39
killermoehrestill running …10:41
killermoehreI like how bzr is logging everything10:42
killermoehreand still running … six minutes now10:44
fullermdThat really sounds excessive.  It's got the CPU nailed the whole time?10:45
killermoehrefullermd: yes11:02
killermoehrefullermd: sorry for the delay. sometimes other work interferes ^^11:03
fullermd.bzr.log probably has some info about the counts of things being dealt with.11:03
fullermdOn my ~37 meg repo here, I've got numbers like11:03
fullermd0.158  repacking 8809 revisions11:03
fullermd3.278  repacking 8809 inventories11:03
fullermd4.600  repacking chk: 8717 id_to_entry roots, 1426 p_id_map roots, 41320 total keys11:03
fullermd14.405  repacking 30226 texts11:03
fullermd29.317  repacking 0 signatures11:03
fullermd(and there it's done)11:03
killermoehrefullermd: http://sprunge.us/MMMK11:05
killermoehrefullermd: I'm killermoehre and /home/killermoehre/tmp/repository/ is the directory where the checkout is11:06
fullermdAlmost 10 minutes to repack 251 texts?  Crazymuffin.11:08
fullermdThe cache overflows aren't _good_, but having 3 of 'em in the whole process doesn't seem like it would make a sizable difference.11:09
killermoehrefullermd: what are those "texts"?11:10
fullermdVersions of files, basically.11:10
fullermdSo 219 revisions and 251 texts means you're changing roughly 1.15 files per commit.11:11
killermoehrefullermd: the devs did11:12
killermoehrefullermd: I never touched the source11:12
fullermdOf course, OTOH, they're obviously pretty big.  251 texts compressing down to 36 megs means (assuming revs etc take up no space, which to a first approximation they do) means that each text after delta compression is taking up about 143k.  That's hyuge.11:12
fullermdCompare my ~37 meg repo with 30k texts, so each is taking up about 1k.11:12
killermoehrefullermd: so what do you suggest? delete the history of this repository?11:14
fullermdSo I suspect there's some significant churn on big files, which usually either means binary or generated stuff (as human-maintained text hardly ever gets THAT big and mobile)11:14
fullermdWell, that would hack around the issue, certainly.11:16
fullermdI don't s'pose it's a sharable repo.11:16
killermoehrefullermd: sharable repo?11:17
fullermdAs in you can give somebody else access to it.11:17
killermoehrefullermd: no. it have to stay in-company11:18
* fullermd nods.11:18
killermoehreI would be killed (at least fired) if I would give access to it for extern11:18
fullermdHey, you gotta sacrifice sometimes   ;p11:18
killermoehrefullermd: with a whole checkout I can go back to any previous revision, right?11:19
fullermdAt any rate, that sorta size sounds a couple sigma off the norm, but I'm still a bit surprised it's that slow.11:19
fullermdEven accounting for my workstation, while not all that screamingly fast, being almost certainly significantly gruntier than your laptop, a factor of 20 difference seems well outside the lines.11:20
fullermd(how's that for tortured sentence construction?)11:21
fullermdYes, you can.11:21
killermoehrefullermd: (I'm german. we invented tortured sentence constructions ;) )11:21
killermoehrefullermd: how can access previous revisions?11:22
fullermdDepending on particulars, usually via update or revert.11:22
fullermdupdate being more for things like "shuffle my whole WT over to this revision for experimentation", revert more for "slam this file back to the state of that rev for committing a backout"11:24
fullermd(as a first approximation; endless details and special cases of course)11:24
killermoehrefullermd: well, I want to find out, which file is why causing this behavior11:25
fullermdHow big is the existing working tree?11:25
killermoehrefullermd: I can't tell the devs "you fucked this up, live with it"11:25
fullermd(a quick mental 'du .' - 'du .bzr' will give you a working approximation)11:25
killermoehrefullermd: 252kB11:27
fullermdBasically, there are 2 possible extremes.  One is that it's a bunch of big files and/or massive changes in each commit.  In that case, well, there's nothing much you can do, it's just how your project goes.11:27
killermoehrefullermd: echo $(($(du -s . | awk '{print $1}') - $(du -s .bzr | awk '{print $1}')))11:27
fullermdThe other is that somewhere back in history, somebody accidentally committed a giant .iso or .core or the like, and then later noticed and removed it because it shouldn't have been there.  In that case, you can in concept rewrite the history without it, and get rid of the deadweight.11:28
killermoehrefullermd: any easy way to find the biggest file in bzr history?11:29
fullermdWell, that's not huge.11:29
fullermdNot directly.  You could try pulling up a 'bzr log -v' then looking through it for 'removed' lines and seeing if any of the files that have been removed look suspicious.11:30
fullermdIt sounds like the latter, really.  252k of size, if it were completely changed every one of the 219 revs, would total up to 55 megs before compression, which is only twice the size of your fully compressed repo.11:30
fullermdSo there's probably some big stuff piled up in the history.11:31
killermoehrefullermd: seems like in the first revision some sql dumps were added, which were removed in the second revision11:34
fullermdSounds like a likely culprit.  A "bzr diff -c1 | diffstat" could confirm it.11:34
fullermd(giving you a quick eyeball of the size of the files)11:34
killermoehrefullermd: 45 files changed, 7351 insertions(+)11:37
killermoehrefullermd: at -c2, I got "41 files changed, 1 insertion(+), 6731 deletions(-)"11:38
fullermdAnd SQL dumps often have ultra long lines, so those ~7000 lines can mean a _lot_ of data.11:38
fullermdYep, I'd call that a safe bet for the bloatsource.11:39
fullermdGetting rid of it would mean redoing the history without those files.  That means everyone's branch/checkouts would need redoing, and anybody's outstanding work in unmerged branches would have to be rebased on top of the redone history.11:40
fullermdWhich may be anything from "trivial" to "fiendishly involved", depending on your particular workflows etc.11:41
killermoehrehrmpf11:41
fullermdDoing it will probably involved some invocation of something in the rewrite plugin, possibly on top of a little manual setup.  But that gets deep into details; I don't have a recipe offhand, so it would probably take some experimentation.11:42
killermoehrefullermd: well, experimentation would be … bad11:43
fullermdWell, obviously you'd make a scratch branch for the experimentation.  It would be _bad_, just potentially time-consuming.11:43
killermoehrefullermd: so I have to say the devs "a other dev not working here anymore screwed off. you're out of luck". i CAN LIVE WITH THT11:44
killermoehresorry for the caps11:44
fullermdThat's why god made devs-who-left-the-company   ;)11:44
killermoehrefullermd: at least it's not my or the machine fault11:45
killermoehrefullermd: thank you very much for your time and help and the insight into bzr. should you ever need help with xfce, don't hestiate to ask in #xfce :D11:47
fullermdnp   :)11:48
fullermdThough I'm afraid ctwm would be terribly disappointed in me if I left it for some newfangled upstart wm   ;>11:49
=== zyga__ is now known as zyga
fullermdvila: Hey, you won First Commit Of 2014.  Does that come with a trophy, or just a little brass nameplate?  ;)14:03
vilafullermd: hehe, no nothing, pure glory ;) (And even there...)14:04
fullermdNot even a cake?  Jeez, what a buzzkill.14:06
=== Kraln- is now known as Kraln

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!