[00:02] https://bugs.launchpad.net/bzr/+bug/303820 [00:02] Ubuntu bug 303820 in bzr "bzr serve fails (IPv6 socket is created)" [Undecided,New] [00:02] lifeless: ah [00:02] now, lessee if it still eats all my RAM [00:05] Peng_: ok if I put that in the branch? [00:09] yup, still does [00:09] * mathrick frees up some RAM to see if it succeeds eventually [00:09] back after doctor visit [00:29] lifeless: Sure. [00:30] okay, so it still fails to serve anything, except this time it doesn't even throw backtraces :( [00:33] lifeless: Although, I've made two completely unnecessary style changes that don't matter at all since the last version I pasted. http://paste.pocoo.org/show/C5sF67TUM5P1Mw9phfTQ/ [00:38] okay, so it reports "no repository", that's all I can gather from the packets [00:42] mathrick: maybe you started it, or limited it to a path, that didn't include the repository directory? [00:48] poolie: nope, it's a straightforward "bzr serve" with no parameters [00:49] iirc it runs in the directory where you invoke it [00:49] it happens reproducibly for that repo [00:49] so you need to make sure you're not standing in one of the branches [00:49] "branches"? [00:49] let's back up [00:49] oh [00:49] it is in a shared repo, yeah [00:50] lemme try with that [00:50] you must cd to the shared repo directory before running bzr serve [00:50] right, it didn't tell me anything when I started it in the wrong dir [00:54] it also continues to insist on eating all my RAM, which I don't appreciate [00:54] it should print something to give you a clue [00:54] is this with 1.10rc1? [00:55] no, 1.9 [00:55] at what point does it use too much memory? [00:55] 1) when I start it in the wrong dir (ie. in the branch dir instead of the shared repo dir) [00:56] 2) when I start it in the shared repo dir [00:56] 2) seems to be more severe [00:56] as soon as you start it? [00:56] yes [00:56] it climbs steadily [00:57] I've started it about 30s ago, and it's at 700M [00:57] wow [00:57] are you on unix or windows? [00:58] 1300M virt, 700M res [00:58] linux, ubuntu [00:58] please press Control-Backslash in the window where you ran bzr serve [00:58] you should get a debugger [00:59] by which i mean, it should say "(pdb) " [00:59] lemme run it with BZR_PDB [00:59] that shouldn't be necessary [01:00] ^\ alone doesn't cut it [01:00] nothing happens? [01:00] nope [01:00] it's rather slow to react to ^C as well [01:01] nope, kill -INT doesn't help either [01:02] it just sits there, allocating like crazy === jamesh__ is now known as jamesh [01:22] mathrick: does strace show anything? [01:22] dunno, I can check [01:24] anyone using bzr-upload + eclipse integration ? [01:24] I got eclipse exception if auto upload on commit is on [01:24] poolie: nope, the last thing it shows is a futex [01:24] and it's still growing in size? [01:24] yes, completely silent [01:25] oh, wait, it clones [01:25] that is bizarre, i don't understand how it could grow without using [01:25] ah [01:26] so it's stat()ing every dir under my ~/Dev, which is a lot of files, but shouldn't really result in such memory usage [01:27] and this is just when you started it, there's no client doing anything? [01:27] yep [01:27] and there are only stat calls? [01:27] I had the same thing when I started it in the wrong dir, btw [01:27] and C-\ doesn't work [01:28] if you ^C do you get a backtrace in ~/.bzr.log? [01:28] [pid 28773] open("/home/mathrick/Dev/Lisp/cl-rdbms/_darcs/patches/20071026164922-6b9e8-dc57c67fcc6c56c8cb8cf9968ae7598378e3a6c2.gz/.bzr/branch-format", O_RDONLY|O_LARGEFILE) = -1 ENOTDIR (Not a directory) [01:28] there are also those [01:28] I'll check [01:28] I can see lstat64, access, occasional open, and brk [01:29] brk being obviously the one responsible for allocations [01:29] jelmer: ping? [01:29] but I dunno where it comes from [01:29] awmcclain, pong [01:30] jelmer: Quick question about how autoppa and bzr-builddeb work together... I noticed you had an autoppa branch dealing with builddeb, but didn't see the attachment in Bug #252774. [01:30] Launchpad bug 252774 in autoppa "bzr-builddeb magic" [Undecided,New] https://launchpad.net/bugs/252774 [01:31] awmcclain, That was just to be able to build the autoppa package itself using bzr-buildde [01:31] it wasn't about integration between the two or anything [01:31] jelmer: Ahhhh. Understood. [01:40] Am I going to screw up my package branch if I run autoppa on the same branch? [02:03] awmcclain, I wouldn't recommend it, autoppa's use of tags is a bit strange [02:04] awmcclain, also, be warned that it throws away pending changes in your working tree [02:04] Hrm, I am trying to add my project to http://bazaar-vcs.org/WhoUsesBzr as it says I can, however it says I am not allowed to edit that page [02:04] Do I need to do something special other than create an account? [02:04] jelmer: All right. I'll branch and use autoppa from that. [02:05] Oh, now it works after logging in for a third time, never mind :) [02:14] mrooney: i don't know why that happened, it normally works first time [02:14] maybe some kind of caching bug [02:15] poolie: just for the record, no backtraces in ~/.bzr.log [02:15] poolie: I have a rather locked down browser with noscript and no cookies by default so I suspect it was something on my end [02:15] even from ^c? [02:15] not even [02:15] and now I need to sleep, I'll followup to the list tomorrow or something [02:16] mathrick: tomorrow, file a bug with strace running from the beginning of the invocation please [02:16] in a directory with no confidential file names [02:16] will do [02:17] I'll try to make a non-confidential repo showing that, if possible [02:17] sadly, the biggest dir in there is closed source [02:17] so I'll have to make something up of similar structure [02:39] I guess that also answers the question of whether bzr.debian.org can start running the smart server :-( [02:39] does launchpad utilise the bzr smart server somehow or is it all custom? [02:40] or does it just have infinite amounts of memory at its disposal ? ;-) [02:44] LP also runs Loggerhead, so my guess is they have infinite amounts of memory. ;P [02:44] What are we talking about? [02:48] jelmer: launchpad does use the smart server [02:49] jelmer: but not entirely directly [02:49] jelmer: jml should be able to answer questions on that [02:49] jelmer: we have infinite amounts of memory and a crack team of sysadmins. [02:51] jelmer: actually, memory is a problem for us. [02:51] I've half-mentioned it to #bzr folk before. [02:52] thumper, jml: Thanks [02:52] I was half-hoping launchpad was using some magic flag in bzr to get the memory usage down [02:52] I would love such a flag. [02:53] jelmer: another issue we have is bzr subprocesses hanging around. [02:53] jelmer: that *might* be an issue with our SSH server though. [02:54] I've seen it on samba.org as well [02:54] There it's mainly related to connections being terminated abruptly [02:54] and some interaction with the fact that the client time and server time are slightly out of sync [02:55] which prevents me from actually breaking locks without ssh-ing in [02:55] interesting. [02:55] * jml needs to write a manifesto or something on hosting Bazaar. [03:00] I think there would be an audience for that [03:03] jelmer: me too. [03:04] jelmer: it probably wouldn't add much to what I said at the London sprint this year, but who knows, it might help someone solve some of my problems for me :) [03:06] jelmer: what problem does bzr.debian,org have with the smart server? [03:06] * mwhudson feels he is missing context [03:06] jml, :-) [03:06] bzr+ssh: works fine [03:06] but that's mainly because processes are short-lived [03:06] and most branches are small [03:06] there's been talk about running bzr:// as well [03:07] You can do bzr:// with inetd so those processes will be short-lived too. [03:07] Or something. [03:07] but you lose the advantage that it doesn't have any startup overhead [03:08] Right. [03:08] I run bzr+http over CGI, meaning there's startup overhead for every request. Whee. :) [03:09] oh right [03:09] yes, we'd like to run bzr:// at some point, and i've had similar thoughts [03:09] i guess you could just fork for each connection... [03:10] Why doesn't bazaar.launchpad.net use HTTPS? The rest of the website does. [03:12] Peng_: because it runs in a different webserver and there's no compelling reason to use HTTPS [03:12] Peng_: and at least one good reason not to. [03:14] jml: What's the reason not to? [03:15] Peng_: https is slower for end users. [03:17] There's a significant difference for bzr over https? [03:18] sending an https request is several times slower than sending over http [03:18] and i would expect substantially slower than sending over ssh [03:19] Huh. [03:19] poolie: I would be mildly interested in actually experimental data on that last point. [03:19] mm [03:19] it should be reasonably easy with -Dhpss [03:19] .. [03:19] poolie: I expect that establishing an SSH connection takes longer. [03:20] poolie: but, my interest is so mild, I could be a central banker. [03:20] haha [03:20] wow nice one spivvo [03:20] bzr -Dhpss info http://bazaar.launchpad.net/~bzr/bzr/trunk does one hpss round trip [03:21] mathrick: you have the bzr-avahi plugin in stalled ? [03:21] robertc@tungsten:~$ ./test-mem bzr 50 500 5000 [03:21] 50,273516,2173.65908694 [03:21] 500,319888,498.259073973 [03:21] 5000,931452,304.59688592 [03:22] Peng_: whats vmpeak in, MB? [03:22] hm, actually not quite so nice, there's no smart server there [03:22] poolie: heh :) [03:23] the interaction between auto-bzr+http, -Dhpss and log+ is a bit confusing [03:24] lifeless: I glanced in /proc a couple times, and it said KB, but I don't know if it's ever something different. [03:24] jml, sending requests across ssh vs http is approximately the same speed, modulo network variability and measurement error [03:24] or, eyeballing the numbers error [03:24] Peng_: ok, so 270MB, 330MB, 930MB, for bzr [03:24] i'd expect https will be awful if you don't keep the connection alive [03:25] its sublinear, which is good, but a GB is awful large :P [03:26] lifeless: You were indexing a copy of bzr.dev? [03:26] yes [03:26] I got very different numbers when I did it. [03:27] When I ran it for 50 and 500, VmPeak was about half of what you got. [03:28] Peng_: 32-bit machine? [03:28] lifeless: Yeah [03:28] 64-bit [03:29] Note to self: Don't upgrade to 64-bit without doubling your RAM... [03:29] man [03:29] I'm doing another more granular run [03:29] this branch of mine is cursed [03:29] get the slope [03:29] push said this to me: [03:29] ShortReadvError: readv() read 0 bytes rather than 185 bytes at 25704 for "00/00/71/72/.bzr/repository/upload/xkxs5h4uy9mq79wr2t1h.pack" [03:30] 500 is 50% longer than 5000, and only 10% more mem than 50 [03:30] so 500 is looking apealing [03:32] mwhudson: i wonder if that's related to spiv's change for caching when writing out packs? [03:32] can you file a bug with the traceback please [03:32] poolie: ok [03:33] mwhudson: and the formats; if poolie's right at least one side will be a stacked branch. [03:34] spiv: the remote side is stacked [03:34] it's all branch7/knitpack5 [03:35] spiv, poolie: https://bugs.edge.launchpad.net/bzr/+bug/303856 [03:35] Ubuntu bug 303856 in bzr "ShortReadvError on pushing stacked branch" [Undecided,New] [03:35] probably a plugin if core works doing this [03:35] try pushing it with --no-plugins [03:36] (plugins can cause a read, its why the cache-change has to be done at transport not the pack writer to work reliably) [03:38] lifeless: same thing with --no-plugins [03:39] mwhudson: ok, one theory down [03:40] what does 'f' do in the new shelve? [03:42] mwhudson: if you edit bzrlib/repository.py, so that InterPackRepo.fetch has "set_cache_size = None" rather than what it currently is, does that fix it? [03:43] spiv: ok [03:45] If it does, it appears that the issue is that for a target repository to take a delta record and turn it into a fulltext it needs to read from a pack file that's still in progress... [03:46] spiv: yes, it will [03:46] Ideally I think that case would be reading that data out of the local repo, rather than the remote one. [03:46] spiv: I don't think thats ideal [03:47] spiv: because that requires a) fullduplex and b) the server asking the client for data it already has - when you consider streaming [03:47] Ok, well that explains why the non-stacked pack-to-pack case can get away with buffering writes and this case can't. [03:48] lifeless: by "that case", I meant the case where the client is doing the file ops on the local and remote repos. [03:48] spiv: mmm, I really hope we can get just one code path to optimise, and having very clear read-from, write-to patterns will help that immensely [03:49] lifeless: right, one code path would also be ideal [03:49] lifeless: there can be conflicting ideals :) [03:49] I'm not going to try decide which is more important, at least not on my week off ;) [03:49] fair 'nuff [03:49] And on that topic... [03:50] * spiv wanders off :) [03:50] (the cricket is going well, btw) [03:51] Oh, before I go... mwhudson, if that does fix it (as I now expect it will), make sure poolie knows so he can revert that change for the 1.10 final release. [03:51] spiv: I didn't know you were on a week off. [03:51] lucky devil. [03:51] jml: accumulating annual leave is good like that ;) [03:51] ah well, at least the wallabies lost [03:52] * spiv really goes [03:52] mwhudson: rugby is a tier 2 sport. [03:52] * poolie looks [03:53] bah [03:53] lifeless: mark was asking something about getting a dodgy nearly-full-duplex connection over http by using 100 continue responses [03:53] i got BzrCheckError: Internal check failed: Newly created pack file has delta references to items not in its repository: [03:53] on pushing again [03:53] i'm pretty skeptical [03:54] also i think this has already been discussed [03:54] lifeless, poolie: how can i fix by broken local branch? [03:54] my [03:54] oh [03:54] actually this is not urgent, and it's probably a silly idea [03:54] it's the remote branch that's messed up, isn't it [03:54] i think so [03:54] mwhudson: branch the parent with a good bzr, then pull your local dodgy branch into the new branch [03:56] lifeless: branching within a repo will be ok? [03:56] mwhudson: no [03:56] mwhudson: the repo is the thing that is confused [03:56] ah [03:56] right [03:58] When you get that "Internal check failed" error, is creating a new repo the only way to fix it? [04:00] well you can also push with an older bzr, which is definitely sneaky :) [04:02] I get it under entirely different circumstances (pulling, and before the "Internal check failed" error was introduced, I got a different one), but I thought I'd ask since it was getting discussed. [04:02] Not to distract from your issues.. [04:08] Peng_: if it's the one about a newly-created pack file then it should cause the transaction to abort and the repository will be unchanged [04:09] Peng_: if you have a repo with a bad delta, there isn't currently a command to fix the delta up [04:09] Peng_: 'bad' meaning 'points outside the repo' [04:11] lifeless: Okay. ...Will there be eventually? [04:11] Peng_: probably, FSVO probable [04:12] Peng_: with a repo, just init a new repo, branch --stacked from the original again for all your branches, and pull --overwrite them all in - scriptable [04:13] BTW, I'm not using stacking in this repo, AFAIK. :) [04:15] Peng_: if its not stacked, you definitely shouldn't have a bad delta. when do you get the error? and what precise error? [04:15] lifeless: It's a large repo (Python), and bzr-svn is involved. [04:24] Hmm, if I'm reading this right, all the bad deltas are for directories. [04:30] lifeless: Still interested? [04:30] Peng_: bug filing time! [04:30] Eep [04:30] Nice timing. [04:30] Anyway, like I said, bzr-svn is involved. It's kind of complicated. [04:32] There's http://svn.python.org/projects/python/ , Python's svn branches. There's also http://code.python.org/python/ , bzr-svn imports of them. [04:33] I created a repo, made my own imports of svn.python.org, and later branched copies of the ones from code.python.org (which, being bzr-svn, are identical to my imports). [04:33] With one code.python.org branch, pulling it dies with one of the "Internal check failed" messages. Before that message was introduced, it died with something less useful. [04:34] If I pull the equivalent svn.python.org branch with bzr-svn, it works fine and generates the revisions. [04:34] So...yeah. [04:35] .bzr.log snippet at http://paste.pocoo.org/show/FZSj4YB6aiyVinJfSWfw/ :) [04:36] Peng_: definitely file a bug [04:36] Peng_: I smell bzr-svn off the cuff [04:36] Peng_: a new repo suffers this? [04:37] lifeless: I haven't tried. [04:49] robertc@tungsten:~$ ./test-mem bzr 50 500 1000 2000 2500 3000 [04:49] 50,273520,2107.63493299 [04:49] 500,319884,498.597379923 [04:49] 1000,398488,373.639508009 [04:49] 2000,552916,305.507139206 [04:49] 2500,622240,302.449185133 [04:49] 3000,721288,304.617379904 [04:49] Peng_: ^ [04:49] flat at 2000+, for bzr.dev [04:50] That's interesting. [04:52] zooming in on 1K to 2K now [04:52] I think we probably need to look at the file group_size too [04:54] Peng_: if I can twist your arm a little more :) [04:56] lifeless: Unless you want to test combinations of both at once, you could open up index.py, change the first group_size line a little so it won't match the regexp (e.g. add another space before the = sign), and it'll start changing the other group_size. [04:57] (or add a space or comment to the end of the line, I guess) [05:15] Peng_: well, I can run combinations outside it [05:16] Peng_: ideally we'd run both and look for a correation, chi-squared regression or whatever [05:16] robertc@tungsten:~$ ./test-mem bzr 1000 1250 1500 1750 2000 [05:16] 1000,398492,372.275660992 [05:16] 1250,424632,334.12329483 [05:16] 1500,484364,306.166346073 [05:16] still running [05:17] but it looks like 1500 hits a bottleneck of some kind [05:18] where it stops getting faster [05:21] poolie: https://wiki.ubuntu.com/EtcUsingEtckeeperSpec [05:29] Peng_: so - [05:29] robertc@tungsten:~$ ./test-mem bzr 1000 1250 1500 1750 2000 [05:29] 1000,398492,372.275660992 [05:29] 1250,424632,334.12329483 [05:29] 1500,484364,306.166346073 [05:29] 1750,526364,304.78441 [05:29] 2000,552908,304.097445011 [05:29] for bzr.dev at least, 1250/1500 is good, at about 240MB for you [05:33] Hm. Is there any reason for the bzr-search package to be in Experimental but not Jaunty? [05:36] none [05:36] its perfect as is, release kthx [05:36] oh, thats me.. hmm [05:36] :P [05:38] Builds, installs, works. Those seem sufficient conditions for asking for a sync at this point. [05:38] sync it baby [05:39] Peng_: trying the current test-mem on python [05:40] Peng_: I'd love to have an option though to control which thing is altered :) [05:40] lifeless: i don't understand your comment about "not related to the Transport abstraction" [05:41] i understood John's [05:41] poolie: uhm [05:41] nm, now i do [05:41] :) [05:42] i think you're just reiterating what he said [05:42] probably [05:42] that it's not on transport because it's local scratch space [05:42] yes, and also that there isn't any motivation (for me at least) to put it on transport [05:44] 'it'? [05:44] management of scratch files [05:45] mm [05:51] poolie: are we talking past each other, or so violently agreeing we're confusing? [05:51] the latter [05:51] it's fine [06:09] lifeless: I'm starting to get out of my depth here. [06:10] Peng_: no worries, tell you what [06:10] Peng_: commit whatever you've got to bzr-search, as e.g. test-mem.py [06:10] Peng_: and throw me a bundle/merge-request on lp/whatever [06:12] lifeless: I actually have been versioning it (though so far the only commits have been comments and stuff), in my hg repo where I throw random stuff. [06:13] Peng_: lol. [06:13] Peng_: we so gotta make bzr more attractive for you, for that [06:14] lifeless: I'm mostly using hg for historical reasons. Although it's nice that "hg convert" can easily spin something out into its own, independent repo. [06:14] (by filtering the history and creating a new repo) [06:16] thread rename pls. [06:28] lifeless: What do you /call/ the two group_size variables? "inventory group_size" and "text group_size" or "revision group_size" or something? [06:28] mathrick: you have the bzr-avahi plugin in stalled ? <-- yes [06:28] should I uninstall it? [06:29] mathrick: its trying to share all your branches [06:29] mathrick: do you have 'bzr-svn' installed too ? [06:29] yes [06:29] Peng_: yes, that makes sense [06:29] mathrick: ok, so here is whats happening [06:29] lifeless: Which ones make sense? :P [06:30] mathrick: there is a bzr-svn bug where 'bzr branches' takes a lot of memory up [06:30] Peng_: 'revision group_size' and 'text group_size' [06:30] mathrick: and bzr-avahi uses 'bzr branches' to find the branches to share [06:30] oh [06:30] I see [06:30] lifeless: The former for _index_revisions and the latter for the other method? [06:30] I'll remove bzr-avahi for now, I don't use it anyway [06:30] mathrick: uninstalling *either* will fix your problem :P. Probably subscribing to the bzr-svn bug would be a good idea. [06:30] Peng_: yes [06:32] lifeless: https://bugs.launchpad.net/bzr-svn/+bug/54253 ? [06:32] Ubuntu bug 54253 in subversion "Excessive memory usage in python-subversion" [High,Fix released] [06:35] btw, is there a way to serve only some branches in a repo? [06:36] mathrick: bzr_access, i think [06:36] lifeless: nice to see you invalidating your own bug 181367 :) [06:36] Launchpad bug 181367 in bzr "bzr update should work in a treeless bound branch" [Low,Invalid] https://launchpad.net/bugs/181367 [06:37] i'm glad i didn't work on it any further [06:37] poolie: is that a command or a plugin? [06:37] poolie: ah yes, well I claim temporary insanity [06:38] poolie: perhaps having it work-when-it-can would be good, but I suspect too-much-effort for now [06:38] missing smiley here: :-) [06:38] that's what i concluded when i looked into it [06:38] mathrick: its a addon in the contrib directory [06:38] mathrick: it's a script in contrib in the bzr distribution [06:39] mathrick: no, I don't think thats the byg [06:39] *bug* [06:39] jelmer: ping; is there a bug for 'bzr branches w/bzr-svn chews ram'? [06:39] poolie: re: invalidating bug, reminds me of an anecdote from one of PL unis, one of the professors was appointed dean *and* rector simultanously. Reportedly at one point he applied for funds to himself and then rejected the application [06:39] lifeless, yes [06:39] lifeless, I think it has "multi-pull" in the description [06:40] PL=poland? [06:40] lifeless, poolie: thanks [06:40] yes [06:40] anyhow, nice to see he can be objective :) [06:40] we should all do so well :) [06:41] amen [06:41] are contrib scripts packaged? [06:41] jelmer: ah, I've seen one like that [06:42] maybe into doc/ [06:42] https://bugs.staging.launchpad.net/bzr/+bug/262513 [06:42] Ubuntu bug 262513 in bzr-svn ""bzr multi-pull" consumes excessive memory" [Low,Fix committed] [06:42] no! [06:42] but they should be [06:42] should I file a bug? [06:42] ubottu, yeah, that's the one [06:42] Error: I am only a bot, please don't think I'm intelligent :) [06:43] i will [06:43] mathrick, that's the same bug as for "bzr branches" [06:43] jelmer: I see, thanks [06:44] jelmer: do you have any idea what causes that? The bug is a bit short on that [06:45] mathrick, No, I haven't spent time debugging it yet [06:45] hmm, it's marked fixcommitted for bzr-svn [06:45] hmm, perhaps I fixed it already [06:46]