=== Mez|OnAir is now known as Mez [00:13] is there a bzr 1.0 backport for gutsy available? (I tried myself, but it doesn't seem to be working right) [00:14] jdstrand: http://bazaar-vcs.org/DistroDownloads [00:14] I just ran dpkg-buildpackage on bzr 1.0~rc2-2, but am getting errors against a 1.0.0.candidate.1 tree [00:15] spiv: thanks! [00:20] jam: I was going to review your Graph.heads optimization, but things seem to be a bit messy-- e.g. it adds preload_parents, which your "Fixes for find_differences()" then takes away. [00:21] hi abentley [00:21] igc: Hi. [00:21] abentley: re pkg_resources, is it easy to include in our code? [00:21] seems to be part of setuptools [00:22] Yes, it is part of setuptools. [00:22] and the pep adding it to python is a future thing I gather [00:22] s/is/in/ [00:22] Yes. [00:22] it looks the right idea ... [00:22] Considering we want bazaar to be installable as an egg, using setuptools doesn't seem too crazy. [00:22] I'm just not overly comfortable making setuptools a dependency just the sake of this [00:23] Well, it may make more sense to reinvent the wheel. [00:24] But I thought it was worth discussing at least. [00:24] I'm pleased you mentioned it [00:24] I wasn't aware of it [00:24] I have no idea how hard or easy it would be to include by itself. [00:25] I can't find anything along those lines after a brief search [00:25] it probably isn't a big deal ... [00:26] but it feels a little like a huge hammer to smash a small nut in this case (to me at least) [00:26] I just want to load doc from files and not python strings :-) [00:27] What it does give you is location-independence for resources. [00:27] I was happy with leaving the docs where they were but it doesn't work with the Windows installer apparently [00:27] If your file is in a zip, getting a python string is better than getting a faile. [00:27] s/faile/file [00:29] true [00:30] pkg_resources is 82k, 2.5k lines. [00:30] So it would be a lot to manage ourselves. [00:31] What we could do is strive for api compatibility. [00:31] Or we could just solve it a simpler way for this case. [00:31] Hmm [00:32] I would like to get ref material out of the UG for 1.0 so ... [00:32] how about I raise an issue about making an api ... [00:33] that is compatible with pkg_resources [00:33] Sounds good to me. [00:33] for 1.0, I'll make the files py ones with big strings in them [00:33] so things don't break in zip files [00:33] Eh? [00:34] well ... [00:34] you can just have an API that returns the contents of a file as a string. [00:34] your point about loading from files will break [00:35] ok [00:35] pkg_resources can give you filenames if you must have them, but this will mean extracting the files from the zip first. [00:36] So IIRC, it's encouraged to retrieve file-like objects or strings. [00:36] I don't need filenames really [00:36] help_topics._load_from_file() currently takes a topic and returns a string [00:36] that's all I care about at the high level [00:37] internally, I get that content via deriving a filename but ... [00:37] Right. So the resource API should return strings or file-like objects, rather than paths. That's all I meant about zips. [00:37] I could derive a string in py module instead say [00:38] You mean stuffing our existing text into py files? [00:38] Doesn't that defeat the purpose of keeping them in doc/en? [00:39] doc/en is no good for alex [00:39] my orginal patch had them there and he wanted them moved [00:39] the windows installer doesn't package doc/*, just the generated files I believe [00:40] I don't think that's a good tradeoff. [00:40] ok ... [00:40] so if we leave the files in doc/en ... [00:40] I don't know enough about the windows installer. [00:40] and mark them as User Reference only topics ... [00:41] then the runtime system doesn't need to find them [00:41] But it must have a way of accessing resources. [00:41] and alex's problem goes away [00:41] Probably pkg_resources, in fact. [00:41] actually, only generate.doc.py needs them -to build bzr_man.txt during make [00:46] igc, interesting mail about including batteries on windows [00:46] is it really unavoidable that there are two somewhat incompatible windows environments? [00:46] that kinda suck [00:46] s [00:47] poolie: yeah - we have some work to do I think [00:47] I saw a complaint yesterday re hg [00:47] the grumble was that it bundled a dozen plugins but people had to turn them on :-( [00:48] in comparison, git has bisect etc enabled by default [00:48] so, in other words ... [00:48] git was deemed low admin because it just worked out of the box [00:49] we shouldn't try to satisfy everyone but ... [00:50] I'm beginning to think we ought to at least be bundling more, perhaps via meta-packages [00:50] disk space is cheap :-) [00:51] it's bandwidth in downloading the stuff that I think people get concerned about [00:51] i think we should too [00:51] even then, that's pretty small [00:52] except perhaps for dependent libraries like qt [00:52] maybe we could offer a "standard" and "minimal" install [00:52] yes [00:53] Compile time and general system dirtiness are concerns too. [00:53] how do you mean? [00:53] particularly on Windows and OS X [00:53] qt, to take your example, takes AGES to build. Longer than Firefox even, last I looked. [00:54] oh, on BSD or Gentoo? [00:54] ("you build qt?" :-) [00:54] i wouldn't expect most people would build it from source [00:54] Well, I don't, no. And I don't want to start because of some third-level dependancy of a package for a VCS I'm going to use the CLI of ;) [00:54] not to neglect those who do have to, but it's probably a minority cas [00:54] e [00:55] but you wouldn't install a binary of it? [00:55] fullermd: I'm good with qbzr, bzr-gtk and bzr-svn being separate installers [00:56] Well, no. But even if so, that falls into the 'system cleanliness' side; I don't like installing extra stuff I'm not going to use, particularly big bunches of extra stuff. [00:56] I mean, just installing bzrtools needs graphviz. [00:56] So I'm looking at py2exe, and there are recipes for including data files. [00:57] it's more about what we bundle as "standard" plugins to match the expectations of functionality people have [00:57] And that pulls in half of X. [00:57] http://www.py2exe.org/index.cgi/AddingConfigFiles [00:57] abentley: I'll take a look [00:57] So I really think this is not a tool limitation. [00:57] Hmm. If I get bzr.dev using rsync, the first thing I receive is bzr.dev/.bzr.backup/? [00:58] fullermd: graphviz is supposted to be optional. Please smack anyone who makes it mandatory. [00:59] abentley: I maintain the port; what sort of masochist do you take me for? ;) [01:00] I could OPTIONS-ify it I s'pose. [01:00] hehe [01:01] I think Bazaar is the only hard dependency of bzrtools. [01:02] The rest are used by one command or another. [01:02] The deps in the port are currently bzr, rsync, and dot. I think I'd just leave rsync; it's not too big, and it's self-contained. [01:03] So you're not supporting baz-import? [01:03] Well, I haven't really done much with the port except update the versions; lulf created it originally. [01:03] Ah. [01:03] I just inherited it and bzr from him. [01:03] Do I know lulf? [01:04] Ulf Lilleengen. [01:04] Think he's still listed as maintainer of the baz port. There's a relaxing job ;) [01:05] Lol. [01:05] Is there a good reason for bzr.dev to contain both .bzr and .bzr.backup? [01:05] Uhm. Make that "Is it on purpose that bzr.dev contains both .bzr and .bzr.backup?" [01:06] Sounds like an accident to me, but not a harmful one. [01:06] It just doubles the download volume. [01:06] :) [01:08] ndim, downloaded using what method? [01:08] rsync? [01:09] igc, i'd like to talk more about batteries-included, but we should do the release first [01:09] poolie: it's a 1.1 thing IMO [01:10] igc, on the movement into the user reference [01:10] poolie: rsync, yes. my local bzr is a little too old. [01:10] (which i know is a different topic) [01:10] ndim, i'll move the backup [01:11] hm, i can't move it myself [01:12] igc, i share jam's view about the desirability of keeping docs in text files rather than inside py files [01:12] if possible [01:12] poolie: I'd prefer that too [01:21] poolie: Have you considered how to un-merge using annotations? [01:22] i.e. given revisions x and y, where y is descended from x, remove all the changes introduced by x? [01:22] abentley, i have not, but it's an interesting idea [01:23] I've been thinking about it as an approach to reverse-cherrypicking, but it seems complicated. [01:26] reverse-cherrypicking is about the only thing that merge3 can do that annotate cannot. [01:38] spiv: ping [01:39] jam: pong [01:40] spiv: do you know what the per-node overhead is for a set() ? [01:41] jam: very similar to a dict [01:41] which is? [01:41] jam: standard PyObject overhead (the PyObject_HEAD fields) plus a couple more fields, plus a "smalltable" [01:42] jam: depending on the number of entries in the set, it'll put all the entries in the PySetObject if they are under a certain threshold [01:42] jam: otherwise it'll malloc separate space for the set items. [01:43] well, I'm doing a set() of the ancestry of nodes in bzr.dev [01:43] so I have 7.3k sets [01:43] (the threshold seems to be 8) [01:43] and 63M nodes inside those sets [01:44] (gdb) p sizeof(PySetObject) [01:44] $1 = 100 [01:44] I'm seeing 29 bytes per entry in the set [01:44] well, lets try it a different wy [01:44] way [01:45] So is just holding the sets in memory the problem, or is it when you try to do an operation on the sets? [01:45] 7.3k * 100 = 730KB [01:45] Well, I just loaded them, and it takes 1.8GB of Ram [01:45] so the PySetObject overhead doesn't seem to be the expense [01:46] But the 28 bytes per entry *in* the set is [01:47] spiv: shouldn't it be something like 10 bytes per node? Something to give you a pointer to the real PyObject [01:47] What are the objects? Strings? [01:47] or am I triggering something funny in the "smalltable" code [01:47] spiv: yes [01:47] but they are bzr.dev revision ids [01:47] so ~60 bytes each [01:47] 15k total [01:47] The "smalltable" code is basically just "if len(set) < 8: store in the struct directly for cache coherence/memory fragmentation goodness" [01:48] hmm... maybe that is 900MB, seems a bit big [01:48] ah, 900 KB [01:48] that makes more sense [01:48] Also, (gdb) p sizeof(PyStringObject) [01:48] $5 = 24 [01:48] spiv: sure, but aren't the PyStringObjects re-used? [01:48] If I have 1 set with 15000 entries [01:49] set1 = set(...15kentries...) [01:49] set2 = set() [01:49] set2.update(set) [01:49] set2.add('a') [01:49] jam: they might be reused, depending on how you construct them. [01:49] jam: there's an easy way to find out [01:49] jam: do a set(id(x) for x in ...) :) [01:51] But if you're building new sets by "set2.update(set1)" like in your example there, then it'd just reference already existing objects, yeah. [01:51] spiv: If I do you set(id(x) ...) and then do a set difference [01:51] I get the same number of nodes [01:51] as just doing a direct set difference [01:54] So "len(my_set)" should always be the same as "len(set(id(x) for x in my_set)))" (assuming the __eq__ of the setitems behave normally). [01:54] spiv: so doing: all = set() [01:54] for rev, nodes in self.ancestries.iteritems(): all.add(id(rev)); all.update(id(r) for r in nodes) [01:55] But what's more interesting is "ids = set(id(x) for x in my_set); ids.update(id(x) for x in my_set2); len(ids)" [01:55] (build up a set of the id of every piece I can find) [01:55] (Pdb) len(all) [01:55] 15499 [01:55] So all of the revision id strings seem to be properly shared [01:55] Ok. [01:55] Next question: are you sure it's these sets that are the problem? [01:56] spiv: well, I can say that it is during creation of these sets that my memory bloats [01:56] spiv: the code is here: http://bzr.arbash-meinel.com/plugins/test_graph/ [01:56] Because it sounds to me like 15k strings of ~60 bytes + 7.3k sets isn't adding up... [01:56] jam: http://www.twistedmatrix.com/users/spiv/countrefs.py [01:56] spiv: I completely agree, that is why I'm confused [01:57] jam: use that or something like it to get a count of the number of instances of various types. [01:58] (Pdb) countrefs.mostRefs(10) [01:58] [(152841, ), (8158, ), (7411, ), (3030, ), (1499, ), (1156, ), (985, ), (779, ), (625, ), (566, )] [01:58] 152k tuples is a bit [01:59] but still not 1.8GB worth of data, IMO [01:59] Well, it depends on how big the tuples are... [01:59] You can find out :) [01:59] all_tuples = [t for t in gc.get_objects() if type(t) is tuple] [02:00] Then you can try e.g. max(len(t) for t in all_tuples) [02:00] Pity there's no "histogram" builtin ;) [02:01] spiv: sum(len(obj) for obj in gc.get_objects() if type(obj) == tuple) [02:01] 379583 [02:01] with 152k tuples [02:01] that puts most of them at 2 [02:01] You can get the median with print sorted(all_tuples, key=len)[len(all_tuples)//2] ;) [02:02] Hmm, so no smoking gun there either. [02:02] It's possible the leak is in a type not tracked by gc (not a "heaptype" or whatever the jargon is). [02:02] print sorted(all_tuples, key=len)[len(all_tuples)//2] [02:02] (None, False) [02:02] weird [02:03] Oh, that printed the median tuple, rather than the length of the median tuple. Heh. [02:03] Still, that supports your "most of them are len 2" calculation :) [02:03] but it is still 2 [02:04] Ok, so the space isn't consumed by the sets, the strings, or the tuples. [02:04] 1500 lists, of total length 10k [02:04] again, not super long [02:04] spiv: can you try the code and see if you get the same effect? [02:05] (it is a bzr plugin, put it in ~/.bazaar/plugins and just run: "bzr test-graph bzr.dev") [02:05] So, at this point I'd try running without the optional C extensions (in case there's a leak there), and without plugins (in case e.g. the svn plugin is getting involved and thus getting its known memory leaks involved). [02:05] Ok, will do. [02:06] Well, I can do that, but the code is generally really simple [02:11] jam: I don't think I want to run that command :) [02:11] why? [02:11] jam: I only have 1G of RAM on this laptop :) [02:11] don't want to use the ram? [02:11] spiv: you could try it on something smaller [02:11] I will :) [02:11] bzrtools works for me [02:11] but doesn't bloat nearly as much [02:12] spiv: on my machine I have definitely tracked it down to the get_ancestry_for_interesting() [02:12] before the loop I'm at 25MB (with the revision graph loaded) [02:12] after the function, I'm at 1.8GB [02:15] spiv: also, python spends a long time during close, I assume doing garbage cleanup [02:17] Strange. I just "fixed" the BB performance issues by rebooting. [02:18] It was at 45% IO wait with nothing running. [02:20] How do you handle shared code in projects? We have a CVS project where some source is shared between us and another organisation. [02:21] spiv: it seems that sets simply wrap a dict in python2.4... [02:21] * data is a dictionary whose values are all True. [02:22] jam: oh, I was assuming you were using 2.5 [02:22] Talden: Generally, you package the shared code separately. [02:22] jam: which has a dedicated set type (based on the dict type's code, but a little leaner) [02:22] spiv: I am, I just have the 2.4 code available :) [02:22] Talden: libraries lend themselves to this treatment especially well. [02:23] spiv: downloading the 2.5 source now [02:23] abentley: We both contribute to the shared components code. Many of our developers would simply end up having to handle two separate projects and shipping code into the dependent project (with a general understanding that it's not maintained there). [02:24] This is acheived in CVS with a little backend hackery to mount the shared code in an appropriate path in the CVS backend. [02:24] jam: I have a bzr-svn import of python trunk and the py3k branch on my laptop :) [02:24] spiv: oooh [02:24] it won't matter for this code [02:24] as it only focuses on merge nodes [02:24] I am NOT promoting backend hackery and am fully aware that this isn't going to fly in DVCS land. [02:25] Talden: okay, I don't understand your requirements yet. [02:25] (1) Neither org can see the others non-shared code. [02:25] (2) We both contribute to the code on HEAD and frequently make contributions on our own branches that make their way to HEAD via merges. [02:25] (3) History of shared content is visible to all (to help make sense of changes originating from two orgs with very different core-business). [02:25] Why would having a separate tree lead to stuff not being maintained? [02:26] abentley: Maybe I've misunderstood your suggestion [02:26] spiv: so if I'm looking at it correctly, the overhead of a set entry should be just 8 bytes, right? [02:27] a 'long' for the hash, and a PyObject* for the key [02:27] So for the shared code, I'm suggesting that should be one branch, and your non-shared code should be another branch. [02:28] Branches or separate ancestry? [02:28] Totally separate ancestry. [02:30] Right... so a developer can't 'mount' the shared code inside the path of their branch of main code right? Or does bzr know to handle nested branches somehow? [02:30] jam: that sounds right to me [02:30] so still, something weird is happening [02:30] 63M keys *8bytes is only 500 MB [02:30] A lot, but I'm seeing almost 4x that [02:32] It's possibly just python's memory management. [02:32] Creating/resizing lots of sets might be leading to memory fragmentation. [02:33] Talden: There has been work on that, but it's not complete. [02:33] There are packages that let you set up a nested tree, but don't keep things up-to-date. [02:40] spiv: well, doing "set1.copy().update(set2)" doesn't seem to improve things [02:40] Talden: if by "mount", you just mean sticking one tree inside the other, that's easy. [02:40] * igc lunch - bbl [02:40] and that hits on 5.6k of 7.3k nodes [02:41] I guess it is actually "s = set1.copy(); s.update(set2); s.add(r)" [02:41] but I guess that involves 2 resizes, and I don't know what sort of internal grow algorithms are used [02:43] spiv: I think you are right about the malloc() issues [02:44] If I change the calls to do: [02:44] jam: so, one interesting point is that sets' internal tables are larger than the number of entries, btw. [02:44] ancestries[revision_id] = ancestry.copy() [02:44] Then it drops to 1.4G [02:44] from 1.8G [02:45] spiv: all of the "dummy" nodes? [02:45] jam: they can be twice to four times larger than the number of entries after a resize, I think; obviously you want some sparseness to avoid too many hash collisions, and provide space for new entries, etc. [02:47] (maybe frozensets are more compact?) [02:47] spiv: they seem to always be powers of 2 [02:47] newsize <<= 1 [02:47] mask = newsize - 1; [02:47] So probably when we trip past 8k revisions [02:48] it jumps to 16k for each revision [02:48] and once we get past 16k (in a few revisions) it will jump to 32k [02:49] jam: well, see also "set_table_resize(so, so->used>50000 ? so->used*2 : so->used*4);" that e.g. set_add_key does. [02:50] so grow by 4x if < 50k nodes [02:50] oh, actually have the hash be 1/4 full if < 50k used nodes [02:50] and 2k full otherwise [02:50] so with 8k nodes, we have 32k tables [02:51] and above that it goes to 64k tables [02:52] and 4:1 is about what I'm seeing [02:52] 8 bytes per node, but an average overhead of 3 blank nodes per real node [02:55] jam: what about "ancestries[revision_id] = frozenset(ancestry)"? I don't think it's any more compact than set.copy, but maybe... [02:55] spiv: just finished that one [02:55] still 1.3G [02:55] so the same as .copy() [02:55] but better than not doing it (1.8G) [02:57] spiv: thanks for helping me track this down [02:57] jam: you're welcome. [02:57] jam: I'll get back to other stuff now I think [02:57] sorry for distracting you for so long [02:57] jam: It's been fun poking at this stuff [02:57] Even if it is hard to get good answers about python memory consumption! [02:58] spiv: yeah, I wish part of the PyObject protocol was to have a "mem_size" variable [02:58] or function for objects [02:58] so you could loop and figure out what was consuming your ram [02:59] I think a "mem_size" method would be helpful, but also deceptive. [03:00] A bit like how looking at the memory stats for processes in "top" is deceptive. [03:02] Because accounting shared memory is hard, if three processes share the same pages, should you say that each process "costs" 1/3 of that memory? That's often a bit too simplistic an analysis. [03:10] jam: so one idea for you: http://rafb.net/p/SRFLNW38.html [03:10] jam: trading memory for time, basically [03:13] jam: 'area'. [03:14] abentley: Thanks for the feedback... I went and looked into the support for nested branches and there doesn't look like much there yet. It looks like some developer handholding will be needed for us to maintain similar code-sharing workflow to what we have now. I guess I'll soon see how much impact this might have on my chances of moving us to bzr. [03:15] jam: (what's this plugin checking, anyway?) [03:17] i386: back in syd ? === hexmode` is now known as hexmode [03:18] lifeless: yep [03:18] Got back this morning [03:18] Feeling pretty good for jetlag [03:18] cool [03:18] etc [03:18] hows things? [03:19] good [03:20] relaxing :) [03:20] Yeah me too [03:20] Im attempting to quit smoking [03:20] so im chewing this special gum [03:22] lifeless: james unplugged the router :/ [03:24] i386_: lol [03:27] See what happens when you quit smoking? :p [03:30] Yeah [03:30] Actually this gum does the trick so far [03:30] except I want to vomit [03:30] eek [03:31] when Lynne quit she just cold-turkeyed [03:31] I didn't exactly _quit_. I just ran out, and haven't bothered to buy another pack in the last 7 years? Something like that... [03:33] I have a friend who advocates a different system. Don't fight it; every time you want a cigarette, just eat one. [03:33] rofl [03:35] hi all [03:36] jam: I mailed poolies details to get the canonical commercial packaging folk to do it for my vacation [03:36] when pushing a new branch up to a server in a new location, (via sftp) should the files transfer as well as the .bzr files? [03:39] lifeless, hi [03:40] RichElswick, no, only the .bzr directory is transferred over sftp atm [03:40] so I would need to transfer the files as well? [03:40] then others would be able to download the version controlled files? [03:41] if others download from that directory, they'll get the whole lot [03:41] igc, re your chapter 7 [03:41] it's good, but it seems a bit out of place in this manual [03:41] ok, which is what I sort of figured. [03:41] must be my ftp account setup then. [03:42] what's going wrong? [03:42] hi poolie [03:43] you can test it out if you have a sec... [03:43] http://detroitcreativetalent.com/SSD/RandomGen/ [03:43] is the location I uploaded the files ot. [03:43] to as a test. [03:43] hm, the problem seems to be with your web server [03:44] bzr info http://detroitcreativetalent.com/SSD/RandomGen/ [03:44] bzr: ERROR: Unknown branch format: ' ... [03:44] it's redirecting to an error page or something [03:46] lifeless, can you tell me what arch.ubuntu.com is still used for, if anything? [03:47] ok, thanks [03:52] poolie: in jan if thats ok, I don't really want to page work in [03:52] or should I say, I really don't want to page work in [04:02] yeah, totally fine [04:02]