[00:00] So yeah, compatibility would suggest checking whether the class supports that parameter. [00:00] like supports_reverse_cherrypick. [00:01] This is weird. I'm trying to bzr rm a symlink to a directory, but it is complaining about the files in the directory the symlink links to [00:01] An no, I'm not appending a slash to the end of the symlink name [00:02] pickscrape: I believe we have a bug :( [00:02] beuno: wb [00:03] lifeless: passing the --force flag does the right thing (the target directory is not affected) [00:03] Not the correct user experience though. I'll raise it in launchpad. [00:06] Loggerhead! I'm going to see if I can even get CLOSE to setting it up..... ;) [00:07] hrmm, does bzr version symlinks? [00:07] jam: why did you change it from using calltree? [00:07] NfNitLoop: yes [00:07] Huh. cool. [00:07] pickscrape: thanks [00:07] lifeless: you mean callgrind?, because I don't have KCacheGrind on win32, you can change it back if you like [00:07] I noticed a bug the other day re: operation on a case-insensitive filesystem. [00:08] is that known? [00:08] I tried adding file x, when it had already been added as X. [00:08] save('filename.callgrind') should give the same result, btw [00:08] and it committed it a second time. [00:08] and then complained that it was missing. :p [00:10] jam: parameterised it [00:10] NfNitLoop: oh, foo. File a bug ? [00:11] NfNitLoop: thats the sort of thing that makes usage for windows users (or FAT under linux) particularly painful. [00:13] yes. I'm all too familiar with file case issues and still was a bit confused as to what was going on. [00:14] I'll try to create a sample and submit that tonight. [00:17] lifeless: latest version of PyBloom has a bloom.resize() function, so we can poke at it if we want. [00:17] I'm done for the night, though. Off to dance class [00:17] lp:~jameinel/+junk/pybloom [00:18] jam: tchau [00:18] jam: what dance style? [00:18] ballroom, aka latin, waltz, swing, etc [00:18] nice [00:18] I keep meaning to get back into that [00:20] so [00:20] parse_lines and string split make iter_random_one cry [00:21] a C parser would be substantially more efficient [00:25] (I'd like to drive the cost of random access way down, because it its a good total-cost proxy) [00:28] Hm. I can't seem to install Paste 1.1.7 [00:29] Moin. [00:29] Er, 1.71. [00:29] Make that Paste 1.7.1. ImportError: No module named setuptools [00:30] I wonder why all the other "python setup.py install"s I did? [00:30] do easy_install setuptools [00:30] somebody wanna raise their voice on the whole GNOME+DVCS stuff? [00:30] tolstoy: Or, better, use your package management system if you have one. [00:30] hey pygi. I liked your post on the matter. [00:30] which gnome+dvcs stuff? [00:31] Odd_Bloke: This is on Solaris, all inside a user account. [00:31] I don't think I have "easy_install" on there. [00:31] james_w, thanks, I just wrote a big comment addressing concerns Jason raised [00:31] Yes, yes, it feels like I've gone back in time decades by not having root on this box. [00:31] name, the discussion of GNOME migration to DVCS [00:32] tolstoy: http://pypi.python.org/pypi/setuptools/ [00:32] Are setuptools not normally part of the Python? And not required for other versions of setup.py? [00:33] (I'm so far away from Python culture these days, alas. Close as I can get is Groovy.) [00:34] tolstoy: I believe that most setup.pys use just distutils. [00:34] But I'm not sure. [00:34] Okay. Cool. That makes sense. [00:35] Odd_Bloke: yes they do [00:35] pygi: thanks, just read it. Jc2k has raised some similar points to you, and I think it's very important, and not something that's being discussed at the moment. I don't have any voice in GNOME to try and steer the conversation, but I'd be happy to comment on a migration plan for bzr. [00:36] pygi: are you a git fan, or is gitorious just very good for what you want? Would you prefer bzr, or is the abstraction of everything what you would like? [00:36] james_w, as I've written in the post, I'm not a fan of either git or bzr [00:36] I use both where I see fit [00:36] (heck, I've even mentored two bzr students in 2006) [00:37] james_w, currently, something like gitorious doesn't exist for bzr [00:37] and no, you can't count LP, because it is not free [00:37] pygi: ah yeah, I forgot that sorry. [00:37] pygi: why is LP not free? [00:37] james_w, the problem is people keep going in circles, rather then discussing important points [00:38] pygi: yeah, that's what lp is trying to do, but it's obviously not going to work for GNOME. [00:38] that's why I hope my post will bring at least put bug in peoples ear :) [00:38] name, because its not Free by FSF definition :) [00:39] james_w, if you wanna work with me, we could surely write a plan for bzr [00:39] I'm all for it [00:39] james_w, it the end, it all comes down to communication and collaboration, as I've mentioned [00:39] these folks forgot what that is lately [00:40] pygi: are you looking for somewhere to put bzr gnome branches? [00:40] yeah, these tools are supposed to help with collaboration, not cause fights. [00:40] pygi: there's a #gnome-bzr channel on gimpnet that may be more appropriate if we want to have longer discussions on this topic, there's a bunch of people there willing to work to make this happen. [00:41] hey thumper [00:41] james_w: hey [00:41] james_w: getting kinda late where you are? [00:41] pygi: where is your post ? [00:41] lifeless, moment :) [00:41] lifeless, http://pygi.wordpress.com/2008/07/01/broken-by-birth/ [00:41] thumper, it [00:42] thumper, it's a little more complicated then that [00:42] james_w, ok, will come there (You have pm too, btw) [00:42] thumper: aye, just finished a meeting. [00:43] james_w: work style meeting? [00:43] thumper: yup, platform team weekly. [00:43] pygi: are you aware that savannah, and alioth support bzr ? [00:44] pygi: (by which I mean, what is 'like gitorious' mean) ? [00:45] lifeless, yes, I am aware that they support bzr [00:45] ok, cool [00:49] lifeless, it's not only about supporting bzr per-se [00:49] pygi: if you're wanting the ability for people to use whatever DVCS they want, then wouldn't svn as the central repo work best? [00:49] it's about making it easier for everyone involved [00:49] from what I understand the svn integration in bzr, git and hg is better than the integration between the 3 [00:50] night all [00:50] Pilky, that's what people are currently doing mostly [00:50] night james_w [00:50] but that means overhead [00:51] true [00:51] Pilky, anyway, that was just a suggestion on how things *could* be handled [00:51] what is important is for people to finally realize they're not going anywhere with useless arguments sometimes even on personal level [00:51] yeah [00:52] the unfortunately reality about we programmers is that when we start liking a tool we'll fight to the death over it ;) [00:52] I know, but sometimes you have to think above tools [00:53] If I was a new folk in the FOSS community, and read planet GNOME, I'd run away from the project as faster as possible [00:53] because of the recent events [00:53] pygi: I think you make good points [00:53] lifeless, any questions or critics perhaps? Those always help flesh out the idea :) [00:54] pygi: there's a reason I don't do a huge amount of open source stuff (or at least large stuff), it's far easier to get your own way when you're the sole dev ;) [00:54] pygi: well, one thing to note is that bzr itself has an abstraction system; it can natively talk about hg and svn and a number of bzr formats [00:54] pygi: (in a way that none of git/hg/svn try to do) [00:54] lifeless, yep, am aware of that =) [00:55] thanks for pointing it out ;) [00:55] for instance, pqm had a multi-dvcs abstraction [00:55] and now we're ripping it out to replace with bzr plugins [00:56] Pilky, I prefer to stay out, and maintain a lot of stuff outside such projects (the only exception is xfburn (xfce project), but it was initially started there, so can't change it for the time being) [00:56] lifeless, I should really look further into pqm branch then (I guess it's on LP as usual) [00:57] pygi: the only "major" open source project I've worked/am working on is BazaarX, everything else is my shareware apps [00:57] Pilky, ah, no Mac here, sorry :) [00:58] well, pqm is not the cleanest code base. Yes its on lp. But the point there was just that /one/ approach would be to write admin toolchains in bzr, as a lingua franca [00:58] much as is being done with svn today, but being D aware [00:58] pygi: heh [00:59] lifeless, understood [00:59] * pygi is thankful for all the comments ;) [01:00] lets hope I can put them to good use :) [01:01] I'd argue that gnome has a gitorious itself /already/ [01:01] just called 'gnome-svn-orius' [01:02] ehm, well, if you can call the current platform they use that way, yes [01:02] well [01:02] what is it about gitorius as a system that could save admin time [01:03] is it outsourcing - but gnome could have gotten a variety of sites to host the svn migration [01:03] lifeless, ehm, no, its not about that [01:03] lifeless, gnome would have its own instance ofcourse, and not *plain* Gitorious [01:03] right [01:03] I've outlined some of the things that would save admins time in my latest comment [01:03] so why would it be easier for the admins :) [01:04] but there are a lot of stuff we could change to make it even easier for them [01:04] thats why I want the discussion [01:04] which I managed to initiate at KDE (sadly I can't attend Akademy, but oh well) [01:05] pygi: right, so my point is 'what would make the sysadmins life easier' is a good discussion and quite orthogonal to changing vcs -except- insofar as different vcs's have different admin overheads [01:05] lifeless, I definitely agree [01:05] lemme c/p excerpt: [01:05] "Maintainers of specific module could approve users for commit access to “mainline” or main repository, giving administrators less work. New modules/projects could be added by existing GNOME developers after their inclusion has been discussed. New translators and documentation writers could get their commit bits much easier" [01:06] as an example ofcourse, I can't count all the use-cases [01:06] I think that thouse things are actually highly automated at gnome already? [01:07] I was thnking of things like bkor having to script some massive dump+load for all 520 svn modules to handle 1.5 [01:07] some things are, some things aren't [01:07] lifeless, that's the part where I say whats needed prior to migration :) [01:07] K [01:08] lifeless, we could have an interface that would allow upgrading all repos to say new bzr format [01:08] pygi: right; not to mention that bzr can do that it self remotely [01:08] indeed [01:09] I just gave that as an example [01:09] :) [01:10] lifeless, I definitely agree that everything you mentioned should be discussed [01:10] lifeless, but before that, we need to make people collaborate the way they can once again [01:11] we can't make them, we can only try to collaborate with them [01:11] well, yes, and by collaborating with them change the way they think about collaboration =) [01:11] which is a little hard when you get told you're stupid or insane for making an informed design decision :) [01:12] lifeless, eh, even though I doubt anybody will listen to whatever I wrote in the post, they'd listen even less if I actually argued technical reasons behind whatever DVCS they choose [01:12] or if I accidentally mentioned that I prefer one over the others [01:12] pygi: sure, not suggesting you should get technical; I was noting that unpleasantness doesn't help collaboration - and there has been plenty in some of the previous months/years about DVCS [01:13] lifeless, and not only DVCS ... [01:14] * pygi should probably do some bzr hacking this summer if he gets the time [01:15] pygi: that would be cool [01:16] pygi: will you be at GUADEC? I'd be happy to point at internals face-to-face if you are [01:17] lifeless, eh, sadly no :( [01:17] ah well [01:18] but I'd be happy to learn more about internals over irc/mail/jabber/whatever :) [01:33] we've done a lot of scalability work since 2007 [01:33] many more things that only look at small subsets of history [01:34] still need to fix inventories to not be full-tree-objects [01:34] lifeless: Looks like someone beat me to it (by a year or so) reporting case insensitive fs issues. https://bugs.launchpad.net/bzr/+bug/77744 [01:34] Launchpad bug 77744 in bzr "win32 bzr is confused by filename case changes" [High,Confirmed] [01:34] I pasted my test as well. [01:34] NfNitLoop: thanks [01:37] *cough* bzr log *cough* [01:37] mwhudson: eh? [01:38] just carping at lifeless's "many more things that only look at small subsets of history" [01:38] aaah. [01:45] mwhudson: *many* not all [01:45] mwhudson: if you compare what we _used_ to do [01:46] yeah, it's much better, i'm just being annoying [01:46] go merge the loggerhead search integration branch or something [01:56] Boy, that's sad. missing took 26 seconds. pull took 9. [01:56] heh. [02:03] Hm. Are these location-based aliases documented somewhere I can't find? [02:06] fullermd: which version of bzr ? [02:06] i know missing got some love recently [02:07] bzr.dev local, 1.5 remote. [02:08] hm [02:08] should be new enough :/ [02:19] bkor: ping [02:20] spiv: ping [02:20] lifeless: pong [02:21] did you end up having time to look at the gnome hooks? [02:21] lifeless: (I suspect Olav will be asleep) [02:21] AfC: so do I but it never hurts to try :) [02:22] gnome hooks? [02:23] Sure. You don't want 'em just wandering around the garden unsupervised... [02:24] spiv: yes - the ones they run on svn, which we'd need to support [02:27] Ah, no, I hadn't looked. [02:28] could I entice you to look? You've done some server-hook stuff before.. [02:28] morning [02:30] hi igc [02:31] * spiv looks [02:31] lifeless: Apologies if I just spammed you with LP merge requests, I was playing around with the interface and didn't realise it generated emails. [02:32] staging.launchpad.net is a handy sandbox for experimenting with. [02:32] I am trying to misuse bzr as a versioned backup program, I have created the remote and local branches, and am running a server for the remote. Is there a tutorial for this type of misuse or do I have to RT(Entire)FM? ps using bzr send I get the error 'bzr: ERROR: No mail-to address specified.' how do I make it go away? [02:33] spiv: Well, the merge requests are valid, so it doesn't matter if they're in LP proper. I just didn't intend to send lifeless a number of emails (which largely mirror on-list discussion anyway). [02:33] Odd_Bloke: :) [02:34] JamesB192: well bzr send is used to create emails, you probably want bzr push [02:35] Ah, OK. [02:37] lifeless: Did you get a merge request (not via LP) that I sent a little while ago. I CC'd the ML but haven't seen any sign of it and am wondering if it got through. [02:37] yesterday? [02:37] Hold on, I'll find a timestamp. [02:37] or earlier today, cause I saw one yesterday and one earlier today [02:38] lifeless: This one was sent, according to GMail, on the last hour (so around 40 minutes ago). [02:38] But I'm using GMail via IMAP, which can occasionally be a bit funky. [02:39] Subject was "[PQM/MERGE] Remove Arch-style naming from tests". [02:40] yes [02:40] uhm, I think its fine [02:40] OK, good. [02:40] The reason I asked was because I was going to reply to it to mention that it depends on my 'fix-tests' branch, but haven't had the chance to do so yet. [02:41] Then I started playing with LP. :p [02:41] :) [02:43] lifeless: well, I've looked. The post_change_branch_tip hook should be able to do the post-commit actions fairly straightforwardly I think. [02:43] lifeless: (and thanks to my push work it turns out that that hook *does* get run server-side now!) [02:44] lifeless: but we probably need to add a pre_change_branch_tip hook as well. [02:45] lifeless: because there's a fair bit of checking of the commit before it is accepted (there must be a log message, PO files should be syntactically correct, etc). [02:45] spiv: I'm really sorry that I hadn't drawn it more visibly to your attention earlier; I think you were in protocol-3 space [02:45] Yeah, I probably was at that time. [02:46] I don't think adding infrastructure for a pre_change_branch_tip hook should be too hard. [02:48] Also, pre-1.6 clients won't trigger that hook, because they will be using VFS RPCs to update the last revision info. I'm not sure how much that will matter. [02:48] spiv: any chance for GUADEC? :P [02:48] spiv: well, we could add a version check policy on the server? [02:48] If it does matter, we could probably hack up a plugin for the server that traps that VFS call and invokes the right hook(s). [02:50] Hmm, we could do that, I suppose, by e.g. writing a plugin so that the server will reject non-protocol-3. [02:50] spiv: don't client give their verison now ? [02:51] Probably no easier than hooking VFS calls that change */.bzr/branch/last-revision, though. [02:51] They do, but as a string. [02:51] That header was intended for logging rather than greater-than/less-than version comparisons. [02:51] But protocol-3 happens to be new in 1.6, as is that verb. [02:53] I'd rather not get into comparing strings like "1.6b3" and "1.7" and "1.10". [02:53] right [02:54] but isn't the string taken from the string made from the tuple ? [02:54] It is bzrlib.__version__, yeah. [02:54] I'd rather introduce a new header if we do that, though. [02:55] So we keep the concepts of user-agent-label and protocol-level-I-speak separate. [02:56] The current header is "Software version", and is intended to be the former. [02:56] Maybe I should just rename it to "User agent" :) [02:56] right [02:57] but - [02:57] there are two levels of protocol [02:57] there is 'how I encode' [02:57] and there is 'what verbs I will try' [02:57] We can add another header pretty painlessly, but we should think about how we want it to work and what use cases we want it to satisfy. [02:57] anyhow [02:58] I would like to be able to say to bkor: 'bzr X on server, bzr X or > on client, plugin Y on server' [02:58] Ideally, we shouldn't need to rely on the client to indicate what verbs it will try, we should just DTRT based on the verbs they do try :) [02:58] and have : ported checks from svn run reasonably fast and appropriately, and reliably. [02:59] Porting the checks from SVN will take a bit of effort. Not gargantuan, but not trivial either. [02:59] yah [03:00] I'm sure bkor would be happy to do that bit [03:00] Although "poke the viewcvs so that it knows about the new revision" bit will hopefully be unnecessary with loggerhead :) [03:00] (though perhaps not as happy as if we did it :P) [03:01] now for the argh! bit. the DVCS talk is on monday :) [03:01] Well, I'd be happy to help port them. Realistically it'll probably be easier for us to work with them than for either side to try port them alone. [03:01] Ok, so you'd like the pre-commit capability done ASAP? [03:01] if its doable this week, then yes. otherwise no - we can say we know what it will take and its coming [03:02] Well, I don't think it'll be hard. I'll spend an hour or so on it to see if it really is feasible that quickly. [03:03] sweet [03:04] I wouldn't be totally shocked if there's a couple of unexpected bumps between here and having it done, but I wouldn't be shocked if it's plain sailing either. We'll see :) [04:05] lifeless: any updates on btree? I haven't seen any commits or posts since I left :) [04:16] jam: just doing C parsing for _leaf_nodes [04:16] jam: 60% of that 110 seconds is claimed to be in the parsing [04:16] lifeless: sure [04:17] If you consider... it takes 1s to read all nodes (iter_all) [04:17] and we get maybe 100-200 nodes per leaf [04:17] and we have a 50% cache rate [04:17] 200 * .5 * 1 = 100s [04:17] lifeless: pyrex or raw C? [04:18] pyrex [04:18] adapting _knit_load_data [04:19] :) [04:19] I'm not sure its wrong [04:19] but profilers - meh [04:19] anyhow, finding out is relatively cheap [04:20] what dance did you do tonight? [04:21] its disconcerting that the three-level test kicks in before the progress bar debounce [04:36] we did rumba and waltz tonight, focusing on underarm turn and breakaway 5-something [04:36] 5position whatever, where your foot is crossed behind the other [04:37] lifeless: yeah, the "nothing is happening" is why I generally run with -v [04:37] lifeless: So I'm poking around at our bloom stuff, and I was noticing that our bottom level rows have more nodes than I think you give them credit for [04:38] I'm seeing bloom filters with 9,000 entries [04:38] I think the bottom level is < 100, but you have at least that many in the internal node [04:38] So to be useful, we would need bloom filters of ... 9000 bytes [04:39] Now, this is for the biggest pack in all packs [04:39] wheee thats quite some compression [04:39] oh right [04:39] yes, I was saying that only the bottom level of filter is really useful [04:39] lifeless: that *is* the bottom [04:40] leaf nodes + 1 internal = 9000 records per internal node [04:40] Otherwise you only have leaf nodes [04:40] oh [04:40] and if you've already read them ... [04:40] so the *only* time these are useful is when you have 1 root node, because you didn't fill it all the way === lamont` is now known as lamont [04:40] eh, this doesn't make sense to me [04:41] lifeless: each internal node can map to 100 other nodes [04:41] right? [04:41] how many keys are in each leaf?, and how many leaves per internal node [04:41] And each leaf node can hold 90 [04:41] so, 9K yes [04:41] but - we *are* seeing IO avoidance already [04:41] I don't know what our FPR is in practice [04:41] I get approximately the same values by just using "Num nodes / row_length" manually looking at the headers [04:42] lifeless: we aren't getting any hits on the big packs [04:42] it is only packs with < 9k records total [04:42] so you have a single root node [04:42] jam: ah, interesting theory [04:42] jam: making indexbench tell us this would be good [04:42] lifeless: -Dindex will print out the bloom statistics for every finalize [04:42] also, trying say a 512byte filter - which will push out some nodes from the internal [04:42] Finished node 1 with bloom filter . [04:43] lifeless: the only problem is the finish_node() has no idea what level it is at, so I had to hack in another debug statement before it gets called [04:43] jam: yes, I added that :P. What I mean is at the external level, to tell us what the aggregate usefulness is of the bottom internal row [04:44] so, there is one other case that it slightly helps, which is the 'last added' node. [04:44] because that one is also not always full [04:46] still, minimal [04:46] ok, there is someting for you to play with [04:47] I shelved the pyx code contents pushed the rest [04:49] jam: ^ [04:49] oh, and iix fits more than tix does into each leaf node [04:49] yeah, I saw the commit, [04:50] I bumped up the bloom to 1K, and we still get 7K entries, and 98.8% full [04:50] Wrote: 4710400 with 1K blooms instead of 4706304 with 256 byte blooms [04:51] so only added 1 page in all of that output :) [04:51] I guess we have some room to spare [04:53] still, 98% is not saving much [04:53] 2K? [04:53] (meep) [04:53] well, we have a few other ones that start hitting 45% full, which is "optimal" [04:53] I'll try 2K next [04:53] but I'm running the benchmarks first [04:53] meep? [04:53] lifeless: note the 'gray level' isn't == FPR [04:55] jam: its the number of bits on; 2% is the number that can possibly miss, and we get 5 chances at that per value [04:55] right [04:55] jam: so 10% on average, or 'not saving much' :) [04:55] sure, but 10% >> 2% [04:55] true [04:56] So, I wasn't translating all the way up to the FPR because we both know the maths :) [04:56] I'm going to give a shot at 2K when this test run finishes, I forgot to supply --no-graph [04:56] anyhow, meep was simply expressing dismay at 2K/4K being bloom [04:57] may also find the chunk code is starting to generate more slack space at that fraction too [04:58] yay gcc [04:58] _parse_btree_c.c:91: warning: â defined but not used [04:58] _so_ not helpful [04:58] well, I just hit AssertionError: Somehow we ended up with too much compressed data, 4221 > 4096 [04:58] with 2K bloom filters :) [04:59] jam: I think that my reserved hack is not quite solid enough [04:59] I'm not quite sure why, as I'm using "2*capacity" [04:59] I guess maybe we don't get our 2:1 compression with < 2K bytes to work with [05:00] dropping to 1.5 [05:02] ultimately, I think we need to make blooms too big to really be useful, which was something I felt before [05:02] It might be a bit better with the md5 bloom [05:02] hard to say at these levels ... [05:04] lifeless: I can see that 2k blooms work "better" than 1k blooms for miss torture [05:04] 309k bloom misses, versus 282k bloom misses [05:04] and it saves ... 13.2s => 12.6s [05:05] search_from_revid is *slower* 1.597 => 1.639 [05:05] get_components is faster 15.143 => 14.394 [05:05] with 24.4k hits versus 19.8k hits [05:07] jam: well the bloom gets compressed [05:07] jam: but a grey one won't compress *that* well [05:07] well, a 100% grey compresses very well [05:07] all 1s :) [05:07] but is useless :) [05:08] also, 100% grey == black! [05:08] yeah, unfortunately optimal blooms don't compress well [05:08] lifeless: black ... or white depending on your terminal color [05:08] and whether you consider 1 whiter or blacker than 0 [05:08] I'm seeing 90% grey at the level 1 nodes for the big packs, and we are down 13k => 8k entries for iix, and 9k => 4.6k for tix [05:09] with 75% for tix [05:09] and a corresponding large increase in l1 nodes [05:09] so broadly [05:09] we can - not use a bloom [05:09] - use a bloom for the entire table, and page it in size [05:10] I would say if we want to use a bloom, it needs to be on a page of its own *next to* the node mapping page [05:10] use blooms for the bottom row only (giving use better fan out high up the tree [05:11] 4096 bytes for 9k records would give us ~4 bits / entry [05:11] (3.64) [05:11] which is in the ~20% FPR range [05:12] lifeless: the one advantage of a whole-pack bloom is that you know what pages to read in, as soon as you know how big the bloom is [05:12] for as many keys as you want to check [05:12]