[00:00] it will make subunit-ls clearer, so if you get the time to review, assume its gone :) [00:00] ronny: have you read "xUnit test patterns"? [00:01] spiv: stacked [merge] sent [00:02] mwhudson: no [00:03] ronny: the first 100 pages or so is well worth reading for provoking thoughts about testing [00:03] hmm, added to my should read list [00:04] it's unfortunate that you have to buy the other 700 pages too :) [00:04] ew [00:18] igc: ping [00:22] edcrypt_: pong [00:25] igc: hi, I the one who made that bug report about views [00:25] igc: what is the view-aware API to iterate on the WT? [00:26] edcrypt_: there probably isn't one ... [00:26] I tend to trap most things at the UI layer inside builtins.py ... [00:27] so that the list of files is automagically passed to commands without needing to change their internals [00:27] edcrypt_: there are exceptions though ... [00:27] igc: shouldn't it filter at the tree? [00:27] diff comes to mind [00:27] igc: so that plugins work [00:28] lifeless:it depends [00:28] whether to use a filtered view needs to be considered on a case by case basis [00:29] igc: WorkingTree..inventory.iter_entries_by_dir() gives all files, but probaly I shouldn't be using it [00:30] edcrypt_: so if you look at the code for cmd_diff in builtins.py, you'll see it passed apply_view flag down to the next layer [00:31] edcrypt_: you certainly can use that API, you'll just need to post filter the results by seeing if the path falls inside the view or not [00:31] thst's not hard - see the examples at the top of builtins.py (like internal_files_for_add say) [00:31] s/thst/that/ [00:32] igc: ok, taking a look, thanks! [00:32] np [00:32] edcrypt_: any you looking at fixing ls for me? :-) [00:33] igc: in what cases is the answer 'no' so far? [00:33] lifeless: merge, log, info at least [00:34] igc: hehe, maybe, I have just learned hwo to create a plugin! [00:37] igc: _check_path_in_view() [00:38] ops, diff._check_path_in_view() [00:40] lifeless: reviewed, bb:approve [00:42] woo [00:42] edcrypt_: might be good to make that a public function in views.py instead of a private one in diff.py. [00:47] igc: indeed [00:57] spiv: ok, thats sent. === abadger19991 is now known as abadger1999 [01:08] luke-jr, hi [01:08] poolie: fyi, here's my plan for today [01:08] luke-jr, any chance you can comment on bug 326278? [01:08] Launchpad bug 326278 in bzr-svn "'bzr log' KeyError" [Undecided,Incomplete] https://launchpad.net/bugs/326278 [01:08] 1. get usertest tweaked with some of the changes on the roadmap [01:09] jelmer: ping [01:09] lifeless, pongz0rz [01:09] 2. ping you and we'll update orchadas [01:09] 3. land content filtering after chatting with you [01:09] * igc bbiab - food calling [01:11] jelmer: could you do a pass over your subunit branches, and submit for merge where relevant [01:12] lifeless, k [01:13] thanks [01:14] igc, sounds good [01:14] my laptop won't boot, i'm guessing because of bug 333073 in jaunty :/ [01:14] Launchpad bug 333073 in linux "very slow boot with lvm-on-dmcrypt" [Undecided,New] https://launchpad.net/bugs/333073 [01:14] that gave me a bit of a setback [01:14] but let's talk anyhow when you're done [01:17] Hi - when I run bzr shelve - I get the following prompt: [01:17] Shelve? [yNfq] [01:18] yes No f???? q???? [01:18] Where can I find more info. bzr help shelve does not say anything. [01:19] q is quit [01:19] does ? do anything? [01:20] no [01:20] no - ? seams to cause no which is the default action - lol [01:20] srsl? :( [01:21] f flips the defaults [01:21] Odd_Bloke: is workng on this I thought [01:22] mwhudson - I jest tested that - f seems to do a "full" diff? [01:23] oh [01:23] i'm wrong then [01:24] ok - it easy to figure out - but irritating. [01:30] it should definitely be able to give you help [01:31] igc, back in 5 === bob2_ is now known as bob2 [01:37] spiv: ok with that sent off I'm looking at the next round trip [01:42] spiv: another regression - we're not using find_repository to do the iteration for us [01:42] we're walking up dirs manually [01:44] lifeless: actually, I'm not sure if that ever worked... [01:44] But yes, it'd be good to fix that. [01:44] Also, coffee is good... [01:44] * spiv does something about that [01:44] I'm fairly sure it did at one time ;) [01:44] but I could be wrong [01:45] Me too :) [01:51] new small lp-open patch just sent to mailing list. [01:52] * jml doubles as an IRC notification bot. [01:53] :) [02:04] spiv: I want to delete InterPackToRemotePack [02:09] igc, i'm free now if you are [02:09] lifeless: +1 (minus a bit for not being right there in the code atm) [02:17] lifeless: so that was only doing two things [02:17] lifeless: (IIRC). calling the autopack RPC, and making sure _walk_to_common_revisions walks 50 revs at a time [02:18] lifeless: the streaming push takes care of the former, I think the latter might be taken care of by the InterOtherToRemote? [02:18] lifeless: if so, I don't see any reason to keep it either [02:21] spiv: yeah [02:21] spiv: thats why I want to delete it; and am doing so ;) [02:21] Ok. +1 :) [02:21] I'm glad to see the set of InterRepos shrinking for once :) [02:22] http://paste.ubuntu.com/121655/ [02:22] I'd like to just toss that at pqm - pass land fail I'll look into it [02:22] spiv: ^ [02:27] lifeless: sure, if it passes, then +1 [02:27] lifeless: or rather, bb:approve :) [02:36] jml: your blug reports footnote doesn't seem to exist [02:39] mwhudson: mea culpa [02:43] i think i just made the annotate view in loggerhead twice as fast [02:44] mwhudson: good [02:44] :) [02:44] by not calling annotate_iter twice! [02:44] * mwhudson sighs [02:46] mwhudson: its amazing how little things can make a difference :) [02:46] mwhudson: I have a suggestion for you [02:46] * mwhudson listens [02:46] this is for testing [02:47] create a little delegate branch [02:47] which logs all the calls made to it [02:47] and has a repository object it can return which likewise does the same for the branches repository [02:47] you can do two things with this [02:47] when asking 'why is X slow' you can see the api calls being made [02:48] and secondly you can write a test that no more than a certain number of calls/exact calls etc are made [02:49] hmm [02:49] the former is useful for exploring [02:49] the latter is useful for locking down a behaviour ratchet [02:51] certainly, something i've been doing lately is removing the layers between loggerhead and bzrlib [02:52] and it was in the process of doing this that i found this problem [02:58] spiv: I'd like to make RemoteRepository.*write_group* no-ops like knit/weave repositories [03:00] lifeless: the _real_repo would still need to have a write group, though? [03:00] well [03:00] lifeless: or are we avoiding enough vfs ops now? [03:01] as a start point we may find points that break [03:01] at the moment though it and repo.lock_write are triggering _ensure_real [03:01] actually [03:01] lock_write doesn't trigger _ensure real [03:01] but its a noop for pack repos [03:03] * spiv nods [03:04] light weight commits will still need vfs [03:04] but bound commit shouldn't [03:05] calls 13 through 18 are RemoteRepository.start_write_group(), in the acceptance test [03:18] spiv: ok, BranchFormat.network_name next I think [03:18] spiv: ping [03:18] bah [03:18] spm: ping [03:18] lifeless: pong [03:18] spm: I think pqm is hung, can you check please? [03:18] sure [03:23] did jelmer's clean-tree patch get reviewed? [03:27] lifeless: it was wedged pretty good. should be rockin again [03:27] spm: did you note the cause? [03:28] looked like the old email thing; but that didn't fix it. the sendmail was then hanging around as a defunct process. ie still wedged. [03:28] ok, I'll try the merge again [03:33] damn. it's rewedged itself. looks like on the original one too. [03:34] poolie: I'm back [03:34] spm: ok, patch is bad anyhow, so I suggest just removing the request from me and killing the bzr child [03:35] spm: you shouldn't ever kill the pqm parent when a test is wedged [03:35] is it just me or does bazaar-vcs.org seem really slow these last few days? [03:36] lifeless: yeah - that was a pebkac. :-( did the chroot kill without a dir specified. clobbered everything :-( x 3 [03:36] spiv: we need a 'refresh your data' api for all repositories [03:36] spiv: because this streaming push can work for knits, but we can't let it try :> [05:17] igc, hi? [05:18] hi polie [05:18] poolie:^ [05:19] poolie: you ready to do this orchadas stuff? [05:19] hey [05:19] i am [05:19] i hit an ubuntu bug 332270 that stopped my laptop booting and messed up my day a bit [05:19] Ubuntu bug 332270 in udev "udev repeatedly generates "change" events for the same block device(s)" [High,Triaged] https://launchpad.net/bugs/332270 [05:20] poolie: yum [05:20] poolie: let's start by having a quick call maybe? [05:20] that's why it's called an alpha [05:20] good idea [05:20] i'll call your home? [05:21] sure [05:22] lifeless: I have some theoretical reasons why multiparent with xdelta is smaller than gc, care to have a call about it tomorrow? [05:22] hello jam [05:22] hi poolie [05:23] just heading off to bed after perusing the ~200 emails that came in over the weekend... [05:23] jam: I'd be delighted [05:23] lifeless: k, ping me when you wake up [05:23] I'm going to sleep now [05:35] jam: ciao [05:36] jam: I'd expect the theoretical reason is deltas composition rather than recipes [05:36] but see recipes for why gc is fast at extraction [05:37] um, bzr just said "bzr: ERROR: exceptions.AttributeError: children" and aborted the commit i told it to do. how do i make it successfully commit? [05:38] maco: what bzr version? [05:39] 1.12 [05:42] maco: uhm [05:43] maco: is there more in ~/.bzr.log ? [05:52] lifeless: it spit out a backtrace [05:52] i'll pastebin it [05:57] kernel panicked === maco_ is now known as maco [06:02] http://paste.ubuntu.com/121690/ [06:02] lifeless: ^ [06:09] maco: can you pastebin the output from 'bzr st' please [06:09] (parent appears not to be a directory) [06:10] just "added: .kde@" [06:11] so .kde is a symlink [06:11] i just made ~/config-backup, cd'd there, and did "bzr init" then tried to do "bzr add ~/.kde/share" and some other directories, but it didn't like that, so i made the symlink [06:11] yes [06:11] ok [06:12] so we'll make a bug for the error, because it shouldn't be happening; but if you're trying to backup your .kde directory, a symlink to it won't do that - bzr will version the symlink [06:12] i didnt give it ~/.kde to version though [06:12] igc: merge request sent (make ls aware of views) [06:12] i gave it ~/.kde/share and ~/.kde/Autostart and ~/.kde/env [06:12] you need to either make ~/.kde a symlink to your bzr tree, or actually bzr init in ~/.kde [06:13] maco: you did 'ln -s ~/.kde' inside config-backup didn't you? [06:13] yes [06:13] edcrypt_: thanks! [06:13] but then i did bzr add on .kde/share, not on all of .kde [06:14] igc: np [06:14] maco: right, so I think I know what the bug is [06:14] maco: however fixing it won't do what you want, so give me a second to record the issue [06:14] was trying to get around it giving "NotBranchError: Not a branch: "/home/maco/.kde/share/"." when i did "bzr add ~/.kde/share" [06:14] ok [06:15] igc, ok, i got a report with just one tool running [06:15] is it not possible to give an absolute path to bzr add? [06:15] now i'll try it with some more though that will be slower [06:15] maco: no, its not possible [06:16] maco: bzr requires that all the files it versions are underneath a .bzr directory containing the 'tree' to be versioned [06:16] maco: roughly, thats where you do 'bzr init' [06:16] maco: so if you want to version ~/.kde/share, you must have done 'bzr init' in one of: ~/.kde/share, or ~/.kde, or ~ [06:18] maco: I've filed a bug for the specific error you're getting, but the actual fault is in 'bzr add' :( [06:19] lifeless: ok thanks. a friend suggested avoiding putting it in ~/.kde in case of the next time i blow away ~.kde but i'll just have to remember to keep a backup of the .bzr that's in there [06:19] maco: put it in ~/.kde like so: [06:20] rm -rf ~/config-backup; cd ~/.kde; bzr init; bzr add [things]; bzr commit -m "start versioning ~/.kde"; bzr push ~/config-backup [06:20] poolie: cool [06:20] maco: now config-backup is a backup [06:20] ah ok good idea. thanks! [06:21] my pleasure [06:30] http://benchmark.bazaar-vcs.org/usertest/summary.html [06:46] if i do "bzr remove" it doesnt remove the file, right? just stops doing version control on it? [06:46] if you do rm --keep it will do that [06:49] or well...how do i tell it to not track anything in share/apps/kmail/*/*? bzr ignore? and will that get rid of its past control of those files or not? [06:49] if you have already added them, and you want to just ignore them in future you need to both remove --keep and also ignore them [06:51] igc, still here? [06:53] ok [06:53] thanks [06:53] poolie: yep [06:54] i'd like to make something that will feed usertest output into rrd [06:54] so we have an over-time plot of results [06:54] er, was i supposed to ignore or remove first? bzr diff still shows stuff from those files [06:54] maco: bzr ignore ./share/apps/kmail/ [06:54] i'd probably start with getting the total scenario time for each tool [06:54] and recording that each time we execute [06:54] maco: ignore and remove [06:57] so firstly ,is this redundant with something you've done [06:57] ok thanks [06:57] poolie: no, I'm yet to produce rrd friendly output [06:57] bearing in mind it has to run on a dapper machine with no added software [06:57] and secondly, what should i start from? [06:57] oh [06:58] actually for the overall time, now that we're running each test separately, it's obviously easy [06:58] to do it from the driver script [06:58] i'll do that first then see how we go [06:58] poolie: ok [06:58] poolie: btw, new usertest just pushed with Description column added to the html report [06:59] poolie: that will reduce the # of times I need to explain what each task/action does :-) [06:59] scalability :) [07:00] rev 151 also fixes the colouring when multiple projects are given in the one html report [07:00] i.e. colouring is per project, not just against the first column :-( === chx is now known as chx_sleeping === picturesque is now known as domas [09:10] * igc dinner [09:44] lifeless: I have a branch to fix the prompt up. I'll send it in now (as the branch it depends on has been merged now). [10:52] yo [11:01] lifeless: FWIW, I always run latest bzr.dev, and the streaming push work hasn't broken anything for me. I don't use stacking though. [11:18] Peng_: that's good to know. Does that include using bzr.dev on the server? [11:34] spiv: Yes. [11:35] Except for once half an hour ago, where the revisions on the client and server were incompatible, so I had to do one push over sftp. [12:05] Peng_: thanks [12:05] Peng_: is it faster? [12:06] lifeless: Sorry, but I don't really pay attention. It takes more than 1 second, so I always do something else while it runs. [12:06] Peng_: fair enough [12:06] I always run it through "time", and yet I never read the results. :P [12:06] LOL [12:16] moin lifeless, Peng [12:19] lifeless: are there any plans to support custom objects in the history store at one point? [12:19] jelmer: Good morning. [12:20] ronny: no; as we discussed before I think we'd need to see more examples of custom things before being able to design a good reliable & sufficient interface [12:20] ronny: so its not 'no, never' its 'no, no _plans_ at the moment, open to things as we see people doing it in plugins' [12:23] k [12:23] lifeless, what happened to the annotated code stuff you were discussing with James in London during the summer? [12:23] can't remember the exact name you were using [12:28] the main use-cases *i* see is signed tags, code-review, testresults [12:30] jelmer: "marks"? [12:31] james_w, yeah, that's it [12:32] I don't know if lifeless has done any work on it, I certainly haven't [12:32] I would love to have it though [12:33] the thing that I'm not sure about is how to store the marks [12:33] "file-id hunk-start hunk-end sha-of-hunk" might work I guess [12:43] jelmer: views [12:43] jelmer: erm [12:43] views/filters/tags something like that [12:43] marks maybe? [12:43] james_w: yeh I started sketching htat [12:43] hmm [12:43] haven't gone far; performance is a major issue first [12:45] ronny: so for all of those, I'd definitely help you get support in the core as hooks, so that plugins can do them [12:45] and once we see what plugins *do* with those hooks look at the design for core inclusion of extensible things [12:50] lifeless: atm im puting together a nosetest plugin for subunit [12:51] lifeless: if you've committed anything I'd love to take a look [12:51] lifeless: I understand that it's not your first priority [12:55] lifeless: subunit could use a setup.py [12:55] ronny: yeh, though it builds native C and shell and stuff; setup.py isn't really a good fit there [12:55] scons isn't either though >-) [12:56] lifeless: at least for the python part [12:56] jelmer: scons was an awful nightmare that I regret [12:56] ronny: certainly thats doable yes [12:56] I think there is a branch [12:56] ronny: http://www.somethingaboutorange.com/mrl/projects/nose/doc/plugin_interface.html#prepareTestResult [12:56] prepareTestResult(self, result) [12:56] seems to be the bit [12:57] lp:~lifeless/subunit/filter has the latest subunit branch I was tweaking [12:58] would you mind if i would put nosetest support directly into subunit ? its basically just replacing the reporter === vednis is now known as mars === mvo_ is now known as mvo [13:15] ronny: I'm not sure what you're asking [13:15] ronny: but note bug 332770 [13:15] Launchpad bug 332770 in subunit "provide a nose extension to output subunit" [Undecided,New] https://launchpad.net/bugs/332770 [13:16] lifeless: thats what i do atm [13:16] which is to say, I'd like it if installing subunit added an option to nose [if nose is installed] [13:16] I don't want subunit to depend on nose though [13:16] I consider it a lower level helper [13:16] ok, so separate repo it is [13:17] ronny: hm? Not sure I follow, however I'm tired :) [13:17] so - chat with you in ~ 7-8 hours [13:17] k [13:17] cu later === montywi|zzz is now known as montywi === mvo__ is now known as mvo === serg_ is now known as serg === beaumonta is now known as abeaumont === chx_sleeping is now known as chx === montywi is now known as montywi|meeting [16:10] morning vila, I hope you had a good weekend [16:11] jam: hehe, hi ! Yes, pretty good, didn't touch a mouse nor a keyboard (didn't happen for a while :) [16:11] jam: how was yours ? [16:11] jelmer: I'm getting timeouts while trying to access a signature file on your dulwich branch [16:11] vila: pretty good [16:12] sat was a kids birthday party [16:12] so it was fun to see Kareem playing with the neighbor kids [16:12] I didn't get back to a comp until late Sunday night as well [16:12] jelmer: lp also seems to be having problems with that branch [16:12] jam: us2 seems to be under heavy load at the moment, one of the admins is looking into it [16:13] jelmer: k. It seems funny since everything works until the .six [16:16] vila: still looking into the mysql stuff? [16:16] jam: on my right screen, yes [16:17] jam: trying to find a better ftp test server on the left one [16:17] jam: I was so close :-/ And had problems *again* with FtpStatResult and the test suite being too demanding for ftp :-/ [16:17] Hmm, it took 438.396 seconds to pull one revision from subvertpy. bzr-svn is still going. Did my connection hiccup? [16:18] jam: the idea was to spend a couple of hours only on it and then address the fact that we can't test under 2.6 anymore until we avoid using medusa there which requires another one [16:19] jam: I hate ftp [16:19] that was rude, ftp doen't like me that much [16:21] Peng_, server is having issues [16:21] vila: Would it be easier to just get medusa working with 2.6 ? [16:21] Oh. Oh! I did read backlog, just not enough to understand it. Alright. :) [16:23] jam: no, far harder as the bug is deep down inside asyncore/asynchat or something (I suspect the problem is roughly that somewhere in an iterator str() objects are special-cased and unicode isn't, ugly business), so the idea was to just stop using medusa for >= 2.6 and find another ftp test server [16:24] jam: I found one that doesn't require running as root, only to find that it returns an empty list when asked to list a non-existing directory and has no problem giving the *size* of a directory [16:24] vila: why would we be using Unicode anywhere... [16:24] at the FTP level, everything should be 8-bit strings, (I think) [16:25] utf-8 (sorry) [16:25] vila: we wouldn't be special casing a utf-8 string [16:25] as it is just another 8-bit string [16:25] but at some point giving utf-8 means handling unicode (from memory) [16:25] jam: I don't remember exactly the problem, certainly it was on server side, and ... well there is little we can do there :( [16:26] vila: I don't know the failures, but everything *under* FTPTransport should be 8-bit strings. I suppose if we have a unicode file on disk there may be other issues [16:26] vila: does everything fail? [16:26] Or is it just a couple tests that we could say "py2.6 + unicode files + medusa is KnownFailure" [16:26] vila: I certainly would rather you spending your time on other things :) [16:27] under 2.6 with medusa, failures are about utf-8 encoded file names IIRC [16:27] vila: exactly, so wrap tests that do exactly that with KnownFailure and get on with your life :) [16:27] jam: I know :-/ I hoped to be done with it far quicker and then... well each bug looks like the solution was close :) [16:28] Can I expect the Mac OS X 10.5 Installer for bzr 1.12 soon? [16:28] hmm, not sure we can peak under the cover to realize we are under 2.6 + medusa + whatever, the idea was to shorcut at get_transport_permutations [16:29] get_test_permutations I meant [16:29] eferraiuolo: "tonight (European time)" according to a post on the mailing list :) [16:30] Peng_: thanks for the info :-) [16:30] jam: to be precise, I have a test server that pass the test suite, with a relaxing constraints patch against bzr.dev, so that's not *that* bad :-) [16:31] jelmer: I have a trivial patch to 'dulwich' which should help the performance of 'apply_delta'. Care to test it out? [16:31] http://paste.ubuntu.com/121930/ [16:32] jam: on the bbc front all the remaining failures will require a bit of work and I wanted to start with a bzr.dev merge which raise to many conflicts to be solved quickly, so *that* was pushed out of the parrallel activities which are now back to the two mentioned above :) [16:32] vila: sure. I'm also submitting the "trailing whitespace" fix [16:32] jam, Is that against the current trunk? James also submitted a performance improvement yesterday [16:32] which is going to be bad for merging into bbc [16:32] jelmer: versus 154 [16:32] it just changes to use a list and append [16:32] and then ''.join() at the end [16:33] jam, ah, cool [16:33] jam, can you "bzr send" or is it easier if I just pick up from pastebin? [16:33] that should avoid O(N^2) performance [16:33] I haven't committed, but I can and I'll send it [16:33] jam: by the way, if *you* are able to merge bzr.dev cleanly in bbc, just tell me :-) [16:33] nice [16:34] there are probably a couple more places in that file where that approach could be used [16:34] jam: Please do :-) [16:34] jam, vila: Also, if you can have a look at my InterBranch patch that would be really helpful for bzr-git.. [16:35] * jelmer is still wondering what the magic trick is to get a patch reviewed [16:35] jelmer: sent [16:35] jam: Thanks :-) [16:35] vila: I'm sure I'm not [16:35] I changed some code in fetch [16:36] which Robert changed with the network streaming stuff [16:36] hi channel [16:36] I added a progress bar [16:36] whats bazaar all about ?? [16:36] jam: yeah, so at least you know *one* side of the conflict as opposed to None :-) [16:37] jelmer: I think it is beer. And submitting patches that directly effect other people. The problem with InterBranch is that I haven't worked out whether the added complexity is worth the benefit for non-core code. I'll try to give it a look [16:37] arshad: freedom :) [16:38] ok . . . . . . but freedom for what ?? [16:38] jelmer: I assume that is the cause of "AttributeError: 'module' object has no attribute 'InterBranch'" with a newer bzr-git? [16:38] arshad: That was a trivial response. To give you a better one, how much do you *know* so I don't repeat myself [16:38] bzr is a version control system [16:39] do you need to know the diff from others [16:39] what a vcs *is*? [16:39] etc [16:40] hope u wont fire me for that [16:41] arshad: no firing, just trying to understand what info you really want [16:42] vila: yeah, robert refactored a lot of stuff that bbc had hacked to get working :( [16:42] So I need to actually understand the new stream code [16:42] before we can really adapt it [16:42] vila: like understanding how to get chk records as part of the new stream [16:42] jam: The good thing is that that should address at least *one* failure :-) [16:43] jam: ouch, not an easy bird then :-/ [16:43] * vila throws some stones at jam to help him :) [16:43] can VCS b made as a career [16:43] ?? [16:43] james_w: yep [16:43] . [16:43] Doesn't CPython optimize += on strings with only one reference, making it faster than ''.join()ing a list? [16:44] jelmer: am I better merging that, or going back in time a little bit on bzr-git? [16:44] arshad: vcs == version control system, try reading http://bazaar-vcs.org a bit [16:44] arshad: you should get many answers there [16:44] where?? [16:45] vila: so one problem I have with your "knownFailure" change to "autopack_rpc_is_used" test. Is that you don't test anything, in case we accidentaly *fix* that. [16:45] jam: It's the only way to make "bzr pull" work on remote git repositories since we can't walk the graph of remote git repositories to determine the revno [16:45] james_w, Better off merging InterBranch, without it you can't pull from remote git branches [16:46] jam: that was quite the only possible route at that time as we were testing that the hpss autopack call was in hpss_calls [16:47] jam: and that required some InterRepo accepting chks which wasn't the right way [16:47] vila: not really [16:47] you could have put the knownFailure later [16:47] around the actual "x in y" test [16:47] anyway, it is fine. [16:47] For right now, I'm pulling out your change, because it conflicts with the new streaming RPCs [16:47] and I don't know how BBC will interact with that yet [16:48] jelmer: will InterBranch allow things like "missing" against the remote branch? I guess they may have to become a fetch to local then missing operation? [16:49] jam: pull out ! pull out ! sounds perfectly right now that the test has changed [17:00] vila: dammit.... we have a serious problem [17:00] the generic fetching code doesn't really seem to support converting on-the fly [17:00] at least not easily [17:01] since it expects to be getting a stream [17:01] that it can just insert on the other side [17:01] and with chk [17:01] it has to convert the repos [17:01] well, convert the inventories. [17:01] james_w: It mainly allows for custom implementations of branch-to-branch operations, such as pull [17:01] james_w: right now bzr-git only overriden update_revisions(), which is the main worker used by pull [17:02] james_w, but in the end, we would indeed end up with a search_missing_revisions() and that would most likely indeed to a fetch :-/ [17:02] james_w, s/to a/do a/ [17:03] yeah, that's a pain [17:03] jam: ouch :-/ [17:03] jelmer: would it be at all possible to extend git's server to help with this? [17:04] not in terms of would they do it, but in terms of whether git could provide the sort of information that is needed? [17:04] james_w, it would, though I'm not sure how likely such changes would be accepted upstream [17:04] I guess having it speak the bzr remote protocol would work :-) [17:04] james_w: :-) [17:05] james_w, Actually, that has crossed my mind.. the git server John has been working on can provide a "bzr" capability, just like "bzr svn-serve" does [17:05] heh [17:05] jelmer: also, would "bzr format-git-patch" make sense? [17:06] and something to then pull then back in? maybe "bzr patch" is enough [17:06] james_w, you mean "bzr send --format=git" ? >-) Yes, I think so [17:06] yeah, that too [17:06] :-) [17:07] jelmer: i thought about custom extensions to git-serve too :) [17:07] making "format-patch" and alias of "send" might be a good idea [17:07] vila: so StreamSink *is* meant to handle some of that, but it only really is defined as handling the case where 1 fulltext == 1 inventory [17:07] which is no longer true for bbc [17:07] so I have to figure out how to work around it [17:08] the stream code seems to think that just "get_bytes_as('fulltext')" would be enough [17:08] and, of course, lifeless and spiv are both sleeping right now [17:08] I may have to punt... :( [17:09] until I can talk to them and see what they were thinking of as a solution for this [17:10] is bzr 1.12's "st" supposed to explode on me? [17:10] Ng, you may want to update your copy of bzr-loom [17:10] Ng: let me guess: it doesn't like 'verbose' and you have installed bzr-loom ? Grr, jelmer is too fast :) [17:11] correct :) [17:14] jam: punting sounds the most reasonable thing to do, is StreamSink._extract_and_insert_inventories where the problematic assumption is being made ? [17:15] vila: that an inventory is contained within a string [17:15] james_w, yeah, I agree [17:15] the idea is that you can do "read_inventory_from_string()" using the source serializer [17:15] the problem is that for chk repos [17:15] you can't just have the little bits you need [17:15] you have to have all the chk pages [17:16] for that inventory [17:16] in order to cast up to an Inventory object [17:16] that can then be serialized back down into whatever format the target requires [17:16] vila: does that make sense? [17:16] jam: can't we just transmit the delta and build from that ? Argh, yes, not all formats can build from a delta :-/ [17:17] vila: so the issue is that "CHKRepo.get_stream()" would want to at best just send the individual pages that changde [17:17] changed [17:17] but the target repo doesn't have *any* pages [17:17] so it wouldn't know how to interpret them [17:17] so CHKRepo.get_stream() sort of needs to do the conversion on *its* end [17:18] jam: so either the target is chk and we send changed pages (>-/) or better the delta or the target is not and... pfff [17:18] The BBC logic just did "for inv in self.source_repo.iter_inventories(revs)" [17:19] vila: right, so I can easily figure out what to do when source_format == target_format [17:19] However, when source_supports_chks != target.supports_chks [17:19] I'm a bit stymied [17:19] I can understand that :-/ [17:19] The code is written such that the *target* does the conversions [17:19] But a Pack-0.92 repository isn't going to know what to do with a chk page [17:20] So I think we need a get_stream() that can tell the target what format it needs [17:20] sorry, tell the *source* [17:20] hmm.. maybe I can cheat and try to do that [17:20] I just worry that I'm breaking the *spirit* of lifeless & spiv's work [17:21] jam: better talk with them then [17:21] though I know they wanted to make sure that the streaming code worked with bbc [17:21] as part of the "if it doesn't, then we haven't done the right thing" [17:21] there is already some XXX in remote.py [17:22] so may be we should just don't try to merge yet (and will regret it later :-) [17:22] vila: well, the other 3 conflicts were easy to resolve... :) [17:23] and if we don't merge revno 4032, when we merge 4033 we're going to have a real pain (it is the whitespace cleanup patch) [17:23] jam: hmmm, can you resolve the last one with a hammer maybe ? [17:23] it is easy enough to do if you merge just before [17:23] because then you know you can revert everything [17:23] and do your own whitespace cleanup [17:24] vila: yeah, I can always put big XXX and say that robert and andrew need to fix this [17:24] :) [17:24] jam: if we end up with test_autopack_or_streaming_rpc_is_used_when_using_hpss failing, well that will make 5 failures instead of 4, not a big deal [17:24] vila :) [17:24] jam: sounds like a good idea, at least they get an almost working bbc [17:25] jam: and they may already know that can't work right now [17:25] jam: and they may already know that bbc can't work right now [17:26] vila: I'll also note that "supports_chk" isn't a sufficient test. As if you have different CHK repos with a different paging structure (different hash, etc) [17:26] then the target repo still doesn't understand the source repository [17:26] pages [17:26] (layered on *top* of the pages is a possible delta format, which could differ but still be understood by both formats) [17:27] like gc+chk255 should be able to understand the wire-stream of a chk255 repo [17:27] but chk16 != chk255 [17:27] me == :( [17:27] jam: yup, and a delta format may be easier to serialize/deserialize for *future* formats too [17:28] vila: I think we are talking different things [17:28] logical inventory diff [17:28] versus byte-wise diff of chk page [17:29] I think defining a "get_stream()" that can return serialized bytes of a logical inventory diff may be the way to go [17:29] though possibly slower than a pure chk => chk could be [17:29] jam: sorry, I went too fast, but you were already saying the byte-wise diff of chk pages was having problems, so going with a logical inv dif... yes, what you said :) [17:30] jam: I'm not sure you can assume that two repos have the needed pages without checking first (and it's true today, I'll be more comfortable if that wasn't required) [17:30] jam: I'm not sure you can assume that two repos have the needed pages without checking first (and *if* it's true today, I'll be more comfortable if that wasn't required) [17:31] of course, I don't want to have to write a logical diff format + serialized form just to get bbc working [17:31] vila: chk pages are the same problem we have today, so I'm not specifically worried [17:31] we assume that if you have the inventory at revision X, then you don't need to transmit it [17:31] in the same way, you don't need to transmit the chk pages for X [17:32] the issue is that if both sides have different chk serializers [17:32] the pages for Y don't mean anything to the other side [17:32] I think that defining new serialized formats shouldn't be harder than defining a % format [17:32] It *is* today but it shouldn't [17:33] vila: anytime you work on bytes-on-the-wire or bytes-on-disk it is hard [17:33] because you need a whole lot of direct tests [17:33] jam: exactly [17:33] to make sure that bzr-X.Y can talk to bzr-X.Z [17:33] I don't see that going away [17:34] I worked in that direction in the transportstats plugin (not the interoperability part, but the easy defined formats) [17:37] argh.... [17:37] jam: but the gap to go there is still rather large :-/ [17:37] I thought I had a decent workaround [17:37] by changing the source to do the reserialization and hand it back [17:38] but it seems the Sink always uses the source-serializer to decode the fulltext bytes [17:38] (which sort of makes sense, but means I don't have a way to do this yet) [17:40] * vila dinner [17:41] hmm... it seems that chk_serializer's inherit from the XML serializers (perhaps wrongly, but they do) [17:41] which means an ugly-ugly hack is to actually write out a xml5 format inventory to the record stream [17:45] jam: will that make it less ugly to use composition instead of inheritance (ignoring it doing that) so that you can provide the right xml serializer ? [17:46] vila: so the ugly hack is that we will read in the CHK inventory, bring it up all the way to a full Inventory object in memory [17:46] and then convert it to some-semi-random not really used XML representation [17:46] and write those bytes out on the wire [17:47] I think I can make it work [17:47] but it certainly isn't what I would consider the "correct" method [17:47] One other possibility, is that get_stream() from a CHK repo [17:47] jam: At least you'll have some meat to talk with lifeless and spiv :) [17:47] could return *all* the pages [17:48] which would mean the target would have enough to work with to do it on its end [17:48] but it would have to transmit and buffer all of those pages somewhere [17:48] until it got to the inv records [17:48] and new what to do with them. [17:48] jam: returning all the pages sounds wrong especially considering that you'll certainly ends up transmitting some of them multiple times [17:48] vila: you can just transmit the set() [17:49] for revisions 10..30 send all referenced chk pages [17:49] hmm [17:49] jam: sounds better [17:49] jelmer: ? [17:49] jam, Trying to figure out what I'm doing wrong benchmarking OOo with bbc and bzr.dev [17:49] vila: yeah, I'm still going to let robert and andrew worry about how to semi-optimize transfering between formats [17:50] I'm seeing a 28x performance gain with bbc [17:50] which seems a little excessive ;-) [17:51] jelmer: not really [17:51] it is an O(N^2) versus O(N) sort of change [17:51] I don't know *what* you are benchmarking [17:51] jam, import from OOo-svn into bzr [17:51] jelmer: oh sorry, it is a log(N) versus N change [17:51] jelmer: so still possible [17:52] with bzr.dev [17:52] bzr.dev gives one revision every 6 or 7 seconds, bbc close to 4 per second [17:52] we have to generate an inventory which contains a line for *every* file [17:52] with bbc [17:52] we only have to update the few pages based on the actual change [17:52] O(tree) operation versus O(changes) [17:53] jelmer: also the bbc pages aren't delta compressed (yet), which means they are also faster to read, etc. [17:53] so *faster* but not *smaller* to store [17:53] jelmer: so my initial feeling is that you aren't doing anything wrong [17:53] as that is really what bbc is *about(* [17:54] I would actually expect the difference to get *more* pronounced as time goes on [17:54] as the larger OOo gets, the size(changes) stays relatively constant for most projects [17:55] right [17:55] jelmer: well, at least if you have an optimized Inventory._make_delta() for bzr-svn [17:55] jam, I do [17:55] jam: Still surprised it matters this much though [17:55] jam, Anyhow, I'm not complaining ;-) [17:56] jam, bbc rocks \o/ === phinze_ is now known as phinze [17:57] jelmer: if you see igc around, I'm sure he'd be interested to hear it [17:57] It seems that OOo is looking again at possibly switching to a DVCS [17:57] jam, This was actually why I was looking into it [17:57] jam: They were seeing slow imports with bzr.dev [17:58] jelmer: yeah, when we imported in the past, it was... 2 weeks to a month? something like that [17:58] Also, I wouldn't be surprised if the generic converter is actually *slower* in bzr.dev than it used to be [17:58] as it was updated in preparation for bb [17:58] bbc [17:59] And I believe Ian was looking at fast-import code in the last couple of days [17:59] as it had also done some internal hackery for performance that may not be as relevant now [18:00] based on what I'm seeing now with bzr-svn (4 revs / second) I should be able to import OOo in 20 hours [18:01] jelmer: that's a good number [18:01] jelmer: is that streaming from their repository? Or do you have an svn dump locally? [18:01] I have a local dump [18:02] That's mainly to avoid hammering their server too much though [18:02] as the process is mainly CPU-bound [18:06] jam: bzr-svn basically does one "svn log -v" followed by "svn up -rX-1:X" for each revision [18:07] so while the extra roundtrip-time should have some impact, it shouldn't be significant [18:07] jelmer: well, bandwidth and latency [18:07] and as you say [18:07] if you try multiple times [18:08] you only have to have dumped once :) [18:08] at 4 revs/sec [18:08] latency becomes an issue [18:08] 250ms [18:08] a latency of 100ms and 2 round trips would cut your conversion speed in half [18:08] certainly didn't matter for bzr.dev at 7s/rev [18:08] jam: Latency to launchpad is 10ms here, not sure how much it is to the ooo svn server [18:37] vila: ok, I've hacked it enough that it seems to be working [18:38] the RPC tests still fail [18:38] though I think we could fix that by allowing chks to be included in the InterPackToRemotePack tests [18:47] * ToyKeeper finds it a little odd that two opposite-sounding options are required to produce the desired output... bzr log --verbose --short [19:08] anyone knows when lifeless will appear? [19:12] anyone know what's up with http://benchmark.bazaar-vcs.org/usertest/usertest.log ? [19:12] ronny: he usually shows up in about 2 hours [19:12] though he almost never actually completely logs off irc [19:12] so I'm not sure if something changed [19:13] mthaddon: what is the problem? [19:13] jam: it seems to be debug output rather than the expected output [19:13] i hacked up a inital nosetest plugin for subunit [19:14] jam: either that or the format has changed radically (it's used to generating performance graphs, so I'd need to alter my parser if the format's changed) [19:18] mthaddon: poolie or igc would be the ones to ask [19:18] If you want, I'll try to mention it when I talk to them later [19:18] mthaddon: best guess is that someone forgot a flag when the updated the benchmarks being run [19:18] cool, thx - I'll send them an email as well [19:45] * jelmer makes a note to buy John beer at the next bzr sprint [19:46] jam: Thanks for the reviews, it's much appreciated! [20:37] hello. is there a way to make bzr use a network proxy setting? [20:57] does nicolas allen irc? [20:57] nicholas, sorry [21:08] mwhudson: I think so === lifeless_ is now known as lifeless [21:22] jam, hi [21:22] Do http connections time out...ever? [21:22] morning poolie [21:23] Peng_: probably, but i would say not as fast as they should [21:23] they should timeout and retry [21:23] since they're all idempotent [21:24] jam, want to talk? [21:25] hey poolie, chatting with robert right now [21:25] ok [21:26] i'll be here all day, ping me if you want a 1:1 [21:27] hi poolie [21:27] hello [21:28] Peng_: we occasionally had http connections hang for ridiculous amounts of time in the branch puller [21:29] jam, btw not sure if you saw but it looks like rockstar will go to the mysql conf [21:30] mwhudson: How long? [21:30] Peng_: days, iirc [21:30] Eh. I'm at 12.5 hours so far. That sounds bad. [21:31] Oh well, I have nothing better to do. :) [21:32] well [21:33] killing and starting again will probably work... [21:34] Sure, but how would that be fun? === thumper_laptop is now known as thumper [21:35] i'm not going to try to guess what you find fun :) [21:38] * mwhudson watches pydoctor trunk process bzrlib [21:39] gawd docutils is SO SLOW === montywi|meeting is now known as monty [22:23] jam: let me know if you want to continue the compression chat [just skype me] [22:24] lifeless: got an initial nose plugin for subunit wired up, currently it imports the reporter [22:25] lifeless: how about a convention for giving stream output/log lines after the trace? [22:26] beuno: hi [22:26] ronny: How would that map into pyunit [let alone junit etc] [22:26] mwhudson, hiya [22:27] beuno: so the masses are complaining about https://bugs.edge.launchpad.net/loggerhead/+bug/253950 [22:27] Ubuntu bug 253950 in loggerhead "Changes view slow to render with large initial import" [High,Triaged] [22:27] lifeless: ignore unless known otherwhise [22:27] ronny: no, I mean *where* would that be exposed [22:28] nosetest tracks things like stdin/stderr (and its nice for additional output) [22:28] ronny: you have a RemoteTestCase which calls result.addFailure(self, RemoteError(self.blah)) [22:28] beuno: a simple fix is to truncate the list of added/modified/... files when the list gets more than a certain length [22:28] ronny: ah [22:28] so; divide and conquer [22:28] mwhudson, sure. Although, to be fair, doing it with ajax should be pretty quick as well. [22:28] beuno: yeah, i guess [22:28] currently my initial wire up doesnt handle that tho [22:29] maybe i should just try that quickly [22:29] there are two use cases here - there is 'get nose discovered tests run with output to subunit', and there is 'accept a subunit stream for display with the nose infrastructure' [22:29] mwhudson, and we could potentially avoid getting that information at all unless needed, gaining some performance? [22:30] doing it nicely will require some infrastructure hacking i guess [22:30] lifeless: i see 2 possible solutions - mangle it into the trace output, or have a new event [22:30] In terms of the former, just putting it in the comment area in the result is fine [though if people should be seeing it real time (e.g. a debugger prompt), put it in the stream in realtime. [22:30] and some nice gifs [22:30] mwhudson, we have most of that in lazr-js ;) [22:30] i should look at lazr-js i guess! [22:31] for the latter, I don't think you need to accept any random stream into nose so much as nose generated ones, so as long as you use the same approach for in as for out it should work nicely [22:31] ronny: yes, I see the same two things; I'm cautious about new events though [22:31] argh [22:31] when will bzr info lp:lazr-js not say "format: unnamed" ? [22:32] ronny: particularly things that are tricky to map into python space - like jml's testtools, I want to see what multiple projects are doing before committing to a design [22:33] mwhudson: [22:33] Repository branch (format: pack-0.92) [22:33] I bet its info not doing the right thing with hpss branches; perhaps filing a bug would help. [22:34] lifeless: got a link to jml's testtools? [22:35] https://edge.launchpad.net/testtools [22:36] lifeless: would seem to be https://bugs.edge.launchpad.net/bzr/+bug/196080 [22:36] Ubuntu bug 196080 in bzr "`bzr info -v bzr://host/branch` hides actual branch/repo format" [Medium,Confirmed] [22:37] mwhudson: k [22:37] thanks [22:37] * beuno -> home [22:37] bbiab [22:37] beuno: so how do i actually use lazr-js, is there documentation or a quickstart somewhere? [22:45] is there any way to get bzr log to show me only the commits made on this task branch? [22:47] afair bzr log -rsubmit:.. [22:47] see bzr help revisionspec [22:55] jam: I've been struggling getting a good conversion of mysql and emacs to --development5 [22:55] emacs fell out with a serializer bug after 10 hours [22:55] s/out/over/ [22:56] mysql completed but then didn't work ... [22:56] and I'm yet to track down why cos ... [22:56] bzr check isn't fast [22:57] jam: I'm wondering if you've had more luck converting these [22:57] igc: I have, but I haven't done a full conversion in a while [22:58] if so, could I ask for you to tar.gz the converted branches and put them on orcadas? [22:58] that's the missing piece for getting meaningful benchmark results [23:00] abentley: thanks for the review [23:00] jml: np [23:09] hi - numpy/scipy developers are discussing dvcs ... I pointed out bzr-svn (they were talking about the shortcomings of git-svn) [23:09] http://projects.scipy.org/pipermail/scipy-dev/2009-February/011205.html [23:10] someone asked "can you point us to a public svn repo where non-trivial branching is happening with bzr?" was wondering if anyone (perhaps jelmer ) could point me to this [23:11] thrope, hi [23:11] thrope, non-trivial branching? [23:11] I thought the point of a DVCS was that ALL branching is trivial ;> [23:12] not sure - thats what the person asked... [23:12] I dont have a lot of experience - but bzrsvn worked well for me and they were complaining about git-svn [23:13] I guess can anyone point to an open source project using bzr-svn 'officially' with a couple of different bzr branches in svn [23:14] thrope, well, there are some projects that contain roundtripped revisions using bzr-svn [23:15] thrope: it seems kindof pointless to have official bzr-svn branches though, since the official repo would always be svn, even if some developers are accessing svn using bzr [23:16] right [23:16] just not sure how to respond to the guys question then [23:17] thrope, if you're looking for example branches in bzr created from svn branches using bzr-svn, see http://bzr-mirror.gnome.org/ [23:19] ok thanks [23:19] thrope: alternatively, I'm interested to hear about svn repositories that don't work in bzr-svn [23:25] * igc out for several hours [23:25] back later hopefully [23:30] spiv: so, what shall we do today? [23:31] The same thing we do every night; try and take over the world. [23:31] lifeless: so I'm currently wikifying my measurements from yesterday [23:31] lifeless: after that I'm happy to head over to Epping [23:31] lifeless: if that suits you? [23:33] hornsby would be better for us today; if epping is better for you though, that is fine too [23:33] lifeless: that's ok [23:34] ok, well I'll pick up food and grab a train [23:37] late morning is ok for you ? [23:40] Sure.