[00:04] lifeless, what has regressed [00:04] jam-laptop, hi [00:04] hi [00:04] poolie: 'bzr update' in a checkout [00:04] the error for trying to commit to a readonly branch? [00:04] poolie: no [00:04] what jam said [00:05] 'bzr checkout http://people.ubuntu.com/~robertc/pack-repository.knits', wait while I push a new rev, 'bzr update' && boom [00:05] I know you looked at this [00:05] iirc there is still a bug open on it [00:05] i haven't fixed it [00:05] oh, ok. [00:05] ipso facto I was confusedo [00:05] it may have been working at some time in the past [00:05] but, it wasn't tested :) [00:06] i have written some format docs on packs [00:06] will send them shortly [00:06] and have done the non-structural review cleanups [00:06] i am going to see whether the structural cleanups i wanted can be done easily or not [00:07] if not easy, they can just be turned into todos or bugs or forgotten maybe [00:07] jam-laptop, ll may have already suggested it, but [00:07] oh. is --forget-merges not in 0.91 either? :( [00:07] maybe you would look at making interpack fetch make sure that the text delta basis is in the destination [00:08] ...odd, initial incremental commit is acting like it isn't using the C extension -- massive amounts of CPU time used while collecting changes. [00:08] dash, it'll be in 0.92 [00:09] i guess I can just write my own script for now [00:09] revno 2000 merged support for --lightweight with http checkouts [00:09] =/ [00:09] dash: heh, sorry :) [00:10] and 2019 claimed to fix it for heavyweight checkouts :( [00:10] ok no its right [00:10] there have been no commits that claim to make heavyweight checkout updates work over http [00:11] 2088 did that for lightweight, so I'd say it was a 'just worked' thing initially [00:12] huh. after "bzr revert" i'm not seeing a pending merge. i guess that's sufficient. [00:12] jam-laptop: what do you think of a dirstate iterator which _sha_from_stat advances, in commit ? [00:12] dash: you'll also have lost your changes ... [00:13] lifeless: which is OK [00:13] lifeless: 15m52s wall-clock for first incremental, 15m27s of that being user time. [00:13] nDuff: that really is suspect isn't it. [00:13] do we list the C extensions anywhere yet ? === mw is now known as mw|out [00:14] nDuff: when it went faster before, how fast was it ? [00:15] poolie: call ? [00:15] in 5m? [00:15] sure [00:18] lifeless: out of my scrollback buffer right now, sadly, and I didn't record it. that said, recommit (of the same thing that just took ~16min) is completing in 1m52s. [00:18] grah [00:18] the only thing that makes sense to me is the C extensions not being picked up [00:19] perhaps we should make that a nag [00:20] ...hmm. seems odd that it would happen only on initial incremental commit and not on recommit. Is there significant work done (prone to throwing an exception based on tree state) while the module is first being loaded? [00:24] * nDuff looks at the code, and yeah -- looks like only an ImportError should result in fallback. [00:25] would trace.mutter() append to .bzr.log under normal conditions, or should I be doing something else to instrument? [00:26] * nDuff looks in trace.py, and decides that warning() is more appropriate to the case anyhow. [00:36] 1m11s initial commit; 12.2s status w/ bad cache (and new files unknown), 7.6s status with good cache (and new files unknown), 25.4s to add unknowns, 12.1s/9.9s initial/subsequent status after adding unknowns... and now the 15min incremental commit isn't reproducing itself. [00:37] morning all [00:40] lifeless: would you use a full iterator, or just another "this was the last cached lookup" like you just added [00:41] nDuff: trace.mutter does to th elog [00:42] jam-laptop: I'm thinking a full generator. Basically lsprof is claiming 10% in _get_entry [00:42] lifeless: I would wonder if it could actually keep exactly in sync with commit [00:42] (partial commits, etc have an effect here) [00:42] jam-laptop: an alternative would be an equivalent of inventory.iter_entries_by_dir(specific_file_ids=XXX) [00:43] with yield_parents=True [00:47] iter_entries_by_dir() seems like a good route to me [00:48] ...hrm, cancel that; still a 15 minute commit, and none of the logging directives I put in in case of an ImportError on the C modules triggered. Time for a callgrind run, I suppose. [00:58] appears to get slower near the end of the change-collection process, incidentally. [01:00] jam-laptop: the iter_entries_by_dir will need to return the disk data; thats really the disconnect we're dealing with. [01:00] jam-laptop: I don't think this is a 2-day piece of code; particularly with win32 sadness. [01:00] jam-laptop: so I'll send a mail right now [01:00] well, the _sha_sum... do you honestly think that would be reasonable? [01:01] It just seems like it is iterating at a very different layer [01:01] and you are expecting it to stay in sync [01:01] I suppose if you pass in the path [01:01] and you know you are always "incrementing" in path [01:01] then it can fast forward until it finds what you need [01:01] well [01:01] But we don't have plain python generators that take more data [01:02] we don't need one [01:02] (until 2.5 or later) [01:02] I guess _sha_... could do the fast forwarding [01:02] iter_entries_by_dir on inventory returns inventory objects [01:02] we need one that returns the key fields and the path_content_summary details [01:03] hmm [01:03] I'll test doing a cache lookup; one second [01:08] jam-laptop: hmm, do I need to update the _id_index in my update_basis patch ? [01:14] jam-laptop: I'm just building a dict of all the stats [01:14] jam-laptop: won't be so hot for partial commits, but ... [01:15] possibly a threshold of lookups to trigger changing form [01:16] nDuff: commit feedback is per-dir, if you have larger dirs at the end that would explain the apparent slowdown, or it may be list accumulation overhead [01:17] still, its two seconds faster on incremental commits [01:21] initial commit down to 1m29 [01:21] wall clock :0 [01:30] lifeless: I've got a .callgrind file; anywhere particular you'd like it uploaded? [01:30] mail me or give me a url [01:30] they gz well [01:34] lifeless: in the mail. [01:34] thanks! [01:39] lifeless: I don't think you want to try to update _id_index, since there is too much stuff going on (some entries will be in the removed parents, etc) [01:39] lifeless: But you probably should reset it [01:39] I think self._id_index = None is fine [01:47] nDuff: holy fuck. is_inside_any is taking 90% time [01:47] nDuff: what parameters did you give commit ? [01:48] lifeless: bzr arguments: [u'--lsprof-file', u'../upstream-portage-tree.current.first_incremental_commit.callgrind', u'commit', u'-m', u'first incremental commit', u'--quiet'] [01:48] nDuff: do you have many deleted paths ? [01:48] probably; let me check. [01:49] thats likely it [01:49] is_inside_any is not that cheap; my strategy to date has been to avoid calling it as much as possible. [01:49] looks like we need a smarter structure than linear searching. e.g. bithection [01:49] brb, food time [01:50] jam-laptop: another dirstate patch for your consideration :) [01:52] lifeless: 8045 removes, 35597 adds, 24211 modifications. [01:52] nDuff: that will be it [01:52] we could make this a list [01:53] deletes[bisect_left(deletes, path)].starts_with(path) [01:53] or a dict matching the tree structure [01:53] jam-laptop: ^ I think bisect is probably good enough, or possibly even best. [01:53] we're going to have to do 100K lookups into this [01:54] lifeless, pack doc sent [01:54] another thing is we know when we're going to have to lookup again - when we add a delete, or when we change directory [01:54] nDuff: Anyhow, I a) know where the problem lies, b) can definately fix it. [01:54] nifty! [01:54] nDuff: its the number of deletes that you have in your tree during the commit. [01:54] thank you for the callgrind [01:55] np. thank *you* for putting up with all my fumbling around. [01:57] know when you might have a chance to address this one? if it'll be a while, I'd like to open a ticket so I can get notice after it's happened. [01:57] please open a ticket [01:58] it may not be a while [01:58] but its worth noting [01:58] I've put the stat cache improvements into pack-repository.knits too, revno.... [01:59] 2853 [01:59] In my Darcs days, this sort of thing would trigger the Red Giant bug, but since it was exponential you could usually get around it be making smaller commits with less files/deletes/whatever. Would that be the case here? [01:59] [Not that you really want to bandy that about] [01:59] AfC: nah, I'll just fix it [02:00] :) [02:05] lifeless: https://bugs.launchpad.net/bzr/+bug/156491 [02:05] Launchpad bug 156491 in bzr "bzr-packs: commit with many deletes spends much time in is_inside_any" [Undecided,New] [02:05] That's no fun. It's hard to feel like your software is working hard, if you don't have to occasionally pander to it. [02:06] fullermd: ah, so you recommend that our tool should be so bad it causes stockholm syndrome? :) [02:07] Well, there are people still using arch... _you_ connect the dots :p [02:07] lifeless: btw, I'm curious, if the explanation is accessible to folks who don't know the codebase -- why is this case hit only on an initial commit, but not on a recommit or other operations which need to determine what is scheduled to be committed/deleted/etc (ie. status)? [02:10] New bug: #156491 in bzr "bzr-packs: commit with many deletes spends much time in is_inside_any" [Undecided,New] https://launchpad.net/bugs/156491 [02:11] spiv, hi [02:13] poolie: good morning [02:13] nDuff: well, its hit on the first incremental case [02:14] nDuff: when both FOO and all the children of FOO are deleted, we like to report only 'FOO deleted' [02:14] thats a significant factor in this [02:16] nDuff: I'm going to review a patch then do the changes review asked for in one of my patches [02:16] then I'll see what I can do about this [02:17] graci [02:17] * nDuff wanders off to do some clustering work. [02:29] spiv, wow, you're right [02:29] i thought i'd checked that this revision was referenced by its own inventory but in fact it is not [02:29] * lifeless grins [02:30] which is why it's good to check [02:30] poolie: yeah. That had me confused for while :) [02:31] I'd assumed that if its own inventory didn't refer to it, then no others could. [02:31] spiv: in theory that is right. [02:32] spiv: I suggest making 'bad' into a fulltext for now, [02:32] with inventory reassembly in the future. [02:32] lifeless: right. But reconcile of course is dealing with broken files... [02:32] this is oooold data [02:32] lifeless: ah, that's a clever idea. [02:32] inventory reassembly is going to be very slow without a fair bit of effort I fear :/ [02:32] right [02:33] not to mention hard [02:33] Hmm, I think I can see how to do it reasonably efficiently. [02:34] so just making it a fulltext is not, by itself, going to totally exclude this problem [02:34] as we talked about on the phone [02:34] But it's not a trivial amount of work... [02:35] poolie: why's that? [02:36] call one of the revisions that references it "bad" [02:37] um, "dependant" (typo) [02:37] suppose the destination already has the revision "bad" [02:37] it probably won't have known to pull that text version [02:37] then, when it comes to pull "dependent" it will think it doesn't need to pull the text version [02:38] I see. [02:39] poolie: but ifi t has bad already, then it has the text already. [02:39] You're now doing checks when pulling to make sure the parent is available, though? [02:39] poolie: *if it has bad already*. [02:39] So this would only happen if the repo was constructed before you added those checks? [02:39] IMO yes [02:41] lifeless, are you sure it will have detected to copy this text when pulling 'bad', even though it's not apparently modified? [02:42] i guess this is easy to test [02:42] I guess we're talking slightly different corner cases. [02:42] What I'm trying to say is 'if andrew makes the text a full text we will not encounter headaches with bzr.dev' [02:44] well, that would be good for now [02:44] even if it may not cover every case [02:45] garh bloody lawn mowers [02:47] jam-laptop: good call on the parent swap test; found a bug [02:48] jam-laptop: ping though [02:51] poolie: doco reviewed [02:53] thanks! [02:53] re your last point about the pack-names file [02:54] igc: can yu review 'use a dict to access stat cache' ? [02:54] the requirement is that, every pack listed there must exist [02:54] igc: also, what part of commit are you hacking on at the moment, so we don't collide. [02:54] poolie: yes, every pack in that file must exist. Every pack that exists *may* be in that file. [02:54] but it is possible to have a window where about-to-be-removed packs are present but not listed [02:54] lifeless: I'll review that soon [02:54] poolie: also during insert, we put the pack before we add to the names list. [02:54] so, you could recreate it, but this might be reintroducing redundant copies of the file [02:55] oh i was going to mention the file sizes too [02:55] poolie: but in that case its arguable that we should add them if we were to notice. [02:55] I'm currently working on bug 115601 [02:55] Launchpad bug 115601 in bzr "no clean abort after bzr+ssh:// connection timeout. ERROR: exceptions.AssertionError: end of file reading from server." [High,Confirmed] https://launchpad.net/bugs/115601 [02:55] as requested by poolie [02:55] igc: coolio :) [02:55] code done - just the tests to do now [02:55] poolie: You might also be introducing inconsistent data in a post-reconcile scenario [02:55] poolie: or data that is being deliberately removed. [02:56] poolie: so I'd really encourage the concept that 'the pack-names file is authoritative, precious, and not reconstructable' [02:57] sure [02:57] that's just the kind of thing i want to capture [02:57] i guess for the iix it is graph parents first too, then compression base [02:58] like for tix? [02:58] yes [02:58] I should have been more explicit [02:58] np [02:58] just brain slippage, i had seen them in the other order === kiko is now known as kiko-zzz [03:30] * igc food [03:46] lunch for me too. [03:50] Good idea. [03:50] * spiv -> food === mneptok is now known as mnepette === mnepette is now known as mneptok [05:41] hi all [05:41] hi [05:41] i have configured the loggerhead as per my bzr branch [05:41] but now i am not knowing how i can view it through the browser [05:42] how to configure it in apache web server [05:45] http://dharmatech.onigirihouse.com/atari-forth.jpg [05:46] lifeless, i'm not so sure that having unlock raise an error if a write group is underway is the best thing, as i said in my review [05:46] i have another reason which is that it tends to hide errors [05:47] to raise exceptions from things expected to be called from finally blocks [05:48] poolie: fair enough, I can buy that, though its good from a correctness point of view [05:48] perhaps a debug option or smething [05:49] or a warning, which can be caught by -Werror [05:52] lifeless: I'll review the 'dict to access stat cache' patch now [05:57] i need help in setting up loggerhead [05:57] indraveni, is there really no documentation? [05:57] is there documentation is loggerhead itself? [05:58] poolie, i couldn't find any good document [05:58] indraveni, mwh is the new maintainer, he should be online in a bit [05:58] except the commented content in the conf file [05:58] ok [05:58] theres no README? or INSTALL? [05:59] lifeless, there is a README which says [05:59] that everything is similar to Tubergears [05:59] and i dont have experience with tubergears [05:59] and for apache conf some steps are given, but i am not able to udnerstnad how it works [06:02] well [06:02] AIUI turbogears can run as its own webserver or as a fastcgi script [06:02] you can probably find out more via google [06:03] * nDuff is generally familiar with turbogears. [06:04] typically, you've got a file start-.py, and a config file with all your installation-specific settings [06:04] ...so starting the server is along the lines of running "start-.py foo.cfg" [06:04] can't say anything about loggerhead specifically, though. [06:05] if you're running through apache, of course, it gets more complicated (unless you're doing it the easy way and using mod_proxy). [06:08] lifeless: hmm, just tried pushing from my local (packs-based) tree to a "bzr serve" instance on the LAN (working against a packs-based shared repo). bzr: ERROR: Could not understand response from smart server: ('error', "") [06:09] nDuff: oh, looks like bad marshalling. [06:09] full stack trace from the server at http://pastebin.com/d26bd26fa [06:09] please file a bug [06:09] it won't be hard to fix, the problem is that this is an optional method, and one that packs don't support [06:10] because they don't use the previous locking model. [06:14] lifeless: https://bugs.launchpad.net/bzr/+bug/156546 [06:14] Launchpad bug 156546 in bzr "unable to push to packs-based smart server" [Undecided,New] [06:15] thanks! [06:21] New bug: #156546 in bzr "unable to push to packs-based smart server" [Undecided,New] https://launchpad.net/bugs/156546 [06:25] spiv: ^ [06:27] ok, think I have the pieces put back together again. [06:27] one more run then onto nDuff's performance issue [06:28] poolie: spiv: I think it would be nice to have some way to be sure the remote server handles optional protocols (like leave_lock_in_place) correctly [06:38] do you mean a way to test it? === jkakar_ is now known as jkakar [06:48] morning all [06:49] hi sab [06:50] hi sabdfl [06:51] hi sabdfl [06:53] lifeless: review sent to the list now [06:56] poolie: is it ok to merge the stat lookup via dictionary patch ? [06:57] poolie: thats the 5 second win on incremental commit [06:57] * poolie looks [06:58] i'd like to merge the pack branch with the small changes, then put the larger ones up for separate review [06:58] ok? [06:58] well this is a trivial change [06:58] not packs per se [06:58] and I've reviewed it poolie [06:58] ''trivial'' conceptually, code wise its something like 20 lines [06:59] right, I'm asking you with your RM hat. [06:59] it looks ok to me [06:59] sure [06:59] poolie.hats().switch("RM") [06:59] :) [07:04] bbiab [07:06] packs branch sent up [07:08] to pqm ? [07:09] yes [07:09] awesome [07:09] tada ! [07:09] indeed [07:09] sabdfl: ^^^ :) [07:09] Woo! [07:10] * vila_ send some champagne [07:11] poolie: does it have the same disk format string ? === vila_ is now known as vila [07:11] yes, still the same [07:11] and still just called 'experimental' [07:12] ok [07:12] is there any test framework support for checking that an assertion other than a deprecationwarning is raised? [07:12] (obviously the format and name needs to be changed) [07:14] *pop* goes the champagne :-) great work lifeless et al [07:15] hm that kind of sucks, the python warning module does not make it very easy to catch them [07:15] I'll wait till friday for the champagne :) [07:15] qantas champagne? [07:15] no, but we're all together friday night [07:15] FSVO friday :) [07:16] also I don't think we get champagne in economny class :( [07:24] hi hi hooray lifeless - well done [07:25] New bug: #156557 in bzr "Bazaar does not obey setgid on Linux" [Undecided,New] https://launchpad.net/bugs/156557 [07:29] current figures are stable - 1m31 wall initial commit, 0m15 incremental, 0m10 partial [07:31] hello [07:33] hello === jamesh_ is now known as jamesh [07:48] poolie: you forgot kthxbye on your commit message :) [07:48] heh [07:48] i should have [07:48] looks like it's copacetic though, deep in the ascii tests [07:49] huh? [07:49] it looks like it will land and not bounce [07:50] http://en.wikipedia.org/wiki/Copasetic [07:53] yes, thats the definition [07:53] righto, I forgot I was going out tonnight [07:53] ciao [07:54] bye [08:01] actually cancelled [08:01] too tired [08:03] reviewing the lp-login patch currently [08:04] woot?! has it landed?! [08:09] mwh, hi, are u tehre [08:10] you want mwhudson [08:11] yes lifeless [08:11] i need help in loggehead, yesterday he was helping me [08:13] morning / evening lifeless [08:13] svn-import finished successfully yesterday :) [08:14] mdke: excellent [08:14] but now I just have .bzr directories, I can't see the files anywhere... [08:14] is there another step I need to do? [08:14] mdke: you can get a working tree in any branch by doing 'bzr checkout .' [08:14] or you can make new branches from these and treat the current directly like a cvs or svn repository [08:14] righty ho [08:15] so bzr branch /local/url new/branch? [08:17] mdke: yup that will work [08:18] mdke: then bzr push /local/url to push back to /local/url [08:18] mdke: or you can do 'bzr checkout /local/url new/tree/' which will make it work like svn - when you commit /local/url gets the changes [08:20] aha, clever [08:20] lifeless: thanks. I will branch then push the branches to LP, once I'm done with that I'll abandon /local/url [08:28] wikipedia says: 'Copacetic may be a descendant of the Hebrew phrase "hakol beseder"', which I read 'How cool bzr' is :) [08:29] well, you have to read bzr with a french accent though, but wikipedia also says that one possible origin is creole, so I think I'm ok :) === meuh_ is now known as meuh === lifeless changed the topic of #bzr to: The Bazaar Version Control System | http://bazaar-vcs.org/ | The packs have landed | Bazaar 0.91 is out - http://bazaar-vcs.org/Download | Please complete the Bazaar User Survey - http://www.surveymonkey.com/s.aspx?sm=L94RvLswhKdktrxiHWiX3g_3d_3d [08:32] Cool! [08:33] a tour de force, lifeless. very well done indeed. [08:34] indraveni, mwh should be awake soon [08:34] poolie, thankyou [08:34] poolie, will wait for him [08:35] sabdfl: thanks!. And boy do we have ideas to improve on this improvement. [08:36] sabdfl: though as I said a while pack, the very core disk layout is now solid IMO (the pack disk format itself). [08:37] pack on the brain eh lifeless - s/pack/back/ :-) [08:37] indeed [08:39] indraveni: hello [08:40] mwhudson, hi [08:40] mwhudson, good morning [08:41] mwhudson, i have edited the loggerhead conf file accordingly but now i am struck with how to make it acess with apache [08:41] ? [08:41] i have apache installled and i have a vhost entry for my bzr repo [08:43] http://bazaar-vcs.org/BazaarFormats doesn't mention packs... [08:43] Lo-lan-do: so add it [08:43] :) [08:43] I'd gladly paste from the output of bzr help formats, but even that doesn't mention them. [08:44] they are still considered experimental [08:44] Lo-lan-do: so write a patch adding documentation for it. [08:44] we have a small amount of work to do to make them be exposed [08:44] Lo-lan-do: in other words, chill, dude. [08:44] indraveni: you need to use ProxyPass or similar [08:44] mwhudson, yes, i saw about it in my README file [08:45] well, I like the enthusiasm [08:45] AfC: I'm totally chilled (especially now jelmer helped me fix my bzr-svn problem), I'm just looking for info on this new shiny format that everyone seems so excited about :-) [08:45] but how do i need to access it through browser ? [08:45] Lo-lan-do: if you want to play, or add docs to the wiki, if you look at my mails to the list about dogfooding they are still valid [08:45] indraveni: here's what we (will) use for testing: http://rafb.net/p/1PMHGs81.html [08:45] Lo-lan-do: its just that you don't need anything other than bzr.dev to experiment [08:46] you need to have "127.0.0.88 codebrowse.launchpad.dev" in your /etc/hosts file [08:47] mwhudson, but in this where is the info about the bazaar repo ? [08:47] indraveni: there isn't [08:48] mwhudson, then how does the apache know where os bazaar [08:48] mwhudson, even there is ocnf related to loggerhead location [08:48] indraveni: um [08:48] indraveni: loggerhead runs an http server, listening on some port or other, 8080 by default [08:49] the conf file i pasted above tells apache to forward requests for codebrowse.launchpad.dev to this loggerhead process [08:50] Lo-lan-do: in bzr.dev [08:50] but where is loggerhead folder in that case in ur system [08:50] indraveni: it doesn't matter! [08:51] there is a new doc - doc/developers/index lists it, which describes packs. [08:51] mwhudson, how does apache know where is loggerhead ? [08:51] lifeless: Thanks, I'll read up on that. [08:52] np [08:53] indraveni: why does it need to? [08:53] it's not running as a cgi script [09:00] mwhudson, when i give bzr.indraveni ( which is my servername in apahce file) its showing nothing [09:01] i am not understanding how loggerhead, apache and bazaar are related [09:01] indraveni: what are you actually trying to acheive here? [09:02] mwhudson, when i configure my apache file with vhost like http://pastebin.ca/747691 [09:02] are you trying to show your branches to the wider internet, just to a few people in your team, just browse them yourself? [09:02] mwhudson, i am able to see the bazaar repos content in browser, using bzr.indraveni as url [09:02] and now , i want to use loggerhead GUI to view the bazaar repo content [09:04]