[00:33] Is there some estimate of when nested (by-reference) trees will become a supported feature? [00:41] is there any way to get a list of files that would be applied in a merge without applying it? `bzr merge --preview` shows a diff, but I'd like to see the files (more like a --pretend?) [00:42] as a quick fix, you could pipe the diff through diffstat [00:47] is that a unix command? [00:48] i ended up just applying the merge and piping stderr to a file, and then reverting it [00:58] fryguy_: yes, it's in a package you can install. The "diffstat" package, as it happens. [00:59] i'm on windows :( [01:00] I feel your pain [01:01] well, I do like developing on windows. c# is a lot of fun.. so it's not all bad [01:03] I did like C# okay. [01:03] ah, might be harder to get diffstat on windows [01:03] Didn't much care for codebehind, but C# was good. [01:27] morning [01:46] Random thought of the moment: I think the decentralized-but-not-quite-distributed development model is far more easily supported by bzr than by any other vcs I've encountered. [01:47] Or to put it another way, I like bound branches. [01:48] take up bonsai as a hobby. [01:48] ;) [01:49] Lulz. [02:02] Kilroo, it's true [02:23] lifeless: yeah, I saw that GIL message. My guess is it doesn't have much relevance for us [02:24] lifeless: because our common case atm is pull from bzr+ssh, which shouldn't involve any threads. [02:24] beep, wrong [02:25] or perhaps [02:25] I was thinking paramiko for a second [02:25] but then I remembered that paramiko isn't used for bzr+ssh [02:25] however [02:25] Right. [02:25] we do have a threading handshake-thunk in the server streaming and client streaming code [02:25] don't we ? [02:25] converting reads to generators [02:25] We do use threads for insert [02:25] and generators to writes [02:25] And for builtin TCP serve. [02:26] right, TCP serve very obviously has threads [02:26] No thread for get_stream*, just insert_stream* [02:26] so [02:26] if the client doesn't read its buffer enough [02:26] client side buffer fills up [02:27] But we've been observing apparent oddness with get_stream [02:27] os socket buffer is full - network will back off [02:27] yes, oddness to [02:27] too [02:27] I filed a couple of bugs I want to chase down, as branch time is hurting the distro [02:27] Even in the inserter case, there's no fixed size buffer. [02:28] As the network thread receives bytes it unwraps them from the HPSS layer and pushes them onto a Queue.Queue [02:29] Which "should" be fast, assuming the other thread(s) aren't starving the network thread of CPU time. [02:29] Yeah, I saw the bugs, they are marked Critical after all ;) [02:29] oh good, for some reason I thought we had a fixedish queue [02:30] the reason the GIL thing stood out to me was that it describes networking threads getting starved out of proportion to the cpu load - AIUI [02:30] Well, from reading from a pipe, we are a bit pessimistic, lots of small reads to avoid blocking. [02:31] When reading from a socket we just always grab 4k or something. [02:31] is bzr+ssh considered to be a pipe or socket ? [02:31] A pipe. [02:31] Anyway, for the bugs you filed, they are streaming from LP, so no threads in the server bzr involved. [02:31] argh [02:32] maybe I should fix [02:32] Conflict: can't delete bzrlib/plugins/bash_completion because it is not empty. Not deleting. [02:32] first [02:32] as its breaking the workflow for 2.0->2.1->trunk merges [02:32] lifeless: you would make many people happy [02:32] * lifeless gets out the economy sized yak shaver [02:32] lifeless: I suggest "moving bzrlib/plugins/bash_completion to lost+found/" ;) [02:35] Hmm. I have a project where I'm interested in determining if any files in a large tree revert to prior states, and being able to scan said directory for changes quickly/efficiently; however, I _don't_ need to be able to retrieve content associated with said former states, just enough metadata to identify them. [02:35] * nDuff is pondering whether bzrlib can be pervented to the purpose. :) [02:35] so, you're happy with a lost+found directory approach ? [02:35] perhaps bzr-lost+found, so as not to collide with the fs one [02:35] nDuff: sure, you could use the dirstate module, or perhaps even working tree 4, directly. [02:36] you may find some areas where we are a bit fuzzy on layers - but we'd be delighted to sharpen them up to serve your purpose [02:36] lifeless: You could just do .saved or something, too. [02:36] lifeless: Somewhat like rpm does. [02:38] lifeless: I'm happy with any improvement [02:38] lifeless, thanks for the encouragement -- I was already reading through the pydoc for bzrlib.dirstate, and will look into the API to the working tree code as well. [02:39] lifeless: I think in practice most people will routinely delete everything in lost+found, and maybe rarely rescue one or two files from it [02:39] mkanat: are there docs on precisely what rpm does ? [02:39] lifeless: Good question. [02:40] lifeless: but "rm -r lost+found" is a much easier way to recover a sane tree than the current situation [02:40] lifeless: Brief search finds an email mention briefly: http://www.redhat.com/archives/rpm-list/2001-July/msg00213.html [02:40] So, I guess I am happy with the lost+found approach :) === AfC1 is now known as AfC [02:41] mkanat: that doesn't seem to address this situation (that a managed directory is being deleted; what should happen if there are unmanaged files in it?) [02:41] mkanat: so, we do ok on single files [02:41] hey [02:41] lifeless: Yeah. [02:41] I just noticed that kdiff3 is back [02:41] how can I use it to resolve merge conflicts? [02:41] mkanat: we're talking about the case where we want to delete a directory X containing a file X/Y that is not versioned [02:42] is it easy? [02:42] mkanat: or is versioned and is modified [02:42] mkanat: the proposal is to move the entire path under a top level bzr-lost+found directory [02:42] lifeless: Well, if it only contains unversioned files and is being deleted, then it's fairly easy to just say "make it .moved or .saved" or something. [02:42] mkanat: and say that we've done that [02:43] lifeless: I imagine that's the most common case--that's the one I've hit. [02:43] mkanat: 99% of the time [NotAMetric statistic] its just a .pyc file so the entire thing should be deleted [02:43] at the moment though, we leave the directory in place, and it gets in the way when you switch back [02:43] lifeless: Unless it's some configuration file that's important to the functioning of some customization on a branch that you're merging into, or something like that. [02:44] mkanat: which is why moving it under a different directory is better - the common case is to nuke all the things; the uncommon case is to want to add the containing directory again and put the file back [02:44] mkanat: sure thats why we can't delete the file outright [02:45] lifeless: Okay. Might want to call it bzr-deleted or something like that, for clarity. [02:45] Or -removed. [02:45] lifeless: I think it will be somewhat surprising behavior, though. [02:45] I'm not sure thats more or less clear [02:45] mkanat: the current behaviour sucks :) [02:46] lifeless: I think lost+found definitely won't be clear to people who don't have that on their OS. [02:47] lifeless: I think I'd find it somewhat more intuitive to rename the directory in place. [02:47] what do you mean ? [02:48] lifeless: I mean, if I were looking for where my files went, and I actually needed them. [02:48] lifeless: I could also nuke them with clean-tree, which is how I nuke things now anyhow. [02:48] so we print out whats happening [02:48] lifeless: But bzr prints out so much stuff when merging, it might get lost. [02:48] mkanat: it doesn't on switch, IME [02:49] lifeless: Oh, that's true. [02:49] mkanat: this is primarily an issue for switch, because you're often changing between radically different code bases. [02:50] lifeless: I think I encountered it with "up" once, maybe, when I was updating with local commits. [02:50] we could introduce the concept of precious files and say 'ignored files can be deleted willynilly', and treat unversioned unignored files as precious [thereby avoiding an initial config file], but we'd still have an issue with that config file story, for intance. [02:50] lifeless: Yeah. [02:51] lifeless: I'm just thinking about the case where, let's say I have two or three of those directories, I switch, and then I do "bzr status", and I see only one directory that I never added. [02:52] lifeless: I think treating it like a contents conflict and making it .moved would be more in line with what I normally expect bzr to do. [02:53] do this [02:53] sit in bzr.dev [02:53] bzr switch 2.1 [02:53] bzr switch 2.0 [02:53] [using whatever layout you want :)] [02:54] oh, but dp a 'bzr selftest nothingtodo' at each step, to compile pyc files [02:54] Okay. [02:55] spiv: the 2.0->2.1 merge could use a NEWS merge magic to move the new items in 2.0.unreleased to 2.1.unreleased [02:55] spiv: any ideas how? file a bug ? [02:57] we should squelch /usr/lib/pymodules/python2.6/lazr/uri/__init__.py:19: UserWarning: Module launchpadlib was already imported from /usr/lib/pymodules/python2.6/launchpadlib/__init__.py, but /usr/lib/pymodules/python2.6 is being added to sys.path [02:57] import pkg_resources [02:58] too [02:58] actualy, lazr should [03:02] lifeless: that would be good, file a bug [03:02] I did already, weeks back [03:02] now to find it [03:02] lifeless: Ah, okay. [03:02] lifeless: (I just went through the process.) [03:02] lifeless: there's the related case of "new item in feature branch, when merged to trunk (or stable branch) move to right place" [03:03] lifeless: See, I think that if there were anything important in there, and they were in a deep directory structure, trying to sync them back into the right place from a lost+found directory would be difficult. [03:04] mkanat: cp -r lost+found/foo ./ would do it quite well [03:04] lifeless: you're possibly thinking of bug 517490? [03:04] mkanat: where foo is a top level dir there, not the whole path [03:04] Launchpad bug 517490 in Bazaar "news_merge confused by added sections (affected: 1, heat: 6)" [Low,In progress] https://launchpad.net/bugs/517490 [03:04] lifeless: On *nix, if you're familiar enough with the command line. [03:05] mkanat: explorer will permit copying to merge things too [03:05] lifeless: True. Does the OS X interface, also? [03:05] mkanat: I believe so [03:05] lifeless: Okay. [03:05] lifeless: Well, why not just not consider them conflicts? [03:06] lifeless: That is, just leave them where they are, unversioned, and don't add a conflict. [03:06] mkanat: thats what we do at the moment, and that is annoying us [03:06] spiv: no [03:06] lifeless: Is that new since 2.0? (I see conflicts in bzr status.) [03:06] spiv: grab 2.1 from lp, merge in 2.0. there is a small conflict - resolve that [its not what I'm talking about]. Then compare the result to what I committed in lifeless/bzr/2.1 [03:07] lifeless: I see how it gets annoying when you move back to bzr.dev. [03:07] lifeless: Renaming them to .moved or .saved would work too, though, wouldn't it? You could do .saved.1 if there was already a .saved there. [03:08] lifeless: I agree that isn't 517490 [03:08] lifeless: just that it's the closest I remember you filing [03:09] mkanat: so if we renamed them to .moved or .saved, (the dir that is), we'd have less issues with successive switching back [03:09] but we'd still clock up an impressive amount of cruft [03:09] lifeless: Yeah, true. [03:10] lifeless: And in the switch case, you probably don't want that stuff sticking around, usually. [03:10] lifeless: What if switching puts that directory in .bzrignore though? [03:10] ideally we'd just delete the dir - but that needs more info than we have [today], or a change in the definition of 'what bzr does to ignored files' [03:11] mkanat: so in branch A its versioned and B its unversioned and ignored ? we still promise not to delete files we can't recreate [03:11] Okay. [03:11] So we use the versioning status of it in branch B to decide whether it goes in lost+found? [03:11] I understand the convenience of it. I still think that it will be surprising to people who don't know the rationale, though. [03:13] mkanat: I think some surprise is inevitable, given the constraints. [03:13] so the current concept I have in my head is: [03:13] when a directory is deleted by a transform and it has children we can't dispose of gracefully, we move the directory [03:14] unlike files *most* conflicts of this nature will be because of build products / editor temporary files (including ones bzr makes) and so on. [03:14] e.g. __init__.py~1~ [03:15] I think Tom may have had it right when he had precious files, in arch. [03:15] so a question here is, is it better to do this now, or do precious files first, or just one of the two things [03:16] if we just do precious files (or junk files - they are symmetrical enough), we could gracefully dispose of more directories. [03:16] and the thing wouldn't be as fantastically annoying [03:17] if we just move directories we can't dispose of further than we do today, we won't get conflicts switching back and forth, but we will still pile up the cruft [03:17] if we do both, we won't get conflicts, and we won't pile up as much cruft [03:18] Avoiding cruft is less important than avoiding conflicts. [03:18] so, I think, I'm going to just make these conflicts a little harsher - to dir.deleted [03:18] if thats not enough we can go for a single place to move the cruft, and-or junk|precious file categorisation [03:19] mkanat: thanks for asking all those questions [03:19] lifeless: Thanks for having the conversation with me. :-) [03:19] If you do dir.deleted, make sure it still solves the problem with multiple levels of deleted dir, each with unmanaged files. [03:21] spiv: naturally - I already had that on my mental tests to write list [03:21] spiv: so - did you see what I mean with the 2.0->2.1 merge? [03:23] lifeless: I haven't yet run those commands, but the description made sense. [03:23] spiv: so, I should file a bug ? Basically I want to copy upwards new things to show where they arrive in the series [03:24] lifeless: yes [03:24] is there a tag for this [03:25] I thought there was, but it doesn't seem to exist anymore. [03:34] https://bugs.edge.launchpad.net/bzr/+bug/583630 [03:34] Launchpad bug 583630 in Bazaar "Support the news mangling we do when merging up across series in newsmerge (affected: 1, heat: 0)" [Wishlist,Confirmed] [03:37] spiv: https://bugs.edge.launchpad.net/launchpadlib/+bug/552419 is the warning to squelch bug [03:37] Launchpad bug 552419 in python-launchpadlib (Ubuntu) "Multiple module import warning from ubuntu-bug (affected: 5, heat: 30)" [Undecided,Confirmed] [03:38] lifeless: ? [03:38] spiv: the launchpadlib already imported thing I was saying we should suppress [03:39] just FYI - gary closed it as invalid, need to find out why [03:39] I'm still lacking context, I don't recall noticing that bug myself. [03:39] oh [03:39] are you running lucid ? [03:39] Yes. [03:40] can you please run feed-pqm bzr (from trunk) [03:40] I see this: [03:40] When would I notice it? Using hydrazine? Or just lp:... lookups? [03:40] ./feed-pqm bzr [03:40] /usr/lib/pymodules/python2.6/lazr/restfulclient/__init__.py:19: UserWarning: Module launchpadlib was already imported from /usr/lib/pymodules/python2.6/launchpadlib/__init__.py, but /usr/lib/pymodules/python2.6 is being added to sys.path [03:40] import pkg_resources [03:40] loaded existing credentials [03:40] bzr lp-propose [03:40] etc [03:40] lp: lookup doesn't use launchpad apis. [03:40] ./feed-pqm bzr [03:40] Yeah, that's what I thought. [03:40] /usr/lib/pymodules/python2.6/lazr/restfulclient/__init__.py:19: UserWarning: Module launchpadlib was already imported from /usr/lib/pymodules/python2.6/launchpadlib/__init__.py, but /usr/lib/pymodules/python2.6 is being added to sys.path [03:40] import pkg_resources [03:40] bah pastefail [03:40] loaded existing credentials [03:42] I don't see that with hydrazine trunk. [03:42] ok, thats *interesting* [03:42] fresh lucid install ? [03:42] what version of setuptools? do you have the launchpad ppa ? [03:42] Depends on what you mean by fresh ;) [03:43] Recently upgraded to lucid (< 1 month), but the system as a whole was originally installed >3 years ago, IIRC. [03:43] ok, not fresh :) [03:43] well, will need to dig deeper [03:44] python-setuptools 0.6.10-4ubuntu1 [03:44] I don't think I have the launchpad PPA [03:53] and pkg-resources? [03:53] hmm [03:53] possibly pycentral or whatever to blame to [03:53] too [03:54] Same version of pkg-resources [03:54] ok [03:54] something to track down later [03:55] for now; time to go pickup bank card, fuel the car - do the stuff that got truncated before [03:55] won't be long [04:10] * igc lunch [04:46] I need a teddy bear [04:46] spiv: got a minute [04:47] actually scratch that, XXX time. [05:55] spiv: so we might want to move to 64K reads on pipes and sockets when we're in streaming mode [05:55] keep the buffers as empty as possible [05:55] or for cleverness value, figure out the current buffer size regularly and start reading in that size [05:55] can scale up quite large [05:55] Yeah. [05:55] MBs [05:56] Although at some point large reads may start hitting perf issues in our code [05:56] ziggy bytes! [05:56] Due to relatively naïve string splitting, etc. [05:56] \o/ partial branch fail -> uploading entire bzr now. [05:56] spiv: yeah. OTOH if we find those we can fix em [05:57] Right. Just something we should be aware of. === tchan1 is now known as tchan [06:20] :( [06:20] :!bzr push lp://staging/~lifeless/bzr/propose-accepted [06:20] 119387kB 74kB/s | Fetching revisions:Inserting stream [06:34] spiv: another data point, we're doing many small writes in that push ^ [06:34] write(7, "b\0\0\f\346B3289\ntexts\n\ngroupcompress-"..., 3307) = 3307 [06:37] lifeless: Hmm :( [06:38] I thought we'd fixed that :( [06:40] - 207016kB 0kB/s | Fetching revisions:Inserting stream [06:40] a tad excessive, methinks [06:41] For bzr.dev? Oof. [06:41] yeas [06:42] now I get to test my patch [06:42] YAK SHAVING [06:42] I hear you have particularly woolly yaks in NZ. [06:44] hmm, trunk is unhappy for lp-propose [06:44] however, dinner, then digging. [06:44] NotFound: Object: , name: u'1.0' [06:45] <> [07:53] back [07:54] ok, so lp-propose in trunk is failing to get an OAuth token [08:02] mgz: did you do the edge->production mass change? [08:02] * lifeless needs to find it again [08:02] http://bazaar.launchpad.net/~bzr-pqm/bzr/bzr.dev/revision/5244/bzrlib/plugins/launchpad/lp_propose.py [08:02] revert that, see if it magically fixes it [08:02] mgz: it does [08:02] mgz: or rather, reverting the entire patch fixed it [08:03] okay, so, either I didn't know what I was doing and broke something, or a change is needed elsewhere as well [08:03] mgz: possibly/probably the latter. Or cached credentials may be at fault. [08:03] mgz: I propose to: [08:03] - back it out now [so dailies don't get broken etc] [08:04] can you see if just backing out that line is sufficient? [08:04] sure [08:04] narrows down what needs looking at. if it's just that, nix it. [08:06] yeah, that makes it work for me [08:06] https://code.edge.launchpad.net/~lifeless/bzr/propose-accepted/+merge/25747 [08:09] mgz: so, roll back that one change and you'll inwestigate ? [08:10] yeah, revert that file then, and maybe someone who knows launchpadlib can say what was wrong [08:10] I think its probably something to do with the beta->1.0 change. Or something. [08:10] or yeah, I can poke further [08:10] so, I think the key elements are: [08:10] lets open a bug on this [08:10] so we can close the one about xmlrpc [08:11] right, makes sense. [08:11] I'll do that [08:11] are there any elements not done for the current bug? it's filed against three projects. [08:11] whats the current bug # ? [08:12] 581670 [08:12] ...I just spent fifteen seconds failing to copy that from my address bar >_< [08:12] mouse/keyboard coordination lacking [08:13] :) [08:13] https://bugs.edge.launchpad.net/bzr/+bug/583667 [08:13] Launchpad bug 583667 in Bazaar "bzr talks to edge API servers to propose merges etc (affected: 1, heat: 0)" [Wishlist,Confirmed] [08:14] * mgz unedges the url to subscribe [08:14] also, naughty, the 581760 change didn't have NEWS entry. [08:14] please always put stuff in NEWS if it is going to matter to anyone. [08:14] didn't really seem user-facing to me, but maybe it is [08:15] new cached credentials [08:15] every user having to re-auth against lp is fairly user facing :) [08:15] ah. [08:15] now, arguably we should use the same token file on edge and prod [08:15] I'll file a bug on that in a sec. === radoe_ is now known as radoe [08:22] mgz: https://code.edge.launchpad.net/~lifeless/bzr/lp-propose-host/+merge/25749 review kplease [08:27] done. [08:31] oh also - for judging whether to put something in NEWS [08:31] we have a programming userbase too; so 'user facing' really means 'end user or plugin/api consumer or devs-should-know' [08:31] if that makes sense [08:34] hm, sure. [08:42] We even have an "Internals" section in NEWS [08:43] For changes likely to be significant for other bzrlib devs, I guess, even if it's not strictly meant to affect end users or 3rd party code. [08:54] mgz: anyhow, enough said about it - I think you need to dial up the 'tell people about it' sensitivity a bit and its all good. [09:13] night all [09:19] gnight [10:04] hmpf, need to get some of my harder stuff to a landable state, not just doing these ten-minute changes [10:17] true [10:17] and I'm going to encourage you on the testtools exception encoding thing too. [10:18] though I think its temporarily under control [10:18] I'm struggling to write that in a non-black magic way [10:18] if you want a teddy bear, tell me your thoughts [10:19] I'm utterly failing to do $fridaynight [10:19] well, to change how testtools formats the traceback, I'm adding a method _exc_info_to_unicode to TestResult [10:20] which is basically just _exc_info_to_str but with traceback.format_exception swapped out [10:21] but means I need to to do one of 1) duplicate the function, 2) monkey-patch, 3) crazy low level stuff [10:22] and it only gets more squiggly from there [10:24] the bits that need to be change are scattered across unittest, traceback, and linecache and not in any coherant manner [10:24] ok [10:24] so lets start at the bottom [10:24] we know we want a fixed linecache ? [10:24] lets say we have a fixed_linecache module [10:24] which we use and wraps linecache [10:25] ignoring how we choose to use it, it can be tested well there, yes ? [10:25] hm, yeah, the other option I was considering there was giving it promoted tuples, which also store the file encoding [10:25] sure, that seems orthogonal to where you put the adaption stuff [10:25] somewhere we need the code that fixes linecache [10:25] so lets get that solid with tests. [10:26] then a fixed traceback, with tests. We could use mocking or dependency injection, or just integration tests for that. [10:26] was it selftest you suggested as a template for whole-source-module tests? [10:26] test_selftest that is [10:26] oh [10:26] bzrlib's test_plugins does something like that [10:27] as does testrepository [10:27] testrepository uses testresources... hmm [10:27] a, testrepository, that was it [10:27] jml: how would you feel about testtools growing a make check dep on testresources? [10:27] but yeah, the test_plugins way makes sense to me [10:28] lifeless: fine by me. [10:28] this is going to be 90% corner cases, but should all end up making sense, and cover the common(ish) case of locale'd os messages [10:30] lifeless: I kind of half-think they should be one project anyway [10:30] mgz: ok, so you could use either - have a look at both. If you want to use testresources, I'll happily have the generic resources from testrepository lifted up into testresources in a new resources package, or something. [10:30] jml: possibly [10:30] man, today was a bug filing day [10:30] I see you've done an impressive number [10:31] five or so just from lp-propose investigations [10:34] GaryvdM: actually bzr-search wasn't quite fixed; Its fixed now. [10:42] lifeless: cool [11:12] spiv: apparently resolve --take-other will delete the dir and its contents [11:13] this is tolerable but still ... icky, I think [11:17] * lifeless pops up a level === mrevell is now known as mrevell-lunch === mrevell-lunch is now known as mrevell [13:15] jam: gmorning [14:55] amanica: Mow about hacking together this Sunday? [14:55] amanica: Is your wife busy? [14:57] mow? [14:57] Hi bialix [14:57] hey Gary :-) [15:09] morning lifeless [15:09] Peng__: history-db just landed in trunk, if you want to check it out :) [15:10] hi GaryvdM :-) , I'll have to check with her, but I'm keen to get some things done this weekend. will have to let you know [15:11] hey amanica! how's your trip back? [15:11] amanica: Ack [15:11] hi bialix, I slept like a baby in the plane. it still took me a couple of days to recover [15:12] bialix, I used some of the russian you tought me today [15:12] yep, the same here [15:12] oh, nice [15:12] good to here [15:12] you make your coleage shy? [15:12] no :-) , but she said she "isn't lazy!" [15:12] work hard, don't be lazy! [15:12] yep :) [15:12] lol [15:13] and what you said? [15:14] I told her the story about why I thought about learning that in russian [15:14] her russian is a bit rusted though. [15:15] bialix, you never tought me how to say goodbye [15:16] bye == poka [15:16] wow [15:16] or even paka [15:16] cool [15:16] goodbye is longer [15:16] do svidanya [15:17] direct translation: for next meeting [15:17] cool thanks [15:19] bialix, amanica: qannotate now has Stable Scroll® [15:20] GaryvdM sweet! [15:21] bialix, amanica : It also trys to keep the selection stable, but is a bit iffy if the selection is inside a modified hunk [15:22]