[00:12] lifeless: What sort of thing would you like to me to put in the developer guide? [00:14] And why this, rather than the many other things we're constantly adding to the codebase? [00:25] abentley: because I remembered to think about it. [00:27] Well, it seems like it would be onerous to do this for everything. [00:27] But where should it go, and what should it say? [00:28] It doesn't seem to belong under "domain classes". [00:29] Also, I've just found an optimization that gets it about as fast as the old implementation. [00:30] On knits. [00:31] cool [00:35] ok 1.0rc1 debs are up; including dapper [00:37] Oh, cool. [00:38] lifeless: Can you give me some guidance on the documenation? I'm otherwise ready to re-submit. [00:39] abentley: one sec, email frenzy [00:39] Okay. [00:40] lifeless: no bzr-svn debs ? would be nice to have them up, at least for gutsy [00:46] jelmer: hmm, I'll see how far I get; I go on leave in two days [00:46] abentley: ok, with you know. [00:47] Fire away. [00:47] so if the answer is 'there is no developer docs 'about merge' at the moment, then you're good - don't worry,. [00:48] starting something that covers the design, motivation, tricks n tips etc for people writing to the collection of apis that combine to make up 'do merge' would be nice at some point., [00:48] I don't remember writing any, so there probably aren't any. [00:49] I can't see any at a glance in doc/developers [00:49] * lifeless -> food [00:49] Okay. Thanks. === kiko-afk is now known as kiko-zzz === mneptok_ is now known as mneptok [01:38] igc: Hi. [01:39] hi abentley [01:39] Would you be able to review my criss-cross documentation? http://bundlebuggy.aaronbentley.com/download_patch/%3C47505D5B.9030702@utoronto.ca%3E [01:40] jam was a bit concerned about the style. [01:40] abentley: sure [01:41] The bit about merge --weave being slow on packs may be obsolete soon. [01:41] ok === doko_ is now known as doko [02:31] Could someone mark https://bugs.launchpad.net/bzr/+bug/172612 as fixed? The patch is in bzr.dev now. [02:31] Launchpad bug 172612 in bzr "new commit is overly verbose" [Low,Confirmed] [02:38] done [02:43] Thanks. [02:51] bbs [03:00] lifeless: have you written up anything in English describing the scenario stuff in bzrlib.tests? [03:03] * igc lunch [03:25] jml: no [03:30] jml, i know a bit about it, i can possibly describe it in documentation [03:30] in fact, i did write something in HACKING, you could look at that [03:30] poolie: I'll take a look. [03:30] I should finish my in-file tweaks. [03:30] questions welcome [03:30] might do that tomorrow === me_too is now known as turbo0O === turbo0O is now known as me_too [06:18] poolie: patch sent [06:29] how do you actually use DVC to commit? [06:31] add a log entry and then hit C-c C-c in the log buffer apparently. [06:31] seems kind of backwards. [06:34] jml: cou can hit 'c' in a dvc-diff or dvc-status buffer, it will open a buffer for the commit message and there you can hit C-c C-c [06:35] vila: ahh, ok [06:35] vila: I guess seeing as I review a diff before I commit anyway, that makes sense. [06:36] jml: yeah the dvc-diff buffer is handy, since it's based on diff-mode you can also very easily revert any hunk by using C-c C-a or mark the files for selective commit === quicksil1er is now known as quicksilver === mrevell is now known as mrevell-sprint [11:55] hi, is there a document that covers running a bzr server using sshd? [11:57] you don't need to run a bzr server there [11:57] just install bzr on the server and bzr will run it on the remote side over ssh [11:58] luks: is there a way to limit the ssh access on the server? [11:59] luks: I don't want to allow full ssh access on my machine. [12:00] um, I'm not sure [12:01] for my subversion server I use the command option in the key configuration: command="/usr/bin/svnserve -R -t -r /srv/subversion" [12:05] ui, you are releasing 1.0 today? congratulations! [12:19] glatzor: you can use something similar for bzr too [12:20] i think it's "bzr serve --inet --allow-writes", but i'm not completely sure [12:20] --directory=/ too. [12:21] (And maybe even something else that gets cut off the edge of the screen.) [12:45] jelmer: sorry, I meant the trunk of 0.4, so "bzr-svn-0.4" branch. [12:45] glatzor: You want to allow bzr access, but not normal shell access? A restricted shell like rsh can do that. [12:48] dato: I would be interested to hear why the trunk version broke [12:48] jelmer: lemme pull and try again. what I saw a couple of days ago is that it didn't do anything after parsing the dump file. [12:59] mwhudson: Peng: so I would need a user/key for every repository [13:10] glatzor: No. [13:10] glatzor: Create as many users as you want. They just all need the proper file permissions on the repo. === kiko-zzz is now known as kiko [14:11] jelmer: there must be something weird in my user setup; as my user, svn-import does nothing, as a newly created user it works. [14:13] dato: does removing ~/.bazaar/subversion.conf help/ [14:13] ? [14:14] lemme check [14:15] yes === cprov is now known as cprov-lunch === mw|out is now known as mw === kiko is now known as kiko-fud [15:03] abentley: ping [15:25] jelmer: so, do you really recommend to use rich-root or rich-root-packs, instead of -with-subtrees ? [15:25] dato: I don't know that jelmer specifically does, but it was a compromise between him and Aaron [15:26] -subtrees is still considered experimental, because it enables nested tree detection [15:26] dato: yep, I do [15:26] right. so I have to choose between compatibility with < 1.0, using an experimental feature, or >1.0 with non-experimental stuff. [15:27] * dato ponders, as this branch is for other people to use. [15:30] oh. [15:30] you cannot `bzr upgrade` -subtrees to rich-root, but you can pull from -subtrees into a rich-root repository :-? [15:42] jam-laptop: ping, who uses bundle format v4 ? It requires seeking backwards (or force me to wrap http.get() into a StringIO) :-/ [15:43] vila: are you working on the ability to apply remote bundles ? [15:44] jelmer: no, on the ability to remove wrapping all http requests into StringIO [15:44] vila: I want to say that v4 is the newest form, so we *all* use them. === jam-laptop is now known as jam [15:44] however, I haven't paged through it thoroughly [15:44] I know Andrew was working on making them more of a producer + consumer [15:44] so he could stream chunks as part of the response [15:45] def get_bundle_reader(self, stream_input=True): [15:45] self._fileobj.seek(0) [15:45] return BundleReader(self._fileobj, stream_input) [15:45] Invalid range access in http://localhost:52409/test_bundle at 28: RangeFile: trying to seek backwards to 0 [15:45] That's the only remaining failing test :-( [15:46] vila: let me check [15:50] vila: the only code that seems to be using that is the bundle code itself [15:50] I'm guessing that BundleInfoV4 reads some of the header, etc to make sure that it really is a bundle [15:51] and then passes the stream into the BundleReader class [15:51] and the BR class was written to start from the beginning [15:51] But I notice that the first thing it does is chomp the first line off the file [15:51] So it is possible that we could just pass a "first line read" flag [15:52] vila: in general, I don't see why it needs to seek, other than maybe multiple calls to get_bundle_reader? [15:52] vila: which specific test? [15:53] TestReadBundleFromURL.test_read_bundle_from_url(HttpServer_urllib) [15:56] the strange thing is that all comments referring to stream_input says stream_input=False is currently faster but everywhere we use stream_input=True... [15:56] vila: that is because = False has to buffer the whole file in RAM [15:57] yeah I know ;-) [15:57] so we chose to be memory conservative [15:57] I also wonder how efficient IterableFile is === cprov-lunch is now known as cprov [15:58] vila: ok, so it seems we have already read the whole file by the time get_bundle_reader() is being called. [15:58] I haven't figured out why yet [15:58] but that is why the seek is needed [16:00] well, I will punt on that one and mark the test as KnownFailure so that we can discuss that during review [16:00] ok [16:01] I think it occur because actually transport.get *seems* to always returns seekable files while ftp always returns StringIOs andd http did too [16:01] vila: we always returned seekable files [16:01] because gzip required it [16:01] gzip.GzipFile [16:01] expects to be able to tell() and seek() [16:01] I think tuned_gzip does not [16:01] seek in both directions ? [16:01] seek backwards, yet [16:01] yes [16:01] it would read a bit [16:02] and then put it back if it didn't like what it found [16:02] but I think that has been worked around in tuned_gzip [16:02] which predictively reads [16:02] but keeps a small buffer [16:02] the test suite triggers nothing related so you should be correct [16:02] vila: do you know pdb very well? [16:02] In gdb there is a way to go until you go up a frame [16:03] (something like Finish) [16:03] but I don't see that in pdb [16:03] r? [16:03] jam: can't answer that ;-) [16:03] ah, thanks radix [16:03] you mean 'r' [16:03] radix: I swear I didn't read your reply ;-) [16:03] hehe :) [16:04] up is handy too to inspect callers contexts [16:05] vila: I think we need it.... [16:05] vila: yeah, I know about up [16:05] I just needed it for all the times we nest function calls [16:05] foo(bar(baz())) [16:05] I want to step into 'foo' [16:05] but it steps me into the others first [16:05] and I don't want to "n" my way out [16:06] list [16:06] sorry [16:06] vila: so.... the bundle code needs to read at least the start of the file [16:07] so that it can get the first line [16:07] to determine the bundle format [16:07] and return an appropriate serializer [16:07] yeah, got that, but then doing a seek to the beginning of the file *rude* [16:07] well, in my tiny bit of testing [16:07] we do: [16:07] for line in f: [16:07] ... [16:08] which we wanted to do so you could embed a bundle in the middle of an email [16:08] and still have it parsed [16:08] (so we ignore everything until we get to the end of the file) [16:08] anyway [16:08] even though "for line in f" has only yielded a single line [16:08] f.tell() [16:08] returns the end of the file [16:10] vila: anyway, I have the feeling that you will break merging from bundles if we cannot seek [16:10] so we won't be able to do "bzr merge http://bundlebuggy...." which is something that *I* do. [16:10] And I believe Aaron does as well [16:10] yeah, I will break something, sure ;-) [16:10] The question is how will we repair it :) [16:11] well, for starters, you are just trying to get rid of the StringIO() wrapper in HTTPTransport.get() [16:11] So far we *pretend* that we stream to be memory conservative *but* we do that by wrapping the url content into a StringIO.... Ecuse me ? [16:11] I'm not sure that is necessary for readv() support [16:11] I agree it is distasteful [16:12] but it is probably okay in the short term [16:12] we generally don't do get() on big files [16:12] indexes, bundles, etc. [16:12] Everything else we use readv() on [16:12] Not necessary, you're right, I will rollback my last change and leave http.get() wraps into a StringIO and be done with it, just a big red flashing FIXME ;-) [16:14] vila: thanks [16:14] If you are doing anything fancy, make a note of it [16:14] but I imagine you are just removing the StringIO wrapper [16:14] from what I can tell [16:14] for line in file: [16:14] Well, the FIXME was already there: # FIXME: some callers want an iterable... One step forward, three steps backwards :-/ [16:14] buffers at 8192 pages [16:15] I'm curious if readline() in the middle would still work correctly [16:15] I will add a better explanaton [16:15] jam: in my case, it's a socket, and yes, read/readline share the same buffer [16:15] weird, the first real read is 8192 [16:15] but the second is 10240 [16:15] could be a read(-1) [16:16] and then 10240 from there [16:16] which can of file/socket ? [16:16] sure, I'm just commenting on the "open()" (file object) behavior [16:16] s/can/kind/ [16:16] and doing "for line in file" [16:16] ok [16:16] start = 0 [16:17] for line in f: [16:17] start += len(line) [16:17] print start, f.tell() [16:17] Gives interesting results [16:17] f.tell() is definitely being read in chunks of (8192, 10240, 10240, ...) [16:17] for a random text file ? [16:18] just something on disk [16:18] yes [16:18] a 238k file in this case [16:19] vila: anyway, if I was to propose a fix [16:19] it would be to change the bundle reading code [16:19] so that it reads a minimal amount of data [16:19] so it can get the header [16:19] and then it passes the header [16:19] along with the file object [16:20] to the serializer which supports that header [16:20] jam: +1 but I will not be able to do that in the coming days (new project starting in RL) [16:20] vila: I'll write up a bug on it [16:20] I search a bit in that direction but saw nothing obvious and came here for help [16:20] jam: thanks [16:23] s/search/searched/ [16:23] vila: does http://dpaste.com/26608/ sound good? [16:23] vila: (re our conversation the other day) [16:24] s/to remote hosts/& with sftp/ [16:24] s/is useful to ensure SSL certificates are always verified/is required to verify SSL certificates/ [16:24] vila: didn't you tell me it was also needed for bzr+ssh? [16:24] dato: rats, yes, but that may well be a bug [16:25] jam: did you read the IRC logs ? It looks like smart/medium.py requires paramiko, aaron and I thought it should be able to use openssh only [16:25] vila: at one point, we could start a bzr+ssh connection without paramiko [16:25] if we can't [16:26] then it is a regression [16:26] kind of hard to test for... [16:26] jam: look at smart/medium.py I didn't test [16:27] vila: I think it is a bad comment [16:27] It used to "from bzrlib.transport import sftp" [16:27] which *does* need paramiko [16:27] 'ssh' should not [16:28] I'm a little concerned that we use "paramiko.SSHException" in the connect_sftp code [16:28] such that if we don't have paramiko, and we fail to connect [16:28] we will get an AttributeError exception [16:28] vila: but other than that, everything should work without paramiko installed [16:28] dato: as jam said ;-) [16:30] dato: so forget my first comment, but please use the second one, it's the only remaining bit that urllib does not handle today and it looks like for some people it's more a problem than an help [16:31] so there are really two kinds of users, those that want certificate verification and those that absolutely don't want them [16:31] vila: ok. re "more a problem than a help" that's what I wanted to reflect in my wording, like "only install it if you're really sure you want such verification" [16:31] vila: I just removed paramiko from my system [16:31] and "bzr log bzr+ssh:// " worked fine [16:31] great [16:31] oh, good [16:32] I'll post a simple patch to the list [16:34] hmm... it is only in connect_sftp() and our sftp subsystem *does* require paramiko to be present... === kiko-fud is now known as kiko [16:38] dato: the reason why we went with Requires for paramiko (IIRC) was that people expected sftp support out of the box [16:38] Since we now at least have bzr+ssh, I suppose we could relax it a bit [16:39] paramiko has always been in Recommends AFAICR, but tools are supposed to install recommends by default. my question the other day was whether to demote pycurl to suggests, which I'm doing right now. [16:40] dato: for Ubuntu, I'm pretty sure it was set to Required [16:40] but I could be remembering that incorrectly. [16:56] New bug: #173689 in bzr "BundleReader cannot stream" [Wishlist,Triaged] https://launchpad.net/bugs/173689 [18:01] dato: the parallel build check showed that some of bzr refuses to build parallel, at least the docs. I think this is because rules calls 'make docs', when it should be '$(MAKE) docs'. [18:02] dato: mind if I correct that? A grep didn't show any other uses. [18:02] james_w: please do [18:03] it might actually cause build failures, or even allow a broken package to be build, if a parallel make is used, but I'm not sure how to test. [18:04] dpkg-buildpackage -j2 ? [18:09] hmm [18:09] so for a test, i want to lock a branch with a given lock id [18:09] this seems to be pretty hard [18:09] james_w: if you don't mind, please use dch -t in the future (leaves the footer alone; so it gets added when opening a new version, and when preparing a final upload) [18:10] dato: ok, thanks for telling me. [18:40] mwhudson: for any branch format, or just for a specific one? [18:41] james_w: Hi James, did you get my merge request for bzr-builddeb? [18:41] because you could just override [18:41] branch.control_files._lock [18:41] jam: er, no, not really [18:43] ? [18:43] otherwise, I guess you could override bzrlib.lockdir.rand_chars [18:43] since that is the final lock token [18:45] jelmer: yeah, the first one is fine, the second I want to think about a little. [18:45] jelmer: also is exposing the file properties planned/wanted/not wanted? [18:46] james_w: planned/wanted. It depends on support for custom file properties in bzr core though [18:47] james_w: what are your thoughts on the second one? [18:48] jam: let me try to explain what i'm trying to do [18:48] jelmer: do you saw bzr-svn selftest results? [18:49] jam: obviously, branch puller on launchpad locks branches while they are being updated [18:50] jam: if things go wrong in a particular way, these branches can be left locked [18:50] k [18:50] with you so far [18:51] so what i'm doing is have the puller use a particular lock id [18:51] and you want to make the branch puller use specially formatted nonces? [18:51] (by setting BZR_EMAIL) [18:51] so for a test, i want to lock a branch with the same id [18:51] (just realize that pack repositories don't have lock ids at that level) [18:52] then run the puller and check that the puller still runs [18:52] (it will look for and break such a lock) [18:52] jam: ah, that's interesting [18:52] jam: so how would you solve this problem? :) [18:52] pack repositories don't take out a physical lock at 'repo.lock_write()' [18:52] time [18:52] they take a physical lock only when they go to update the 'pack-names' file [18:52] jelmer: I guess I'll just wait for that, it's not a massively urgent to implement this. I suppose I could guess in the meantime as well, and there is always the command line option. [18:53] moin moin [18:53] morning lifeless [18:53] mwhudson: are you trying to get a custom lock token (aka nonce) [18:53] privet [18:53] mwhudson: or are you trying to set the username inside the lock to that specified by BZR_EMAIL [18:53] jelmer: as for the second one, I think it's a fine aim, I just want to work out if it is the most natural way to do it. Your final example command line is great, so I guess it may well be. [18:53] mwhudson: I think you are trying for the latter [18:53] such that peek_lock() will give you back what you want [18:54] versus having token = repo.lock_write() give you back what you want [18:54] mwhudson: do you understand the difference? [18:55] jam: i don't think i care [18:55] bialix: hi [18:55] jelmer: evening [18:55] jam: i want to be able to say "is this branch locked? if so, did i lock it?" [18:56] then, "if all that, break the locks" [18:56] bialix: all tests should actually be succeeding, from what I understood from reports [18:56] james_w: ok [18:56] if a custom nonce is an easier way of achieving that, then maybe that's what i want to be doing [18:56] mwhudson: well the string returned by "branch.lock_write()" is only part of the whole info on disk [18:57] but lock_write() will spin rather than returning right away [18:57] (default timeout of 5 minutes, etc) [18:57] jam: right [18:57] are you trying to avoid that as well? [18:57] (you *can* set bzrlib.lockdir._DEFAULT_TIMEOUT_SECONDS if you want) [18:57] i'd like to, yes [18:58] jelmer: I has firewall on my machine. do you want me to run tests one more time with disabled firewall? [18:58] mwhudson: so... Branch.lock_write() will take out a physical lock right away, so it is probably best to think about starting from there. [18:58] my impression is that lockdir.py has all this configurability and flexibility and branches have lock_write() that takes no arguments [18:58] mwhudson: pretty much [18:59] and I assume you are trying to do it without too much surgery into bzrlib itself, right? [18:59] jam: well, i want to deploy this code within the next two weeks, so yes [19:00] mwhudson: pack repositories have no over-the-wire locks at all [19:01] hm, suddenly log in bzr.dev is very slow? [19:01] mwhudson: if the hpss decides to lock a pack repository itself, it will be because the hpss is performing the write. [19:01] lifeless: this is for the mirror puller [19:01] mwhudson: ok; it will take a lock during insertion of the repository for a couple of ms [19:01] dato: with 1.0rc1 or bzr.dev? [19:01] lifeless: so the hpss doesn't come into this [19:01] jam: bzr.dev [19:02] lifeless: it does sound like this is not going to be a problem for packs [19:02] bug #172975, bug #172567 [19:02] jam: logging in bzr.dev [19:02] Launchpad bug 172975 in bzr "bzr log slower with packs" [Medium,Triaged] https://launchpad.net/bugs/172975 [19:02] Launchpad bug 172567 in bzr "Per-file log is *very* slow with packs" [Undecided,Incomplete] https://launchpad.net/bugs/172567 [19:02] (or at least, not a very real one) [19:02] mwhudson: the branch object will have a longer lock held open [19:02] jam: thanks [19:02] dato: I have a workaround in 1.0rc1 [19:02] we are trying to implement the "right fix" for bzr.dev [19:02] ok [19:02] jam: ah it did get merged straight to 1.0 ? [19:02] lifeless: oh, well, we'll have the same problem then [19:02] jam: good [19:02] lifeless: I believe martin put it in the 1.0 branch [19:03] jam: I checked bzr.dev; didn't see it there. I don't know if 1.0's code has the fix or not. And he's off today. [19:03] I'll peek [19:03] jam: anyhow, I have a considerably better idea of how to get my point across :). I might write something up about it [19:04] he did a "merge updates for 1.0" [19:04] jam: did he? good. :) [19:04] but I'm not sure what that included [19:04] (and, of course, bzr log is slower now :) [19:05] lifeless: it looks like my workaround is *not* in 1.0rc1 [19:05] jam, lifeless: so now i'm more confused than i was when i first asked in here :) [19:06] (that i started work 11 hours ago may be related to this) [19:07] mwhudson: so... basically you want to be able to trap when a branch is already locked, peek to see if it was another process that you would have been controlling, and break it automatically when necessary [19:07] jam: yes [19:07] bialix: nah, it shouldn't do any network access at all [19:07] interesting, we don't even use the per-branch config username, just the global one... [19:08] bialix: what version of windows are you on? [19:08] xp sp2 [19:08] xp home actually [19:08] mwhudson: So, you want to depend on bzr's branch locking, but be willing to ignore it :) [19:08] mwhudson: don't we write the pid in the extra metadata? [19:09] when opening a branch [19:09] I would probably monkey patch .lock_write() [19:09] so that it used self.control_files.attempt_lock() [19:09] and if that fails, then you can peek, etc. [19:09] lifeless: not really sure how that helps? [19:09] mwhudson: 'when is it safe to break_lock' ? [19:10] mwhudson: so you know that all locked branches are from other pullers right? [19:10] mwhudson: when you find a locked branch, you can check the host name, see if it is your current host, and then check to see if the pid is still alive [19:11] jam: it has to always be the current host [19:11] if host == 'localhost' and pid is not present (as given by ps, or whatever) [19:11] then you know you have a stale lock [19:11] umm, is there some kind of semantic difference between RevisionSpec.in_branch, in_store and in_history? [19:11] jam: the public area is not directly writable by users [19:11] they are all aliases, but I guess each was meant for something different? [19:11] lifeless: well, no not necessarily, sometimes for better or worse operators end up touching branches [19:11] mwhudson: on the public side the most operators should do is delete them :) [19:12] lifeless: not going to argue with that [19:12] luks: the idea was in_branch -> in the left side history. in_store -> in the repository, in_history -> in the ancestry of the branch. or something. [19:12] lifeless: this complication wasn't my idea :) [19:12] luks: overly engineered we suspect [19:12] lifeless, luks: probably true, but I think we only implemented in_store... :) [19:13] maybe the code should just break all locks it finds [19:13] and a lot of code expects in_store ability [19:13] mwhudson: sounds aggressive :) [19:13] mwhudson: do you mutex branches against concurrent pullers? [19:13] lifeless: lost my nickserv password i'm afraid, can't pm you here [19:13] I finds 'em, and I breaks 'em [19:13] (until it expires :) [19:13] lifeless: yes [19:14]