/srv/irclogs.ubuntu.com/2006/07/27/#launchpad-meeting.txt

=== lifeless [n=robertc@ppp245-86.static.internode.on.net] has joined #launchpad-meeting
=== ddaa [n=ddaa@82.109.136.116] has joined #launchpad-meeting
=== jelmer [n=jelmer@a62-251-123-16.adsl.xs4all.nl] has joined #launchpad-meeting
=== ddaa [n=ddaa@82.109.136.116] has joined #launchpad-meeting
ddaahello there05:10
jelmerhi05:10
jelmerIs right now ok for you for the meeting? We can postpone it to this evening if you like. I don't mind05:10
ddaayes, just need to clarify the issues with SteveA first05:12
ddaasorry for the delay05:13
jelmerno worries, just let me know when you're ready05:13
ddaalet's do it05:17
=== jamesh [n=james@82.109.136.116] has joined #launchpad-meeting
ddaajelmer: last time we talked, I gave you a short summary of the issue05:18
ddaajelmer: I would you to explain it back to me, to make sure we are in sync05:18
jelmerddaa: Sure05:18
jelmerAs I understand it, you're looking into improving importd so it can recognize the special properties set on Subversion branches using bzr-svn.05:19
jelmerIs that correct?05:20
ddaathat's probably the shortest way of framing the issue05:20
jameshjelmer: my understanding is that the main feature we are after is for people working off branches we publish as vcs-imports on launchpad.net can land changes to SVN05:20
jelmerjamesh: Ok, that's quite a bit different05:21
jameshjelmer: the question is whether to do it by making your plugin handle our current vcs-imports code, or whether to change vcs-imports to be compatible with your code05:21
jelmerjamesh: I assume invalidating all the existing vcs-imports is not an option?05:21
jelmerfor example, by changing the revision id generation algorithm?05:22
jameshwe are discussing that.05:22
jelmerIf you haven't seen it yet, there's a short summary of the SVN properties bzr-svn sets on http://samba.org/~jelmer/bzr/mapping.txt05:22
ddaajelmer: I'd rather we not use your svn import system05:23
ddaait has nice properties, in that it does not require a central authority to work05:23
ddaabut it also has issues with referential integrities in the face of changes in the conversion code05:24
jelmerddaa: I'm not suggesting replacing importd by bzr-svn, but those are some of the properties importd will have to set05:24
jameshi.e. if you find a problem with your revision and file ID mapping and need to change it05:24
jelmerif you want to allow people to take an importd-generated branch and push it to subversion05:24
ddaalet me have a quick look, so do not talk _entirely_ out of my ass05:25
jelmerjamesh: There's a version number that's part of the generated revision id that I'll increment whenever I change the mapping code; that hasn't been necessary so far.05:26
jelmerjamesh: The revision ids are used for informational purposes by bzr-svn, they're not just random identifiers.05:27
ddaaokay05:27
ddaaAIUI there are two important processes in what your plugin does05:28
ddaathe first one is the "commit to svn" process. Its inputs are: 1. a bzr branch 2. a svn url. Its output is 1. a svn repo with a commit in it that includes some magical metadata.05:29
ddaathe second one is the "merge with svn" process. Its inputs are: 1. a bzr branch that was initially imported from svn, and may have a number of native bzr commits after the last imported revision 2. a svn url to a repository which includes a revision created by "commit to svn", and which may include some of the native commits on the bzr branch. Its output is a bzr working tree which is merged in a history sensitive way with the svn repo.05:32
jelmerYeah, I guess dividing it that way makes the most sense from a launchpad POV05:35
ddaaexcuse me, the guys are chatting around05:37
ddaaI'll move to some quieter place05:37
jelmerok05:38
ddaaback05:38
ddaawhat we would like to have in the end is the ability to:05:38
ddaa1. commit to svn from a bzr branch imported from svn, but with non-deterministic ids05:39
ddaato do that meaningfully, we might have to alter our import tool to provide the commit tool with some special metadata it needs to do its job05:40
ddaaa highly desirable property here, is that it should not be a requirement for _all_ the revisions in the history of the bzr branch to have this special metadata05:41
ddaabut that's only a secondary requirement05:42
ddaawe can throw away all the existing svn import, as a one-off event05:42
jelmerOk, that makes sense. 05:43
jelmerSo, in order to be able to do a commit, given a certain SVN branch and a Bazaar branch derived from that SVN branch, the bzr-svn plugin needs:05:43
jelmer* A common revision that contains a branch path, revision number, uuid and file ids that can be mapped back to paths in the Subversion repository05:46
jelmerAt the moment, the branch path, revision number and uuid are simply "stored" in the revision id, but I guess you could store them somewhere in Bazaar revision properties05:47
ddaa2. import svn into a bzr branch, that uses non-deterministic ids, sets the parent_ids in revision that were comitted using the bzr-svn plugin, and preserves the file-ids for files that were introduced on the svn repo using the bzr-svn plugin.05:47
jelmerddaa: I doubt 2 is very well feasible05:48
ddaajelmer: okay, so we could start storing that information into, say, a "svn-revision" revision property, and once a launchpad import has a single revision with this property, it could be used to branch off and commit back to svn?05:49
jelmerddaa: Yes, but if the revision ids are non-deterministic, there are a couple of weird problems that'll show up05:50
ddaaI'm all electronic ears.05:50
jelmermostly, if you have two bazaar branches created from different svn branches in the same repository05:51
ddaayes?05:51
jelmerthen, you merge branches/foo into trunk using bzr-svn05:52
jelmerlater on, you can still merge whatever branch was created by importd for branches/foo05:52
ddaaI sense trouble here, but can you spell out what the problem will be?05:53
ddaafrom the user perspective, what is the failure mode you are concerned with?05:54
jelmerbecause that revision (the same revision) has different revision ids05:56
jelmerthe fact that you will get the same history twice05:57
jelmerbut with different revision ids05:57
jelmerI'm not sure how much of a problem that is for you05:57
jelmerit's not a very big problem, it's just not very nice05:58
ddaadoes svn actually _tracks_ merge?05:58
jelmerwell, bzr-svn does05:59
jelmeror at least, the revision ids of the merged revisions are stored, the revision itself aren't05:59
jelmerSubversion 1.5 will actually track merges, and as it looks now, bzr-svn should be easily able to support that05:59
ddaajelmer: in the end, the problem you just mentioned only happens because there are two competing mappings from svn to bzr revision ids: the deterministic ones you produce, and the random ones importd produce.06:01
jelmerddaa: Sure, so I guess as long as you don't mix the two, there isn't much of a problem.06:01
jelmerddaa: I guess checkouts don't work on importd branches?06:02
ddaajelmer: I do not understand that question about checkouts.06:02
jelmerddaa: You can't have a branch bound to a vcs-import, right?06:03
jameshjelmer: well, checkouts fail for read-only bzr branches right now, independent of who made them06:03
ddaawhat jamesh said06:03
jameshah.  You mean so that commits go directly to svn?06:03
jelmerjamesh: Yes. There'll be no need for bzr-svn to support direct commits to svn for importd-branches06:04
jelmeronly push06:04
jelmerIf I understand the system correctly06:04
ddaathat sounds like it, except push is a bit too limited06:05
jameshthat's pretty much -- just get the delta committed to svn with the metadata needed to recognise the merge06:06
jelmerok, that should be doable if importd sets the right metadata in the revision properties06:06
ddaaI just realised something that your approach gives that importd cannot readily give06:07
jelmerwhat is it?06:07
ddaaUsing your system, to merge a bzr branch to svn, you do a checkout of svn, merge your external branch there, and commit.06:08
ddaayou have the opportunity to resolve any conflict at the merge point06:08
ddaaAnd once you have comitted, you immediately get the import branch that includes your merge.06:09
ddaaPutting importd in the loop to assign non-deterministic yet "canonical" revision ids force the workflow to become:06:09
ddaa1. do an atomic "merge and push" operation to the svn repository, that only accept clean merges 2. wait for importd to come around updating its branch to get the revision you just comitted06:11
ddaajelmer: is that correct, or do you see a way to avoid this loss in agility?06:11
ddaaoh! yes, there would be a way to avoid it!06:12
ddaaWhen doing the atomic "merge and push" thing, stick the "canonical bzr revision id" in a property of the svn revision.06:13
jelmerddaa: well, you'll always commit locally and push to svn - never commit directly to SVN, unless we add a 'commit-svn' command or something06:13
jelmerright, that used to be the original idea for push-as-merged. Ideally revision id aliases would be used for that06:14
ddaalet's stay away from revision ids aliases for now06:14
ddaaif abentley says "it confusing the heck out me", that means it's definitely too complicated for mere mortals to understand06:15
jelmerddaa: I don't think that was about revision id aliases - probably about file id aliases06:15
jelmerddaa: anyhow, how would you know what revision id importd is going to assign?06:16
ddaain that case, the revision id to use by importd would be encoded explicitly in the svn revision.06:16
jelmerthough that makes it deterministic06:17
ddaayeah your are right06:18
ddaawrong track06:18
ddaaI was on crack06:19
jelmerAre you opposed strongly to deterministic revision ids? Or mainly to breaking the existing revision ids?06:19
ddaaone minute06:20
ddaaOkay...06:26
ddaaI think your determinitic revision id approach is necessary to get something that people will want to use.06:26
ddaathe push-then-wait-for-launchpad is, IMO, not something people will realistically want to use06:27
ddaahowever, to use those deterministic revision ids, we need additional guarantees06:28
ddaathe way it currently works, we have a potentially open-ended number of namespace transitions that may affect all svn revisions06:29
ddaathat's not acceptable in that it makes users of svn branches pay for svn features they do not use06:29
ddaafor example, consider svn symlinks06:30
jelmerso, to summarize: you don't want a bug in the mapping code force all vcs-import users to ditch their existing branch copies?06:30
ddaayes06:31
ddaafixing the bug should only bump the revision id namespace of revisions whose ancestry contains at least one revision affected by the bugfix06:31
ddaanote that this is not limited to bugs, it also matter for new features06:32
ddaalike nested trees and stuff like that06:32
ddaaor file properties06:32
jelmerright06:32
ddaacan you deliver functionality in your plugin to provide that logic right away?06:33
jelmerThe easy way would be to just iterate over the branch and compare all the existing revisions with the ones on the SVN branch06:35
jelmer(where the ones on the SVN branch are created using code with the bug fixed)06:36
jelmerbut I guess it makes sense to provide an upgrade mechanism, much like the one for data formats in bzr at the moment06:36
ddaajelmer: I think there's more to it06:36
ddaaconsider the independent import06:36
jelmerindependent import using bzr-svn, you mean?06:37
ddaayes06:37
ddaain that case, you need to determine how many namespace bumps each revision needs to undergo in order to be compatible with other imports06:38
jelmerhmm06:38
jelmerback in a minute06:39
ddaaI'm starting to think that, maybe, it would be simpler (as in more robust) to actually go the git way, and use content-addressed revision ids.06:39
ddaajelmer: cool, I need some nicotin06:39
jelmerddaa: that's not really an option for bzr-svn as it requires you to fetch the full svn revision just to figure out what the revision id should be06:47
ddaathat fixes my main problem at least06:48
ddaawhen comitting to svn, you have the full revision handy, so you can readily hash it06:48
ddaaand importd is very good at sucking the data out of svn servers06:49
jameshddaa: do you hash on the data you preserve, or all the data SVN stores?06:49
ddaathe hash would have to be computed on what the bzr revision looks like, after being transformed by bzr-svn06:49
jameshso if later versions of bzr-svn preserve more data you'd get different hashes?06:50
ddaathe alternative is to guarantee that for all revision, we can reliably count the number of required namespace version bumps, using any bzr-svn version ever released 06:51
ddaawhich seems too strong a promise to make06:51
ddaajamesh: yes06:51
jelmerddaa: Keeping a hash is probably a good idea - I'm not sure whether it should be part of the revision id though06:52
jelmercan't we just rely on the revision sha1?06:52
ddaaunless you can come up with a good way to detect and recover from loss of referential integrity of revision id, I think we need to put the hash in the revision id.06:53
jelmerI simply can't put the hash in the revision id for bzr-svn, it's way too expensive06:53
ddaajelmer: what if launchpad provides you readily imported branches, and handles re-importing them now and then?06:54
ddaayou do need to suck in that data anyway when pulling or branching from a svn repo, dont' you?06:55
jelmerddaa: big loss of functionality - bzr-svn works stand-alone right now, and svn commits appear immediately in the bzr-svn plugin06:55
jelmerddaa: Yes, but if I just run 'bzr log' on a remote branch, that's only a couple of roundtrips now, even on a very big repository06:55
jelmerddaa: if you add a hash in the revision id, Branch.revision_history() will slow down big time06:56
ddaajelmer: can you propose a different way to reliably handle the requirement of limiting revision-id changes to those revision that are affected by changes in the conversion logic?06:57
ddaaSo far I see only two options:06:57
jelmerddaa: I hope so.. that would be the ideal solution, so I'm trying to think of something06:57
ddaa1. merge-and-push then wait for importd06:58
ddaa2. put a hash in the revision id06:58
ddaanone of them are ideal, so I'm certainly open to better ideas06:58
jelmerwhere (1) would be with a non-deterministic revision id, (2) with a deterministic revision id?06:58
ddaajelmer: yes06:58
ddaaI think log-directly-from-svn could be handled using a special revision id namespace07:00
ddaabecause then the actual contents of the revision do not really matter07:00
jelmerddaa: but how are you going to detect you're in a log-directly-from-svn?07:00
jelmerddaa: log isn't really special, it's just a bunch of get_revision() calls07:01
ddaamaybe have "bzr log svn://something" be interpreted as "bzr log svn+logonly://something"07:02
jelmerthat's just plain ugly :-)07:02
ddaayes07:02
ddaaWhat I'm looking for here, is something that would make the sabdfl happy.07:02
ddaaTo slightly rephrase what he told me: "we are not paying for the svn plugin because it's cool, but because we want some feature working"07:04
ddaaso, working trumps pretty07:04
jelmerWell, bzr-svn works very well without importd... (-:07:04
jelmer(1) and (2) both sortof defeat the purpose of bzr-svn, as (1) doesn't allow mixing of merges from bzr-svn and importd and (2) slows down bzr-svn too much07:05
ddaaThe reason I'm talking to you here, is that I do not think that in its current form it can work reliably enough for us.07:05
ddaabut maybe some less anal launchpad folks will disagree with me.07:05
ddaawhere reliability means, mainly, preserving referential integrity of revision ids and avoid breaking imports that are not affect by changes in the import logic07:07
ddaaFinally, I do not personally believe that having launchpad assign "canonical" bzr ids to imports from other vcs increases the value of launchpad in a useful way, but some people in the company seem not to share that conviction yet.07:09
jelmerWhy exactly can't it work reliably? I can see the issue with mapping changes and I am convinced that should be fixed, but I'd rather not cripple bzr-svn for it.07:09
ddaaBecause you have not told me yet how to fix that issue in another way.07:10
ddaaI'm totally open to suggestions.07:10
ddaaIf you need time to think about it, that's fine with me.07:11
ddaajamesh: anything you would like to add?07:11
jelmerWell, one of the original ideas behind bzr-svn (as it read on the wiki) was that it worked without depending on Launchpad. Another option would be to simply extract the commit code of bzr-svn and create a launchpad-svn-commit plugin without complicating the bzr-svn code.07:12
ddaaMh, which wiki?07:12
jelmerlaunchpad - page BzrRoundTrippingSvn, though I can no longer access that07:13
jelmerddaa: I'm still trying to think of a way to properly detect problems.07:13
jameshddaa: my feeling is that jelmer's system is the right general direction, but I agree that we need to address the future in the bzr <-> svn mapping spec07:14
ddaajelmer: the idea of the launchpad-svn-commit plugin is one of the option I see  now. But it's far from ideal.07:15
ddaain that this functionality will just suck07:15
jelmerddaa: If we can figure out some way to deal with upgrades but still have deterministic revision ids, that'd be by far the best solution imo.07:16
ddaajelmer: I agree with jamesh that your design goes in the right general direction07:16
ddaajelmer: how can we help in the figuring out?07:17
jelmerddaa: I'm not sure - I'm currently trying to generalize the problem a bit.07:18
ddaaI mean, how can we help in terms of gathering resources: like attention from core bzr devels working at canonical.07:21
jelmerddaa: Ah, in that sense. I think it would be a good idea to discuss this with Martin, Robert or John.07:22
ddaaI understand that you need to sit back and think about the issues, but when I report to the company, I want to be able to give a date when I will have some more information.07:22
jelmerddaa: Sure, I understand. Perhaps it's a good idea to have another discussion on, say, monday?07:24
ddaaif you expect to have some progress on that issue monday, great07:24
ddaabtw, I hereby invite you to the launchpad-bazaar meeting07:24
ddaaWhich is held at 1000UTC on monday in that very channel07:25
jelmerCool, I'll be there.07:26
ddaaWith SteveA, mpool, lifeless (usually), jamesh, spiv and myself.07:26
jelmerOk, I'm off to dinner. See you on monday.07:27
ddaajust for clarification, this meeting is intended to be a short discussion to assign tasks and resources07:27
ddaadesign discussion should take place outside of the meeting07:27
jelmerok07:27
ddaathank you for your time07:27
ddaahave a nice dinner07:27

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!