[00:09] Hmm [00:09] I was going to push 2.3b2 to the beta-ppa [00:10] But now I think it would be better if I first figured out how to make bzr selftest run as part of the build [00:10] But I'm too sleepy for that, so later. [00:15] mgz: around? [00:32] poolie: I've been thinking about the underlying problem [00:32] poolie: and I've come up with two repeating problems [00:33] ok [00:33] 1) we have no initial idea if we are trying a write or read operation [00:33] 2) we don't have a clear way to ask "does this exist" without poking the filesystem [00:33] by removing the FileNotFound errors [00:33] it opened a world of hurt [00:34] which stopped being able to push branches [00:34] and I can't think of a clean way to get errors out of translatePath [00:34] with lead me to think that we should have an earlier check [00:34] which can have better "smarts" [00:35] I wish we had some smart command that allowed us to say: I want branch "X" for writing [00:35] or I want branch "Y" for reading [00:35] although even that probably isn't enough [00:35] * thumper is thinking stacking [00:35] we could then do our smart branch lookup [00:35] with clean semantics early on [00:36] i agree specifying that earlier would be good [00:36] it would also let us give a better message on http [00:36] * thumper nods [00:36] what are the contracts and constraints of translatePath ? [00:36] lifeless: well... [00:37] lifeless: probably best talked through if you're willing [00:42] what do you mean by "without poking the filesystem"? [00:44] jelmer: are you around atm? [00:44] peitschie: yep [00:45] jelmer: cool. a quick query, I'm recreating our svn repo from scratch, but want to preserve the history from the existing bzr branches.... is fetch ghosts likely to reintroduce any history issues? [00:46] jelmer: or is there a better way to get the bzr history linked up in the new repository? theres quite a few branchs (~400 i think...) so manually is a little bit painful :) [00:48] jelmer: a bit more context might help for you... specifically, we've encountered bug #485601 again... I didn't create the new repo last time as suggested so am doing it now :) [00:48] Launchpad bug 485601 in Bazaar "missing chk node(s) for id_to_entry maps (affected: 2, heat: 13)" [Medium,Incomplete] https://launchpad.net/bugs/485601 [00:52] jelmer: argh... realised that missed some vital joining bits as well! What I *mean* to say, is I am creating a new bzr shared repo from the existing svn trunk [00:52] peitschie: yeah, fetch ghosts should do the right thing in that case [00:53] jelmer: cool! that will make this a lot less painful than I feared [00:54] thumper: i'd like to see a bug saying what the behaviour ought to be case by case [00:55] jelmer: oh... that was a fun experience [00:55] jelmer: I can't fetch ghosts... as this encounters bug #485601 again [00:55] Launchpad bug 485601 in Bazaar "missing chk node(s) for id_to_entry maps (affected: 2, heat: 13)" [Medium,Incomplete] https://launchpad.net/bugs/485601 [00:55] :-( [00:55] i suppose that makes sense... lol... now just gotta find a way to fool things... [01:00] hmm... there seems to be a specific checkin that has poisoned the broken repo [01:00] as soon as I attempt to pull any branch across that references this broken rev, things go dodgey [01:01] awww... i scared jelmer off :( [01:01] jelmer: if I can hunt down the exact svn checkin that causes this rev error... would that be of any assisstance with the missing chk nodes bug? [01:03] peitschie: I'm not sure - didn't somebody (you?) already post a script for reproducing the chk node bug? [01:03] Yhea - Productive night for me. - Goodnight all... [01:03] jelmer: I haven't been able to yet. I posted some more thoughts about it... but I keep struggling to be able to reproduce [01:03] g'night Gary [01:05] jelmer: i'm pretty much shooting in the dark with the latest one, as I spent ages the other night trying various permutations to try and get the crash. I've posted up on that bug the exact workflow we use essentially... but haven't been able to narrow down why it works 99% of the time [01:05] jelmer: bzr-svn is frustratingly stable sometimes :D [01:07] peitschie: a way to reproduce the chk node bug or even a public repository that demonstrates the issue would be a big help. [01:08] peitschie: unfortunately this is a combination of a bzr-svn and a bzr bug; bzr (imho at least) shouldn't be giving an exception this low in the stack, it should give a sane exception when called with invalid parameters. [01:08] peitschie: and of course bzr-svn is at fault for calling it with invalid parameters in the first place. [01:08] jelmer: i know! (programming is my day job too... without reproduction steps it's usually a lost cause) problem is I don't know what I'm looking for really. I can verify which part of the check is failing on the insert, and have dabbled with dropping break-points in various spots to see the stack... but my knowledge of bzr is lacking heavily [01:10] jelmer: that would correlate with my impressions too... this is a joint bug of some type. I can make the exception throw out the chk node it's complaining about... but don't know what to do with it from there. That is why I was wondering if potentially identifying the file/commit/? that this is related too and comparing this to the svn properties or similar might give some hints? [01:11] jelmer: of course... i'm still hopelessly unbazaared so have been lurking here for help [01:14] it should be possible to convert the chk node back to the related file id [01:15] I need to get some sleep though.. I'm usually around during Australian mornings, or perhaps one of the other bzr devs can help debug. [01:17] jelmer: g'night! [01:17] jelmer: fair enough :). Thanks for your help! sleep well [01:17] Hmm, I suppose my chk-used-by hack could be extended to look at more than just root keys. [01:17] spiv: hi and goodnight (-: [01:17] :) [01:18] spiv: would you have some time to give me a hand again? I've recreated a clean repo from svn trunk, and am trying to branch history across to the new repo... but keep hitting the same chk nodes issue [01:24] peitschie: hmm, what can I do though? Extend chk-used-by to look at more than just root nodes? [01:25] spiv: thas pretty much all I was hoping for if you could :) [01:26] oh... wait... thats a little cool [01:26] if i fetch the right revision set, I get "Cannot add revision(s) to repository: missing referenced chk root keys" [01:28] peitschie: ah! [01:28] peitschie: that's good :) [01:28] spiv: if I branched an svn branch that suffered from bug #522637... would bazaar automatically recover from that bug on the way through the branching? or will I then need to reconcile the repo afterwards...? [01:28] Launchpad bug 522637 in Launchpad Bazaar Integration "BzrCheckError: Cannot add revision(s) to repository: missing referenced chk root keys (affected: 16, heat: 89)" [Low,Triaged] https://launchpad.net/bugs/522637 [01:29] peitschie: 'bzr branch' won't usually automatically correct that [01:29] peitschie: but is the source branch an svn branch? [01:31] spiv: yes. The "Central" source control is svn, so that has trunk. So what I did was branch straight from that trunk again. I'm wondering though if the revision data bzr is finding in there is incorrect tho... and so I am essentially creating a freshly broken repository [01:31] peitschie: ok, that's very interesting [01:31] spiv: running the chk-used-by i have now gotten 3 revisions in the broken one that use it [01:31] peitschie: because AFAIK bzr-svn is causing CHK records to be created on the fly (because the underlying SVN repo doesn't use that) [01:32] peitschie: which version of bzr itself are you using? [01:32] hi spiv [01:33] spiv: we only just upgraded to 2.2.1 in the last few weeks. We've been running on 2.0 & 2.1 for the last 3ish months.... with around 3000 commits to svn in that time... I think roughly 9000 commits to the bzr repo [01:34] 2.2.1 is new enough to have the fix to 522637, so it shouldn't be generating non-canonical CHKs that cause the error you just saw. [01:35] spiv: that is correct... but I suspect the bug happened prior to 2.2.1 being released. Looking at this stuff, the bug happened on a commit to svn trunk on sept 1st [01:36] spiv: what I mean by that is the missing sha1 is referenced by that commit at the earliest [01:37] spiv: which happened to be a commit to svn directly, so the network repo would have gotten second hand info about that commit when it updated from svn.... mayb a non-canonical chk was created and inserted along side that revision? [01:37] peitschie: hmm [01:37] I wouldn't have thought that the SVN repo would be storing references to CHKs at all, though [01:38] spiv: let me go look :) [01:38] Because that's an implementation detail of bzr's 2a format [01:38] And not something that would make sense to store in SVN, because it has its own mechanisms already to store trees of files. [01:38] i know bzr-svn stores a bunch of properties along with the checkin [01:39] from previous glances it looks like bzr-svn stores enough information to be able to track base revisions and such [01:41] Right [01:41] The CHK stuff is basically just a storage optimisation, though [01:42] It's possible to fully represent the data bzr records without it (just it will tend to be less efficient to do so) [01:43] FWIW grepping the bzr-svn source I don't see any obvious sign that it would ever store CHK info in a SVN repository. [01:43] ahh k [01:43] well... the data stored against the rev is things like: [01:43] slidetraytemperature-20100901004344-1c1ttvdesj7oryzv-1 [01:43] 13860@464e02c1-a7d9-0310-851b-f51ca712b586:products%2FBondAdvance%2Ftrunk%2FBond%2FInstrument%2FInstrumentControl%2FController%2FMessageAdaptors%2FHeatersAdaptor.cs [01:44] ahh [01:44] wait [01:44] so it was merged into svn eventually [01:44] but the revision was directly committed to bzr first [01:44] then svn just had the merge committed [01:46] it looks like the referenced chk root key was introduced as part of a merge [01:47] that is... the first time this referenced sha appears is when i merged from svntrunk to my feature branch [01:48] Ok, and the bzr repo you are branching into is the same one that has this other bzr branch that has been merged into svntrunk? [01:48] yep [01:49] essentially... we have svntrunk, then a network repo that also has a copy of svn trunk + all feature branches [01:49] And this bzr data pre-dates you upgrading to bzr 2.2.1? [01:49] then each dev has a local repo with a bound branch to their feature [01:49] yep... this data pre-dates 2.2.1 [01:49] If so, it's fairly likely that following the repair instructions on #522637 will fix it. [01:49] I did run the repair tool for that bug tho [01:50] so this is the repo after that tool has been run across it [01:50] and at this point, if I recreate the local devs repo, they can commit to their feature only until someone checks into svntrunk [01:51] at this point, the dev that committed to svntrunk can continue to commit as per normal, but every other dev hits the missiing chk nodes bug [01:51] (to recreate teh local dev repo, I simply branch the NETWORK bazaar version of svntrunk into a new repository, so the dev gets all the bzr history with their repo) [01:52] Did the [01:52] repair tool log any lines like "Non-canonical CHK map for ..." in your ~/.bzr.log? [01:53] Also, could you elaborate on "at this point" a bit more? [01:54] Each dev has their own bzr repo, I assume? [01:55] (i didn't connect that was ur tool btw. i do appreciate all your work here... i'm in awe of the tool you guys have built here) [01:55] But as soon as they pull/merge a revision from svntrunk that was committed by another dev using bzr-svn they encounter this bug? [01:56] so.. the setup is as you said [01:56] And precisely on which operation do they get an error? [01:56] each dev has their own repo, but the feature branchs are actually bound branches that point to the network one (so we keep things backed up in case an individual pc dies) [01:57] (ugh, my grammar is terrible!) [01:57] *usually* the dev will hit this bug when they attempt to merge & commit changes from svn trunk into their feature branch [01:57] e.g., bzr merge ..\svntrunk && bzr commit -m "Merging trunk" [01:58] it will then blow up with bug #522637 at this point, and refuse to commit [01:58] Launchpad bug 522637 in Launchpad Bazaar Integration "BzrCheckError: Cannot add revision(s) to repository: missing referenced chk root keys (affected: 16, heat: 89)" [Low,Triaged] https://launchpad.net/bugs/522637 [01:58] So the merge succeeds, but the commit fails? [01:58] yep [01:58] That's extra bizarre :) [01:58] they can then simply throw away the commit, and keep working as normal and committing... but can never get changes in from svn trunk [01:58] And they are definitely using 2.2.1? [01:58] i never realised how close bizarre and bazaar sounds until ppl say that all the time in real life lol [01:59] they are now... and the same issue still occurs [01:59] Hmm. [02:00] What if they run the repair tool (on their local repo *and* the bound repo) between the merge and the commit? [02:00] (And does the repair tool log anything about non-canonical chk maps in ~/.bzr.log?) [02:01] due to how long the repair tool takes I haven't tried that [02:01] i checked my logs and the first run of the tool has been lost unfortunately [02:01] Yeah, I understand :( [02:02] i was hoping that once the network repo was fixed, each dev then only needs ~30mins to get their working sets back up again [02:02] hmm [02:02] so it appears that either way, teh reconcile has still left this revision hanging in the network repo that causes the missing chk root keys [02:03] This is sounding very much like 522637, though, or something closely related. [02:03] When you say "network repo", you mean the repo that devs have their branches bound to? Or the SVN repo? [02:03] the one their branches are bound to [02:04] oh wait... i meant to say way prior... the merge works, but the commit actually blows up with bug #485601... not the one i had there [02:04] Launchpad bug 485601 in Bazaar "missing chk node(s) for id_to_entry maps (affected: 2, heat: 13)" [Medium,Incomplete] https://launchpad.net/bugs/485601 [02:05] *but* if I attempt to branch the networked version of svntrunk (this is all bzr only... not svn interaction), I get bug #522637 [02:05] Launchpad bug 522637 in Launchpad Bazaar Integration "BzrCheckError: Cannot add revision(s) to repository: missing referenced chk root keys (affected: 16, heat: 89)" [Low,Triaged] https://launchpad.net/bugs/522637 [02:06] Hmm. [02:07] if it'd help i can draw up a pretty diagram and instructions lol... it is a slightly convoluted workflow to explain :) [02:08] Hmm. The distinction between those two errors might not be very important: one is that the root node of a CHK map is missing, the other is that a non-root node is missing. [02:08] It's otherwise a sanity check at the same point in the code. [02:09] thats what i started suspecting. dont know if that helps anything much tho [02:09] Either way, it suggests that at some point CHK maps are either not being generated correctly (i.e. the wrong nodes), or generated incompletely, or that we are losing nodes during a fetch. [02:10] All 3 options should be equally impossible :) [02:10] The slightly different errors are certainly interesting, and I think a useful hint. [02:11] But I'm pretty sure they are a manifestation of the same root cause. [02:12] What's weird is that these checks are run on every write to the repository. [02:12] i'm looking at the earliest rev to reference these and seeing where it came from [02:12] So if the merge works, and it does, then the data being pulled across by merge is apparently complete. [02:13] mm... it's not a big merge either [02:13] (Not necessarily *correct*, bug 522637 can create data that passes this check but causes trouble for new data generated against the bad data) [02:14] Launchpad bug 522637 in Launchpad Bazaar Integration "BzrCheckError: Cannot add revision(s) to repository: missing referenced chk root keys (affected: 16, heat: 89)" [Low,Triaged] https://launchpad.net/bugs/522637 [02:14] has 2 revs... 1 rev is a single binary file, second rev is a few sources + a bunch of binary files [02:14] Only file modifications [02:14] ? [02:14] 1st is a file modification, 2nd are all new files [02:14] Hmm. [02:15] Ok, at this point I'd really like to know that it's not another case of 522637. [02:16] I can't see how, but obviously some assumptions need questioning at this point :) [02:16] https://code.edge.launchpad.net/~jameinel/+junk/bzr-check-chk is a plugin that will be useful [02:16] getting it now... [02:17] You can give it a range of revisions, and it can tell you if they are broken in a way that reconcile --canonicalize-chks will repair [02:17] range... it works by branch or repo? [02:17] Branch [02:18] So you can use it to relatively cheaply check if e.g. 'bzr merge' from svntrunk has this corruption. [02:20] i'm running it over the networked repo's trunk at the moment [02:20] then i'll do it in the branch it first appeared in [02:20] In fact, it seems to me that if 'bzr merge ..\svntrunk && bzr commit' fails on commit but not merge that it's likely that either the new revisions fetched by 'merge' are broken, or the most recent parents of those are, or one of the recent parents of the current tip you are trying to commit on are. [02:21] that would reflect how it appears to behave [02:21] Note that tool is actually slower than 'bzr reconcile --canonicalize-chks' per revision, so it'll only be faster to use it if you give a range of revisions rather than letting it check the whole repo. [02:22] if i then delete the individual dev's repo, and create a brand new clone from the network repo again, the problem goes away... until the next bzr checkin lands in svn that both the dev and the remote repo retrieve [02:23] ok... check-chk on trunk hasn't turned up anything for the revision range where the affected root key was merged in [02:23] i've widened the range to work earlier slightly and am running again [02:23] Basically at this point I'd really like narrow down where the corruption is being introduced. [02:24] Because so far in isolation all the individual pieces sound fine. [02:25] ahh the fun of complex interactions :) [02:25] And e.g. glancing at the bzr-svn source it seems to be fetching revisions from SVN to bzr in a totally reasonable way, so I wouldn't tend to suspect it. [02:25] Except that the only people reporting this bug are using bzr-svn... [02:26] i wonder if it is something related to two different repos with slightly different history available coming up with a slightly different revision somehow [02:27] svn would b the key then, because of it's effect on ghosting various parts of teh revision [02:27] under normal bzr operation, the situation doesn't seem quite as common as for bzr-svn [02:27] (the ghosting situation that is :)) [02:28] Possibly, yes. I mean, in principle that should be fine! But obviously *something* isn't fine. [02:30] on a side note, are you aware of any tools for generating a bunch of changes for a branch? I've been considering throwing together a fuzzer of sorts so that I can set up a repository and essentially simulate a whole bunch of changes being made... working on the theory that i will eventually hit this again [02:33] No, not really. [02:33] I mean, we have a perfectly decent API ;) [02:33] oh... that check-chks doesn't work with a branch that has ghosts by the way [02:33] i didn't want to go through the api for fear that it was bzr-svn's use of said api that was doing something "unusual" [02:34] Oh, interesting. I guess I can see that. [02:34] well bzr-svn is an implementation of the api [02:34] so its a bit special in that regard [02:35] i can't run check-chk on the clean checkout of svntrunk due to the ghost thing... is there any easy way we can fix that? [02:36] peitschie: specify revisions one by one? [02:36] lifeless: thats wat was one of the questions... one of teh bugs is only affecting bzr-svn users, which suggests bzr-svn's usage is a little different from standard bzr somehow... likely in some very subtle way... hence why i'm thinking of fuzzing via the file system directly rather than the bzr api [02:37] peitschie: I think you misunderstand me. [02:37] peitschie: bzr-svn is an *implementation* of the API [02:37] peitschie: its not a consumer of it in the usual sense, at all. [02:38] peitschie: this is what makes bzr-svn glue in so nicely to the core [02:38] lifeless: ahh. sorry :S. gotchya [02:40] spiv: so far it hasn't reported any issues with either the new or old repo [02:42] Hmm, I wonder if bzr-svn isn't roundtripping inventories perfectly. [02:44] is there any way to verify that you can think of? [02:46] * spiv adds a note to the bug while he's thinking of it, so jelmer will see it [02:46] Basically, compare the CHK root keys before and after roundtripping. [02:47] hmm [02:47] i can probably try and branch the revisions just prior to this issue from the broken dir [02:47] i.e. make a revision in bzr repo A, commit/push it to the SVN repo, and fetch it into bzr repo B. I suppose if you're seeing this bug, and this theory is right, you already have examples of this. [02:48] Then use the bzr APIs to tell you what the CHK root keys for that revision's inventory are in A vs. B. They are supposed to be the same. [02:48] the challenge is finding the exact combination that causes it to break :). things work the large majority of the time... so it is a specific sequence of somethings that does it [02:48] You can probably see how to adapt bzr-check-chk or similar to just print the key for a given revision. [02:49] yep [02:50] (If this theory is right, then it will pass the 'canonical-form' check, because the bug isn't the layout of the CHK, it's that actual inventory contents for a given revision ID are not the same in the two repos.) [02:56] (all check-chks came back without any issues btw) [02:56] Ok, that's good to know. Thanks for that. [03:05] * spiv -> lunch [03:32] spiv: let me know wen ur back... i could use a few more pointers :) [04:01] spiv: i found the dodgey rev... now i need to get the details out of it somehow [04:04] peitschie: back [04:04] spiv: wb :) [04:04] peitschie: oh nice [04:04] Tell me more about this dodgy rev! [04:04] spiv: it's rather ... unexpected. the key revision is a pure bzr rev that has stacked on an svn rev [04:05] i don't know how to get other debug info out of it yet... i'm quickly testing checking out these revs against trunk [04:05] once this single bzr rev gets into a branch, every branch from that point onwards shows the missing chk node entry bug [04:05] A pure bzr rev in the same repo it was created it? Or a pure bzr rev that has travelled via the svn repo? [04:05] (when i try and branch it into my clean repo) [04:06] pure bzr [04:06] so the dev branched off svn trunk (a rev the repo already had), then performed a checkin to their feature branch (pure bzr) [04:06] i can pull the svn rev into the clean repo [04:06] i cant pull the dev checkin to the repo [04:07] "pull the svn rev into the clean repo" -- pull from svn -> bzr, or from the first bzr repo? [04:07] first bzr => second (new clean from svntrunk) bzr repo [04:08] Ok. I'd definitely like to know if the chk root keys for that rev in both repos match. [04:09] And I assume both the bzr rev and the rev from svn trunk both pass check-chk? [04:10] checking now [04:11] these revs were way outside the range i was checking before [04:11] back in early june sometime [04:12] What do the two revs look like in terms of changes? Modifies, adds, deletes? [04:12] the svn rev just prior is adding 2 files [04:13] the first bzr checkin after is deletes & modifications [04:13] check-chks says nothing special for those two [04:20] im currently branching at various trunk revs to see what happens once that revision lands back into trunk [04:21] ag... duh.. thats not gonna help because the new repo has all trunk revs >.< [04:21] ok... whats the best way to get all the info we can about these two revisions that cause the problem? [04:23] Modify bzr-check-chk to print the root keys [04:24] (even if they pass the check) [04:24] they pass the check without any issues [04:24] and then use that to make sure the root keys for those revs are the same in both repos [04:24] ahh... thats gonna b fun [04:25] the second commit is only present in one repo... i can't get it in the other [04:25] Hmm, yeah. Well, just check that one. [04:25] I think it will match, but I'd like to check my assumptions :) [04:31] sorry it's taking so long [04:31] i've dropped pdb into ther and am trying to find what i want on the inventory :) [04:32] inv.id_to_entry.key() and inv.parent_id_basename_to_file_id.key() [04:32] thnx! [04:33] I don't suppose you're able to send me a copy of the repo that exhibits the problem? [04:34] its 1.2Gb... so it's a little difficult [04:35] oh... thats not so good i suspect [04:35] shas for the svn checkin in question are different [04:37] Hi [04:37] I'm getting trouble similar to https://bugs.launchpad.net/bzr/+bug/377261 [04:37] Launchpad bug 377261 in Bazaar ""AssertionError: name u'COMPILING' already in parent" in _generate_inventory when running "bzr update" (affected: 5, heat: 21)" [High,Confirmed] [04:38] The difference is that I got different stack backtrace than that bug [04:38] Should I file another bug? [04:39] echo-area: probably a good idea. It is easier to mark a bug as a duplicate than split a bug out :). Make sure you mention the other bug in it as well to make it easier for whoever is checking to see [04:39] peitschie: Okay, I'll do that. Thanks [04:39] peitschie: ah, so we are getting close after all! [04:40] spiv: it is starting to look that way a bit. wat next? [04:40] peitschie: that probably does explain why the next revision that depends on it can't be fetched [04:41] peitschie: if you don't mind sharing some of your data, [04:41] peitschie: send me the output of print inv.id_to_entry._dump_tree(True) and inv.parent_id_basename_to_file_id._dump_tree(True) for both [04:42] spiv: i can probably talk to people.. I work at a commercial company so things will need to be a little careful. Whats the easiest way for you to get 1.2Gb...? [04:42] spiv: even better :) [04:42] spiv: so you want that of the rev in question correct? [04:42] Yes, and from both repos [04:42] The output will, I hope, be different :) [04:46] And actually, it'd be great if you can send me the output for the parent revision (or revisions?) of that rev-from-svn too. [04:46] yep... so you want the rev that breaks things, and it's parent rev in both sides? [04:47] is 7zip ok for the file? It's way smaller than zip... [04:50] I *think* so. bzip2 is also good. [04:50] spiv, there is a 7z in ubuntu [04:51] spiv: I'm attempting to send it now... i'm behind a proxy tho, so if this fails i can email it [04:52] email is fine, andrew.bennetts at canonical.com [04:52] My IRC client says 'DCC unknown ctcp XXXX from peitschie ...' [04:52] So I guess that's not going to work :) [04:54] yups :)... emailing now [04:54] it's sent [04:54] Thanks! [04:55] now heres to hoping it's useful in some way... [05:01] Received ok, I think this is going to be helpful [05:01] What is r4686 is the rev from svn? [05:02] thats the one that magically starts breaking things [05:03] so i have a checkin on a branch just past that point, which i cannot branch into my "clean" bzr repo [05:03] that reminds me, do you want the info from the checkin just past 4686 as well? [05:04] So interestingly 4685 differs in both revs too [05:05] i've been chasing those back... [05:05] er, in both *repos* [05:05] they were differening back to the first bzr checkin made [05:05] so up until the first checkin, it's all identical. once the first checkin was made, the files immediately start differing [05:06] The inventories have the same files and directories, with the same IDs, with the same contents and the same lengths and the same executable bits... [05:06] *but* some of those files and directories have different revision IDs [05:06] i thought so [05:07] Which is the inconsistency that is causing all your trouble.s [05:07] So, how precisely were these two repositories created? [05:08] so... repo 1 (broken) was created with a check out of the entire svntrunk [05:08] that then served as our network repo since then (early june) [05:08] repo 2 was created by rechecking svntrunk out a few days ago... that has had no other use at this point [05:09] as soon as the first bzr checkin is made to the repo, the bzr info that i'm dumping changes [05:10] peitschie: hmm! [05:11] So adding a new bzr revision to the SVN repo is causing future bzr checkouts from that repo to get different results for old revisions? [05:11] it is looking like it! [05:11] i am wondering... we got bitten by https://bugs.launchpad.net/bzr-svn/+bug/587819 ages ago [05:11] Launchpad bug 587819 in Bazaar Subversion Plugin "race condition when doing a bzr commit to an svn repo (dup-of: 515850)" [Undecided,New] [05:11] Launchpad bug 515850 in Bazaar Subversion Plugin "Revisions added while a commit is happening are ignored (affected: 4, heat: 23)" [High,Triaged] [05:12] i had to manually delete all bzr revision properties from the svn history [05:12] i wonder if someone forgot to delete their svn cache prior to a checkin... [05:12] and if that might have poisoned the stack [05:14] peitschie: ooh, sounds messy! [05:15] peitschie: that sounds plausible [05:15] peitschie: I'm out of my depth on that degree of SVN-fu, though. [05:15] peitschie: I think at this point we need jelmer for further diagnosis [05:15] spiv: i suspected we were hitting that point as well :) [05:16] I've added some more comments to the bug. We're definitely much much closer now, thank you very much for your persistence. [05:16] And patience :) [05:16] it's a pleasure :).. i like working with you guys [05:16] with those dumps that I sent... I dont mind if you pass them out to a dev or too.. just make sure they don't get too far out of ur sight please :) [05:17] Ok, I'll pass them onto jelmer as well [05:17] You should probably add a bug comment with the suspicions about svn caches and deleting old bzr rev props [05:19] yep... will do [05:19] if it is related to that... that took a scarily long time to bite :S [05:37] spiv: wb :)... another quick query... is it worth me doing a check-out of svn using an older version of bzr (2.0 or 2.1) to see if that is different too? [05:38] Hmm, I don't think so. But then I still don't quite know what's going wrong, so what would I know? :) [05:39] hehe... i know that feeling :S [05:39] i'll hold off then and see how far jelmer can get on wat we've given him [05:39] thanks again for all your help digging that out... It would have taken me all week to do that without your guidance! [05:43] If my own experience with these bugs is any guide, longer probably ;) [05:44] aint that the truth... it scares my poor little mind thinking about it :S === IBZ is now known as jfroy [06:23] lifeless: I'm around now if you're still after me. [06:24] mgz: yeah, you fixed timing data from parallel stuff [06:24] I was wondering if any of that was still in bzrlib, or if its all in testtools nows [06:25] it's moved to testtools basically. [06:25] https://code.launchpad.net/~gz/bzr/use_testtools_timings_625594/+merge/36784 [06:25] that's the bzr mp that does it. [06:29] thanks [07:06] hi all ! [07:07] poolie: did you get an email for the shortcuts failures ? Can I have a copy ? [07:08] vila: yes, lp:~gz/bzr/escape_selftest_console_output_633216 depends on a non-broken-with-unicode testtools version [07:08] poolie: regarding the http tests, the code is not called because the http client coalesce the offsets. The tests need to be tweaked to avoid that, but doing so led to a hang when I briefly tried yesterday. Why it hasn't been noticed before is... unfortunate :-/ [07:08] and I'd like to see the failure spiv got with it as well. [07:09] mgz: yup, hence my ping in the review comments [07:09] spiv: ^ [07:10] poolie: and to avoid problems, please send this mail to my canonical address [07:11] poolie: I reported the problem to my ISP but I'm not holding my breath, *one* postmaster for millions of users to handle false positive spams... hmmm, does it blend^W scale ? [07:11] should make pqm post a summary to the mp [07:12] hi vila, i did get a failure mail [07:12] vila: sent [07:12] mgz we should use tarmac [07:12] or that. [07:12] vila, i'm also reviewing your config mp [07:12] or, at least, work out why not to use it [07:13] vila, well, it's nice we noticed now [07:14] poolie: noticed what ? [07:14] noticed the http test wasn't reached [07:16] ha, yes, we may just get rid of the tests, they try to ensure a pretty unlikely regression and the hang may be a real bug in the test server [07:25] poolie: if you can't "work out why not to use it" I'd be happy to land it and work on the *next* steps ;) [07:28] poolie: ok, got the PQM failures mail. That's insane, what the hell is different on the pqm instance... [07:33] vila, i meant either use Tarmac or get a good idea why not [07:33] but maybe not today [07:34] poolie: did Tarmac allows running the test suite before landing ? [07:34] it should [07:34] i haven't tried this week [07:34] ok [07:34] i mean i haven't tried yet; i hope to try this week or next week before UDS [07:35] have you touched fastimport at all? [07:46] never [07:47] hmm, at least nothing I can recall, I may have fixed some test failures though ;) [07:57] PQM has code to comment on the MP [07:58] and rockstar has overhauled the API - a small tweak to use the new API next week, and it should be usable. [07:58] using tarmac would be good too [07:59] poolie: I've merged your testscenarios patch; it needed a little tlc to be up to your normal standards, so I did that as I merged. [08:00]