=== r0bby is now known as robbyoconnor === r0bby is now known as robbyoconnor [07:29] morning people! [07:32] * fullermd obviously isn't one ;p [07:33] you're a category to yourself perhaps fullermd? [07:37] I'm autoontological! [08:21] mgz, vila, jelmer, jml: I put up a branch to handle: bug# 1046284 [08:21] bug #1046284 [08:21] Launchpad bug 1046284 in Bazaar "TooManyConcurrentRequests when committing to lightweight checkout" [Undecided,In progress] https://launchpad.net/bugs/1046284 [08:21] In doing so, I'm starting a new branch for bug #1046697 [08:21] Launchpad bug 1046697 in Bazaar "missing integration tests of lightweight checkout and remote repository" [Medium,Triaged] https://launchpad.net/bugs/1046697 [08:21] Which so far has found at least 2 bugs in lightweight checkouts with remote repositories. [08:21] jam: cool :) [08:21] So I'm trying to decide how much to split this up. [08:22] Right now, I'm thinking to do 1 branch per small fix, and then have an overall integration branch that runs the full permutation tests. [08:22] That should leave us with easy to review small steps, and a test suite that ensures all those are correct when we're done. [08:22] vila,mgz,jelmer: Would you prefer I wait until I've fixed all the holes and have one major branch to review? [08:22] also, the initial fix is targeting 2.5 [08:23] however, all these cleanup fixes could just be done in bzr.dev if we feel that is a 'safer' way to do it. [08:23] Thoughts? [08:24] out of blue, I'd say: target a final branch with all permutations passing and do as many intermediate branches you feel are small enough to review ? So basically, you're already doing that, go for it ;) [08:24] now, if you hesitate about 2.5 vs dev: [08:25] if 2.5 is deployed on lp, you'll get bug reports soon enough and would need to fix those bugs anyway, so go for 2.5 [08:25] the alternative being rolling back from 2.5 on lp, I don't think you'll hesitate long ;) [08:26] I don't mind seeing small intermediate branches, even if there's a rollup that's what lands in the end [08:26] vila: so far they are all client side fixes [08:26] so while the initial bug is exposed by bzr-2.5 on the server, it is only the client that needs updating. [08:26] Anyway, I'm happy to improve our test coverage and fix the holes that we've had a long time. [08:26] They aren't strictly regressions. [08:27] But meh, targeting 2.5 is cheap and easy. [08:27] If it was "we should target 2.1" I might fuss a bit. [08:27] client only but uncovered because lp has been upgraded, so to me, that's still part of the 2.5 experience, even for the clients [08:28] if < 2.5 clients encounter the issue, well, they'll have to upgrade [08:28] vila: so for 'get_file_text' it won't impact anyone who is running <2.5 because they won't try the new rpc [08:29] for the other bugs... I've only uncovered 1, which is that "WT.bzrdir.sprout()" is broken if you have a RemoteBranch. [08:29] but I'm sure we've had it for a while. [08:29] pretty good then [08:29] And upgrading LP would have triggered it. [08:29] I'm assuming I'll run into more bugs, though, as part of updating the test suite [08:29] clarification: lp has been upgraded or not ? [08:30] vila: lp has been upgraded [08:30] we added 1 new rpc, which triggers the bug in a bzr-2.5.1+ client calling WT.get_file_text() [08:30] to which version ? lp:bzr/2.5 or still with lp specific additional fixes ? [08:30] in a lightweight checkout of a remote repo. [08:30] vila: have to ask jelmer, but I think just stock 2.5.1 [08:30] it's a pretty specific thing, that most people haven't done because the performance was terrible [08:31] now the performance isn't so terrible, but it breaks [08:33] mgz: but isn't it the recommended emacs setup ? [10:40] have we got docs for the new way of doing :policy = appendpath stuff? [10:40] an example locations.conf that sets the push_location of a lp branch sensibly would be nice [10:44] ah, is all under configuration-help [11:05] grmbl, xchat seems to remove highlighting indication after disconnects [11:05] jam: yes, 2.5.1 [11:08] bug 1046773 confuses me [11:08] Launchpad bug 1046773 in Bazaar "can't push into new project" [Undecided,New] https://launchpad.net/bugs/1046773 [11:08] he claims this is against launchpad as a smart server, but I fixed bug 722416 in 2.4b1 [11:08] Launchpad bug 722416 in Bazaar "Smart server transmits MemoryError as ('error', '')" [High,Fix released] https://launchpad.net/bugs/722416 [11:21] mgz: note that 'error' is any generic "We didn't know about this error in advanced". It may not be MemoryError. [11:21] mgz: also note that he is using 'bzr-2.5b1' so still a beta release. [11:22] I would ask him to update to bzr-2.5.1 and confirm that the bug still happens. [11:22] we could have changed an RPC verb between b1 and 2.5-final or something like that. [11:27] but as part of that fix, I made the smart server return the full error class name [11:27] which is all server side. [11:28] does launchpad actually keep bzr logs for smart server access? I can't find any. [11:31] hi [11:31] could somebody help with this stacktrace: [11:31] http://pastebin.ubuntu.com/1188779/ [11:31] maybe a known error? [11:31] mmrazik: so we recently upgraded bzr on Launchpad, such that it now closes the connection if you leave it idle for more than 5 minutes. [11:32] I think if you either upgrade bzrlib locally, it will auto-reconnect. [11:32] Or just reconnect yourself. [11:32] jam: thanks! let me try [11:32] I think sidnei was mentioning earlier that tarmac opens a connection, runs the test suite, and then tries to commi tit. [11:33] jam: yes. Thats this case as well [11:33] jam: do you know what version of bzrlib should I use? [11:33] the box is something old (?oineric) [11:37] mmrazik: I'm sure the connection retry logic is in bzr-2.5 [11:37] I thought there was an argument at the time that we were going to backport it to bzr-2.1 (stable series) [11:37] but I then was on rotation, so maybe that never happened? [11:37] (i don't see a release-notes indicating the retry logic was in any older stable release) [11:38] mgz, jelmer, vila: Do you remember what happened with the retry logic? Is my statement sound? (it is in 2.5, and we never backported it to a 2.1 release?) [11:38] jam: Yeah, I remember us backporting it to at least 2.2 too [11:38] but I'm not sure if we ever did a release with those changes [11:39] how can I reconnect manually? Can you point me to some docs? [11:39] jelmer: Yeah, I think I did the work, but we wanted to wait to see how it sorted out. [11:39] mmrazik: https://launchpad.net/~bzr/+archive/ppa can get you a newer bzr for oneiric [11:39] jam: thanks [11:39] * mmrazik tries [11:39] ppa:bzr/ppa I believe [11:39] mmrazik: for the manual reconnect, I would just say re-open the branch object. [11:39] But I don't know the internals of tarmac all that well. [11:40] I'll try the ppa first [11:40] ah, I think I know what is happening, and what we didn't expect. [11:40] the server is closing the ssh connection, and the ssh subprocess notices, cleans up and goes away [11:40] and then bzr tries to talk to the ssh process that isn't there anymore. [11:41] which is why we are getting EPIPE instead of ConnectionReset. [11:41] that explaing the EPIPE [11:41] yep [11:41] mmrazik: if you want to do a quick test, you might try forcing "BZR_SSH=paramiko" in the environment. [11:41] And see if that triggers ConnectionReset instead. [11:41] Just as a 'my hypothesis is sound' :) [11:42] then again, you may have credentials with openssh, and nothing will work with paramiko [11:42] what is paramiko btw? [11:42] I'm trying it right now... [11:44] mmrazik: paramiko is a python ssh implementation. [11:45] (we especially use it on Windows where openssh is likely not to be present.) [11:53] jam: with paramiko it seems to be stuck at this stage: [11:54] 26057kB 0kB/s | Revert phase:Apply phase:adding file 0/1 [12:04] mhm... the same stack trace with bzrlib 2.5.1 :-/ [12:07] mmrazik: still EPIPE? strange.as there shouldn't be a subprocess for paramiko. [12:07] ah, you mean with 2.5.1 [12:07] jam: yes, EPIPE with 2.5.1 [12:07] mgz, jelmer: https://code.launchpad.net/~jameinel/bzr/2.5-remote-wt-tests-1046697/+merge/123061 is a rollup of all the fixes for lightweight checkouts and a remote repository. [12:08] it ended up only 459lines, so you might just review that. [12:14] jam: if I have bzrlib.branch object, what can I do to re-connect [12:14] * mmrazik is trying to read the API but doesn't find anything [12:14] mmrazik: I think there are 2 things to try, depending on how comfortable you are hacking python code and testing it. [12:15] lets try :) [12:15] I think one issue is that we are using a socket_pair rather than a pipe to communicate to the subprocess, so we didn't expect EPIPE. [12:15] mmrazik: so to just reconnect manually,y ou can do "mybranch = mybranch.bzrdir.open_branch()" I think. [12:16] that would be at the tarmac level. [12:16] yep [12:16] just trying that [12:16] I need at least some quick hack to make the system work [12:18] at the bzrlib level, I would probably do something like: http://pastebin.ubuntu.com/1188861/ [12:18] I think the issue is that the retry logic is there, but it isn't treating EPIPE as a connectionreset failure. [12:20] mmrazik: I would guess the traceback is slightly different (the line numbers at least don't match up) can you paste the new traceback? [12:21] jam: http://pastebin.ubuntu.com/1188864/ [12:23] mmrazik: hmm... it looks like it is actually retrying at that point... [12:23] the failure is occuring at line 13 of: http://pastebin.ubuntu.com/1188866/ [12:23] note that we've caught a ConnectionReset and are retrying the request. [12:25] mmrazik: my wife is giving me the "I've been ready to leave for 15 minutes" look, so I have to go now, but I'm happy to work with you more tomorrow. [12:25] jam: thanks! [12:25] mgz, jelmer: ^^ if you can get a chance to follow up on this. We *might* need to rollback bzr-2.5 but hopefully we can do a client side fix instead. [12:25] jam: *nod* [12:50] ...I'm confused by the status of this reconnect bug, [12:50] it's paramiko only, or for any ssh connection? [12:50] mgz: I tried paramiko but it didn't really help. The stack trace went away but tarmac "hang" at certain stage [12:51] so its any ssh connection [12:51] I'm just trying paramiko with the newest bzrlib [12:51] to see if there is a change === deryck is now known as deryck[lunch] === deryck[lunch] is now known as deryck === yofel_ is now known as yofel