[01:05] vila, jam, lifeless: IIRC my change to --parallel was to make it allocate by round-robin [01:13] hi spiv === wallyworld is now known as Guest64579 [05:27] I'm curious what might be done to help bzr take advantage of multi-core systems. [05:27] I guess Python is a serious roadblock in here :/ [05:27] Does bzr even doing anything very parallelizable? [05:27] Other than the test suite? :P [05:28] what needs to be [05:28] paralleized [05:29] honestly [05:29] * AfC is watching a bzr process at "fetching revisions 1566/264178" at about 20 per second on one core, and watching three other cores of this server idle. [05:30] Gonna be a while, this one. [05:30] thats probably because [05:30] its limited by your download speed [05:30] AuroraBorealis_: its local [05:30] not because its single threaded. [05:30] AfC: And how exactly would you multithread that? [05:30] The system is not in iowait at all. [05:30] ive copied the bazaar source code which is like 37000 revisions [05:30] in seconds... [05:31] It's a quad Xeon. It's not a slow system. Anyway [05:31] Peng: that is, of course, a fascinating question [05:31] Peng: I don't know how much the Python runtime is an obstacle here. [05:32] Peng: but, on e.g. import loads, mapping any one revision would conceivably be independent of mapping any other revision. [05:32] python probably is not the obstacle [05:32] I imagine at some point there is the whole "sequence adjacent revision texts" thing [05:33] branching the bazaar source code which is at 38,000 revisions took less than a minute [05:33] i feel thats acceptable [05:33] shared or non-shared repo? [05:34] AuroraBorealis_: if it's CPU bound, then yes, the Python runtime's global locks will be an obstacle (if threading is the approach one uses to parallelism). [05:34] 6000 down. Only 258 thousand to go. [05:35] welll, if it's cpu-bound /and/ parallelisable :) [05:35] bob2: heh [05:35] dunno why its taking so long for you. [05:35] what are you branching [05:35] Linux [05:35] it's 10x as many revs as bzr [05:35] AuroraBorealis_, one minute to branch into a new unshared repository? [05:35] wait [05:35] i have it locally [05:35] so yes [05:35] well i didnt put it into a repo [05:36] and where are you getting linux in a bazaar repo [05:36] it uses git [05:37] lots of people import lots of things [05:37] i mean going from git->bzr sounds like it could take a while [05:37] Interesting data points: getting the kernel from github was 15 minutes. Getting the vcs-import on Launchpad was 1 hour 20 min. I'm assuming that was lp's net I/O fault, but we'll see later today. [05:39] barely relatedly, 1:04 to git clone linux-2.6 with cold cache, 40s with warm [05:39] bob2: that is interesting, thanks [05:39] lp to au is tedious :/ [05:40] bob2: yeah. I'm getting tired of hearing all the reasons why Launchpad can't mirror content. [05:43] sounds like a problem with lp rather then bazaar :> [05:43] Just hit the repack at 10k revisions. That was exciting. [05:45] whats the link to clone the kernel [05:45] kernel.org is...down [05:46] AuroraBorealis_: here [05:46] $ git clone https://github.com/torvalds/linux.git linux [05:47] AuroraBorealis_: then, see https://bugs.launchpad.net/bzr-git/+bug/861973 as you'll hit a bug when you try to bzr branch from the local copy [05:47] Ubuntu bug 861973 in Bazaar Git Plugin "crashes when cloning local git repository with tags pointing at Tree objects" [High,Fix committed] [05:47] of course the git gui sucks [05:47] [the local copy being necessary since currently released bzr-git crashes when talking to github. THAT is fixed in the ppa:bzr/daily (yeay) though this bug's fix isn't there yet] [05:48] again, going through bzr->git seems like some problems... [05:48] just sayin [05:50] cause not only bazaar has to support itself but now it has to flawlessly support every other version control software out there [05:52] also github is sure taking its dear sweet time. siqq no progress indicators [05:56] yeah. cloning from github does NOT take 1 minute =P [06:20] hi all ! [06:21] AuroraBorealis_: as I said above, I got it in ~15 minutes. [06:50] * fullermd vilas at wave. [06:52] fullermd: :) [06:57] well [06:57] the fast-export output for the linux kernel is currently 15 gb [06:57] i might run out of drive space o.o [07:00] morning all [07:01] jam: _o/ [07:02] jam: no hang today, you're getting closer, the hang(s?) is scared and hides ;) [07:03] Maybe it's just hanging him out to dry. [07:03] vila: well, in testing, it was hanging around 2pm, and then stopped around 5pm. Apparently it is related to the phase of the moon. [07:03] jam: also note that windows has never hung which is a good sign [07:04] jam: testing locally you mean ? How ? Otherwise, yes, of course it's related to the moon... [07:04] :) [07:05] vila: yes, locally [07:05] and on devpad [07:05] I could reliably get it to hang, and then it started always working... [07:06] jam: infuriating [07:06] yep [07:07] but being able to trigger it more and more is a sure sign you're making progress [07:07] don't let it drive you nuts, you'll win in the end, it knows that... [07:10] jam: which server is involved in the hang, the pipe or the TCP one ? [07:34] vila: tcp, we generally don't use the pipe one in testing. [07:36] jam: did you consider using cethread there ? I seem to remember some hangs being caused by uncaught exceptions... [07:49] vila: cethread? [07:49] CatchingExceptionThread? [07:49] * vila nods [07:49] I don't think that is this specific problem, but i'm still digging. [07:49] bzrlib.cethread [07:50] I'm not getting an uncaught exception traceback on the terminal [07:51] I think I encountered cases where you don't get tracebacks (can't remember the details, that was... hairy) [07:54] morning all [07:55] mgz: _o/ [08:02] 'morning [08:08] jelmer: what's the process for landing things for bzr-builddeb? [08:08] (thanks for reviewing) [08:09] mgz: once you have approval, you can land them manually by doing one merge per MP into trunk [08:10] hm, I'll need to join the right team [08:12] does that mean bugging james_w? [08:16] yep [08:16] * mgz gets out his butterfly net [08:29] hello mgz, jelmer, europa [08:29] hey poolie! [08:29] hi poolie [08:38] hah, netsplits [08:38] mgz: one of the other bzr-builddeb-hackers should also be able to land it for you until you get commit access [09:02] jelmer: _o/ [09:02] jam: hi [09:02] jam: did you see my follow-up to https://code.launchpad.net/~jelmer/bzr/vf-fileids-altered-by-revisions/+merge/76851 ? [09:02] . o O (Using semaphores during net storm sounds appropriate) [09:02] jelmer: approved [09:03] jam: Thanks! [09:12] vila: wrt https://code.launchpad.net/~jr/bzr/plugin-test-failure/+merge/77569 [09:12] vila: isn't the test for export-pot ? [09:12] yes, but applied to a plugin [09:13] i.e. it should run if the plugin is available and not run it's not there, so putting it in plugin test suite makes sense [09:13] now it may also makes sense to have another test that we fail gracefully when we try to export-port an unknown plugin [09:14] vila: it should run in either case I think; it's a test for 'bzr export-pot', not for bzrlib.plugins.launchpad [09:14] vila: perhaps rather than using launchpad the test should register a dummy plugin [09:14] would work too but that seems more complex than two simpler tests [09:15] the dummy plugin stuff is brittle [09:15] vila: it seems wrong to put a export-pot test in the launchpad plugin testsuite though [09:15] really ? Why ? [09:15] vila: I would never think to look there if I was looking for export-pot tests [09:15] haa [09:15] hm. [09:15] vila: I'd have to check the testsuite of every other plugin too to see if it happens to test export-pot [09:16] I think it's a question of whether this is just a test that any plugin works at all [09:16] vila: using requiresFeature in the tests seems like the right thing to me [09:16] or whether every plugin that grows translatability will want a test that it works [09:16] mgz: it is [09:16] well, it there are bits specific to plugins in export-pot, it makes sense to run that kind of test for all plugins [09:16] if the former, the current location and a skip seem fine [09:16] if the latter, there should be a testcase class that plugins can subclass and use [09:17] mgz: +1 [09:17] that'll make plugin autors life easier to have already written tests for them [09:17] authors [09:18] mgz: in this case, it's the former [09:25] jelmer: and you won't test the unknown plugin case === yofel_ is now known as yofel [09:56] vila: The unknown plugin situation deserves its own testcase [10:18] okay, all caught up on launchpad mails, found some interesting bug reports [10:40] ModuleAvailableFeature from tests is deprecated, anyone know what replaces it? [10:42] Riddell from tests.features instead? [10:42] Riddell: use it from features [10:43] right, just a move [10:43] bzrlib.tests.features.ModuleAvailableFeature [11:58] hi jelmer [11:59] hi Noldorin [11:59] jelmer, how's it going? [12:00] jelmer, you seem to disappear each time i poke you hehe [12:00] alright, how are you ? [12:00] pretty good [12:00] working on various things; putting bzr-git aside for now [12:00] but speaking of which, how's progress? :-) [12:03] jelmer, [12:11] Noldorin: nothing new yet [12:12] jelmer, ah okay. any time soon though you think? :-) [12:12] Noldorin: no idea, sorry [12:12] okay [12:12] it might be very soon, might take a month or so [12:12] haha [12:12] it just depends on what else comes my way [12:12] very specific [12:12] oh well [12:12] fair enough [12:13] jelmer, it's okay, i understand. you seem quite over-workd to be fair :-P [12:13] shame there isn't someone else helping you out [12:15] Noldorin: nothing stopping you :) [12:15] jelmer, except my lack of knowledge of both python and bzr-git? :-P [12:15] that's fixable [12:16] i mean...the code is well-written with doubt...but not exactly well commented heh [12:16] alas, i don't have the many hours required for such a task [12:17] vila: so, I found out why it was hanging, now to figure out what to do next. [12:17] Specifically, if any client causes an exception in SmartTCPServer_for_testing it causes the server to stop [12:18] which also has the side effect that the next connection sees "oh, I'm stopping" [12:18] but doesn't actually close the connection [12:18] and because we keep a list of ".clients" the socket stays around and doesn't get garbage collected [12:18] so the client just hangs indefinitely [12:24] jam: which exception ? [12:24] vila: I'm not sure on that part yet, but this fixes the hang: http://paste.ubuntu.com/699711/ [12:25] I think there was a failure in the previous request (which might eventually bubble up) [12:25] but we don't get there, because the next request is denied and just hangs [12:25] that's probably the race [12:26] vila: right, now we might start seeing test *failures* after this [12:26] but we shouldn't see hangs [12:26] which I'm happier with :) [12:27] hehe, progress ! [12:27] jelmer, okay, so i'm actually a little bored now. if i can make this fix within an hour or two, maybe it's just worth it :-P [12:27] jelmer, that is, if you could provide any specific details how that function needs to be rewritten...would help a lot [12:28] i know you summarised already, which helps ;-) [12:29] vila: and I *think* what is happening is that we get 3-4 connections to the smart server during the test case [12:29] and some of those start a connection, and then never make another request [12:29] so they eventually timeout as an idle connection [12:29] which is actually true [12:29] they are dead, but just never called disconnect() [12:30] probably related to daemon threads which should be joined instead during tests (at least) [12:30] however, if any of those trigger ConnectionTimeout, that is an exception which gets passed to the server and shuts it down [12:30] because you don't have the "handle a connection timeout" logic in SmartTCPServer_for_testing that is in SmartTCPServer [12:30] we re-implemented the '.serve()' stack [12:30] well, there was no timeout when it was implemented [12:31] vila: sure, though that also meant we just stayed connected on dead connections forever. [12:31] so I think the sequence is [12:31] known thread leaks [12:31] we make an initial request to the server [12:31] and never disconnect [12:31] that connection is then 'idle' [12:31] if the rest of the test takes longer than 4.0 seconds to finish [12:32] when I mentioned the issue, the answer was, we don't want to change the server design because we don't care [12:32] the idle thread wakes up and disconnects the client [12:32] which has a side effect of shutting down the server as a whole [12:32] which causes further requests to get side lined [12:32] causing a hang without the patch, and probably a test failure with ti. [12:32] so ConnectionTimeout just needs to be added to ignored_exceptions [12:33] vila: right, something like that [12:33] I'm still poking around there [12:33] vila: though not exactly that [12:33] because handle_error is the one shutting down the server [12:33] not CEThread [12:33] CEThread would raise it as an exception during thread.join [12:34] but before we get to the point of reaping old connections [12:34] we've already shut down the server because one request failed [12:36] I think you have been mixing two designs [12:36] if ConnectionTimeout is not an error, handle_error should never see it [12:37] vila: I'm not 100% sure about ConnectionTimeout, I am pretty sure about handle_request causing a hang if the server is in shutdown mode. [12:40] jam: I don't quite get why a client can connect if the server is shutting down, isn't the listening socket closed as soon as the server shuts down ? [12:40] vila: *not* if it is shutting down because of handle_error [12:40] vila: If you call "stop_server" then it tries to connect to its own socket to force the server to stop accepting connections [12:40] ha right ! [12:40] but if you call handle_error() [12:40] it just sets "self.stopping" [12:40] but it is stuck in "self.accept()" [12:40] socket.accept() [12:40] because only a single client could fail before your patch [12:40] so it will accept *1 more connection* [12:40] that it won't respond to [12:41] vila: yeah probably [12:41] well, that's more than probably :) [12:41] oh, another small bug [12:41] vila: http://paste.ubuntu.com/699723/ [12:41] the test runs in the main thread so only one client can run at a time, therefore only one client can fail at a time [12:41] can you point out why close_request won't get called? [12:42] same constraint [12:42] vila: handle_request has a blanket "raise" in it [12:42] the server shutdown takes care of closing [12:43] where ? [12:43] vila: bzrlib.tests.test_server.TestingTCPServerMixin.handle_error [12:44] and ? [12:44] vila: we get an exception, we 'handle it' by re-raising it, which means you won't call close_request() though the code sure looks like it would [12:44] Noldorin: I'm not sure if I can really provide much more details without spending a considerable amount of time on it myself [12:44] no [12:45] Noldorin: either way, it would be worth reading up on the bzr and git internals if you want to have a stab at this [12:45] vila: TestingTCPServerMixin.handle_request calls process_request in a try/except [12:45] the code calls handle_error and then close_request, if an execption is raised in handle_error, close won't be called [12:45] jelmer, hmm okay sure. anything i should read beside the general code and bug report itself? [12:45] vila: except handle_error calls "raise" [12:45] re-raising the existing error [12:45] that's what I just said [12:45] jelmer, even a rough pseudocode implementation of the new implementation perhaps...? ;-) [12:45] jam: and I said previously: the server shutdown takes care of closing [12:46] jam: which was based on the same assumption that a single client can fail at a time [12:46] vila: so, it violates the SocketServer paradigm to have handle_error raise an exception. Sure, it may get caught later. [12:46] Noldorin: the repository structure (how a tree is built up, etc) [12:46] but it can also cause hangs [12:46] what paradigm ? [12:46] because the client is never disconnected [12:46] Noldorin: sorry, I'm not entirely sure what the pseudocode would have to look like yet either [12:46] heh okay [12:47] that's fine [12:47] so say the client makes a bad request, that triggers an exception on the server [12:47] the client then hangs forever waiting for the server to respond [12:47] because the server didn't close the connection. [12:47] doesn't happen in tests [12:47] vila: we haven't triggered it yet in tests [12:47] you can't [12:48] unless you introduce a new kind of failure like you did with the timeout [12:48] vila: if process_request itself was buggy it would be triggering this [12:48] it happens that process_request immediately deferrs to a thread for the actual processing. [12:48] by design [12:48] vila: sure, but if there were a bug in that code, it would cause hangs [12:49] vila: also you have "t.pending_exception()" so if an exception happens at thread start, it will also cause hangs. [12:49] which is why there should be no bug there :) [12:49] again, I haven't tracked down where the actual failure is causing the code to then hang, I'm pointing out that the current implementation is likely to hang if there are failures in a few key aspects of the code. [12:50] rather than getting a test suite failure that we can debug [12:50] well, I suggest focusing on this new failure mode you've introduced first [12:50] vila: I have to get an actual failing test first [12:50] which involved not having the test suite hang [12:50] On windows, because the hang is in a sock.recv() you can't even ^C out of it to get a traceback. [12:50] use cethread to check which exception is shutting down the server then [12:51] jam: the TCP server on windows ? [12:51] vila: socket.recv() is one of those "blocking and cannot be interrupted" calls on Windows [12:51] this is actually during "self.make_repository()" [12:51] jam: the TCP server on windows ? [12:51] vila: the *client* code is blocked [12:52] jam: the TCP server on windows ? [12:52] jelmer, heh. i guess that 'in progress' is for me? ;-) [12:52] no, just in general since there's a testcase and some initial work [12:52] vila: Repeating yourself doesn't make me understand what you are asking. Vs what? [12:53] jam: is that when using the TCP server on windows ? [12:53] vila: yes [12:53] as opposed to the pipe server, there are only 2 [12:53] vila: as I said way back when, we generally don't use the pipe server in testing [12:53] This is a per_interrepository test. [12:53] But per_repository where Repository format is Remote triggers the same code paths [12:54] We have a SmartTCPServer_for_testing running, it received a request but decided it didn't want to respond because it was shutting down. [12:54] jelmer, ok sure ;-) [12:54] The client is then blocked waiting for the server to respond (or disconnect it) [12:54] On Windows, the client is blocked so badly you have to kill the process. [12:54] On Linux, you can at least do ^C and get a traceback in ~/.bzr.log [12:55] yeah, on linux too [12:55] not always when a hang happens [12:55] vila: for this particular hang, ^C worked fine for me [12:55] it is how I pinpointed the hanging part [12:55] hmm, lucky you then ;) [12:57] jam: so, IIUC, if the client is in a blocking read, you should not shut down the server [12:57] jam: or if you do so, you should close the connected socket in a way that will unblock the client [12:57] vila: right, I think "closing sockets on shutdown" is the most reasonable process. [12:58] and currently the code has a few bugs where during exceptions that will stop the server, it doesn't close the connections [12:59] apart from the timeout case, I think the test server is quite well tested in this regard [12:59] the only problematic case being th TCP smart server because it didn't track the client connections [12:59] an exception during process_request will cause the server to not disconnect the client [13:00] that includes process_request_thread because SocketServer was also written as "self.handle_error(); self.close_request(request)" [13:01] that was the right thing to do as long as a single client thread could fail [13:02] if this is not true anymore for your server, you should probably overload verify_request and handle_error [13:24] hm, not sure if it'd be better to give instructions or just do the changes. [14:09] vila: so interestingly enough, you had a test case for "test_server_fails_while_serving_or_stopping", but you had wrapped the client.read with a socket timeout and suppressed the socket error. [14:09] It turns out, that if you don't need the timeout if you close the connection. [14:09] however, there is a catch [14:09] if you use SocketServer.StreamRequestHandler [14:09] it creates an "rfile" [14:10] which is *another handle to the socket* [14:10] so just closing the 'request' object isn't sufficient [14:10] you have to either a) Not use StreamRequestHandler, or b) somehow get at the handler to tell it to close its handle because you had an error. [14:11] alternatively, if you don't raise an exception in handle_error, the StreamRequestHandler goes out of scope naturally, and closes its connection [14:11] but with an exception, I think it ends up in the traceback, and thus doesn't get gc'd [14:13] given that we avoid StreamRequestHandler everywhere else because makefile doesn't play very nicely with other bits (in general) [14:13] it would seem reasonable to just avoid it. [14:13] and replace all read/write with recv/send ? [14:14] there is only one or two read write calls anyway. [14:14] The comment on the timeout was that "the server may not get cycles" but the reality was that the server got the cycles, it just wasn't disconnecting the client. [14:15] which comment ? [14:17] yeah, makefile is more trouble than it's worth [14:20] That bzr branch of a local git to local bzr tree that I kicked off earlier is at 8h53m22s elapsed, 245 minutes CPU time. Half way done. [14:31] hmm [14:31] * jelmer ponders deprecating make_branch_and_tree [14:32] jelmer: oh, hey [14:32] jelmer: I applied poolies workaround tag deletes [14:32] jelmer: and I'm churning though it now. [14:32] AfC: cool [14:33] AfC: note that his bug shouldn't crop up if you clone from the remote git branch directly [14:35] jelmer: yeah, I couldn't do that [at the time] because bzr-git choked on github URLs (or, rather, the 405 redirect thereby) [14:35] jelmer: ppa:bzr/daily appears to have fixed that, I discovered subsequently. [14:35] AfC: git:// should work in older versions of bzr-git, too [14:35] AfC: but yeah, http/https weren't supported earlier [14:35] jelmer: plus, old habit - I got tired of things crashing :/ [14:36] AfC: yeah, this is all still pretty experimental stuff.. [14:40] hang on lucid :-/ [14:40] I was hoping to *not* see that as it means it can happen on pqm too :-/ [14:41] vila: I don't want to hear these things! [14:41] AfC: sorry, I wasn't talking to you :) [14:41] heh [14:45] * AfC adds 8GB of swap to his server in hopes that bzr will not OOM on me [15:24] Finally, https://launchpad.net/bzr/+milestones starts to become readable... [15:47] vila: still there? [15:47] it looks like any bzrlib usage now breaks if bzrlib isn't explicitly initialized [15:49] I've filed a bug [15:51] ha, I had some doubts about that :-/ [15:52] jelmer: assign it to me, I ~know what is going on [15:53] jelmer: sorry about that :-/ [15:53] jelmer: how did you encounter it ? [15:53] vila: I was writing a simple test script when I hit it: http://pastebin.ubuntu.com/699875/ [15:54] ha. Add 'with bzrlib.initialize():' :-p [16:11] Hi guys, I'm trying to use bzr builder to do a dailydeb for a source tree that has a debian/ dir that differs from the packaging branch.. [16:11] Is there a way to tell nest-part to just do a forcible merge? === 64MAAICBP is now known as mtaylor [16:17] jelmer: where did you file this bug ? === soren_ is now known as soren === deryck is now known as deryck[lunch] [16:24] well. converting the linux kernel git repo to bazaar has taken 9 hours so far and only 1/6th done haha [16:25] no i lied its half done === marienz_ is now known as marienz [16:29] all this in the name of science! === vila is now known as science [16:30] :O\ [16:30] AuroraBorealis: thanks ! === science is now known as vila [16:30] someone here was saying a local branch with a tons of commits took forever [16:30] so i'm trying it out [16:32] but i think he was trying to branch the git repo which means it had to convert it, which of course is taking forever [16:42] jelmer: if you get a moment, please do merge my bzr-builddeb branches [16:43] jam: you can review your own fix for bug 856731 if you like [16:43] Launchpad bug 856731 in Bazaar "out of memory Fetching revisions:Inserting stream:Estimate on AIX" [Medium,In progress] https://launchpad.net/bugs/856731 [16:49] Who would be the best person to ask about build recipes? I am having a hell of a time with one right now. :-/ [16:49] james_w: ping? === deryck[lunch] is now known as deryck [17:22] mgz: will do [17:33] jelmer: still there ? [17:34] jelmer: I'm running a full test run before submitting my fix for bug #863401 [17:34] Launchpad bug 863401 in Bazaar "now requires explicit initialization" [Critical,Confirmed] https://launchpad.net/bugs/863401 [17:57] anyone else having problems commiting with bazaar explorer? bzr: ERROR: exceptions.TypeError: unsupported type u'Collecting changes [0] - Stage' [17:58] does it happen with the command line bazaar? [17:58] AuroraBorealis: no.. works perfect [17:59] hmm [17:59] latest version of explorer and all that? [17:59] 1.2.1 [17:59] and all that :) [18:00] http://paste.pocoo.org/show/eUq2c1UEqxfo3wHvM2eE/ [18:00]