[08:25] re [09:12] Good morning all! [09:18] niemeyer: hiya [09:30] heya niemeyer, you're on early :) [09:30] fwereade__: Yeah, a bit :) [09:46] * rog finally makes his network connection work again [09:47] rog: Welcome back [09:48] niemeyer: hi! [09:49] niemeyer: you're up earlier than usual... [09:49] rog: hiya [09:49] TheMue: morning! [09:50] rog: Yeah, a bit [09:50] niemeyer: any chance you could have another look at those CLs? i'm hoping to get something pushed this week. [09:52] rog: Not right now [09:52] niemeyer: k [09:55] niemeyer, btw, there was a conversation the other day I think you missed about gozk panicing in C when a Conn is closed (and something else is busy in C code) [09:55] fwereade__: Ugh, that's not good [09:55] niemeyer, would fixing this just be a matter of a RWMutex on handle accesses, or is it more subtle? [09:55] fwereade__: Well.. I'd need some more information about the crash [09:57] niemeyer, I've seen it in a couple of cases, but the common feature is that some goroutine is always somewhere like: launchpad.net/gozk/zookeeper._C2func_zoo_wexists(0x36edf80, 0x36ef380) [09:57] fwereade__: Ok, do you have a full traceback? [09:58] niemeyer, here's the one I just saw [09:58] niemeyer: i had a brief look at this - it just happens when zk derefs the nil handle, because handle access isn't mutexed, as fwereade__ suggestes [09:58] niemeyer, http://paste.ubuntu.com/855178/ [09:58] suggests [09:59] niemeyer, the set line shouldn't be the problem, because that conn is not closed [10:01] fwereade__: Is this looping while you're calling Close on the zk Conn? [10:01] niemeyer, yes [10:02] fwereade__: That's going back to the point we talked about [10:02] fwereade__: We shouldn't do that [10:02] fwereade__: We shouldn't ever close a connection behind the back of logic that we're managing ourselves [10:03] niemeyer, ok, maybe I'm missing something [10:03] fwereade__: That's true for tests, and also for real logic [10:04] fwereade__: We can even implement locking to prevent the nil pointer reference at some point, but that's not the most critical issue [10:04] fwereade__: Ok, but do you recall that conversation? [10:04] niemeyer, ok, in that case I guess the tests I was trying to write are not actually relevant, because I'm really trying to test deal-with-session-event handling that doesn't exist yet [10:05] Feb 22 12:09:11 fwereade_: We should never, ever, ever, leave background logic unattended while we're messing around with the system state in disruptive ways [10:05] fwereade__: Hmm, can you expand on that? [10:05] niemeyer, yes, I do; the reason I saw this one just now is because all I *thought* I had active on that conn was a watch [10:06] niemeyer, I was trying to verify that the watch channel got closed when the underlying ZK connection got yanked out from underneath [10:06] fwereade__: Ok, that's an interesting case.. maybe we have something to fix then [10:06] fwereade__: Can you please paste waitFor? [10:07] niemeyer, http://paste.ubuntu.com/855189/ [10:09] fwereade__: Sorry, I thought that was where the loop was.. TestDisconnectAliveWatch then, I guess [10:10] Ah, this is the guy blowing up: launchpad.net/gozk/zookeeper._C2func_zoo_set2(0x0, 0x36ef360) [10:10] niemeyer, yep [10:10] fwereade__: Please paste the test and (*Pinger).run [10:10] niemeyer, before I paste this: I know the stuff in the goroutine comment is wrong :p [10:10] niemeyer, [10:10] http://paste.ubuntu.com/855193/ [10:11] fwereade__: Cool :) [10:11] niemeyer, http://paste.ubuntu.com/855195/ [10:13] niemeyer, wait, you're right, it is set and not exists, bugger, I'm at least partially on crack (in a way that I was not hitherto aware of) [10:13] fwereade__: You know it's wrong in which way? [10:13] niemeyer, "this should be equivalent" [10:16] niemeyer, ie, what we should actually be doing is *not* just closing the conn until we've dealt with everything [10:17] niemeyer, but if it is indeed the set that's blowing up, anmd it looks like it is, then I'm completely confused because that's using a different conn to the one that gets closed [10:17] fwereade__: Not really [10:17] fwereade__: Oh, wait.. yes [10:18] niemeyer, so I would appear to be crackful in an entirely fascinating and original way :/ [10:18] fwereade__: But! [10:18] fwereade__: Please paste Close [10:18] fwereade__: I bet you have a race [10:18] niemeyer, oh balls, I try to hit the target node again [10:19] niemeyer, (to give clients as long a time as possible before they see a timeout) [10:19] niemeyer, (naively trusting that if the connection is borked it'll just error and I can ignore it) [10:19] fwereade__: That's fine [10:20] fwereade__: But a race is not fine.. please paste Close [10:21] Interesting, http://cocode.io/beta/ will provide collaborative coding. It's written in Go knows Go as a highlighted lang. Sadly yet only GitHub, no Bazaar (or others). [10:22] fwereade_: That's fine [10:22] fwereade_: But a race is not fine.. please paste Close [10:23] fwereade_, fwereade__: Are you having wifi issues too? [10:23] fwereade_, fwereade__, fwereade: Are you having wifi issues too? :-) [10:23] niemeyer, seems so :p [10:24] niemeyer, http://paste.ubuntu.com/855206/ [10:24] niemeyer, if you didn't see above [10:24] fwereade_, fwereade__, fwereade: I always knew you were not a single person.. [10:24] fwereade: Yeah, you have a race indeed [10:25] fwereade: Can you please have a look at this post: http://blog.labix.org/2011/10/09/death-of-goroutines-under-control [10:25] niemeyer, I'm a globally distributed collection of outsourced drones, but don't tell anyone [10:25] fwereade: I'll highlight a few things afterwards, if they're not obvious [10:25] niemeyer, cool, thanks [10:25] * fwereade reads [10:27] niemeyer: where's the race? [10:28] niemeyer, I remember reading that when you wrote it, and totally forgot about it until now [10:28] niemeyer, it seems clear, but I'd appreciate the highlighting all the same [10:31] oh yeah, in TestDisconnectAliveWatch. i was looking in the pinger methods [10:36] grar [10:37] fwereade_: Are you here? :) [10:38] niemeyer, er, probably [10:38] fwereade_: Ok :) [10:38] fwereade_: So, quicklY! :-) [10:38] fwereade_: Observe the way Stop is implemented in the blog post [10:39] fwereade_: It sends a fatal error, and *waits* [10:39] niemeyer: next round of https://codereview.appspot.com/5671055/ [10:39] TheMue: Thanks, I'll have a look at that next to rog's [10:39] niemeyer: yep, thx [10:40] fwereade_: You're sending a stop signal, and ignoring the fact that, naturally, stopping isn't instantaneous [10:40] niemeyer, yeah; I had a "closed" channel I was waiting on originally, but it was pointed out that since closing was unbuffered I wasn't getting any benefit from waiting on closed after sending to closing [10:41] fwereade_: Uh.. this is bogus [10:41] fwereade_: Oh, or maybe not.. let me see again [10:41] niemeyer: it looked ok to me :-) [10:41] niemeyer, I'm not saying there weren't subtle bugs in what I was doing, or in what I am doing ;) [10:42] niemeyer: i *hope* it's not bogus otherwise my understanding of channels is fundamentally flawed... [10:43] rog: No, you're right.. as long it's unbuffered, that's fine [10:43] phew [10:44] fwereade_: The race is in the test itself [10:44] yup [10:44] fwereade_: No, sorry.. crack again.. [10:45] altConn is in a different end [10:45] Hmm [10:46] niemeyer: isn't is a race that waitFor(c, altConn, path) is called concurrently with altConn.Close() ? [10:46] (not that i've looked at waitFor) [10:47] rog: No.. the pinger is blowing up with a nil handle [10:48] Despite anything else, there's a race here in that the handle is being cleaned up underneath its feet [10:48] niemeyer: that still looks like a race to me [10:48] rog: :-) [10:48] niemeyer: even if it's not the thing that's blowing up [10:49] rog: If the logic in the test is sane, the Close won't happen before the kill [10:50] rog: if.. [10:51] niemeyer: oh yes, i hadn't seen connect - i was assuming that the session event was the connection event, not the teardown event. [10:51] but presumably connect waits for the first session event [10:51] rog, yes [10:51] rog: That's what I'm assuming too [10:56] fwereade_: how reproducible is this? [10:56] rog, not very :( [10:56] rog, it'll happen eventually [11:00] fwereade_, niemeyer: AliveW is busy on altConn at the same time that altConn is closed, right? [11:00] so that looks like a race to me [11:01] rog, yes; so that Close could happen at any time, and that's definitely a problem [11:01] i.e. the server gets killed, the goroutine receives the session event and closes altConn, at the same time AliveW happens to do a Change [11:02] rog, there are similar tests that really are *only* watching when the conn gets closed, and they're fine; yes [11:02] rog: If that's the case, then the connection is being Closed before kill() [11:02] niemeyer: i don't think so [11:03] niemeyer: the kill happens, then the close happens, then AliveW does something [11:03] rog: AliveW is run before kill.. [11:03] niemeyer: AliveW has a goroutine [11:03] niemeyer, I guess it's not impossible that there's another session event confusing me [11:03] rog: OH, does it? [11:03] Why? [11:03] To transform the event, I guess [11:03] niemeyer: yup [11:03] But it should be listening only [11:03] On a channel.. that's not supposed to blow up [11:04] fwereade_: Can you please paste the whole file, or just push the branch? [11:04] yeah [11:04] niemeyer, sure [11:04] niemeyer, ~fwereade/juju/go-presence-nodes [11:05] niemeyer, when we're watching for aliveness we can't just dumbly watch a channel [11:05] niemeyer, we need to keep rewatching when we get changes before timeouts [11:05] fwereade_: Ho ho ho [11:05] niemeyer, technically, above, when we're watching for *deadness* [11:06] niemeyer, have I misunderstood something fundamental? [11:06] fwereade_: I'm just curious about the rewatching.. that may explain it [11:06] niemeyer, (also we may need to rewatch when waiting for liveness too, it's a bit of an edge case, but...) [11:07] fwereade_: Yeah, there's a race there.. [11:07] fwereade_: You're firing goroutines that use zk in a way that closing the connection will ignore completely [11:08] niemeyer, huh, I thought that closing the conn would close all the watches, and I'd see that [11:08] niemeyer: i don't think there's a way to do it without either having a way to Close an AliveW or by making zk.Close concurrent-safe [11:08] niemeyer: personally, i think i'd opt for the latter [11:08] niemeyer: (after all, all the other routines are ok to call concurrently) [11:09] s/routines/zk methods/ [11:10] rog: Yes, there may be something to change in gozk, but there's something to be handled in presence itself that I'm trying to understand [11:11] niemeyer: presence seems ok to me, but i'm probably missing something [11:11] fwereade_: Why do we need awaitDead and awaitAlive to loop? [11:12] niemeyer, awaitDead to refresh the data watch until I get a timeout without the data watch having fired [11:12] fwereade_: Ah, right.. it's the opposite.. it fires when it doesn't get a watch [11:12] Hmm [11:12] niemeyer, awaitAlive to handle the case where a known-unresponsive node is deleted -- it's still dead but not in the same way and IMO I shouldn't really alert the client to say "yeah, nothing changed" [11:14] Well, looks like we need our Watcher type back.. [11:15] niemeyer: if zk Close was concurrent-safe, everything would be fine [11:15] So that we can stop this logic in a sane way [11:15] niemeyer: just like closing a net.Conn [11:15] niemeyer: then the logic just stops as a matter of course [11:16] rog: What do you mean by concurrency safe? [11:16] niemeyer: i mean that you could Close a zk conn concurrently with executing some other operation on the conn [11:16] niemeyer, it feels like one of the advantages of gozk is that the watches can just get auto-closed, and it would be really nice if we could follow the same convention here [11:16] rog, nitpick, technically it's all the other methods that aren't concurrent-safe ;) [11:17] rog: That's not enough [11:17] fwereade_: they're concurrent-safe with each other... [11:17] niemeyer: no? [11:17] fwereade_: Exactly [11:17] rog: What fwereade_ said [11:17] niemeyer: i don't understand. wouldn't a mutex around the handle access be acceptable? [11:18] niemeyer, it still seems to me that a RWMutex on handle around C calls would be enough; what am I missing? [11:18] niemeyer: (every method would have to respect it, of course) [11:18] rog: This is something else, not what you originally said [11:18] niemeyer: that's what is necessary to make it ok to call Close concurrently, AFAICS [11:18] niemeyer: which is what i was suggesting [11:18] rog: You're still missing the point.. [11:19] niemeyer, so am I I think [11:19] rog: It's not just about concurrency [11:19] * rog usually does [11:19] rog: Close(); Set(foo).. BOOM! [11:19] niemeyer: no. [11:19] niemeyer: not if Set checks to see if handle is nil and then returns an error [11:19] rog: Dude.. please make up your mind [11:20] niemeyer, wouldn't we just grab a read lock, check for nil, and error back cleanly? [11:20] fwereade_: exactly [11:20] fwereade_: YES! [11:20] fwereade_: We'd just do that.. that's not what was being said so far! [11:20] niemeyer, ha, I thought it was [11:20] rog: What do you mean by concurrency safe? [11:20] niemeyer: i mean that you could Close a zk conn concurrently with executing some other operation on the conn [11:20] niemeyer: i just said "make Close concurrent-safe". that doesn't imply that Close is the only method that needs to change. [11:21] rog: This is not about concurrency! Serial operations blow up! [11:21] niemeyer: well, that's a different (although related) issue. [11:21] niemeyer, Close might block for a while (or *possibly for ever if enough is going on...) but that seems preferable to unrecoverable panics in C if if a hamfisted amateur like myself is trying to use the library [11:21] rog: Yes, it's the issue we're seeing [11:22] niemeyer: i think we're seeing an issue because of concurrency, no? [11:22] rog: This is not about concurrency! Serial operations blow up! [11:22] niemeyer: is that the case in this example? i thought we were doing things in two separate goroutines [11:23] rog: This is not about concurrency! Serial operations blow up! [11:23] rog: You can inline.. Close(); Set(); BOM! [11:23] rog, niemeyer: sorry, lamb chops are hot on the table, back shortly :) [11:24] niemeyer: that's not what's happening in this issue though, right? [11:24] rog: Yes, it is [11:24] niemeyer: in the same goroutine? [11:24] rog: No.. the fact there are two goroutines is completely irrelevant [11:25] niemeyer: to my mind, it's not. if we were in a single goroutine it would be easy to avoid doing an operation after Close. [11:25] niemeyer: it's the concurrency issue that makes it harder [11:26] rog: Sorry, but a "concurrency issue" is something else than what you have in mind [11:26] niemeyer: ok, fair enough [11:26] niemeyer: so... [11:26] rog: You can put a lock across every single function in gozk, and it will still blow up [11:26] niemeyer: only if they don't check that handle is not nil before calling into C, no? [11:27] rog: Yes, and that's not a concurrency issue.. this is about using a connection after it's been closed. [11:27] rog: and make that not panic [11:27] niemeyer: agreed. but just doing that won't fix our problem either. [11:27] niemeyer: because then there would be a race. [11:29] rog: Sure, because you can't trust on the test for the handle to be valid without a lock [11:29] niemeyer: exactly [11:37] fwereade_: So.. [11:37] niemeyer: i think a RWMutex would probably do the job just fine [11:37] on zk.Conn, that is [11:38] fwereade_: You're right, we must definitely fix gozk itself so that we protect from that kind of issue [11:38] fwereade_: I'm just still wondering if we should change the interface for that [11:38] fwereade_: I guess it's fine to keep it running it in background, and simply error out, given that this is purely a read operation without any side effects [11:40] niemeyer: that was my thought. [11:40] fwereade_: Btw, I'd prefer to not have the (s state) as a parameter to the await functions [11:41] fwereade_: There's actually a bug in that regard in the current logic [11:42] fwereade_: No, there isn't, sorry [11:42] fwereade_: You're passing the parameter as a copy, which is safe [11:42] fwereade_: Still, I don't see why the parameter is necessary [11:43] niemeyer: it saves passing the path and timeout parameters as separate arguments [11:44] niemeyer: but i'm with you [11:47] rog: Yeah, I think the organization there needs some tweaking for clarification [11:51] niemeyer: i wondered if things might be clearer if there was an internal watcher type which held the params that are being passed around. [11:52] rog: I was wondering about that too, but this is makes it seem like there wouldn't be much benefit compared to what is there now: [11:52] s, zkWatch, err = newStateW(conn, s.path) [11:52] niemeyer: s, err := w.getStateW() ? [11:52] rog: The only bit that is actually constant is the path [11:53] yeah [11:53] rog: Right, w is the path.. [11:53] rog: Which renders fwereade_ design elegant [11:53] it was the other methods i was really thinking about (waitAlive and waitDead) [11:54] rog: Right.. as methods, they would have to live on a type that is simply the path [11:54] rog: Because timeout and aliveness changes [11:55] niemeyer: that's fine. they can still be fields in the waiter type. it will change when they change. [11:55] niemeyer: (it's not used concurrently) [11:55] rog: There's no point in having them as fields.. it's only used in that one function [11:55] maybe "node" might be a better name for the type [11:56] rog: Yeah, path :) [11:56] rog: type node string, where node is the path [11:57] rog: That won't improve much what fwereade_ has there now, though. The design feels sound on that level [11:57] * rog is thinking about it [11:57] rog: Well.. I guess we could have the connection on it too [11:58] Which would make it more interesting [11:58] niemeyer: yeah [11:58] type node string [11:58] niemeyer: and the watch channel to send to [11:58] func (n node) readState() (alive, timeout, error) [11:58] func (n node) readStateW() (alive, timeout, watch, error) [11:58] func (n node) get() error [11:58] sorry [11:58] func (n *node) get() error [11:58] func (n node) awaitAlive [11:58] func (n node) awaitDead [11:58] func (n *node) getW() error [11:59] rog: What's get? [11:59] rog: get error? :) [11:59] niemeyer: gets the current state of the node. stores it in n. [11:59] niemeyer: getW also gets a wait channel and stores that in n too. [11:59] rog: That's a bad name then.. update() maybe [11:59] sure [12:01] and updateW() [12:01] rog: Yeah, that should improve things indeed, +1 [12:02] cool [12:02] * fwereade_ reads back... [12:02] rog, fwereade_: So.. I'd just like to take an eagles view on the whole issue to close it down, if that's ok [12:02] We have one bug, and one design approach we're agreeing that breaks a prior rule [12:02] * rog feels those sharp eyes on him from far far above. [12:02] :) [12:03] The bug is, as we discussed at length, the fact closing and using a connection is a blow up in gozk for sure. We must fix that. [12:03] The design decision is a more subtle one [12:04] The prior goal I was personally attempting to ensure is that we never have background logic under our control running in parallel with other stuff that will break down the concurrently executing logic [12:04] Under that design approach, we'd need a Stop() method on the watcher [12:05] So that we can stop it.. that would avoid the gozk bug too [12:05] But, we're agreeing to not do that, and instead fix gozk, and establish a new condition: [12:05] If the background logic is entirely innocuous, it's fine to allow it to die a cold death [12:07] sounds good to me. as long as the logic *will* die, and not be left around as garbage [12:07] niemeyer, that sounds sensible to me personally, but then I would say that [12:07] AliveW is one example of that.. by itself, it's doing nothing. If it blows up in the background, there are zero side effects, _as long as the logic that is using the watch does nothing_!! [12:07] niemeyer, this feels consistent with every other ZK watch [12:08] It's _NOT_ ok to have someone waiting on the watch resulting from AliveW, and doing further actions like changing the FS or whatever else, despite the fact the connection has been closed [12:08] and the logic using the watch will just see a close on the channel, which means an error, so hopefully this kind of logic can cascade [12:08] fwereade_: Agreed [12:08] niemeyer, excellent [12:08] fwereade_: agreed [12:08] rog: No, that's exactly what we don't want [12:09] rog: We'd have to reevaluate the use of the watch, if it's also in background [12:09] niemeyer: no? i'm imagining some other innocuous logic layered on top of AliveW [12:09] rog: For it to be alright, it'd have to be innocuous too [12:09] niemeyer: yes, definitely [12:09] rog: Cool, we're in agreement then [12:09] yup [12:10] So, those are the relevant decisions [12:10] Glad we're in agreement, phew :) [12:10] fwereade_: In addition to that, we were nitpicking over the interface around the state type [12:11] fwereade_: Which is not terribly important, but that might be interesting to sort out if you still have the energy [12:11] niemeyer, I saw that and I'm not strongly invested in what I currently have; the fields seemed to go well together, and to stop the function signatures breaking lines, and so I decided it was good :) [12:11] fwereade_: rog was suggesting a node type, which feels nice to clarify the state handling [12:12] fwereade_: Something like this is the status quo of the discussion: [12:12] type node struct { conn, path, timeout, alive } [12:12] func (node) update() error [12:12] func (node) updateW() (watch, error) [12:13] *node [12:13] Sorry, yes [12:13] func (*node) waitAlive() [12:13] func (*node) waitDead() [12:13] (the last two with the necessary return types and parameters) [12:13] niemeyer, LGTM [12:13] thanks guys, productive discussion :-) [12:14] fwereade_: waitAlive and waitDead would be implemented on top of the former two [12:14] fwereade_: Likewise! [12:14] i keep on wondering if waitAlive and waitDead might not be better named as deadWait and liveWait respectively [12:14] i.e. name them after the state they start in rather than the state they're waiting for. [12:14] dunno [12:14] rog: -1.. these are nouns [12:14] we need verbs [12:14] yeah [12:15] i want to call waitDead waitDeath [12:15] but can't think of an appropriate name for the other one [12:15] rog, I considered similar but "waitBirth" wasn't right [12:15] rog, indeed [12:15] waitAlive and waitDead feels great to me [12:15] yeah [12:16] And on that node, I'll do some reviews.. [12:16] that's fine. just thought i'd mention it in case others has similar thoughts. [12:16] niemeyer, would you like me to take a look at gozk as well? [12:16] Actually, let me finish answering emails first [12:16] niemeyer, I don't want to overload you :) [12:16] i could do gozk [12:16] fwereade_: Yes, please [12:16] or rog.. whatever works [12:16] or fwereade_ sure [12:17] i'm a bit ahead of myself anyway currently [12:17] * fwereade_ bows out gracefully [12:18] fwereade_: how about you refactor presence (again!) and i'll do zk. [12:18] rog, give me a shout if it becomes a hassle in any way and I can grab it [12:18] rog, but that sounds good to me [12:18] * rog thinks is shouldn't be too much problem. [12:18] s/is/it/ [12:27] niemeyer: hi [12:27] andrewsmedina: Hey [12:35] fwereade_, niemeyer: ok, zk changes made. now for the harder bit... the testing. [12:36] rog, good luck :) [12:38] * rog thinks of a devious way of forcing a zk request to take a long time. [12:44] niemeyer: in the last weekly version Go isn't work with two packages in same project [12:46] andrewsmedina: in some of your projects you put a example.go which is a different package. [13:08] rog, just a thought; I have a couple of `switch event.Type`s without defaults; if I get totally unexpected events, it would probably be sensible to `close(watch)`, right? [13:08] fwereade_: you could probably panic [13:09] fwereade_: it's part of the contract that the correct event types should be returned [13:09] fwereade_: if they're not, then something's very broken [13:09] the contract with zk, that is [13:09] rog, in that case I think explicitly panicking may be excessively defensive [13:10] fwereade_: i'd prefer to see a panic rather than a silent failure in that case [13:10] rog, ok, just `panic(event)` then? [13:10] fwereade_: panic(fmt.Errorf("unexpected event %v", event)) [13:10] or something like that [13:11] rog, ok then :) [13:11] it's nice to see the thing that caused the panic [13:12] rog, sounds good [13:16] rog, hm, I guess I can lose the forward() stuff once we have a Close-safe gozk [13:16] fwereade_: yeah. that'll be good. [13:16] fwereade_: i could push a version that you could use until i've done the tests, if you like. [13:17] rog, a good thing but also sort of a shame, that stuff made me very happy :) [13:17] rog, I guess that would be useful [13:17] fwereade_: it's still there for when you need it again... [13:17] fwereade_: in fact i think i might be using that technique in the zk tests... (my "devious way") [13:18] rog, indeed, the pleasure of deleting code still outweighs the pleasure I took in the code to begin with [13:18] rog, nice :) [13:18] fwereade_: i'm gonna have a forwarder and stop all incoming traffic. then i know that the reply can't come back quickly... [13:20] fwereade_: in fact i might steal your code directly, since it's still there... [13:20] rog, go for it :) [13:24] So, solved a merging prob, 76 tests passed [13:25] TheMue: nice [13:26] rog: yeah, it goes on, piece by piece [13:26] TheMue: i bet you're glad you speeded up the tests now :-) [13:35] rog: definitely [13:36] rog: when zk is cold (first start) about 11 sec, then about 6.3 sec. [13:37] TheMue: how long if you run just one test? [13:37]