[03:11] does juju assume all machine units have a different IP? [03:20] i think so [03:39] bigjools: assume in what way? [03:40] thumper: I don't know the nature of the assumptions, I just need to know if it does as it affects how we'd implement some Azure stuff [03:40] Azure is not a "flat" service [03:40] bigjools: I don't think anything is indexed on IP [03:40] however it does use the address to talk to the machines [03:41] yeah I figured [03:41] it is what is propagated between units in relations [03:41] there is some work pending which mgz is starting around machine addressability [03:41] to remove some of the assumptions [03:41] and have clear expectations [03:41] bigjools: so touch base with mgz for more info [03:41] Azure gives multiple VMs the same IP address and load balances unless you do things differently [03:42] ?! [03:42] thumper: right thanks - Gavin was talking to him already [03:42] you should be fine then :) [03:42] In Azure, you get this sort of structure: Service -> Deployment -> VM / VM / ... [03:43] each "service" has its own IP [03:43] rather than the VM [04:00] Any reviewers in the house? I've got a big one waiting: https://codereview.appspot.com/11027043 [04:02] way to attract reviewers :) [04:03] Any worse than the built-in "please take a look"? :) [04:31] * thumper does a mega move [04:42] Can anyone help with this panic in the stdlib please? http://play.golang.org/p/zRZh_cXGy9 [04:43] * davecheney looks [04:47] I think I was doing the wrong thing to what I need anyway, but I thought it was interesting that I can panic it :) [04:47] it's because you're trying to marshall a []string into [04:47] [04:48] where ... is whatever []string marshalls too in xml [04:48] yes I was expecting [04:48] * davecheney scratches [04:48] head [04:48] i dunno if that is valid xml [04:48] two secs [04:48] might not be [04:48] i don't think it is [04:48] but it's a shit poor error message in that case [04:48] what I really wanted was an array of [04:49] thanks for looking [04:49] bigjools: np [04:49] * davecheney ponders http://www.w3.org/TR/REC-xml/#attdecls [04:49] it's been a while since i've had to read this train wreck [04:50] XML is the spawn of satan anyway [04:50] bigjools: so youi wanted [04:50]

[04:50] [04:50] [04:50] yes [04:50]

[04:50] I have that now [04:51] ok [04:51] by making a sub-struct array [04:51] yup, that is the solution [04:51] the impdendence mismatch between XML and Go is pretty high [04:51] Gustavo did some work to make it less crap before 1.0 shipped [04:51] it has raised some eyebrows [04:51] but it's still margian [04:51] marginal [04:52] you can't use attr with chains, either :( [04:53] chains ? [04:53] xml:Node1>Node2 [04:53] there is a syntax [04:53] it is certainly non intuative [04:53] ie `xml:"Node>SubNode,attr"` [04:54] i think that is a bridge too far [04:54] i don't go in that package much [04:54] * bigjools idly wonders if xpath would have been less of a shit sandwich [04:55] I believe our man in Brazil might have a solution to that [04:55] http://blog.labix.org/2013/06/07/efficient-xpath-for-go [04:55] not tried yet [05:03] review incomming $ bzr diff -r branch::parent | wc -l [05:03] 2315 [05:04] 59 files changed, 348 insertions(+), 363 deletions(-) [05:06] * davecheney shrieks [05:06] this branch has a high likelyhood of clashing with someone [05:06] i guess that is how you got your nick [05:06] davecheney: I bit the bullet for jam [05:07] davecheney: this branch kills state.Tools [05:07] ; [05:07] and makes everything use tools.Tools [05:07] and moves agent/tools.go into tools/agent.go [05:07] lbox is currently working out the diff [05:08] jam: I wanted to mess around with tools a little [05:08] jam: in particular, uploading pre-built tools for the local provider [05:08] into the local storage [05:08] so I needed to refactor a few methods anyway [05:08] seemed stupid to do it in the old place [05:08] Rietveld: https://codereview.appspot.com/11028043 [05:08] thumper: per the request from fwereade he really likes the idea of an "agent" top level package, rather than a "tools" one. [05:08] entirely mechanical [05:09] So I was going to pull environs/agent top level and replace tools with it. [05:09] Though still have agent/Tools instead of state/Tools. [05:09] jam: well... someone can change the files later [05:09] this is done [05:09] lots of tests fixed [05:09] jam: well, the work is half done [05:09] baby steps, no? [05:10] I have to admit, the review looks impressive [05:11] thumper: rather than having to do 2 massive conflicty changes like this, can you just 'bzr mv tools agent' and then use 'agent.Tools' everywhere in your patch, but otherwise leave it the same? [05:11] We don't have to move the rest of the agent stuff [05:11] just call it agent.Tools so we don't have to have a mass rename again. [05:11] thumper: or would you rather see tools.Tools *and* and agent/* package? [05:12] jam: there are lots of other tools methods too [05:12] jam: I'm happy to have agent [05:12] jam: as long as it doesn't bring in dependencies on environs or state [05:12] thumper: I definitely want to nuke state.Tools, I'm just trying to figure out what the best we can replace it with for now, so we can hopefully avoid changing 50+ files again :) [05:12] jam: just moving the dir will cause more problems [05:13] as there are some places that have environs/agent and tools [05:13] thumper: right, so don't move the dir. This is more about "replace state.Tools with something better" and agent/ as a top level package probably makes more sense than tools/ (I think) [05:13] if you want s/tools/agent/ it will have to wait for tomorrow [05:13] thumper: np, I'll look at your patch and bring it up with William today, so we can settle it out. [05:13] I sort of like "tools.Tools" [05:13] I think agent.Tools makes more sense IMO [05:13] but I think there is more than just tools that we want to cluster there. [05:14] jam: except heaps of tests has local variables called tools [05:14] thumper: yeah [05:14] and Tools is pretty generic [05:14] * thumper nods [05:14] so agent.Tools makes it clear it is the jujud stuff [05:14] not just any thing [05:14] I have to go put on some veg and rice [05:14] later folks [05:14] thumper: later === tasdomas_afk is now known as tasdomas [07:19] jam: Hi John… sorry to bother you with this again… but could you please… update lp:gwacl again :) (This time I'll send an email to the list as well because we won't update gwacl in the next few days… hopefully). [07:21] rvba: updating from 166 to 168, does that sound correct? [07:22] jam: yep. ta. [07:23] rvba: for https://code.launchpad.net/~rvb/juju-core/az-destroy/+merge/173657 this is one of those cases where with mocking, if something changes will we notice? [07:25] rvba: looking at something like "buildDestroyServiceResponse", you are describing what gwacl does under the covers, which certainly feels like a test that should be in gwacl. [07:26] Would it make more sense to mock the gwacl layer itself, and assert what functions in gwacl are being called? [07:29] jam: as I said on the MP, checking which requests have been done is indeed a tradeoff: it forces you to know what gwacl does under the covers to some extend… but it's better than mocking the gwacl layer because if a method changes it's behavior in any significant way, the tests will fail. [07:30] rvba: but isn't that what the gwacl tests are for? It feels like if you have to fix something in gwacl, causing it to change its behavior, you then have to fix the tests (without actually fixing the juju-core code). [07:31] (eg, a False Positive test failure) [07:33] jam: that's right, it's a bit clumsy in that regard… but I'm not sure mocking the gwacl layer entirely is feasible. [07:34] rvba: well 'feasible' means it would probably have to be designed from the start as 'interface' things that you then swap out for something different. [07:34] Not necessarily feasible at this point. [07:34] jam: hey [07:34] you can't really monkey patch in go, so you need to build in the abstraction layer [07:34] hi dimitern [07:34] Right. [07:34] We've used that pattern in some cases. [07:35] rvba: though to be fair, often abstraction layers leave you with really nice decoupled and coherent code :) [07:35] jam: i don't think the changes to deployer would've caused the issues reported upgrading 1.10->dev [07:35] jam: i tested that specifically live on ec2 [07:38] dimitern: It was more being hopeful that we had an answer to why upgrade failed rather than thinking it would really fix it. [07:39] jam: hmm.. actually come to think of the tests i did - it was 1.11 -> 1.11.2, so don't know about 1.10 -> 1.11.2 [07:39] jam: the problem with the design you suggest is that, ultimately, it means you are creating a test double in the provider's code. So it might work nice and clean (compared to the testing strategy we have in place now) at first and then turn into a hairy piece of code that is awkward to maintain, especially in the provider's testing code (as opposed to on gwacl's side). [07:40] rvba: sure, I'd certainly rather see a hypothetical double exist in gwacl than maintained in juju-core. I'm just trying to get a handle on what the tests add vs failure modes, etc. [07:40] r [07:40] rvba: if you mock at the wrong level, all you end up with is "the code looks like the code" [07:41] at the same time, you do want to end up with "things are hooked up in a sane manner". [07:41] jam: Don't get me wrong, I think you've got a valid point. But what we've done here is a tradeoff that stemmed from our experience with the MAAS provider where we decided to create a full test double in the GO MAAS lib. [07:42] The testing code of the provider was nice an clean. But the amount of work required to develop and maintain a test double in gomaasapi wasn't worth it. [07:42] At least that was our conclusion :). [08:00] rvba: reviewd: https://codereview.appspot.com/11034044/ [08:01] mgz: poke [08:01] jam: ta! [08:46] jam: hey [08:46] mgz: wow, I actually get to see you during a day :) [08:46] well, hear from you at least [08:47] yeah, trying to get ack on earlier footing [08:47] *back [08:50] fwereade: ping [08:50] dimitern, pong [08:50] fwereade: you wanted to talk about the deployer api stuff? [08:50] dimitern, ah yeah [08:50] dimitern, can we do a g+in 10 mins or so please? [08:51] fwereade: sure, just ping me when you're free [08:52] fwereade: State.Sync() vs State.StartSync() can I get some context as to what they mean? [08:53] jam, both cause the state to get the latest content from the transaction log [08:53] (what the difference is in practical) [08:53] jam, StartSync returns immediately; Sync doesn't return until all events have been delivered by the watcher package [08:53] fwereade: so I think I understand that Sync() waits for it to complete before you do your assertions, and StartSync does not. [08:54] fwereade: but I don't quite understand why worker/uniter/filter_test.go uses both [08:54] jam, note that this does not *necessarily* imply that all consequences have come to pass, though [08:54] jam, we usually StartSync before things we don't expect to happen and Sync before things we do [09:02] fwereade: is there a way to write a generic fuction that can be passed a "<- chan Type" ? [09:02] I know I can write <-chan interface{} [09:02] But I don't think you can pass a <-chan Type as that parameter [09:02] right? [09:03] jam, afraid not, as far as I'm aware === _mup__ is now known as _mup_ [10:24] dimitern, hey, I forgot to say I merged api-use-entities; did you have a moment to review api-life-result? [10:29] fwereade: just reviewed [10:30] dimitern, cheers dude [10:30] fwereade: it seemed a bit weird not to have Machine.Refresh() anymore [10:30] dimitern, if it's not used, I don't want it cluttering up the api [10:31] dimitern, cost of reimplementing it one day is dwarfed by the mental load on everybody of keeping it around [10:31] fwereade: it was to be used by the provisioner I think [10:31] fwereade: but yeah, fair point [10:31] dimitern, I *think* the usage profile is different there [10:31] dimitern, and will probably go by another facade regardless [10:32] fwereade: yeah [10:32] fwereade: provisioner also uses WatchEnvironConfig [10:33] which I know you wanted to change, though *I* don't immediately know how to restrict it. [10:33] jam: but that's bound to change [10:33] jam, all I expect is that we'll use your WatchFor watcher and directly get the EnvironConfig when it's needed [10:33] jam: I think the idea is to have a notifywatcher for that and then call EnvironConfig() to read it when changed [10:34] dimitern, and jam's already implemented that watcher ;) [10:34] fwereade: yep! [10:35] fwereade: fairy nuff [10:35] fwereade: I'd like to get your input on: https://codereview.appspot.com/11035043/ [10:35] As far as having a testing/* object for testing how a channel is operating [10:36] (akin to state/testing/NotifyWatcherC but for things that aren't Watcher objects.) [10:36] In theory NotifyWatcherC could be implemented in terms of the testing/* object, but I didn't go that far) [10:41] jam, I think that's very nice, even if I needed to reread curryResolvedMode when I encountered it [10:41] jam, I'd be +1 on extending it in that direction [10:41] fwereade: from what I can tell, go1.1 has some more Select/Chan based stuff in reflect, which would let us get around writing currying functions. [10:42] but I didn't want to dig into that until it was actually useful :) [10:43] jam, quite right too (and it'll be interesting to see how much of a difference that makes... it's nice that reflection exists, but I've never really felt the code that uses it is particularly clean or expressive) [10:43] fwereade: yeah, reflect seems to mostly gain you "1 function to handle many types" , which interface{} mostly gets you [10:44] except when it doesn't like channels [10:44] but go 1 didn't have reflected channels :) [10:45] fwereade: if you can think of a better name than NotifyAsserterC I'm happy to change it, btw. [10:45] Any reviewers available? I need a second review for https://codereview.appspot.com/11027043/ [10:46] * fwereade shrugs -- if you think of a better one then go for it :) [10:47] jtv2, ack [10:48] thx [10:48] jam: wow, one of those 100+ files +1/-0 lines reviews [10:50] dimitern: fortunately most of those +1 is a blank line :) [10:50] dimitern: might be a case where it is easier to review on Launchpad. [10:51] dimitern: I'm not too concerned about getting them all perfect, just getting them 'good' so people will crib from existing ones. [10:54] jam: i'll take a look on lp [10:54] though "j j j j j j j j j j j j " isn't too hard to type :) [10:54] but does take a lot of page reloads [10:54] jam: so not all gocheck refs are changed to "gc" [10:55] dimitern: I didn't get to that part, just changing the ordering of imports, and clarified that we want to do "gc" [10:55] that is going to touch a lot more content, rather than just imports [10:55] jam: yeah, good point [10:55] I also didn't get to the other 280 files. [10:57] fwereade: http://paste.ubuntu.com/5858056/ is what a trivial folding-into-WatcherC looks like [10:57] the primary differences: [10:57] a) It will always call c.State.Sync() even for AssertNoChange [10:57] (when not Lax) [10:58] b) The naming of AssertNoChange vs AssertNoReceive [10:58] means that it proxies the call [10:58] c) We lose the "watcher did[not do] something" in error messages [10:58] because NotifyAsserter talks about channels. [10:58] worth it? [10:59] jam: do we now have the StringsWatcher we talked about? [10:59] dimitern: no [11:00] dimitern: I thought that was on your plate :) [11:00] jam: ah, i see :) [11:00] jam: i was wondering whether to implement the UnitsWatcher in apiserver or do something more generic [11:01] fwereade: what do you think? ^ [11:01] in any case it can be done i 2 steps, which is easier for me anyway [11:01] dimitern: I think for the params adding a StringsWatcher interface, and then implementing UnitsWatcher in terms of that. [11:01] since LifeCycleWatchers are going to want it as well, right? [11:02] jam: hmm or implement UW and then extract an iface from it :) [11:03] dimitern: so the steps I did were: (1) add a state/* interface (NotifyWatcher) (2) add a corresponding api/params NotifyWatcher interface along with NotifyWatcherResult, etc. (3) add a client-side implementation of api/watcher/notifyWatcher, (4) Add a worker/NotifyWorker to consume these things into actions. [11:03] you should be able to do something quite similar for StringsWatcher [11:04] jam: yeah, except it's a little foggy what should it have right now [11:04] which might help break it down into steps to complete and get landed. I was generally working on 1-2 branches at a time to make sure that the design was reasonable [11:04] dimitern: IMO StringsWatcher interface is just NotifyWatcher with a different Changes() [11:04] jam: I should be able to copy most of the the notifywatcher, except for inital event handling and changes [11:05] dimitern: right [11:05] jam: and then LifecycleWatcher and UnitsWatcher will use StringsWatcher interface [11:05] dimitern: we *could* go with an ObjectWatcher that had: Changes() <-chan interface{} [11:06] jam: let's not do that just now :) [11:06] but generic channels are a bit hard (as seen by my recent patch) [11:06] dimitern, jam, yeah, let's make it Strings [11:06] dimitern: right: func NewLifecycleWatcher(...) StringsWatcher [11:06] IMO [11:07] fwereade: Strings? [11:07] fwereade: you mean StringsWatcher or? [11:07] dimitern, StringsWatcher over the API, yeah [11:08] dimitern: StringsWatcher rather than ObjectWatcher [11:08] fwereade, jam: oh yeah, absolutely [11:22] jtv2, reviewed [11:36] thanks fwereade [12:09] dimitern, jam: if I were to say "relation tags", what would spring to mind? [12:09] fwereade: i see where are you going :) [12:09] fwereade: I'd be making something up, but a handle for a single relation [12:09] *or* a relations endpoints defined in terms of tags [12:10] dimitern, jam: the main issue is that relation keys (may) contain spaces, and this doesn't fit well with our existing taggery [12:10] fwereade: relation-wordpress:db-mysql:server ? [12:10] fwereade: should the spaces matter? after the "relation-" prefix anything valid goes [12:10] dimitern, I fear that - is valid in relation name as well as service [12:11] dimitern, fair point, maybe spaces in tags are ok, but if feels a little bit off [12:11] fwereade: sure, but we still prefix it [12:11] fwereade: so how about @ or $ or # or some other separator? [12:11] it is true that any-given-string may accidentally match [12:11] but as long as we only actually pass a real Tag into a Tag field [12:11] dimitern, jam: I was wondering how you guys would feel about relation-%d [12:11] it must have the "start-" be unique [12:12] fwereade: ah, actually better [12:12] fwereade: not worse than machine-% anyway [12:12] fwereade: how are relations addressed currently, just by id? [12:12] jam, ha, well, they have an id and a key [12:12] jam: internally; otherwise by name [12:12] jam, the key is derived from the endpoints [12:13] fwereade: relationDoc struct {Key string `bson:"_id"`} [12:13] jam, stringed in a canonical order, separated by spaces [12:13] jam, yeah, exactly [12:13] fwereade: "derived" is the tricky part [12:13] I think that means they have a "Key" which is acutalyl an id [12:13] but I guess they also have an Id [12:13] ugh [12:13] implies ordering [12:13] jam, tell me about it [12:14] fwereade: so does that mean that "Key" as derived from endpoints is actually indexed by Mongo (because it is the _id), but Id as an int is not? [12:14] jam, it is [12:14] that would be the only reason I can think of to settle what reference to use in a tag [12:15] something that can be looked up via index [12:15] so relation-%d is fine, as long as %d is indexed, IMO [12:15] jam, Id is, er, probably not indexed, but should be [12:15] the id is unique and unambiguous [12:15] fwereade: I guess that is what our 2k service tests are for, right? [12:15] what it's not is "user-friendly" [12:16] dimitern: I don't think anyone is going to be manually entering them, are they? [12:16] dimitern, yeah, good point, I think that swings it even if it's not the nicest [12:16] They will be discovered via api requests. [12:16] jam, dimitern: and relation keys are somewhat unfriendly to enter anyway [12:16] jam: no, but these ids are only internal and never displayed (hmm maybe in hook env vars only) [12:16] fwereade: relation keys seem far harder to *type* because of the derived nature [12:16] jam, dimitern: what with having to know all names and roles and that [12:16] dam, yeah [12:17] jam, dimitern: hmm, *except* [12:17] fwereade: if there is a way to convert id to name in a tool it doesn't matter really [12:18] jam, dimitern: the LifecycleWatcher uses _id, therefore Key, and it would be nice to be able to easily convert watcher results into tags [12:18] just relation-name returns derived name and copy/paste that if you need it? [12:18] dimitern, I think the human-readability is a bit of a red herring maybe [12:18] dimitern, in charms there's a different vocabulary (sigh) derived from the id [12:19] fwereade: well, for display purposes we're not using either anyway [12:19] fwereade: lifecyclewatcher could be taught to return Tags in the api... [12:19] fwereade: we use verbose yaml dump [12:19] jam, yeah, I am wondering about that [12:19] jam, but the same consideration applies to some degree at least [12:19] jam, it should be convenient to convert them [12:20] fwereade: as in, you should be able to convert them without a db query? Though aren't they used to query the db? [12:20] jam, and with state/watcher we're stuck with _id-based watches at the lowest level [12:21] jam: you don't need a db query - you already have all the services in the relation struct, so RelationTag can be simply implemented [12:21] jam, most of them probably will be, yeah, but having to read the whole object to get an id that we can subsequently use to read it again feels, er, suboptimal [12:21] fwereade: well the API is intended as a point of Abstraction so we don't have to worry about what is in the DB quite as much. [12:21] jam, dimitern, relation.Tag is easy whatever we choose, Key and Id are both present [12:23] jam, ah, hmm, nice theory, practical concern [12:23] jam, I don't think we can depend on the db docs still existing when we want to convert [12:23] fwereade: why didn't we have relation tags before? there wasn't any need for them through the api? [12:23] jam, so we my be stuck with Key-based ones [12:24] dimitern, yeah, can't auth as *or annotate* a relation [12:24] jam, dimitern: so I think we're stuck with Key [12:25] fwereade: why would we ever need to auth as a relation? [12:25] dimitern, quite [12:25] dimitern, that's why it's not an auther [12:25] not even an otter :D [12:25] dimitern, but we never annotate them either, which is how this never got considered [12:25] ;p [12:26] fwereade: hmm.. that might be useful i guess for the gui [12:26] dimitern, jam: what's the worst that can happen if we put a " " in a tag, I wonder [12:26] dimitern, maybe, but they didn't do it because they didn't need it afaik [12:27] fwereade: the worst? Someone uses it in a shell script and it splits on the extra space. [12:27] fwereade: spaces are scary.. c'mon it's not 70s anymore [12:27] And that causes rm -rf to run on their personal machine [12:27] and that causes them to go on a killing spree triggering a nuclear war [12:27] any script should assume spaces might be present and use "" [12:28] fwereade: The only legitimate concern I would have is the charm using shell case [12:28] I don't know that relation-tag would even be exposed there. [12:28] dimitern, jam: *that* won't happen because the charm doesn't see tags [12:28] dimitern, jam: but the spectres yu raise still bug me [12:29] fwereade: can't we simply s/ /_/ or something? [12:29] dimitern, jam, hmm, maybe [12:29] both ways [12:29] dimitern, jam, I have a horrible feeling we don't restrict relation names much :/ [12:29] fwereade: it wouldnt' be guaranteed unique if you munge it. [12:29] intelligently ofc - check for existence of the result [12:30] relationship_foo and relationship foo [12:30] fwereade: +1 to that [12:30] fwereade: the transports we currently use for talking about stuff are all space-safe. [12:30] "JSON" string, etc. [12:30] shell is the only place I can think of that could be a problem. [12:30] fwereade: we should at least restrict spaces and a few other chars we could use as separators [12:31] jam, dimitern: yeah, I'll try to check that out [12:31] jam, dimitern: it's grown beyond what I should do in this CL anyway [12:31] fwereade: proposal: restrict " ", "@" and/or "#" in relation names? [12:31] jam, dimitern: will surprise us when we hit the uniter facade [12:32] dimitern, I'd much rather restrict it harder, to basically what we have in service names [12:32] even "/" as we use it already as a separator [12:32] fwereade: ah, good point [12:32] dimitern, weird to have those in rel tags when we sub them out in unit tags [12:32] give the people too much freedom in name choices and all hell breaks loose ;) [12:33] dimitern, jam: future plans: restrict charm relation names, implement relation tags with s/ /$some_forbidden_char/ [12:33] dimitern, today plans: ignore relation tags [12:33] fwereade: sgtm [13:14] fwereade, dimitern: I'd like a second pair of eyes on something with cmd/jujud/upgrade.go. It has this fancy "delayedTomb()" logic, that AFAICT isn't actually doing anything. [13:15] it is designed that if delayedtomb->Dying is set, it sleeps for a time and then calls actualtomb->Kill() [13:15] however, the delayed tomb is only a local variable === teknico1 is now known as teknico [13:33] jam: where? [13:33] cmd/jujud/upgrade.go [13:33] look for tomb := delayedTomb() [13:33] ok [13:33] I don't see anything that could try to stop tomb rather than lots of stuff that would stop Upgrader.tomb [13:36] anyway, I'm done for today. I was just trying to see if I needed to mimic that code, but it doesn't seem to actually be used. [13:36] jam: it looks fishy [13:37] dimitern: if it set self.tomb or something like that, then I could see it being used [13:37] jam: and it's used in several places as noDelay(), but i'm yet to figure out why [13:37] dimitern: it might be that I mixed the order around. [13:37] jam: hmm [13:37] So if someone calls Kill on Upgrader.Kill() it takes a while to actually call tomb.Kill [13:38] so the download loop takes 5 min to respond to a Stop request [13:39] jam: i'm not sure but in live tests i observed a lot less than that [13:39] ah, the idea is that Runner will see a task fail (which actually fails because we need an upgrade) so we still allow this task to download new tools. [13:39] dimitern: upgraderDelay is a var that is probably overridden in tests [13:39] jam: seems likely [13:39] jam: anyway, have a good evening then [13:47] jam, oh, christ, that stuff [13:48] jam, I don't think it's necessary with a sanely configured worker.Runner [13:48] dimitern, jam: you know how any error in any task takes down every task? [13:48] dimitern, jam: the kill delay was a "solution" to that problem [13:49] fwereade: i thought we were changing this with the task runners though? [13:49]