[00:06] ericsnow: ping, you still around? [00:06] wwitzel3: more or less ;) [00:07] ericsnow: our cat got bit by a dog so I had to run off to the vet for most of the day, but I'm back now if you want to hand off [00:07] wwitzel3: sorry to hear that [00:07] wwitzel3: cat okay? [00:08] ericsnow: she is well, yeah, couple set of stiches and a cone of shame [00:08] wwitzel3: if you like, you can give my cleanup patch a once over: http://reviews.vapour.ws/r/2927/ [00:08] wwitzel3: ha, cone of shame [00:10] ericsnow: that diff seems to have a lot of stuff in it, is all of this part of the cleanup? [00:10] wwitzel3: I have some comments for your LXD provider code, but that can wait until tomorrow :) [00:10] wwitzel3: well, it was smaller before fwereade made a number of (reasonable) suggestions :) [00:11] wwitzel3: but yeah, it really is that big [00:11] wwitzel3: sorry [00:11] wwitzel3: a *lot* of extra test coverage [00:11] ericsnow: what about all the stuff in apiserver/common/errors? [00:11] * ericsnow checks [00:12] wwitzel3: alas, yes [01:19] Bug #1507601 changed: TestStartTerminationWorker fails on gccgo [01:28] Bug #1507601 opened: TestStartTerminationWorker fails on gccgo [01:34] Bug #1507601 changed: TestStartTerminationWorker fails on gccgo [01:34] thumper: menn0 waigani wallyworld do either of you have a minute to review this for the release http://reviews.vapour.ws/r/2980/ [01:35] sinzui: lgtmandt [01:35] thank you wallyworld [02:00] axw: be there in 1 minute [02:00] wallyworld: me too [02:12] lololol - juju-core's complimentary commercial subscription expires on 2015-11-22. Proprietary projects can use Launchpad by purchasing a commercial-use subscription which costs US$250/year/project. [02:16] somebody just broke something in launchpad [03:21] davechen1y: ping [03:43] Can I get a review for http://reviews.vapour.ws/r/2981/ ? [03:46] * thumper looks [03:49] * thumper sighs [03:49] clicked the wrong button again [03:49] * thumper wanted the fix it and ship it button [03:51] oh blargh [04:14] kids calling [04:14] laters folks [04:14] see y'all Tuesday (public holiday monday here) [04:34] wallyworld, axw could I get a sanity check on this revision: http://reviews.vapour.ws/r/2981/ [04:35] It's late and I don't want to mess things up again :) [04:35] sure [04:40] cherylj: seems fine, although I'd probably just use a regex to match the whitespace [04:43] cherylj: sorry, just got off the phone, what was the issue? [04:45] oh, i see - version string length [04:45] axw: would you be willing to implement just using a regex? [04:45] it's midnight here and I really want to go to bed [04:45] cherylj: sure [04:46] thank you, axw. This is for bug 1509032 [04:46] Bug #1509032: Juju doesn't support is own version of 1.25.0 [04:47] or even just strip the trailing spaces [04:48] cherylj: not clear how I trigger the failure? [04:48] set juju/juju/version/version.go, const version = "1.25.1" [04:49] you have to be on 1.25 and get my latest commits from today [04:49] cherylj: I see, thanks [05:07] wallyworld: if you have a moment, http://reviews.vapour.ws/r/2982/ [05:07] sure [05:09] axw: yup, +1 [07:28] dimitern: ping [07:28] anyone around to do a very quick (and urgently needed) review? [07:29] http://reviews.vapour.ws/r/2983/ [07:39] rogpeppe, sure, looking [07:39] dimitern: thanks! [07:40] dimitern: (FWIW it follows the exact same pattern used in the other providers) [07:42] rogpeppe, ship it! :) [07:50] cherylj, any chance you're still around? [07:50] dimitern: hey there, do you have time for a chat? [07:52] wallyworld, I do [07:52] https://plus.google.com/hangouts/_/canonical.com/tanzanite-stand [07:52] wallyworld, omw [09:02] dimitern: standup? [09:03] omw [09:03] fwereade, joining standup? [10:06] wallyworld: ping [10:06] axw: ping [10:06] hey [10:06] wallyworld: hey [10:06] wallyworld: the broken upgrade issue [10:07] that old thing [10:07] wallyworld: we've worked out that *part* of the problem is that "ignore-machine-addresses" is broken for lxc containers [10:07] wallyworld: and there's an easy fix (we think) [10:07] wallyworld: so I'll create a new bug for that specifically and work on it [10:08] ok [10:08] lxc containers *only* have machine addresses, so ignoring them is bad... [10:08] yeah, containers don't have provider addresses [10:08] I've added a note to the main launchpad bug about this [10:08] just keeping you up to date [10:09] yay for easy fix - get 1.24 and 1.25 working and fix 1.26 "properly" [10:09] thanks for update, much appreciated [10:10] I'll add a reply to the rt too === \b is now known as benonsoftware [10:10] ty [10:29] Bug #1509292 opened: "ignore-machine-addresses" broken for containers [10:32] Bug #1509292 changed: "ignore-machine-addresses" broken for containers [10:41] Bug #1509292 opened: "ignore-machine-addresses" broken for containers [11:05] frobware, I think jam's comment on deploy time bindings just made my day :) [11:05] dimitern: how do I get "juju upgrade-juju --upload-tools" to work? [11:06] dimitern: http://pastebin.ubuntu.com/12901704/ [11:06] dimitern, need to work out why I don't get any doc notifications [11:06] dimitern: client is my new version, server is 1.20.14 [11:06] frobware, not actually implementing (but speccing properly) the "connects" side of the metadata, will save good 2 to 3 weeks of work before jan [11:07] voidspace, so this is coming from the apiserver I guess? try with --debug? [11:08] voidspace, if that's the case, try $ juju upgrade-juju --upload-tools --series trusty [11:08] dimitern: http://pastebin.ubuntu.com/12901718/ [11:09] this tells me you're on wily [11:09] dimitern: tried that already, --series is ignored [11:09] I'm really not [11:09] oh ffs.. [11:09] 15.04 [11:10] voidspace, juju status | grep agent-version ? [11:10] we need to solve this so peter can try the new binaries (let alone actually testing it ourselves) [11:10] agreed [11:10] 1.20.14.1 [11:11] hmm [11:11] so where's wily coming from then [11:11] presumably simplestreams [11:11] where else could it come from [11:12] it could be hardcoded [11:12] in the client [11:12] ah right [11:12] most likely, as 1.20 was pre-vivid, right? and it's not complaining about it [11:12] indeed, if I set the tools-url to something invalid I still get the same error [11:12] so it must come from the client [11:13] well, no - if I grep our codebase for wily it doesn't appear [11:13] we must have a series list that the server checks [11:14] we have indeed - version/supportedseries.gho [11:15] yeah, just found it [11:15] dimitern: and wily isn't in seriesVersions [11:15] just try adding "wily" to ubuntuSeries map rebuild and retry [11:15] dimitern: let me try adding it there [11:15] :) [11:16] voidspace, and if that works we should give both juju and jujud in the tarball for peter [11:16] dimitern: kk [11:17] hmm... still the same error [11:17] voidspace, hmm I suspect you need to do apt-get upgrade and make sure distro-info is up-to-date [11:17] voidspace, looking at the comment on seriesVersions, check: $ cat /usr/share/distro-info/ubuntu.csv [11:17] mine does show wili [11:17] wily [11:17] dimitern: on machine-0 [11:18] on your machine [11:18] mine does too [11:18] (well, try first on yours, then try on machine-0 I guess :) [11:18] this error looks like it's coming from the server [11:18] yeah - the "400" is a http status I guess [11:19] that directory doesn't exist on machine-0 [11:19] doing an upgrade anyway [11:20] nothing to upgrade [11:20] voidspace, try dist-upgrade as well [11:20] did [11:37] voidspace, well, juju set-env logging-config '=TRACE' && retry and paste the machine-0.log for the deploy call? [11:38] dimitern: I'm going to see if 1.22 has the same issue [11:39] dimitern: if I can upgrade from 1.22 that will suffice for testing [11:39] dimitern: and they do report failed upgrades from 1.22 - so they can test that scenario too [11:39] (if upgrading from 1.22 with upload-tools works) [11:41] voidspace, +1 [11:45] dimitern: seemed to work [11:45] at least the upload worked I mean... [11:45] we'll see if the issue is fixed [11:45] nice! [11:47] 1.20 is too old anyway [11:47] yep [11:50] * dimitern steps out for a while [12:14] dimitern: nope, still the same error - just reported in a slightly different way [12:15] dimitern: in fact, I can't even upgrade from 1.22 to 1.24 due to this error [12:15] dimitern: must be due to the already uploaded tools [12:15] I'll reset [12:18] dimitern: as wily is imminent I wonder if it's just recently been added to a stream / version list somewhere which is only *now* causing this problem to show [12:33] dimitern: I bet I can still test this fix anyway ("ignore-machine-addresses" with containers is probably broken even without an upgrade) [12:33] dimitern: and treat the upgrade problem as a separate issue [12:42] voidspace, seems like it it would be a good step along the way [12:59] frobware: yeah [13:21] Bug #1509353 opened: local provider service cannot be enabled after disabling [13:49] grrrrrrrrrrrrrrrrr RB apparently just ate a long reply [13:51] fwereade: that has happened to me a few times. Seemsl ike if you click on a link to look at the code, your replies go away [13:52] natefinch, dammit, thanks, hopefully I will learn from it [13:52] fwereade: but you don't really need a long reply. You can just click the ship it button on my review. I'll know what you mean [13:52] lol [13:53] fwereade: but seriously... I always open the code links in new tabs now, that seems to be the only safe way to do it [13:54] natefinch, so what happened there, I think, was that I had two concurrent replies to previous reviews open, and then remembered I also had a review, and that would be a good place to put the topp-level comments, so I pressed the "edit review" button; when I returned only one of the replies was there [13:54] :\ [13:54] thats never fun when you're working and the software is actively trolling you [13:55] still better than lp or github for reviews :) [13:56] debateable [13:56] it's amazing no one's made a significantly better review system... they all kind of suck in their own special ways. [13:57] which one you like best is mostly dependent on which suckiness you find least annoying [13:58] natefinch, ha, yeah [13:58] i feel this way about most software natefinch [13:58] lazypower: lol yep [14:00] natefinch, btw, we should talk more about tests -- I *think* that our differences on BaseSuite and in/out-package tests come from a common source [14:00] fwereade: I definitely think we both want the same thing in the end, we are just approaching it from different directions :) [14:00] natefinch, which could reasonably be summarised as "fwereade is unreasoningly paranoid and grumpy about all developers, including himself" [14:01] fwereade: ditto for me :) [14:01] natefinch, so in all cases, what you are doing is fine in isolation, but presses my buttons because it feels very easy for other people to break them unintentionally [14:02] natefinch, BaseSuite doesnn't protect against anything in your code [14:03] natefinch, but it (or maybe better, here, I think, IsolationSuite) *does* protect against the stuff other people might do in the future [14:03] natefinch, network access, writing env vars, etc etc [14:03] natefinch, and I have approximately zero faith in our collective ability to avoid doing stuff like that [14:03] natefinch, because it seems to keep cropping up [14:04] natefinch: planning time [14:04] natefinch, and log-capture-by-default STM to be cheap compared to the monstrous hassles of trying to repro an intermittent failure that doesn't capture logs [14:05] katco`: oops, coming [14:05] natefinch, and so BaseSuite/IsolationSuite feel to me like just Good Habits because they're a tiny cost for a low-probability large benefit, that still IMO has a positive expected value [14:05] natefinch, I guess now is not the best time [14:06] natefinch, read at your leisure [14:07] natefinch, similarly, testing in-package is fine if you're disciplined and don't fossilise the code around the implementation details; but in practice, over time, the tests and the code leak into one another [14:09] natefinch, and then, when you're testing out-package and you get the urge to use export_test, you can actually use that as a smoke detector for higher-level problems with your design -- very very often, an implicit dependency that you "only want to change for tests" and therefore don't make an explicit dependency [14:09] natefinch, and the urge is understandable but IMO wrong [14:09] natefinch, almost anything you need to export_test you could instead represent as explicit config [14:10] natefinch, which may appear to broaden the interface [14:10] natefinch, but I contend that it actually just shows you a truer picture of the breadth, which was always there, but some of it was hidden [14:11] natefinch, and it's only by dragging the whole set of connections into the light that we can start to make good decisions about severing them [14:12] dooferlad, you about? Can we have a quick chat about your NUCs? [14:43] I need help with actions and the jsonschema, can anyone help me out? [14:44] Mostly, what are the valid "types" in actions? [14:44] I get string, int, boolean, but what else is there? mainly, is there enum [14:46] marcoceppi_: last I checked, they're all strings [14:46] marcoceppi_: hmm, it's in the schema but not sure what they implemented in the go library for it. [14:46] frobware: dimitern: I can confirm that deploying a container with "ignore-machine-addresses" on fails with 1.24 (no need for an upgrade to demonstrate the bug) [14:46] frobware: dimitern: testing the fix now [14:46] natefinch: well, no, there are definitely ints and booleans [14:47] marcoceppi_: https://github.com/juju/gojsonschema/blob/master/json_schema_test_suite/enum/schema_1.json [14:47] natefinch: ^ [14:47] voidspace, and that is just this bug - https://launchpad.net/bugs/1509292 - or something else? [14:47] Bug #1509292: "ignore-machine-addresses" broken for containers [14:47] \o/ [14:47] rick_h__: thanks! [14:47] voidspace, great! then we can confirm the fix works, and prepare a tarball with patched binaries for peter to try on the actual HW [14:47] marcoceppi_: so the library should support it, not sure how it's exposed/used through actions [14:47] marcoceppi_: but that's the tool used so hopefully helps a bit more at least [14:48] marcoceppi_, rick_h__: https://bugs.launchpad.net/juju-core/+bug/1456703 [14:48] Bug #1456703: action-set converts everything to strings [14:48] rick_h__: that's cool, turns out I don't need enums atm, but good to know [14:48] natefinch: ah, I care slightly less about that, I was talking instead about the actions.yaml file format [14:48] natefinch: hmm, but can you do a enum of strings then? [14:49] rick_h__: not sure [14:50] marcoceppi_: probably just needs some testing to see how it actually works... but I was getting numbers as strings, which was screwing things up, but maybe display in the GUI wouldn't be affected. [14:51] jw4: ^ [14:57] frobware: correct [14:58] dimitern: well, he can try it - but I don't think he can upgrade to it [14:58] until there's a released one [14:59] voidspace, hmm.. that might be true, but I'm *sure* there's a way to fix that "invalid series "wily"" error [14:59] dimitern: you would hope so [14:59] I haven't found it yet [15:10] dimitern: frobware: the fix works [15:10] I'll add a test [15:11] voidspace, \o/ [15:12] voidspace, before you leave today, please update the original bug with links to the separate bug we filed and your proposed fix [15:12] dimitern: done the first bit already [15:12] will add a link to the branch [15:13] dimitern: doesn't look to me like IgnoreMachineAddresses is tested... [15:13] it's not called from any tests anyway [15:14] dimitern: ah, found the tests [15:14] voidspace, yeah, it's tested for newMachiner [15:16] ok, I'm starting to celebrate the weekend and the productive week ;) [15:16] dimitern: enjoy [15:17] voidspace, thanks ;) [15:33] ericsnow: wwitzel3: natefinch: not sure what's going on. looks like i have connectivity, but having trouble with hangouts [15:33] katco`: maybe log out and back in to google? [15:34] voidspace, please could you point me at your fix [15:35] voidspace, forget it, found it in the /other/ bug [15:35] frobware: just pushed a test [15:35] frobware: test needs improving - but the test passes with the fix and fails without it [15:51] anybody see general gmail connectivity issues - seem to be plagued with chrome failures most of this afternoon [15:51] frobware: hmm... not that I've noticed [15:51] I'm using FF though [15:52] voidspace, I spent at least one hour earlier in the day where things just went "aw, snap" [15:52] anyone fancy a ridiculous review? this merges a feature branch. it has no commits in that have not already been reviewed. https://github.com/juju/juju/pull/3590 [15:53] it would make my week if i could get this merged today [15:53] rogpeppe2: if they're all already reviewed we've done the merge without further review... [15:53] voidspace: awesome! $$merge$$ coming up! [15:55] fwereade: hey are you still around? [15:56] rogpeppe2: if you want to return the favour, here's a really short review [15:56] http://reviews.vapour.ws/r/2992/ [15:56] voidspace: i have 4 more minutes until i have to go... [15:56] rogpeppe2: if you can't do it no problem :-) [15:57] rogpeppe2: it's *possible* to do that one in four minutes, but you might want to spend a bit more time on the test [15:57] rogpeppe2: the code change is like 3 lines... [15:57] frobware: http://reviews.vapour.ws/r/2992/ [16:00] voidspace, I made one comment but need to change browsers... as it keeps going snap! [16:00] voidspace: sorry, small review but i'd need a while to understand the context [16:00] rogpeppe2: k, thanks [16:02] rogpeppe2: you wouldn't default to true, you'd default to the environment value and then override with false for a container [16:02] rogpeppe2: you think that would be better? [16:02] maybe [16:03] rogpeppe2: I've switched to that with a comment [16:06] rogpeppe2: oops, those comments should have gone to frob [16:06] voidspace, much better [16:06] frobware: ^^ [16:06] Bug #1509419 opened: 1.25-beta1 no matching tools error message spam [16:06] voidspace, I think you need somebody with additional +1 on the test [16:06] frobware: ok [16:07] TheMue: I think you're OCR today, if you're still around [16:07] voidspace: np [16:07] voidspace: have a good w/e [16:07] rogpeppe2: o/ [16:07] you too [16:07] TheMue: http://reviews.vapour.ws/r/2992/ === jam1 is now known as jam [16:12] can I somehow export a struct only for testing? [16:12] Bug #1509419 changed: 1.25-beta1 no matching tools error message spam [16:15] Bug #1509419 opened: 1.25-beta1 no matching tools error message spam [16:18] bogdanteleaga: the usual pattern is to export an alias to the struct in export_test.go [16:20] voidspace: yeah, but I'm getting " is not an expression" [16:20] it does work for functions [16:20] ah [16:21] bogdanteleaga: export a function that returns a struct instance? [16:21] NewStruct [16:21] bogdanteleaga: with `type SomeStruct someStruct`? [16:22] or that... [16:22] that doesn't work depending on what exactly you're doing with the struct [16:23] if it's being passed in places that accept an interface it's fine [16:23] if you're testing the struct itself it should be fine [16:25] mgz_: if you have time a quick review would be appreciated, would be nice to land it today [16:25] mgz_: http://reviews.vapour.ws/r/2992/ [16:26] voidspace: looking [16:26] mgz_: yup, that seems to do it [16:26] voidspace: I'm still mystified as to why 2992 is special [16:27] mgz_: special? [16:27] there's a little banner at the top of the page [16:27] mgz_: oh yeah [16:27] how odd [16:27] just because I'm special I guess [16:28] classic off-by-eight error? [16:28] a palindromic number? [16:28] mgz_: 2002 has a banner too [16:28] aha [16:28] and 2112 [16:30] do we support local provider on Centos? [16:30] Bug #1509419 changed: 1.25-beta1 no matching tools error message spam [16:30] natefinch: nope [16:31] mgz_: ok, I thought as much, but wanted to know for sure [16:31] voidspace: code changes look good to me, need to look at the original ignoremachineaddresses branch to work out the actual logic [16:31] bogdanteleaga: what are the plans over centos and containers? [16:32] I'm pretty out of the loop concerning that one [16:32] I know there were some talks around templates and whatnot [16:32] you'd be better off asking gabriel or alexis about that [16:33] gotcha [16:33] Bug #1509419 opened: 1.25-beta1 no matching tools error message spam [16:35] wow, juju/testing seems to depend on juju/utils [16:36] bug 1463480 [16:36] Bug #1463480: Failed upgrade, mixed up HA addresses 1.22:Fix Committed by thumper> [16:39] voidspace: okay, your change seems fine, but we really need to kill this config option [16:39] the logic around address setting is getting ridiculous [16:39] mgz_, choice leads to madness! :) [16:48] mgz_: thanks [16:48] mgz_: I know nothing about the origins of the hack [16:48] voidspace: the bug is fun reading [16:48] mgz_: ah, right [16:48] will try and dig it out [16:49] mgz_: I'll port this to 1.25 and on monday discuss with dimiter what to do about master [16:49] voidspace: sounds like a good plan [16:49] mgz_: at the moment if you use ignore-machine-addresses then containers are broken - so we can't leave it like this for long [16:54] anyone know why we've never written code to replace downed units? Seems like a pretty basic need - to keep your service at the capacity you told Juju you wanted [16:55] (automatically replace, that is) [17:12] EOW [18:10] ericsnow: ping [18:10] wwitzel3: hey [18:11] wwitzel3: otp [18:11] ericsnow: ok, let me know when you have a second [19:08] natefinch, fwiw, the minunitsworker does that, but the functionality never got exposed via the CLI iirc [19:09] fwereade: so like, you just need a flag to enable it or something? [19:09] natefinch, the api accepts minunits, it just never made it into the cli [19:10] natefinch, it happened a looong time ago [19:10] fwereade: that's crazy [19:12] natefinch, also worth remembering that our direction has been explicitly focused on providing the primitives rather than building in too much hard-to-observe cleverness [19:12] fwereade: with all the talk of schedulers from other orchestration tools, seems like people want at least some basic cleverness [19:13] natefinch, I have a notion that we were expecting to pick it up alongside --scale-with and similar things that we didn't do for those reasons [19:13] natefinch, I'm not disagreeing, but I think there are valid arguments against building them in too deep [19:14] natefinch, built-in strategies *are* confining, and tend towards pressure to add more for every scenario out there [19:15] natefinch, that said, we could do a better job of exposing attachment points for external strategies [19:16] natefinch, although... machine placement and unit placement *do* give you quite a lot of control [19:16] fwereade: seems like if we give some default behavior and allow it to be overridden, we get the best of both worlds [19:17] fwereade: for people that want it to just work, they can use the default, and for people that want to do something else, they can use something else (or just do it manually like they have to now) [19:17] natefinch, zoom out enough and that's what we do, I think -- the intention is that just spamming commands with the fewest possible args *will* get you reasonably sane/predictable results [19:18] natefinch, you can certainly make a case that there's room for in between the two extremes [19:18] fwereade: the whole point is not to require a human at a keyboard to both notice and fix problems [19:19] fwereade: especially when the most reasonable thing to do is blindingly obvious [19:23] natefinch, fwiw, in machine/unit scenarios, the trouble is that the problem is rarely that blindingly obvious [19:23] natefinch, just because juju's lost track of a unit does not *necessarily* imply that that unit has gone away [19:24] natefinch, and when we can't be sure about the nature of a problem -- which is the general case -- automated responses are risky [19:24] fwereade: that's why you let the admin decide whether or not to automatically create a new one or not. They get to decide which is worse - having one too many units/machines or one too few [19:24] fwereade: I'm guessing most admins would rather have one too many than one too few. [19:25] natefinch, for instance, the instance-replacement code that we used to have got dropped after it emerged that nobody was using it on purpose, and only consciously experienced negative effects (eg stopping an instance to do something out of band, whoops, juju started a new one) [19:26] fwereade: might require some tools - a juju pause-unit or something.... but mostly I'm looking for feature parity to our competitors... and one of the first things people always ask is "will you bring up a new unit if one goes down?" And I think we lose a lot of people when we say no. [19:31] natefinch, so, yes, given a clear definition of "down", you can at least come up with predictable rules there [19:31] Bug #1496972 opened: juju bootstrap fails to successfully configure the bridge juju-br0 when deploying with wily 4.2 kernel [19:31] fwereade: might also be a case for better introspection into our providers... juju should be able to ask EC2 if an instance is down, for example. [19:31] natefinch, but I would reiterate that when we tried it, people actively disliked it [19:32] natefinch, yeah, I can't remember how much instancepoller keeps track of offhand [19:35] natefinch, but, honestly, it's also annoyingly tricky to do usefully for any stateful workload [19:35] natefinch, storage possibly makes that better, for some forms of storage at least [19:35] fwereade: right, that's definitely the hardest bit [19:37] natefinch, and even then... I think I'm quite comfortable focusing on the atoms -- i.e. if I had to divide effort between "migration" and "deciding when to migrate" I think I'd be pretty happy putting all my eggs in the first basket [19:39] fwereade: that's true [19:39] natefinch, still, though, your point about what people are asking for remains unanswered [19:40] natefinch, if it is turning people off we should be addressing it [19:41] fwereade: I wouldn't say I have my finger on the pulse of the devops community, but I've definitely seen people ask for having the number of units maintained, at least. And if it's low-hanging fruit because the backend stuff already exists.... [19:43] natefinch, I don't think I've actually looked at the code for, uh, 30-odd months, so I can't really vouch for much -- in particular, I'm not sure it's working at the level we actually want it to [19:44] natefinch, it might mesh well with a machine-reaper for dead instances... but that is its own can of worms [19:46] fwereade: ahh, didn't realize it was quite that old :) [19:47] natefinch, something like that, yeah, round about the atlanta sprint [19:47] natefinch, it was a time of many decisions, not all of which have turned out optimal in the long run [19:48] natefinch, ...and fwiw, the machine-replacer code was gone before even then, so I suspect our audience today has little in common with our audience then [21:15] just pulled in -proposed juju, and so far so good team :thumbsup: nice work so far! [21:18] lazypower: yay! thank you :) [21:20] :D [21:20] lazypower, thanks!