=== ehw is now known as Guest9026 [01:50] wallyworld: in case you're wondering why I didn't land the simplestreams changes yet, I found a bunch of tests I forgot to update :) [01:51] ah np :-) [01:59] * thumper -> school run [02:51] wallyworld: please don't remove the names when done [02:52] thumper: ok, i saw some names had already been removed for things besides mine so i thought i'd tidy it up [02:52] looks neater :-) [02:52] * thumper fixes [02:52] it isn't how its doen [02:52] do we care about the names? [02:52] done [02:52] once done [04:48] jam: I had it as two tests, but changed it because I keep getting told to ;) I agree - will change it back to two [04:48] axw: having worked through the Uniter tests, I find them very hard to debug when things go wrong [04:48] because the Nth item in the test is failing [04:49] and the log is 1000 lines long [04:49] yeah, I find this problematic too [05:05] wallyworld: poke [05:06] for https://code.launchpad.net/~jameinel/juju-core/faster-passwords/+merge/193667 I realized that if an agent logs in with the "slow" hash, we can just rewrite it there to the fast one [05:07] (or if we want something with salt, etc, etc) === gary_poster is now known as gary_poster|away [05:33] jam: hi [05:34] hey wallyworld [05:34] you want me to look at that review again? [05:41] jam: so did you need me to do anything re: the above poke? [05:42] wallyworld: Sorry, I haven't finished responding to all the feedback, but I wanted to ask if a change seemed reasonable. [05:42] Namely [05:42] when running entity.PasswordValid [05:42] if we see that the PasswordHash in the DB is the old form [05:42] just rewrite the DB to the new form [05:42] we can trivially compute the hash [05:42] because the agents always just pass in the full password [05:43] right, i thought you were looking to do that. seems reasonable to me, since cost is trivial and it will incrementally upgrade the db [05:44] wallyworld: what about "salt" for UserPasswords [05:44] sounds like fwereade would like to see that added [05:44] doesn't seem hard, though it means adding another field to the DB [05:44] that makes sense to me too - o'd personally feel better with it [05:44] i'd [05:44] wallyworld: is it worth doing for agent passwords? [05:45] that's a harder question [05:45] also, is it worth doing something like a len(password) >= 18 for the fast version? [05:45] wallyworld: so "worth it" is just in the "so the code paths are similar" [05:45] I'm 99% sure it isn't worth it from an actual increased security [05:45] i'd like that since were relying on long enough password = hard enough to brute force [05:45] (len(password) maybe) [05:46] does the salt for agent passwords add any tangible benefit? [05:46] cf the extra complexity [05:47] i guess if the code is the same anyways.... [05:47] wallyworld: no [05:47] salt is a "prevent someone from precomputing 1B password hashes" [05:47] of known user-likely passwords [05:47] the whole point is we don't have known user-passwords for agents [05:47] yeah [05:47] so i wouldn't do it [05:48] we could still use same code [05:48] ie look up salt and use it if there [05:48] i think mongo returns empty for non existent fields [05:49] wallyworld: I'm sure we can tell and be compatible [05:50] so then, add salt for user passwords, check length of agent passwords, rewrite out of date hashes [05:50] do we pass password over the wire in plain text? [05:50] i guess we do? [05:51] wallyworld: yes. though we have a TLS connection by that point [05:51] ok [05:51] wallyworld: I know fwereade also talked about using CA signed client certs for agents [05:51] because that also helps in the case of "recovery" mod. [05:51] mode [05:52] ok [05:52] but you'll still want *some* sort of user identity token/password/thingy [05:52] yeah [05:52] because you don't want machine-1 agent pretending to be machine-0 [05:52] since machine-0 gets all the passwords [05:52] yep :-) [05:52] or machine-N [05:52] right [05:53] where N is a HA state server [05:53] wallyworld: yeah, I think recovery will need some thought about security [05:53] indeed [05:53] one can argue the attack surface is minimized by requiring a user to engage the mode [05:53] and it could even require Admin registration sort of thing [05:54] I don't know how much we want to automate all of recovery [05:54] so EOUTOFSCOPE for now :) [05:54] yep [05:54] there's lots of prior art for this sort of thing too i think [05:54] let's not reinvent the wheel [06:02] wallyworld: oh, the other bit about requiring min length of Agent passwords [06:02] is it is going to disrupt the test suite a lot [06:02] because we have tests that set the password for machine to "test-password" [06:02] which is a lot less than the 24 bytes we normally have [06:02] s/test-password/test-password1234567890 :-) [06:03] wallyworld: well that, and it *might* bite us in backwards compatibility mode [06:03] since we can't change the actual password [06:03] we can change what we *store* [06:03] but we don't have a "that Login is valid, but you need to create a new Password now" [06:04] true [06:04] utils.RandomPassword has generated 24-byte passwords for a long time now, though [06:04] i reckon it's worth trying === Guest9026 is now known as ehw === ehw is now known as Guest25962 === Guest25962 is now known as ehqw === ehqw is now known as ehw [08:47] mornin' all [08:51] fwereade: ping [08:51] morning rogpeppe [08:51] axw: yo! [08:55] rogpeppe: should everything prefer to use state.Machine.Addresses() rather than go to environs.Environ.WaitDNSName()? [08:55] I'm updating juju ssh to use the API; it uses WaitDNSName currently [08:55] axw: yes, it should definitely use state.Machine.Addresses [08:55] ok [08:56] axw: or Unit.Address(es?) when appropriate [08:56] yup [08:56] cool [08:56] rogpeppe: main problem now is that NewAPIConn doesn't pass secrets... I guess I'll do that now [08:57] axw: ah yes, that definitely needs to happen [08:57] axw: you mean, it doesn't push secrets if it's the first connection, right? [08:57] rogpeppe: yup [08:57] there's a TODO [08:59] axw: ha, the TODO just above it is very stale... [08:59] axw: i'm just wondering if there's a nicer way to do secret pushing that doesn't incur an extra round trip [09:00] rogpeppe: the API server could return an error that says "I haven't got my secrets yet"? and then we push and retry? [09:00] axw: something a little like that, yes [09:01] morning [09:01] morning TheMue [09:01] * TheMue fights with a mail backlog of one week :) [09:04] TheMue: hiya [09:04] hiya axw and rogpeppe [09:05] axw: one alternative i'm considering is that the Login response, rather than failing, returns a "lacking secrets" status [09:06] axw: that can be cached locally in the api.State and queried. [09:06] rogpeppe: ah ok. I don't know the internals well to know how separated they are... [09:06] sounds sensible [09:07] so instead of a GetEnvironment/SetEnvironment, it'd just push secrets during login [09:07] axw: yes [09:08] axw: so we might add an environ config argument to api.Open [09:08] axw: which is allowed to be nil, but if it is, then the connection will fail if it's the first API connection [09:09] axw: in that case in fact it might work well to have Login fail [09:10] axw: hmm, not sure though [09:10] axw: depends whether we want the login message to contain the environ config [09:12] rogpeppe: if it's just the first connection, why not just have the login proceed, and then have the server request the secrets (via a special error)? [09:12] then a second message [09:12] it's only once [09:13] axw: so the error implies "login has actually succeeded (despite the error) but secrets are needed" ? [09:14] rogpeppe: yeah. perhaps confusing, but that's one option anyway :) [09:14] axw: i definitely don't mind a second message to push the secrets [09:18] axw: one question that arises from this: is there ever a case where we want to allow some request to the API server *without* pushing the admin secrets? [09:18] axw: because if we push environ config with Login, that will be ruled out [09:18] axw: but that might well be a good thing [09:19] axw: because then there's no way that any client can do anything at all with an environment with no secrets [09:20] hmm [09:20] axw: the other thing that i'm thinking about is how does the server know when secrets have been pushed [09:23] axw: the most straightforward approach is simply to get the environment config and see if there are any secrets in it [09:23] axw: but perhaps there might be an environment that has no secrets [09:23] axw: ha, that's actually not a problem, i realise [09:23] rogpeppe: I thought the idea was an environ's config must be invalid if it doesn't have its secrets [09:23] axw: yeah, it is [09:23] axw: but we don't even need to create the Environ [09:23] rogpeppe: as for allowing no secrets... sounds preferable to require them always, but I don't know if there's a case or not [09:23] axw: because we've got EnvironProvider.SecretAttrs [09:23] axw: so if that returns nothing, we know that we don't require secrets to be pushed [09:23] ah yeah, I see. then we can distinguish an invalid env from one with no secrets [09:24] axw: yeah [09:24] axw: although... [09:24] axw: perhaps it might be a good plan to actually validate the environ [09:24] axw: something we can't do currently [09:25] can't? [09:25] rogpeppe: why can't we? [09:26] axw: because clients talk directly to mongo [09:26] axw: so there's nothing stopping a dodgy client pushing a bad environ config [09:27] ah ok, I see [09:27] there's no good time to do it currently [09:28] axw: the other thing that occurs to me is that we could cache "secrets pushed" in the apiserver.Server (because it can only go from false to true), meaning that any api server would only need to check once [09:28] axw: but that's just an optimisation (but one that's not possible if the secrets checking is done client-side) [09:30] rogpeppe: is all of this going to break the GUI horribly? [09:30] axw: i don't think so [09:30] axw: because AFAIK the GUI can't currently make the first connection anyway [09:30] axw: and the Login call can be changed in a totally backwardly compatible way [09:31] cool [09:38] axw: so to summarise, how does this sound? http://paste.ubuntu.com/6363731/ [09:39] rogpeppe: sounds great. [09:41] rogpeppe: are you planning to do this yourself, or are you working on other things? [09:41] axw: i'm currently oriented more towards the HA stuff - if you feel like doing this, it would be great. [09:42] sure, I will look into it (probably in the morning) [09:42] axw: just to let you know, I'm currently poking a lot of stuff underneath login (PasswordHash) stuff [09:42] it probably won't conflict, but you might want to wait a sec on it [09:42] jam: ok no worries [09:42] jam, fwereade: how does the above plan look to you? [09:42] thanks [09:43] rogpeppe: I *think* it is all unnecessary. thumper was quite keen on changing "juju bootstrap" to wait until it can connect to the API server [09:43] in which case [09:43] bootstrap does all the work [09:43] and then we don't have to do it for every API connection. [09:43] jam: i don't think that's viable [09:43] rogpeppe: because ? [09:44] jam: what happens if someone interrupts "juju bootstrap" ? [09:44] rogpeppe: they have to start over [09:44] they don't have more than 1 machine at that point [09:44] so we aren't destroying an environment that is well set up anyway [09:44] Or we allow "juju bootstrap" to start where it left off [09:45] rogpeppe, jam: the plan was to catch interrupts and takes the machine down if it's interrupted [09:46] fwereade: i'm not sure that's great actually - what if the network is down? does that mean you can't interrupt bootstrap? [09:46] rogpeppe, jam: blocking bootstrap actually has a lot of advantages -- no silly secrets dance, ability to create storage in the environment instead of the provider [09:47] rogpeppe, it just fails [09:47] fwereade: I'm a big fan, plus the fact you can give the user feedback about how far it gets [09:47] rogpeppe, don't think it's any worse than the network going down during a normal bootstrap [09:47] jam, rogpeppe: indeed, useful feedback during bootstrap is also awesome [09:47] rather than trying to do that at every "juju status" or "juju deploy" or ... etc [09:48] jam, in a sense that's just an extension of the secrets dance, but yeah, would be good to drop it entirely, no argument [09:50] fwereade: FWIW i think we can create storage in the environment instead of the provider anyway, can't we? [09:50] jam, rogpeppe: the question is *when* thumper is likely to do this, because we need some solution for the cli-api work [09:51] fwereade, rogpeppe: going back to another discussion, I'm going back to the PasswordHash stuff, and splitting it into a UserPasswordHash(password, salt) and AgentPasswordHash(password) [09:51] where we allow CompatPasswordHash, but if that succeeds, we then change the DB to set it to the new methods [09:51] fwereade: well, 'juju status' and 'juju deploy' are going to be some of the last steps we actually finish :) [09:51] rogpeppe, well, we do, for the manual provider -- but that's a blocking bootstrap ;) [09:51] we can set someone on it, even if it isn't thumper [09:52] jam: i don't think you can salt user passwords until the entire CLI is API, can you? [09:53] jam, heh, axw springs to mind given that he did the manual stuff -- it's just "make everything else work like manual bootstrap" :) [09:53] heh [09:53] jam, axw: modulo *also* needing provider storage -- or some alternative mechanism -- to store the bootstrap info [09:53] rogpeppe: so we don't salt the Mongo password, so I don't think that actually changes, it is just when someone *does* connect via the API, we look up the hash + salt. [09:54] jam, fwereade: per my email before, lack of secrets via API kinda blocks work I'm doing [09:55] I can move onto destroy-environment maybe [09:55] but otherwise, I could look at the secrets/bootstrap business [09:55] jam: currently we do hash the mongo password, but i guess we could use a known salt for that [09:56] jam: (in fact that's what we do currently) [09:56] rogpeppe: CompatPasswordHash() uses the same UserPasswordHash(password, FIXEDSALT) [09:58] jam: seems reasonable [09:59] jam: while we're about it, can we increase the password strength? [09:59] rogpeppe: 18-bytes of entropy is about 2^53 or so. We could easily go up to 24 (and get 32-byte base64 passwords). which gets us up into 2**72. [10:00] I may be wrong on the exact values [10:00] but < 2^64 today, and >2^64 with a size bump. [10:00] rogpeppe: the code itself says "we stick to 18-bytes because mongo uses md5sum anyway" [10:00] jam: i'm thinking we could usually use 256-bit random passwords and hashes [10:01] s/usually/usefully/ [10:01] rogpeppe: I honestly don't think that improves our actual security, but yes, we could make it really big. [10:02] #1 mongo is still the most critical part [10:02] as getting *that* password gives you everything [10:02] jam: yeah, but cracking the user password might give you access to other environments [10:02] rogpeppe: doesn't matter, we don't set the user password [10:03] and these passwords that we are generating are only good for a given agent [10:03] so no leakage [10:04] we're using sha512 as our hash, so we have room internally [10:04] if mongo is using md5sum [10:04] then that would be 128 bits [10:05] but 18 bytes of entropy = 2^144 (i was doing the math wrong before) [10:05] so we're already better than md5 [10:06] yeah, i thought your numbers looked weird (i thought you perhaps meant 10^53) [10:07] rogpeppe: I was doing 8 bytes == 8^8 rather than 256^8 [10:08] jam: or 2^(8 * 8) [10:08] jam: (easier just to work in bits, i reckon) [10:09] rogpeppe: so because we take the raw bits and put it into base64 encoding, the useful bits are 18 bytes, 24 bytes and 30 bytes [10:09] since those leave us with a base64 password that doesn't have '=' padding. [10:09] jam: tbh a 144 bit random password is probably ample [10:10] rogpeppe: for our attack surface I think it is more than ample myself [10:10] jam: that's not gonna be the way that someone breaks into our system [10:10] rogpeppe: yeah, I was considering it when my math was bad, because 2^64 isn't great security. but 2^144 is perfectly fine [10:11] jam: yeah [10:37] fwereade: are you around for a chat about the HA stuff? [10:37] fwereade: ah, it's standup in a mo actually [10:46] rogpeppe, mgz, fwereade: standup [10:46] ta [10:46] fwereade: standup? [10:46] fwereade: https://plus.google.com/hangouts/_/calendar/am9obi5tZWluZWxAY2Fub25pY2FsLmNvbQ.mf0d8r5pfb44m16v9b2n5i29ig [11:50] * TheMue => lunch [12:32] does juju support tokens in config.yaml? I.e. in cases where one value is dependant on another one (like base path and subfolders)? [12:38] tasdomas: nope [12:38] mgz, right, thanks === gary_poster|away is now known as gary_poster [14:16] sinzui: did thumper fill you in on the lxc/local-provider developments? [14:16] No, but I saw the branch merge [14:17] sinzui: I did a test run after the branch merged and the local provider still failed, but I did not have time to check whether it was the same issue as before. [14:17] abentley, are you using mysql in the test? [14:17] sinzui: Yes. [14:18] rogpeppe1: fwereade: https://code.launchpad.net/~jameinel/juju-core/faster-passwords/+merge/193667 has been updated. It now sets a Salt for User passwords and uses clearly denoted AgentPasswordHash vs UserPasswordHash vs CompatPasswordHash [14:19] jam: looking [14:19] I haven't had a chance to test live upgrades, but I have every belief things will JustWork [14:22] jam: one thing that occurs to me as *potentially* useful in the future, if we have many "users" that are actually agents, is that if we've generated the admin secret automatically (i.e. it's got lots of entropy) we could eschew the salting. [14:22] jam: something to think about for the future, perhaps [14:24] rogpeppe1: well, eventually we'll get real users [14:24] we do often generate admin-secret today [14:24] jam: indeed. [14:24] but, meh, salting is cheap, I'm only looking to change this stuff for the AgentPasswordHash changes [14:25] jam: salting is cheap, but UserPasswordHash is not. at some point in the future, we *might* come to a situtation where we've got many agents (the GUI is one example) that reconnect when an API server goes down, and hence use lots of CPU resources when doing so [14:26] jam: so, i guess i'm not really talking about the salting per se [14:26] jam: anyway, it was just a thought that occurred to me; ignore me :-) [14:28] sinzui: my test was invalid, because it started with 1.16.x, which isn't expected to have a fix yet. [14:28] ah [14:29] abentley, we could add stable branches to the test? juju/1.16 will build a 1.16.3 client and server [14:30] sinzui: Certainly. Did that temporarily for thumper yesterday. [14:32] sinzui: Are you in the stand-up? I switched my urls around. [14:33] jam: why is environs/cloudinit.go using CompatPasswordHash ? [14:34] rogpeppe1: that is the old "use the hashed password until we can use the real one" [14:34] abentley g+ is asking me to juggle three identities [14:34] sinzui: Fun. [14:34] jam: why can't we use a salted password there too? [14:35] rogpeppe1: that would require changing what we pass to cloud-init, I think [14:35] jam: (after all, it's actually one of the most insecure places that the admin password is kept) [14:35] jam: yes, it would. [14:35] jam: is that a problem? [14:35] which is something I wasn't as comfortable with because it isn't hidden behind the api [14:36] jam: i'm not that comfortable seeing "Compat"PasswordHash being used in a place where it looks like it will not be deprecated. [14:36] rogpeppe1: regardless, the data we write to cloud init gets rewritten anyway, [14:36] jam: how do you mean? [14:36] rogpeppe1: I'm fine changing the name back, or having multiple names for the same thing. [14:36] rogpeppe1: once an agent is up, it resets its password [14:37] so it won't match what is in cloud-init [14:37] and bootstrap changes the admin password back to the real password rather than the hashed password [14:37] jam: but the admin password is still hashed in cloud-init, no? [14:38] jam: so someone that gets access to the cloud-init data (probably not too hard) can still brute-force the non-salted admin password AFAICS [14:38] jam: if we *are* going to salt user passwords, i think that's probably one of the most important places to do it [14:40] rogpeppe1, jam: any place we *can* salt passwords, we should. It's not computationally expensive, and even if we're not too worried about that vector of attack, it certainly can't hurt. [14:40] rogpeppe1: so I dont think anything I've done precludes us adding salt there, and I think bootstrap is particularly a place where it is easy to break compatibility accidentally [14:44] jam: yeah, it *would* mean you couldn't use a new juju to bootstrap with old tools [14:45] jam: but perhaps you could change the occurrences of CompatPasswordHash to call UserPasswordHash with the known constant salt - then it's more obvious what's going on, perhaps. And a TODO in the code would be nice too. [15:19] jam: you have a review [15:21] sinzui: As we move to have more sets tests running, we'll want to have multiple versions of juju running concurrently. Which makes me think we need to chroot (not lxc because that would break local provider). [15:23] abentley, I understand [15:39] I am getting a strange panic when running go test on juju-core/worker/uniter [15:39] http://pastebin.com/ER6GUuza [15:40] (this is juju-core trunk) [15:42] sinzui: just realized what time it is. are we 1x1ing? [15:44] jcsackett, sorry, had another meeting === rharper_ is now known as rharper [16:03] sinzui: AFAICT, it's impossible to run "make clean", because "go list -e -f '{{.Dir}}' launchpad.net/juju-core" doesn't find anything. I could do "bzr clean-tree --unknown --ignored" [16:15] abentley, the make-recipe-and-package script creates a new directory with the revision in the name, so I don't understand how we could be reusing a built tree. [16:17] sinzui: Yesterday, I was testing 1.16 in the tree normally used for trunk. With bad luck, a 1.16 revno could match a trunk revno. [16:18] ah@ [16:18] abentley, I have experienced that just after I created the 1.16 branch [16:20] sinzui: I also trigger the tests manually, without waiting for the revno to update, but that's probably less of an issue. [16:23] jcsackett: are you around? can we talk in a bit? === natefinch is now known as natefinch-afk [16:47] sinzui: I'm adding the clean-tree anyway to reclaim disk space. [16:47] Yay [17:53] bac: sorry, i missed your message earlier. i can chat now, if you like. [18:01] hi jcsackett [18:01] now is good [18:02] bac: g+? [18:02] jcsackett: https://plus.google.com/hangouts/_/72cpjm0fnhduq36l7pim15v0mk?hl=en === natefinch-afk is now known as natefinch [18:22] sinzui: can you join in on the above g+? ^ [18:29] sinzui: nm. [18:40] * rogpeppe1 is done for the day. [18:41] g'night all [19:37] sinzui: would you have a moment to review my one-line migration script? i'll get someone else to look at the rest. https://codereview.appspot.com/21790045 [20:06] sinzui: can you look at https://code.launchpad.net/~jcsackett/charmworld/rollback-422/+merge/193999 today? [20:33] bac, jcsackett I can start the reviews now [20:33] sinzui: awesome, thanks. [20:33] sinzui: please do jc's first [20:42] jcsackett, r=me [20:43] sinzui: thanks. [20:45] bac: i'll ping you when i've qa'ed it on staging. [20:45] sinzui: morning [20:45] sinzui: thoughts on 1.16.3? [20:47] bac: LGTM. Thank you for updating the migration template [20:47] cool. thanks for looking at the migration stuff sinzui. [20:48] jcsackett: any problem with me landing my branch now or do you want me to wait? i can walk the dog now and do it later [20:48] bac: I am just happy we remember that es-update is automatically run for us [20:49] bac: i don't think our branches collide, so it should be fine. [20:55] thumper, These bugs are fix releases in stable. they are fixed in trunk. I think I can mark these as fix released because everyone has the fix https://bugs.launchpad.net/juju-core/+bugs?search=Search&field.importance=Critical&field.status=New&field.status=Incomplete&field.status=Confirmed&field.status=Triaged&field.status=In+Progress&field.status=Fix+Committed [20:56] sinzui: well, kinda, are we going to roll out 1.17? [20:56] should we make them fix released when we release them? [20:57] I was think of doing it this week, but doing 1.16.3 might exhaust me [20:57] well, bug 1246556 is in 1.16.2 [20:57] <_mup_> Bug #1246556: lxc containers broken with maas

[20:57] so I think that should be fix released [20:57] hmm, I see the 1.16 task is released [20:58] personally I'd rather not have them marked fix released until we have a release with the fix [20:58] fix committed is enough to say they are in trunk [21:00] thumper, I think only your addition last night is unreleased as well as the complicated maas bug [21:01] sinzui: I was concerned that we'd break people in a charm school at ODS [21:01] as the local provider is broken for everyone [21:01] due to an old mistake and a precise update [21:01] bac: qa-ok. [21:02] bac: how did you cleanup the bad old review jobs last time? my terminal-fu is weak, and we have processes that won't die. [21:02] thumper, yes, but not every charm is affected. I wont push back on the releases, but each releases of stable delays other work. It took days to get 1.16.2 to every place it had to be [21:03] sinzui: hmm, not every charm, but the local provider is broken now [21:03] for everyone except those compiling trunk [21:04] no install hooks complete [21:04] because apt is left in an incomplete state [21:04] look at https://bugs.launchpad.net/juju-core/+bug/1240709. I think this is fix released for everyone because the juju-core maintains stable and devel trees. [21:04] <_mup_> Bug #1240709: local provider fails to start

[21:04] jcsackett: it was in such a sad state i just rebooted the charmworld instance [21:06] thumper, I typed into the to wrong channel 30 minutes ago: thumper I can do it. I was hoping that that CI would be up today. but abentley and I have had to rethink the branch + revno tactics used with jenkins [21:06] * thumper nods [21:06] Normally I wouldn't be too concerned, but with ODS and charm school... [21:06] could look real bad [21:07] thumper, remember that the release takes between 8 hours and many days [21:07] :( [21:07] I don't want to be rushed it it has to be rushed because we don't control the builders and the copy step just adds more hours to the release [21:09] thumper, If I had off a package in a few hours, I can hope that I have something to republish to end users in the first hours of my morning [21:09] s/had/hand/ [21:09] sinzui: I think we wouldn't need to push new tools [21:09] sinzui: as 1.16.3 should get 1.16.2 tools [21:10] which would work fine [21:10] the only change is in the local provider cloud-init config [21:10] because juju-core (client and server) are deployed by the client on the same machine? [21:10] yeah [21:11] the local provider always pushes local tools [21:11] it is horrible [21:11] and we should fix it to be nicer [21:11] but it works for now [21:11] thumper, this is an awkward position to take because we have gone out of our way to ensure the jujuds are published before users get the clients [21:11] :) [21:11] The number mismatches are scary [21:12] I understand [21:12] was just trying to make things easier [21:12]