| * thumper looks | 00:00 | |
| thumper | sinzui: no. sorry | 00:02 |
|---|---|---|
| thumper | sinzui: but I have felt that our upgrade story is too flakey | 00:03 |
| thumper | I have plans (in my head) | 00:03 |
| thumper | and half written down | 00:03 |
| sinzui | yeah. Sometimes I need to wait forever for it to happen. I cannot predict if it is slow or will fail | 00:03 |
| sinzui | but I think the 14 -> 16 upgrades are better than 12 -> 14 | 00:04 |
| thumper | :) | 00:05 |
| === bjf is now known as bjf[afk] | ||
| === gary_poster|away is now known as gary_poster | ||
| === gary_poster is now known as gary_poster|away | ||
| axw | thumper: I'm updating the API to support pushing secrets now. I was chatting with rogpeppe yesterday, and we discussed having a negotiation thing during login. Then fwereade and jam reminded us that you were keen on making bootstrap blocking, and secrets-pushing can go there | 03:52 |
| axw | so... I'm just going to add an API call to push secrets. sound ok? | 03:53 |
| axw | we'll unconditionally call it when making an API connection for now, and later just during bootstrap | 03:53 |
| thumper | sure | 03:53 |
| thumper | and yes we want bootstrap synchronous | 03:53 |
| thumper | it would solve all manner of bootstrap issues | 03:53 |
| thumper | sure | 03:54 |
| thumper | wallyworld: FYI - https://codereview.appspot.com/22320043 | 04:04 |
| wallyworld | thumper: looing | 04:05 |
| wallyworld | looking even | 04:05 |
| * thumper goes to help with dinner, then swim | 04:05 | |
| thumper | I'll be back later this evening | 04:05 |
| === thumper is now known as thumper-afk | ||
| wallyworld | +1000 to synchronous bootstrap. a lot of us have been wanting that for ever | 04:06 |
| wallyworld | so long as there's feedback as it executes | 04:06 |
| davecheney | WHOOOOOOOOOOOOT! | 04:15 |
| axw | ooh | 04:16 |
| axw | yay for output :) | 04:16 |
| sodre | ping ? | 06:13 |
| sodre | what is the standard way of including charmhelpers in a charm ? | 06:21 |
| sodre | i.e. should I branch charmhelpers and check it in together with the rest of my charm? | 06:22 |
| bradm | sodre: thats what I've done in my charms, but I don't know if its considered best practice | 06:23 |
| sodre | bradm: thanks. | 06:28 |
| jam | sodre: from everything I've seen you copy charmhelpers into your code | 06:35 |
| sodre | jam: alright :) That's how I'll do it then. | 06:36 |
| jam | axw: I don't think we want to call it uncondionally during api.Open | 06:38 |
| axw | jam: why not? | 06:40 |
| jam | axw: we only need to pass secrets 1 time, rather than the 100 times you ever call status/etc8 | 06:40 |
| axw | jam: sure, but detecting that requires feedback from the server - seems like a lot of work for something that's going to go away soon | 06:41 |
| axw | we currently go to state each time we connect to see if we need to do it | 06:41 |
| axw | oops, race dector'd my machine | 06:48 |
| axw | detector'd even | 06:48 |
| jam | race decor'd sounds more interesting | 06:48 |
| axw | jam: if you replied, I missed it. last message I have is "we currently..." | 06:49 |
| jam | no, I didn't say anything | 06:49 |
| axw | ok | 06:49 |
| axw | jam: I realise doing it every connection sucks, so if there's a non sucky way, I'm open to suggestions | 06:50 |
| jam | axw: is it that hard to add an extra field to Login that says now that you've logged in, can you send the credentials | 06:50 |
| jam | ? | 06:50 |
| axw | jam: the Login code knows nothing about environments or secrets. It's probably not hard, just felt a bit wrong. I don't know that area of the code well, though | 06:52 |
| axw | well, it knows the state object of course | 06:52 |
| jam | axw: so I think we need a state/api/client level interface that supports passing secrets in, no doubt about that. | 06:53 |
| axw | jam: ah, you're suggesting a result from Login that says the next call should be PushSecrets? | 06:55 |
| jam | axw: looking at Login today, all it can return is an err | 06:55 |
| jam | and if we changed *that* behavior | 06:55 |
| jam | it would break things like the GUI | 06:55 |
| axw | yup | 06:55 |
| jam | I suppose that might be ok, since the GUI can't ever send secrets? | 06:55 |
| axw | one idea that came up was returning a special error | 06:55 |
| axw | right | 06:55 |
| jam | so you have to always connect via another means 1st time anyway | 06:55 |
| jam | I'd really like to run it by gary_poster|away or someone else on his team | 06:56 |
| jam | I think they're all mostly US time based | 06:56 |
| axw | ok | 06:57 |
| jam | axw: so the Client api for pushing up secrets is needed regardless, we'll likely use it in bootstrap, etc | 06:57 |
| jam | axw: do you know about "directory.canonical.com" ? | 06:57 |
| axw | jam: yeah, I was thinking you were talking about the idea of adding secrets into the Login call itself, which I think is too much work | 06:57 |
| axw | jam: sure do | 06:57 |
| jam | so frankban is the only one in EU timezone. Huw is in AU, but he's more Web/UI focused. | 06:58 |
| jam | axw: so I wouldn't put the secrets *into* Login, because I would avoid putting them on the wire as much as possible | 06:59 |
| jam | I was hoping Login had a nice way of telling you something after you login | 06:59 |
| jam | rather than only returning an err | 06:59 |
| jam | and that would give us a way to extend it with "please upload secrets" without telling clients to shove off | 06:59 |
| jam | err => most clients will probably just disconnect right away | 06:59 |
| axw | jam: yeah. I think GUI requires it to be there already, but I can check with gary_poster|away | 07:00 |
| jam | axw: I actually don't think it strictly does, though to install the GUI you would have had to pass the secrets | 07:00 |
| jam | because, you have to "juju deploy" the gui | 07:00 |
| jam | so actually, I think we're ok there | 07:00 |
| axw | well, sorry, that's what I meant | 07:00 |
| axw | implicitly requires :) | 07:01 |
| jam | sorry I'm mostly catching up to something that you've probably thought more about | 07:01 |
| jam | so I think short term putting the err return in Login is ok | 07:01 |
| axw | that's cool | 07:01 |
| jam | because we don't have much juju client stuff that connects via the api in older versions | 07:02 |
| jam | just add-unit and get | 07:02 |
| jam | and you are very unlikely to call either of those | 07:02 |
| jam | until you've run some other command | 07:02 |
| jam | which is why we could get away with it :) | 07:02 |
| axw | hehe | 07:02 |
| jam | axw: it *was* why I picked those. That and they were pretty straightforward. | 07:02 |
| jam | axw: if you don't mind investigating a bitd | 07:02 |
| jam | bit | 07:02 |
| jam | can you see if we can add a return parameter to Login that isn't an err? | 07:03 |
| jam | and see if old clients complain about it | 07:03 |
| axw | jam: sure, will do | 07:03 |
| jam | axw: because if we can do that compatibly, I can see a use case for doing other things at Login time | 07:03 |
| jam | giving hints about compatibility, etc | 07:03 |
| axw | I'll get the new API call working first, then go onto the Login stuff and look at that first | 07:03 |
| jam | so adding a struct to the return will be generally beneficial | 07:03 |
| jam | but if we need to go the err route, I think that's ok | 07:04 |
| axw | sounds good | 07:04 |
| jam | axw: I would even go as far as propose the API call so that we can land it | 07:04 |
| jam | so you don't have to have that pending in a branch | 07:04 |
| jam | unless you think it might need tweaking based on your other results | 07:04 |
| axw | we'll see, tho I expect the only thing that'll change is where it's called | 07:05 |
| jam | (one thought, bootstrap doesn't ever connect as a Client today, so it might just push the secrets direct into the DB, though the fact that we want bootstrap to hang around until the API is up, maybe it is sane to have it do the Client login as well...) | 07:05 |
| jam | axw: in which case, it would just be the first thing that logged in | 07:05 |
| jam | axw: I don't really like it being an err as a general quality thing | 07:06 |
| axw | yeah that did occur to me - we could just leave that bit alone once it's implemented for the general case | 07:06 |
| jam | but I could live with it | 07:06 |
| axw | yup, fair enough | 07:06 |
| axw | it's not an error, after all | 07:06 |
| jam | axw: so looking at state/api/Login today, it just passes "nil" for the response struct | 07:07 |
| axw | jam: yeah, but does adding one after the fact break the clients? that's what you're asking me to investigate, right? | 07:08 |
| jam | axw: right. I'm poking around right now :). If resp == nil ; resp = &struct{}{} and then we still call conn.codec.ReadBody(resp, isRequest) | 07:09 |
| jam | so it looks like | 07:09 |
| jam | current clients will still consume a body response, and just throw it away | 07:09 |
| jam | which is fine | 07:09 |
| axw | cool | 07:09 |
| jam | axw: but we should probably test that in practice :) | 07:09 |
| axw | and clients ignore extraneous fields, right? | 07:10 |
| jam | axw: generally, yes | 07:10 |
| jam | JSON.Unmarshal loves throwing away unrecognized things | 07:10 |
| axw | heh | 07:10 |
| jam | axw: AFAICT there's no way to pass it a Strict flag | 07:10 |
| * jam => away for lunch and groceries | 07:13 | |
| axw | later | 07:14 |
| jam | axw: to summarize before I go. It looks like putting it as an extra parameter is going to be just as much work and just as compatible with the existing clients, rather than changing the Err, so I'd rather do that. | 07:15 |
| jam | but, as always, testing with 1.16 against a running patched 1.17 is probably worthwhile | 07:15 |
| axw | jam: agreed, I will go with that. | 07:15 |
| axw | (and test) | 07:15 |
| dimitern | morning all | 07:48 |
| === thumper-afk is now known as thumper | ||
| thumper | fwereade, jam: do you guys have time for a catch up hangout? | 07:54 |
| fwereade | thumper, sure | 07:58 |
| thumper | fwereade, jam: https://plus.google.com/hangouts/_/76cpjgbfmbsvbg46v349h6l5ug?hl=en | 08:01 |
| wallyworld | dimitern: hi there. how close are you to finishing the get-environment migration to the api? | 08:32 |
| dimitern | wallyworld, get-env is done, finishing set-env now and proposing both | 08:35 |
| wallyworld | dimitern: great :-) cause i need the ability to get environ config via api and i figure your work will include that | 08:35 |
| dimitern | wallyworld, yeah | 08:36 |
| dimitern | wallyworld, basically, you can use client.api.state.EnvironConfig to get it, where client is the "c" receiver of the method | 08:37 |
| wallyworld | ok, thanks | 08:38 |
| wallyworld | i need EnvironConfig() exposed in state/api/client | 08:39 |
| axw | wallyworld: is that for add-machine? | 08:39 |
| wallyworld | yeah | 08:39 |
| wallyworld | the manual provisioning | 08:39 |
| axw | ah | 08:39 |
| * wallyworld -> dinner, bbiab | 08:41 | |
| jam | fwereade: I take it you and tim already finished up? | 08:54 |
| jam | wallyworld, dimitern: I think we need to think reasonably carefully about exposing all of EnvironConfig via the Client API. Because it ends up having environ credentials which we don't want to expose to something like the GUI (I think). | 09:05 |
| wallyworld | jam: i need it to create an environs.Environ instance. we could audit the usage of that to see what's really needed. it will be messy | 09:06 |
| jam | wallyworld: what do you need for Environ given you are doing manual provisioning? | 09:06 |
| jam | (I have the feeling it is accidentally using something it shouldn't) | 09:07 |
| jam | just so it can do something like get AuthorizedKeys | 09:07 |
| wallyworld | jam: it's used in the manual provisioning code in a few places | 09:07 |
| jam | which we ran into for the Provisioner | 09:07 |
| wallyworld | setting up auth is one such usage i think | 09:07 |
| wallyworld | finding tools is another | 09:07 |
| wallyworld | i can't recall the third offhand | 09:07 |
| jam | wallyworld: given what you've been saying about avoiding provider storage at all, is that something we are trying to get away from? | 09:08 |
| jam | anyway, I can see it being necessary today, though I'd like us to understand what we really need, because we've run into several cases where we really didn't want it | 09:08 |
| wallyworld | possibly. i'm initially looking to just do a straight port | 09:09 |
| wallyworld | agreed about seeing what we really need | 09:09 |
| wallyworld | will get messy, so looking to do it in stages | 09:09 |
| jam | wallyworld: right, the main problem with just the straight port is that we can expose secrets that we don't want to commit to supporting | 09:09 |
| wallyworld | well, we use State now to get a full env in the current code | 09:10 |
| wallyworld | so the cmd has secrets today doesn't it? | 09:10 |
| jam | wallyworld: yes, but it has straight DB access today. The issue is that this isn't the only point that uses Client | 09:10 |
| jam | GUI uses it directly, for example | 09:10 |
| rogpeppe1 | mornin' all | 09:11 |
| jam | and we are trying to support juju -the-commandline that doesn't have raw access to EC2 creds | 09:11 |
| wallyworld | jam: ok, and you are saying as soon as we add that to the api the gui could misuse it | 09:11 |
| jam | so that you can share your env in a limited fashion | 09:11 |
| jam | wallyworld: I'm not saying the GUI would, because i trust Gary et al, but it is *an* api that is leaking security in a place where we weren't before | 09:11 |
| rogpeppe1 | jam: what about the case where a GUI client actually needs to change cloud credentials? | 09:11 |
| jam | EnvironConfig specifically | 09:12 |
| wallyworld | jam: yes, that's what i meant | 09:12 |
| jam | is something we've been really careful about sharing | 09:12 |
| rogpeppe1 | jam: or in fact, and client | 09:12 |
| jam | when it came to the agents | 09:12 |
| rogpeppe1 | s/and/any/ | 09:12 |
| jam | wallyworld: I think the specific security hole is some *other* code running in the same machine where the GUI (etc) are running and getting access to secrets. | 09:12 |
| jam | certainly that was the Uniter issue | 09:12 |
| * wallyworld nods | 09:12 | |
| jam | we didn't want any charm to be able to get access to EnvironConfig | 09:12 |
| wallyworld | jam: ok, so we may need to avoid the EnvironConfig via the api and stick to current method and refactor | 09:13 |
| wallyworld | jam: but get-environment uses it afaik | 09:13 |
| jam | rogpeppe1: that may be a use case that we need to support. *My* point is that we need to actually discuss it and decide from a security perspective rather than just saying "oh just expose this, no problem" | 09:13 |
| dimitern | jam, wallyworld, we already have a way to mask out secret attributes in environ config - take a look at the provisioner api's EnvironConfig | 09:13 |
| rogpeppe1 | jam, wallyworld: i think that for the time being at any rate (pending user roles/permissions) we should not take away the ability for a client to see and change the environ configuration | 09:13 |
| rogpeppe1 | jam: if we don't, we're breaking backwards compatibility | 09:14 |
| jam | wallyworld: from what I've understood from fwereade, get-environment and set-environment do far too much. | 09:14 |
| wallyworld | i can believe that | 09:14 |
| jam | and that is partially because we've sort of just shoved everything in EnvironConfig | 09:14 |
| rogpeppe1 | jam: i guess it's possible we want to do that, but we should at least audit current users to check how much they rely on get-environment | 09:14 |
| wallyworld | let's discuss after standup with dimitern et al | 09:14 |
| fwereade | jam, wallyworld: well, set-environment is terribly flaky by nature, but I think that's separate | 09:15 |
| jam | wallyworld: like you can trigger an upgrade via set-environment | 09:15 |
| fwereade | jam, wallyworld: yeah, that's something I would like to be very careful to avoid | 09:15 |
| fwereade | jam, wallyworld: we may or may not have code that tries to prevent that | 09:15 |
| jam | rogpeppe1, wallyworld: I would *guess* the #1 use case for it is to handle setting constraints, because we don't expose another way to set whole-environment values | 09:15 |
| rogpeppe1 | i totally agree that there are somethings in the environ config that really should not be | 09:15 |
| jam | and you set whole-env values at bootstarp time | 09:16 |
| fwereade | jam, wallyworld: but the state method does basically depend on people calling it "right" | 09:16 |
| wallyworld | yeah | 09:16 |
| fwereade | jam, there are no constraints in environ config actually | 09:16 |
| rogpeppe1 | jam: you can set constraints with it? | 09:16 |
| jam | fwereade: isn't that set in "juju set-environment" ? | 09:16 |
| fwereade | jam, nope, set-constraints | 09:16 |
| jam | it may not be EnvironConfig but part of that command? | 09:16 |
| jam | ah, k | 09:16 |
| rogpeppe1 | jam: if your environment credentials have been compromised and you need to change them, set-environment would seem like the right thing to use | 09:17 |
| dimitern | fwereade, wallyworld, jam, rogpeppe1: https://codereview.appspot.com/22210044/ set/get-environment | 09:18 |
| rogpeppe1 | jam: and for debugging, i've certainly relied on get-environment before, for sanity checking (otherwise there's no way of actually finding out what environ config the agents are really using) | 09:18 |
| rogpeppe1 | jam: but we should certainly consider (at least) erasing secrets from the return of get-environment | 09:19 |
| wallyworld | dimitern: tests? | 09:22 |
| fwereade | rogpeppe1, jam: I *think* that's dependent on user roles/permissions, isn't it? | 09:22 |
| rogpeppe1 | fwereade: yeah, that's my thought | 09:22 |
| rogpeppe1 | fwereade: and currently we treat the client as admin, so they get everything | 09:23 |
| jam | fwereade: so I don't have a particular stake in the matter, I just wanted to make sure it got discussed | 09:23 |
| jam | fwereade: it would seem a strong end-run around sharing a .jenv that didn't have env creds | 09:23 |
| jam | if the first thing you could do | 09:23 |
| jam | is run "juju get-env" | 09:23 |
| jam | and have access to them | 09:23 |
| rogpeppe1 | fwereade: given they can deploy to machine 0 and read the database, there's not really any security hole that we're avoiding by making the environ config available | 09:24 |
| rogpeppe1 | jam: i think that having less privileged users is something we are aiming for, but we're not there yet | 09:24 |
| fwereade | rogpeppe1, jam: true, but I long-term expect people to create relatively restricted users and share .jenvs authing those users as themselves, rather than having anyone actually share their jenv files directly | 09:25 |
| jam | rogpeppe1: so I think with your work about api.Open we could actually support it for many commands very soon | 09:25 |
| dimitern | wallyworld, there are tests already that pass without change | 09:25 |
| jam | fwereade: sure, but if we don't actually think about security at get-environment, then it is a pretty clear whole | 09:25 |
| jam | hol | 09:26 |
| jam | hole | 09:26 |
| jam | anyway, meh. as I said, I wanted us to be aware of it more than anything | 09:26 |
| wallyworld | dimitern: for the new apis? EnvironmentGet/Set? | 09:26 |
| dimitern | wallyworld, oh, right | 09:26 |
| jam | add-machine sounds like something that should explicitly *not* have an Environ since we aren't provisioning from the environ | 09:26 |
| dimitern | wallyworld, will add, thanks | 09:26 |
| wallyworld | dimitern: we've been adding new tests for those new things we add | 09:26 |
| wallyworld | np | 09:26 |
| jam | but I can see where it just used it so far, so we can keep doing so | 09:26 |
| wallyworld | jam: right, so we need to understand how env is used in manual provisioning, cause that's why add-machine needs it | 09:27 |
| wallyworld | add-machine per se doesn't need it | 09:27 |
| wallyworld | but add machine calls into the manual provisioning api | 09:28 |
| jam | wallyworld: right, LXC Provisioner was asking for it too, and the *only* thing it pulled out of it was authorized keys | 09:28 |
| jam | which thumper had to work to fix | 09:28 |
| wallyworld | manual does that too | 09:28 |
| wallyworld | plus some other stuff | 09:28 |
| jam | because the MaaS environ can't be created if you strip secrets out of it | 09:28 |
| jam | so we'll run into this again with not being able to manually provision with MaaS if we just did the same thing. | 09:28 |
| wallyworld | sounds so | 09:29 |
| wallyworld | i've just read the manual provisioning code today for the firt time | 09:29 |
| axw | wallyworld, jam: a couple of things spring to mind: state/api.Info (could be gotten another way), and finding tools | 09:31 |
| jam | wallyworld: sure. So I would encourage you to see if you can do add-machine without a full EnvironConfig because it shouldn't really be using it. If it isn't worth the time, then we can hack it in | 09:31 |
| jam | wallyworld: so we do already have a Tools api for agents, I wonder how hard it would be to expose that for an "I want a machine over here". | 09:32 |
| davecheney | is it call the null provider | 09:32 |
| davecheney | the manual provider | 09:32 |
| axw | jam, wallyworld: don't we need the full env config for the provisioned machine's machine config though...? | 09:32 |
| davecheney | or the ssh provider | 09:32 |
| davecheney | please help | 09:32 |
| jam | davecheney: I'm *pretty* sure we settled on Manual provisioner. However, this is still about manually registering a machine to an env that could have a regular provider. | 09:33 |
| jam | axw: I'm pretty sure we don't | 09:33 |
| davecheney | jam: what is SABDFL decided to call it ? | 09:33 |
| davecheney | i understand there was a ruling | 09:33 |
| jam | axw: Uniter and Provisioner today don't have access to all of EnvironConfig | 09:33 |
| jam | davecheney: he liked the phrase "manual registration" | 09:34 |
| davecheney | well, that isn't helpful | 09:34 |
| jam | and not having the word "Provider" in there | 09:34 |
| jam | because nothing is *providing* machines | 09:34 |
| davecheney | +1 for semantic correctness | 09:34 |
| jam | davecheney: so the last bit I remember was "not having provider: in the environ config and using the term manually register new machines" | 09:34 |
| davecheney | -1,000 for usefullness :) | 09:34 |
| fwereade | axw, we should in general never need any of the stuff in environ config | 09:36 |
| fwereade | axw, authorized-keys is the only generally useful bit of it that springs to mind | 09:36 |
| axw | fwereade, jam: yeah I was just revising, that's only used for manual bootstrap | 09:36 |
| fwereade | axw, and *that* is pretty dumb and broken regardless :( | 09:36 |
| jam | axw: right. manual bootstrap is pretty different | 09:37 |
| fwereade | axw, because *surely* the *useful* concept is "authorized users" | 09:37 |
| fwereade | dimitern, commented | 09:37 |
| dimitern | fwereade, thanks | 09:42 |
| dimitern | fwereade, not sure I get your comment about Key/Keys - it's what the command is supposed to do, no? | 09:42 |
| fwereade | dimitern, yeah, but I'm not sure it's actually very useful at an API level | 09:43 |
| fwereade | dimitern, I would prefer to keep the api small and orthogonal as much as possible | 09:43 |
| fwereade | dimitern, baking into it "and this is a use case that's sometimes helpful for the CLI doesn't seem like much of a win, when the cli can filter just fine" | 09:43 |
| fwereade | dimitern, it's kinda like `juju get` which I have a horrible feeling returns the crazy nonsense that the command outputs | 09:44 |
| fwereade | dimitern, the api for `juju get` | 09:44 |
| fwereade | dimitern, which is no use to man nor beast, and even if it is it should be constructed for the specific context it's used in, out of clear sensible atoms, rather than forcing that interpretation on every api user | 09:45 |
| fwereade | dimitern, anyway I could be convinced that Keys was sensible | 09:46 |
| fwereade | dimitern, not so much Key | 09:47 |
| dimitern | fwereade, but there's no need for Keys | 09:48 |
| dimitern | fwereade, it's either one key or all of them | 09:48 |
| dimitern | fwereade, are you saying to have Keys and use only one or zero keys when calling the command and have the rest of the logic moved to the command itself (key == "" -> all keys, else just that key) ? | 09:49 |
| dimitern | fwereade, I thought we want to make the client API as close as possible to the CLI, logic included, so the GUI can provide the same features? | 09:50 |
| fwereade | dimitern, at the moment there's no need for keys because we have a set-env that was hacked together without much thought and it only uses one | 09:51 |
| jam | dimitern: so I would guess the point is to think about what is actually useful in an API. Today the CLI is actually deficient in that area. If I want 3 settings, I have to do 3 round trips, when you can easily implement the API to support multiple in one pass. | 09:51 |
| dimitern | fwereade, set-env accepts multiple key=value args | 09:51 |
| fwereade | dimitern, then all the more reason to allow us to ask for multiple values in get, surely? | 09:52 |
| fwereade | jam, exactly | 09:52 |
| dimitern | fwereade, ok, I'll change EnvironmentGet to accept zero or more keys | 09:53 |
| fwereade | jam, dimitern: we're trying to write a sane api that happens to allow for the CLI functionality | 09:53 |
| fwereade | jam, dimitern: rather than just writing an API for the CLI | 09:53 |
| dimitern | fwereade, and the CLI get-env will use it like that, except there'll be the bit that if key == "" -> return the map, otherwise, return just the value of the given key | 09:53 |
| fwereade | dimitern, sgtm, I'm not *that* much against Keys ;) | 09:55 |
| dimitern | fwereade, ok, because frankly I can't see the alternative | 09:55 |
| fwereade | dimitern, just ask for the whole env config? | 09:56 |
| dimitern | fwereade, always? | 09:56 |
| dimitern | fwereade, it seems unnecessary trafficwise | 09:56 |
| jam | dimitern: as a general guide, if it is less than 64kB it isn't worth filtering | 09:56 |
| fwereade | dimitern, gut feeling says that's simplest and always adequate, but maybe all those extra bytes add up to an annoying degree | 09:56 |
| jam | as the bandwidth cost == latency of a round tirp | 09:56 |
| jam | if its big, then we can do somethnig | 09:57 |
| dimitern | ok, I'll get rid of Key/Keys and return the whole lot always then | 09:57 |
| fwereade | dimitern, cheers | 09:57 |
| jam | that's at least the numbers I've found in the past | 09:57 |
| jam | 100ms at 1M/s = 100 | 09:58 |
| jam | 100kb I guess is 12kB | 09:58 |
| jam | though I've got 16MB and I've seen 300ms ping | 09:58 |
| === rogpeppe1 is now known as rogpeppe | ||
| rogpeppe | jam: just saw your conversation with axw about client secret-pushing. this is what i suggested yesterday - it seems to me that this is more-or-less where you ended up, is that right? http://paste.ubuntu.com/6363731/ | 10:04 |
| === BradCrittenden is now known as ba | ||
| === ba is now known as bac | ||
| axw | rogpeppe: except not putting it into Login | 10:05 |
| axw | I started down that path this morning, it got a bit messy | 10:05 |
| axw | Login will (after my next CL) respond with a flag saying that secrets need to be pushed | 10:05 |
| axw | but it won't require it there and then | 10:05 |
| rogpeppe | axw: ah, ok | 10:06 |
| rogpeppe | axw: where did the messiness come from? | 10:06 |
| axw | rogpeppe: possibly just my misunderstanding of the API stuff, but it seemed wrong to add knowledge of environments in there. Now that I think of it though, it's going to have to anyway... | 10:07 |
| rogpeppe | axw: yeah, that's what i was thinking | 10:07 |
| axw | well, either approach will not be much of a change from what I've got now anyway | 10:07 |
| rogpeppe | axw: cool | 10:08 |
| axw | rogpeppe: the other thing I wondered was about validating config that soon | 10:08 |
| jam | axw: I'm wondering if we want it in api.Open or if we could do it one layer higher | 10:08 |
| jam | NewAPIConn is what the current CLI code ends up using, and it has the ENviron there | 10:09 |
| axw | is it possible the config is wrong at that point? then how do we fix it without being able to login? | 10:09 |
| jam | axw: my gut instinct at https://code.launchpad.net/~axwalk/juju-core/api-push-env-secrets/+merge/194083 was that it looks like the wrong level | 10:09 |
| rogpeppe | axw: if the config is wrong, you shouldn't be able to push it into the environment | 10:09 |
| axw | jam: I originally had it higher, in fact. It seemed better to me to keep the Login-results-stuff inside api.Open | 10:10 |
| rogpeppe | axw: you can fix it by pushing a correct config | 10:10 |
| axw | rogpeppe: true, unless the correctness changes over time :p | 10:10 |
| jam | axw: but having state/api know anything about environs secrets is ... dissapointing | 10:10 |
| axw | rogpeppe: ah, so I was just planning to push the secrets, not the entire config | 10:10 |
| rogpeppe | axw: well, having a bad config doesn't make the environment inaccessible | 10:10 |
| axw | jam: true, but the apiserver is going to have to anyway, right? | 10:11 |
| axw | rogpeppe: I'm saying if we require login to successfully Validate config before letting the API client through | 10:11 |
| axw | (back to our discussion yesterday) | 10:12 |
| rogpeppe | jam: i think that the api will need to know about environ secrets even when bootstrap is synchronous | 10:12 |
| rogpeppe | axw: is that problematic? | 10:12 |
| jam | rogpeppe: axw: so it feels very strange to push "people connecting to the API should have environ and secrets" when we *really* want to get the environ state from the api | 10:12 |
| axw | rogpeppe: only if it's feasible that the first time you connect, config is bad. I think it's probably not, like you said. | 10:12 |
| jam | and it is only because of the handoff that we need it for the *very first* login. | 10:13 |
| rogpeppe | jam: only the first client connecting needs environ and secrets | 10:13 |
| jam | rogpeppe: needs, but everything else looking at the function will go "where do I get these bits" | 10:13 |
| rogpeppe | jam: we allow nil to be passed | 10:13 |
| jam | rogpeppe: "allow nil to be passed" is not very good API design. | 10:13 |
| rogpeppe | jam: and when bootstrap is synchronous, we can have a separate API call | 10:14 |
| rogpeppe | jam: rather, a separate entry point in juju | 10:14 |
| rogpeppe | jam: sometimes i think it's ok | 10:14 |
| rogpeppe | jam: when the thing really is optional | 10:15 |
| rogpeppe | jam: so we'd provide it as an argument only if we find a BootstrapConfig in the .jenv. | 10:15 |
| rogpeppe | jam: hmm, actually there's perhaps another possibility | 10:16 |
| rogpeppe | jam: we can actually know if we're the first to connect, because the .jenv file has no cached API address (well - that will be the case when we do actually cache API addresses...) | 10:17 |
| rogpeppe | jam: what's your suggestion for the (juju) package API? | 10:18 |
| jam | rogpeppe: so we could just do a "Do you need secrets" call in NewAPIConn, and if it returns true, then call Client.SetSecrets() | 10:19 |
| jam | though even there, NewAPIConn is used by agents | 10:19 |
| jam | I think | 10:19 |
| axw | jam: the agents use api.Open directly | 10:19 |
| jam | axw: k, that would be pretty clear that the 99% case is not wanting to pass in the extra data, so we probably don't want to put it there. | 10:20 |
| jam | axw: AFAICT the only thing we gain is 1 round trip which we are going to get rid of with doing it at bootstrap | 10:20 |
| rogpeppe | jam: it looks to me as if NewAPIConn isn't used by anything except NewAPIClientFromName | 10:21 |
| jam | rogpeppe: it is a point where we already have the data we need, and it is called by NewAPIConnFromName which most (all?) CLI cmds will be using to connect. | 10:22 |
| axw | jam: sorry, confused. are you suggesting that I go back to doing the unconditional API call for the client? | 10:22 |
| jam | axw: a thought, api.Open() could cache some status onto the api object itself, so that a follow up "api.DoIneedSecrets" doesn't have to do a round trip | 10:22 |
| jam | but we don't expose it directly in the return of api.Open | 10:22 |
| jam | axw: well it was a "CheckIfWeNeedSecrets" unconditionally. | 10:23 |
| jam | though the putting it in a result from Login and then caching that on the api.State object. | 10:25 |
| rogpeppe | if it's going to go away, i don't really mind an extra round trip for every API connection | 10:26 |
| axw | jam: what's the big difference between unconditionally calling "PushEnvironmentSecrets" instead of "CheckIfWeNeedSecrets" and t hen "PushEnvironmentSecrets" | 10:26 |
| axw | the former does nothing if it's not needed | 10:26 |
| jam | rogpeppe: well today we get to remove about 5 round trips by caching the API address :) | 10:26 |
| jam | but after that it shows up more | 10:26 |
| jam | axw: passing secrets over the wire when they aren't neededx | 10:27 |
| rogpeppe | jam: we're talking about something which we're going to fix anyway, right? | 10:27 |
| jam | rogpeppe: right, I'm saying we add 1 round trip here, but then we take away 5 over there pretty soon, and eventually get back to this 1 | 10:27 |
| rogpeppe | jam: so cluttering the API as little as possible seems like a reasonable approach | 10:27 |
| dimitern | rogpeppe, should DeepEquals work and return 2 for 2 identical map[string]interface{} values? | 10:29 |
| axw | sorry guys, gotta go help get my daughter ready for bed. I may be on a bit later | 10:29 |
| rogpeppe | jam: we can probably use exactly the same logic that the mongo connection uses currently - no need to change the API at all | 10:29 |
| rogpeppe | dimitern: yes | 10:29 |
| dimitern | rogpeppe, s/return 2/return true/ | 10:29 |
| dimitern | rogpeppe, well it doesn't for some reason | 10:29 |
| jam | dimitern: are you sure about int types | 10:29 |
| jam | int 2 != int64 2 | 10:29 |
| rogpeppe | dimitern: they're not identical then :-) | 10:29 |
| jam | interface{} is really bad about this | 10:29 |
| jam | and the output in the log looks like | 10:29 |
| jam | {a: b} != {a: b} which is very WTF | 10:30 |
| dimitern | rogpeppe, I checked them key by key http://paste.ubuntu.com/6369724/ | 10:30 |
| rogpeppe | dimitern: ah, there's one other thing: it won't return true if some values are not comparable ISTR | 10:30 |
| dimitern | rogpeppe, perhaps like ca-cert? | 10:30 |
| rogpeppe | dimitern: gospew is useful for diagnosing stuff like this | 10:30 |
| jam | dimitern: I would bet api-port is an 'int' on one side and something like 'int32' or 'int64' somewhere else | 10:30 |
| jam | or if this is data you got out of JSON, even float74 | 10:31 |
| rogpeppe | dimitern: i bet it's a JSON issue | 10:31 |
| jam | float64 | 10:31 |
| jam | which writes out as int if it doesn't have a decimal | 10:31 |
| rogpeppe | dimitern: remember that any numbers in json arrive as float64, as jam says | 10:31 |
| dimitern | both ports look like ints | 10:31 |
| jam | I've hit that in the past | 10:31 |
| jam | printf(float64(1)) == 1 | 10:31 |
| jam | not 1.0 | 10:31 |
| dimitern | damn! | 10:31 |
| jam | which is sort of a shame | 10:31 |
| jam | I've *definitely* been bit by that in the past | 10:31 |
| dimitern | so passing anything through the api which returns interface{} having possibly ints is doomed | 10:32 |
| jam | well, "doomed" to get cast into a float | 10:32 |
| jam | :) | 10:32 |
| jam | unless you put it into a struct | 10:32 |
| jam | we had that for "juju get" as well | 10:32 |
| rogpeppe | jam: well printf("%v %v %v", int8(1), int16(1), int(1)) prints 1 1 1 | 10:32 |
| jam | rogpeppe: http://play.golang.org/p/Spp9lwRovO | 10:33 |
| jam | right, the issue is | 10:33 |
| rogpeppe | dimitern: i've considered the possibility of doing UseNumber for all json stuff we send on the wire | 10:33 |
| jam | Printf("%v", float64(1.0)) | 10:33 |
| jam | also returns just | 10:33 |
| jam | 1 | 10:33 |
| jam | so you can't tell it is a float | 10:33 |
| jam | vs returning 1.0 | 10:33 |
| dimitern | rogpeppe, what's UseNumber | 10:34 |
| rogpeppe | jam: yeah - i'm just pointing out there are *lots* of possible types there too. just fixing it for floats wouldn't fix it for those too | 10:34 |
| rogpeppe | dimitern: http://golang.org/pkg/encoding/json/#Decoder.UseNumber | 10:34 |
| rogpeppe | dimitern: BTW gospew reference: github.com/davecgh/go-spew/spew | 10:35 |
| rogpeppe | dimitern: spew.Dump(someVal) is useful for seeing everything (recursively and with types) inside a value | 10:36 |
| jam | dimitern: see rev 1722 of juju-core where I mention that cmd.Get also suffers from auto-cast to float64 | 10:36 |
| dimitern | ok, thanks rogpeppe | 10:37 |
| jam | dimitern: note that the discussion from juju get is that clients just shouldn't count on it being an Int object | 10:40 |
| jam | so you could also go that way | 10:40 |
| jam | otherwise you need to run the type decoding on the client side | 10:40 |
| jam | after getting the map | 10:40 |
| jam | even UseNumber | 10:40 |
| jam | just gives you stuff that you can then chose what you cast it to | 10:40 |
| rogpeppe | jam: i *think* i'm marginally in favour of using UseNumber throughout | 10:41 |
| rogpeppe | jam: it means that we can transmit int64 losslessly too | 10:41 |
| jam | rogpeppe: UseNumber means we just have strings so you still don't have an actual int or float | 10:42 |
| rogpeppe | jam: indeed, but at least it's explicitly something else | 10:42 |
| jam | rogpeppe: as I said before, you then *still* need to run the "interpret everything as an explicit type" code on the client | 10:42 |
| rogpeppe | jam: and we can make the schema package work with json.Number | 10:43 |
| jam | so the decision 2 months ago was, don't worry about it. | 10:43 |
| jam | it doesn't change the cli | 10:43 |
| jam | as we are printing it out into JSON anyway | 10:43 |
| jam | or possibly YAML | 10:43 |
| jam | so the only thing that might care is someone calling the API directly, and they're already screwed | 10:43 |
| rogpeppe | jam: how so? | 10:44 |
| rogpeppe | jam: or do you mean calling the api package directly? | 10:44 |
| jam | rogpeppe: they just have a string on the wire that is a float, So all the code we have to try and cast thing into appropriate types, but on the wire its just a string | 10:44 |
| rogpeppe | jam: it doesn't look like a string on the wire | 10:45 |
| jam | rogpeppe: anyway, quite a bit of effort for almost 0 gain | 10:45 |
| rogpeppe | jam: it looks like a normal json number | 10:45 |
| jam | rogpeppe: it doesn't look like an int or a float on the wire. it looks like a sequence of bytes | 10:45 |
| jam | aka "string" | 10:45 |
| rogpeppe | jam: it fits exactly the usual json syntax for a number | 10:45 |
| jam | rogpeppe: sure, but it *isn't* a float or an int | 10:45 |
| rogpeppe | jam: no? i'm not sure i see the difference. | 10:45 |
| rogpeppe | jam: json only has one number type | 10:46 |
| jam | the fact that an object is an int or a float needs to be transmitted separately | 10:46 |
| jam | which means we have to re-implement the code to interpret what the value of this field should be | 10:46 |
| jam | and it was determined in the last discussion | 10:46 |
| jam | that it wasn't actually worth doing that | 10:46 |
| jam | I don't quite see how that's changed | 10:46 |
| jam | UseNumber doesn't actually help until we have a way to cast it | 10:46 |
| rogpeppe | jam: ah, i see what you mean | 10:46 |
| jam | so yes, *if* we implement client-side casting, then we can UseNumber at that point | 10:47 |
| rogpeppe | jam: yeah, i think i agree that we can transmit all numbers as exactly the same type | 10:47 |
| rogpeppe | jam: and we can define that to be float64 | 10:47 |
| jam | rogpeppe: so in the config structure, it does define what the actual type of that number is | 10:48 |
| jam | and we have code to do the work, and it runs behind the API, but then all that info gets discarded on the wire, so we have to do it again on the client | 10:48 |
| rogpeppe | jam: standup? | 10:48 |
| jam | yep, sorry | 10:48 |
| dimitern | fwereade, added tests https://codereview.appspot.com/22210044/ | 10:54 |
| fwereade | dimitern, cheers | 10:54 |
| * TheMue => lunch | 11:35 | |
| === rvba` is now known as rvba | ||
| === gary_poster|away is now known as gary_poster | ||
| rogpeppe | natefinch: ping | 13:15 |
| natefinch | rogpeppe: hey | 13:15 |
| rogpeppe | natefinch: hangout? | 13:15 |
| natefinch | sure | 13:16 |
| rogpeppe | natefinch: https://plus.google.com/hangouts/_/calendar/am9obi5tZWluZWxAY2Fub25pY2FsLmNvbQ.mf0d8r5pfb44m16v9b2n5i29ig?authuser=1 | 13:16 |
| dimitern | fwereade, thanks for pointing out agent-version - it was totally possible to change it with EnvironmentSet - now it's not and I added a specific test about it | 14:15 |
| dimitern | fwereade, updated https://codereview.appspot.com/22210044/ | 14:15 |
| fwereade | dimitern, sweet, tyvm | 14:16 |
| rogpeppe | jam: do you fancy a quick chat about the charm streaming stuff? i was just about to write yet another reply, but thought perhaps a higher bandwidth connection might be useful | 14:33 |
| mattyw | fwereade, thanks for looking at my mps - hope it's not wasting too much of your time, I'll take another look at them tomorrow | 14:41 |
| fwereade | mattyw, dude, it's my job :) | 14:41 |
| fwereade | mattyw, not a waste at all | 14:41 |
| mattyw | fwereade, sweet! did you see my reply, what do you mean by dirty? the commit history?? | 14:42 |
| fwereade | mattyw, it just looked like the proposed branch hadn't had history merged in | 14:42 |
| fwereade | mattyw, so I can't (easily) tell which are your changes and which just happened in the interim | 14:43 |
| fwereade | mattyw, just merge trunk through the pipeline and propose again | 14:43 |
| mattyw | fwereade, oh ok, I hadn't noticed that, I'll take another look | 14:43 |
| mattyw | who's a good person to talk to about the charm store? | 14:47 |
| rogpeppe | mattyw: what do you want to talk about? | 14:48 |
| sodre | smoser: is example-sync still working ? | 15:34 |
| TheMue | strange, shut eth0 on a ec2 machine, but status still reports it as running | 15:39 |
| mgz | TheMue: that's from the instance existing, not the state reporting though, no? | 15:40 |
| TheMue | mgz: no, status reporting | 15:41 |
| TheMue | mgz: from the perspective of ec2 it is still running, which is ok | 15:41 |
| TheMue | mgz: juju ssh fails as expected | 15:42 |
| TheMue | mgz: pinging the machine fails too, so I would expect that also machine 0 would fail to reach the machine :/ | 15:44 |
| mgz | TheMue: 'running' is an ec2 instance state, not a juju machine status | 15:44 |
| mgz | one thing `juju status` does is call DescribeInstances and report the instance state of all instances juju is managing | 15:45 |
| TheMue | mgz: yep, ok, wrong wording, agent-state is still started, not down | 15:45 |
| TheMue | mgz: the instance state is correct with running | 15:46 |
| mgz | TheMue: try for instance, euca-stop-instances on that, then see what it reports | 15:46 |
| mgz | "for instance"? too many instances. | 15:46 |
| TheMue | mgz: :) | 15:46 |
| TheMue | mgz: I tried to simulate a netsplit. stoping or terminating a machine is fine for the status reporting. | 15:47 |
| jcsackett | :q! | 15:50 |
| jcsackett | exit | 15:50 |
| jcsackett | ...i hate when i get windows mixed up. | 15:51 |
| mgz | :D | 15:51 |
| fwereade | bbiab | 15:52 |
| TheMue | yeah, bootstrap node has no connection to the agents on that machine anymore. and status reports still that everything is fine. | 15:52 |
| benji | :w plan-to-use-jcsackett's-window-management-skills-against-him.txt | 15:53 |
| benji | oops | 15:53 |
| mgz | ehehe | 15:54 |
| jcsackett | truthfully it's less window management and more "what has focus" issues. :-P | 15:55 |
| benji | sloppy focus is the only way | 15:55 |
| natefinch | Man, I really want to be able to use juju to deploy juju's mongodb in HA :/ | 16:04 |
| mgz | natefinch: you are not the first to have this thought | 16:04 |
| natefinch | mgz: chickens and their stupid eggs | 16:05 |
| natefinch | ok, so what we do, is use the manual provider to bootstrap the bootstrap for a real provider... | 16:07 |
| * natefinch checks out the mongodb charm to see if there's anything he can crib | 16:11 | |
| === natefinch is now known as natefinch-afk | ||
| sinzui | jamespage, I just released https://launchpad.net/juju-core/+milestone/1.16.3 to address the cgroup-lite bug that affects lxc | 16:26 |
| * sinzui doesn't want to do any more 1.16 releases | 16:26 | |
| mgz | sinzui: jamespage is at the openstack summit, so I think he's probably not online right now | 16:37 |
| sinzui | :) | 16:37 |
| === mind_ is now known as WEBBRANDON | ||
| === WEBBRANDON is now known as webbrandon | ||
| bac | hi sinzui | 18:53 |
| sinzui | Hi bac | 18:53 |
| bac | sinzui: can you join us in #juju-gui? | 18:53 |
| * rogpeppe is done for the day | 19:18 | |
| rogpeppe | g'night all | 19:18 |
| natefinch-afk | morning thumper | 20:19 |
| thumper | morning | 20:19 |
| natefinch-afk | what color do *you* want the bike shed to be? :) | 20:19 |
| === natefinch-afk is now known as natefinch | ||
| thumper | which shed are we talking about? | 20:20 |
| thumper | if it is the ensure-ha chat | 20:20 |
| thumper | I'm likely to dive in :) | 20:20 |
| thumper | but firstly | 20:20 |
| natefinch | yeah ,ha | 20:20 |
| thumper | *STOP CROSS POSTING EVERYONE!!!!* | 20:20 |
| natefinch | wow, I missed that roger had posted to juju, not just juju-dev | 20:23 |
| thumper | since I don't use gmail, I get everything showing twice | 20:24 |
| natefinch | well, you have no one but yourself to blame for that one ;) | 20:25 |
| === BradCrittenden is now known as bac | ||
| bac | hi marcoceppi | 20:47 |
| marcoceppi | o/ bac | 20:47 |
| bac | hey marcoceppi, we need a change to charmtools proof to make the endpoint an option to the proof call. i've got a branch that seems to work that i'd like to propose | 20:48 |
| marcoceppi | bac: couldn't you guys just call proof with --offline and then call your endpoint seperately? | 20:49 |
| bac | marcoceppi: long term that's probably a good solution | 20:52 |
| bac | marcoceppi: please have a look at https://code.launchpad.net/~bac/charm-tools/proof-endpoint/+merge/194227 and see what you think. | 20:54 |
| marcoceppi | :\ I don't like it, but it's nothing to do with the code, it's just exactly the reason I put the offline flag within proof in the first place | 20:56 |
| marcoceppi | Once this is in, it's pretty much there until a 2.0 | 20:57 |
| bac | marcoceppi: we're not offline. we just need to configure the address of the server. | 20:57 |
| marcoceppi | well | 20:57 |
| marcoceppi | bac: the idea behind offline was for charmworld. Since charmworld knows it's own api it won't need charm-tools to do it since it can invoke it itself | 20:57 |
| bac | marcoceppi: i don't know the history but i don't understand how offline has anything to do with charmworld. i thought it was to allow people to use your tool when offline. the two seem orthogonal. | 20:59 |
| marcoceppi | bac: offline was just the way the feature was named, as it turns off remote checking | 21:00 |
| rick_h_ | marcoceppi: the thing with offline is we still need to run proof and then run our own checks | 21:02 |
| rick_h_ | marcoceppi: so we need to double the calls, build a combined list of issues, etc | 21:02 |
| rick_h_ | marcoceppi: the hope was that using proof in 'online always' mode would make sure we're all seeing the same messages/list of issues | 21:03 |
| rick_h_ | marcoceppi: but we can break it apart. | 21:03 |
| rick_h_ | marcoceppi: we'll work on that path. | 21:03 |
| marcoceppi | OTP, rick_h_ if the URL is different then it should be run outside of that | 21:03 |
| rick_h_ | marcoceppi: rgr | 21:03 |
| marcoceppi | rick_h_ bac: I don't mean to be disagreeable, however, I don't have time to cut this as a 1.1.3 before I start traveling given my current workload | 21:03 |
| rick_h_ | marcoceppi: understand | 21:04 |
| bac | marcoceppi: i've retracted my MP. i understand your concerns. | 21:18 |
| marcoceppi | bac: thanks, sorry again. I really want to accomodate you guys as much as possible - let me know if there's anything else I can help with o/ | 21:19 |
| * thumper -> gym | 22:49 | |
| === bradm_ is now known as bradm | ||
| davecheney | all: which pocket is Juju going to ship in 14.04 ? | 23:40 |
| davecheney | mramm: which pocket is Juju going to ship in 14.04 ? | 23:56 |
| wallyworld_ | thumper: since you are ocr, https://codereview.appspot.com/22580043 plus i wouldn't mind a quick hangout if you are free | 23:58 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!