/srv/irclogs.ubuntu.com/2012/07/05/#juju-dev.txt

davecheneyniemeyer: howdy00:08
davecheneysorry i'm a bit slow off the mark today00:08
davecheneym_3 and I are checking out a coworking space in the city today00:08
davecheneyniemeyer: short status report, got ec2 local tests working with a provisoiner working last night00:09
davecheneybut there are a few problems that need to be solved before I can propose it00:09
niemeyerdavecheney: Heya00:09
niemeyerdavecheney: Neat, sounds good00:09
davecheneyniemeyer: the main one is we ask ec2test for the DNS name of machine/000:09
davecheneywhcih returns something liek i-3.example.com00:09
davecheneywhich breaks the local test because it then uses that hostname in the stateinfo passed to anyone who wants it00:10
davecheneyso I need to add some kind of hook to either ec2test to say 'always return localhost' as the dns name, or to be able to override it if the test is running in local mode00:11
davecheneyniemeyer: the second problem is ec2 is hard coded to expect the zkport to be 218100:11
davecheneybut that is simpler to fix00:11
davecheneyi'll propose a fix real soon now00:11
niemeyerdavecheney: Hmm00:12
niemeyerdavecheney: Wait, you mean you're putting a worker within the ec2 tests?00:12
davecheneyniemeyer: by worker, do you mean provisioner ?00:13
niemeyerdavecheney: a juju-core/worker00:13
davecheneyniemeyer: yes, the tests that roger left bootstrap an environment then try to do a state.AddMachine00:14
niemeyerdavecheney: Within the *ec2* tests?00:15
niemeyerdavecheney: I may be wrong, and will be happy to look at it, but it doesn't sound entirely right00:15
davecheneyniemeyer: yes, roger extended jujutest/livetest to start a machine in the provisioned environment00:15
davecheneyhang on, will get a link00:15
davecheneyniemeyer: sorry, m_3 just turned up00:21
davecheneyhttps://codereview.appspot.com/6347044/ << not a proposal00:21
niemeyerdavecheney: No worries, I'll actually have to step out for dinner00:21
davecheneythis is a grab bag of rogers stuff and my additions00:21
davecheneyniemeyer: the key change is https://codereview.appspot.com/6347044/patch/10001/1101200:22
davecheneyspecifically, if hasProvisioner then testprovisioner00:22
niemeyerdavecheney: I'm indeed a bit lost.. in a sentence or two, what is this testing?00:58
davecheneyniemeyer: the changes to jujutest test that after bootstrapping, a provisioner is running, and it can start an instance with state.AddMachine01:05
niemeyerdavecheney: I can't see how we could possibly be testing that without a functional test01:06
niemeyerdavecheney: How can we be testing that after bootstrap a provisioner is running? This is entirely dependent on details of ec2 itself01:06
davecheneyyes, i haven't got that to work in the ec2 amazon tests yet because of the push secrets problem01:08
davecheneythis only works when we make the local provisioning agent in process for local_test.go01:08
davecheneybut in theory, once the secrets are pushed (faux juju deploy) the test will also pass on the real ec201:08
niemeyerdavecheney: The point is that there's nothing being tested there01:56
niemeyerdavecheney: The live real tests can do that on real EC201:57
niemeyerdavecheney: If we fake the precise point we want to test, what's the purpose of the test?01:57
davecheneyniemeyer: this test should (ha!) work on the real ec2 as well01:58
davecheneyso at the moment all I can prove is the test case works when we fake out all the moving parts behind it01:59
davecheneythat is, localtest continues to work with a test plan that expects a provisioner02:00
davecheneyonce the pushing of secrets is fixed, we can remove the 'hasprovisioner' field as it will always be true02:00
niemeyerdavecheney: I understand, but this doesn't seem to be going in a good direction02:02
niemeyerdavecheney: All this faking isn't buying us stability or anything02:02
niemeyerdavecheney: It's messing up the code instead02:02
davecheneyniemeyer: should I put this on hold until we know how to fix the pushing of secrets and can do a real test in a real ec2 ?02:03
niemeyerdavecheney: No, I think we should fix the problems we have to fix for getting the real EC2 running02:03
davecheneyniemeyer: cool, that means resolving the secrets problem02:04
niemeyerdavecheney: The secrets problem is resolved in deploy02:04
niemeyerdavecheney: There's no *real* problem.. we just need to do the same thing we used to do02:04
niemeyerdavecheney: Sorry, I'm kind of stating that with the belief I know what's going on, and maybe I'm wrong02:05
niemeyerdavecheney: So please don't take that at face value.. feel free to bring other points02:05
davecheneyniemeyer: thats cool02:05
davecheneythe problem i think I'm solving is02:05
davecheney1. get boostrapping working -- which I believe is done, juju bootstrap works02:06
davecheney2. prove it with tests02:06
davecheney2 is tricky, and this CL is part way towads solving it02:06
niemeyerdavecheney: There are also several bits in the same CL that should be broken down so we can debate about them (and integrate them!) in isolation02:06
niemeyerdavecheney: e.g. there's a fix for cloudinit, etc02:07
davecheneyniemeyer: yes, i'll peel off those bits02:07
davecheneyjust at the moment i'm taking another pass at the s3 location constraint proposal02:07
davecheneybecause I want to commit that before telling everyone to please go get -u02:08
davecheneythen i'll start to pull apart this CL02:08
niemeyerI find this, for instance, unexpected:02:08
niemeyer  87                 err = t.seedSecrets(env)02:08
niemeyer  88                 c.Assert(err, IsNil)02:08
niemeyer  89                 if t.HasProvisioner {02:08
niemeyer  90                         t.testProvisioning(c, st)02:08
niemeyer  91                 }02:08
niemeyerdavecheney: This is in TestBootstrap02:10
davecheneyyes, all environments should have a provisioner, but we can't do that yet02:10
davecheneysorry, all providers02:10
niemeyerdavecheney: So why would we be seeding secrets if we're testing bootstrap?02:11
davecheneyniemeyer: right, i understand02:11
davecheneyI will make a new test case02:11
niemeyerdavecheney: In fact, not only seeding secrets, but hacking the environment configuration arbitrarily02:11
davecheneyniemeyer: agreed, this is to work around the fact that bootstrap doesn't push the complete environment configuration into the state02:12
davecheneyif it did, then this wouldn't be a problem02:12
niemeyerdavecheney: But this is TestBootstrap! :)02:12
niemeyerdavecheney: It does not, and it should not02:13
davecheneyunderstood, i'll rework it02:14
niemeyerdavecheney: If the idea is to test a real ec2 deployment on live tests, we should call fwereade_'s conn.Deploy02:14
niemeyerdavecheney: and let it do the work02:14
davecheneyniemeyer: i'll try that02:15
niemeyerdavecheney: There's another issue with the CL approach as well that is good to keep in mind02:17
niemeyerdavecheney: One of the goals of jujutest is for it to be reusable across providers02:17
davecheneyyes02:17
niemeyerdavecheney: So we have to watch out for provider-specific behavior there.. in some cases we may need to hook onto the provider02:18
niemeyerdavecheney: Via a callback or whatever02:18
niemeyerdavecheney: But ideally we should be able to take these tests, plug on the OpenStack or Compute Engine backend, and have them working02:18
davecheneyniemeyer: yes02:19
davecheneyjust checking what abuse i've done to jujutest02:19
niemeyerdavecheney: The configuration is the only thing that stands out to me02:20
davecheneyniemeyer: the complication is the ec2 environ has two modes02:20
davecheneylocal fake ec2 mode and amazon mode02:20
davecheneysupporting that is hard02:21
davecheneyfor example, fake ec2test server gives back DNS names which are impossible02:21
davecheneyie, i-3.example.com02:21
davecheneywhich can't be used as value StateInfo addresses02:21
niemeyerdavecheney: Right, but the question is where do we draw the line02:22
niemeyerdavecheney: I don't think it'd pay off if we simulated the *whole* provider02:23
niemeyerdavecheney: How do we actually deploy a machine?02:23
niemeyerdavecheney: How do we get unit agents running and interacting?02:23
niemeyerdavecheney: Faking out all of that stuff would have us with white hair by then02:23
davecheneyand as you say, doesn't actually test anything02:24
niemeyerdavecheney: It could test, the interaction between the pieces, but indeed not the *deployment* of the pieces, since that's all faked indeed02:24
niemeyerMeaning it could be entirely broken and we'd think it's alright02:24
niemeyerdavecheney: So, to begin with, I'd be happy with02:25
davecheneyit's really about continuing the illusion of the ec2 provider in local mode02:25
niemeyerif !t.CanOpenState { return }02:25
niemeyerdavecheney: At the top of the test that checks the real provisioning agent with conn.Deploy02:25
niemeyerdavecheney: run that with real Amazon only02:25
niemeyerdavecheney: This means we're testing stuff *for real*, cloud-init, running of commands, upstart script, and Inc.02:26
davecheneyniemeyer: ok, i'll try that02:26
niemeyerdavecheney: Of course, we won't be able to run that all the time02:26
niemeyerdavecheney: But for that we have the unit tests, closer to the implementations themselves02:26
niemeyerdavecheney: As soon as we have stuff working like that, I'll be happy to setup an integration test somewhere that does the boring work for us02:27
niemeyerdavecheney: I mean, integration test runner02:27
niemeyerdavecheney: LiveTest is already an integration test02:28
niemeyerLiveSuite, that is02:28
davecheneyniemeyer: thank you for your time this evening02:30
niemeyerdavecheney: Sure thing, I'm glad to have that stuff moving forward.. we're about to have something real working, and that's quite exciting02:31
davecheneyi'm going to take a break on this CL and work on the s3 location one for a little02:31
davecheneyniemeyer: it's been working for over a week02:31
davecheneyactually, no02:31
davecheneythta is not true02:31
davecheneyonce we have deploy it works02:31
davecheneyalso, ... value *net.OpError = &net.OpError{Op:"remote error", Net:"", Addr:net.Addr(nil), Err:0x28} ("remote error: handshake failure")02:32
davecheneyfu amazon02:32
niemeyerYeah, Amazon has been going through some bad times02:33
davecheneyniemeyer: changing gears02:34
davecheneyI'd like to change the s3 tests to hit a number of regions02:34
davecheneynot just USEast02:34
davecheneyis there anyting in juju-core that I can crib for reusing a test ?02:34
niemeyerdavecheney: Should be easy to have a setting on the suite type on s3i_test.go, and instantiating a couple of them with different values at the Suite(&S{..}) call time02:35
davecheneyI'm thinking jujutest/tests.go02:35
davecheneyniemeyer: yes, that is what I want to do02:35
niemeyerdavecheney: But if it's s3, s3i_test.go is the thing02:35
niemeyerTheMue, fwereade: Mornings!10:04
TheMueniemeyer: Morning10:05
TheMueniemeyer: You're up early today.10:05
niemeyerTheMue: Yeah, couldn't sleep for whatever reason10:05
TheMueniemeyer: So used the time for a relaxed start with a good breakfast10:07
niemeyerTheMue: Yeah, exactly10:08
fwereadeniemeyer, heyhey10:24
TheMuefwereade: Heya10:27
niemeyerfwereade, TheMue: How're things going guys? Good progress there?10:40
TheMueniemeyer: Think so, yes. Setting up my tests.10:40
fwereadeniemeyer, not too bad, getting default-series handling right is a touch fiddly10:40
fwereadeniemeyer, had been hoping to chat to dave this morning, but missed him10:41
niemeyerfwereade: Aw10:42
niemeyerfwereade: What's the trickiness around it?10:42
fwereadeniemeyer, at the heart of it is now we initialize state10:43
fwereadeniemeyer, how10:43
fwereadeniemeyer, the current Initialize does basically nothing10:44
fwereadeniemeyer, what I would like to do is (among the other stuff) to put in a complete environ config, excluding only secrets, at bootstrap time10:44
niemeyerfwereade: That sounds great10:45
fwereadeniemeyer, but that has distressingly far-reaching consequences10:45
niemeyerfwereade: Oh?10:45
fwereadeniemeyer, I'm hoping that I will be able to see a way to break this down into 2 changes with hindsight10:45
fwereadeniemeyer, well, the whole bootstrap pipeline needs to change to accept that complete environ config10:46
niemeyerfwereade: How so?10:46
niemeyerfwereade: Bootstrap() is in the environ, and the environ knows its own config10:46
fwereadeniemeyer, cloud-init and initzk10:47
niemeyerfwereade: So on that side at least, we're good10:47
niemeyerfwereade: Right, on the other side we have to load it10:47
niemeyerfwereade: A base64 option to initzk should do the deal10:47
fwereadeniemeyer, yeah, that's basically done10:48
fwereadeniemeyer, also, we need to be able to figure out which bits are secret and which aren't10:48
fwereadeniemeyer, and ATM it feels like we can't sanely get away without some sort of distinction between a loosely-specified user config and a strict real config... but we'll see how we go there10:49
niemeyerfwereade: The only thing that knows about that is the environ10:50
fwereadeniemeyer, (I feel it is a profoundly bad thing to have the remote components inferring anything at all about their config, and they should not even accept junk like authorized-keys-path)10:50
niemeyerfwereade: Handly enough, when sending the environ itself can filter out secrets10:50
niemeyerfwereade: given that this is an implementation detail behind Bootstrap10:50
fwereadeniemeyer, sure; but I think Conn will also need to get the secrets so it update state on first access10:51
fwereadeniemeyer, none of these things are intrinsically painful, but I have yet to see how to turn it into a very clean set of CLs10:52
niemeyerfwereade: I don't think it needs to know those details10:52
niemeyerfwereade: (so far)10:52
niemeyerfwereade: It just has to send the config it has10:52
niemeyerfwereade: So we can make the operation be "send initial config if never done before" rather than "send secrets"10:52
fwereadeniemeyer, hmmmmm, so maybe we shouldn't set anything in /environment on bootstrap at all?10:54
fwereadeniemeyer, I would be a lot more comfortable if an initialized state implied valid data, even in the /environment node, eschewing only secrets10:54
niemeyerfwereade: Your idea of sending an initial configuration when we have none sounds good, but it doesn't solve the entire problem, so we can postpone it for the moment I guess10:55
niemeyerfwereade: These two options do not conflict with each other, though10:55
niemeyerfwereade: "Bootstrap with config-secrets" & "send initial config if never done before" can be friends :)10:56
niemeyerSorry10:56
niemeyerfwereade: "Bootstrap with config <minus> secrets" & "send initial config if never done before" can be friends :)10:56
fwereadeniemeyer, consider environment-specific settings that should not change, such as the control bucket10:56
fwereadeniemeyer, hm, we'll be broken anyway if that changes, won;t be able to find the bootstrap machine anyway10:57
* fwereade ponders10:57
fwereadeniemeyer, I guess it just feels kinda icky to ever *overwrite* env settings without checking them10:58
fwereadeniemeyer, it feels smarter to send public settings OAOO, and then to update them with secrets alone OAOO10:59
niemeyerfwereade: Okay, I see your point, agreed11:00
niemeyerfwereade: This can be solved in a somewhat clean way, I suppose, if we introduce ConfigSecrets in the Environ interface, returning []string11:01
fwereadeniemeyer, anyway, it is progressing nicely enough, and I'll know better how to split it all up cleanly before too long11:01
fwereadeniemeyer, yeah, something like that11:01
niemeyerfwereade: I'd be happy with that11:02
TheMueLunchtime11:13
niemeyerTheMue: Enjoy!11:14
Aramhey all.11:24
niemeyerAram: Morning11:25
fwereadeAram, heyhey11:31
fwereadeniemeyer, I just had a thought... what would happen if we made State.EnvironConfig() return an environs.EnvironConfig? allthe consequences I can currently see are positive ones11:33
fwereadeniemeyer, we wouldn't be able to casually update it as we currently do, thanks to it currently being a ConfigNode11:34
niemeyerfwereade: How do we Write it?11:34
niemeyerfwereade: Stat.SetEnvironConfig?11:35
niemeyerState11:35
fwereadeniemeyer, I had been thinking of a distinct State.UpdateConfigSettings(map[string]interface{})11:35
fwereadesorry UpdateEnvironConfig11:35
niemeyerfwereade: Does it replace or does it append?11:35
niemeyerfwereade: I suppose it replaces, so Set would be better.. anyway, +111:36
fwereadeniemeyer, cool, thanks11:36
fwereadeniemeyer, I had originally been thinking of appending, but that's by the by11:36
niemeyerfwereade: That'd mean we'd need an extra interface for removing keys, and it'd be different from the interface we have on Environ for the same purpose12:13
fwereadeniemeyer, true12:14
fwereadeniemeyer, not sure when we actually want to do that though12:14
niemeyerfwereade: Hmm.. all the time?12:15
niemeyerfwereade: I mean.. what other way would a config be updated?12:15
niemeyerfwereade: Maybe you have something else in mind that I've missed12:15
fwereadeniemeyer, I mean I don't know when we delete keys from an environ config12:15
fwereadeniemeyer, I guess I'm missing something?12:16
niemeyerfwereade: We surely are able to have missing keys, right?12:16
niemeyerfwereade: Maybe we shouldn't allow that, or should say that missing keys and a key set to "" are the same?12:16
niemeyerfwereade: Either way, the effect is the same.. Set would replace12:17
fwereadeniemeyer, IMO we shouldn't really ever have an /environment with missing keys (that aren't secrets)12:17
fwereadeniemeyer, we should do all our fuzzy guessing at defaults OAOO, when we load the config12:18
niemeyerfwereade: I don't think that's the case today, and it doesn't feel sane to force the user to have all possible keys, even those not wanted, set12:18
fwereadeniemeyer, having missing keys in the provider config has always been an irritation in the python12:18
niemeyerfwereade: In which sense?12:19
fwereadeniemeyer, self.config.get("region", DEFAULT_REGION) all over the place, occasional self.config["region"]s that blow up at unexpected times12:19
fwereadeniemeyer, it's just ugly12:19
fwereadeniemeyer, and I don't see why we shouldn't construct complete configs with real values as soon as they enter the system12:20
niemeyerfwereade: Hmm12:22
niemeyerfwereade: This is bogus indeed12:22
fwereadeniemeyer, yeah... I was somewhat upset when I saw we'd copied those flaws over12:23
niemeyerfwereade: The point, though, is that some values may be actually missing. You seem to be implying what I said above regarding having unset keys behaving as "", though12:23
niemeyerfwereade: Not all values have defaults12:23
niemeyerfwereade: In some, the "unset" value is signal12:24
fwereadeniemeyer, hmm, yes; I don't see any of those in go atm though12:24
niemeyerfwereade: That's not relevant.. the system is far from complete. We're talking about what we want to happen in those cases.12:24
niemeyerfwereade: We also can't default to "", because our config is typed12:25
fwereadeniemeyer, fair point, ok, but consider a setting that *does* have a significant effect, like ec2-uri12:26
niemeyerfwereade: Yep, good example12:27
niemeyerfwereade: What's your feeling about it?12:27
fwereadeniemeyer, that an unset ec2-uri is a really bad signal for "not really EC2, modify behaviour appropriately" :)12:28
fwereadeniemeyer, it's all very well in a user config12:28
fwereadeniemeyer, but internally I would far rather have "ec2-uri" always containing a real value, and another "this-is-not-really-ec2" value that controls the different behaviour12:28
niemeyerfwereade: I'd rather not, to be honest12:29
niemeyerfwereade: Or maybe I do.. hmm12:29
niemeyerfwereade: digesting, just a sec12:29
fwereadeniemeyer, it does feel like squishing far too much subtle meaning into a single value12:29
niemeyerfwereade: I don't understand that.. it's the endpoint we're talking to.. how's that subtle?12:30
fwereadeniemeyer, that bit is fine... but let me find a code ref12:31
fwereadeniemeyer, ok, its impact is limited, because I got annoyed and did this:12:32
fwereade    @property12:32
fwereade    def using_amazon(self):12:32
fwereade        return "ec2-uri" not in self.config12:32
fwereadeniemeyer, and even that is not heavily used12:32
niemeyerfwereade: This is wrong, actually12:33
niemeyerfwereade: Because it may be set to an endpoint in Amazon12:33
fwereadeniemeyer, it's wrong if someone sets ec2-uri explicitly12:33
niemeyerfwereade: I don't understand this particular issue12:33
niemeyerfwereade: Why!?12:33
niemeyerfwereade: It certainly wasn't the intention when I put it there12:33
fwereadeniemeyer, that is what AFAIK it has always been used for12:33
fwereadeniemeyer, I always considered it somewhat nasty12:34
niemeyerfwereade: The goal of this setting is to send the endpoint manually, as its name indicates12:34
niemeyerfwereade: I don't get the leap on how it can be something else than that12:34
fwereadeniemeyer, so, yes, it would be *even* better if we didn't have this ugly and frequently-incorrect overloading of meaning12:34
fwereadeniemeyer, I would guess that someone doing non-ec2 ec2 provider work made an unhelpful assumption12:35
fwereadeniemeyer, I stipulate that a this-is-not-amazon setting would be superior in every way12:35
niemeyerfwereade: Yeah, that sounds like the heart of the issue we've been brainstorming on12:35
niemeyerfwereade: If we need such a setting, +112:36
fwereadeniemeyer, but then, assuming we fix that12:36
niemeyerfwereade: Do we need such a setting?12:36
fwereadeniemeyer, possibly, and possibly not, I would have to look through a bit more carefully12:36
fwereadeniemeyer, but, ok, consider a correct ec2-uri setting12:37
niemeyerfwereade: Okay12:38
fwereadeniemeyer, how do we gain by leaving that as "", when we could just as easily infer the real value *only* when parsing user configs, and ensure that the contents of /environment are always correct and need no further substitutions?12:39
niemeyerfwereade: Because when a user questions for the configuration of their environment, I'd rather tell them what they've set12:39
niemeyerfwereade: We also lose the ability to do changes12:40
niemeyerfwereade: It's fine to adapt values we've set ourselves.. it's not fine to hack user-given values in general12:40
fwereadeniemeyer, I feel that argument might not apply all that well to omitted settings... don't they imply "figure it out for me", rather than "I wish to communicate with the fairyland EC2 which has no endpoint?" ;)12:42
niemeyerfwereade: Exactly12:42
niemeyerfwereade: What I mean is that "figure out for me" does not mean "figure out for me just when I bootstrap"12:43
fwereadeniemeyer, ok, I've caught up, I think I agree there :)12:44
fwereadeniemeyer, so there are indeed some settings that can reasonably be left unset12:44
fwereadeniemeyer, and still have a valid config12:44
fwereadeniemeyer, how does authorized-keys fit in here?12:46
fwereadeniemeyer, I'm pretty certain that should be an environment setting12:46
fwereadeniemeyer, and I'm also pretty sure that existence of authorized-keys-*path* in the /environment node indicates that we've screwed up pretty badly12:47
niemeyerfwereade: Agreed on both12:47
niemeyerfwereade: keys-path should be translated to keys as we do today12:47
fwereadeniemeyer, ok; would you agree that an authorizedKeysPath field on environConfig is an unwarranted temptation to foolishness?12:48
* niemeyer looks12:51
niemeyerfwereade: Which environConfig?12:51
fwereadeniemeyer, ec212:52
niemeyerfwereade: You mean providerConfig?12:53
fwereadeniemeyer, er, sorry, yes12:53
niemeyerfwereade: np, just to make sure we're on the same page12:54
niemeyerfwereade: Do you have alternative suggestions?12:54
fwereadeniemeyer, essentially, that we have 2 paths for config parsing -- fuzzy user configs and strict internal configs12:55
fwereadeniemeyer, strict configs can have empty values as discussed above ofc, but we do need to do slightly different work when interpreting user configs and when checking for validity of internal ones12:56
niemeyerfwereade: Feels like a lot of code and work for a one-off field12:56
fwereadeniemeyer, similar considerations IMO apply to region12:57
niemeyerfwereade: Hm.. why?12:58
niemeyerfwereade: The path is special because we load from the filesystem on one side, and don't care about the other side12:58
fwereadeniemeyer, because region is an env property that should be immutable12:58
niemeyerfwereade: The region doesn't feel similar12:58
fwereadeniemeyer, the "what if the default changes" argument applies in the opposite direction12:58
niemeyerfwereade: The path being changed is irrelevant12:59
niemeyerfwereade: A region being changed can be caught up on SetConfig and refused12:59
niemeyerfwereade: Seem simple12:59
niemeyerSeems12:59
fwereadeniemeyer, I was referring to the "changing defaults" argument you made wrt ec2-uri12:59
niemeyerfwereade: Erm.. I'm even more confused now13:00
fwereadeniemeyer, if we change the ec2-uri default it is a bad thing to have baked it in from the start13:00
fwereadeniemeyer, if we change the region default it is a bad thing NOT to have baked it in from the start13:00
niemeyerfwereade: I'm not suggesting we change anything right now..13:00
niemeyerfwereade: I was pointing out reasoning why having unset fields matters13:01
niemeyerfwereade: The region may be hardcoded yeah, because it can't be changed13:01
fwereadeniemeyer, ok, and that then implies again that if we get an /environment without a region, we have screwed up, just as we have if we get an /environment with an authorized-keys-path13:02
fwereadeniemeyer, and hence is IMO support for the idea that user configs and internal configs are different enough that we want separate paths for interpreting them13:03
niemeyerfwereade: Sorry, I don't see the relation between these two points.13:04
fwereadeniemeyer, both should translate to the same type13:04
niemeyerif region == "" { region = "us-east-1" }13:04
niemeyerfwereade: Why are we discussing this? :)13:04
niemeyerfwereade: The path case is special.. the region case is trivial13:05
fwereadeniemeyer, because I think that interpreting an /environment config in exactly the same way as one from environments.yaml is sloppy and risks bugginess13:07
niemeyerfwereade: I've been trying to demonstrate that this is not the case.. preventing a region from changing is if oldRegion != newRegion && oldRegion was set { return error }13:08
niemeyerfwereade: We don't need a new code path for these three lines13:08
niemeyerfwereade: Do I misunderstand?13:08
fwereadeniemeyer, since you're talking about changing regions at runtime, I think so13:09
niemeyerfwereade: Okay, so please bear with me while I adapt to reality13:09
fwereadeniemeyer, earlier you pointed out that some settings should remain empty, and not be baked in at parse time, on the basis that we may wish to change the defaults in future and having them baked into the env config would therefore be bad13:10
fwereadeniemeyer, agreed?13:10
niemeyerfwereade: Can we please focus on the specific issue of region being changed?13:10
niemeyerfwereade: Otherwise we'll derail I think13:10
fwereadeniemeyer, this is directly relevant to region changes13:11
niemeyerfwereade: Yeah, so how is region changed difficult to do with current infra?13:11
fwereadeniemeyer, do you agree with how I characterised your position above?13:11
niemeyerfwereade: THat's the core point13:11
niemeyerfwereade: Okay, I guess you do want to review the points made?13:13
fwereadeniemeyer, I am laser-focussed right at the moment, because I think it will help our communication13:13
niemeyerfwereade: Ok13:14
fwereadeniemeyer, do you agree with my statement that " earlier you pointed out that some settings should remain empty, and not be baked in at parse time, on the basis that we may wish to change the defaults in future and having them baked into the env config would therefore be bad"13:14
niemeyerfwereade: More precisely,13:14
niemeyerfwereade: So the point I made is that having unset values allows us to change the values *when that's wanted*, and I also said that it makes sense to allow values to be unset because some settings have unset values naturally, and we can't have a value like "" for unset because we have typed values in the configuration, and also that we want to show users the configuration that they've made13:15
fwereadeniemeyer, maybe we should skip sideways a little -- what is the distinction between a "naturally unset value" and an unset value that indicates that some default substitution should be made?13:18
fwereadeniemeyer, can you give me an example of the first kind?13:18
niemeyerfwereade: A naturally value is any value that is unset, and should remain as so because we want to know it's unset13:19
fwereadeniemeyer, but I can think of two cases where failing to bake them in will kill environments (even if one of them is historical)13:19
fwereadeniemeyer, that is, will kill environments if we ever change the default settings13:20
niemeyerfwereade: I don't understand.. suppose we have a new "environment-description" field13:20
niemeyerfwereade: By default we call it "No description for this environment."13:21
niemeyerfwereade: We've made a typo.. now we want to fix it13:21
niemeyerfwereade: Bingo.. we change the code so that the unset value now returns the proper description.13:21
fwereadeniemeyer, ok, that is an example where there are distinct benefits to deferring the substitution rather than doing so at parse time13:22
niemeyerfwereade: If the user has put his own description, though, we don't want to touch that.13:22
fwereadeniemeyer, I am aware of these examples13:22
fwereadeniemeyer, you already explained it perfectly wrt ec2-uri13:22
niemeyerfwereade: Yeah.. so can we go back to region?13:22
fwereadeniemeyer, absolutely! what happens when we change the default ec2 region and it has been left unset?13:22
niemeyerfwereade: We don't change it13:22
fwereadeniemeyer, never ever?13:23
niemeyerfwereade: How can we possibly change the region of a running environment?13:24
fwereadeniemeyer, well, yeah, that would be crackful13:24
fwereadeniemeyer, how can we possibly predict thatus-east-1 will be the appropriate default region for ever and ever?13:24
niemeyerfwereade: We can't, we shouldn't, and as far as we know, we don't.. !?13:25
fwereadeniemeyer, well, if we allow region to remain unset, how can we prevent a default region change from causing a *runtime* region change when that env is upgraded?13:25
niemeyerfwereade: Why don't we set the region?13:26
niemeyerJul 05 10:04:14 <niemeyer>      if region == "" { region = "us-east-1" }13:26
fwereadeniemeyer, well, we do; but that's a setting that the user did not make13:27
fwereadeniemeyer, you seemed to be arguing that we should not do that13:27
niemeyer<niemeyer> fwereade: So the point I made is that having unset values allows us to change the values *when that's wanted*13:27
fwereadeniemeyer, agreed and understood13:28
niemeyerJul 05 10:04:14 <niemeyer>      if region == "" { region = "us-east-1" }13:28
niemeyerfwereade: I said that :)13:28
fwereadeniemeyer, you *also* argued, and reiterated, that we should not show the user settings that they did not set13:28
niemeyerfwereade: True13:28
fwereadeniemeyer, these positions are in conflict13:29
niemeyerfwereade: The region case is fine, though13:29
niemeyerfwereade: and I've been saying that for a while13:29
niemeyerJul 05 10:01:50 <niemeyer>      fwereade: The region may be hardcoded yeah, because it can't be changed13:29
niemeyerfwereade: That said, there's a bit missing it seems, which is why I was trying to focus on how we do that in practice and avoid side-tracking13:30
fwereadeniemeyer, I have shown you 2 ways in which a user config is a stupid /environment config -- authorized-keys-path and region -- and I can name more (acquired-mgmt-class and available-mgmt-class, off the top of my head)13:31
niemeyerfwereade: It doesn't look like there's a place for us where we get an Environ to intercept a configuration that is about to be put in SetConfig, in a way that it has both the old and the new config13:31
niemeyerfwereade: Sorry, that path has proven unhelpful already.. I've been repeatedly pointing out that region is easy to solve, and you've failed to provide any facts for why it's not13:32
niemeyerfwereade: If there's a reason why region is hard to solve, please tell me as I do want to understand13:32
niemeyerfwereade: But focus on it.. no backtracking13:32
fwereadeniemeyer, region is not hard to solve in the small13:33
fwereadeniemeyer, the fact is that certain values, one of which is region, can be safely omitted from environments.yaml but should not be omitted in /environment13:34
fwereadeniemeyer, and that certain values that can be set in environments.yaml *should* be omitted in /environment13:34
fwereadeniemeyer, I am arguing that there is a meaningful distinction between parsing the two environment configuration styles, and that using the same code to do so is going to bite us one day13:36
niemeyerfwereade: I find much more likely that duplicating the same configuration parsing with different semantics will bite us.13:37
niemeyerfwereade: If we have problems, let's address them13:37
niemeyerfwereade: The one thing I see missing is a way to handle a configuration delta for an environment13:38
niemeyerfwereade: At no point the Environ has both old and new config in a way it could refuse it13:38
fwereadeniemeyer, agreed13:39
niemeyerfwereade: I suggest introducing ValidateConfigChange(oldConfig, newConfig)13:39
niemeyerfwereade: In the EnvironProvider interface13:40
niemeyerfwereade: And explicitly state that oldConfig will be nil when it's first set13:41
niemeyerfwereade: So that we can introduce logic in there which can act on initial setup13:41
niemeyerfwereade: Such as the handling of -path13:41
fwereadeniemeyer, hmmm, ok, I think I see where you're going13:45
fwereadeniemeyer, this would indeed address the potential problem of incompatible config replacements13:47
niemeyerfwereade: and give an entry point for handling the harcoding of region, and the weirdness of path13:48
fwereadeniemeyer, ie, exploding if we see missing region or existing path?13:52
niemeyerfwereade: Missing region is fine, we hardcode it to us-east-1 on first setup13:52
fwereadeniemeyer, sure, ok, thinko13:53
fwereadeniemeyer, but seeing an attempt to replace with missing, or otherwise change it13:53
niemeyerfwereade: existing path is an interesting question..13:53
niemeyerfwereade: I think we should ignore the setting13:53
niemeyerfwereade: and clean it up after we load it onto -keys13:54
niemeyerfwereade: So the env deployment only ever sees -keys13:54
niemeyerfwereade: Or!13:54
niemeyerfwereade: Perhaps even better13:54
niemeyerfwereade: We can load it and compare to the current -keys, and update it if so13:55
niemeyerfwereade: But never let -path itself pass onto the state13:55
niemeyerfwereade: We don't even have to compare I guess13:55
fwereadeniemeyer, wait, when are we replacing env configs at runtime on the client?13:55
niemeyerfwereade: juju set-env?13:56
niemeyerfwereade: and on the first deploy13:56
niemeyerfwereade: That stuff is done in the client13:56
niemeyerfwereade: Server side is too late..13:56
fwereadeniemeyer, ahhh, ok, so only on the first SetConfig13:57
niemeyerfwereade: juju set-env should do something like: 1) State.EnvironConfig; 2) update EnvironConfig with keys provided in the cmd line; 3) call ValidateConfigChange(old, new); 4) if all good, call SetEnvironConfig on State13:58
fwereadeniemeyer, where's the Environ?13:59
niemeyerfwereade: No where in that picture..14:00
fwereadeniemeyer, ah, I think I misunderstood you when you said "EnvironProvider interface" above14:01
niemeyerfwereade: Yeah, I meant EnvironProvider interface indeed, not Environ14:01
fwereadeniemeyer, sorry :)14:01
niemeyerfwereade: By the time it reaches the Environ it should be sweet14:01
fwereadeniemeyer, ok, I need to think about this some more -- I do see it will be useful14:08
fwereadeniemeyer, a semi-related thought: why do we error out when parsing environments.yaml if any environ has a problem? (unless that problem is "unknown type" in which case we continue)14:30
niemeyerfwereade: Convenience, probably.. I don't see either way as being specially better than the other14:31
fwereadeniemeyer, how would you feel about ignoring invalid environments in the same way we do ones with unknown providers?14:34
niemeyerfwereade: Sounds fine14:34
niemeyerfwereade: Assuming it's not the one in use, of course14:35
fwereadeniemeyer, indeed so, I need to double-check how that's handled, but it should be fine14:35
fwereadeniemeyer, ok; *then*, how would you feel about dropping authorizedKeysPath from ec2.providerConfig, and loading them into .authorizedKeys when we encounter them?14:36
fwereadeniemeyer, with what I suggested above, a bad setting won't prevent other envs from loading; but it will mean we don't have this lurking field that is profoundly wrong if used on the server side14:37
fwereadeniemeyer, (yeah, attempts to Open that environ will fall over wit the original error)14:39
niemeyerfwereade: ValidateConfigProvider takes two configurations, which were previously parsed, in a single location14:50
niemeyerfwereade: Considering this, it's not clear to me what you're suggesting14:50
niemeyerSorry, I meant ValidateConfigChange14:50
niemeyerfwereade: Where would the -path be translated into -keys?14:53
fwereadeniemeyer, I'm suggesting we do that when we load the environments.yaml, and wondering whether you would consider that to be evil14:53
niemeyerfwereade: Where would that be?14:54
fwereadeniemeyer, ec2.NewConfig14:54
niemeyerfwereade: Hmm14:55
fwereadeniemeyer, (ok, so technically just a fter we load the environments.yaml ;))14:55
niemeyerfwereade: Seems alright14:55
niemeyerfwereade: We can easily move if we find a reason why that's bad at some point14:56
fwereadeniemeyer, cool, it will make me much more comfortable, at least at the moment :)14:56
niemeyerfwereade: That's great :)14:56
fwereadeniemeyer, cheers14:56
niemeyerLunch time15:50
* niemeyer waves17:15
fwereadeniemeyer, ping18:17
niemeyerfwereade: pongus18:17
fwereadeniemeyer, I've been wondering about juju.Conn.Environ18:18
fwereadeniemeyer, and have been unable to figure out whether it matters that it is potentially out of sync with the state18:18
fwereadeniemeyer, I think the answer is probably "not yet, but it will do as soon as we can extract config information from it"18:23
niemeyerfwereade: Do you have a scenario that is concerning you, so that I can better approach your thinking?18:23
fwereadeniemeyer, well, a while ago IIRC we tentatively agreed that DefaultSeries would be a sensible thing to add to Environ18:24
fwereadeniemeyer, if it is, then as soon as the deploy command tries to get it (as loaded from environments.yaml) rather than from the /environment-backed config18:25
fwereadeniemeyer, ...then we have a problem18:25
fwereadeniemeyer, I am -1 on that now18:25
fwereadeniemeyer, we probably need DefaultSeries, and the other common properties, on EnvironConfig18:26
fwereadeniemeyer, and since we can't get an EnvironConfig from an Environ it thus doesn't matter18:26
fwereadeniemeyer, since the only thing we will need from Environ is Secrets18:27
fwereadeniemeyer, and that will only be relevant when the State environ config has no secrets of its own to overwrite18:27
niemeyerfwereade: Hmm18:28
fwereadeniemeyer, so, on reflection, what I'm really asking is "are we OK with adding DefaultSeries, Name, Type, etc, (AuthorizedKeys?) to EnvironConfig?"18:28
niemeyerfwereade: We can't get an EnvironConfig from an Environ?18:28
fwereadeniemeyer, don't think so, may be blind18:28
fwereadeniemeyer, nope, no access18:29
niemeyerfwereade: A bit surprising, but I guess it's ok.. we need the config to create the environ in the first place18:29
niemeyerfwereade: Which means we must already have it18:29
fwereadeniemeyer, I think this is actually a good thing, there's no need to know any of that information (other than the secrets) from the Environ18:29
niemeyerfwereade: So, to your question18:30
niemeyerfwereade: What do you mean by "DefaultSeries", more precisely?18:30
niemeyerfwereade: What is that, a constant, a method, ..?18:30
fwereadeniemeyer, I mean the value of the default-series key, returned from a method that is part of the EnvironConfig interface18:31
niemeyerfwereade: No, that's bogus18:31
niemeyerfwereade: Wait, wait18:31
niemeyerfwereade: Not in the way I initially thought.. I see you selected the things that are meaningful cross env18:32
niemeyerfwereade: Hmmm18:32
fwereadeniemeyer, yeah, that was the idea18:33
niemeyerfwereade: Not sure.. it feels like we'll be increasing the API surface, and in a way faking it out because we do need to know the string values in more than one place (set-env, parsing, etc)18:33
niemeyerfwereade: What's the goal?18:34
fwereadeniemeyer, proximate goal is "implement deploy as required" ;)18:34
fwereadeniemeyer, for that I need the default-series one way or another18:34
fwereadeniemeyer, and I'm not quite sure that it's better to have State.EnvironDefaultSeries than it is to have State.EnvironConfig return an EnvironConfig that I can then ask18:35
niemeyerfwereade: Sure, but what I'm comparing against is config.Get("default-series")18:35
fwereadeniemeyer, ofc I could just use the ConfigNode directly :)18:35
niemeyerfwereade: Which you can then ask anyway18:35
niemeyerfwereade: Didn't we talk about that yesterday, or was it Dave?18:36
fwereadeniemeyer, I guess I just hate having the environment config node available for direct manipulation outside the state package18:36
niemeyerfwereade: I see18:36
niemeyerfwereade: So, I like the principle of what you want too18:36
fwereadeniemeyer, you seemed cautiously +1 on State.EnvironConfig returning an actual EnvironConfig earlier :)18:37
niemeyerfwereade: Let's try to solve the key issue I mentioned above then18:37
niemeyerfwereade: Yes, I still am18:37
niemeyerfwereade: It sounds reasonable.. we just have to solve that mismatch in a nice way18:37
fwereadeniemeyer, jolly good :)18:37
niemeyerfwereade: We have two places (config and cmdline) talking about env-config in terms of string keys18:37
niemeyerfwereade: How do we translate that?18:37
fwereadeniemeyer, well, I'm not 100% sure on this, but for context's sake consider get_serialization_data in python18:38
fwereadeniemeyer, I don't think we *have* to expose it in the same way, but it might be quite nice18:39
fwereadeniemeyer, ha, no, forget it18:39
fwereadeniemeyer, the keys in that are not necessarily the valid keys for the env18:40
niemeyerfwereade: Right18:40
niemeyerfwereade: We're translating key <=> method18:40
niemeyerfwereade: and there's the extra complexity that Environ is an interface18:41
niemeyerfwereade: Hmm18:42
niemeyerfwereade: That translation is exactly what NewConfig does18:42
fwereadeniemeyer, those schema.FieldMaps are looking like they suddenly want to be reused somehow :)18:44
niemeyerfwereade: They are already being used, precisely within NewConfig18:44
fwereadeniemeyer, yeah; and they're doing ~exactly the job we want them to do in env-set18:45
fwereadeniemeyer, except some can't be changed18:45
niemeyerfwereade: Well, and that's exactly the method we should use there?18:46
fwereadeniemeyer, how do we properly error on unrecognised settings?18:46
niemeyerfwereade: NewConfig does18:46
niemeyerfwereade: (already)18:46
fwereadeniemeyer, no it doesn't18:46
fwereadeniemeyer, there's a commented-out test saying "should we error?"18:47
niemeyerfwereade: Ah, sorry, ok.. it's the opposite.. it errors on required+not found18:47
fwereadeniemeyer, I would be +1 on making that stricter though18:47
niemeyerfwereade: =118:47
niemeyerfwereade: +118:47
niemeyerfwereade: =1 is great, though :-)18:47
fwereadehaha18:47
fwereadeniemeyer, ok, with more strictness, it's trivial18:48
fwereadeniemeyer, grab serialization data from existing config, poke in new key, reparse, and ValidateConfigChange18:48
niemeyerfwereade: Woohay!18:48
fwereadeniemeyer, ok, I'll make the parsing stricter18:49
niemeyerfwereade: Reading it, ValidateConfig should be clear enough, btw18:49
niemeyerfwereade: The parameters will make the semantics clear18:49
fwereadeniemeyer, with (old, new EnvironConfig), yeah18:49
niemeyerfwereade: Right18:49
fwereadeniemeyer, bah, keyword18:50
fwereadeniemeyer, we'll think of something :)18:50
niemeyerfwereade: Interestingly, it's not18:50
fwereadeniemeyer, oh! sweet!18:51
niemeyerfwereade: new, make, delete, etc18:51
niemeyerclose18:51
fwereadeniemeyer, yeah, all functions :)18:51
fwereadeniemeyer, hadn't appreciated that properly until just now :)18:51
niemeyerfwereade: Yeah, handy18:52
fwereadeniemeyer, anyway, I'd better be off for a bit18:52
fwereadeniemeyer, ttyl18:52
niemeyerfwereade: Have a great one19:00
fwereadedavecheney, ping23:26
davecheneyfwereade: ack23:28
fwereadedavecheney, I'm planning to change State.EnvironConfig and associated watcher to return actual environs.EnvironConfig~s; any thoughts/objections?23:30
davecheneyfwereade: is that type already in the tree ?23:30
fwereadedavecheney, yeah, it's the interface implemented by all env config types23:30
fwereadedavecheney, so hopefully the provisioner will just be able to SetConfig(newConf) and not worry about errors23:31
davecheneyfwereade: sounds excellent23:31
davecheneymy concern would be in the type of the thing that is passed to the SetConfig (ish) method in the actual provider23:32
fwereadedavecheney, assuming we manage to avoid monumental scrwups like somehow saving environments for the wrong type, anyway ;)23:32
davecheney+123:32
fwereadedavecheney, expand on concern?23:33
davecheneyfwereade: i'm guessing that inside, say, the ec2 environ, it still gets the config object of it's coerced type ?23:36
davecheneyif that is true, then I have no objection23:36
fwereadedavecheney, the Environ.SetConfig methods all cast to the appropriate type, and as I said if we don't manage to screw that up we're golden23:37
fwereadedavecheney, the plan is not to even bother sending configs that aren't valid23:38
davecheneyi originall had a type assertion at that point, but gustavo felt it was appropriate to just let the runtime panic23:38
davecheneywhich is pretty reasonable given how hard you would have to fuck that up23:38
fwereadedavecheney, yeah, I think that's way over on the "programmer error" end of the spectrum23:38
fwereadedavecheney, f7u12: state.AssignmentPolify, state.Info; both used by environs; import loop23:46
fwereadedavecheney, not so bad, I just need to move them somewhere, but eww23:47
fwereadedavecheney, I'll decide where tomorrow, sleeping now ;)23:47
davecheneyurgh23:48
davecheneyfwereade: want to leave it with me ?23:48
fwereadedavecheney, no worries, I'll deal with it :)23:48
davecheneykk23:48
fwereadedavecheney, (both of them seem to me to be reasonable members of environs instead; sanity check?)23:49
davecheneyniemeyer: thanks for the review23:49
davecheneyi'll do a followup fix for the lower case naming thing (fscking us-east-1)23:50
davecheneyfwereade: is there a commit syntax for lp that closes issues ?23:53
davecheneyFixes issue 1234. ?23:53
fwereadedavecheney, err, I forget, something like that; but I think lbox takes -bug 12345623:54
davecheneyahh, cool23:54

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!