[02:27] * thumper tries to formulate some replies [02:31] thumper: did you want to look at my local-storage auth change before I land? I have fwereade's LGTM [02:31] axw: have I reviewed it? [02:31] thumper: no, just fwereade [02:31] * thumper is replying to the upgrade emails [02:32] nah, I trust you, and if fwereade has acked it, that is good enough for me [02:32] okey dokey [02:32] cool [02:32] thumper: the gist is, using TLS client certs for authentication; authentication=authorisation [02:33] * thumper nods [02:58] school run time [03:16] axw, wallyworld: reminder on the code review time in 15 min [03:16] yep [03:21] thanks [04:26] ugh [04:27] having a very meh day [04:36] thumper: I didn't really understand this statement in your email- "I think I agree with William here, these are more associated with a [04:36] provider rather than just agent config." [04:36] axw: I was thinking that we wouldn't need to put it in the config [04:36] my intention was for the provider to control what jobs get put into the MachineConfig, which drives agent.Config creation [04:36] but in state [04:36] thumper: yes, but the bootstrap process puts the jobs into state [04:37] the machine jobs are defined in state [04:37] are we talking about the bootstrap jujud process? [04:37] or the machine agent? [04:37] yep [04:37] um [04:37] what's the difference? [04:37] cmd/jujud/bootstrap.go [04:37] one is 'jujud bootstrap' the other is 'jujud machine' [04:38] currently does an InjectMachine with hard-coded set of jobs [04:38] there is a key/value store in format 1.16 [04:38] oh right. I'm talking about the former [04:38] for the agent config [04:38] so if you use that, you don't need to teach the format anything new [04:39] hmm yes, it just felt a bit wrong to put it into something so free-form... but I suppose the machine agent doesn't care about this at all [04:39] right [04:39] but I do get your meaning [04:39] could be command line params to the bootstrap ? [04:40] although that isn't much better [04:40] and then creates a mix of data sources [04:44] thumper: they've all got pros and cons, so I guess key/value will do; at least there's no format change required then [04:44] * thumper nods [04:44] thumper: it sounds like we're thinking along the same lines regarding upgrades [04:44] good [04:45] I was thinking more after I sent my email last night, and that the code that I will need to add for this could be used for state schema upgrades [04:45] even the multi-version jump? [04:45] yes definitely [04:45] good [04:45] I think that previously that has been in the "too hard" basket [04:45] having to hop along like that sounds horrible [04:45] but it definitely needs to be done [04:45] I keep thinking back to what we did with postgresql [04:45] where we had a number of patch files [04:46] a similar thing could be done for our version changes [04:46] for each minor version, we have a slice of functions to call [04:46] that will modify state based on what was left by the previous call [04:46] yep that makes sense [04:46] so going from say 1.16 -> 1.18 would just require the one stack to be called [04:46] but 1.16 -> 1.22 (say) would run three, one after the other [04:47] * thumper waves hands [04:47] magic [04:47] :) [04:48] thumper: so I'm vaguely thinking that there'll be an environs/upgrade package, which will handle generic upgrades, and defer to an optional EnvironUpgrader interface, which an Environ may implement [04:48] haven't thought through in much detail yet [04:49] environs/upgrade would do schema upgrades [04:49] sounds good to me [04:49] EnvironUpgrader would do provider-specific things, like adding jobs [05:03] axw: I'm wondering whether state/upgrade should do schema upgrades given that state has the core "state" [05:04] thumper: yes, probably [05:05] * thumper calls a wash on today, back tomorrow === axw_ is now known as axw [07:12] mornin' all [07:15] hey rogpeppe [07:15] axw: hiya [07:22] nice to see juju getting a shout out in this talk https://www.youtube.com/watch?v=sYukPc0y_Ro [07:23] rogpeppe: yeah, saw that this morning - very nice! :) [07:39] axw: last night thumper mentioned an email thread about upgrades - where is that? [07:40] rogpeppe: I emailed fwereade and rogpeppe only; I'll forward to you if you're interested [07:40] err not rogpeppe ) [07:40] axw: please [07:40] :) [07:40] thumper [07:41] actually I'll just CC the list, since it's grown into something that everyone will probably care about [07:41] good plan [07:42] wallyworld: hiya [07:42] hi [07:42] wallyworld: you've replied to this, but i don't think you've pushed your changes yet : https://codereview.appspot.com/13842044 [07:42] i have [07:43] i pushed to lp [07:43] wallyworld: could you re-propose too, please? [07:43] it's been merged also [07:43] :-( [07:43] wallyworld: so i can see the changes in codereview [07:43] i don't have that branch easily assessicle right now [07:43] look at lp [07:43] ok [07:43] bit stressed trying to get @^%^@!^%@! gpg working [07:44] i can't generate a private key have have Go use it [07:44] wallyworld: but in general it's nice to have the pushed version in codereview because the link is in the commit message [07:44] guess so [07:44] i hate codereview [07:44] wallyworld: you can't generate a private key? [07:44] too hard to use [07:44] wallyworld: funny, i think it's really good, but there y'go [07:45] i can generate fine'but when i try and use it in go, it complains it is encrypted [07:45] wallyworld: what library are you using? [07:45] so i have a private key block which i got from pgp, then i use keyring, err := openpgp.ReadArmoredKeyRing(bytes.NewBufferString(signedMetadataPrivateKey)) [07:46] i trying to generate some signed data to test with, i have both public and private key blocks from a test key i generated in gpg [07:46] errors.InvalidArgumentError = "signing key is encrypted" ("openpgp: invalid argument: signing key is encrypted") [07:46] is the error [07:47] i'm not sure how to make a signing key that is unencrypted [07:48] rogpeppe: i hate codereview because i can't see the whole diff at once, so it's hard for me to navigate around the changes [07:48] wallyworld: i like codereview because i can see the whole file in context (and all the comments) [07:48] wallyworld: but it would be nice if it was much faster [07:49] yeah [07:49] rogpeppe: i got to go and buy dinner, but if you had a clue as to how I can get a pulic and private key i can use with openpgp in go that would be awesome [07:50] wallyworld: i'll have a look [07:50] thanks [07:50] i have a test private key but that comes from g a go lib test [07:50] wallyworld: did you see this bug BTW? https://bugs.launchpad.net/juju-core/+bug/1229839 [07:50] <_mup_> Bug #1229839: provider/ec2: LiveTests.TestBootstrapWithDefaultSeries is broken [07:50] and the data is precanned [07:50] i replied [07:51] wallyworld: so you haven't generated the private key yourself? [07:51] we have had that SetToolsPrefix in the code for a long time' [07:51] wallyworld: it's hideous! [07:51] rogpeppe: i have, but go refuses to import as [07:51] per the above error [07:52] rogpeppe: i have a chunk of text wrpped with -----BEGIN PGP PRIVATE KEY BLOCK----- etc [07:52] but i can't use ReadArmoredKeyRing to get something i can sign stuff with [07:52] wallyworld: what command line did you use to generate your gpg private key? [07:52] wallyworld: just so i can reproduce your issue [07:52] rogpeppe: i used seahorse and exported it [07:53] * rogpeppe hasn't heard of seahorse [07:53] i generated a sign only key [07:53] it's a gpg gui [07:53] rogpeppe: but i guess "gpg --gen-key" would have worked [07:54] wallyworld: that's ok, i can never remember the gpg usage either :-) [07:56] wallyworld: I just generated one, it worked for me [07:56] with --gen-key [07:56] wtf [07:57] can you read it using ReadAmouredKeyRing [07:57] without getting an error [07:57] wallyworld: yep [07:57] did you type in a passphrase when you generated it? [07:58] axw: i'm stupid [07:58] the reading in bit qorks [07:58] works [07:58] it's the signing that fails [07:58] plaintext, err := clearsign.Encode(&buf, keyring[0].PrivateKey, nil) [07:58] fails [07:58] ah [07:58] using the keyring read in [07:59] wallyworld: did *you* type in a passphrase? [07:59] will seahorse allow you not to? [07:59] no, i'll try command line [08:00] (gpg --gen-key assigned one even when I said not to) [08:00] not sure if that will force it [08:01] i how do i get an unencrypted private key is the question [08:01] don't know why Go needs that [08:01] anyway, got to run to get dinner, back layter [08:05] wallyworld: call PrivateKey.Decrypt(passphrase) [08:06] wallyworld: see the comment on the PrivateKey.Encrypted field (http://godoc.org/code.google.com/p/go.crypto/openpgp/packet#PrivateKey) [08:07] rogpeppe: if you didn't see it yet, I forwarded the upgrade email chain to juju-dev [08:07] axw: thanks [08:20] axw: here are some notes i made on major-version upgrades a while ago: http://paste.ubuntu.com/6153562/ [08:26] rogpeppe: heh, pretty much what I wrote I think :) nice to know I wasn't off base [08:26] the pending flag bit is a bit different [08:26] axw: yeah, it seemed pretty similar [08:26] davecheney, still around? [08:26] rogpeppe: I think thumper was suggesting that the API server just lock out all connections, and API clients keep attempting to reconnect [08:27] so the "pending flag" is in effect cleared when they can finally connect [08:27] I kinda like that approach [08:31] axw: just shutting off access to all agents seems a little draconian. [08:31] axw: but it does have some advantages [08:32] axw: like you don't have to wait for everyone [08:32] axw: but i think it might be good to let all the agents download their new version before we start to upgrade [08:33] axw: that way we can have a clean break [08:34] rogpeppe: what if the API server fails during upgrade? how would you roll back the tools on the machine agents? [08:34] or do you mean, download but don't action [08:35] axw: both are possible [08:35] jam: i have 2 simplestreams mps, which fix the issues you were talking about, i'm hoping to get these into 1.15 https://codereview.appspot.com/13899044/ https://codereview.appspot.com/13888043/ [08:35] i got to go make dinner, but back later [08:35] axw: if we make sure that at least the new API is backwardly compatible to the point of finding out the agent version, then agents can downgrade if the upgrade fails [08:36] that sounds reasonable [10:01] fwereade, hey [10:01] fwereade, quick question [10:02] fwereade, if we don't take an environ in NewSimpleAuthenticator in the provisioner, how should we get the state and api infos? The manual provisioner needs a state connection I think, so we can't just return nil stateInfo [10:46] fwereade: standup ? [10:46] https://plus.google.com/hangouts/_/3d4586c62aa1310a0c3f40960494578688c86f1a [10:50] fwereade: ^^ [11:26] huh, hangout experiencing difficulties [12:16] * TheMue => lunch [12:30] fwereade, ping [13:33] sinzui: ping for standup [14:49] natefinch, https://codereview.appspot.com/13802045/ LGTM with comments (and a followup?) -- let me know what you think [14:50] natefinch, wait, sorry, I seem to have skipped some files [14:52] natefinch, added to it [14:56] natefinch, maybe not LGTM any more, but should be reasonably simple to address [14:57] natefinch, I think you just need to make a distinction between nil tags and empty tags [15:10] fwereade: good point about masking tags [15:41] ping [15:42] :) [15:42] i meant fwereade ping [15:42] was going to ask :) [15:45] dimitern, heyhey [15:45] dimitern, looking [16:17] fwereade: good call on empty tags... we actually aren't handling it well at all. tags= is getting treated as a single empty string tag, rather than an empty list [16:20] erk [16:21] mgz, eh, that's what review are for [16:21] ironcically, the "round trip" serialization tests don't catch this (I didn't have a test for it, but I added one, that still passes) [16:21] natefinch, thanks for checking it out [16:21] fwereade: yeah, but I really should have looked out for that >_< [16:21] because []string{""} serializes the same as []string{} [16:22] so, I wrote a specific test for it to check that tags= gets deserialized into []string{} ... which currently fails, but I'll fix that. TDD, woo! [16:23] yeay! [16:23] * fwereade cheers [16:33] fwereade: fancy a small review (ExtraConfig in configstore)? https://codereview.appspot.com/13912043/ [16:33] mgz, dimitern, TheMue, natefinch: ^ [16:34] looking [16:34] me too [16:39] rogpeppe: one minor comment, otherwise LGTM [16:40] TheMue: i'd prefer not to define yet another map[string]interface{} type [16:40] TheMue: i'm wondering about consolidating all of them to one type [16:40] TheMue: but in the meantime it's nice to avoid the type conversion if it's used with testing.Attrs, for example [16:41] rogpeppe: as those types doesn't cost a lot i like them explicit to give a better semantic expression [16:41] TheMue: what does that mean? [16:42] rogpeppe: especially when you set data not directly near to the function call [16:42] TheMue: sorry, i'm not sure i get you [16:43] rogpeppe: you can say cfg := ExtraConfig{"foo": 1234} then do something else, maybe add some more config, and later call SetExtraConfig(cfg) [16:43] rogpeppe: in this case already the initialization shows the usage of the data [16:43] TheMue: it's never going to be used that way [16:44] rogpeppe, +1 on naming types like map[string]interface{} [16:44] rogpeppe, we have too many too generic ones already [16:44] rogpeppe: sure for every developer until eternity? ;) [16:44] dimitern: i think map[string]interface{} expresses exactly what is required [16:44] dimitern: but i'm thinking of creating utils.Attrs and making *everything* use that [16:45] rogpeppe: it's technical, yes, but it doesn't say anything about the intention [16:45] rogpeppe, i don't want to argue, just giving my opinion [16:45] dimitern: thanks [16:45] TheMue: neither does "int", but we don't retype every int [16:45] TheMue: the argument name or function name is usually good enough [16:45] rogpeppe: will start to refactor :D [16:47] rogpeppe: but if it is enough you don't even need to create a utils.Attrs [16:48] rogpeppe: then let as stay with it as it is [16:48] TheMue: the current advantage to using an unnamed type is that it's compatible with other named types. [16:49] rogpeppe: that's right [16:50] TheMue: and also, given that config.Config.AllAttrs returns map[string]interface{}, I think my function should accept that exact type, because that's where its attributes are coming from [16:50] rogpeppe, there's already testing.Attrs btw [16:50] many attrs ;) [16:51] rogpeppe, and it's map[string]interface{}, so maybe while deciding to refactor all such cases should be considered in the codebase [16:51] dimitern: i know - i made it :-) [16:51] * TheMue has to step out, somehow i'm stuck with my testing and will start freshly tomorrow morning [16:51] dimitern: i'd move it to utils [16:51] dimitern: and then make everything use it [16:52] rogpeppe: +1 for utils, yes [16:53] so, good n8 everybody [16:58] mgz: about the overall intention of the ExtraConfig stuff: the intention is that the extra configuration attributes are created only once, at Prepare time; subsequent operations should not add attributes, as that may well be racy. [16:59] mgz: adding endpoint address information should not be racy, as everyone will be trying to save the same info, so it's idempotent [17:00] rogpeppe: okay, I sort of see [17:00] but then how does that distinguish from just not letting the config get overritten once in place? [17:00] *w [17:00] (having the bool "has this been created" flag, that is) [17:01] mgz: well, create fails if the info already exists [17:01] mgz: so you can only set extra config attrs if you are the one that first created the info [17:02] mgz: ... does that make sense? i'm not sure i understood your question [17:03] hm, sort of, I think this is implementation detail rather than something that will break things later anyway [17:04] man, I love tests. I found like 4 bugs with the tests I wrote in the last hour [17:04] natefinch: :D [17:04] mgz, fwereade: btw, you both missed the fact that constraints.IsEmpty had no tests, and actually had a bug in it ;) [17:05] I complained about the indenting, which should give me half points... [17:05] rofl [17:06] the indenting is as gofmt would have it, all hail gofmt [17:06] but but... it makes no sense ;_; [17:06] mgz: it's grouping [17:07] mgz: it's showing explicitly with the indenting what the implicit order of operations will do [17:08] mgz: gofmt does similar stuff when it's all on one line by using spaces or no spaces to show what will get calculated together, it's actually pretty awesome [17:09] fwereade, there it is the secret attrs patch https://codereview.appspot.com/13916043 [17:10] oops, just pastebin'ed all my aws security credentials [17:10] natefinch: I'd understand if it was just the indent after the ||, it's the indent after the first && that makes no sense [17:12] fwereade: FYI here's what the environment info file looks like with *all* the attributes stored in it: http://paste.ubuntu.com/6155286/ [17:12] mgz: it makes more sense if you put another || at the end like this: http://pastebin.ubuntu.com/6155319/ [17:14] okay, but that still took a little bit of thinking about, I'd be breaking out the brackets if I got that far [17:14] mgz: yes, in general... but it's handy if you think you know what it's doing , and turn out to be wrong... at least you get a hint of it. [17:15] mgz: this kind of thing has already saved my butt: https://groups.google.com/forum/#!msg/golang-nuts/qFtA6g8AIxE/oumyR1ytP4MJ [17:15] rogpeppe, hmm, I dislike that a bit less than I expected to [17:16] fwereade: i think it has some definite advantages - it's nice being able to see everything in one place, minus all the config initialisation magic [17:16] natefinch, ouch, I was so happy to see IsEmpty everywhere [17:16] fwereade: well, there's a test for it now, no harm, no foul. :) [17:17] * fwereade strokes his chin thoughtfully [17:17] (given that I was the wrong that wrote it and failed to write a test.....) [17:17] s/wrong/one [17:21] rogpeppe, it just feels like it's asking to replicate the Conn problems [17:21] fwereade: the Conn problems? [17:21] rogpeppe, the Environ [17:21] rogpeppe, which is wrong [17:21] rogpeppe, and which people keep trying to use because it's so ridiculously convenient [17:22] fwereade: well, we'll want to purge the environment from the environ info when we can [17:22] fwereade: (in fact, it might even be a command) [17:22] fwereade: really, we only want to keep it around until after the first connection [17:23] fwereade: but there's that problem with what happens if you lose contact with the last-contacted API address [17:23] fwereade: but that's a problem for everyone, and most people *won't* have the extra config attrs [17:24] fwereade: so i guess i'd prefer to work on that problem (finding API endpoints without any environ credentials), and just throw the credentials away at the first opportunity [17:25] rogpeppe, well, up to a point -- we *do* still have the creds around in environments.yaml and it would be silly not to use them [17:25] fwereade: well, one person does, yes [17:26] fwereade: but by fixing the issue for other people, we fix it for that person too [17:26] rogpeppe, yeah -- taking that knowledge (along with eg control-bucket) away from that user is not helpful I think [17:26] fwereade: i'm not sure how much mechanism is worth adding here [17:27] rogpeppe, I accept that a global fix would be ideal [17:27] rogpeppe, but today we still need actual environ access for the CLI in a bunch of situations [17:27] fwereade: indeed [17:27] fwereade: i don't see what difference it makes how many attributes you've got in that file though [17:28] fwereade: you've either got the environ attrs or not [17:28] fwereade: keeping them all in one place makes our life simpler [17:28] fwereade: (and probably users' lives too) [17:33] rogpeppe, I need to sleep on it I think [17:33] wtf is params.StatusData ? [17:33] rogpeppe, at the moment I still feel that if yu can get a complete environ config from that file alone we screwed up [17:33] dimitern, dict of useful info attachable to status [17:33] somebody landed this and changed machine.SetStatus() underneath [17:33] dimitern, relation hook status reporting is particularly bad [17:33] ok, so do need to care about it in the provisioner? [17:34] dimitern, I think you can ignore it safely for now, nothing sets it on a machine [17:35] fwereade, ok thanks [17:36] rogpeppe, I do see the advantages in rendering to the One True Format, once, explicitly, though [17:36] fwereade: that's interesting. i'm not sure *how* that means we've screwed up, given that environments.yaml+info = the same thing [17:36] rogpeppe, the problem is that it looks like a sane config but most if the time it is not [17:37] rogpeppe, I have a thought though [17:37] rogpeppe, what if we called it creation-config [17:37] rogpeppe, that might make its status a bit more apparent [17:37] fwereade: that's not a bad idea [17:38] fwereade: or bootstrap-config [17:38] rogpeppe, it's still a bit risky in the same way Conn.Environ is, but it's better than before [17:38] rogpeppe, perfect [17:39] I'm getting a weird test failure in state/service_test.go in TestUpdateConfigSettings [17:39] code that is no where near anything I've touched [17:39] rogpeppe, validate and complete it OAOO, record it locally so we can finish bootstrap nicely, subsequently stick to changing the state-servers field and fall back to bootstrap-config just in case it offers a save for a screwed situation [17:40] fwereade: i could imagine a command, say clear-bootstrap-info, which would delete the local bootstrap config [17:40] fwereade: although it's probably unnecessary [17:40] rogpeppe, I'm not sure it deserves a UI [17:40] rogpeppe, anyone who can be trusted to run it can handle deleting a few lines;p [17:40] fwereade: yeah [17:41] fwereade: what *might* be nice though is a way of exporting the file without the bootstrap-config [17:41] rogpeppe, yeah, I think that definitely deserves a UI [17:41] rogpeppe, import/export environment basically [17:42] fwereade: yeah [17:42] rogpeppe, but I think that vocab's taken so we should think of something else [17:42] fwereade: endpoint [17:42] fwereade: export-endpoint, import-endpoint ? [17:43] rogpeppe, I'm wondering whether it's more like an identity almost [17:43] fwereade: it would be great if it contained the environ's UUID [17:44] rogpeppe, that's a good point [17:44] mgz, fwereade, rogpeppe, dimitern: help with this test failure? I don't know where it's coming from - http://pastebin.ubuntu.com/6155445/ [17:44] rogpeppe, we do something annoying like defer its creation to bootstrap-state now, don't we? [17:44] fwereade: tbh it would be nice if the UUID was generated at bootstrap time [17:44] fwereade: yeah [17:44] fwereade: i argued against that at the time :-) [17:44] rogpeppe, well forseen [17:45] rogpeppe, eh, we can always add that to the info file, along with the state-servers, rather than jam it into config [17:45] fwereade: that's true [17:45] rogpeppe, just pick it up a bit later [17:46] natefinch, probably you need to merge trunk and frank's changes for updating default/empty settings values? [17:46] fwereade: although it would be nicer to verify that the other end has that UUID rather than taking it for granted we're talking to the right one [17:46] dimitern: hmm... I did just merge trunk.. let me double check [17:47] rogpeppe, agreed that Prepare-time UUID generation would be nicer [17:47] rogpeppe: any chance to not double space the cert? [17:48] jam: that's just how goyaml produces multiline strings [17:48] jam: if you wanted to patch goyaml... :-) [17:50] rogpeppe: is it part of the yaml spec? in that if you don't double space you end up with everything on-one-line ? [17:50] jam: inside single quotes, yes [17:50] rogpeppe: you can always use a custom type with a MarshalYAML or whatever the func is. [17:50] jam: yaml has the most abstruse quoting rules ever [17:50] jam: that can't help us, unfortunately [17:51] jam: because MarshalYAML returns interface{} containing values which will be marshalled according to the usual yaml marshalling rules [17:51] jam: GetYAML, i mean [17:52] * rogpeppe has reached eod [17:53] dimitern: all set. looks like it was old cruft from go install that needed to be rebuilt. [17:54] natefinch, ah, yeah - we keep bumping into that [17:54] fwereade, any chance of looking at https://codereview.appspot.com/13916043 ? [17:55] fwereade: seems like we have a plan [17:55] dimitern, sorry, got lost on the stack, doing it now [17:55] dimitern: yeah, I've hit it before, just didn't think of it this time... I just try never to do go install anymore... it's too fraught with problems like that [17:55] rogpeppe, I think so [17:55] rogpeppe, thanks [17:55] fwereade: cool. [17:55] fwereade: i'm happy. [17:55] natefinch, I tend to always do go install in cmd/juju and cmd/jujud before doing a live test with --upload-tools [17:56] g'night all [17:56] natefinch: I always use "cd cmd/juju; go build; ./juju ..." which means it won't ever find "jujud" next to it :) [17:56] dimitern, nice, LGTM [17:57] jam: yeah, I've taken to the go build; ./juju thing too [17:57] fwereade, cheers, the next one coming up shortly [17:58] dimitern, awesome, I am workecueing and will be around for a bit [17:59] fwereade, nice! :) [17:59] mgz, fwereade: updated the maas-tags stuff with some more tests and bug fixes and now we handle nil tags vs empty list of tags properly https://codereview.appspot.com/13802045/ [18:09] fwereade, and the last one https://codereview.appspot.com/13919043 [18:36] natefinch, LGTM [18:36] dimitern, looking [18:37] fwereade, thanks! [18:38] dimitern, natefinch: if you're both around, I think we may want to verify that we can still deploy a container on maas with the changes [18:38] and I believe natefinch to have something resembling reliable maas access [18:41] dimitern, pending verification, LGTM [18:41] natefinch, you just need "juju deploy wordpres --to lxc:0" or something and make sure it works [18:41] fwereade, but natefinch will need to pull my branch, right? [18:42] * dimitern brb [18:53] fwereade, you a maas live test specifically? [18:54] dimitern, that'sd where we need that providsioner [18:54] sorry knuckle typing [18:54] pig fat [18:54] :) [18:54] I can do a live tests on ec2 or the local provider (or both) tomorrow [18:55] ec2 runs no lxc proisioner now I Thought [18:55]