=== vladk is now known as vladk|offline [00:12] no caps letters in services names. [00:12] * hazmat` wonders why [00:12] url-able? [00:14] rick_h_, urls can include caps letters. lwn.net/Articles/587373/#Comments [00:15] hazmat`: sure, but it's generally nice to make Comments and comments not do different things [00:15] humbug ;-) [00:59] axw: hiya, when you have a minute, can we have a hangout? [01:00] wallyworld_: heya, sure just a moment [01:01] ok, i'll set one up for when you are ready [01:01] https://plus.google.com/hangouts/_/7acpj2ql7v7hcjv3aea5s03kog [01:02] wallyworld_: ready... it says "the party is over", did you close it? [01:03] no [01:03] hmm, let me try again [01:03] i sent an invite [01:07] wallyworld_: I think I lost you [01:07] nope [01:26] * thumper digs out the ice pack [01:28] thumper: stop hitting yourself [01:29] I was even wearing my head gear [01:40] menn0: are you reusing branches? [01:40] i was just writing to you about that :) [01:40] i'm using the same branch because it's a continuation of the same work [01:41] not the right approach? [01:41] (or could I have used lbox propose --new-bug?) [01:43] thumper: ? [01:43] --new-bug is something different [01:43] I tend to use new branches, based of the old ones, for continuation of work [01:44] this way, branches linked to bugs, merge proposals are simpler, and lbox doesn't get confused [01:44] the code review there now shows it as already approved [01:44] yeah i saw that :( [01:44] more a function of our tools and workflow [01:44] so to recover... delete the review and try again from a new branch [01:44] ? [01:44] nah, don't worry for this one [01:45] just try to remember for the next [01:45] ok [01:45] and we should write that in the developer documentation :-) [01:46] writing that down now ... [01:47] lunch time ... [01:53] wallyworld_: are you still wanting to catch up weekly? [01:53] sure, why not [01:54] ok, I'll just go make some lunch [01:58] wallyworld_: seems this is P1 -- https://bugs.launchpad.net/juju-core/+bug/1316869 [01:58] <_mup_> Bug #1316869: juju sync-tools destroys the environment when given an invalid source [01:58] wallyworld_: do you want me to look at this, or keep looking at the LXC thing? [01:58] axw: i thought at first glance that env one was a non issue? [01:58] but maybe they want to keep the newly created env [01:59] wallyworld_: no I don't think so. after looking at the log more closely, it shouldn't be destroying [01:59] in that case fix that one first [01:59] we'll need to get a release done with that, both 1.18 and 1.19 i expect [02:32] axw: before you start looking at the fast lxc stuff again, can you please ping me? [02:42] wallyworld_: will do [02:43] ta, we need to tweak the implementation === hatch__ is now known as hatch [03:46] anyone interested in reading over my developer doc on writing tests? https://codereview.appspot.com/96110043/ [03:56] thumper: this document looks very useful... thanks [03:57] np [04:40] thumper: review just submitted. very useful! [04:49] menn0: ta [05:20] thumper, morning/evening [05:20] fwereade: just being called away.. bbs [05:20] thumper, ok :) [05:30] fwereade: back [05:30] kitchen/cooking emergency [05:30] thumper, no worries [05:30] thumper, thanks for the test docs, just writing some comments [05:30] fwereade: did you want a quick hangout? [05:31] fwereade: I'm just adding a section on checkers, and will refer to the other checkers package [05:31] thumper, cool; and yes I do, in say 5-10 mins? [05:32] sure [05:55] fwereade: morning, I'd like to run something by you that I discovered while doing the type introspection. [05:56] jam1, just in with thumper at the moment, will let you know when I'm out [05:56] fwereade: yeah, no prob [06:01] if anyone has 1.18 locally, can you please try running BootstrapSuite.TestTest in cmd/juju? [06:04] axw: you know you can always switch to 1.18, right? [06:05] jam1: I possibly don't [06:05] :) [06:05] axw: you want the tip of the 1.18 branch? [06:06] jam1: sorry, maybe I misunderstood. I know I can clone lp:juju-core/1.18 [06:06] is there a way to switch to 1.18 within the same repo? [06:06] I guess just switch... it's just a branch isn't it [06:06] axw: I'm not sure how your files on disk are layed out, but if you are using a lightweight checkout or colocated branches, you can. [06:06] axw: but also the 1.18 branch is merged into trunk [06:06] so you can "bzr update -r juju-1.18.2" [06:07] jam1: cool, thanks [06:07] axw: you probably need to run "godeps -u dependencies.tsv" because of differences [06:07] jam1: I was actually interested in someone reproducing an error I was getting on the 1.18 branch, but this is useful info anyway [06:07] I think I've tracked down the error in the mean tim [06:07] time* [06:07] axw: hmm.. tags are missing, I'll get them put in, just a sec [06:20] looks like people have been backporting and not keeping trunk up to date with 1.18, I'll spend some cycles to bring them in sync again. [06:21] jam1, https://plus.google.com/hangouts/_/gux2ehqneizflyei3np3mfnyzya?hl=en [06:42] fwereade: so if I *just* do my one line change to unwrap interfaces, there is an error, because it builds up methods using the reflect method Index, and the concrete type's method indexes don't match the interfaces interfaces (for obvious reasons). but I think with the other changes, it will build up that method mapping later so it won't be a problem [06:43] investigation still neede [06:47] critical bug fix up for review, if someone has a moment: https://codereview.appspot.com/92140043 [07:04] axw: i'd look but i'm off to soccer. i'll look when i get back if not reviewd by then. bigger than i thought it would be [07:05] wallyworld: yeah, unfortunately touches lots of little bits [07:05] thanks [07:05] wallyworld: I will take a look at the lxc stuff now [07:08] axw: ok. i talked to tim today and he mentioned that you don't *need* the btrfs stuff. a straight file copy of the template will do. i think the current code may do that if clone is enabled but btrfs is not used but i haven't had a chance to check. so we may be able to (for now) leave out the btrfs stuff [07:09] wallyworld: ok. I will test out the bare minimum, and make any changes required for that. then we can see what else is needed [07:09] yep ok. thanks very much. i think the main thing missing is with add-machine. in which case, i think not using btrfs is the right way to go [07:10] * wallyworld -> soccer now [07:10] fwereade: confirmed. with the rest of my patch there isn't a failing test that we end up "exposing" more than we intended... [07:10] so I guess that means I'm allowed to do whatever I want, right? [07:44] jam1, oops, missed you: yes, sgtm [07:45] fwereade: your embedding of an interface into a concrete type did work as expected (reflect.Value(value.Interface()) doesn't give you the fully underlying type) [07:45] jam1, cool, the universe works how my brain thought it did, this is reassuring ;) [07:48] axw, commnted on https://codereview.appspot.com/92140043/ [07:49] ta [08:04] oo, ferry is moving [08:05] fgz: ferry gz ? that's new [08:06] or france gz... in like 5 hours [08:06] fgz: premature naming, if you ask me... :) === fgz_ is now known as fgz [08:19] fwereade: we have CodeNotImplemented for when you try to call an API that doesn't exist (Client.Foo is not available) [08:19] fwereade: should I reuse that code for when Client(2) doesn't exist at all? [08:21] I guess today everything returns "params.ErrBadId" [08:21] jam1: what happens currently if you try to... [08:21] fgz: ^^ [08:21] fgz: only because each implementation does it directly [08:21] I guess you can just accept both that and a new more apt code if desired [08:21] it isn't part of the RPC spec, whereas CodeNotImplemented is used [08:22] fgz: well on the APIServer itself it has a mapping that ErrBadId ==> params.CodeNotFound [08:24] jam1, I don't think I have a coherent position there. I can't see a real distinction between BadId and NotImplemented in practice, but I would prefer not to wantonly change behaviour as we progress [08:24] fgz: so today you probably actually get CodeNotFound if you pass an unknown version, and you get CodeNotImplemented if you pass a valid version but to a Method that doesn't exist [08:24] fwereade: digging deeper shows it to be CodeNotFound [08:24] jam1, however I think the latter is weaker than the former because AFAIK nobody is depending on BadId [08:25] fwereade: I believe client side we translate both into NotFound, checking [08:25] jam1, NotFound does not strike me as coherent, but I suppose that doesn't help [08:25] fwereade: well, we didn't find the Facade you asked for :) [08:26] fwereade: I'm not sure about top level missing the entire facade either [08:26] jam1, yeah, but the broader the domain of NotFound the less meaningful it is [08:26] fwereade: you get CallNotImplementedError if the Facade isn't there [08:26] jam1, time was you could depend on NotFound meaning that some entity referenced by your request -- implicitly or explicitly -- was missing [08:26] same as if the Method is not there [08:27] fwereade: right [08:27] CodeNotFound ==> data in the DB isn't there [08:27] jam1, if we can make some baby steps back in that direction I would be happy [08:27] CodeNotImplemented => API isn't there [08:27] jam1, +1 [08:27] fwereade: should we just continue to enforce CodeNotImplemented for missing Id, or should we have an CodeAPIVersionNotImplemented [08:28] I think I prefer just using CodeNotImplemented [08:28] with enough context you have a chance to figure it out [08:28] jam1, I think not -- "not implemented in this version" does not imply "implemented elsewhere" [08:28] jam1, well, it kinda *does*, but that doesn't map onto reality [08:28] jam1, stick with CodeNotImplemented [08:29] * fwereade bbs [08:29] fwereade: sgtm. Unfortunately the actual error we get across the pipe isn't typed, just a message string and a Code [08:29] but I can live with that. [08:29] and for Compat purposes, ErrBadId => CodeNotImplemented [08:29] we've never needed to deal with it before [08:30] but we can put a compatibility shim in the client side for mapping {Message: ErrBadId, Code: CodeNotFound} => CodeNotImplemented if we need to (I think) [08:31] sounds reasonable to me [08:42] jam1, +1 [08:46] fwereade: so... can we break the API and start returning CodeNotImplemented for all ErrBadId 's ? [08:46] Nothing (today) should be sending anything but "" for Id [08:46] as all currently exposed APIs return ErrBadId in that case [08:47] I'm reasonably ok with saying that until now passing Id != "" was undefined behavior, so we get to define it. [08:47] wallyworld: I have created a new branch that picks out just the fast-lxc-clone bit from yours, but I've done it a little differently. I've created a new provisioner API, ContainerManagerConfig, which returns a map[string]string that configures the container manager === vladk|offline is now known as vladk [08:48] wallyworld: also I called it use-lxc-clone (maybe change to lxc-use-clone), otherwise people will just see "fast" and stop thinking [08:48] But it is true that when we update clients with versioning they're going to sometimes get ErrBadId, CodeNotFound (old server) and sometimes CodeNotImplementedError, CodeNotImplemented, but I think that is ok [08:48] wallyworld: seems to be working in MAAS, but I need to do a bit more work to handle upgrades [09:14] jam1, sorry, I keep not seeing notifications, that's the trouble with trying to write, 80% of my tie is spent staring into the middle distance [09:14] fwereade: np [09:14] I do have another Q for you as well. [09:14] jam1, I think we just have to put up with the variation for now, and just be consistent as we progress [09:14] If you have a function that returns an interface you end up with a reflect.Type that describes just that interface. [09:15] I'm trying to figure out if I can come up with an object [09:15] such that reflect.TypeOf will return that same Type [09:15] fwereade: what I mean is, we *could* continue to return ErrBadId [09:16] fwereade: since that is maintaining the existing API [09:16] but I think that is actually a bad API and nobody depends on it, so we can sanely change it out for returning CallNotImplementedError [09:16] and teach the client code to accept either one [09:16] but have the server code start returning the better values [09:16] jam1, +1 to that, but I'm still furrowing my brow re the reflect stuff [09:17] fwereade: did what I say make sense and you don't know how to do it? [09:17] The way the existing code got the reflect.Type was doing: [09:17] reflect.TypeOf((*InterfaceMethods)(nil)).Elem() [09:17] jam1, I *think* so, yes [09:17] I was hoping I could get rid of all the reflect.TypeOf calls and put them in one place [09:18] but var interfaceValue InterfaceMethods = &SimpleMethods{}; reflect.TypeOf(interfaceValue) == SimpleMethonds [09:18] jam1, yeah, I'm not sure how to do that [09:19] fwereade: fwiw I never figured it out either [09:19] I was originally trying to do something like it for "interface{}" [09:19] but my attempts there just led me to get "nil" [09:40] fwereade: https://codereview.appspot.com/98090043 when you get the chance [09:43] jam1, fwereade: what's the current story about gcc-go? are we planning to use that for all our builds, or just for power, since it's required there? I wasn't sure where that discussion landed.. [09:43] natefinch: for trusty we went with golang-go where we can and gcc-go where we can't [09:43] natefinch, while the performance differential is as significant as it is, best toolchain for the platform [09:43] I'm not sure where the status quo ends up [09:43] certainly distro is pushing for gcc-go everywhere [09:44] natefinch, ideally I'd have a single toolchain across the board, but I'm not interested in making things suckier than they have to be as a point of principle [09:44] fwereade: I hear you. In an ideal world, gcc-go would be good enough for everyone, but yeah... it's not. [09:57] axw: that's great, thanks. for the demos, I don't think we need to worry about upgrades as such [09:58] wallyworld_: yeah I've just added some code to ignore API-not-implemented errors [09:58] just about to push [09:58] ok, i'll take a look and let the maas folks know [09:59] good morning everyone [10:01] hey perrito666 [10:03] wallyworld_: https://codereview.appspot.com/96140044 [10:03] great thanks, looking [10:07] voidspace, menn0: meeting? [10:08] wwitzel3: ping [10:08] axw: so you tested with add-machine of a kvm container and then deploy to lxc inside that? [10:09] perrito666: pong [10:10] I privmsgd you [10:10] wallyworld_: I created a vmaas, and add-machine'd in one of the machines [10:10] cool [10:11] wallyworld_: the top-level machines are on kvm, but not using the kvm provisioner [10:11] yep [10:11] i'll have a comment or two about adding a doc comment to exported methods, but if it works it's good enough for the maas folks to try from source [10:22] gah [10:22] coming [10:29] jam1: "Be careful that you use different queue file names for the second action, else you will mess up your system." [10:29] jam1: I was not doing that ^ [10:34] natefinch: so are you intending to continue attending the UTC 10 meeting next week ? Or are you going to split off ? [10:35] perrito666: join the moonstone hangout re: replicaset stuff [10:35] jam1: is that the daily? I was going to split off the daily one. [10:35] yes [10:37] I wonder if for cross pollination, it would work to send one person each day to another team's standup. [10:39] so like monday, someone from my team would go to the onyx standup, tuesday, someone else would go to the sapphire standup, etc [10:42] what is a good way to generate some logging on a pre-existing mysql node? .. [10:42] wwitzel3: oops sorry missed the message, going [10:48] wwitzel3: you want Juju logging, or mysql loggign? [10:48] because you can "restart jujud-unit-2" or whatever the agent name is [10:49] jam1: I want to test this rsyslog change before putting it in to the code, so I want to trigger some unit mysql logging. [10:49] wwitzel3: right, I think if you just bounce the running agent, you'll get "I restarted" logging [10:50] wwitzel3: or you can do "juju set" to change some conf item [10:50] jam1: thanks [10:54] hi alexlist`, really sorry to hear about what happened. I hope you're doing well. [10:55] jam1: that worked, thanks, and it seems to be reliably logging to all the state machines now. [10:55] wwitzel3: yay [10:55] rsyslog is *magic* :) [10:55] "..else you will mess up your system" [11:01] wwitzel3: did you do a "standard" maas install - or us virtual maas? (or both?) === perrito6` is now known as perrito666 [11:06] jam1: natefinch: both you guys have in progress bugs against 1.18.3. are you able to get them completed so that we can merge the fix for the critical P1 issue when done and get a 1.18.3 release out? [11:07] wallyworld_: I think mine is the bug you actually fixed [11:07] https://bugs.launchpad.net/juju-core/+bug/1306537 [11:07] <_mup_> Bug #1306537: LXC local provider fails to provision precise instances from a trusty host

[11:07] voidspace: I just used a standard trusty image and choose the maas option, for my provider [11:07] has your branch associated with it [11:08] "bootstrap supported series" [11:08] wallyworld_: is it just that you intended to backport to 1.18 but didn't get it? [11:08] ah could be [11:08] wallyworld_: I'll reassign it to you [11:08] sure thanks. i should have looked close than just the miletone page [11:09] wallyworld_: mine was released in 1.19.1, possibly just needs backporting [11:09] ok [11:09] i'll backport tomorrow [11:15] voidspace: also my latest push to 008-ha-rsyslog has node logging to all state machines working [11:18] wwitzel3: cool, thanks [11:19] https://codereview.appspot.com/100270043/ .. his is for the nodes logging to any state servers that exist when the node is created. [11:19] tested in non-HA as well [11:20] now I'm going to look at refactoring the watcher for rsyslog for re-writing the config file when state servers are added/removed. [11:21] great [11:21] jam1: I have your notes from yesterday about using an API call to get the environ config bits needed, instead of using it directly [11:24] wwitzel3: do you need more detail about it? [11:26] jam1: don't think so, though I'm just starting to dig in [11:26] wwitzel3: sounds good [11:28] wwitzel3: so you specify a queue filename per state server [11:28] wwitzel3: do you have an eta on your vmaas setup notes? [11:28] wwitzel3: interesting [11:28] voidspace: yeah, you have to, else you will mess up your system. [11:28] wwitzel3: ah, that was the discussion earlier [11:29] lovely... [11:29] wallyworld_: I will flesh them out today [11:30] \o/ thanks. but of course don't put them ahead of real work :-) [11:32] wallyworld_: I've been meaning to do it, I've just got a pile of notes tossed in to a doc now, so I'll turn them in to something real [11:32] great, looking forward to it [11:38] axw, where was that destroy env logic in sync tools.. it was source diving for it through the sync code yesterday... and it was really non obvious where it was. [11:39] s/it/i was source [11:54] axw, hazmat`: environFromName [11:54] has a cleanup func it returns [11:54] which is supposed to only delete it if the env can't be looked up, but there was a bug WRT EnvName being an empty string [11:55] so one function couldn't find it, but the next function could [11:57] wwitzel3, voidspace: reviewed https://codereview.appspot.com/100270043/ [11:58] jam1, you have a bunch of LGTMs [12:03] fwereade: did you see rog's comment on that CL ? [12:05] wwitzel3: do you have a link to the docs warning about the ActionQueueFileName stuff? [12:07] jam1, sorry, which? [12:13] fwereade: on https://code.launchpad.net/~jameinel/juju-core/reflect-only-facades/+merge/218764 [12:14] comment: https://code.launchpad.net/~jameinel/juju-core/reflect-only-facades/+merge/218764/comments/521422 [12:14] I did reply that I intend to do runtime introspection, instead of static introspection [12:15] fwereade: also, that specific branch can't land just yet because we *do* static introspection of srvRoot in the state/apiserver code to make sure types are registered with no Discarded Methods. [12:16] so I was going to push forward on getting a registry so that we can do the lookups [12:17] fwereade: also, should I push forward on making "Facade" a name that gets used in code and errors strings, etc. Right now the RPC package thinks in terms of: "unknown object type %q" when you ask for something that isn't accessible. [12:21] what's the cause of "WARNING failed to find 1.19.2 tools, will attempt to use 1.19.1"? [12:21] I had this during the sprint, but I can't remember how I solved it [12:21] voidspace: "bzr bootstrap --upload-tools" as 1.19.2 isn't actually released yet IIRC [12:22] jam1: it's simpler than that [12:22] jam1: I didn't bootstrap with --upload-tools [12:22] jam1: thanks :-) [12:23] and I need to - the whole point of this exercise is to test something in the dev version... === vladk is now known as vladk|offline [12:41] * voidspace lunches === vladk|offline is now known as vladk === vladk is now known as vladk|offline === vladk|offline is now known as vladk === vladk is now known as vladk|offline === rvba` is now known as rvba [14:00] fwereade: not sure I grok your comment; I have added lxc-use-clone to the immutable config attributes list [14:00] fwereade: if/when we want to make it dynamic, I will take it out and add code to react to changes [14:00] but for now it's set-at-bootstrap and that's it [14:00] axw, then I must just be an idiot and have misread where it was [14:00] axw, sorry [14:01] fwereade: nps [14:02] fwereade: also, I totally agree regarding inversion of EnvCommandBase [14:02] fwereade: so any thoughts on runtime self-description vs static self-description? I still feel that having the name separate from the version is better for us, though arguably the final type we are going to be returning is going to have the type in its name. [14:02] sorry, have the version in its name [14:02] (ClientAPIv2 or somesuch) === Ursinha is now known as Ursinha-afk [14:05] jam1: separate seems like it's almost universally better than combined. Combining things is trivial, splitting them is usually more difficult. === alexlist` is now known as alexlist [14:07] jam1, yeah, I just saw that comment in mail -- I do not consider munging the version number into the method name to be "reasonable", and I'm pretty sure we can do all the API description generation dynamically just fine [14:14] natefinch: isn't that what regex is for :) [14:15] jam: now you're just yanking my chain :) [14:19] natefinch: well, I prefer the split form, and since 3 out of 4 devs polled agree, I'll push forward with it [14:34] jam, dammit, I borked my voip and forgot about it -- can you make the cross-team while I rush to figure out how to fix it up again? [14:35] axw: wallyworld_ any of you guys around now? [14:35] or natefinch, since jam's not meant to be working now [14:35] fwereade: I can do that [14:35] fwereade: can you give me a link? [14:41] natefi [14:41] natefinch: it is the canonical Conference call [14:41] since there are like 30 people [14:41] jam: fwereade gave me the info [14:42] jam: it was still in "waiting for the leader" mode, so we figured it wasn't happening [14:51] sometimes I really wish we were better about being more specific with our wording. [14:52] fwereade: I have a task: Agent conf needs to store all addresses (hostports), not just private addresses. I presume this means API hostports? [14:52] heh, the only thing the rsyslog worker grabs from the EnvironConfig is the SyslogPort [14:52] natefinch, yeah -- it's to prevent the suicide-in-manual-environments problem [14:52] oh and the CAcert [14:53] natefinch, although, if state hostports are kept there too, fix them as well [14:53] fwereade: fair enough, will do [14:53] natefinch, otherwise we *will* get complaints about suiciding manual HA environments === Ursinha-afk is now known as Ursinha [14:54] jam: so were you saying that in the actual SetUp of the rsyslog worker, it shouldn't use WatchForEnvironConfigChanges? Or just in Handle, it should no longer use EnvironConfig()? Or both? [14:54] wwitzel3: both [14:54] jcastro: your issue with deployed bundles not being placed correctly has now been fixed an deployed. enjoy. [14:54] It should have an API for Watching the actual RSyslog changes (which might be backed by the EnvironConfig watcher) [14:54] you mean placed as in the boxes? [14:55] but the Agent side should have a proper "this is what I care about" API [14:55] jam: I see, got it. Thank you. [14:57] wwitzel3: as an example you can look at WatchAPIVersion on the Upgrader interface [14:57] the API is WatchAPIVersion(), but the internal watcher is actually WatchForEnvironConfigChanges. [14:58] that gives us the ability to say "we're generating too many notifications when API hasn't actually changed" and fix it in the future [14:58] rather than breaking the contract [14:58] jam: yep, looking at it now. Then we can later tune it [14:58] right [14:58] wwitzel3, and in general *only* state servers should get to see environ config [14:58] wwitzel3, it's (usually) got cloud account keys in it [14:58] fwereade: seems sane :) [14:59] wwitzel3: fwereade: right. Also in the case of rsyslog you actually care about 2 things, because you want the config for the ports, etc, but you also want the list of addresses. [14:59] wwitzel3, we can't rationally stop the state servers from seeing them, but we shouldn't let anyone else ;) [14:59] And I *think* we could sanely hide that with 2 watchers inside the APIServer and merge any notification events to 1 watcher in the client [14:59] jam, sounds sane to me [15:00] jam: sounds good [15:00] fwereade, jam: thanks [15:00] fwereade: fwiw we explicitly blank them for any agent that doesn't have JobManageEnviron, but it is still too much a "giant back of unrelated data" when we want the API to be around just what we actually want for rsyslog [15:01] jam, yeah, but the blanking sucks imo [15:01] fwereade: I agree, just that it isn't an *actual* security hole ATM [15:01] jam, and it forces secrets to be strings, and it doesn't stop people from hacking up not-quite-working Environ objects, and, eww [15:02] jam, true [15:02] * fwereade needs to visit shops, bbiab [15:03] is there anything in place to update mongo machines entry for machine 0 if machine-0 ip changes? [15:05] perrito666: I believe so, yes. I would have to go back through it, but I think that's the peer grouper's responsibility [15:07] are there feature toggles/flags in juju-core? [15:07] jcw4_: not really [15:07] natefinch: mm peergrouper is the one making my life hard, apparently it sets my replicaset to old ip values instead of the new ones, I believe it is getting wrong info from machines bc the machine-0 entry has old address data, so I wanted to update that so peergrouper stops messing with me [15:08] jcw4_: what specifically are you thinking of? [15:08] natefinch: tx. Just trying to think of the best way to combine regular small commits with introducing a whole new feature (i.e. actions) [15:09] jcw4_: the easiest way to do it is to simply not expose the CLI command until it's ready. That's our usual practice. "in but not on" [15:09] natefinch: bodie_ and I have made a number of changes but we're not anywhere close to a full feature. [15:10] natefinch: yeah, maybe we'll just have to do an MR with what we have soon... no cli, but state server changes [15:13] jcw4_: generally that is ok, as long as a compatible client won't corrupt things [15:13] jcw4_: there is stuff like environ config which can be used for feature flags [15:13] jam: interesting. [15:13] but there isn't something like "go build -WITH_SUPPORT_FOR_FOO" [15:14] jam: rick_h_ gave bodie_ and I a bit of a demo of their process, which includes feature flags. [15:14] jcw4_: It sort of depends how invasive the changes are, if you are in your own collections it should be fine, if you have to change existing docs then we have to be careful with how we'll end up doing an upgrade from previous stable to new code. [15:15] yeah, we have one collection (actions) that's new, but we're also adding actionids to units... possibly an issue [15:15] jcw4_: sure, but they aren't quite as concerned about multiple client versions making requests. or at least they are but at a different level (FF vs IE vs Safari, etc) [15:15] jam: good point [15:15] jcw4_: the main problem there is that we need an upgrade path [15:16] so if you start with a 1.18 environment, we need a way to get to the new code with action ids [15:16] natefinch: is currently tasked with sorting out db schema migrations [15:16] during upgrade [15:16] because we have most clients behind the API [15:16] except the API servers themselves [15:16] and HA complicates a migration procedure a bit [15:16] jam: backward compatibilty and upgrade path seem a little tricky. Are there any tests or scenarios that are commonly used to test upgrades? [15:17] jcw4_: CI does upgrade testing, bootstraps with 1.18 and then does an upgrade to dev tip [15:17] jam: I see, and that is just when a change gets landed right? [15:17] jcw4_: it isn't something we do well internally yet. [15:17] no CI before landing in trunk? [15:17] (a bit obvious I guess) [15:17] jcw4_: we run the unit test suite before landing trunk (go test), but the CI suite is run async on lots of platforms and clouds [15:18] jam: I see. [15:18] jcw4_: so we won't do an official release unless CI says everything is green [15:18] but we might break upgrade/etc for a short while. [15:19] jcw4_: for a schema change, ideally we'd have the code written such that it can handle either schema [15:19] and then once we know all the agents are upgraded to know how to handle either schema [15:19] we can update the DB to the new schema [15:20] I *think* the current migration is to actually take the system offline, upgrade the agents, see that they are upgraded, then stop the db, copy the data to a safe backup, and then rewrite in place. [15:20] but natefinch and wallyworld's teams are the ones task with actually doing that work [15:20] jam: so forgiving if collections or keys are missing and not changing by removing stuff? Interesting about migration path [15:21] jcw4_: it is a common way to leap-frog update your API servers and DB backend, though we might not need to go through quite as much effort [15:21]