[01:30] <axw> wallyworld, waigani: standup
[03:40] <davecheney> https://bugs.launchpad.net/juju-core/+bug/1281394
[03:40] <_mup_> Bug #1281394: uniter failed to run non-existant config-changed hook <juju-core:New> <https://launchpad.net/bugs/1281394>
[03:42] <davecheney> today's WTF
[04:10] <axw> davecheney: is that plain old juju built with gc, or is it built with gccgo?
[04:15] <davecheney> bog standard golang-go
[04:15] <davecheney> axw: not really supposed to log public bugs about the other thing
[04:15] <davecheney> yet
[04:15] <axw> yah ok
[04:15] <axw> hrm
[04:20] <axw> weird, I dunno how this ever worked. seems to be looking for the complete wrong error type. I'll dig more later
[04:20] <davecheney> axw: if you point me in the right direction
[04:20] <davecheney> i can fix this one muself
[04:20] <davecheney> it doens't sound that hard
[04:20] <axw> davecheney: worker/uniter/context.go, runCharmHook
[04:21] <axw> at the end of the function it's asserting the error is of type *os/exec.Error
[04:21] <axw> AFAICT, that's only ever returned by os/exec.LookPath
[04:22] <davecheney> that is stupid
[04:24] <davecheney> axw: ummm ...
[04:25] <davecheney> 2014-02-18 04:11:37 INFO juju.state.apiserver apiserver.go:135 [F] user-admin API connection terminated after 18m27.150333066s
[04:25] <davecheney> 2014-02-18 04:11:42 INFO juju.state.apiserver apiserver.go:131 [10] API connection from 124.149.62.81:43651
[04:25] <davecheney> 2014-02-18 04:13:51 INFO juju.state.apiserver apiserver.go:135 [10] user-admin API connection terminated after 2m8.222108526s
[04:25] <davecheney> 2014-02-18 04:18:30 INFO juju.state.apiserver apiserver.go:131 [11] API connection from 124.149.62.81:43665
[04:25] <davecheney> 2014-02-18 04:20:38 INFO juju.state.apiserver apiserver.go:135 [11] user-admin API connection terminated after 2m8.067538006s
[04:25] <davecheney> 2014-02-18 04:21:34 INFO juju.state.apiserver apiserver.go:131 [12] API connection from 124.149.62.81:43671
[04:25] <davecheney> 2014-02-18 04:21:41 INFO juju.state.apiserver apiserver.go:135 [12] user-admin API connection terminated after 6.261804724s
[04:25] <davecheney> 2014-02-18 04:21:43 INFO juju.state.apiserver apiserver.go:131 [13] API connection from 124.149.62.81:43673
[04:25] <davecheney> two things
[04:25] <davecheney> and more importantly
[04:25] <davecheney> why did someone decide to use %x there
[04:25] <davecheney> ooooooooooooooooooooooooooooooh
[04:25] <davecheney> i know what they are
[04:25] <davecheney> those are juju ssh connections
[04:25] <davecheney> why is it leaving the connection open while ssh is running
[04:25] <davecheney> i thought that was fixed
[04:26]  * axw shrugs
[07:45] <hazmat> that config-changed hook error is odd
[08:02] <rogpeppe> axw: yes, that import sorting change was made mechanically, but the program was just a quick hack - it's not really suitable for using in production
[08:02] <axw> rogpeppe: ok
[08:02] <rogpeppe> axw: in particular, it doesn't cope with comments well (the bugbear of go reformatting)
[08:02] <axw> ah yes
[08:02] <rogpeppe> axw: it doesn't actually use go/ast - it cheats
[08:03] <rogpeppe> axw: did you see my question on golang-nuts, BTW?
[08:03] <axw> well, we don't necessarily have to have something that makes source changes automated - just something to check they're wlel formatted
[08:03] <axw> rogpeppe: I did not
[08:03]  * axw looks
[08:04] <rogpeppe> axw: subject was "gofix and comment consistency"
[08:04] <axw> yup, reading now
[08:05] <axw> I had this issue with gocov, just gotta dredge the old source code to see if I actually solved it :)
[08:05] <rogpeppe> axw: BTW in case you might find it useful, the sortimports code is this: http://paste.ubuntu.com/6953264/
[08:06] <axw> thanks :)
[08:07] <axw> rogpeppe: https://github.com/axw/gocov/blob/1f9133a687283076e46f1fb56ad58a76bbd7392b/gocov/instrument.go#L273
[08:07] <rogpeppe> axw: ah... i'll try that
[08:07] <rogpeppe> axw: thanks!
[08:07] <axw> but.. you said you're ot using go/ast?
[08:07] <axw> not*
[08:08] <rogpeppe> axw: not for sortimports, but i am for the errgo changes - i did a gofix module for that
[08:08] <axw> oh in this case
[08:08] <axw> right
[08:22] <rogpeppe> axw: unfortunately that solution (clearing out the File comments) doesn't seem to work for me - it ends up deleting the comment
[08:22] <axw> hrm
[08:23] <rogpeppe> axw: the weird thing is that they must have encountered this issue in previous gofix modules
[08:23] <rogpeppe> axw: i suppose it's just possible that there was no previous gofix that made the source code longer
[08:24] <axw> entirely possible... not really sure why the comments would disappear tho
[08:50] <TheMue> fwereade: Thx for your response to rogpeppe. The number of /var/log/juju in trunk is indeed high, the handling somewhat curious. Should be refactored, but in a different CL.
[08:51] <fwereade> TheMue, I think that'd need to be a prereq for this one, thoug
[08:52] <TheMue> fwereade: Could be, yes.
[08:54] <fwereade> TheMue, rogpeppe: thoughts re direct regexp filtering?
[08:54] <rogpeppe> fwereade: as far as apps knowing our log format:  we could just pick a format and stick to it - a prefix-based format should be reasonably future-proof. in a way, you can think of the format as the interface definition.
[08:55] <rogpeppe> fwereade: i had thought that it would be nice if command-line clients didn't have to use regexps though - we could potentially map from tag-set to regexp on the client side
[08:56] <fwereade> rogpeppe, it feels to me like tags are the language of the API, and if we want to filter by logging-entity we should darn well name it in the usual way over the API
[08:57] <fwereade> rogpeppe, regexps feel useful as well, but for different reasons
[08:57] <TheMue> rogpeppe: why the difference here? people would otherwise use it with a pipe to grep
[08:58] <rogpeppe> fwereade: it seems to me that people are going to become dependent on the exact format of the log output whether we have explicit tag-level filtering or not
[08:58] <fwereade> rogpeppe, dependingon the content of individual messages feels qualitatively different from depending on the format of all messages
[08:59] <TheMue> rogpeppe, fwereade: remembering my own admin time I always took first an unfiltered look into the logs and then drilled down by the optimal greps
[08:59] <rogpeppe> fwereade: i'm not sure i get that
[09:00] <TheMue> rogpeppe, fwereade: because it isn't always one machine, one unit. errors can have different reasons, on a higher level
[09:00] <TheMue> rogpeppe, fwereade: and I need easy ways to aggregate them
[09:00] <rogpeppe> fwereade: the thing is that even if they filter by a few explicit units, the log output still needs to identify the message from each individual unit
[09:01] <rogpeppe> fwereade: so the client-side code may well still need to know which line pertains to which unit in the filter
[09:02] <fwereade> rogpeppe, that's easily solved by having a connection per entity ;p
[09:02] <rogpeppe> fwereade: that's not a good idea
[09:03] <rogpeppe> fwereade: it doesn't scale well
[09:03] <TheMue> rogpeppe, fwereade: using tags how do I get the log entries of units 5, 7 and 9 of one service in one run to see the timeline and there how errors correlate, e.g. stablishing relations?
[09:03] <fwereade> TheMue, how would you get that today?
[09:04] <fwereade> TheMue, it seems very optimistic to assume you'll see perfect ordering once it's been through rsyslog
[09:04] <TheMue> fwereade: passing it as regex so that I've got a server-side filtering and only those are delivered (regardless if cli or api)
[09:04] <rogpeppe> fwereade: BTW there's nothing to say that this interface *is* dependent on the exact log file format - we can always transform the text before filtering and delivering to the client
[09:04] <TheMue> fwereade: not a perfect one, but a nearer one than doing three individual calls
[09:05] <fwereade> TheMue, I don't think that's a logically sound argument
[09:05] <fwereade> TheMue, if it might be wrong, it might be wrong
[09:05] <fwereade> TheMue, you either depend on it or you don't, and you can't with either aproach
[09:05] <TheMue> fwereade: shure it might be wrong, but it's the practical approach by admins
[09:05] <TheMue> fwereade: you wonna create a data warehouse
[09:06] <TheMue> fwereade: in this case we have to create an entity with dedicated fields in the database
[09:06] <fwereade> TheMue, kind of, yes -- at the very least I'd like this API to have a chance of continuing to work if and when we do use grown-up logging
[09:06] <TheMue> fwereade: and an atomic tme sync on all nodes
[09:06] <rogpeppe> fwereade: i think there's a lot to be said for providing some concrete definition of the log format here, because people *will* see it
[09:06] <TheMue> fwereade: and a good dwh api
[09:07] <rogpeppe> fwereade: this is perhaps a good moment to do that.
[09:07] <TheMue> fwereade: but do we really want this? is this a requirement that currently exists or is it wishful thinking? who are the stakeholders for it and how do they work?
[09:08] <fwereade> TheMue, I am one of the stakeholders because having juju be all amateur and flaky makes me sad
[09:08] <TheMue> fwereade: hehe, nice argument
[09:08] <rogpeppe> fwereade: if we move to moving JSON-based (or other "richer format") logging at some point, we can still provide the current interface, i think.
[09:08] <fwereade> TheMue, rogpeppe: my judgment was, and remains, that we don't have enough time or enough need *in this cycle* to do a proper logging database
[09:08] <TheMue> fwereade: sure it would be more powerful than flat files
[09:09] <TheMue> fwereade: but then you talk about a feature which isn't really cheap
[09:09] <fwereade> TheMue, rogpeppe: but I still think it's important to provide an API at a level of abstraction that maps to the entities we're actualy playing with
[09:09] <fwereade> TheMue, which is why we're not doing it this sysle
[09:09] <fwereade> er s/sysle/cycle/
[09:10] <TheMue> fwereade: to define such a kind of api ask yourself what you really want to know from your environment
[09:10] <TheMue> fwereade: make a list of all questions
[09:10] <TheMue> fwereade: those define the constraints of the solution
[09:11] <fwereade> TheMue, I know at the very least that I will sometimes want to see all the log output of a single tagged entity
[09:11] <TheMue> fwereade: and you're right, to have a good solution it quickly gets larger than possible in this cycle
[09:11] <rogpeppe> fwereade: i get that you're concerned that the level that we're specifying filtering doesn't map to some of the entities that we might want to filter on, but the problem is that the filtered output is also part of the API
[09:11] <fwereade> TheMue, which is why I'm pushing back on doing everything via regexp
[09:11] <TheMue> fwereade: how does using a regexp hindering you to get exactly that output?
[09:11] <fwereade> rogpeppe, yeah, I rather liked the idea of us transforming the output so we can impose a format on the way out
[09:12] <fwereade> TheMue, rogpeppe: it means that once a client is using a prefix regexp we can't (say) move the uniter into the machine agent without a load of extra hassle
[09:12] <fwereade> TheMue, rogpeppe: there'll be hassle anyay
[09:13] <fwereade> TheMue, rogpeppe: but that will make it worse, I think
[09:13] <rogpeppe> fwereade: how will it make moving the uniter into the machine agent harder?
[09:14] <TheMue> fwereade: if you talk about a "client", about what *exactly* do you talk? men, machines?
[09:14] <fwereade> TheMue, rogpeppe: if we specify *what output we're interested in* rather than *certain coincidental features of that output* we get to change things freely internally and just need to fix them at the boundaries
[09:14] <fwereade> TheMue, rogpeppe: both
[09:14] <rogpeppe> fwereade: ah, each log entry is prefixed with the file it came from?
[09:15] <fwereade> rogpeppe, yeah -- and while I think we will need to hack that about a bit regardless, having external clients seeing those details directly will just further constrain us
[09:15] <TheMue> fwereade: as I said, admins are used to drill down. more problematic are hard coded machines to me. where are they and what exactly do they expect from the log output?
[09:15] <fwereade> TheMue, you're writing one, aren't you? the CLI client...
[09:15] <TheMue> fwereade: here our log output is more than often enough not machine readable
[09:16] <TheMue> fwereade: the CLI is used by the admin who is drilling down
[09:16] <TheMue> fwereade: he is used to do so, with all current log formats
[09:16] <TheMue> fwereade: that's part of her job since years
[09:16] <fwereade> TheMue, the admin who has to remember what version of juju is running on the server side to get the right messages? that seems unlikely to lead to general rejoicing
[09:17] <TheMue> fwereade: to support error analysis and monitoring by machines we need an absolutely different way of logging informations
[09:17] <fwereade> TheMue, in case it's not clear I am 100% opposed to making people type `filter=^machine-1` on the CLI
[09:18] <TheMue> fwereade: tal /var/log/messages  | grep "something"
[09:18] <fwereade> TheMue, I would much rather see it in juju language: `1 2 3/lxc/2 wordpress/7 mysql/3`
[09:18] <TheMue> fwereade: but I'm interested in ... | grep ' ERROR '
[09:19] <fwereade> TheMue, yeah, that's a completely different conversation
[09:19] <rogpeppe> fwereade: so if they do that and then see the log output, how do they know which log line corresponds to which entity?
[09:19] <TheMue> fwereade: or ... | grep failed
[09:19] <fwereade> TheMue, you can and should be able to filter by regexp
[09:19] <fwereade> TheMue, but regexp filtering is not appropriate for determining message source
[09:19] <TheMue> fwereade: sorry, different opinion by own experience here
[09:20] <TheMue> fwereade: often enough I start by "I don't know anything, so take a look."
[09:20] <TheMue> fwereade: and then, by getting hints in larger logs, I take deeper looks
[09:20] <rogpeppe> fwereade: for example, I have suggested to the GUI team that if they're monitoring the log output from several units (perhaps hundreds) that they should use only a single log stream, and then demultiplex it at the client side
[09:20] <TheMue> fwereade: and those are not only single machines or units
[09:20] <fwereade> TheMue, no objections there
[09:21] <TheMue> rogpeppe: good idea too, didn't thought of it so far
[09:21] <rogpeppe> fwereade: one possibility that came to mind earlier: we could provide an API call that returns a regexp suitable for filtering the log according to a set of specified entities
[09:21] <fwereade> TheMue, rogpeppe, AFAICS it's still not an argument for making clients filter by prefix
[09:22] <fwereade> rogpeppe, that's maybe interesting, but it feels like the time spent to implement it would be better spent on just writing the API at the appropriate level in the first place
[09:23] <rogpeppe> fwereade: that would enable clients to demultiplex the log stream themselves, which just specifying the filter in terms of entities would not
[09:24] <rogpeppe> fwereade: (assuming that's what you mean by "writing the API at the appropriate level")
[09:24] <fwereade> rogpeppe, but transforming on the way out *would* I think allow demuxing
[09:24]  * TheMue feels sad - having a now nicely working solution on API and CLI, with local provider and <= 1.16 support
[09:25] <rogpeppe> fwereade: transforming into what format on the way out?
[09:25] <TheMue> rogpeppe: some kind of entities where the tag is a field, like the timestamp, the logging class and the message
[09:26] <rogpeppe> TheMue: it's not your fault - this is stuff that has not been thrashed out before
[09:26] <fwereade> rogpeppe, possibly the one we have right now, just prefixing by entity tag -- but *officially* in that format, not just by conincidence
[09:26] <rogpeppe> fwereade: agreed
[09:26] <TheMue> rogpeppe: the solution, as it is, would already help people *today*, while changes later surely can be done
[09:27] <rogpeppe> fwereade: i would like to take the time to define a format.
[09:27] <rogpeppe> fwereade: but i agree with TheMue - what's there now is useful and better that what we already have
[09:28] <rogpeppe> fwereade: (and also we want the regexp stuff too, so it's not like we'd be throwing anything away in the future)
[09:28] <fwereade> TheMue, well, what we *originally* had was --include and --exclude by entity
[09:28] <rogpeppe> fwereade: in pyjuju?
[09:28] <fwereade> TheMue, rogpeppe: yes; was it not clear that we were still trying to go for feature parity here?
[09:29] <fwereade> TheMue, rogpeppe: the existing implementation was known to be a filthy hack
[09:29] <rogpeppe> fwereade: sure - i'd like that.
[09:29] <fwereade> TheMue, rogpeppe: extending that hackery is less than ideal
[09:29] <rogpeppe> fwereade: but i don't want to push TheMue's branch back by another couple of weeks while we sort all this out
[09:29] <fwereade> TheMue, rogpeppe: especially since we'll be stuck with it for 5 years
[09:30] <fwereade> TheMue, rogpeppe: IIRC TheMue's first implementation at least had include-by-tag, because I explicitly asked for it
[09:30] <rogpeppe> fwereade: i *think* that the current branch could reasonably be the basis for something that *does* provide explicit tag-based filtering
[09:30] <TheMue> fwereade: --include / --exclude? sorry, what do I miss?
[09:30] <fwereade> rogpeppe, ok, but anything we land in the next couple of days is going to be in 1.18
[09:31] <fwereade> TheMue, pyjuju
[09:31] <TheMue> fwereade: ah, ok
[09:31] <rogpeppe> fwereade: i think that's ok - we can explicitly state that the log file format will probably change
[09:31] <fwereade> TheMue, rogpeppe: why even land it if it can't help the gui?
[09:31] <TheMue> fwereade: using the current format it should be no problem to add a flag --tag that creates a filter
[09:32] <fwereade> TheMue, the CLI shouldn't be using tags as input, should it?
[09:32] <TheMue> fwereade: why doesn't it help the gui?
[09:32] <TheMue> fwereade: the gui can pass filters too
[09:32] <fwereade> TheMue, because if the format is going to change, the gui will need to change as well
[09:33] <TheMue> fwereade: that's called change management, yes *scnr*
[09:33] <fwereade> TheMue, this is *exactly why* I care about doing it by tag
[09:33] <TheMue> fwereade: what do we do if the tag format changes?
[09:33] <fwereade> TheMue, and why we had a couple of rounds on the original CL in which I tried to make sure you put the transformation from tag to prefix at the right layer
[09:34] <fwereade> TheMue, I thought this *had* been hashed out
[09:34] <rogpeppe> fwereade: blame me
[09:34] <fwereade> TheMue, we don't change the tag format
[09:34] <TheMue> fwereade: just found "A camel is a horse designed by committee.", maybe that matches here
[09:35] <TheMue> fwereade: if we don't change the tag format, why aren't you sure that we don't change the log format?
[09:35] <fwereade> rogpeppe, TheMue: I do not care about blaming anybody, we should all be fundamentally aligned here...
[09:35] <TheMue> fwereade: my experience is that everything can change over time and the process has to handle it
[09:35] <rogpeppe> fwereade: one possibility: we specify that, whatever format we decide on subsequently, we will define it so that the entity tag is at the start of the line
[09:35] <fwereade> TheMue, the tag format is the language of the API, and we're stuck with it
[09:36] <fwereade> TheMue, the log format has changed several times already
[09:36] <rogpeppe> fwereade: but leave everything else unspecified for now
[09:36] <rogpeppe> fwereade: that then makes the current scheme ok for future use
[09:37] <TheMue> fwereade: sounds like plan, but I've seen often enough things changing in long running projects. typical maintenance troubles
[09:37] <TheMue> fwereade: that's why I'm not as sure about it as you are
[09:37] <fwereade> rogpeppe, except that we *still* have to figure out what regexps refer to entities and unpick them if//when we use a distributed log db
[09:38] <TheMue> fwereade: and that's why I see no problem in change but I *very high* need in doing everything possible to enable changes
[09:38] <fwereade> rogpeppe, if we at least separate the channels for entity specification and message filtering we're in a slightly better position
[09:38]  * TheMue gets philosophical
[09:38] <rogpeppe> fwereade: that's actually not hard to do
[09:38] <rogpeppe> fwereade: http://golang.org/pkg/regexp/#Regexp.LiteralPrefix
[09:39] <fwereade> rogpeppe, `|`?
[09:39] <rogpeppe> fwereade: hmm, except that won't quite work
[09:39] <rogpeppe> fwereade: yeah :-)
[09:40] <fwereade> TheMue, I don't follow -- ISTM that having clients operate directly wrt an internal format detail makes change harder -- expand please?
[09:40] <fwereade> TheMue, I am +1000 on designing for change where possible
[09:40] <fwereade> TheMue, if I felt we were doing that to an adequate standard I would not be fussing like this
[09:40] <rogpeppe> fwereade: that's an interesting point BTW
[09:41] <rogpeppe> fwereade: (the possibility of a distributed-by-entity log file)
[09:41] <fwereade> rogpeppe, ah I hadn't thought of it that way actually :)
[09:41]  * fwereade had started feeling smug for a moment there
[09:42] <fwereade> rogpeppe, TheMue: the heart of my argument is, I think, what I said a few lines back: entity specification and message filtering are different things, and we shouldn't conflate them
[09:43] <rogpeppe> fwereade: ok, i think i'm sold
[09:43] <rogpeppe> fwereade: but...
[09:43] <rogpeppe> fwereade: i still think that this CL is a good foundation for putting entity specification in the API
[09:44] <rogpeppe> fwereade: i think it's pretty much a strict subset of that
[09:44] <rogpeppe> fwereade: so can i suggest we get it merged and then move on with adding explicit entity specification in the command line and the API?
[09:45] <TheMue> fwereade: I'm still searching for the use case where specifying the entity tag would help me
[09:45] <fwereade> rogpeppe, TheMue: I could live with that *if* we weren't just about to release 1.18 and have to carry what lands
[09:45] <fwereade> TheMue, foo/1 is in an error state, I want to see the logs for foo/1
[09:45] <TheMue> fwereade: yes, I can do so
[09:45] <fwereade> TheMue, your *example* is "I want to see the logs for machine 1" ;p
[09:45] <rogpeppe> fwereade: we can explicitly say that entity specification is not supported yet
[09:46] <rogpeppe> fwereade: but it still gives people (and command-line clients in particular) a useful feature
[09:46] <TheMue> fwereade: yes, I can filter for it, no problem, but I can even filter deeper or more flexible
[09:47] <fwereade> TheMue, rogpeppe: entity filtering is honestly trivial to add afaics, compared to the issues with /var/log/juju
[09:47] <rogpeppe> fwereade: you're probably right. but i'd still do it as a separate CL, and then maybe land them together
[09:47] <fwereade> TheMue, and you could also do so with direct entity specification
[09:48] <fwereade> TheMue, if you want complete flexibility you can still do it by watching the environment logs
[09:48] <rogpeppe> fwereade: i think it's a bit tough to say that a single CL has to land a complete feature
[09:48] <fwereade> TheMue, and filtering from there
[09:49] <fwereade> rogpeppe, I'm not saying it does, necessarily, but if we release with this particular half-feature it encourages clients like the GUI to start depending on prefix filtering
[09:49] <rogpeppe> fwereade: in that specific case it's easy - we tell them not to depend on it
[09:49] <TheMue> fwereade: sure I can, but I don't see the advantage
[09:50] <fwereade> TheMue, correct me if I'm wrong: the GUI use case is "show logs for X entity"
[09:50] <TheMue> fwereade: operators simply grep through logs or use smnp with according monitoring applications (what even would be by far better)
[09:51] <TheMue> fwereade: what they can do by passing the filter
[09:51] <TheMue> fwereade: yes
[09:51] <fwereade> TheMue, if the only way they can do that is by depending on implementation details, we have a problem, because we're suddenly unable to change those impl details without breaking clients
[09:52] <TheMue> fwereade: isn't implementing an API with a specific "tag" argument some kind of changeable implementation detail too?
[09:52] <fwereade> TheMue, so the GUI shouldn't have a "show logs" button on an entity? it should just have a logging pane where users type regexps?
[09:52]  * TheMue things of the hard changes from zk to mongo, from state to api ...
[09:53] <fwereade> TheMue, the tag format is *explicitly* the unambiguous entity identifier for use in the API
[09:54] <fwereade> TheMue, if we have to change that then *everything* breaks
[09:54] <TheMue> fwereade: "show logs" uses the regex for the current log format (could be changed when the log format ever changes) and "drill down" allows to enter a regex
[09:54] <TheMue> fwereade: like the change from zk to mongo
[09:54] <TheMue> fwereade: or the change from py to go
[09:54] <fwereade> TheMue, so you'd rather add an API that converts from tag to reges, and make people use that, than just specifythe tag?
[09:54] <TheMue> fwereade: once again, I've seen enough software changing during my life
[09:55] <fwereade> TheMue, believe me, so have I
[09:56] <TheMue> fwereade: users sees button, gui code in case of only one entity passes the correct filter for it (like it would pass the correct tag). additional ways provide more flexibility
[09:56] <fwereade> TheMue, but I am somewhat baffled that you're advocating depending on an implementation detail, and pooh-poohing objections by saying that a defined and published external identifier ormat might also change
[09:56] <fwereade> TheMue, *how does the gui know what the correct format is*?
[09:56] <fwereade> TheMue, regexps good
[09:56] <fwereade> TheMue, do not drop regexp support
[09:57] <fwereade> TheMue, just, please, do not confuse it with entity specification
[09:58] <TheMue> fwereade: how does entering tags *and* filters work?
[09:58] <TheMue> fwereade: double grepping for tag first and then the regexp in there?
[09:59] <fwereade> TheMue, given the current log format, yes, I think so
[09:59] <TheMue> fwereade: and when the regexp contains a ^, is it the beginning of the rest of the message behind the tag?
[10:00] <rogpeppe> [09:58:50] <TheMue> fwereade: double grepping for tag first and then the regexp in there?
[10:00] <rogpeppe> that seems like the model i was thinking of
[10:00] <rogpeppe> the tag is a first-level filter
[10:00] <fwereade> TheMue, it *might* be nice if that were the case, yes, but I understand that's tedious
[10:01] <rogpeppe> fwereade: i don't think that should be the case, FWIW
[10:01] <TheMue> fwereade: I already have a time problem as Curtis asked Abel and me to support Dave
[10:01] <rogpeppe> fwereade: i think the regexp should exactly match against the lines produced
[10:01] <TheMue> fwereade: so no problem to enlarge the time problem :D
[10:01] <fwereade> rogpeppe, TheMue: yeah, it's probably a slightly different thing -- messagefilter=
[10:02] <fwereade> TheMue, haha
[10:02] <fwereade> TheMue, rogpeppe: ISTM that adding a query arg and grepping by that is not a major burden
[10:02] <rogpeppe> fwereade: i really think this CL could land though
[10:02] <rogpeppe> fwereade: it's not hard to do, no
[10:03] <fwereade> TheMue, rogpeppe: I remain much more worried about the log-location stuff
[10:03] <rogpeppe> fwereade: i'm just replying to that
[10:03]  * TheMue passes his branch to rogpeppe for doing the not so hard to do stuff ;)
[10:03] <fwereade> TheMue, rogpeppe: ISTM that it's not hooked up to rsyslog at all, and that this is a problem
[10:03] <TheMue> fwereade: yeah, the whole logging needs a rework
[10:04] <TheMue> fwereade: especially that the part of the rsyslogd and all-machines.log is a fire-and-forget
[10:04] <rogpeppe> fwereade: the idea is not that log-location is arbitrarily specifiable
[10:04] <fwereade> TheMue, but adding log-location adds a whole new way to break it
[10:04] <fwereade> rogpeppe, but this CL makes it so
[10:04] <rogpeppe> fwereade: yes - i think that can be worked around though
[10:05]  * fwereade frets but waits to read the response
[10:05] <rogpeppe> fwereade: by, for instance, checking in Prepare
[10:05] <TheMue> fwereade: the log-location has been needed to give information only available during bootstrap to the state server later
[10:05] <TheMue> fwereade: but I'm not happy about it too
[10:05] <TheMue> fwereade: it's only for the beast "local provider"
[10:06] <TheMue> fwereade: we won't have that problem if machine-0 of the local provider would be a lxc container too
[10:06] <TheMue> fwereade: but maybe enough other probs then ;)
[10:06] <fwereade> TheMue, yeah, but nested lxc is a beast
[10:06] <fwereade> TheMue, we need something running on the host to add more containers on the host
[10:06] <rogpeppe> fwereade: why would it need to nest?
[10:07] <rogpeppe> fwereade: that's true
[10:07] <TheMue> rogpeppe: containers -to 0
[10:07] <rogpeppe> TheMue: that doesn't need to work - you can't do that to *any* of the machines in the local provider
[10:07] <fwereade> TheMue, not even that
[10:07] <fwereade> TheMue, just adding machines 1, 2, 3
[10:08] <TheMue> fwereade: hmm, a kind of extra daemon talking to the provisioner? no, too heavy
[10:08] <rogpeppe> fwereade: but it could be a very limited-scope thing, rather than the whole shebang
[10:08] <TheMue> fwereade: yep, just seen your note
[10:08] <fwereade> TheMue, yeah, I've been round that argument a couple of times in my head :)
[10:08] <TheMue> fwereade: come on, mr brain, you can do it in a couple of hours, can't you?
[10:08] <fwereade> rogpeppe, we *could* implement a daemon that exposed a cloud-provider-like API and just run that on the host, it's true
[10:09] <fwereade> TheMue, ;p
[10:09] <fwereade> rogpeppe, but we didn't
[10:09] <rogpeppe> fwereade: that would have been my inclination
[10:09] <fwereade> rogpeppe, might even have been better in hindsight, but hey ho, it is what it is
[10:10] <rogpeppe> fwereade: so, to get back to log-location, what's a decent way for the local provider to pass the location of the log file to the machine-0 API server?
[10:11] <fwereade> rogpeppe, modulo upgrades, put it on the machine in state?
[10:11] <fwereade> rogpeppe, given upgrades, break layering :/
[10:11] <TheMue> rogpeppe, fwereade: I've chosen the config to not extent the environment entity
[10:12] <rogpeppe> fwereade: one though i had was that it could go in the agent config file?
[10:12] <rogpeppe> s/though/thought/
[10:12] <fwereade> rogpeppe, that sounds plausible, keep going
[10:12] <rogpeppe> fwereade: it would then be passed into the api server by the machine agent
[10:13] <fwereade> rogpeppe, yeah, I think I like that
[10:13] <rogpeppe> fwereade: and set up at bootstrap time by the provider
[10:13] <rogpeppe> darn, i nearly suggested that, but thought that log-location was actually fine in practice
[10:13]  * TheMue now even more thinks of passing the stuff to rogpeppe 
[10:14] <rogpeppe> TheMue: i'm very sorry
[10:15] <TheMue> rogpeppe: who demands changes has to do them ;P
[10:15] <rogpeppe> TheMue: fwereade then :-)
[10:15] <TheMue> rogpeppe: h5
[10:17] <fwereade> $ juju deploy rogpeppe -n 5
[10:17] <fwereade> ERROR: blah blah
[10:17] <fwereade> dammit
[10:17] <rogpeppe> ERROR: no units available
[10:18]  * fwereade wonders if he can maybe get dimitern onto it -- jam?
[10:19] <mgz> fwereade: no jam back quite yet
[10:21] <TheMue> mgz: yep, finding pics from capetown in my fb timeline from time to time
[10:21] <TheMue> s/from/og/
[10:21] <TheMue> s/og/of/
[10:21] <TheMue> aaargh
[10:21] <mgz> TheMue: weel, he's back in dubai, but out right now
[10:22] <TheMue> mgz: ah, ok
[10:28] <fwereade> TheMue, what is it you'll be doing for davecheney
[10:28] <fwereade> ?
[10:31] <TheMue> fwereade: quickly forwarded you the mail by curtis. we not yet started, need still access to the according machines to use for development
[10:42] <mattyw> any volunteers for helping me work out what's going on inside juju/api.go:212 (apiConfigConnect)
[10:43] <mgz> mattyw: what's confusing you there?
[10:44] <mgz> summary is we try to connect to more than one api server at once
[10:44] <mgz> and if we get more than one error,
[10:44] <jam> mgz: hey, I finally made it
[10:44] <mgz> prefer displaying the more specific ones
[10:45] <mgz> jam: just in time for standup - 1:1 after?
[10:45] <jam> mgz: well, I'm meeting with nate after, but otherwise I would
[10:45] <mattyw> mgz, ok - I'm struggling to see where the two connections happen - I'm implenting the id work - so you can add users and login at the command line-  and at the moment entering an invalid login doesn't fail - because it falls back to the user-admin
[10:46] <mgz> mattyw: l156/l174
[10:47] <mgz> and utils/parallel/try.go
[10:47] <TheMue> rogpeppe: could you add a sketch of you log-location alternative to the CL comments?
[10:50] <rogpeppe> fwereade: standup?
[10:51] <mattyw> mgz, thanks very much - I think I can see the next thing I need to do
[10:53] <jam> wallyworld: standup?
[12:32] <rogpeppe> dimitern, fwereade, jam: this was the algorithm i had in mind: http://paste.ubuntu.com/6954172/
[12:33] <dimitern> rogpeppe, looking
[12:33] <rogpeppe> it's not easy to implement efficiently though if we're filtering many units
[12:34] <rogpeppe> s/units/tags/
[12:34] <rogpeppe> the filtering operation is O(number of tags)
[12:35] <dimitern> looks reasonable in general
[12:36] <rogpeppe> if we made the regexp match only the text following the tag, then we could be considerably more efficient (two regexp matches for any line, regardless of the number of units) but dealing with '^' in the regexp would be awkward
[13:03] <jam> rogpeppe: couldn't you always just mix the regexes together with '|' ?
[13:03] <jam> (^foo)|(^bar) should work fine
[13:04] <rogpeppe> jam: not quite
[13:04] <rogpeppe> jam: because the filter regexp starts from the beginning of the line, and that's also where the tag regexp starts
[13:04] <jam> rogpeppe: also, I'm not sure if your tagFilter matching is *quite* right because "unit-1" ends up matching "unit-10" which I don't think is intentional
[13:05] <rogpeppe> jam: you could do something like this though: http://paste.ubuntu.com/6954294/
[13:05] <rogpeppe> jam: yeah, the prefix should include the final :
[13:06] <rogpeppe> jam: the above code won't work with '^', but it's not too hard to make it work, i think, although it's slightly icky
[13:07] <jam> rogpeppe: I also thought we wanted to support "service" as well, which would be a prefix without the :, but probably with the -
[13:08] <rogpeppe> jam: yeah, definitely
[13:08] <rogpeppe> jam: "service-wordpress" would turn into the prefix "unit-wordpress-" i think
[13:09] <rogpeppe> jam: the same kind of thing applies to the second example
[13:44] <rogpeppe> jam, dimitern: couldn't resist making it work: http://paste.ubuntu.com/6954469/
[13:45] <dimitern> rogpeppe, nice!
[13:46] <rogpeppe> dimitern: the icky bit is line 118
[13:46] <dimitern> rogpeppe, and 123 i guess
[13:46] <rogpeppe> dimitern: well, that's only a consequence of the other
[13:47] <rogpeppe> dimitern: in practice this is not actually a problem
[13:47] <rogpeppe> dimitern: it just feels a bit dubious
[13:48] <dimitern> rogpeppe, i'll take that into account when i'm doing debug-log
[13:53] <rogpeppe> dimitern, fwereade: one thing we didn't consider in this was what the command-line interface should look like
[13:54] <dimitern> rogpeppe, --include (optional, assumed it is) and --exclude (if given), each having tag[=regexp] perhaps?
[13:54] <fwereade> rogpeppe, list of entities by name-not-tag; empty implies environment; explicit --excludes; I like the name[=regexp] idea
[13:54] <fwereade> ?
[13:55] <rogpeppe> i also like the name[=regexp] idea
[13:56] <rogpeppe> although perhaps ":" might seem more apt
[13:57] <fwereade> rogpeppe, I think I have a slight preference for `=` but I'm aware that --exclude=foo-bar is a bit weird
[13:57] <fwereade> --exclude=foo=bar
[13:58] <rogpeppe> juju debug-log --exclude 0:API wordpress mysql/1
[14:00] <rogpeppe> juju debug-log exclude :API
[14:00] <rogpeppe> juju debug-log --exclude :API
[14:00] <rogpeppe> rather
[14:01] <rogpeppe> my preference for : is mainly from the fact that it's a very different kind of association from the other places = is used on the command line
[14:02] <dimitern> what's the ":" in there?
[14:02] <marcoceppi> any thoughts on this? https://bugs.launchpad.net/juju-core/+bug/1281583 I'm pretty sure I can hack it in, was wondering if there was any objections
[14:02] <_mup_> Bug #1281583: Juju should NOT put a .git directory in $CHARM_DIR <juju-core:New> <https://launchpad.net/bugs/1281583>
[14:02] <rogpeppe> dimitern: it's my preference instead of "="
[14:03] <dimitern> rogpeppe, well, it's not entirely expected from a command-line argument formats
[14:03] <rogpeppe> dimitern: is ":" special to the shell?
[14:03] <dimitern> rogpeppe, not really, it's a path separator
[14:04] <rogpeppe> dimitern: only under windows
[14:04] <dimitern> rogpeppe, but it looks weird and non-*nix standard for CLI
[14:04] <rogpeppe> dimitern: and that's not really relevant here, i think
[14:04]  * rogpeppe tries to think of other places that use "=" as a separator like that
[14:05] <dimitern> rogpeppe, --exclude=foo=bar is the same as --exclude foo=bar as far as parsing is involved
[14:05] <rogpeppe> dimitern: most places use it to associate attributes with values, don't they? this isn't quite that
[14:05] <rogpeppe> dimitern: i agree there
[14:05] <dimitern> rogpeppe, the only places in a CLI where i think is common is to define host:port or ipv6 addresses
[14:06] <dimitern> marcoceppi, fwereade is working on replacing the need for git entirely in the uniter
[14:06] <rogpeppe> dimitern: usually the identifier on the lhs of an "=" is non-optional; i guess we could make that the case here though.
[14:07] <rogpeppe> dimitern: so "juju debug-log =someregexp" becomes illegal and you have to do "juju debug-log environment=someregexp" instead
[14:09] <rogpeppe> dimitern: somehow the equality sign seems slightly intuitively wrong, as there's no assignment or equality; we're saying "here's a tag and a filter that applies within that tag"/
[14:09] <rogpeppe> another possibility might be to use square brackets: wordpress[regexp]
[14:11] <marcoceppi> dimitern fwereade: will this land for 1.18?
[14:11] <dimitern> marcoceppi, possibly
[14:12] <dimitern> rogpeppe, how about pairs of --include/exclude name --filter regexp ?
[14:13] <rogpeppe> dimitern: not keen - it would be pain to do, and it doesn't really fit well with standard flag parsing logic
[14:13] <rogpeppe> s/pain/a pain/
[14:13] <dimitern> marcoceppi, the initial step is proposed (in charm package), but the uniter changes are not yet
[14:14] <dimitern> rogpeppe, [] will interfere with the regexp itself
[14:14] <rogpeppe> dimitern: not really
[14:14] <rogpeppe> dimitern: it's trivial to strip off before sending the regexp
[14:15] <dimitern> rogpeppe, i mean from the ux perspective - is it part of it? is it not? should I escape it? pretty much the same questions I keep asking myself with the weird emacs grep regexps
[14:16] <rogpeppe> dimitern: i agree that the plain string concat is simpler
[14:29] <dimitern> rogpeppe, we can get even weirder --wordpress[=regexp] (i.e. --<name>[=regexp])
[14:29] <rogpeppe> dimitern: no we can't :-)
[14:36] <dimitern> rogpeppe, or configure-style --with-<name>[=regexp] and --without-<name>[=regexp]
[14:36] <rogpeppe> dimitern: that's still bad :-) flags should have a small known set of names
[14:37] <dimitern> rogpeppe, or just plain old -i (--include) name1[=regexp1] name2[=regexp2] -e (--exclude) name3[=regexp] ...
[14:37] <dimitern> i.e. everything after -i (--include) [default] is name-regexp pairs, same for -e (--exclude)
[14:38] <rogpeppe> dimitern: tbh, i think william's idea of having excludes mentioned explicitly seems reasonable.
[14:38] <rogpeppe> dimitern: the only point i quibbled was whether to use "=" or ":" as a separator
[14:38] <dimitern> rogpeppe, if it's just a *minor* quibble, then can you live with = ? :)
[14:40] <rogpeppe> dimitern: i guess. still seems a bit wrong to me, but we'll see what people think.
[14:41] <dimitern> rogpeppe, what other people? you want to discuss it on the ML ?
[14:42] <rogpeppe> dimitern: well, we probably should. i was really thinking of peoples' reactions when they see the new feature
[14:43] <dimitern> rogpeppe, right, that'll blow a minor implementation detail way out of proportion, but i guess it's worth at least announcing it :)
[14:44] <rogpeppe> dimitern: how about writing the command docs as you'd get them from juju help debug-log
[14:44] <rogpeppe> ?
[14:44] <dimitern> rogpeppe, sgtm
[14:45] <dimitern> rogpeppe, i'm about to write one about the environment/api urls/allwatcher, so I'll write about this one as well
[14:45] <rogpeppe> dimitern: cool
[15:03] <dimitern> rogpeppe, so how does the allwatcher report changes that include an environment tag?
[15:03] <dimitern> rogpeppe, I couldn't see that in params.Delta
[15:03] <rogpeppe> dimitern: good question - one mo, i'll have a look
[15:08] <rogpeppe> dimitern: hmm, looks like it's the UUID-based environ tag
[15:08] <rogpeppe> dimitern: the only place that does that is annotation changes
[15:09] <dimitern> rogpeppe, can you point me to the code?
[15:09] <rogpeppe> dimitern: the relevant piece is in the State.Environment method
[15:10] <dimitern> rogpeppe, ta!
[15:10] <rogpeppe> dimitern: where it initialises the annotator value with env.Tag()
[15:11] <rogpeppe> dimitern: perhaps you could talk to the GUI guys about making their code compatible with both "environ" and "environ-343ab464"
[15:13] <dimitern> rogpeppe, but how does the response to AW.Next() look like if there's a change to env's annotations?
[15:13] <dimitern> rogpeppe, where's the actual tag included?
[15:15] <rogpeppe> dimitern: the Delta.Entity field will be an AnnotationInfo containing the tag
[15:17] <dimitern> rogpeppe, ah, I see, cheers
[15:31] <fwereade> mgz, sorry to hassle you when I know you're doing bugs, but I was wondering about https://code.launchpad.net/~dstroppa/juju-core/joyent-provider-storage/+merge/200851 -- can this be landed now?
[15:32] <fwereade> TheMue, btw, not sure we said explicitly: dimitern is taking over your debug-log branch, so you can go do hardware enablement with a clear conscience :)
[15:35] <mgz> fwereade: I think so, but that does entail bringing in the gojoyent library
[15:35] <fwereade> mgz, ahh... and I'm not sure I ever looked at that properly >_<
[15:35] <fwereade> mgz, level of confidence?
[15:37] <natefinch> man, we really need a --quiet or something for the tests... when a bunch of things fail I just get a billion lines of useless log messages that make it impossible to actually see what tests are failing.
[15:37] <mgz> fwereade: I think we can do it would risking breaking anything else
[15:37] <mgz> *without
[15:37] <mgz> ...that was a confusing braino
[15:37] <fwereade> mgz, I managed to parse it :)
[15:38] <fwereade> mgz, ok, cool
[15:38] <fwereade> mgz, the joyent provider doesn't get compiled in anyway, does it?
[15:38] <mgz> we've not added it to the magic place yet, no
[15:39] <fwereade> mgz, ok, cool, please do land it in your, uh, Copious Free Time...
[15:39]  * fwereade has to go see laura doing rudimentary flamenco, later all
[15:39] <natefinch> fwereade: sounds like fun :)
[15:39] <mgz> okay, I shall queue it up
[15:45] <TheMue> fwereade: thx for info, I already thought so following the discussions here
[15:47] <sinzui> I anyone working  on bug 1271937
[15:47] <_mup_> Bug #1271937: Use juju-mongodb when the package is available <arm64> <ci> <mongodb> <ppc64el> <trusty> <juju-core:Triaged> <https://launchpad.net/bugs/1271937>
[15:48] <TheMue> fwereade: and if my branch so far, as well as the Tailer, will find their way into Juju I'm happy
[15:48] <natefinch> sinzui: I'm working on landing it
[15:48] <natefinch> sinzui: hitting a ton of spurious test failures
[15:50] <sinzui> natefinch, oh good. CI ran out of disk space over the weekend. Once your branch is merged and CI passed it, I will release. Can you ping me when your branch is merged?
[15:50] <natefinch> sinzui: sure thing
[15:50] <sinzui> Thank you!
[16:36] <natefinch> mgz, rogpeppe, dimitern: easy review?  https://codereview.appspot.com/65460043/
[16:36] <dimitern> natefinch, looking
[16:36] <natefinch> dimitern: ta
[16:36] <mgz> dimitern wins
[16:41] <rogpeppe> natefinch: reviewed
[16:42] <natefinch> mgz: looks like rogpeppe wins where it counts ;)
[16:45] <dimitern> natefinch, reviewed
[16:46] <dimitern> it counts how thorough you review as well :P
[16:46] <natefinch> rogpeppe, dimitern: thanks
[16:46] <natefinch> dimitern: haha yes, definitely
[16:49] <rogpeppe> dimitern, natefinch: not entirely sure about "PrependPath", as it doesn't indicate the "patch" nature of it
[16:50] <dimitern> rogpeppe, natefinch, the thing that irks me a bit is that seeing s.PatchPath(blah) doesn't immediately tell me what happens to blah
[16:50] <rogpeppe> dimitern: yeah, i definitely see that too
[16:50] <dimitern> rogpeppe, but I'm +1 for ExecPath
[16:50] <rogpeppe> dimitern: i'm trying to think of a better way
[16:51] <natefinch> rogpeppe, dimitern:  I like PatchEnvPath better than PatchExecPath but I agree, getting some "prepend" in the name would be nice
[16:52] <rogpeppe> natefinch: one possibility: s.PatchEnvPath(dir, testbase.Prepend)
[16:52] <rogpeppe> vs s.PatchEnvPath(dir, testbase.Overwrite)
[16:52] <dimitern> natefinch, rogpeppe, that sgtm
[16:52] <rogpeppe> it makes it slightly longer, but the use is rare enough that i don't think it matters
[16:53] <arosales> rogpeppe, fwereade, sinzui could you guys give me a very high level features targetted for 1.18
[16:53] <rogpeppe> and i'm sure there are places where we actually want to overwrite rather than prepending
[16:53] <natefinch> rogpeppe: except that almost everywhere (possibly 100% everywhere), we prepend, not overwrite
[16:53] <sinzui> arosales, the 1.17.x release notes
[16:54] <arosales> sure but I think we still have some in the queue correct?
[16:54] <sinzui> arosales, https://launchpad.net/juju-core/+milestone/1.17.0 https://launchpad.net/juju-core/+milestone/1.17.1 https://launchpad.net/juju-core/+milestone/1.17.2
[16:54] <rogpeppe> natefinch: yeah - but it reads nicely, and actually perhaps some of those places don't actually want to inherit the system path
[16:55] <arosales> sinzui, thanks any major features targetted post 1.17.2?
[16:55] <natefinch> rogpeppe: but I'm not going to spend time to figure out what tests really don't want to inherit the system path.  All I was trying to do was wrap up some common code to do things the right way.   Complicating the interface because sometime we might need to overwrite the path does not seem like a good idea.
[16:55] <sinzui> arosales, juju-mongodb
[16:56] <rogpeppe> natefinch: that's ok - i was only trying to think of a better name. PatchPrependEnvPath doesn't read so well.
[16:57] <sinzui> arosales, juju-run, API-every-where is the hidden feature for developers.
[16:57] <rogpeppe> natefinch: or maybe it's actually ok, i'll leave it to your judgement
[16:57] <natefinch> rogpeppe: I know, it's terrible, that's why I was just going to do PatchEnvPath....  I dunno
[16:57] <arosales> sinzui, thanks
[16:58] <rogpeppe> natefinch: i think it reads quite nicely with the extra arg though
[16:58] <rogpeppe> natefinch: which is why i suggested it
[16:58] <natefinch> rogpeppe: yeah, if we could default it to prepending, that would be cool, but I hate adding a parameter when we're never actually going to set it differently (at least for now).
[16:59] <rogpeppe> natefinch: PathEnvPathByPrependingDir ?
[16:59] <rogpeppe> s/Path/Patch/
[17:01] <natefinch> rogpeppe: can we somehow make it a FactoryBean, too? ;)
[17:02] <natefinch> rogpeppe: PatchEnvPathPrepend seems ok
[17:02] <rogpeppe> natefinch: i'd be ok with that
[17:02] <natefinch> rogpeppe: cool.
[17:02] <dimitern> i was going to suggest just that earlier in fact ;)
[17:32] <hazmat> ugh.. regex
[17:32] <hazmat> why
[17:32] <hazmat> just multi-include multi-exclude
[17:33] <hazmat> with machines/units/services
[17:33] <hazmat> i thought that got resolved 12 hrs ago
[17:33] <hazmat> dimitern, ^
[17:34] <dimitern> hazmat, because that's what we discussed today and hence the mail to ask for comments :)
[17:53] <hazmat> sent
[18:49]  * rogpeppe is done for the day
[18:49] <rogpeppe> g'night all
[20:55] <wallyworld> fwereade: can i poke you to take another look at https://codereview.appspot.com/63730043/ when you next get a chance? i think all the issues have been fixed
[22:06] <fwereade> wallyworld, looking
[22:11] <fwereade> wallyworld, LGTM, tyvm
[22:11] <wallyworld> awesome, thanks for looking
[22:11] <fwereade> wallyworld, np
[22:12] <fwereade> wallyworld, would you let waigani know that I'd really like to see something that builds on the axw's Policy stuff for in-state environ validation? and that I *think* that, with replaceSettingsOp, that can actually nail all the issues I'm currently aware of around environ config changes, and it would make me ever so happy ;)
[22:13] <wallyworld> ok, we like to make you happy
[22:13]  * fwereade approves of this :)