[08:54] <TheMue> rog: morning
[08:54] <rog> TheMue: hiya
[13:15] <niemeyer> Alow
[13:25] <TheMue> niemeyer: hiya
[13:25] <TheMue> niemeyer: my propose of the continued machines based on yesterdays charms (has been a branch of that) is in
[13:33] <niemeyer> TheMue: Thanks
[13:33] <niemeyer> TheMue: I may only get to it tomorrow
[13:33] <niemeyer> TheMue: I'll try to stick to store work today, since I've been clearly failing at it
[13:34] <TheMue> niemeyer: ok, no prob, better keep concentration
[13:34] <TheMue> niemeyer: would part of state should I take next?
[13:38] <rog> lunch
[13:39] <rog> (outside in the sun!)
[13:43] <niemeyer> TheMue: Not sure..
[13:43] <TheMue> rog: enjoy. sun? (envy) here it is grey and wet
[13:43] <niemeyer> TheMue: Actually, wasn't there something to be done as a left over from the current work?
[13:43] <niemeyer> I recall something like that, but can't remind of it now
[13:44] <niemeyer> TheMue: Ah, yes.. zk => st
[13:44] <TheMue> niemeyer: state into all entities, but that's only very small
[13:44] <niemeyer> TheMue: Yeah, but has to be done
[13:44] <TheMue> niemeyer: yep, exactly
[13:44] <niemeyer> TheMue: Besides that, just pick something that moves the state forward
[13:44] <TheMue> niemeyer: ok, will do so
[13:45] <niemeyer> TheMue: Preferably in areas that do not conflict with changes going on in Python (relations)
[13:45] <niemeyer> andrewsmedina: Any progress there?
[13:45] <andrewsmedina> niemeyer: yes
[13:45] <niemeyer> andrewsmedina: Nice
[13:45] <TheMue> niemeyer: yep, no relations
[13:46] <andrewsmedina> niemeyer: I'm finishing the tests for port charm metadata subordinate to Go
[13:55] <niemeyer> andrewsmedina: Very neat
[13:55] <niemeyer> andrewsmedina: Thanks for working on that
[13:58] <andrewsmedina> niemeyer: :D
[14:26] <rog> wondering if anyone is up for sanity-checking a possible approach to making my ec2test fake ec2 server simulate eventual consistency issues
[14:26] <rog> fwereade, jimbaker?
[14:27] <fwereade> rog, I'm trying to finish something off atm, I'll ping you when I'm free
[14:27] <rog> fwereade: thanks
[15:50] <niemeyer> Lunch time here..
[15:53] <fwereade> rog, sorry that took so long; can I be of service?
[15:54] <rog> fwereade: yes, thanks. gimme 1 minute.
[15:56] <rog> fwereade: bakc
[15:56] <rog> ck
[15:56] <rog> fwereade: so...
[15:56] <rog> fwereade: the object is to simulate the "eventual consistency" semantics of the ec2 server.
[15:56] <rog> fwereade: to that end, the idea is that all operations operate on an out-of-date version of the server data.
[15:57] <rog> fwereade: when a client makes a call, if it's a read-only request, we return the results from the out-of-date version.
[15:57] <rog> fwereade: if it's a modification request, we allocate any unique ids (instance ids, group ids etc) from the current version. then we queue up the operation to be executed later.
[15:57] <rog> fwereade: after every call, we execute one operation from the queue.
[15:57] <rog> fwereade: does that sound reasonable to you?
[15:57] <fwereade> rog, yes, I think so
[15:58] <rog> fwereade: i can't help feeling that there might be a large bear lurking in the room somewhere
[15:59] <rog> guess i can only try and see
[15:59] <fwereade> rog, I'm wondering though whether it might be good to have some way of varying the behaviour -- sometimes immediate consistency, sometimes irritatingly delayed
[15:59] <rog> fwereade: the idea is to allow the length of the queue to be specified when the server starts
[15:59] <rog> fwereade: i don't think having random behaviour is going to be good for reproducing failed tests
[15:59] <fwereade> rog, the trouble is you don't *really* want to do that *randomly*, or you're on a miserable path to unreproducible test failures
[16:00] <fwereade> snap :p
[16:00] <rog> fwereade: so i think we'd try all tests with "totally consistent" and "very delayed" behaviour
[16:01] <rog> fwereade: one question i can't quite decide on is if there are significant tests that might fail with random consistency but neither of the above scenarios
[16:01] <fwereade> rog, that's what's bugging me
[16:01] <fwereade> rog, I can see a plausible route towards repeatable random tests
[16:02] <jimbaker> rog, this sounds like what we do in the testing in the python code. not uniformly mind you
[16:02] <jimbaker> in terms of delays
[16:02] <rog> fwereade: what about concurrency?
[16:02] <fwereade> rog, where we store the seed that generated the queue-frobbing and can attach that
[16:03] <rog> fwereade: that assumes we generate requests deterministically.
[16:03] <fwereade> rog, ok, close-enough-to-repeatable to probably be a bit easier to reproduce
[16:03] <rog> fwereade: hence my question "what about concurrency"?
[16:03] <fwereade> rog, quite so; it may very well be that the difference is slight
[16:03] <rog> fwereade: i dunno.
[16:03] <fwereade> rog, and doesn't justify the effort
[16:04] <rog> fwereade: the other thing is that the above approach doesn't work if you can jump backwards and forwards in time.
[16:04] <fwereade> rog, in what sense?
[16:04] <rog> fwereade: because you need to keep several versions of the server state
[16:04] <rog> fwereade: with this approach we just keep one version and apply deltas
[16:04] <fwereade> rog, ah got you
[16:05] <rog> fwereade: hmm
[16:05] <rog> fwereade: although
[16:06] <rog> fwereade: if delta application is deterministic
[16:06] <rog> fwereade: then we can keep several versions and apply deltas to each of them as appropriate
[16:07] <fwereade> rog, sorry, I think I'm lost...
[16:07] <rog> fwereade: we could have several versions of the state, each representing the state at some point within the queue of pending operations
[16:08] <rog> fwereade: as deltas move through the queue, they get applied to each state in turn. or maybe not in turn... to simulate the random ordering
[16:09] <rog> fwereade: assuming s1+delta1+delta2 == s1+delta2+delta1 it should be ok
[16:09] <fwereade> rog, yes, I understand that bit; ah, yes, ok, we can have a separate queue of operations for each... er, time-slice of the server
[16:09] <rog> fwereade: unfortunately that last isn't quite true, although it is for most things AFAICS
[16:10] <fwereade> rog, thinking more though; it might be good to propose it with just the discussed boundary-condition checks, which will be useful if not complete
[16:10] <rog> fwereade: for instance, if the user does AddSecurityGroupAuth followed by RevokeSecurityGroupAuth, what's the result if the ops get reordered?
[16:11] <fwereade> rog, finding some happy medium between randomness and repeatability can wait
[16:11] <fwereade> rog, I have no idea, and that worries me
[16:11] <rog> fwereade: yeah, i think so. and that probably makes it easier to do something more sophisticated later on if we deem it useful
[16:11] <rog> fwereade: i *wish* there was some documentation on what semantics you can expect from ec2
[16:12] <fwereade> rog, hopefully
[16:12] <rog> fwereade: i can't find anything at all. even discussions.
[16:12] <rog> fwereade: (which i find hard to believe - surely people encounter this stuff all the time?)
[16:12] <fwereade> rog, I think the semantics are "you'll get what you're given, and you'll like it"
[16:12] <jimbaker> rog, maybe take a look at http://www.allthingsdistributed.com/ ?
[16:13] <jimbaker> also there is some docs on expected semantics for s3, fwiw
[16:14] <rog> jimbaker: did you have any particular blog post in mind?
[16:14] <fwereade> rog, heh, what if it's just that it's a *really* rare and unreproducible occurrence, so nobody ever hears about it or quite believes it when it happens?
[16:14] <rog> fwereade: it's not rare!
[16:14]  * fwereade dons tinfoil hat
[16:14] <jimbaker> rog, unfortunately no, but it is the sort of thing werner discusses on his blog
[16:14] <rog> fwereade: it happens a lot of the time in my tests
[16:14] <rog> jimbaker: ok, thanks.
[16:14] <fwereade> rog, ah, ok, I didn't realise
[16:15] <rog> fwereade: otherwise i don't think i'd've know
[16:15] <rog> n
[16:43] <rog> fwereade: one minor comment on https://codereview.appspot.com/5672067/
[16:45] <fwereade> rog, not convinced by that; how about a "checkedClock" (or something) field on the type, as a compromise?
[16:46] <rog> fwereade: i'm not sure that's right. if the file did not at first exist, you never need to check the clock.
[16:46] <fwereade> rog, hm, good point
[16:47] <fwereade> rog, Idon;t really like the idea of threading it through all those calls though :/
[16:47] <rog> fwereade: i think that the calls in Alive and AliveW are in the unique position of knowing that the clock needs to be checked, hence my suggestion.
[16:47] <rog> fwereade: it's only two levels...
[16:47]  * fwereade looks at the code again
[16:56] <fwereade> rog, yeah, it turns out ok
[16:56] <rog> fwereade: cool
[16:56] <fwereade> rog, how about calling setAlive setState instead? it now always sets both .alive and .timeout
[16:58]  * rog has a look
[17:01] <rog> fwereade: seems reasonable. (although i think it always has set both alive and timeout)
[17:02] <fwereade> rog, it went through a brief recent incarnation in which it didn't even bother to read the timeout more than once
[17:02] <rog> fwereade: ah
[17:02] <fwereade> rog, changing it back made the name change seem like a good idea
[17:03] <rog> fwereade: yeah, i think setState is a better name for what it does
[17:04] <fwereade> rog, cheers, reproposed ;)
[17:08] <fwereade> rog, EOD I'm afraid; if you have a spare moment I would appreciate it if you would have a little look at https://codereview.appspot.com/5695082 and let me know the perceived crackfulness of the general idea
[17:08] <rog> fwereade: cool!
[17:09] <rog> fwereade: yes, i'll do that
[17:09] <fwereade> rog, er, expect another one in a sec, I appear to have ballsed up a bit
[17:09] <rog> fwereade: looks better with firstTime, to my mind - that logic becomes a bit more obvious.
[17:09] <rog> fwereade: thanks for changing it.
[17:11] <fwereade> rog, a pleasure :)
[17:12] <fwereade> rog, huh, no I didn't balls it up; somehow I was trying to link against something stale, I guess
[17:13] <fwereade> rog, so don't expect an update soon
[17:13] <rog> fwereade: ok. i'll try to ignore the extraneous diffs
[17:14] <rog> fwereade: where's the implementation of relation-get etc again?
[17:14] <fwereade> rog, in python?
[17:14]  * fwereade racks brains
[17:14] <fwereade> rog, juju.hooks.commands
[17:15] <rog> fwereade: no, that wasn't the question.
[17:15] <rog> fwereade: i was wondering what files the new code for " Rough proposal for implementation of relation-get and similar commands." was in
[17:15] <rog> fwereade: presuming it's somewhere in that CL
[17:15] <fwereade> rog, ah sorry: none of them are *implemented* yet
[17:15] <rog> fwereade: server/protocol looks new
[17:16] <fwereade> rog, but it provides a mechanism by which a command like relation-get can get the unit agent (which knows all the appropriate state) to execute the command for it and pipe back the results
[17:16] <rog> fwereade: yeah, that's what i was looking for. that's in cmd/server, right?
[17:17] <fwereade> rog, yeah
[17:18] <rog> fwereade: instant initial reaction: why not use the rpc package?
[17:18] <rog> fwereade: rather than doing our own marshalling etc
[17:18] <fwereade> rog, huh, because I ballsed up on due diligence :p
[17:19] <rog> fwereade: i think this code can be really really small
[17:19] <fwereade> rog, then that's doubly awesome
[17:19] <fwereade> rog, it was very instructive writing it all but less code is always better than more code ;)
[17:19] <rog> fwereade: the unix socket stuff will stay :-)
[17:20] <fwereade> rog, cool
[17:22] <rog> fwereade: but yeah, i think a few methods exported on a relation type and passed to rpc.Server.Register will probably do the job
[17:23] <rog> fwereade: the only thing that won't give you will be commands that stream.
[17:23] <rog> fwereade: are there any like that?
[17:23] <fwereade> fwereade, some certainly dump data back on stdout
[17:24] <fwereade> er, *rog*, ^^
[17:24] <rog> fwereade: lots and lots of data? or don't-care-too-much-if-it's-all-in-memory data?
[17:25] <fwereade> rog, and in general I'd expect/hope that --verbose and --debug should work (and thereby potentially generate monster stderrs too)
[17:25] <fwereade> rog, not sure how big it could get
[17:26] <rog> fwereade: i've an idea for how that could work ok with rpc
[17:27] <fwereade> rog, well, we could do exactly the same in terms of how we pipe stuff back
[17:27] <rog> fwereade: yeah
[17:27] <fwereade> rog, the streams feel to me like a good thing
[17:27] <rog> fwereade: if you're around for a minute or so more, i'll paste a little example
[17:28] <fwereade> rog, please do
[17:31] <rog> fwereade: http://paste.ubuntu.com/860798/
[17:31] <rog> fwereade: slightly contorted by the necessary form of rpc methods
[17:32] <rog> fwereade: but the idea is that you call New which gives you a handle on a running command. then you can call other methods to get info on that command, including reading from its output streams
[17:32] <rog> fwereade: Cmd is the type that would be registered as an rpc handler
[17:33] <rog> fwereade: maybe Cmd.Result might be better named Cmd.Wait
[17:34] <rog> fwereade: and if we were concerned about malicious clients, we could put a random capability string in the Running id.
[17:38] <fwereade> rog, seems less clear to me but that's probably a matter of familiarity; I shall meditate upon it
[17:39] <rog> fwereade: i'll quickly sketch an implementation
[17:39] <rog> fwereade: i think it'll be pretty small.
[17:40] <fwereade> rog, don't rush, it has been noticed by others that it's my eod
[17:40] <rog> fwereade: no probs.
[17:40] <rog> fwereade: i've only got 20 minutes myself...
[17:40] <rog> fwereade: see ya tomorrow morning
[17:41] <fwereade> rog, and you :)
[18:39] <andrewsmedina> niemeyer: hi
[18:41] <niemeyer> andrewsmedina: yo
[18:43] <andrewsmedina> niemeyer: for port this test (http://dpaste.org/9MrTY/) I need create another dummy charm?
[18:44] <niemeyer> andrewsmedina: You can copy and change an existing one.. some tests already have the needed utilities to do that easily
[18:45] <niemeyer> andrewsmedina: Hold on.. I'll find an example
[18:46] <andrewsmedina> niemeyer: dont have yet in Go a equivalent for change_sample
[18:46] <niemeyer> andrewsmedina: We actually do
[18:46] <niemeyer> andrewsmedina: Check charm_test.go
[18:47] <niemeyer> andrewsmedina: If you have a reader r, you can do something like
[18:47] <niemeyer> hackYaml := ReadYaml(r)
[18:47] <niemeyer> hackYaml["subordiate"] = true
[18:48] <niemeyer> newr := hackYaml.Reader()
[18:49] <andrewsmedina> niemeyer: I will try this
[18:49] <niemeyer> andrewsmedina: It's not as practical as change_sample yet, but feel free to implement a function so that it is
[18:49] <niemeyer> andrewsmedina: For example: changeSample(func(yh *yamlHacker) { yh["subordinate"] = true })
[18:51] <andrewsmedina> niemeyer: is missing only this use case
[19:07] <niemeyer> Booom!
[19:07] <niemeyer> Summer storm..
[22:39] <andrewsmedina> niemeyer: can you help me?
[22:39] <niemeyer> andrewsmedina: Maybe.. :) What's up?
[22:40] <andrewsmedina> niemeyer: All tests are ok
[22:40] <andrewsmedina> niemeyer: but
[22:42] <andrewsmedina> niemeyer: in python (http://dpaste.org/9MrTY/) when a charm is not a proper subordinate, is returned the charm path in the excetion
[22:47] <hazmat> andrewsmedina, and?
[22:47] <hazmat> this is problematic because?
[22:48] <andrewsmedina> hazmat: I dont have path in go implementation
[22:48] <niemeyer> andrewsmedina: Sounds fine.. just don't return it
[22:49] <andrewsmedina> niemeyer: I return a simple error message?
[22:49] <niemeyer> andrewsmedina: Simplicity FTW
[22:50] <andrewsmedina> niemeyer: I will send the commit in minutes :p
[22:50] <niemeyer> andrewsmedina: Superb :)
[22:50] <andrewsmedina> niemeyer: sorry for my noob questions
[22:50] <niemeyer> andrewsmedina: Totally fine
[22:50] <andrewsmedina> Im a noob in go =/
[22:53] <niemeyer> andrewsmedina: Not to worry.. there are a handful of people only that know Go for more than a couple of years, so you're not too far behind :)