=== andrewsmedina_ is now known as andrewsmedina === andrewsmedina_ is now known as andrewsmedina [08:54] rog: morning [08:54] TheMue: hiya === Leseb_ is now known as Leseb [13:15] Alow [13:25] niemeyer: hiya [13:25] niemeyer: my propose of the continued machines based on yesterdays charms (has been a branch of that) is in [13:33] TheMue: Thanks [13:33] TheMue: I may only get to it tomorrow [13:33] TheMue: I'll try to stick to store work today, since I've been clearly failing at it [13:34] niemeyer: ok, no prob, better keep concentration [13:34] niemeyer: would part of state should I take next? [13:38] lunch [13:39] (outside in the sun!) [13:43] TheMue: Not sure.. [13:43] rog: enjoy. sun? (envy) here it is grey and wet [13:43] TheMue: Actually, wasn't there something to be done as a left over from the current work? [13:43] I recall something like that, but can't remind of it now [13:44] TheMue: Ah, yes.. zk => st [13:44] niemeyer: state into all entities, but that's only very small [13:44] TheMue: Yeah, but has to be done [13:44] niemeyer: yep, exactly [13:44] TheMue: Besides that, just pick something that moves the state forward [13:44] niemeyer: ok, will do so [13:45] TheMue: Preferably in areas that do not conflict with changes going on in Python (relations) [13:45] andrewsmedina: Any progress there? [13:45] niemeyer: yes [13:45] andrewsmedina: Nice [13:45] niemeyer: yep, no relations [13:46] niemeyer: I'm finishing the tests for port charm metadata subordinate to Go [13:55] andrewsmedina: Very neat [13:55] andrewsmedina: Thanks for working on that [13:58] niemeyer: :D [14:26] wondering if anyone is up for sanity-checking a possible approach to making my ec2test fake ec2 server simulate eventual consistency issues [14:26] fwereade, jimbaker? [14:27] rog, I'm trying to finish something off atm, I'll ping you when I'm free [14:27] fwereade: thanks === Leseb_ is now known as Leseb [15:50] Lunch time here.. [15:53] rog, sorry that took so long; can I be of service? [15:54] fwereade: yes, thanks. gimme 1 minute. [15:56] fwereade: bakc [15:56] ck [15:56] fwereade: so... [15:56] fwereade: the object is to simulate the "eventual consistency" semantics of the ec2 server. [15:56] fwereade: to that end, the idea is that all operations operate on an out-of-date version of the server data. [15:57] fwereade: when a client makes a call, if it's a read-only request, we return the results from the out-of-date version. [15:57] fwereade: if it's a modification request, we allocate any unique ids (instance ids, group ids etc) from the current version. then we queue up the operation to be executed later. [15:57] fwereade: after every call, we execute one operation from the queue. [15:57] fwereade: does that sound reasonable to you? [15:57] rog, yes, I think so [15:58] fwereade: i can't help feeling that there might be a large bear lurking in the room somewhere [15:59] guess i can only try and see [15:59] rog, I'm wondering though whether it might be good to have some way of varying the behaviour -- sometimes immediate consistency, sometimes irritatingly delayed [15:59] fwereade: the idea is to allow the length of the queue to be specified when the server starts [15:59] fwereade: i don't think having random behaviour is going to be good for reproducing failed tests [15:59] rog, the trouble is you don't *really* want to do that *randomly*, or you're on a miserable path to unreproducible test failures [16:00] snap :p [16:00] fwereade: so i think we'd try all tests with "totally consistent" and "very delayed" behaviour [16:01] fwereade: one question i can't quite decide on is if there are significant tests that might fail with random consistency but neither of the above scenarios [16:01] rog, that's what's bugging me [16:01] rog, I can see a plausible route towards repeatable random tests [16:02] rog, this sounds like what we do in the testing in the python code. not uniformly mind you [16:02] in terms of delays [16:02] fwereade: what about concurrency? [16:02] rog, where we store the seed that generated the queue-frobbing and can attach that [16:03] fwereade: that assumes we generate requests deterministically. [16:03] rog, ok, close-enough-to-repeatable to probably be a bit easier to reproduce [16:03] fwereade: hence my question "what about concurrency"? [16:03] rog, quite so; it may very well be that the difference is slight [16:03] fwereade: i dunno. [16:03] rog, and doesn't justify the effort [16:04] fwereade: the other thing is that the above approach doesn't work if you can jump backwards and forwards in time. [16:04] rog, in what sense? [16:04] fwereade: because you need to keep several versions of the server state [16:04] fwereade: with this approach we just keep one version and apply deltas [16:04] rog, ah got you [16:05] fwereade: hmm [16:05] fwereade: although [16:06] fwereade: if delta application is deterministic [16:06] fwereade: then we can keep several versions and apply deltas to each of them as appropriate [16:07] rog, sorry, I think I'm lost... [16:07] fwereade: we could have several versions of the state, each representing the state at some point within the queue of pending operations [16:08] fwereade: as deltas move through the queue, they get applied to each state in turn. or maybe not in turn... to simulate the random ordering [16:09] fwereade: assuming s1+delta1+delta2 == s1+delta2+delta1 it should be ok [16:09] rog, yes, I understand that bit; ah, yes, ok, we can have a separate queue of operations for each... er, time-slice of the server [16:09] fwereade: unfortunately that last isn't quite true, although it is for most things AFAICS [16:10] rog, thinking more though; it might be good to propose it with just the discussed boundary-condition checks, which will be useful if not complete [16:10] fwereade: for instance, if the user does AddSecurityGroupAuth followed by RevokeSecurityGroupAuth, what's the result if the ops get reordered? [16:11] rog, finding some happy medium between randomness and repeatability can wait [16:11] rog, I have no idea, and that worries me [16:11] fwereade: yeah, i think so. and that probably makes it easier to do something more sophisticated later on if we deem it useful [16:11] fwereade: i *wish* there was some documentation on what semantics you can expect from ec2 [16:12] rog, hopefully [16:12] fwereade: i can't find anything at all. even discussions. [16:12] fwereade: (which i find hard to believe - surely people encounter this stuff all the time?) [16:12] rog, I think the semantics are "you'll get what you're given, and you'll like it" [16:12] rog, maybe take a look at http://www.allthingsdistributed.com/ ? [16:13] also there is some docs on expected semantics for s3, fwiw [16:14] jimbaker: did you have any particular blog post in mind? [16:14] rog, heh, what if it's just that it's a *really* rare and unreproducible occurrence, so nobody ever hears about it or quite believes it when it happens? [16:14] fwereade: it's not rare! [16:14] * fwereade dons tinfoil hat [16:14] rog, unfortunately no, but it is the sort of thing werner discusses on his blog [16:14] fwereade: it happens a lot of the time in my tests [16:14] jimbaker: ok, thanks. [16:14] rog, ah, ok, I didn't realise [16:15] fwereade: otherwise i don't think i'd've know [16:15] n [16:43] fwereade: one minor comment on https://codereview.appspot.com/5672067/ [16:45] rog, not convinced by that; how about a "checkedClock" (or something) field on the type, as a compromise? [16:46] fwereade: i'm not sure that's right. if the file did not at first exist, you never need to check the clock. [16:46] rog, hm, good point [16:47] rog, Idon;t really like the idea of threading it through all those calls though :/ [16:47] fwereade: i think that the calls in Alive and AliveW are in the unique position of knowing that the clock needs to be checked, hence my suggestion. [16:47] fwereade: it's only two levels... [16:47] * fwereade looks at the code again [16:56] rog, yeah, it turns out ok [16:56] fwereade: cool [16:56] rog, how about calling setAlive setState instead? it now always sets both .alive and .timeout [16:58] * rog has a look [17:01] fwereade: seems reasonable. (although i think it always has set both alive and timeout) [17:02] rog, it went through a brief recent incarnation in which it didn't even bother to read the timeout more than once [17:02] fwereade: ah [17:02] rog, changing it back made the name change seem like a good idea [17:03] fwereade: yeah, i think setState is a better name for what it does [17:04] rog, cheers, reproposed ;) [17:08] rog, EOD I'm afraid; if you have a spare moment I would appreciate it if you would have a little look at https://codereview.appspot.com/5695082 and let me know the perceived crackfulness of the general idea [17:08] fwereade: cool! [17:09] fwereade: yes, i'll do that [17:09] rog, er, expect another one in a sec, I appear to have ballsed up a bit [17:09] fwereade: looks better with firstTime, to my mind - that logic becomes a bit more obvious. [17:09] fwereade: thanks for changing it. [17:11] rog, a pleasure :) [17:12] rog, huh, no I didn't balls it up; somehow I was trying to link against something stale, I guess [17:13] rog, so don't expect an update soon [17:13] fwereade: ok. i'll try to ignore the extraneous diffs [17:14] fwereade: where's the implementation of relation-get etc again? [17:14] rog, in python? [17:14] * fwereade racks brains [17:14] rog, juju.hooks.commands [17:15] fwereade: no, that wasn't the question. [17:15] fwereade: i was wondering what files the new code for " Rough proposal for implementation of relation-get and similar commands." was in [17:15] fwereade: presuming it's somewhere in that CL [17:15] rog, ah sorry: none of them are *implemented* yet [17:15] fwereade: server/protocol looks new [17:16] rog, but it provides a mechanism by which a command like relation-get can get the unit agent (which knows all the appropriate state) to execute the command for it and pipe back the results [17:16] fwereade: yeah, that's what i was looking for. that's in cmd/server, right? [17:17] rog, yeah [17:18] fwereade: instant initial reaction: why not use the rpc package? [17:18] fwereade: rather than doing our own marshalling etc [17:18] rog, huh, because I ballsed up on due diligence :p [17:19] fwereade: i think this code can be really really small [17:19] rog, then that's doubly awesome [17:19] rog, it was very instructive writing it all but less code is always better than more code ;) [17:19] fwereade: the unix socket stuff will stay :-) [17:20] rog, cool [17:22] fwereade: but yeah, i think a few methods exported on a relation type and passed to rpc.Server.Register will probably do the job [17:23] fwereade: the only thing that won't give you will be commands that stream. [17:23] fwereade: are there any like that? [17:23] fwereade, some certainly dump data back on stdout [17:24] er, *rog*, ^^ [17:24] fwereade: lots and lots of data? or don't-care-too-much-if-it's-all-in-memory data? [17:25] rog, and in general I'd expect/hope that --verbose and --debug should work (and thereby potentially generate monster stderrs too) [17:25] rog, not sure how big it could get [17:26] fwereade: i've an idea for how that could work ok with rpc [17:27] rog, well, we could do exactly the same in terms of how we pipe stuff back [17:27] fwereade: yeah [17:27] rog, the streams feel to me like a good thing [17:27] fwereade: if you're around for a minute or so more, i'll paste a little example [17:28] rog, please do [17:31] fwereade: http://paste.ubuntu.com/860798/ [17:31] fwereade: slightly contorted by the necessary form of rpc methods [17:32] fwereade: but the idea is that you call New which gives you a handle on a running command. then you can call other methods to get info on that command, including reading from its output streams [17:32] fwereade: Cmd is the type that would be registered as an rpc handler [17:33] fwereade: maybe Cmd.Result might be better named Cmd.Wait [17:34] fwereade: and if we were concerned about malicious clients, we could put a random capability string in the Running id. === Leseb_ is now known as Leseb [17:38] rog, seems less clear to me but that's probably a matter of familiarity; I shall meditate upon it [17:39] fwereade: i'll quickly sketch an implementation [17:39] fwereade: i think it'll be pretty small. [17:40] rog, don't rush, it has been noticed by others that it's my eod [17:40] fwereade: no probs. [17:40] fwereade: i've only got 20 minutes myself... [17:40] fwereade: see ya tomorrow morning [17:41] rog, and you :) [18:39] niemeyer: hi [18:41] andrewsmedina: yo [18:43] niemeyer: for port this test (http://dpaste.org/9MrTY/) I need create another dummy charm? [18:44] andrewsmedina: You can copy and change an existing one.. some tests already have the needed utilities to do that easily [18:45] andrewsmedina: Hold on.. I'll find an example [18:46] niemeyer: dont have yet in Go a equivalent for change_sample [18:46] andrewsmedina: We actually do [18:46] andrewsmedina: Check charm_test.go [18:47] andrewsmedina: If you have a reader r, you can do something like [18:47] hackYaml := ReadYaml(r) [18:47] hackYaml["subordiate"] = true [18:48] newr := hackYaml.Reader() [18:49] niemeyer: I will try this [18:49] andrewsmedina: It's not as practical as change_sample yet, but feel free to implement a function so that it is [18:49] andrewsmedina: For example: changeSample(func(yh *yamlHacker) { yh["subordinate"] = true }) [18:51] niemeyer: is missing only this use case [19:07] Booom! [19:07] Summer storm.. [22:39] niemeyer: can you help me? [22:39] andrewsmedina: Maybe.. :) What's up? [22:40] niemeyer: All tests are ok [22:40] niemeyer: but [22:42] niemeyer: in python (http://dpaste.org/9MrTY/) when a charm is not a proper subordinate, is returned the charm path in the excetion [22:47] andrewsmedina, and? [22:47] this is problematic because? [22:48] hazmat: I dont have path in go implementation [22:48] andrewsmedina: Sounds fine.. just don't return it [22:49] niemeyer: I return a simple error message? [22:49] andrewsmedina: Simplicity FTW [22:50] niemeyer: I will send the commit in minutes :p [22:50] andrewsmedina: Superb :) [22:50] niemeyer: sorry for my noob questions [22:50] andrewsmedina: Totally fine [22:50] Im a noob in go =/ [22:53] andrewsmedina: Not to worry.. there are a handful of people only that know Go for more than a couple of years, so you're not too far behind :)