[07:31] morning [07:39] TheMue, heyhey [07:39] fwereade__: hello [07:40] TheMue, just in case you have perspective on this: ISTM that there is nothing preventing a charm from publishing multiple relations with the same name, so long as they have different roles... do you know if this is intentional? [07:41] fwereade__: sorry, dunno. net yet went deeper into relations constraints [07:42] TheMue, no worries :) [07:42] fwereade__: phew ;) [07:43] TheMue, it's just that I saw you at the exact moment I was deciding I didn't know the answer myself :) [07:45] fwereade__: yeah, I would have liked to give you the missing link [08:16] fwereade__, TheMue: mornin' [08:17] wrtp: bonjour [08:49] moin. [09:06] Aram: moin moin [09:16] Aram, wrtp, TheMue: does anyone know why the juju-info relation has global scope? [09:17] bcsaller, actually, if you're around, then you will be sure to know: ^^ [09:19] fwereade__: sorry again, but it seems i have to take a deeper look at relations to get a better understanding [09:57] fwereade__: that does seem a bit odd [09:57] fwereade__: isn't the whole point of juju-info to support subordinates' relationship with their principal? [09:57] wrtp, that had indeed been my conception of it [10:05] wrtp, hmm, I'm suddenly wondering whether we should be occupying the juju-* namespace for hooks as well as relations [10:06] wrtp, otherwise people could implement hooks for the provider side of juju-info [10:06] fwereade__: hmm, interestin [10:06] g [10:06] wrtp, which becomes icky if those hooks suddenly appear during a charm upgrade, with a bunch of hooks already "run" [10:07] fwereade__: i'm not quite sure i understand [10:08] wrtp, it might not be a big deal, I'm not quite sure [10:09] wrtp, I can't figure out whether there's any value in implementing provider-side hooks for juju-info [10:09] fwereade__: i don't think we can occupy the juju-* name space entirely, otherwise you won't be able to implement requirer-side hooks for it [10:09] fwereade__: hmm, i can see that there *might* be [10:10] wrtp, ha, just realised [10:10] fwereade__: although it's perhaps dodgy. [10:10] wrtp, even if there is value it's fundamentally dodgy anyway [10:11] wrtp, if we have multiple relations with the same name then the "what hook do we run" question becomes, I think, insoluble [10:11] fwereade__: why would we have multiple relations with the same name? [10:12] wrtp, we already do... everything implicitly provides juju-info, and some things also require it [10:12] fwereade__: doesn't a principal provide exactly one juju-info relation? [10:12] wrtp, yes, but so does a subordinate [10:12] wrtp, everything provides exactly one [10:12] fwereade__: ah, so you've got the same relation in provider and requirer roles [10:12] wrtp, everything requires one or zero [10:13] wrtp, yeah, essentially, but they're not really the same relation [10:13] fwereade__: yeah sorry, [10:13] fwereade__: i meant the same relation name [10:14] wrtp, cool [10:14] fwereade__: so it makes sense that you can't specify a juju-info hook [10:14] wrtp, no, but yu have to [10:14] wrtp, otherwise the whole relation is worthless [10:14] fwereade__: because if you're requiring a juju-info hook, you'd name the relation something different, no? [10:15] fwereade__: tbh i haven't yet seen a good use for the juju-info relation [10:16] wrtp, I haven't done an exhaustive check but I've seen at least two charms with a require named juju-info [10:16] fwereade__: did you see what they used it for? [10:17] wrtp, here is one one example http://jujucharms.com/~david-duffey/precise/ddclient/hooks/juju-info-relation-joined [10:18] wrtp, here is another: http://jujucharms.com/~ted/precise/application-start/hooks/juju-info-relation-joined [10:21] fwereade__: it's interesting that they both specify scope: container [10:22] fwereade__: i suppose that actually it makes sense to allow non-locally-scoped juju-info relations, because requirers can get local scope if they want [10:23] wrtp, I'm not sure we get anything out of that ability though [10:23] wrtp, I suspect it just misleads [10:23] wrtp, I *think* that what I would like is: [10:23] fwereade__: it provides the ability for one service to watch any other service in a generic way [10:26] wrtp, every subordinate implicitly requires it; every principal implicitly provides it; write all the hooks you want; don't allow cross-role name collisions [10:27] fwereade__: how many of those statements are true today? [10:27] wrtp, just about none :/ [10:27] fwereade__: i'm not sure about the first one [10:27] wrtp, except possibly for write-all-the-hooks-you-want, but it's worthless due to name collisions [10:28] fwereade__: why should every subordinate implicitly require it? [10:28] wrtp, the subordinates don't have to do anything with it, just as principals don't have to do anything with the one they implicitly provide [10:28] fwereade__: i thought *all* services provided a juju-info relation [10:28] fwereade__: it kinda makes sense to me that that might be the case [10:29] wrtp, yeah, I think that subordinates shouldn't [10:29] fwereade__: why not? [10:30] fwereade__: i think it could make sense to think of juju-info as something entirely orthogonal to the subordinate-principal relationship [10:30] wrtp, because we have charms that both provide and require juju-info, and that's obviously crack [10:30] wrtp, that's a nice idea, but I can't see how to make it work [10:31] wrtp, (without changing existing charms...) [10:31] wrtp, ok, alternative that might work [10:31] fwereade__: i'm not sure. if we say that all charms implicitly provide juju-info, and juju-info relation hooks refer to the requirer side, i think that *might* work [10:31] wrtp, everything implicitly provides juju-info unless it explicitly requires juju-info [10:33] wrtp, apart from anything else, surely the fact that we can have unresolvably ambiguous relation specs is enough to sink the idea that name collisions are ok [10:33] fwereade__: one possible way would be to move in a backwardly-incompatible way and prohibit people declaring a relation with the name "juju-info" [10:34] fwereade__: given that very few people seem to use this functionality so far, that might not be so bad [10:34] fwereade__: and then all the problems go away [10:34] wrtp, could do, I guess -- but that feels like a punt on the big issue, which is that we apear to allow relation name collisions in charms [10:35] fwereade__: you can declare several relations with the same name? [10:35] fwereade__: that does seem like crack [10:35] wrtp, yeah [10:35] fwereade__: i wonder if any charms do that [10:35] fwereade__: (other than the ones that declare juju-info, of course) [10:35] wrtp, any with a juju-info relation do [10:35] wrtp, ;p [10:35] wrtp, I hope not :) [10:36] fwereade__: i think it makes sense to reserve all relations with the prefix "juju-" [10:36] fwereade__: or actually... [10:37] fwereade__: yeah, it makes sense to do that; then we can add more in the future without breaking charms. [10:38] fwereade__: but i think people could still write juju-info hooks [10:38] fwereade__: i don't really see why that's a problem yet [10:39] wrtp, they are actually all meant to be reserved, I think :/ [10:39] fwereade__: if we insist that noone can redeclare juju-info, then we don't have any ambiguity [10:39] fwereade__: in which case that ddclient charm is bogus, right? [10:40] fwereade__: (the other one is ok, as it doesn't name the relation "juju-info") [10:41] wrtp, hell, the hook naming totally threw me there [10:41] wrtp, yeah, that sounds sensible to me [10:41] wrtp, I could swear I'd seen code explicitly forbidding explicit juju-* relations [10:42] fwereade__: we don't know that that charm actually works... === Aram2 is now known as Aram [10:46] wrtp, haha :/ I had hoped that charm store entry procedures were a little stricter, but I guess we already know they aren't all perfect [11:13] wrtp, can you think of any reason not to insert the implicit relation in ReadMeta? [11:14] fwereade__: that's a good question [11:14] fwereade__: my gut says it's probably not a great idea, but... [11:15] wrtp, yeah, I too feel discomfort, but I don't think any other place works [11:17] fwereade__: can't it be done in state, when the charm is added? [11:18] fwereade__: i think the charm package should prohibit the declaration of relations with a juju- prefix, but i'm not sure it should actively declare them. [11:19] fwereade__: that seems to me like something that's appropriate for the system implementation to do. [11:19] fwereade__: i'm kind of thinking that at some point in the future, we might see charms being used in different contexts, some of which might have a different set of implicitly declared juju- relations. [11:20] wrtp, you may be right... I guess it's just that I'd prefer that the unit agent be able to determine whether a potential relation is valid by reading its own charm instead of having to hit remote state for it [11:20] wrtp, actually that's no bug deal it's just a method that takes a charm.Charm [11:20] wrtp, yeah, sgtm, cheers [11:20] fwereade__: cool [11:21] fwereade__: gustavo may well think differently, of course! [11:21] wrtp, I understand that caveat to be implicit in everything I hear [11:21] wrtp, or think ;) [11:21] lol [11:24] wrtp, hm, the python explicitly disallows *provides* with a juju-* name or interface [11:24] fwereade__: i think it should disallow requires too [11:24] wrtp, ISTM that we should disallow any juju-* name (or just "juju") [11:24] wrtp, but I'm not sure we have any reason to mess with the interface [11:24] fwereade__: any juju- name [11:25] wrtp, so "juju" should be allowed? [11:25] fwereade__: yeah, why not? [11:26] wrtp, because it is misleading to see a "juju-info-relation-joined" hook [11:27] fwereade__: i'm not sure i see why [11:27] wrtp, I'm almost tempted to say that `starts with "juju"` should be the criterion [11:28] fwereade__: why is juju-info-relation-joined misleading, even if there is a juju relation? [11:28] wrtp, hold on, it's crack anyway [11:28] wrtp, application-start has one of those [11:29] wrtp, but it'll never get run afaict [11:29] fwereade__: why not? [11:29] wrtp, the only explicit relation it has is juju; it may run if it uses its implicit juju-info relation [11:30] fwereade__: if you have a juju-info hook, then it must refer to the provider side, right? (assuming you're not allowed to explicitly declare a juju-info relation) [11:31] fwereade__: so there's no ambiguity and... no problem? [11:35] wrtp: btw, regarding our discussing about " and =, = and == were in 10th edition unix, just catched my eye yesterday :). [11:36] wrtp, guess so... still worried about the inconsistencies on charm upgrade when we add such hooks [11:36] also, the rc manual in 10th edition manual mentions plan 9, I wonder when did plan 9 development started. [11:36] Aram: yeah, i think i used "=" first; then i added "==" (without knowing about the 10th edition version); then i moved to rc and started using - and -- instead [11:36] fwereade__: what kind of inconsistencies are you thinking about? [11:37] fwereade__: it would be easy to disallow juju-info hooks in the first instance, anyway [11:38] Aram: i remember seeing plan 9 around just after i'd left university, around '91. charles had built his own clone of it (called Orbit) before it was made publicly available. [11:39] really? did he make any code available? [11:40] Aram: i might have a pirated copy on magtape somewhere :-) [11:40] Aram: he got the whole comp sci year using it for a year or so [11:50] wrtp, most relations won't even be joined if the charm doesn't explicitly provide them; but the juju-info relation will be processed as usual [11:51] wrtp, if we upgrade to a version that does provide hooks, it won't see the expected events [11:51] wrtp, it will see departs of units that never joined, etc [11:51] fwereade__: i see. [11:52] wrtp, this *might* not be a big deal in practice but it involves dropping guarantees that STM to be useful [11:52] fwereade__: i think that that's probably good enough reason to disallow implementation of juju-* hooks [11:53] fwereade__: it kinda makes sense that the system is providing all interactions with juju- relations. [11:54] fwereade__: one could see it as implicitly declaring the necessary hooks [11:54] fwereade__: so it doesn't make sense for a charm to redefine them [11:55] wrtp, yeah, this sounds reasonable [11:55] wrtp, that then means that we should disallow "juju" as a relation name as well, otherwise we won't be able to write juju-relation-joined hooks [11:56] s/otherwise/because/ [11:56] fwereade__: i'm not sure i follow [11:57] wrtp, disallow implementation of juju-* hooks => juju-relation-joined not ok => cannot meaningfully declare a "juju" relation [11:58] fwereade__: i don't get the first inference. we disallow implementations of hooks for relations with a juju- prefix. but the relation name in juju-relation-joined is "juju" not "juju-" [11:59] wrtp, ah, I thought yu were advocating the simpler "no hook can start with `juju-`" rule [11:59] wrtp, which honestly I would prefer [12:00] fwereade__: i think it's easier to think in terms of relation names [12:00] fwereade__: (HasPrefix(name, "juju-") || name == "juju") is more complex and arbitrary, IMHO, than just the HasPrefix test [12:01] wrtp, I think specifying that no files in charm/hooks can start with "juju-" is the absolute simplest way to get what we need [12:01] fwereade__: and i don't see that we'll ever need to define a "juju" relation. [12:01] wrtp, why should we allow others to? [12:02] fwereade__: because it's a convenient name to use for the requirer side of a juju-info relation? [12:02] wrtp, doesn't this presuppose that we won't have other juju- relations? [12:02] fwereade__: i don't think so. [12:02] wrtp, it's only a good name so long as juju-info is the only juju- one available [12:03] fwereade__: a better name would probably be principal-info, right enough [12:04] fwereade__: tbh i wouldn't mind if we just reserved all juju* names. [12:04] niemeyer, heyhey [12:04] niemeyer: yo! [12:04] wrtp, that's what I'd prefer [12:04] Good morning! [12:08] niemeyer, I was wondering, did you see the prereq of the CL you approved last night? I have a suspicion it may have got caught in the miscommunication [12:08] fwereade__: Let me check [12:08] niemeyer, https://codereview.appspot.com/6678046/ [12:08] niemeyer, you have reviewed it once butthat was a while ago [12:09] fwereade__: Ah, sorry.. I didn't realize it was a pre-req.. I reviewed the other one in an attempt to unblock you, but I has fail [12:09] niemeyer, np at all, I have other things I have been thinking about usefully [12:10] niemeyer, I will definitely want a call about charms and relations at some point [12:10] fwereade__: I'm game at any time [12:10] niemeyer, it's just that I'm getting peckish and will probably disappear for luch soonish ;p [12:10] fwereade__: Sounds like a good plan :-) [12:12] niemeyer, ok, I will be back in a little while, hopefully the various issues will settle into some sort of order in my mind :) [12:14] niemeyer: a bootstrap can "succeed" several times, because the only way it can find out if the environment is already bootstrapped is by reading the environment's storage, which might not be available because of eventual consistency. [12:14] niemeyer: it's possible that moving to using tags might help this issue, but i'm not sure. [12:15] wrtp: We shouldn't allow for that to happen [12:15] wrtp: Otherwise it's ad-hoc [12:16] niemeyer: do you have an idea for how we might prevent it? [12:16] niemeyer: we could put a sleep at the start of Bootstrap, i suppose [12:16] wrtp: We can read the information we just wrote, for example [12:16] wrtp: Before confirming it worked [12:17] fwereade__: how does that help? [12:17] niemeyer: how does that help? [12:18] fwereade__: (didn't mean to address that to you, sorry) [12:18] niemeyer: BTW could we have a chat about TLS certs some time today? [12:21] wrtp: Well, the idea is ensuring data we just wrote is visible [12:21] niemeyer: doing that does not ensure that, unfortunately [12:21] niemeyer: a read can succeed and then fail [12:22] niemeyer: (there's one live test that prints a *sigh* message when that happens, and i see it reasonable often) [12:23] niemeyer: it would probably reduce the frequency of Bootstrap succeeding twice in succession, but i don't think it would stop it entirely. [12:26] wrtp: It's not just about Bootstrap.. this isn't right either: [12:26] r, err = storage.Get(name) [12:26] if err == nil { [12:26] break [12:26] } [12:26] niemeyer: i was just looking at that [12:26] niemeyer: yes, that's wrong. [12:26] niemeyer: storage.Get does the retry anyway, so it's unnecessary for the tests to do that [12:26] wrtp: Eventual consistency sucks to deal with, and we should take the bullet to offer people a sane API [12:26] niemeyer: i agree. [12:27] niemeyer: but in some cases i'm not sure it's possible [12:27] wrtp: If accessing data is *entirely* random, then I can't see why people use S3 [12:27] niemeyer: it's not entirely random. it works... eventually. [12:28] wrtp: Yep, so we should wait until it does [12:28] niemeyer: for the record, i see the *sigh* message relatively often. [12:28] niemeyer: we do! [12:29] niemeyer: but in these cases, the request succeeds, so there's nothing to wait for [12:29] wrtp: Obviously a request succeeding doesn't seem to mean much [12:30] niemeyer: if we delete a file, we can sometimes fetch it even after the delete request. that request is succeeding, but it shouldn't. it will fail eventually though, but the thing doing the Get can't know that it was expected to fail and hence retry. [12:32] wrtp: I understand what eventual consistency means, and I also understand that S3 has it [12:34] niemeyer: i'm not sure how you expect me to fix the problem while removing the loops in jujutest then [12:39] wrtp: I'm thinking as well [12:42] wrtp: So, I think the issue is on loading, rather than on writing [12:43] wrtp: It'd suck to wait a long time for things to show up when we're writing sequentially several charms or whatever [12:43] wrtp: But it seems that in general files we read from S3 are in well known locations, that should be there [12:44] niemeyer: i don't yet see where you're going with this [12:44] wrtp: Bootstrap doesn't tell you that there was a previous bootstrap because it fails to get the previous file [12:45] wrtp: We already have a retry strategy on Get.. why is it not working? [12:45] niemeyer: look at the start of the Bootstrap method [12:45] niemeyer: we fail when the Get *succeeds* [12:46] wrtp: Bootstrap doesn't tell you that there was a previous bootstrap because it fails to get the previous file [12:46] niemeyer: yeah, i was wrong there [12:47] niemeyer: it can only know if there was a previous bootstrap if it can get the file which was created by the previous bootstrap. [12:47] Yep [12:47] niemeyer: actually, Bootstrap should do the loop itself [12:48] niemeyer: because it knows that it's expecting the file not to be there. [12:48] niemeyer: that would erase the jujutest bootstrap loop at any rate, if not the others., [12:52] niemeyer: it means Bootstrap will always take 5 seconds when the environment is already bootstrapped, but that might not be a problem. [12:52] wrtp: Hmm [12:53] wrtp: Why would it do the loop itself? [12:53] wrtp: Get does the loop already [12:53] niemeyer: Get only loops if the file is not found [12:53] wrtp: We just have to improve Get until it works more reliably [12:53] wrtp: Yes, and that's what we need [12:53] niemeyer: in this case the file *is* found, so it won't loop [12:54] niemeyer: and that's a problem for Bootstrap, because it gives an error when the file is found. [12:54] wrtp: If the file is found, Bootstrap can fail [12:54] niemeyer: exactly. [12:54] wrtp: Immediately [12:54] wrtp: I mean it *can* fail [12:54] niemeyer: that's a problem if we do a Destroy followed by a Bootstrap. [12:55] wrtp: Destroy kills the whole bucket [12:55] niemeyer: yes, but we might still be able to fetch that bucket even after it's been killed [12:55] niemeyer: which will cause Bootstrap to fail inappropriately. [12:55] wrtp: Okay, so perhaps we should make Destroy more reliable instead [12:56] wrtp: Because that's the weak link [12:56] niemeyer: so we make Destroy always take 5 seconds? [12:57] wrtp: We can have slightly more deterministic logic by having it retry until it can no longer see the file in a few different tries [12:57] wrtp: That means we don't slow down tests much [12:57] niemeyer: that doesn't necessarily mean that a subsequent fetch won't still succeed [12:57] wrtp: Heh [12:57] wrtp: It won't ever mean that, whatever we do [12:58] wrtp: Even hours later S3 can still pull off a TA-DA! moment [12:58] wrtp: But we can at least try to bring some sanity [12:58] niemeyer: yeah [12:58] niemeyer: but it means we still need the Bootstrap loop if we're to make the tests reliable [12:59] wrtp: I don't see why.. bootstrap uses loadState which uses Get [12:59] niemeyer: ... which can succeed even after the bucket has been destroyed and we've verified that we get a 404 error. [13:00] wrtp: We've just said we'd improve Destroy? [13:00] wrtp: What has to change in Bootstrap? [13:00] niemeyer: that's the "verified that we get a 404 error" bit [13:00] niemeyer: nothing has to change in Bootstrap. i'm talking about the loop in the test that calls Bootstrap. [13:00] niemeyer: (the one that triggered this discussion) [13:00] wrtp: Erm.. [13:01] wrtp: Yes.. I'm a bit lost now.. all we've been saying above is to make things more reliable [13:01] wrtp: If we still have to loop in tests, the whole point is moot [13:01] niemeyer: yes, but it won't be that reliable, even with the change above. [13:01] wrtp: Why? [13:02] niemeyer: because verifying that the Get fails after Destroy doesn't verify that get Get called in Bootstrap won't subsequently succeed. [13:02] s/get Get/the Get/ [13:05] niemeyer: personally, i *think* the loop in the test is the lesser evil. much better would be if we could talk to *something* in amz that was fully consistent. [13:05] wrtp: If the Get sequentially fails several times at Destroy, and then Bootstrap can see the file again, too bad.. let the test blow [13:06] wrtp: Much better is if we don't use S3 at all.. that's where we have to go [13:06] niemeyer: how many times do we try? what if this causes the test to fail 10% of the time? [13:06] wrtp: We improve it until it's 1% or 0.01% [13:06] niemeyer: i'm not entirely sure that avoiding S3 will help here. [13:06] wrtp: Erm? [13:07] wrtp: Everything we've been talking about for the past hour is about S3 [13:07] niemeyer: what's the alternative? tags? surely they'll suffer from a similar problem? [13:07] wrtp: An internal storage [13:07] niemeyer: how does that help with Bootstrap? [13:07] wrtp: True.. [13:07] wrtp: Either way, let's not derail [13:07] wrtp: The test is great.. it's showing the API is flaky.. there's nothing to fix there [13:08] niemeyer: what about the other tests that loop? [13:08] wrtp: Same thing.. we have that shows storage.Get failing, and you say it fails often [13:08] wrtp: Sounds to me like we should improve storage.Get too [13:09] niemeyer: by causing it to fetch several times? [13:09] niemeyer: it does already retry on error [13:09] wrtp: Or perhaps try more often [13:10] wrtp: We should have a live test specifically for it, showing whether it works reasonably well or not [13:10] niemeyer: ISTM that these tests are doing something that people won't be doing much in normal usage - we don't really care if something succeeds when it shouldn't most of the time. [13:10] wrtp: ISTM that we don't know how people will be using it [13:11] wrtp: destroy+bootstrap is not uncommon at all, for example [13:14] niemeyer: i'm not sure what the Get test you're suggesting would do, and how Get might be improved. i don't think we want to retry on Get. nor, probably do we want to slow down every Put or Delete by doing a sequence of Gets after it. [13:15] s/retry on Get/retry when Get succeeds/ [13:16] wrtp: It would verify how reliable Get looks like when we put something [13:17] wrtp: We want to solve the instability you see in the test by fixing the API, rather than by looping in the test [13:18] wrtp: We want to avoid looping in the test, because it shows very unstable behavior in the API itself.. we don't want that to be our high-water mark for all the providers. [13:18] niemeyer: i'd like to fix the API (and we *have* fixed it when things fail), but i'm not sure how we can fix it without slowing everything down to full-eventual-consistency pace. [13:19] wrtp: I've just explained [13:19] niemeyer: i fully agree that it's unfortunate [13:19] niemeyer: ok, so suppose i write the test and it shows that in 40% of cases, a Get succeeds when it should not. [13:19] niemeyer: what then? [13:20] wrtp: Then that's REALLY BAD isn't it!? [13:20] wrtp: If Destroy+Bootstrap fails 40% of the times, we suck I think [13:20] niemeyer: that's S3 for you :-( [13:20] wrtp: Nope.. that's juju developers for you [13:20] wrtp: S3 seems to work fine for a lot of people [13:21] wrtp: storage.Get succeeding is fine [13:21] niemeyer: we can easily make Destroy+Bootstrap fully reliable by making Bootstrap wait for eventual consistency to resolve before returning an error, as Get does. [13:21] wrtp: It's the other case that is a lot more rare: there are very few spots where we care about content being *deleted* [13:22] niemeyer: Get("nonexistent-thing") will currently always take 5 seconds. [13:22] wrtp: I don't see how that's relevant [13:22] niemeyer: i agree. but that's what these tests are showing. [13:22] wrtp: Bootstrap uses loadState, which uses Get.. Bootstrap is fine as it is [13:23] s/showing/testing/ [13:24] niemeyer: our current technique for avoiding eventual consistency issues is to try to avoid errors. [13:24] niemeyer: so, if we're about to return an error, we try a few times to make sure that we've *really* got an error. [13:24] niemeyer: ISTM that Bootstrap fits into that category. [13:25] wrtp: Sorry.. can we get to actual use cases so we can start to funnel the conversation towards agreement? [13:25] wrtp: We're talking for more than an hour, so it's time to reach some conclusions [13:26] wrtp: Destroy+Bootstrap fails and we want it to work.. [13:27] wrtp: We can make that more reliable by having Destroy wait until the file at least looks gone [13:27] niemeyer: my suggestion is to make Bootstrap not fail until the bootstrap bucket definitely isn't disappearing [13:28] wrtp: I don't know what that means [13:28] wrtp: Probably because thre are three negatives [13:28] niemeyer: your suggestion is *more* reliable, but it still isn't reliable. i believe my solution is reliable (well, as reliable as our Get heuristics) [13:29] niemeyer: this is what i suggest for the start of the Bootstrap method :http://paste.ubuntu.com/1285011/ [13:30] wrtp: We're almost in agreement [13:30] niemeyer: the down side is that a failed Bootstrap gets slower. but that's what happens with a failed Get too, so comparable cases, i think. [13:32] wrtp: The difference is that you're doing the verification that should be dong in Destroy within Bootstrap [13:33] wrtp: That's effectively what this is doing.. it's waiting until Destroy actually takes place.. we can do that in Destroy itself [13:34] niemeyer: i'm not sure we can do that and ensure that the subsequent Bootstrap will succeed, without waiting for the full eventual consistency timeout. [13:35] niemeyer: and perhaps that's actually best - we'd just delay destroy-environment for a while. [13:36] wrtp: That's true [13:36] wrtp: Okay, sounds good [13:36] niemeyer: so we add a sleep to Destroy? [13:37] wrtp: What? :) [13:37] niemeyer: or... what sounds good to you? [13:37] wrtp: Your proposal.. [13:37] niemeyer: ok, cool. [13:38] wrtp: I think storage.Get likely deserves some improvement too to sort out that issue you see frequently [13:38] niemeyer: i'll still need to leave the loops in some of the other jujutest tests, i think. [13:38] wrtp: I'd prefer to nail down problems rather than looping [13:38] wrtp: For all the reasons we already covered [13:38] wrtp: We're nailing one of them [13:38] niemeyer: i only see it frequently because the test is checking Get after Delete, which i don't think is something that we care too much about in real-world code. [13:39] wrtp: Let's solve that first one, and then we see the others [13:39] wrtp: So let's not do that [13:39] niemeyer: remove that test? [13:39] wrtp: Well, I suppose this is testing Delete? [13:39] niemeyer: yes [13:40] niemeyer: there's also one in the test that tests List, i think. [13:41] wrtp: List is a tough one, because we don't know what we're waiting for.. [13:41] niemeyer: in fact it's all in TestFile [13:41] niemeyer: yup [13:43] wrtp: I guess those loops for the storage method itself are fine, at least for now [13:43] wrtp: It's the environment interaction we care the most about [13:43] niemeyer: yeah [13:43] niemeyer: i'm happy to have Bootstrap work a little more reliably. [13:44] wrtp: Me too [13:44] wrtp: I think storage.Get may need some tweaking too [13:45] wrtp: But we'll see [13:45] niemeyer: what kind of tweak are you thinking of? [13:45] wrtp: Perhaps just reducing the time between retries [13:45] wrtp: So we don't increase the overall time further but improve the method a tad [13:46] niemeyer: i'm not sure that helps here. we're talking about the case where it doesn't retry at all, because the Get succeeds. [13:47] niemeyer: we could retry even when the fetch succeeds, but i *think* that's unnecessary. [13:47] wrtp: I was considering something else, but it doesn't really matter right now.. sorry for the noise. [13:47] niemeyer: np [13:48] niemeyer: i'm sorry i'm bad at explaining things! [13:49] wrtp: I don't think you're bad at explaining things. [13:51] niemeyer: it felt like i wasn't explaining things well above, but we got there! [13:53] wrtp: I don't think it was a problem in the explanation.. [13:54] niemeyer: nothing has to change in Bootstrap. i'm talking about the loop in the test that calls Bootstrap. [13:54] wrtp: That was 1h ago.. [13:54] wrtp: It took 1h for us to agree to solve the actual problem without looping in tests. [13:56] that was in response to talking about making Destroy try the get, i think [13:56] niemeyer, take a look at http://paste.ubuntu.com/1285056/ -- I think it roughly covers the areas I'm thinking about [13:58]