=== bodie__ is now known as bodie_ [00:34] perrito666: thanks :) glad to hear it [02:00] axw: a minute late, just finishing with tim [02:00] wallyworld: sure, just ping when you're ready [02:00] I'll go make some tea [02:09] axw: bring your tea [03:01] thumper: can you have a look at this please? http://reviews.vapour.ws/r/2366/diff/ [03:02] * thumper looks [03:05] thumper: i've realised that there's yet one more PR required: for the api package [03:05] k [03:06] thumper: even though Juju won't call WatchAllEnvs itself we should probably have it implemented (WatchAll is) [03:06] everything takes too long... [03:06] yeah [03:34] Bug #1339967 changed: juju: cacheAPIInfo uses the wrong tag when storing the environ UUID [03:40] Bug #1339967 opened: juju: cacheAPIInfo uses the wrong tag when storing the environ UUID [03:40] * thumper heads out to drop down two laptops to the repair person [03:46] Bug #1235217 changed: support 1.14 environments without .jenv files [03:46] Bug #1339967 changed: juju: cacheAPIInfo uses the wrong tag when storing the environ UUID [03:46] Bug #1427505 changed: apiserver/networker: test failure due to map ordering [03:52] Bug #1235217 opened: support 1.14 environments without .jenv files [03:52] Bug #1427505 opened: apiserver/networker: test failure due to map ordering [04:04] Bug #1235217 changed: support 1.14 environments without .jenv files [04:04] Bug #1427505 changed: apiserver/networker: test failure due to map ordering [05:44] Bug #1484789 opened: add-machine fails with hosted environments [06:52] TheMue: if you get a chance, could you look at http://reviews.vapour.ws/r/2360/ [07:47] voidspace: morning, will do now [07:48] TheMue: morning [07:48] thanks [08:23] voidspace: you've got a review [08:32] Bug #1484833 opened: juju sync-tools fails to complete as environment is not bootstrapped (catch 22 situation!) [08:48] any takers for a trivial review? http://reviews.vapour.ws/r/2363/ [08:51] voidspace, dooferlad, TheMue, axw, wallyworld ^^ [08:52] dimitern: *click* [08:56] TheMue, ta! [08:59] dimitern: reviewed [08:59] TheMue: thanks for the review [08:59] voidspace: yw [08:59] dimitern: TheMue: dooferlad: be with you in a few minutes [09:00] voidspace, ok [09:00] TheMue, thanks! [09:00] oh, yes, we're starting now, have to fetch my headset [09:11] voidspace, ping [09:11] dimitern: ome [09:11] *omw === mup_ is now known as mup [11:55] voidspace, ping [11:59] voidspace, looking at what was scheduled and agreed to deliver for the MVP, subnet add (rather than create) is expected to land, FWIW [12:00] voidspace, and just space create, and both of subnet|space list - that's it [12:00] (along with constraints ofc) === MerryChristmas is now known as benonsotfware === benonsotfware is now known as benonsoftware === dooferlad_ is now known as dooferlad [12:47] dimitern, voidspace, TheMue: I am back this afternoon. Will take a look at the CI test reviews I have got and the board to see the new plan. Anything urgent that should take priority? [12:48] dooferlad: ah, you're back, fine. how do you feel? [12:50] Bug #1484930 opened: Leader's upgrade-charm hook is not run first [12:52] dooferlad, awesome! [12:53] TheMue: a bit foggy, but mostly fine [12:53] dooferlad, if you don't feel well though, better not though [12:53] dooferlad, nothing is *that* important ;) [12:53] dimitern: +1 [12:54] dimitern: I will stop if I think I need to. I can at least get on with low brain effort tasks. [12:55] dooferlad, only you can know that :) [12:56] dooferlad, I'll leave it to your judgment [12:56] dimitern: sure, but once you have run out of paid sick days, it sure feels like the company wants you back :-| [12:57] Bug #1484930 changed: Leader's upgrade-charm hook is not run first [12:59] dooferlad, it happens sometimes, but I'm sure we've not gone there yet [12:59] dimitern: oh, I have run out. [13:01] dooferlad, don't worry about it for now [13:03] Bug #1484930 opened: Leader's upgrade-charm hook is not run first [13:39] dimitern: why do we want a reference count for state.subnet? We can just perform a simple mgo query to see which spaces are using a subnet (if that is what it is for) [13:42] dooferlad, it's about which subnets have running (etc.) instances [13:42] dimitern: still sounds like a DB query... [13:43] dooferlad: in which case the relationship between instances and subnets needs to be stored somewhere [13:43] dooferlad: in fact that's true anyway [13:45] voidspace, dooferlad, otp, sorry [13:46] voidspace: That is what I thought. Seems a shame to add what seems to be a workaround to something that could easily be managed by our single source of truth. [13:46] dooferlad: well, we don't yet have a single source of truth - that's a big part of that card [13:47] dooferlad: references could actually be "list of machine ids" I guess [13:48] voidspace: that is backwards. Each machine is part of 1+ spaces, so you get the machine document to store the spaces it is in. [13:48] dooferlad: it needs to know subnet, not just space [13:48] voidspace: then you can query spaces that contain subnets, then machines in those spaces [13:48] Actually, you are right, you want to store subnets that a machine is in [13:49] dooferlad: that doesn't tell you which machine is using any specific subnet [13:49] right [13:49] I suggested putting it on the machine record, dimitern wasn't so kee [13:49] then if you move a subnet between spaces you don't need to change anything [13:49] *keen [13:49] he'd have to explain why [13:49] part of it maybe transactional [13:50] dooferlad: our transaction simulation doesn't actually work across collections [13:50] dooferlad: so keeping things like reference count in their own collection avoids some problems [13:51] dooferlad: for IPAddresses we store machineId (and instanceId) on the IPAddress [13:51] http://blog.labix.org/2012/08/22/multi-doc-transactions-for-mongodb - "Operations may span multiple collections" [13:51] dooferlad: operations can, the transaction guarantees aren't the same I believe [13:52] dooferlad: that's what I was told, although that documentation doesn't seem to agree with it :-) [13:52] I am going with the documentation. [13:54] dooferlad: good luck :-) [13:55] dooferlad: there maybe another (better) reason to prefer reference counts in their own collection [13:55] dooferlad: and it may or may not apply to storing subnet <-> machine relationships [13:56] not amazingly helpful I realise... but I know fwereade prefers we do that for reference counts at least [13:56] My problem is this: we have a database. You don't need reference counts if you have the right relations. If you have reference counts you have to work hard to make sure they are right. [13:57] dooferlad: ah, see doc/hacking-state.txt [13:57] dooferlad: the reason for a separate doc is for the sake of watchers [13:57] watchers don't need to be notified when reference count changes and *watching* only works at the whole doc level [13:58] I believe... [13:58] voidspace, dooferlad, instance state != machine life [13:58] dimitern: what does that have to do with it? [13:59] dimitern: quick review of http://reviews.vapour.ws/r/2369/ ? isn't complex [13:59] voidspace, dooferlad, that's why we need a refcount + txn.Ops on it to avoid removing subnets in use [13:59] (even if no instance are yet/anymore running on it) [13:59] dimitern: we're not proposing anything to do with instance state [13:59] dimitern: we're saying that we need to store machine ids using subnets *anyway* [14:00] TheMue, sure, looking in a moment [14:00] dimitern: in which case a subnet is in use if the number of alive machines using it > 0 [14:00] dimitern: so you don't need a reference count, you count alive machines [14:00] dimitern: using the db [14:00] voidspace, you can't store machine ids that don't exist yet though [14:01] dimitern: isn't the machine id created straight away? [14:01] dimitern: it's instance id that is created later [14:01] voidspace, i.e. as a side-effect or provisioning with space constraints [14:02] dimitern: as soon as you provision with contraints a machine will be created, right? and I don't know what you mean by "side-effects" [14:02] dimitern: picking a subnet is part of provisioning - i.e. part of machine record creation [14:02] voidspace, I mean a service's space requirements indirectly should keep its subnets in state [14:03] dimitern: a service constraint for undeployed services won't keep any specific subnets in use [14:03] voidspace, and by requirements I realize constraints are a bad example (except maybe in environment constraints) [14:03] natefinch: standup [14:03] voidspace, but for bindings of a service [14:03] voidspace, once we have them (that was the intent behind --networks) [14:04] dimitern: in which case we add the machine id to all subnets they are "using" [14:04] voidspace, so even if we avoid adding refcounts and use machineIds-to-subnets, we'll need it pretty soon [14:04] I don't think so [14:04] we *still* need to do that, even if we use reference counting - so that we could decrement [14:04] (and not having it will keep the subnet up *until* the dead machine doc is in state) [14:05] (by that I mean - store machine id) [14:05] dimitern: in order to decrement properly you have to store somewhere the machine id referencing subnets [14:05] dimitern: agreed? [14:05] voidspace, ok, first, I absolutely agree we need to be able to link machines to subnets (many-to-many) [14:05] even if it's an "implicit" reference as you describe [14:06] dimitern: and if we have that link then the reference count *must* equal the number of links [14:06] so the reference count is always redundant [14:06] voidspace, and the original and perhaps not great model was to use machine's NICs (which have subnets) for that [14:06] dimitern: ok, that sounds good - we can do that [14:06] any reason not to? [14:06] we have those in state already [14:07] voidspace, nope - refcount shouldn't stop you from removing a subnet referenced only by non-alive machines [14:07] voidspace, the only reason not too (not a good one) is we'll need to update (add) NICs too [14:08] right [14:08] NICs are already added by container address allocation [14:08] yeah, adding NICs on machine creation is non-zero amount of work :-/ [14:08] complexity, which can be avoided by a machineSubnetsC for the time being [14:08] why do we need subnets --> machines? Again, this is a query. [14:09] dooferlad: in order to determine if the subnet is in use [14:09] dooferlad, that's what we're discussing [14:09] ericsnow, I moved our 1x1 to monday so that you can focus on moonstone planing [14:09] dooferlad: so you can tell if it is safe to move the subnet to a different space or remove it altogether (we have a subnet remove command) [14:09] dooferlad, to be able to do that query, you need less data than to ensure it stays like this during an multi-op insert [14:10] dimitern: ah, interesting. *thinks* [14:10] dooferlad: or are you saying "subnets --> machines" is a query. In which case a query of *what*, that information doesn't exist unless we store it. [14:10] alexisb: thanks! [14:11] voidspace: I get that, but it feels like you should store in the machine document, what subnets it is part of. [14:11] dooferlad, yeah, mongo makes trivial design discussions so much spicier :D [14:11] then everything flows from that [14:11] dooferlad, preferably not on the machine document [14:12] well, wherever the machines network information is. I don't have that in my head yet and need to look it up. [14:13] dimitern: why not? (out of interest) [14:13] dooferlad, voidspace, having it there means any change to it potentially triggers a lot of watchers and workers [14:13] I think my argument is that if you have x is a member of y, you should be able to determine that by x storing that it is a member of y. If you store that y has x, y, z as members then if x goes away you need to update that membership list. [14:13] dimitern: but it doesn't change after machine creation [14:13] dooferlad, voidspace, it changes rarely, but always significantly [14:14] dimitern: when does it change? [14:14] dooferlad, voidspace, e.g. when space membership changes or subsequent deployments [14:14] dimitern: space membership of a *machine*? [14:14] dimitern: how is that possible [14:15] dooferlad, voidspace, you deploy another service from a different space on the same machine [14:15] dimitern: and if it's significant *shouldn't* watchers be triggered... [14:15] dimitern: which would require adding an extra NIC, which we won't support initially [14:15] dimitern: I believe [14:15] or can you bind multiple subnets to the default NIC on an EC2 instance? [14:15] dooferlad, voidspace, that brings a new NIC on a subnet in that other space, triggers a lot of address and other changes already [14:16] the *right* thing to do is to use NICs to model this [14:16] so maybe we should do that [14:16] voidspace: you don't need physical NICs to have >1 IP address (or addressable containers wouldn't work). I wouldn't worry about physical interfaces (or anything that looks like them) [14:17] dooferlad: EC2 has a concept of a NIC and different instance types have a maximum number of supported NICs [14:17] voidspace, it's not a change of the machine itself (as juju sees it) - exactly like and why we're not storing what services we deploy to a machine [14:17] dooferlad: and a max number of IP per NIC [14:17] voidspace: yes, I know [14:17] (only indirectly from their units) [14:17] dooferlad: I think only one subnet per NIC, I may be wrong [14:18] dooferlad: I'm not worrying about physical NICs :-) [14:18] dooferlad, I know - that's yet to be decided (the model expects 1-nic-per-space on machines) [14:18] dooferlad: I'm worrying about modelling machines on multiple spaces in EC2 [14:18] voidspace: That EC2 specific logic should be in one tidy place. It is a restriction on what we can do, not an addition. [14:18] with one-nic-multiple-ips some things will be simpler, but impossible on EC2 [14:19] dooferlad: well, sure [14:19] we seem to have sidelined from the main discussion [14:19] we have a model of NICs in state already [14:19] NICs hold the relationship between subnets and machines [14:19] indeed. back to the point... [14:20] so they seem like the right level of abstraction for what we need [14:20] and they conveniently match what we're trying to model anyway... [14:20] sounds good :-) [14:20] dooferlad, voidspace, it's not just EC2 specific, but AWS is an important target [14:21] dimitern: I completely agree. We just need to make sure that EC2 is a subset, not some blob on the side. [14:21] yeah, our model needs to *work* on EC2 - just not necessarily be entirely bound by it's limitations [14:21] and NIC in juju terms is more like a linux device than a Elastic Network Interface [14:22] dooferlad, of course, we started from the free-for-all openstack model and AWS restrictions constrained the model in a few ways, but made it better in others [14:26] so, we have IPAddress. That contains a SubnetId and a MachineId. So, we can find all the IPAddress documents that reference a SubnetId before we delete it and use that as the reference count, right? [14:27] dooferlad: IPAddress is only used for "container address allocation" [14:27] dooferlad: not (yet) as a general IPAddress store [14:27] erk [14:27] well, that is horrid. [14:27] because it's only for containers that we specifically allocate an address (if the feature flag is on) [14:27] we get them automatically for machines [14:28] what do we use for machines? [14:28] so no need to request or model them [14:28] to store them that is [14:28] well, to work out which is the current public/private address for a machine we spit on our finger and stick it in the wind [14:28] which is why we have all sorts of bugs around the private/public address of machines appearing to change... [14:29] machineDoc has two slices of IP addresses. [14:29] (we ask the machine OS and the provider what addresses it thinks it has and report those - holding them on the machineDoc) [14:29] right [14:29] but no CIDRs [14:29] great [14:30] oh, hang on... [14:30] we have a bug we have to fix reqarding machine IP addresses apparently changing [14:30] have to fix *soon* [14:30] so moving to a model that actually works would be better [14:31] the address struct that machine uses could become ipaddressDoc. [14:31] that would make life easier... [14:32] then we use that plus a list of machines not yet provisioned that have been allocated a subnet [14:33] which I assume is generated from another query looking at constraints [14:33] (assuming we can't just look at constraints) [14:53] katco: dang.... somehow my copy of the link failed, can you re-paste? [14:56] dooferlad: dimitern: thx for reviews. the whitebox test does not rely on BaseSuite, so no PatchValue() this time. [14:57] Bug #1484993 opened: TestBootstrapReusesAffinityGroupAndVNet: no tools are available for your environment [15:00] Bug #1484993 changed: TestBootstrapReusesAffinityGroupAndVNet: no tools are available for your environment [15:03] Bug #1484993 opened: TestBootstrapReusesAffinityGroupAndVNet: no tools are available for your environment [15:06] TheMue, you can still call it from juju/testing [15:07] TheMue, at least it does the same thing [15:09] dimitern: will have a look, thought it's a method of an embedded type, which isn't used here [15:09] dimitern: in the moment the patched attempt strategie like you said by PatchAttemptStrategies isn't working [15:09] TheMue, it is a method of BaseSuite via CleanupSuite, which calls PatchValue inside [15:11] dimitern: so I would have to embed the CleanupSuite into the whole suite only for this one patching of a variable I directly have access to [15:11] TheMue, what a moment [15:11] dimitern: will see how the side effects are [15:11] TheMue, s/wait/ [15:11] yep [15:11] TheMue, environSuite (in whitebox test) uses PatchValue already [15:11] the whitebox tests are a bit ... different ;) [15:12] TheMue, the other environSuite (non-whitebox one) directly embeds BaseSuite [15:12] looking provider/maas on master [15:15] so, back, just had phone, sorry [15:16] dimitern: yeah, for the non-wb I've seen, but missed it here [15:19] TheMue, whitebox tests effectively use providerSuite, which already uses PatchValue as it embeds CleanupSuite via Fake(Juju)HomeSuite [15:20] dimitern: yes, deep nested embedding, almost like the "good" old OOP [15:21] TheMue, luckily rogpeppe's godef is amazing for navigating deeply nested types [15:22] dimitern: have to see how I can integrate it into my editor macros [15:23] TheMue, +1 [15:23] right EOW [15:23] happy weekend everyone [15:24] dimitern: hehe, this bad foo == 5, gc.Equals, true is a leftover from some former approaches [15:25] dimitern: PatchValue works fine now [15:28] dimitern: even optimised it using variables for retries and enoughRetries to patch it only once, final problem is now the strategies patching [15:28] TheMue, great! [15:30] TheMue, just use PatchValue for the strategies [15:30] dimitern: yep, thought the same [15:31] TheMue, patching it once and using a retries chan for reporting back [15:33] Bug #1485013 opened: MgoSuite tests must be run with MgoTestPackage [15:33] Bug #1485017 opened: add the ability to open terminal in hook context [15:33] dimitern: hmm, think it's more simple now, will push it so that you can have a final look [15:35] dimitern: so, updated [15:35] TheMue, looking [16:08] dimitern: thx for review, will now merge it [16:09] TheMue, go for it [16:34] dimitern: I added the cards for dummy, maas and ec2. the original one is still there, I grouped them at the end of the lane, so you'll easier find them. I also change the "master card" to a user story, the points are still in there. IMHO these have to be removed due to the points on the individual cards now. [16:35] * TheMue grumbles, has branch cannot land due to a fix blocker *sigh* [16:35] TheMue, you could fix it ;) [16:36] alexisb: hehe, sure, but then I have to fix my wife too [16:36] I assigned it to Ian because it is his PR that caused the regression but really it is up for anyone to fix that is affected by the block [16:36] heh understood [16:36] it is friday evening :) [16:37] alexisb: yep, but sill will have a look [16:43] TheMue, got it, thanks! [16:43] dimitern: yw [18:27] Bug #1485071 opened: juju deploy and add-machine not installing agent [19:38] ericsnow: wwitzel3: sorry, also forgot we have to take care of this bug: https://canonical.leankit.com/Boards/View/114568542/116652767 [19:40] oh right [20:33] ericsnow: what do you want to do about the problem with my untrack code and ids that could conflict if they're for different workload types?