[02:34] wallyworld: ping [02:34] wot [02:38] wallyworld: hangout? [02:38] ok [02:38] https://plus.google.com/hangouts/_/72cpj771u1klaslrripsbvshss?hl=en [02:39] wallyworld: ?^^ [03:33] * thumper is trying to work out the least shitty way to do something [03:33] there doesn't seem to be a clean way to get the machine agent data dir from the api server client [03:33] since this is needed to get the ssh identity file location... [03:33] I could pass the *agent.Config through about 4 layers [03:34] or... [03:34] oh, we [03:34] ew [03:34] I could use a global :-) [03:34] axw, wallyworld, davecheney: opinions? [03:35] noooo. not a global [03:35] pass the agent config pointer? [03:35] could we attach it as a stuct attr somewhere/ [03:35] ? [03:35] it'll need to go machine agent -> api server -> root -> api [03:36] wallyworld: it is the three or four somewheres... [03:36] i know there are other places where agent.Config is passed around [03:36] so doing it that way at least is consistent [03:36] yeah... [03:36] maybe see how it is done elsewhere? [03:37] wallyworld: it is passed into the worker [03:37] but it is normally only one level deep [03:38] the api server is somewhat nested [03:38] ah ok. i couldn't recall how it was used [03:38] what about adding it to the state struct of whatever [03:38] thumper: I wonder if the key should end up getting plonked in the client key dir that I'm introducing, which will automatically get picked up by utils/ssh [03:39] not sure, just hand waving [03:39] axw: but the key is on the state server, not the local machine [03:39] axw: two very different ssh bits [03:40] thumper: the client key dir can be changed. it doesn't have to be ~/.juju/ssh [03:40] just a thought, not necessarily a good one [03:40] axw: yeah, but I don't know which directory it is [03:40] that is the problem :) [03:40] thumper: at startup? [03:41] axw: well the machine agent knows, but that information isn't passed down to the api server client yet [03:41] just trying to work out how to get it down there... [03:41] I think just adding the param is a good start [03:41] better to store a single pointer to an already created structure [03:41] than to pass down a string [03:41] right, but the machine agent can initialise the client key dir to be whatever it knows about, and then when the api server uses utils/ssh, it'll get picked up. maybe I don't understand correctly [03:42] ah... wat? how does utils/ssh pick up the client key dir? [03:43] axw: currently we write the generated private key into $datadir/system-identity [03:43] thumper: it's a global, unique per process thing [03:44] wallyworld: there ya go, use a global :) [03:44] heh [03:44] :-) [03:45] normally I wouldn't advocate, but in this case it seemed cleanest. hasn't been reviewed yet though :) [05:20] * thumper avoids the global [05:20] and dances through the apiserver weirdness [05:46] * thumper throws in the towel for today [05:46] I'm pretty sure I have a way forward now... [05:46] at least [05:46] night all === OutOfControl is now known as benonsoftware === rvba` is now known as rvba [08:08] jam: i'm about to head out tonight to a function, so will miss the stand up. i've got about 4 pipes in progress, but none ready to propose yet [08:08] wallyworld_: k, anything you want to summarize for the group? [08:09] i've been adding a bunch of instrafstructure for the charm update work, api service pieces, and splitting it all up, plus refoactoring further the charm store interface and test mock [08:10] jam: also,i've had no feedback from martin on that bug and mp we discussed, i'm keen to hear something from hiom about that [08:12] * wallyworld_ runs away === gnuoy` is now known as gnuoy [08:13] mornin' all [09:42] I may be a few mins late for the standup, but I should be around for it [10:41] rogpeppe: I think you asked about the fireworks in Dubai: http://www.youtube.com/watch?v=Oo1JwImshw8&list=WLD47DA262F88F1497 [10:41] Its a better view than I had, since a lot of it is from above so you get to see the palm shape [10:43] jam: thanks [10:43] jam: that's quite a lot of fireworks... [10:43] indeed [10:43] where we were, we only saw the primary palm island on edge. [10:43] so a couple low shots is what it looked like to uss [11:09] welcome natefinch, how did you get here in the end ? [11:11] rogpeppe: do you know if that was 1.16.3 against a 1.17 server, or a 1.17 against a 1.16.3 server? [11:11] jam: ip address... something must be wonky with my DNS [11:11] jam: the latter [11:11] It doesn't reproduce for me just yet, but it looks like my /usr/bin/juju got updated over the vacaiton [11:14] sudo apt-get install juju-core=1.16.3-0ubuntu0.13.10.1~ctools0 [11:14] and now 'juju status' using 1.17 looks as described [11:15] mgz: http://paste.ubuntu.com/6708528/ [11:15] mgz: ^^ [11:17] jam: ta [11:23] mgz: but I also see it after doing "juju upgrade-juju -e local --upload-tools" with juju 1.16.5 [11:23] okay, so it's probably not fixed for them [11:24] mgz: well "its fixed" was that they reverted to the 1.16.5 client, I bet [11:24] mgz: but you're going to file the bug, right, or shall I do it and assign it to you ? [11:25] its certainly considered a critical bug that juju status in the 1.17 series is incompatible with the 1.16 series, just like add-machine, etc. [11:25] I'm currently finishing add-machine [11:25] we removed a struct I was using :( [11:34] jam: if you could file it, I will assign to me [11:39] mgz: https://bugs.launchpad.net/juju-core/+bug/1266734 [11:39] <_mup_> Bug #1266734: juju status incompatible in 1.17.0 against 1.16

[11:39] that's a bug in _mup_, Triaged by gz [11:39] I assigned it to you, and it is triaged, but it certainly isn't triaged *by* you :) [11:39] _mup_: make me a sandwich [11:40] ah, mup doesn't respond to pokes anymore :( [11:42] :( [11:57] axw: I'm getting "environment is no longer alive" trying to add a machine to a 1.16 system. [11:57] Did we mess up the default value for environment life? [11:58] mgz: ^^ that *might* be a hint as to what is going wrong? If juju 1.17 is saying "oh this env is dying, I won't probe any deeper", or something like that [12:27] jam: I had tested when there was no life in the document... but it does sound like it points to that [12:35] axw: it is the assertAliveOp() [12:35] after debugging some more [12:35] axw: Life() still returns Alive [12:35] because 0 is the default value [12:35] which is the enum Alive [12:35] but assertAliveOp() asserts that "life == 0" when it "== nil)" [12:35] ha [12:36] jam: yeah, I thought it was working in the db level too. sounds like I messed up my testing [12:36] sorry [12:36] mgz: sounds like unrelated to status [12:36] yeah, that's what I was coming to with adding logging in status here [12:36] axw: well, I created a bug and assigned it to you already :) [12:37] thanks, will get onto it in the morning [12:37] https://bugs.launchpad.net/juju-core/+bug/1266748 [12:37] <_mup_> Bug #1266748: Environments.Life() in juju-1.17.0 incompatible with juju-1.16.*

[12:37] everyone can triage it! [12:37] mgz: more importantly *I* can make anyone triage it :) [12:38] I'll try a hack fix and see if that make status happy [12:40] axw: ugh, and your latest change to auth keys also conflicts with the changes to add machine... sigh [12:40] guess I'll look more tomorrow [12:41] I refactored that code so it operated in a "poll for information, then set that information" [12:41] rather than interleaving it [12:41] (so that when we go to set it via the API we can fallback to directly setting it in the DB) [12:41] axw: note, what is your compatibility with 1.16 here with the ssh keys stuff? [12:43] jam: I thought it was polling first to be honest - there were some late changes when merging with rogpeppe's updates, maybe it got broken around then. will have to investigate some more later [12:43] umm [12:43] jam: are you referring to the CL I put up today? [12:43] axw: you just landed code today that breaks my patch for bug #1253631 [12:43] <_mup_> Bug #1253631: juju add-machine in trunk is incompatible with 1.16

[12:44] when I mean "poll", I just mean, gather up the information we need to send in the add-machine request and do all the gathering before we start talking to the db. [12:44] yeah, I thought that was being done [12:45] * axw looks at what he merged today [12:45] axw: I created a new function "gatherMachineParams" that did all of the NewUUID, ParseIP, hostAddresses, checkProvisioned, etc [12:45] and Arch, and etc [12:46] it was a bit interleaved in the function before [12:46] anyway, I can sort it out [12:46] jam: ah, sorry [12:46] just one more time that this stuff got broken (ian did it last time by getting rid of State.AddMachine) [12:46] jam: as for 1.16 compat, the stuff I've done is all client side so far [12:46] axw: I just need to get this landed so it is on everyone else to keep it working :) [12:47] axw: sure, but changing where SSH keys are found is going to break if someone uses 1.17 and then 1.16, right? [12:48] axw: InitUbuntuUser sounds a bit "mutating state" while we are "gathering information" (might be necessary, I'm not sure exactly yet) [12:48] jam: sorry, I don't understand. if they use 1.17 and then 1.16...? [12:48] axw: from what I can tell, if I use 1.17 client in a 1.16 environment, and do "juju add-machine" it is going to set up that machine in a way that 1.16 may not understand [12:50] jam: all that InitUbuntuUser does is ensures that ubuntu exists on the machine, and if not creates it and makes it a passwordless sudoer [12:51] so that's not going to break anything older - this is only done at add-machine/bootstrap time anyway [12:51] the CL I put up today is about using additional keys from the client side; the additional public keys are added to authorized-keys in the env config [12:52] so no changes needed on the server side there either === gary_poster|away is now known as gary_poster [13:43] jam: I realised what happened with my testing. The assertion used to be !dead, then it got changed during review [13:44] and I didn't retest [14:37] natefinch: i've been wondering about IP address changes with respect to replica sets [14:37] natefinch: do you know how we can deal with that? [14:37] natefinch: (at the moment i don't think it's possible to change the IP address of a member, right?) [14:43] natefinch: ping? [14:46] rogpeppe: here now [14:46] natefinch: what do you think of the above? [14:46] rogpeppe: I don't think you can change the ip address, but you can remove and re-add [14:46] natefinch: you *should* be able to change the ip address, at least according to this: https://groups.google.com/forum/#!topic/mongodb-user/_EUC63MJfQA [14:47] natefinch: unless i'm misinterpreting === hatch_ is now known as hatch [14:48] rogpeppe: sorry, stupid USB ethernet adapter keeps making my computer hang up. Didn't see anything after what I posted [14:49] [14:46:58] natefinch: you *should* be able to change the ip address, at least according to this: https://groups.google.com/forum/#!topic/mongodb-user/_EUC63MJfQA [14:49] [14:47:07] natefinch: unless i'm misinterpreting [14:52] rogpeppe: you're right. I thought I had seen somewhere that you oculdn't change the host on a member, but maybe I'm thinking of something else. Looks like you can: http://docs.mongodb.org/manual/tutorial/change-hostnames-in-a-replica-set/ [14:53] nate_finch: i think that means, unfortunately, that our current strategy of keying by address is not valid [14:54] nate_finch: jam also brought up the point that it would probably be better if we didn't denormalise the replica set voting info [14:54] nate_finch: which means, i think, that we want to be able to map from replicaset Members to *state.Machine [14:55] nate_finch: i think we can probably use the tagging functionality for that [14:55] nate_finch: we'd associate a tag holding the machine id with the replica set member [14:57] rogpeppe: so, there is a problem with replicaset.Set, if you modify the address of a member, yes. I thought I had mostly copied that function from the javascript code, but maybe there's a subtle difference. I can double check. [14:58] rogpeppe: I'm not sure what you mean by denormalizing the voting info [14:58] nate_finch: do the ids have to be strictly numbered from zero? [14:59] rogpeppe: I don't believe so, I was just following the way the javascript code works. [15:00] rogpeppe: so you want the id to be the same as the machine id? [15:00] nate_finch: i'm not sure if that's a good idea [15:00] nate_finch: but we could provide access to the Tags functionality [15:01] nate_finch: and state something like: if the id is greater than zero, it is treated as explicitly set [15:01] nate_finch: otherwise it will be automatically generated [15:03] nate_finch: then a client of the package can inspect the tags and determine which members match, according to the tags, and when updating, it can reuse the original indexes for previously existing members [15:03] nate_finch: does that make some kind of sense? [15:04] rogpeppe: what are we gaining? Just being able to figure out what replica member corresponds to what machine, if its address changes? [15:04] nate_finch: it means we can use the replicaset member list as the canonical source of what machines are voting [15:05] nate_finch: rather than needing to trust that state reflects that correctly [15:07] rogpeppe: definitely using the information in the replicaset member list as the canonical source of who is voting is the way to go, since it *is* the canonical source :) However, I'm trying to figure out when the host name in the member list would be wrong, and how this would help in that situation. obviously if the hostname is wrong in mongo, then that member isn't voting anyway, right? [15:09] nate_finch: consider what will happen if a machine's address changes. the address updater will change the machine's address in state, and we will need to change the appropriate replica set member to reflect the new address [15:09] nate_finch: that member can't vote while it has the wrong address, but it's still a valid member (it's analagous to a network error cutting off that machine, i think) [15:09] rogpeppe: can't that just be another thing the address updater does, change it in the replica config? [15:10] nate_finch: i don't want two things both changing the replica config [15:10] nate_finch: seems like a recipe for race conditions [15:12] rogpeppe: hmm.. true. I'm just not a fan of having things out of sync like that. [15:13] nate_finch: well, it's just one part of the address propagation [15:13] nate_finch: the machine's address was out of date for a while too [15:14] rogpeppe: I'm not averse to having the machine id be the replica id. That's not a terrible solution, though maybe tags are more flexible in case of weird edge cases [15:14] nate_finch: i don't want to tie us to having integer machine ids [15:15] rogpeppe: ahh, as opposed to random strings? [15:15] "random" = arbitrary === nate_finch is now known as natefinch [15:15] nate_finch: yeah [15:15] rogpeppe: good point. [15:18] rogpeppe: I don't know that tags is the right solution. Couldn't we just add a value to the machine document in state to hold replica set member id? [15:20] natefinch: using tags seems like a direct reflection of what we want to do [15:20] natefinch: the other way around isn't so clearly correct to me [15:21] natefinch: because the replica set must be updated independently from the machine doc [15:21] natefinch: so there's a potential for things to get out of step [15:21] natefinch: what are your reservations about using tags here? [15:22] rogpeppe: tags are human-readable information used for selecting machines. I don't think replica set id is really anything anyone will ever want to see in tags. [15:23] natefinch: i don't think it's incompatible with that [15:24] natefinch: { "juju-machine-id": "12" } [15:25] natefinch: they don't look like they're for human-readable info to me, BTW [15:25] natefinch: mostly it looks like they're used for write- and read-concern configuration [15:25] natefinch: http://docs.mongodb.org/manual/tutorial/configure-replica-set-tag-sets/ [15:26] rogpeppe: crap, we were talking about different things. I thought you meant tags in state on the machine docs [15:26] rogpeppe: yes, that sounds fine [15:26] natefinch: ah, cool [15:27] natefinch: sorry for the misunderstanding [15:27] natefinch: i'd forgotten state.Machine had tags :-) [15:27] rogpeppe: haha no big deal [15:30] fwereade, we should probably have a chat this week about the next stage of user ids in juju [15:40] natefinch: would you be able to move forward with that? [15:43] natefinch: or i could if you're otherwise busy [15:46] rogpeppe: I can do that. [15:47] natefinch: thanks! [15:48] natefinch: just to be clear, i'm thinking of this: [15:48] // Set changes the current set of replica set members. Members will have their [15:48] // ids set automatically if they are zero. [15:48] abentley: and a change to a Tags field to Member [15:48] oops [15:48] s/abentley/natefinch/ [15:49] rogpeppe: That's good, 'cause I had no clue... [15:49] abentley: :-) [15:50] rogpeppe: yep, sounds good [15:51] natefinch: and the automatic ids we'd choose would start from 1, not 0 [15:51] rogpeppe: right [15:58] mattyw, for sure [15:58] mattyw, I'm sprinting today and flying tomorrow [15:59] fwereade, no problem, shall I put something in the diary for thursday or friday or would you prefer to wait till next week? [15:59] mattyw, if I come online tomorrow pm before you go, grab me -- otherwise it may have to be thursday [15:59] mattyw, let's do thursday, it's a meetingy day for me anyway [15:59] fwereade, ok cool - I'll put something in the diary [15:59] fwereade, thanks very much [16:00] mattyw, a pleasure, sorry it can't be sooner [16:00] fwereade, not at all - I have plenty to do in the meantime [16:00] mattyw, cool [16:05] * rogpeppe grabs a late lunch === dames is now known as thedac [17:31] natefinch: important thing i've just noticed, (from http://docs.mongodb.org/manual/tutorial/configure-replica-set-tag-sets/) the type of the Tags field should be map[string]string [17:42] natefinch: do you have a strong idea as to what we should do if we find extra members in the peer set that don't correspond to machines? [18:05] rogpeppe: yeah, I was making the Tags map[string]string . Not sure what to do if we find extra members. Honestly, I think if we just ignore that condition it'll basically take care of itself, which is to say, normally we'll just ignore them, if the normal process is [18:05] natefinch: it's not quite that straightforward [18:06] natefinch: for the time being i'm just going to return an error if there extra members [18:06] natefinch: i've added a TODO comment with various options [18:06] rogpeppe: that's probably fine... it shouldn't really ever happen unless people go messing with the replicaset by hand, in which case, not mucking with it is probably the right thign to do [18:07] yeah [18:07] natefinch: BTW i just wanted the replica set id from the CurrentStatus results - it's not there and not documented, but it appears in the actual results [18:08] natefinch: do you think it's reasonable to add it as a field in the Status type? [18:09] rogpeppe: sure, no harm [18:09] natefinch: one mo, i'll just check that it is the expected id [18:10] rogpeppe: probably we should just export a ReplicaSetStatus struct that includes the data from replSetGetStatus [18:11] natefinch: isn't that what Status is? [18:11] rogpeppe: status just returns the members right now [18:11] rogpeppe: the member statuses that is [18:11] natefinch: ah yes [18:12] rogpeppe: brb diaper [18:12] natefinch: i don't care much at the moment [18:12] (not mine ;) [18:18] rogpeppe: back [18:18] natefinch: confirmed that _id is the replica set id [18:19] natefinch: i thought it was strange that it wasn't there, otherwise there's no way of cross-correlating statuses with members [18:20] natefinch: unless you assume that there's a one-to-one correspondence in the arrays, but that's not documented either [18:23] rogpeppe: you mean between the members and their statuses in replSetGetStatus? I presume there's a 1:1 relationship. I'd be surprised if there were not [18:24] natefinch: i would too, but i can't find anything that documents that fact [18:24] natefinch: and the fact that we can't fetch them both at the same time makes me naturally wary [18:24] natefinch: so given that the _id field is provided, i'd rather latch onto that [18:25] rogpeppe: that's fine === _bjf is now known as bjf [19:01] * rogpeppe is done for the day [19:01] g'night all [19:02] rogpeppe: g'night === hatch_ is now known as hatch [19:53] morning folks [20:44] o/ natefinch [20:45] natefinch: care for a catch up call? [20:53] thumper: sorry, just got surprise acquisition of the kids for a bit. When my wife gets back, I'll be able to catch up, but kind of occupied at the moment, unfortunately [20:53] natefinch: np [21:24] thumper: still want to catch up? [21:24] natefinch: yeah [21:25] natefinch: https://plus.google.com/hangouts/_/7acpicc4tp7juqvoctqj0i9qb4?hl=en [21:49] natefinch: you have frozen [21:50] * thumper goes to tell the dog to be quiet... === gary_poster is now known as gary_poster|away [23:35] decisions... decisions... [23:35] do I hook up my crazy shit and try it [23:35] or write more tests first [23:35] ... [23:59] davecheney: you around?