[01:33] davecheney: question for you [01:33] wallyworld_: shoot [01:34] on trunk, i try this: juju set-env some-bool-field=true [01:34] it fails [01:34] expected bool, got string("true") [01:34] o_o [01:34] have you seen that? [01:35] i haven't use that command [01:35] certainly never with bool fields [01:35] do we even support them ? [01:35] which charm ? [01:35] there's code there to parse a string to bool, but it appears to not be called at the right place [01:35] this is setting an env config value [01:35] ahh [01:35] i bet nobody ever tried [01:35] cf. the horror show that is the environment config [01:35] and updating it after the fact [01:36] yeah, appears so :-( [01:36] time for a bug report [01:36] or it could be fallout from moving to api [01:36] could be [01:36] the only bool env field I know of is [01:36] use-ssl [01:36] thanks, just wanted to check before raising a bug [01:36] or the use-insecure-ssl [01:36] i think you've got a live one [01:36] there's also development [01:36] ]and a new one i am doing [01:36] provisioner-safe-mode [01:37] which will tell provisioner not to kill unknown instances [01:37] wallyworld_: i think nobody has ever tried to change a boolean env field after deployment [01:37] :-( [01:38] we've only even had that use insecure ssl one and you need that to be set for bootstrapping your openstack env [01:38] ok, ta. bug time then [02:26] sinzui: any word on 1.16.5 / 1.17.0 ? [02:47] wallyworld_, axw: if you're unable to get into garage MaaS, you could probably ask bigjools nicely if you can use his equipment. [02:48] jam: he's been busy supporting site [02:48] and only has small micro servers [02:48] axw: from what I inferred when Nate got access, it was essentially smoser just "ssh-import-id nate.finch" as the shared user on that machine. [02:49] wallyworld_: sure, but I don't think we're testing scaling, just that the backup restore we've put together works [02:49] w/ MaaS [02:49] sure, but we need at least 2 virtual instances, not sure how well that will be handled [02:50] wallyworld_: well, I wasn't suggesting using VMs on top of his MaaS, just using the MaaS [02:51] wallyworld_ you already have access [02:51] yes i do. i was waiting for your on site support effortd to wind down [02:51] consider it down [02:51] you seemed stressed enough, didn;t want to add to it [02:51] hi bigjools [02:51] when the guy you're helping f*cks off mid-help, I consider it done. [02:51] ouch [02:52] :/ [02:52] wallyworld_: you could come round as well if you want direct access [02:52] so i'm currently working on one of the critical bugs [02:52] bigjools: so do you know someone who already has Garage MaaS access to the shared user? From what I can tell the actual way you get added is by adding your ssh key to the "shared" account [02:52] was hoping to get that done before i looked at the restore doc [02:53] "needing to be in the group" seems like a red herring [02:53] jam: I have access, want me to add anyone? [02:53] me and axw :-) [02:53] me please [02:53] bigjools: axw, wallyworld_, and ? [02:53] heh [02:53] lp ids please [02:53] wallyworld [02:53] ~axwalk [02:53] bigjools: I haven't done the other steps, but ~jameinel is probably good for my long term health [02:55] ok you're all in [02:56] I am having a lunch break, if you need me wallyworld_ can just call me [02:56] hooray. thanks bigjools [02:56] np [02:57] axw: the other bit that I've seen, is that you might have a *.mallards.com line in your .ssh/config with your normal user, but you need to still use the other User shared [02:58] if the *.mallards line comes first, it overrides the individual stanza [02:58] jam: I explcitly tried logging in as shared@ [02:58] i can ssh in now [02:58] it works for me now too [02:59] axw: so I think you have to be in the iom-maas to get into loquat.canonical.com, but to get into maas.mallards you just get added to the shared account [03:00] that would seem to be the case [03:00] axw: as in, I'm trying and can't get to loquat [03:00] axw: can you update the wiki? [03:00] jam: right, you need to get IS to do that [03:00] I would get rid of the "host maas.mallards" line in favor of the *.mallards line [03:00] jam: sure - "step 3: ask bigjools to add you to the shared account"? ;) [03:01] axw: ask someone who has access to run "ssh-import-id $LPUSERNAME" [03:01] as the shared user [03:01] axw: Hopefully we can make it a big enough warning for IS people to realize they aren't managing that acccount [03:03] thanks for setting them up bigjools [03:04] * jam is off to take my son to school [03:50] ping -> https://code.launchpad.net/~dave-cheney/goose/001-move-gccgo-specific-code-to-individual/+merge/196643 [03:55] axw: thanks for the review [03:55] now I can close this issue [03:55] nps [04:04] davecheney: you forgot to set a commit message on lp:~dave-cheney/goose/goose [04:04] https://code.launchpad.net/~dave-cheney/goose/goose/+merge/196471 [04:04] I'll put one in there [04:05] it should get picked up in 1 min [04:05] and then you can approve your above branch [04:06] ah [04:06] thanks [04:06] i was wondering whta was going on [04:06] i didn't realise you added the commit message for me [04:15] * axw froths at the mouth a little bit [04:15] wtf is going on with garage maas === philipballew is now known as philip [05:30] axw: isn't maas server supposed to be localhost given Nate's instructions? [05:30] You're generally supposed to be running a KVM (virtual MaaS) system just on one of the nodes [05:31] in Garage Maas [05:31] the main reason we use g-MaaS is because the nodes there have KVM extensions and are set up for it [05:31] in theory you could do it on your personal machine [05:31] jam: maas-server gets inherited by the nodes [05:32] they'll just try to contact whatever you put in there [05:32] (e.g. localhost) [05:32] you need to put in an absolute address [05:33] axw: ah, sure. So 10.* whatever, but not 'localhost' [05:33] yup [05:33] axw: I can imagine that maybe bootstrap works, or some small set of things, but then it doesn't actually work together [05:33] seems like the provider should be able to figure it out itself, but I dunno the specifics [05:34] jam: bootstrap doesn't even work- the node comes up, but the cloud-init script tries to grab tools from localhost [05:35] axw: well "juju bootstrap" pre-synchronous works, right? Just nothing else does :) [05:35] "the command runs and exits cleanly" [05:35] yes :) [06:02] wallyworld_: how's bug #1254729 coming? [06:02] <_mup_> Bug #1254729: Update Juju to make a "safe mode" for the provisioner [06:03] jam: we hit small bug where juju set-env something-boolean={true,false} [06:03] didn't work as expected [06:03] I saw that part, didn't know you were working on it with him [06:03] i think wallyworld_ is in that rabbit hold atm [06:03] jam: been stuck on some stuff inside the provisioner task. i think i've got a handle on it. issues with knowing about dead vs missing machines [06:03] when I saw, [06:03] You could cheat and make it an int [06:03] i mean wallyworld_ [06:04] and when i say we, i mean ian [06:04] yeah me [06:04] :) [06:04] * davecheney ceases to 'help' [06:04] wallyworld_: so you mean we "should kill machines that are marked dead" but not "machines which are missing" ? [06:04] davecheney: thanks for being supportive [06:04] yeah [06:05] sort of [06:05] we have a list of instance ids [06:05] wallyworld_: I'm guessing thats "we asked to shutdown a machine, wait for the agent to indicate it is dead, and then Terminate" it [06:05] and knowing which of those are dead vs missing is the issue, due to how the code is constructed [06:05] but we were detecting that via a mechanism that wasn't distinguishing an instance-id we don't know about from one that we asked to die [06:06] wallyworld_: I don't think you mean "missing", I think you mean "extraneous" [06:06] yeah [06:06] the code was destroying the known instance id too soon [06:06] agent for $INSTANCE-ID is now Dead => kill machine, unknown INSTANCEID => do nothing. [06:26] jam: I've just started a new instance in MAAS manually - shouldn't machine-0 be killing it? [06:26] it's been there for a little while now, still living [06:26] axw: you're using 1.16.2+ ? [06:26] jam: 1.16.3 [06:26] axw: did you start it manually using the same "agent_name" ? [06:27] jam: yeah, I used my juju-provision plugin [06:27] jam: do you know how I can confirm that it's got the same agent_name? [06:27] axw: some form of maascli node list [06:28] axw: it has been a while for me, might want to ask in #maas [06:28] jtv and bigjools should be up around now [06:28] nodes list doesn't seem to show it [06:28] ok [06:28] axw: if nodes list doesn't list it, it sure sounds like it isn't running [06:29] jam: no I mean it doesn't show agent_name [06:29] the node is there in the list [06:30] axw: try "maascli node list agent_name=XXXXX" [06:30] it looks like it isn't rendered, but if supplied it will be used as a filter [06:32] that worked [06:33] jam: the new one does have the same agent_name [06:33] axw: so my understanding is that we only run the Provisioner loop when we try to start a new unit. You might try add-unit or something and see if it tries to kill of the one you added [06:33] ah ok [06:33] thanks [06:38] axw: did it work? [06:38] jam: not exactly; I tried to deploy to an existing machine. it only triggers if a machine is added or removed [06:38] makes sense [06:39] anyway, it was removed [06:39] so I'll go through the rest of the steps now [06:39] axw: so I've heard talk about us polling and noticing these things earlier, but with what ian mentioned it actually makes sense [06:39] the code exists there to kill machines that were in the environment but whose machine agents were terminated [06:40] yup [06:40] and it had the side effect of killing machines it never knew about [06:40] which we decided to go with [06:50] * axw watches paint dry [06:54] axw: ? [06:54] provisioning nodes does not seem to be the quickest thing [06:55] axw: provisioning in vmaas I would think would be reasonably quick, no? [06:55] jam: it's likely the apt-get bit that's slow, but *shrug* [06:56] it's definitely not quick [06:56] I will investigate later [06:58] axw: the fix for that is to smuggle the apt-cache details into your environment [06:58] however when you're on one side of the world [06:58] and the env is on the other [06:58] it's unlikely that there is a good proxy value that will work for both you and your enviornment [07:19] davecheney: garage maas is in Mark S's garage, so I think it would be both reasonably close and have decent bandwidth to the datacenter (I could be completely wrong on that) [07:31] * jam heads to the grocery store for a bit [07:58] mgz, rogpeppe: any updates re agent-fixing scripts? [08:23] fwereade: i've got a script that works, but i don't know whether mgz wanted to use it or not [08:23] fwereade: i phrased it as a standalone program rather than a plugin, but that wouldn't be too hard to change [08:23] rogpeppe, I don't see updates to the procedure doc explaining exactly how to fix the agent and rsyslog configs [08:24] rogpeppe, documenting exactly how to fix is the most important thing [08:24] rogpeppe, scripting comes afterwards [08:24] rogpeppe, sorry if that wasn't clear [08:24] fwereade: ah, ok, i'll paste the shell scripty bits into the doc [08:24] rogpeppe, <3 [08:25] fwereade: I just finished running the process (manually) on garage MAAS [08:25] I keep writing garaage [08:25] anyway [08:25] all seems to be fine [08:25] I missed rsyslog, now that I think of it [08:25] axw, ok, great [08:26] fwereade: sent out an email with the steps I took [08:26] axw, if you can be around for a little bit, would you follow rog's instructions for fixing those please, just for independent verification? [08:26] fwereade: sure thing [08:27] axw, so did the addressupdater code not work? [08:27] fwereade: the what? [08:27] axw, you said you fixed addresses in mongo [08:27] ah, maybe I didn't need to do that bit? [08:27] axw, rogpeppe: addresses should update automatically once we're running [08:27] ok [08:28] rogpeppe, can you confirm? [08:28] fwereade, axw: it seemed to work for me [08:29] rogpeppe: no worries, I was just poking in the database and thought I'd have to update - I'll put a comment in the doc that it was unnecessary [08:30] fwereade: hmm, i realised i fixed up the rsyslog file, but didn't do anything about restarting rsyslog... [08:31] axw, well, technically, we don't know it was unnecessary [08:32] axw, rogpeppe: I am a little bit baffled that the "one approach" notes seem to have been used instead of the main doc [08:32] fwereade: i didn't suggest that [08:32] fwereade: my mistake, I just picked up the wrong thing [08:32] rogpeppe, I know you didn't suggest that bit [08:33] fwereade: i thought dimitern had some notes somewhere, but i haven't seen them [08:33] rogpeppe, they're linked in the main document [08:33] rogpeppe, axw, dimitern: fwiw I have no objection to writing your own notes for things, this is good [08:34] fwereade: just trying to fill in the hand wavy "do X in MAAS" bits :) [08:34] rogpeppe, axw, dimitern: but if they don't filter back into updates to the main doc -- and if they're left lying around without a big link to the canonical one -- we end up with contradictory information smeared around everywhere [08:34] sure [08:35] rogpeppe, axw, dimitern: eg axw trying to use rogpeppe's incorrect mongo syntax [08:36] fwereade: tbh dimitern's isn't quite right either, currently [08:36] dimitern: shall i update it to use $set ? [08:36] rogpeppe, dimitern: fixing your notes is fine if you want [08:36] fwereade: my notes were fixed when you mentioned the problem FWIW [08:37] fwereade: I'll run through the main doc and see if I can spot any problems [08:37] fwereade: it was just a copy/paste failure [08:37] rogpeppe, dimitern, axw: but the artifact we're meant to have *perfect* by now is the main one [08:37] rogpeppe, I don't mind what notes you make, so long as it's 100% clear that they're not meant to be used by anyone else, and they link to the canonical document [08:38] rogpeppe, and I'm pretty sure mramm and I were explicit about using something that understands yaml to read/write yaml files [08:39] rogpeppe, sed, for all its joys, is not aware of the structure of the document;) [08:39] fwereade: does it actually matter in this case? we know what they look like and how they're marshalled, and the procedure leaves everything else unaffected - it's pretty much what you'd do using a text editor [08:40] fwereade: i wanted to use something that didn't need anything new installed on the nodes [08:40] fwereade, sorry, just catching up on emails [08:40] fwereade, yes, the $set syntax should work [08:40] fwereade: and i'm not sure that there's anything yaml-savvy there by default [08:41] rogpeppe, crikey [08:41] rogpeppe, well if that's the case I withdraw my objections [08:42] rogpeppe: pyyaml is required by cloud-init, so it's on there [08:42] rogpeppe, objections backin force [08:42] but... IMHO sed is fine here [08:43] * axw makes everyone hate him at the same time [08:43] * rogpeppe leaves it to someone with less rusty py skills to do the requisite yaml juggling [08:43] * fwereade flings stones indiscriminately [08:43] axw: did you check with anyone in #maas if maas-cli still doesn't support uploading? The post from allenap was from June (could be true, and you could have experienced it first hand) [08:43] rogpeppe, did you hear from mgz at all yesterday? [08:43] jam: the bug is still open, so I didn't [08:43] but [08:44] I couldn't get it to work [08:44] fwereade: briefly - he'd been offline, but i didn't see his stuff [08:45] fwereade: mgz posted his plugin to the review queue [08:46] fwereade: I'll just update the address in mongo back to something crap and make sure the addressupdater does its job; so far the main doc is fine, tho I had to add the quotes into the mongo _id value filters [08:46] axw: as long as its "I tried and couldn't, then I found the bug" I'm happy. vs if it was "I found the bug, so I didn't try" [08:47] jam: defintely the former :) [08:48] axw, thanks for fixing the main doc :) [08:48] np [08:48] axw, and let me know if the address-updating works as expected [08:48] will do [08:49] jam, axw: it doesn't support uploading still [08:50] bigjools: thanks for confirming [08:50] thanks bigjools [08:51] fwereade: confirmed, addressupdater does its job [08:51] sorry for the confusion [08:52] axw: did you have to set LC or LC_ALL when doing mongodump ? [08:53] axw: or is it (possibly) set when you ssh into things [08:53] jam: I did not, but I didn't check if it was there already; I'll check now [08:54] thx [08:54] not set to anything [08:54] dunno why it didn't affect me [08:55] axw: one thought is that you only have to set it if you don't have the current lang pack installed (which a cloud install may not have) ? not really suer [08:58] jam, rogpeppe: hey, I just thought of something [08:58] fwereade: oh yes? [08:58] ? [08:59] jam, rogpeppe: we should probably be setting *all* the unit-local settings revnos to 0 [08:59] fwereade: i thought of something similar yesterday actually, but not so nice [08:59] fwereade: that would be a good thing to do [09:00] rogpeppe, yeah, it was inspired by your comments yesterday, it just took a day for it to filter through [09:00] fwereade: I don't actually know what revnos you are talking about. Mongo txn ids? [09:00] fwereade: that gets you unit settings, but what about join/leave? [09:00] jam: the unit agent stores some state locally [09:00] jam: so that it can be sure to execute the right hooks, even after a restart [09:01] rogpeppe, join/leave should be good, the hook queues reconcile local state against remote [09:01] fwereade: great [09:01] fwereade: do config settings need anything special? [09:01] rogpeppe, config settings should also be fine thanks to the somewhat annoying always-run-config-changed behaviour [09:01] rogpeppe, we have a bug for that [09:02] fwereade: currently we can treat it as a useful feature :-) [09:02] rogpeppe, indeed :) [09:03] axw: when you did your testing, did you start machine-0 before updating the agent address in the various units? [09:03] fwereade: it would be interesting to try to characterise the system behaviour when restoring at various intervals after a backup [09:03] jam: no, I started it last [09:04] fwereade: e.g. when the unit/service was created but is not restored [09:04] jam: sorry, I'll add that step in :) [09:04] jam: actually [09:04] I lie [09:04] I did start it first [09:04] fwereade: i suspect that's another case where we really don't want to randomly kill unknown instances [09:04] axw, dimitern, rogpeppe, mgz, *everyone* -- *please* be *doubly* sure that you test the canonical procedure [09:05] axw: actually we *wanted* to do it last [09:05] rogpeppe, well, there's no way to restore those things at the moment anyway [09:05] jam, why? [09:05] axw: so update machine-0 config, start it, then go around and fix the agent.conf [09:05] fwereade, ok, i'm starting a fresh test with the canonical procedure now [09:05] fwereade: didn't you want to split "fixing up mongo + machine-0" from "fixing up all other agents" ? [09:05] fwereade: agreed, but the user might have important data on those nodes [09:06] jam: yeah that's what I did, sorry [09:06] axw: sorry, "when I say do it last" it was confusing what thing "it" is [09:06] axw: start jujud-machine-0 should come before updating agent.conf [09:06] jam, I think we suffered a communication failure -- you seemed to be suggesting he should fix agent confs before starting the machine 0 agent [09:06] thanks [09:07] fwereade: yes. I think we all agree on what should be done :) [09:07] jam: I fixed mongo, started machine-0, fixed provider-state, fixed agent.conf [09:07] axw: I'm copying some of your maas specific steps into the doc [09:07] cool [09:07] rogpeppe, this is true, hence https://codereview.appspot.com/32710043/ -- would you cast your eyes over that please? [09:08] fwereade: looking [09:08] rogpeppe, there's not much opportunity to fix them, it's true [09:08] fwereade: I used my plugin to provision the new node; how are people expected to do it without it (and get a valid agent_name)? [09:08] rogpeppe, and the rest of the system should anneal so as to effectively freeze them out [09:09] rogpeppe: fwereade: are we actually suggesting run "initctl stop" rather than just "stop foo" ? [09:09] jam, I don't think so [09:09] we do it differently at different points in the file [09:09] fwereade: yeah [09:09] jam, where didinitctlcomefrom? [09:09] "sudo start jujud-machine-0" but [09:09] "for agent in *; do initctl stop juju-$agent" [09:09] fwereade: in the main doc, I think rogpeppe put it [09:09] I'll switch it [09:09] jam: i generally prefer "initctl stop" rather than "stop" as i think it's more obvious, but that's probably just me [09:10] jam: the two forms are exactly equivalent i believe [09:10] rogpeppe, it's just you :) [09:11] rogpeppe, i preferred service stop xyz before, but now i find stop xyz or start xyz pretty useful [09:11] rogpeppe, and i don't think they are quite equivalent [09:11] * rogpeppe thinks it was rather unnecessary for upstart to take control of all those useful verbs [09:11] rogpeppe: honestly, I think they are at least roughly equivalent, but we should be consistent in the doc [09:11] dimitern: no? [09:12] dimitern: [09:12] % file /sbin/stop [09:12] /sbin/stop: symbolic link to `initctl' [09:12] main problem *I* had with "service stop" is I always wanted to type it wrong "service stop mysql" vs "service mysql stop" [09:12] I still am not sure which is correct :) [09:12] rogpeppe, initctl is the same as calling the script in /etc/init.d/xyz {start|stop|etc..} [09:12] jam: initctl stop mysql [09:12] rogpeppe, whereas start/stop and service are provided by upstart [09:13] dimitern: yeah, I confirmed rogpeppe is right that stop is a symlink to initctl [09:13] at least on Precise [09:13] dimitern: i believe that stop is *exactly* equivalent to initctl stop [09:14] dimitern: try man 8 stop [09:14] rogpeppe, hmm.. seems right [09:14] dimitern: (it doesn't even mention the aliases) [09:14] dimitern: that's why i like using initctl, as it's in some sense the canonical form [09:14] rogpeppe: can you double check the main doc again. I reformatted the text, and reformatting regexes is scary :) [09:14] rogpeppe, but again, I usually am too lazy to type more, if I can type less :) [09:14] https://docs.google.com/a/canonical.com/document/d/1c1XpjIoj9ob_06fvvGJz7Jm4qS127Wtwd5vw_Jeyebo/edit# [09:15] dimitern: this is a script :-) [09:15] jam: looking [09:15] rogpeppe: actually, it is a document describing what we want other people to type [09:15] again, it doesn't matter terribly, but we should be consistent [09:15] jam: i don't expect anyone to actually type that [09:16] rogpeppe: that is what this doc *is about* actually [09:16] rogpeppe: right down what the manual steps are to get things working [09:16] and then maybe we'll script it later [09:16] jam: i realise that, but surely anyone that's doing it will copy/paste? [09:16] jam: rather than manually (and probably wrongly) type it all out by hand [09:16] rogpeppe: well, C&P except they have to edit bits, and its actually small, so they'll just type it, and ... [09:16] jam: i wouldn't trust anyone (including myself) to type out that script by hand [09:17] like "8.3" ADDR=<...>" [09:17] they *can't* just C&P [09:17] jam: i deliberately changed it so that the only bit to edit was that bit [09:18] rogpeppe, btw for the copy/paste to work we need to use the correct arguments, like --ssl instead of -ssl ;) [09:18] dimitern: good catch, done [09:23] so... have we stopped the "stay on the hangout" bit of the day ? [09:25] fwereade: I used my plugin to provision the new node; how are people expected to do it without it (and get a valid agent_name)? [09:25] jam, i for one find it a bit distracting tbo [09:25] (just wondering if I should proceed to fix it or not) [09:26] axw: maascli acquire agent_name=XXXXX [09:27] jam: ah :) [09:27] then I shall just let that code sit there for now [09:28] jam: do you think it's worth putting that in the doc? [09:34] axw: well, if you don't mind testing it and finding the exact right syntax, then I'd like it in the doc [09:35] jam: I'll see what I can do before the family gets home [09:36] jam, wallyworld_: reviewed https://codereview.appspot.com/32710043/ [09:36] fwereade: ^ [09:38] rogpeppe: i'll read your comments in detail - the changes i made were what i found i had to do to make the tests pass [09:39] wallyworld_: what was failing? [09:39] otherwise it had issues distinguishing between dead vs extra instances [09:39] a number of provisioner tests [09:39] concerning seeing which instances were stopped [09:40] your proposed code may well work also [09:40] wallyworld_: so that original variable "unknown" didn't actually contain the unknown instances? [09:40] jam: there are other things that StartInstance does for MAAS too, like creating the bridge interface [09:41] wallyworld_: i would very much prefer to change as little logic as possible here [09:41] rogpeppe: it also contained dead ones i think from memory [09:41] cause the dead ones were removed early from machines map [09:41] jam: tho I guess this is moot if they're just doing a bare-metal backup/restore [09:43] axw: so... we should now if this stuff works by going through the checklist we've created. If we really do need something like juju-provision, then we should document it as such. [09:44] jam: the problem is that step 1 is vague as to how to achieve the goal [09:45] axw: so 1.1 in the main doc is about "provision an instance matching the existing as much as possible" [09:45] jam: yeah, how? maybe it's obvious to people seasoned in maas, I don't know [09:46] axw: as in *we need to put it in there* to help people [09:46]