[00:01] adam_g, ok, if i hide my real ~/.ssh directory, i am running into problems. it's not clear to me this is really necessary to do however [00:03] (juju will report ERROR Invalid SSH key repeatedly) [00:04] jimbaker: what i did: cp -r $HOME/.ssh /tmp/new/.ssh ; export HOME=/tmp/new ; juju bootstrap and verified the correct ssh public key was injected to the instance (ssh -i /tmp/new/.ssh/id_rsa.pub ubuntu@host worked) [00:05] jimbaker: after bootstrap, 'juju status' ends up launching an ssh session. $OLD_HOME/.ssh/known_hosts is updated and AFAICS $OLD_HOME/.ssh/id_rsa consulted instead of $HOME [00:05] so yeah, ends in invalid key [00:07] adam_g, ok this looks like i was able to duplicate what you're seeing [00:09] adam_g, i wonder if this is poorly interacting with ssh-agent [00:34] Ugh, handling departed/broken gracefully really is tedious sometimes. :-P [00:35] * SpamapS decides to see if a good solution bubbles up whilst staring into the abyss [00:48] :) [00:48] adam_g, so far all i can determine is that the real .ssh is certainly sticky [00:49] i tried playing around with starting new instances of ssh-agent, but this didn't solve the problem for me [00:59] adam_g, it seems to me that ssh insists on the real user home directory, likely for security reasons. there are certainly options for controlling this. for example, maybe we could use ~/.ssh/config and the Host directive to better control what's going on [00:59] adam_g, another possibility is to allow for juju to use a specific known hosts file per environment. i had a branch that did that, but it was much more involved in terms of ssh key mgmt. maybe if it just did the known_hosts stuff, that could be reasonable [01:19] jimbaker: yeah, i think its a feature of the ssh client more than anything. im able to work around it by tweaking my schroots [02:26] adam_g, sounds reasonable [03:41] I'm dealing with a user question where I'm not sure the problem they're having is really related to the question. Could anyone here shed some light on the connection failure here? http://askubuntu.com/questions/162255/does-juju-really-need-two-nodes [03:42] It looks similar to this one, but lacks the notice about a deadline being exceeded: http://askubuntu.com/questions/141687/juju-error-server-refused-to-accept-client?rq=1 === objectiveous_ is now known as objectiveous [04:10] jcastro, are you still here? [04:11] jtv: those are pretty tough questions :-P [04:11] jtv: #maas might have more helpful people [04:12] Thanks. My problem though is that it's all a bit too far removed from maas. [04:12] jtv: the questions you pasted are maas specific [04:12] Juju bootstrap… it might not even be talking to maas yet for all I know! [04:13] it is [04:13] bootstrap is the only step that talks to maas [04:13] except destroy-environment, which does the inverse :) [04:13] So any idea what connection it is that is failing? [04:14] jtv: if you ssh into that machine, do you see zookeeper running? Also do you see juju processes running? [04:15] I don't have access to any of these machines, but when you say “that machine,” which one do you mean? [04:15] I'm not very familiar with juju. [04:17] jtv: the second question, juju was able to ssh in [04:17] jtv: it just failed on the ZK connection [04:17] have to go afk for a bit.. but you should be able to ssh to that machine [04:17] Juju was able to ssh into the system it created, but not to connect to ZK? [04:18] IQinIT-node01 I think [04:18] It may be helpful to talk about which machine roles are involved. But go afk first! === almaisan-away is now known as al-maisan === TheRealMue is now known as TheMue === zyga is now known as zyga-afk [09:37] morning === zyga-afk is now known as zyga === al-maisan is now known as almaisan-away [11:45] g'monring [11:47] jamespage, just watched your presentation, re q/a unexpected removal of a node is detected, but a new unit or machine is not automatically launched. [11:58] hazmat: re: detailed google storage info, man that kicks absolute arse [11:58] :) [11:58] i already have a google storage account too, just never used it much, might start now :) [12:04] imbrandon, me too [12:24] hazmat: imbrandon: I wonder if adding cloudflare on top of my stuff being served on the google stuff would actually make it perform worse [12:24] after reading hazmat's link it seems they have a bunch of that stuff built in [12:26] jcastro: it does, i actually turned cloudflare off on omg [12:26] jcastro: it keeps the site up if its on rinky-dink hardware but otherwise it throttles as well [12:27] interesting [12:27] jcastro: so its actually slower if you have a robust site [12:28] imbrandon: yeah I was wondering if 2 CDN things would just end up clobbering [12:28] in other words if you dont have to worry about your shit falling over face first if you get slash-dotted cuz you have google storeage backing or juju nginx scaling ,your better off without cloudflare [12:29] I was kind of hoping their static server config would give me automagic spdy but it's just normal http [12:29] yea the 2 cdn dont really hurt, but their throttleing does, it keeps connections alive so they dont fail but at some point arround ~100 connections the are serialized [12:29] ~100 concurent* [12:30] so like connections 100 to 150 ( 50 connections ) sit and wait for the first 100 to finish [12:30] but are kept alive tcp so they dont fail [12:30] etc [12:30] very cool setup but if you got the backing to do better its best not to have it on === almaisan-away is now known as al-maisan [12:48] hazmat, hrm - is it not? OK - I must have seen a different behaviour then [12:51] jamespage, hmm.. i take it back, it will relaunch the machine [12:51] but not as a new unit [12:51] which feels a bit questionable... [12:58] hi everyone. We're interested in using juju as part of an entirely automated process--part of an integration test for a script that configures a machine, fwiw. Juju is nice as an abstraction that can be pointed to a variety of backends for the test. This works very well mostly, because juju ssh and juju scp accept all arguments to their wrapped commands, so we can use -o to pass ssh config values to turn off interactive p [12:58] rompts. However, juju deploy won't let us pass these -o options for when it is trying to connect to the 0 machine initially. Is there a command-line workaround for this (that is, other than modifying your .ssh/config)? Alternatively, if I filed a juju bug for this feature would it have a chance of actually being accepted? [13:00] hazmat, I used this when I had issues starting instances on an openstack deployment [13:01] gary_poster, yes re bug, its inconsistent behavior.. for the other machiens your passing it via unit name identifiers or via machine id? [13:01] anything that went to failed state on build out got deleted by my instance reaper [13:01] and juju just started up a new instance for the unit [13:01] I think [13:01] ... [13:01] I'll have to check again [13:01] hazmat, cool, will file the bug then. we are using machine id [13:02] gary_poster, so its only machine 0 that it has an issue, but you can reference other machines by id and it works? [13:02] hazmat: only on deploy it looks like [13:02] jamespage, yeah.. i just walked through the code it does appear to restart if the instance id is not present in the provider (which filters out some states) [13:02] imbrandon, ? [13:03] hazmat, well, it is only in juju deploy, which under the covers connects via ssh. when you then use juju ssh or juju scp, even on machine 0, it works fine [13:03] he said "juju ssh -o" works "juju deploy -o" does not [13:03] oh.. i misunderstood [13:03] that's expected behavior [13:04] hazmat, so no workaround except modifying .ssh/config, and feature bug would not be interesting? [13:04] gary_poster, sorry that's not a bug [13:04] imbrandon, thanks [13:04] :) [13:04] afk-ish, brb [13:05] hazmat, I didn't expect it was; it was a "hey would this feature/use case (where the use case is full automation of deploy) be something that juju would be willing to support) [13:05] gary_poster, just use juju ssh prior to deploy [13:05] gary_poster, no [13:05] gary_poster, its not something juju would be willing to support [13:05] hazmat, I'm making a mess of this. :-( Oy [13:06] It is juju bootstrap [13:06] that has the problem [13:06] so juju ssh won't help [13:06] gary_poster, even better install a subordinate for your config, ie puppet or saltstack [13:06] gary_poster, i don't really understand what the end goal is here.. "part of an integration test for a script that configures a machine" [13:07] hazmat, we want to be able to fully atomate juju [13:07] okay [13:07] gary_poster, and that means? [13:07] hazmat, rest of my answers sound OK? [13:07] jamespage, definitely, nicely done [13:08] as part of that we need to call bootstrap (and later destroy-environment) as part of the automation. bootstrap requires manual ssh accepting of new key in ec2 case (non-lxc generally I think). [13:08] hazmat, thanks! [13:09] bootstrap cannot currently be automated, therefore, unless you modify .ssh/config [13:09] gary_poster: thats best handled in the .ssh/config [13:09] gary_poster, and the problem with that modifying .ssh/config is? [13:10] gary_poster, we do automated tests of charms on multiple providers across ec2, local, etc via juju automation. [13:11] hazmat, imbrandon, the problem is that we'd rather not modify a user's .ssh/config just to run our tests--especially when you are making a blanket statement like "all aws instances for all purposes will have this behavior" which afaik is the only way to do it [13:11] (then you also have to make those statements for other providers) [13:11] (but that's not a practical issue for us right now) [13:11] gary_poster: if users running it != automated [13:11] unless i missed something here [13:12] editing the ci servers $HOME/.ssh/config is much diffrent [13:12] imbrandon, fair enough. We want the automation to run it, but we want devs to be able to run it too. charm test example: you want automated tests, but you want people to be able to run the tests locally [13:12] gary_poster, so what's automated about this? [13:13] a user is running it.. so they have their own ~/.ssh and their own ~/.juju environments.yaml [13:13] gary_poster: sure let them kick it off manuly from the ci server, thats easy in jenkins [13:13] agreed that ci .ssh/config is fine, which is what we are doing [13:13] gary_poster, so your delivering what to people to run the tests? [13:15] its also unclear why you want the ssh to the bootstrap node on deploy, in this case.. it sounds like a regular environment, and an automation script to launch the test env and run the tests. [13:16] and then possibly a separate front end script for the jenkins runner, that setups the keys and environments.yaml as needed [13:16] hazmat, python callable. Nice easy story is "set up your .juju config with a special environment named foo, configured to run tests whereever makes sense for you". Another nice part of that would be "configure that environment to not ask questions when you bootstrap". A less nice story is "and also modify your .ssh/config to ignore keys from AWS" [13:16] so your saying its okay to modify ~/.juju for a user but not okay to modify ~/.ssh ? [13:16] just have the script set $HOME [13:17] gary_poster: what exactly are you wanting to pass? just suppression of the known_hosts prompt or other bits too? [13:18] hazmat, ok to modify .juju and not .ssh: yes, because in .juju you can constrain everything to a specially named environment. .ssh is blanket for (for instance) AWS. [13:18] because juju should probably just integrate smoser's cloud-init method of verifying host keys [13:19] mgz: yes, just -o 'StrictHostKeyChecking=no' -o 'UserKnownHostsFile=/dev/null' or similar [13:19] gary_poster, this is confusing in terms of the original question.. but the delivery script/callable can set $HOME and setup a private .ssh and .juju as needed [13:20] okay, so there's a better way of doing that, +/- the amazon delay on supplying console output [13:20] which is to acutally verify the key reported by the cloud-init during instance boot and insert it in ~/.ssh/known_hosts [13:21] hazmat, ack. Yes, confusing, I said deploy initially and meant bootstrap and took everything down the wrong rabbit hole, I'm sorry. [13:21] mgz, that sounds very nice [13:21] I use this for fully automated deployment of launchpad on canonistack [13:22] mgz, so console output scraping for the host fingerprint? [13:22] hazmat, ack that $HOME would work. I'll propose it. mgz, that would have to happen in juju though, right? We, as juju clients, can't do that? [13:22] hazmat: yup, works really nicely with openstack, may be less good against aws [13:23] kees also suggested it at uds.. the only issue with that on ec2 is the delay can be a bit atrocious.. [13:23] (The $HOME is not as easy to do as what I proposed, so as a user I'd still prefer the kind of automation mgz suggests) [13:23] so I heard 4 minute intervals on output update? and I guess sshd is generally up and waiting before the first 4 minutes [13:24] well before, often [13:24] mgz, it can take longer, its quite variable [13:24] so adding wait time after boot wouldn't be great [13:24] mgz, re output availability, i've hit 10m b4 [13:25] can poll at 5s intervals with openstack and get nearly live reporting :) [13:26] hazmat: I guess the alternative is a juju config option, to 1) prompt/die as currently 2) accept any host key blndly per gary's args 3) use smoser's trick [13:26] and leave which one up to the user [13:27] the current behaviour when connecting to a new host on an ip you've seen before is obnoxious [13:27] agreed [13:27] mgz, regarding ssh keys. one thing that would be completely possible for juju is what kirkland's cloud-sandbox does. [13:28] it generates ssh public/private key on the local system. passes those through user-data to cloud-init. [13:28] I think I'd be ok with waiting for this use case, fwiw [13:29] connects to the host expecting those credentials, and then destroys them and creates new ones. [13:29] so the ones that were sent insecurely via user-data are usable for less than a minute. [13:30] right, was going to say, putting the private key in user-data isn't ideal [13:30] but its fine if you destroy it right away. [13:30] and for everything but the bootstrap node, they can then post the public keys to the bootstrap node. [13:30] and your client can aske the bootstrap node for the new nodes public keys. [13:30] so you only need the dance once. [13:31] by which I mean, for this use case, I'd be ok with what you'd need to do for mgz's solution: waiting longer than usual for a trustable machine without having user interaction. For other usecases, less so. [13:50] is it possible to tell juju to look at a different environments.yaml file? [13:54] jml, not without $HOME manipulation.. juju does respect JUJU_ENV for named env usage [13:54] hazmat: thanks. [13:54] fwiw, we're experimenting with using juju to fire up our services for integration tests [14:12] how can I tell if a local environment is bootstrapped? [14:13] never mind. destroyed the wrong one :( === zyga is now known as zyga-food === al-maisan is now known as almaisan-away === almaisan-away is now known as al-maisan [14:59] if you deploy a charm as serviceA, and then deploy it as serviceB, does it does the same copy of the charm? [14:59] i.e. if you modify the charm between the two deploys, would the second need -u to use the changes? [15:04] is there a valid charm name regex? === objectiveous_ is now known as objectiveous [15:15] what's the juju equivalent of: [15:16] trace.warning("some message to the user about something breaking") [15:16] trace.log_exception_quietly() [15:17] basically giving a nice clean error without traceback for common failures, but preserves the traceback in log/verbose mode [15:26] can you use jitsu to watch for destroy-service to complete? === zyga-food is now known as zyga === salgado is now known as salgado-lunch === al-maisan is now known as almaisan-away === almaisan-away is now known as al-maisan === al-maisan is now known as almaisan-away [16:22] http://i.imgur.com/x7zO0.jpg , that is all ... [17:12] mgz log.warning, log.debug [17:12] mgz with traceback module formatted exception [17:13] into the debug [17:19] SpamapS: btw https://launchpad.net/~webstack/+archive/php/+packages [17:20] imbrandon: cool [17:21] i still like the idea of a charmers ppa, but thats for now [17:21] 386 built, amd64 still in queue === salgado-lunch is now known as salgado === zyga is now known as zyga-afk [19:59] hrm [19:59] there's something very broken with debug-hooks [19:59] sometimes debug-hooks just sits there, and no more hooks execute [20:00] yea and if you exit go luck ever running it again [20:00] on that environment [20:00] exactly [20:01] hazmat: ^^ debug-hooks seems to have something wrong.. flags not cleared or something [20:09] ugh, and restarting the unit agent made it worse [20:09] now no hooks run [20:09] even when debug-hooks isn't running [20:10] 2012-07-12 13:09:45,743 INFO Waiting for unit to come up. [20:10] ah.. you can't relaly restart the unit agent [20:12] man where is the creat project page [20:12] grrr [20:12] front page [20:12] first thing [20:12] oh the one place i no look [20:12] "Register a project" [20:25] <_mup_> Bug #966617 was filed: supply high level design documents < https://launchpad.net/bugs/966617 > === pdtpatrick_ is now known as pdtpatrick