[00:01] <jimbaker> adam_g, ok, if i hide my real ~/.ssh directory, i am running into problems. it's not clear to me this is really necessary to do however
[00:03] <jimbaker> (juju will report ERROR Invalid SSH key repeatedly)
[00:04] <adam_g> jimbaker: what i did: cp -r $HOME/.ssh /tmp/new/.ssh ; export HOME=/tmp/new ; juju bootstrap and verified the correct ssh public key was injected to the instance (ssh -i /tmp/new/.ssh/id_rsa.pub ubuntu@host worked)
[00:05] <adam_g> jimbaker: after bootstrap, 'juju status' ends up launching an ssh session. $OLD_HOME/.ssh/known_hosts is updated and AFAICS $OLD_HOME/.ssh/id_rsa consulted instead of $HOME
[00:05] <adam_g> so yeah, ends in invalid key
[00:07] <jimbaker> adam_g, ok this looks like i was able to duplicate what you're seeing
[00:09] <jimbaker> adam_g, i wonder if this is poorly interacting with ssh-agent
[00:34] <SpamapS> Ugh, handling departed/broken gracefully really is tedious sometimes. :-P
[00:35]  * SpamapS decides to see if a good solution bubbles up whilst staring into the abyss
[00:48] <jimbaker> :)
[00:48] <jimbaker> adam_g, so far all i can determine is that the real .ssh is certainly sticky
[00:49] <jimbaker> i tried playing around with starting new instances of ssh-agent, but this didn't solve the problem for me
[00:59] <jimbaker> adam_g, it seems to me that ssh insists on the real user home directory, likely for security reasons. there are certainly options for controlling this. for example, maybe we could use ~/.ssh/config and the Host directive to better control what's going on
[00:59] <jimbaker> adam_g, another possibility is to allow for juju to use a specific known hosts file per environment. i had a branch that did that, but it was much more involved in terms of ssh key mgmt. maybe if it just did the known_hosts stuff, that could be reasonable
[01:19] <adam_g> jimbaker: yeah, i think its a feature of the ssh client more than anything. im able to work around it by tweaking my schroots
[02:26] <jimbaker> adam_g, sounds reasonable
[03:41] <jtv> I'm dealing with a user question where I'm not sure the problem they're having is really related to the question.  Could anyone here shed some light on the connection failure here?  http://askubuntu.com/questions/162255/does-juju-really-need-two-nodes
[03:42] <jtv> It looks similar to this one, but lacks the notice about a deadline being exceeded: http://askubuntu.com/questions/141687/juju-error-server-refused-to-accept-client?rq=1
[04:10] <jtv> jcastro, are you still here?
[04:11] <SpamapS> jtv: those are pretty tough questions :-P
[04:11] <SpamapS> jtv: #maas might have more helpful people
[04:12] <jtv> Thanks.  My problem though is that it's all a bit too far removed from maas.
[04:12] <SpamapS> jtv: the questions you pasted are maas specific
[04:12] <jtv> Juju bootstrap… it might not even be talking to maas yet for all I know!
[04:13] <SpamapS> it is
[04:13] <SpamapS> bootstrap is the only step that talks to maas
[04:13] <SpamapS> except destroy-environment, which does the inverse :)
[04:13] <jtv> So any idea what connection it is that is failing?
[04:14] <SpamapS> jtv: if you ssh into that machine, do you see zookeeper running? Also do you see juju processes running?
[04:15] <jtv> I don't have access to any of these machines, but when you say “that machine,” which one do you mean?
[04:15] <jtv> I'm not very familiar with juju.
[04:17] <SpamapS> jtv: the second question, juju was able to ssh in
[04:17] <SpamapS> jtv: it just failed on the ZK connection
[04:17] <SpamapS> have to go afk for a bit.. but you should be able to ssh to that machine
[04:17] <jtv> Juju was able to ssh into the system it created, but not to connect to ZK?
[04:18] <SpamapS> IQinIT-node01 I think
[04:18] <jtv> It may be helpful to talk about which machine roles are involved.  But go afk first!
[09:37] <bloodearnest> morning
[11:45] <hazmat> g'monring
[11:47] <hazmat> jamespage, just watched your presentation, re q/a unexpected removal of a node is detected, but a new unit or machine is not automatically launched.
[11:58] <imbrandon> hazmat: re: detailed google storage info, man that kicks absolute arse
[11:58] <imbrandon> :)
[11:58] <imbrandon> i already have a google storage account too, just never used it much, might start now :)
[12:04] <hazmat> imbrandon, me too
[12:24] <jcastro> hazmat: imbrandon: I wonder if adding cloudflare on top of my stuff being served on the google stuff would actually make it perform worse
[12:24] <jcastro> after reading hazmat's link it seems they have a bunch of that stuff built in
[12:26] <imbrandon> jcastro: it does, i actually turned cloudflare off on omg
[12:26] <imbrandon> jcastro: it keeps the site up if its on rinky-dink hardware but otherwise it throttles as well
[12:27] <hazmat> interesting
[12:27] <imbrandon> jcastro: so its actually slower if you have a robust site
[12:28] <jcastro> imbrandon: yeah I was wondering if 2 CDN things would just end up clobbering
[12:28] <imbrandon> in other words if you dont have to worry about your shit falling over face first if you get slash-dotted cuz you have google storeage backing or juju nginx scaling ,your better off without  cloudflare
[12:29] <jcastro> I was kind of hoping their static server config would give me automagic spdy but it's just normal http
[12:29] <imbrandon> yea the 2 cdn dont really hurt, but their throttleing does, it keeps connections alive so they dont fail but at some point arround ~100 connections the are serialized
[12:29] <imbrandon> ~100 concurent*
[12:30] <imbrandon> so like connections 100 to 150 ( 50 connections ) sit and wait for the first 100 to finish
[12:30] <imbrandon> but are kept alive tcp so they dont fail
[12:30] <imbrandon> etc
[12:30] <imbrandon> very cool setup but if you got the backing to do better its best not to have it on
[12:48] <jamespage> hazmat, hrm - is it not? OK - I must have seen a different behaviour then
[12:51] <hazmat> jamespage, hmm.. i take it back, it will relaunch the machine
[12:51] <hazmat> but not as a new unit
[12:51] <hazmat> which feels a bit questionable...
[12:58] <gary_poster> hi everyone.  We're interested in using juju as part of an entirely automated process--part of an integration test for a script that configures a machine, fwiw.  Juju is nice as an abstraction that can be pointed to a variety of backends for the test.  This works very well mostly, because juju ssh and juju scp accept all arguments to their wrapped commands, so we can use -o to pass ssh config values to turn off interactive p
[12:58] <gary_poster> rompts.  However, juju deploy won't let us pass these -o options for when it is trying to connect to the 0 machine initially.  Is there a command-line workaround for this (that is, other than modifying your .ssh/config)?  Alternatively, if I filed a juju bug for this feature would it have a chance of actually being accepted?
[13:00] <jamespage> hazmat, I used this when I had issues starting instances on an openstack deployment
[13:01] <hazmat> gary_poster, yes re bug, its inconsistent behavior.. for the other machiens your passing it via  unit name identifiers or via machine id?
[13:01] <jamespage> anything that went to failed state on build out got deleted by my instance reaper
[13:01] <jamespage> and juju just started up a new instance for the unit
[13:01] <jamespage> I think
[13:01] <jamespage> ...
[13:01] <jamespage> I'll have to check again
[13:01] <gary_poster> hazmat, cool, will file the bug then.  we are using machine id
[13:02] <hazmat> gary_poster, so its only machine 0 that it has an issue, but you can reference other machines by id and it works?
[13:02] <imbrandon> hazmat: only on deploy it looks like
[13:02] <hazmat> jamespage, yeah.. i just walked through the code it does appear to restart if the instance id is not present in the provider (which filters out some states)
[13:02] <hazmat> imbrandon, ?
[13:03] <gary_poster> hazmat, well, it is only in juju deploy, which under the covers connects via ssh.  when you then use juju ssh or juju scp, even on machine 0, it works fine
[13:03] <imbrandon> he said "juju ssh -o" works "juju deploy -o" does not
[13:03] <hazmat> oh.. i misunderstood
[13:03] <hazmat> that's expected behavior
[13:04] <gary_poster> hazmat, so no workaround except modifying .ssh/config, and feature bug would not be interesting?
[13:04] <hazmat> gary_poster, sorry that's not a bug
[13:04] <hazmat> imbrandon, thanks
[13:04] <imbrandon> :)
[13:04] <imbrandon> afk-ish, brb
[13:05] <gary_poster> hazmat, I didn't expect it was; it was a "hey would this feature/use case (where the use case is full automation of deploy) be something that juju would be willing to support)
[13:05] <hazmat> gary_poster, just use juju ssh prior to deploy
[13:05] <hazmat> gary_poster, no
[13:05] <hazmat> gary_poster, its not something juju would be willing to support
[13:05] <gary_poster> hazmat, I'm making a mess of this.  :-( Oy
[13:06] <gary_poster> It is juju bootstrap
[13:06] <gary_poster> that has the problem
[13:06] <gary_poster> so juju ssh won't help
[13:06] <hazmat> gary_poster, even better install a subordinate for your config, ie puppet or saltstack
[13:06] <hazmat> gary_poster, i don't really understand what the end goal is here.. "part of an integration test for a script that configures a machine"
[13:07] <gary_poster> hazmat, we want to be able to fully atomate juju
[13:07] <hazmat> okay
[13:07] <hazmat> gary_poster, and that means?
[13:07] <jamespage> hazmat, rest of my answers sound OK?
[13:07] <hazmat> jamespage, definitely, nicely done
[13:08] <gary_poster> as part of that we need to call bootstrap (and later destroy-environment) as part of the automation.  bootstrap requires manual ssh accepting of new key in ec2 case (non-lxc generally I think).
[13:08] <jamespage> hazmat, thanks!
[13:09] <gary_poster> bootstrap cannot currently be automated, therefore, unless you modify .ssh/config
[13:09] <imbrandon> gary_poster: thats best handled in the .ssh/config
[13:09] <hazmat> gary_poster, and the problem with that modifying .ssh/config is?
[13:10] <hazmat> gary_poster, we do automated tests of charms on multiple providers across ec2, local, etc via juju automation.
[13:11] <gary_poster> hazmat, imbrandon, the problem is that we'd rather not modify a user's .ssh/config just to run our tests--especially when you are making a blanket statement like "all aws instances for all purposes will have this behavior" which afaik is the only way to do it
[13:11] <gary_poster> (then you also have to make those statements for other providers)
[13:11] <gary_poster> (but that's not a practical issue for us right now)
[13:11] <imbrandon> gary_poster: if users running it != automated
[13:11] <imbrandon> unless i missed something here
[13:12] <imbrandon> editing the ci servers $HOME/.ssh/config is much diffrent
[13:12] <gary_poster> imbrandon, fair enough.  We want the automation to run it, but we want devs to be able to run it too.  charm test example: you want automated tests, but you want people to be able to run the tests locally
[13:12] <hazmat> gary_poster, so what's automated about this?
[13:13] <hazmat> a user is running it.. so they have their own ~/.ssh and their own ~/.juju environments.yaml
[13:13] <imbrandon> gary_poster: sure let them kick it off manuly from the ci server, thats easy in jenkins
[13:13] <gary_poster> agreed that ci .ssh/config is fine, which is what we are doing
[13:13] <hazmat> gary_poster, so your delivering what to people to run the tests?
[13:15] <hazmat> its also unclear why you want the ssh to the bootstrap node on deploy,  in this case.. it sounds like a regular environment, and an automation script to launch the test env and run the tests.
[13:16] <hazmat> and then possibly a separate front end script for the jenkins runner, that setups the keys and environments.yaml as needed
[13:16] <gary_poster> hazmat, python callable.  Nice easy story is "set up your .juju config with a special environment named foo, configured to run tests whereever makes sense for you".  Another nice part of that would be "configure that environment to not ask questions when you bootstrap".  A less nice story is "and also modify your .ssh/config to ignore keys from AWS"
[13:16] <hazmat> so your saying its okay to modify ~/.juju for a user but not okay to modify ~/.ssh ?
[13:16] <hazmat> just have the script set $HOME
[13:17] <mgz> gary_poster: what exactly are you wanting to pass? just suppression of the known_hosts prompt or other bits too?
[13:18] <gary_poster> hazmat, ok to modify .juju and not .ssh: yes, because in .juju you can constrain everything to a specially named environment.  .ssh is blanket for (for instance) AWS.
[13:18] <mgz> because juju should probably just integrate smoser's cloud-init method of verifying host keys
[13:19] <gary_poster> mgz: yes, just -o 'StrictHostKeyChecking=no' -o 'UserKnownHostsFile=/dev/null' or similar
[13:19] <hazmat> gary_poster, this is confusing in terms of the original question.. but the delivery script/callable can set $HOME and setup a private .ssh and .juju as needed
[13:20] <mgz> okay, so there's a better way of doing that, +/- the amazon delay on supplying console output
[13:20] <mgz> which is to acutally verify the key reported by the cloud-init during instance boot and insert it in ~/.ssh/known_hosts
[13:21] <gary_poster> hazmat, ack.  Yes, confusing, I said deploy initially and meant bootstrap and took everything down the wrong rabbit hole, I'm sorry.
[13:21] <gary_poster> mgz, that sounds very nice
[13:21] <mgz> I use this for fully automated deployment of launchpad on canonistack
[13:22] <hazmat> mgz,  so console output scraping for the host fingerprint?
[13:22] <gary_poster> hazmat, ack that $HOME would work.  I'll propose it.  mgz, that would have to happen in juju though, right?  We, as juju clients, can't do that?
[13:22] <mgz> hazmat: yup, works really nicely with openstack, may be less good against aws
[13:23] <hazmat> kees also suggested it at uds.. the only issue with that on ec2 is the delay can be a bit atrocious..
[13:23] <gary_poster> (The $HOME is not as easy to do as what I proposed, so as a user I'd still prefer the kind of automation mgz suggests)
[13:23] <mgz> so I heard 4 minute intervals on output update? and I guess sshd is generally up and waiting before the first 4 minutes
[13:24] <gary_poster> well before, often
[13:24] <hazmat> mgz, it can take longer, its quite variable
[13:24] <mgz> so adding wait time after boot wouldn't be great
[13:24] <hazmat> mgz, re output availability, i've hit 10m b4
[13:25] <mgz> can poll at 5s intervals with openstack and get nearly live reporting :)
[13:26] <mgz> hazmat: I guess the alternative is a juju config option, to 1) prompt/die as currently 2) accept any host key blndly per gary's args 3) use smoser's trick
[13:26] <mgz> and leave which one up to the user
[13:27] <mgz> the current behaviour when connecting to a new host on an ip you've seen before is obnoxious
[13:27] <gary_poster> agreed
[13:27] <smoser> mgz, regarding ssh keys. one thing that would be completely possible for juju is what kirkland's cloud-sandbox does.
[13:28] <smoser> it generates ssh public/private key on the local system. passes those through user-data to cloud-init.
[13:28] <gary_poster> I think I'd be ok with waiting for this use case, fwiw
[13:29] <smoser> connects to the host expecting those credentials, and then destroys them and creates new ones.
[13:29] <smoser> so the ones that were sent insecurely via user-data are usable for less than a minute.
[13:30] <mgz> right, was going to say, putting the private key in user-data isn't ideal
[13:30] <smoser> but its fine if you destroy it right away.
[13:30] <smoser> and for everything but the bootstrap node, they can then post the public keys to the bootstrap node.
[13:30] <smoser> and your client can aske the bootstrap node for the new nodes public keys.
[13:30] <smoser> so you only need the dance once.
[13:31] <gary_poster> by which I mean, for this use case, I'd be ok with what you'd need to do for mgz's solution: waiting longer than usual for a trustable machine without having user interaction.  For other usecases, less so.
[13:50] <jml> is it possible to tell juju to look at a different environments.yaml file?
[13:54] <hazmat> jml, not without $HOME manipulation.. juju does respect JUJU_ENV for named env usage
[13:54] <jml> hazmat: thanks.
[13:54] <jml> fwiw, we're experimenting with using juju to fire up our services for integration tests
[14:12] <jml> how can I tell if a local environment is bootstrapped?
[14:13] <jml> never mind. destroyed the wrong one :(
[14:59] <james_w> if you deploy a charm as serviceA, and then deploy it as serviceB, does it does the same copy of the charm?
[14:59] <james_w> i.e. if you modify the charm between the two deploys, would the second need -u to use the changes?
[15:04] <jml> is there a valid charm name regex?
[15:15] <mgz> what's the juju equivalent of:
[15:16] <mgz> trace.warning("some message to the user about something breaking")
[15:16] <mgz> trace.log_exception_quietly()
[15:17] <mgz> basically giving a nice clean error without traceback for common failures, but preserves the traceback in log/verbose mode
[15:26] <jml> can you use jitsu to watch for destroy-service to complete?
[16:22] <imbrandon> http://i.imgur.com/x7zO0.jpg  , that is all ...
[17:12] <hazmat> mgz log.warning, log.debug
[17:12] <hazmat> mgz with traceback module formatted exception
[17:13] <hazmat> into the debug
[17:19] <imbrandon> SpamapS: btw https://launchpad.net/~webstack/+archive/php/+packages
[17:20] <SpamapS> imbrandon: cool
[17:21] <imbrandon> i still like the idea of a charmers ppa, but thats for now
[17:21] <imbrandon> 386 built, amd64 still in queue
[19:59] <SpamapS> hrm
[19:59] <SpamapS> there's something very broken with debug-hooks
[19:59] <SpamapS> sometimes debug-hooks just sits there, and no more hooks execute
[20:00] <imbrandon> yea and if you exit go luck ever running it again
[20:00] <imbrandon> on that environment
[20:00] <SpamapS> exactly
[20:01] <SpamapS> hazmat: ^^ debug-hooks seems to have something wrong.. flags not cleared or something
[20:09] <SpamapS> ugh, and restarting the unit agent made it worse
[20:09] <SpamapS> now no hooks run
[20:09] <SpamapS> even when debug-hooks isn't running
[20:10] <SpamapS> 2012-07-12 13:09:45,743 INFO Waiting for unit to come up.
[20:10] <SpamapS> ah.. you can't relaly restart the unit agent
[20:12] <imbrandon> man where is the creat project page
[20:12] <imbrandon> grrr
[20:12] <SpamapS> front page
[20:12] <SpamapS> first thing
[20:12] <imbrandon> oh the one place i no look
[20:12] <SpamapS> "Register a project"
[20:25] <_mup_> Bug #966617 was filed: supply high level design documents <juju:New> <juju (Ubuntu):Triaged> <juju (Ubuntu Precise):Triaged> <juju (Ubuntu Quantal):Triaged> < https://launchpad.net/bugs/966617 >