jimbakeradam_g, ok, if i hide my real ~/.ssh directory, i am running into problems. it's not clear to me this is really necessary to do however00:01
jimbaker(juju will report ERROR Invalid SSH key repeatedly)00:03
adam_gjimbaker: what i did: cp -r $HOME/.ssh /tmp/new/.ssh ; export HOME=/tmp/new ; juju bootstrap and verified the correct ssh public key was injected to the instance (ssh -i /tmp/new/.ssh/id_rsa.pub ubuntu@host worked)00:04
adam_gjimbaker: after bootstrap, 'juju status' ends up launching an ssh session. $OLD_HOME/.ssh/known_hosts is updated and AFAICS $OLD_HOME/.ssh/id_rsa consulted instead of $HOME00:05
adam_gso yeah, ends in invalid key00:05
jimbakeradam_g, ok this looks like i was able to duplicate what you're seeing00:07
jimbakeradam_g, i wonder if this is poorly interacting with ssh-agent00:09
SpamapSUgh, handling departed/broken gracefully really is tedious sometimes. :-P00:34
* SpamapS decides to see if a good solution bubbles up whilst staring into the abyss00:35
jimbakeradam_g, so far all i can determine is that the real .ssh is certainly sticky00:48
jimbakeri tried playing around with starting new instances of ssh-agent, but this didn't solve the problem for me00:49
jimbakeradam_g, it seems to me that ssh insists on the real user home directory, likely for security reasons. there are certainly options for controlling this. for example, maybe we could use ~/.ssh/config and the Host directive to better control what's going on00:59
jimbakeradam_g, another possibility is to allow for juju to use a specific known hosts file per environment. i had a branch that did that, but it was much more involved in terms of ssh key mgmt. maybe if it just did the known_hosts stuff, that could be reasonable00:59
adam_gjimbaker: yeah, i think its a feature of the ssh client more than anything. im able to work around it by tweaking my schroots01:19
jimbakeradam_g, sounds reasonable02:26
jtvI'm dealing with a user question where I'm not sure the problem they're having is really related to the question.  Could anyone here shed some light on the connection failure here?  http://askubuntu.com/questions/162255/does-juju-really-need-two-nodes03:41
jtvIt looks similar to this one, but lacks the notice about a deadline being exceeded: http://askubuntu.com/questions/141687/juju-error-server-refused-to-accept-client?rq=103:42
=== objectiveous_ is now known as objectiveous
jtvjcastro, are you still here?04:10
SpamapSjtv: those are pretty tough questions :-P04:11
SpamapSjtv: #maas might have more helpful people04:11
jtvThanks.  My problem though is that it's all a bit too far removed from maas.04:12
SpamapSjtv: the questions you pasted are maas specific04:12
jtvJuju bootstrap… it might not even be talking to maas yet for all I know!04:12
SpamapSit is04:13
SpamapSbootstrap is the only step that talks to maas04:13
SpamapSexcept destroy-environment, which does the inverse :)04:13
jtvSo any idea what connection it is that is failing?04:13
SpamapSjtv: if you ssh into that machine, do you see zookeeper running? Also do you see juju processes running?04:14
jtvI don't have access to any of these machines, but when you say “that machine,” which one do you mean?04:15
jtvI'm not very familiar with juju.04:15
SpamapSjtv: the second question, juju was able to ssh in04:17
SpamapSjtv: it just failed on the ZK connection04:17
SpamapShave to go afk for a bit.. but you should be able to ssh to that machine04:17
jtvJuju was able to ssh into the system it created, but not to connect to ZK?04:17
SpamapSIQinIT-node01 I think04:18
jtvIt may be helpful to talk about which machine roles are involved.  But go afk first!04:18
=== almaisan-away is now known as al-maisan
=== TheRealMue is now known as TheMue
=== zyga is now known as zyga-afk
=== zyga-afk is now known as zyga
=== al-maisan is now known as almaisan-away
hazmatjamespage, just watched your presentation, re q/a unexpected removal of a node is detected, but a new unit or machine is not automatically launched.11:47
imbrandonhazmat: re: detailed google storage info, man that kicks absolute arse11:58
imbrandoni already have a google storage account too, just never used it much, might start now :)11:58
hazmatimbrandon, me too12:04
jcastrohazmat: imbrandon: I wonder if adding cloudflare on top of my stuff being served on the google stuff would actually make it perform worse12:24
jcastroafter reading hazmat's link it seems they have a bunch of that stuff built in12:24
imbrandonjcastro: it does, i actually turned cloudflare off on omg12:26
imbrandonjcastro: it keeps the site up if its on rinky-dink hardware but otherwise it throttles as well12:26
imbrandonjcastro: so its actually slower if you have a robust site12:27
jcastroimbrandon: yeah I was wondering if 2 CDN things would just end up clobbering12:28
imbrandonin other words if you dont have to worry about your shit falling over face first if you get slash-dotted cuz you have google storeage backing or juju nginx scaling ,your better off without  cloudflare12:28
jcastroI was kind of hoping their static server config would give me automagic spdy but it's just normal http12:29
imbrandonyea the 2 cdn dont really hurt, but their throttleing does, it keeps connections alive so they dont fail but at some point arround ~100 connections the are serialized12:29
imbrandon~100 concurent*12:29
imbrandonso like connections 100 to 150 ( 50 connections ) sit and wait for the first 100 to finish12:30
imbrandonbut are kept alive tcp so they dont fail12:30
imbrandonvery cool setup but if you got the backing to do better its best not to have it on12:30
=== almaisan-away is now known as al-maisan
jamespagehazmat, hrm - is it not? OK - I must have seen a different behaviour then12:48
hazmatjamespage, hmm.. i take it back, it will relaunch the machine12:51
hazmatbut not as a new unit12:51
hazmatwhich feels a bit questionable...12:51
gary_posterhi everyone.  We're interested in using juju as part of an entirely automated process--part of an integration test for a script that configures a machine, fwiw.  Juju is nice as an abstraction that can be pointed to a variety of backends for the test.  This works very well mostly, because juju ssh and juju scp accept all arguments to their wrapped commands, so we can use -o to pass ssh config values to turn off interactive p12:58
gary_posterrompts.  However, juju deploy won't let us pass these -o options for when it is trying to connect to the 0 machine initially.  Is there a command-line workaround for this (that is, other than modifying your .ssh/config)?  Alternatively, if I filed a juju bug for this feature would it have a chance of actually being accepted?12:58
jamespagehazmat, I used this when I had issues starting instances on an openstack deployment13:00
hazmatgary_poster, yes re bug, its inconsistent behavior.. for the other machiens your passing it via  unit name identifiers or via machine id?13:01
jamespageanything that went to failed state on build out got deleted by my instance reaper13:01
jamespageand juju just started up a new instance for the unit13:01
jamespageI think13:01
jamespageI'll have to check again13:01
gary_posterhazmat, cool, will file the bug then.  we are using machine id13:01
hazmatgary_poster, so its only machine 0 that it has an issue, but you can reference other machines by id and it works?13:02
imbrandonhazmat: only on deploy it looks like13:02
hazmatjamespage, yeah.. i just walked through the code it does appear to restart if the instance id is not present in the provider (which filters out some states)13:02
hazmatimbrandon, ?13:02
gary_posterhazmat, well, it is only in juju deploy, which under the covers connects via ssh.  when you then use juju ssh or juju scp, even on machine 0, it works fine13:03
imbrandonhe said "juju ssh -o" works "juju deploy -o" does not13:03
hazmatoh.. i misunderstood13:03
hazmatthat's expected behavior13:03
gary_posterhazmat, so no workaround except modifying .ssh/config, and feature bug would not be interesting?13:04
hazmatgary_poster, sorry that's not a bug13:04
hazmatimbrandon, thanks13:04
imbrandonafk-ish, brb13:04
gary_posterhazmat, I didn't expect it was; it was a "hey would this feature/use case (where the use case is full automation of deploy) be something that juju would be willing to support)13:05
hazmatgary_poster, just use juju ssh prior to deploy13:05
hazmatgary_poster, no13:05
hazmatgary_poster, its not something juju would be willing to support13:05
gary_posterhazmat, I'm making a mess of this.  :-( Oy13:05
gary_posterIt is juju bootstrap13:06
gary_posterthat has the problem13:06
gary_posterso juju ssh won't help13:06
hazmatgary_poster, even better install a subordinate for your config, ie puppet or saltstack13:06
hazmatgary_poster, i don't really understand what the end goal is here.. "part of an integration test for a script that configures a machine"13:06
gary_posterhazmat, we want to be able to fully atomate juju13:07
hazmatgary_poster, and that means?13:07
jamespagehazmat, rest of my answers sound OK?13:07
hazmatjamespage, definitely, nicely done13:07
gary_posteras part of that we need to call bootstrap (and later destroy-environment) as part of the automation.  bootstrap requires manual ssh accepting of new key in ec2 case (non-lxc generally I think).13:08
jamespagehazmat, thanks!13:08
gary_posterbootstrap cannot currently be automated, therefore, unless you modify .ssh/config13:09
imbrandongary_poster: thats best handled in the .ssh/config13:09
hazmatgary_poster, and the problem with that modifying .ssh/config is?13:09
hazmatgary_poster, we do automated tests of charms on multiple providers across ec2, local, etc via juju automation.13:10
gary_posterhazmat, imbrandon, the problem is that we'd rather not modify a user's .ssh/config just to run our tests--especially when you are making a blanket statement like "all aws instances for all purposes will have this behavior" which afaik is the only way to do it13:11
gary_poster(then you also have to make those statements for other providers)13:11
gary_poster(but that's not a practical issue for us right now)13:11
imbrandongary_poster: if users running it != automated13:11
imbrandonunless i missed something here13:11
imbrandonediting the ci servers $HOME/.ssh/config is much diffrent13:12
gary_posterimbrandon, fair enough.  We want the automation to run it, but we want devs to be able to run it too.  charm test example: you want automated tests, but you want people to be able to run the tests locally13:12
hazmatgary_poster, so what's automated about this?13:12
hazmata user is running it.. so they have their own ~/.ssh and their own ~/.juju environments.yaml13:13
imbrandongary_poster: sure let them kick it off manuly from the ci server, thats easy in jenkins13:13
gary_posteragreed that ci .ssh/config is fine, which is what we are doing13:13
hazmatgary_poster, so your delivering what to people to run the tests?13:13
hazmatits also unclear why you want the ssh to the bootstrap node on deploy,  in this case.. it sounds like a regular environment, and an automation script to launch the test env and run the tests.13:15
hazmatand then possibly a separate front end script for the jenkins runner, that setups the keys and environments.yaml as needed13:16
gary_posterhazmat, python callable.  Nice easy story is "set up your .juju config with a special environment named foo, configured to run tests whereever makes sense for you".  Another nice part of that would be "configure that environment to not ask questions when you bootstrap".  A less nice story is "and also modify your .ssh/config to ignore keys from AWS"13:16
hazmatso your saying its okay to modify ~/.juju for a user but not okay to modify ~/.ssh ?13:16
hazmatjust have the script set $HOME13:16
mgzgary_poster: what exactly are you wanting to pass? just suppression of the known_hosts prompt or other bits too?13:17
gary_posterhazmat, ok to modify .juju and not .ssh: yes, because in .juju you can constrain everything to a specially named environment.  .ssh is blanket for (for instance) AWS.13:18
mgzbecause juju should probably just integrate smoser's cloud-init method of verifying host keys13:18
gary_postermgz: yes, just -o 'StrictHostKeyChecking=no' -o 'UserKnownHostsFile=/dev/null' or similar13:19
hazmatgary_poster, this is confusing in terms of the original question.. but the delivery script/callable can set $HOME and setup a private .ssh and .juju as needed13:19
mgzokay, so there's a better way of doing that, +/- the amazon delay on supplying console output13:20
mgzwhich is to acutally verify the key reported by the cloud-init during instance boot and insert it in ~/.ssh/known_hosts13:20
gary_posterhazmat, ack.  Yes, confusing, I said deploy initially and meant bootstrap and took everything down the wrong rabbit hole, I'm sorry.13:21
gary_postermgz, that sounds very nice13:21
mgzI use this for fully automated deployment of launchpad on canonistack13:21
hazmatmgz,  so console output scraping for the host fingerprint?13:22
gary_posterhazmat, ack that $HOME would work.  I'll propose it.  mgz, that would have to happen in juju though, right?  We, as juju clients, can't do that?13:22
mgzhazmat: yup, works really nicely with openstack, may be less good against aws13:22
hazmatkees also suggested it at uds.. the only issue with that on ec2 is the delay can be a bit atrocious..13:23
gary_poster(The $HOME is not as easy to do as what I proposed, so as a user I'd still prefer the kind of automation mgz suggests)13:23
mgzso I heard 4 minute intervals on output update? and I guess sshd is generally up and waiting before the first 4 minutes13:23
gary_posterwell before, often13:24
hazmatmgz, it can take longer, its quite variable13:24
mgzso adding wait time after boot wouldn't be great13:24
hazmatmgz, re output availability, i've hit 10m b413:24
mgzcan poll at 5s intervals with openstack and get nearly live reporting :)13:25
mgzhazmat: I guess the alternative is a juju config option, to 1) prompt/die as currently 2) accept any host key blndly per gary's args 3) use smoser's trick13:26
mgzand leave which one up to the user13:26
mgzthe current behaviour when connecting to a new host on an ip you've seen before is obnoxious13:27
smosermgz, regarding ssh keys. one thing that would be completely possible for juju is what kirkland's cloud-sandbox does.13:27
smoserit generates ssh public/private key on the local system. passes those through user-data to cloud-init.13:28
gary_posterI think I'd be ok with waiting for this use case, fwiw13:28
smoserconnects to the host expecting those credentials, and then destroys them and creates new ones.13:29
smoserso the ones that were sent insecurely via user-data are usable for less than a minute.13:29
mgzright, was going to say, putting the private key in user-data isn't ideal13:30
smoserbut its fine if you destroy it right away.13:30
smoserand for everything but the bootstrap node, they can then post the public keys to the bootstrap node.13:30
smoserand your client can aske the bootstrap node for the new nodes public keys.13:30
smoserso you only need the dance once.13:30
gary_posterby which I mean, for this use case, I'd be ok with what you'd need to do for mgz's solution: waiting longer than usual for a trustable machine without having user interaction.  For other usecases, less so.13:31
jmlis it possible to tell juju to look at a different environments.yaml file?13:50
hazmatjml, not without $HOME manipulation.. juju does respect JUJU_ENV for named env usage13:54
jmlhazmat: thanks.13:54
jmlfwiw, we're experimenting with using juju to fire up our services for integration tests13:54
jmlhow can I tell if a local environment is bootstrapped?14:12
jmlnever mind. destroyed the wrong one :(14:13
=== zyga is now known as zyga-food
=== al-maisan is now known as almaisan-away
=== almaisan-away is now known as al-maisan
james_wif you deploy a charm as serviceA, and then deploy it as serviceB, does it does the same copy of the charm?14:59
james_wi.e. if you modify the charm between the two deploys, would the second need -u to use the changes?14:59
jmlis there a valid charm name regex?15:04
=== objectiveous_ is now known as objectiveous
mgzwhat's the juju equivalent of:15:15
mgztrace.warning("some message to the user about something breaking")15:16
mgzbasically giving a nice clean error without traceback for common failures, but preserves the traceback in log/verbose mode15:17
jmlcan you use jitsu to watch for destroy-service to complete?15:26
=== zyga-food is now known as zyga
=== salgado is now known as salgado-lunch
=== al-maisan is now known as almaisan-away
=== almaisan-away is now known as al-maisan
=== al-maisan is now known as almaisan-away
imbrandonhttp://i.imgur.com/x7zO0.jpg  , that is all ...16:22
hazmatmgz log.warning, log.debug17:12
hazmatmgz with traceback module formatted exception17:12
hazmatinto the debug17:13
imbrandonSpamapS: btw https://launchpad.net/~webstack/+archive/php/+packages17:19
SpamapSimbrandon: cool17:20
imbrandoni still like the idea of a charmers ppa, but thats for now17:21
imbrandon386 built, amd64 still in queue17:21
=== salgado-lunch is now known as salgado
=== zyga is now known as zyga-afk
SpamapSthere's something very broken with debug-hooks19:59
SpamapSsometimes debug-hooks just sits there, and no more hooks execute19:59
imbrandonyea and if you exit go luck ever running it again20:00
imbrandonon that environment20:00
SpamapShazmat: ^^ debug-hooks seems to have something wrong.. flags not cleared or something20:01
SpamapSugh, and restarting the unit agent made it worse20:09
SpamapSnow no hooks run20:09
SpamapSeven when debug-hooks isn't running20:09
SpamapS2012-07-12 13:09:45,743 INFO Waiting for unit to come up.20:10
SpamapSah.. you can't relaly restart the unit agent20:10
imbrandonman where is the creat project page20:12
SpamapSfront page20:12
SpamapSfirst thing20:12
imbrandonoh the one place i no look20:12
SpamapS"Register a project"20:12
_mup_Bug #966617 was filed: supply high level design documents <juju:New> <juju (Ubuntu):Triaged> <juju (Ubuntu Precise):Triaged> <juju (Ubuntu Quantal):Triaged> < https://launchpad.net/bugs/966617 >20:25
=== pdtpatrick_ is now known as pdtpatrick

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!