| jimbaker | adam_g, ok, if i hide my real ~/.ssh directory, i am running into problems. it's not clear to me this is really necessary to do however | 00:01 |
|---|---|---|
| jimbaker | (juju will report ERROR Invalid SSH key repeatedly) | 00:03 |
| adam_g | jimbaker: what i did: cp -r $HOME/.ssh /tmp/new/.ssh ; export HOME=/tmp/new ; juju bootstrap and verified the correct ssh public key was injected to the instance (ssh -i /tmp/new/.ssh/id_rsa.pub ubuntu@host worked) | 00:04 |
| adam_g | jimbaker: after bootstrap, 'juju status' ends up launching an ssh session. $OLD_HOME/.ssh/known_hosts is updated and AFAICS $OLD_HOME/.ssh/id_rsa consulted instead of $HOME | 00:05 |
| adam_g | so yeah, ends in invalid key | 00:05 |
| jimbaker | adam_g, ok this looks like i was able to duplicate what you're seeing | 00:07 |
| jimbaker | adam_g, i wonder if this is poorly interacting with ssh-agent | 00:09 |
| SpamapS | Ugh, handling departed/broken gracefully really is tedious sometimes. :-P | 00:34 |
| * SpamapS decides to see if a good solution bubbles up whilst staring into the abyss | 00:35 | |
| jimbaker | :) | 00:48 |
| jimbaker | adam_g, so far all i can determine is that the real .ssh is certainly sticky | 00:48 |
| jimbaker | i tried playing around with starting new instances of ssh-agent, but this didn't solve the problem for me | 00:49 |
| jimbaker | adam_g, it seems to me that ssh insists on the real user home directory, likely for security reasons. there are certainly options for controlling this. for example, maybe we could use ~/.ssh/config and the Host directive to better control what's going on | 00:59 |
| jimbaker | adam_g, another possibility is to allow for juju to use a specific known hosts file per environment. i had a branch that did that, but it was much more involved in terms of ssh key mgmt. maybe if it just did the known_hosts stuff, that could be reasonable | 00:59 |
| adam_g | jimbaker: yeah, i think its a feature of the ssh client more than anything. im able to work around it by tweaking my schroots | 01:19 |
| jimbaker | adam_g, sounds reasonable | 02:26 |
| jtv | I'm dealing with a user question where I'm not sure the problem they're having is really related to the question. Could anyone here shed some light on the connection failure here? http://askubuntu.com/questions/162255/does-juju-really-need-two-nodes | 03:41 |
| jtv | It looks similar to this one, but lacks the notice about a deadline being exceeded: http://askubuntu.com/questions/141687/juju-error-server-refused-to-accept-client?rq=1 | 03:42 |
| === objectiveous_ is now known as objectiveous | ||
| jtv | jcastro, are you still here? | 04:10 |
| SpamapS | jtv: those are pretty tough questions :-P | 04:11 |
| SpamapS | jtv: #maas might have more helpful people | 04:11 |
| jtv | Thanks. My problem though is that it's all a bit too far removed from maas. | 04:12 |
| SpamapS | jtv: the questions you pasted are maas specific | 04:12 |
| jtv | Juju bootstrap… it might not even be talking to maas yet for all I know! | 04:12 |
| SpamapS | it is | 04:13 |
| SpamapS | bootstrap is the only step that talks to maas | 04:13 |
| SpamapS | except destroy-environment, which does the inverse :) | 04:13 |
| jtv | So any idea what connection it is that is failing? | 04:13 |
| SpamapS | jtv: if you ssh into that machine, do you see zookeeper running? Also do you see juju processes running? | 04:14 |
| jtv | I don't have access to any of these machines, but when you say “that machine,” which one do you mean? | 04:15 |
| jtv | I'm not very familiar with juju. | 04:15 |
| SpamapS | jtv: the second question, juju was able to ssh in | 04:17 |
| SpamapS | jtv: it just failed on the ZK connection | 04:17 |
| SpamapS | have to go afk for a bit.. but you should be able to ssh to that machine | 04:17 |
| jtv | Juju was able to ssh into the system it created, but not to connect to ZK? | 04:17 |
| SpamapS | IQinIT-node01 I think | 04:18 |
| jtv | It may be helpful to talk about which machine roles are involved. But go afk first! | 04:18 |
| === almaisan-away is now known as al-maisan | ||
| === TheRealMue is now known as TheMue | ||
| === zyga is now known as zyga-afk | ||
| bloodearnest | morning | 09:37 |
| === zyga-afk is now known as zyga | ||
| === al-maisan is now known as almaisan-away | ||
| hazmat | g'monring | 11:45 |
| hazmat | jamespage, just watched your presentation, re q/a unexpected removal of a node is detected, but a new unit or machine is not automatically launched. | 11:47 |
| imbrandon | hazmat: re: detailed google storage info, man that kicks absolute arse | 11:58 |
| imbrandon | :) | 11:58 |
| imbrandon | i already have a google storage account too, just never used it much, might start now :) | 11:58 |
| hazmat | imbrandon, me too | 12:04 |
| jcastro | hazmat: imbrandon: I wonder if adding cloudflare on top of my stuff being served on the google stuff would actually make it perform worse | 12:24 |
| jcastro | after reading hazmat's link it seems they have a bunch of that stuff built in | 12:24 |
| imbrandon | jcastro: it does, i actually turned cloudflare off on omg | 12:26 |
| imbrandon | jcastro: it keeps the site up if its on rinky-dink hardware but otherwise it throttles as well | 12:26 |
| hazmat | interesting | 12:27 |
| imbrandon | jcastro: so its actually slower if you have a robust site | 12:27 |
| jcastro | imbrandon: yeah I was wondering if 2 CDN things would just end up clobbering | 12:28 |
| imbrandon | in other words if you dont have to worry about your shit falling over face first if you get slash-dotted cuz you have google storeage backing or juju nginx scaling ,your better off without cloudflare | 12:28 |
| jcastro | I was kind of hoping their static server config would give me automagic spdy but it's just normal http | 12:29 |
| imbrandon | yea the 2 cdn dont really hurt, but their throttleing does, it keeps connections alive so they dont fail but at some point arround ~100 connections the are serialized | 12:29 |
| imbrandon | ~100 concurent* | 12:29 |
| imbrandon | so like connections 100 to 150 ( 50 connections ) sit and wait for the first 100 to finish | 12:30 |
| imbrandon | but are kept alive tcp so they dont fail | 12:30 |
| imbrandon | etc | 12:30 |
| imbrandon | very cool setup but if you got the backing to do better its best not to have it on | 12:30 |
| === almaisan-away is now known as al-maisan | ||
| jamespage | hazmat, hrm - is it not? OK - I must have seen a different behaviour then | 12:48 |
| hazmat | jamespage, hmm.. i take it back, it will relaunch the machine | 12:51 |
| hazmat | but not as a new unit | 12:51 |
| hazmat | which feels a bit questionable... | 12:51 |
| gary_poster | hi everyone. We're interested in using juju as part of an entirely automated process--part of an integration test for a script that configures a machine, fwiw. Juju is nice as an abstraction that can be pointed to a variety of backends for the test. This works very well mostly, because juju ssh and juju scp accept all arguments to their wrapped commands, so we can use -o to pass ssh config values to turn off interactive p | 12:58 |
| gary_poster | rompts. However, juju deploy won't let us pass these -o options for when it is trying to connect to the 0 machine initially. Is there a command-line workaround for this (that is, other than modifying your .ssh/config)? Alternatively, if I filed a juju bug for this feature would it have a chance of actually being accepted? | 12:58 |
| jamespage | hazmat, I used this when I had issues starting instances on an openstack deployment | 13:00 |
| hazmat | gary_poster, yes re bug, its inconsistent behavior.. for the other machiens your passing it via unit name identifiers or via machine id? | 13:01 |
| jamespage | anything that went to failed state on build out got deleted by my instance reaper | 13:01 |
| jamespage | and juju just started up a new instance for the unit | 13:01 |
| jamespage | I think | 13:01 |
| jamespage | ... | 13:01 |
| jamespage | I'll have to check again | 13:01 |
| gary_poster | hazmat, cool, will file the bug then. we are using machine id | 13:01 |
| hazmat | gary_poster, so its only machine 0 that it has an issue, but you can reference other machines by id and it works? | 13:02 |
| imbrandon | hazmat: only on deploy it looks like | 13:02 |
| hazmat | jamespage, yeah.. i just walked through the code it does appear to restart if the instance id is not present in the provider (which filters out some states) | 13:02 |
| hazmat | imbrandon, ? | 13:02 |
| gary_poster | hazmat, well, it is only in juju deploy, which under the covers connects via ssh. when you then use juju ssh or juju scp, even on machine 0, it works fine | 13:03 |
| imbrandon | he said "juju ssh -o" works "juju deploy -o" does not | 13:03 |
| hazmat | oh.. i misunderstood | 13:03 |
| hazmat | that's expected behavior | 13:03 |
| gary_poster | hazmat, so no workaround except modifying .ssh/config, and feature bug would not be interesting? | 13:04 |
| hazmat | gary_poster, sorry that's not a bug | 13:04 |
| hazmat | imbrandon, thanks | 13:04 |
| imbrandon | :) | 13:04 |
| imbrandon | afk-ish, brb | 13:04 |
| gary_poster | hazmat, I didn't expect it was; it was a "hey would this feature/use case (where the use case is full automation of deploy) be something that juju would be willing to support) | 13:05 |
| hazmat | gary_poster, just use juju ssh prior to deploy | 13:05 |
| hazmat | gary_poster, no | 13:05 |
| hazmat | gary_poster, its not something juju would be willing to support | 13:05 |
| gary_poster | hazmat, I'm making a mess of this. :-( Oy | 13:05 |
| gary_poster | It is juju bootstrap | 13:06 |
| gary_poster | that has the problem | 13:06 |
| gary_poster | so juju ssh won't help | 13:06 |
| hazmat | gary_poster, even better install a subordinate for your config, ie puppet or saltstack | 13:06 |
| hazmat | gary_poster, i don't really understand what the end goal is here.. "part of an integration test for a script that configures a machine" | 13:06 |
| gary_poster | hazmat, we want to be able to fully atomate juju | 13:07 |
| hazmat | okay | 13:07 |
| hazmat | gary_poster, and that means? | 13:07 |
| jamespage | hazmat, rest of my answers sound OK? | 13:07 |
| hazmat | jamespage, definitely, nicely done | 13:07 |
| gary_poster | as part of that we need to call bootstrap (and later destroy-environment) as part of the automation. bootstrap requires manual ssh accepting of new key in ec2 case (non-lxc generally I think). | 13:08 |
| jamespage | hazmat, thanks! | 13:08 |
| gary_poster | bootstrap cannot currently be automated, therefore, unless you modify .ssh/config | 13:09 |
| imbrandon | gary_poster: thats best handled in the .ssh/config | 13:09 |
| hazmat | gary_poster, and the problem with that modifying .ssh/config is? | 13:09 |
| hazmat | gary_poster, we do automated tests of charms on multiple providers across ec2, local, etc via juju automation. | 13:10 |
| gary_poster | hazmat, imbrandon, the problem is that we'd rather not modify a user's .ssh/config just to run our tests--especially when you are making a blanket statement like "all aws instances for all purposes will have this behavior" which afaik is the only way to do it | 13:11 |
| gary_poster | (then you also have to make those statements for other providers) | 13:11 |
| gary_poster | (but that's not a practical issue for us right now) | 13:11 |
| imbrandon | gary_poster: if users running it != automated | 13:11 |
| imbrandon | unless i missed something here | 13:11 |
| imbrandon | editing the ci servers $HOME/.ssh/config is much diffrent | 13:12 |
| gary_poster | imbrandon, fair enough. We want the automation to run it, but we want devs to be able to run it too. charm test example: you want automated tests, but you want people to be able to run the tests locally | 13:12 |
| hazmat | gary_poster, so what's automated about this? | 13:12 |
| hazmat | a user is running it.. so they have their own ~/.ssh and their own ~/.juju environments.yaml | 13:13 |
| imbrandon | gary_poster: sure let them kick it off manuly from the ci server, thats easy in jenkins | 13:13 |
| gary_poster | agreed that ci .ssh/config is fine, which is what we are doing | 13:13 |
| hazmat | gary_poster, so your delivering what to people to run the tests? | 13:13 |
| hazmat | its also unclear why you want the ssh to the bootstrap node on deploy, in this case.. it sounds like a regular environment, and an automation script to launch the test env and run the tests. | 13:15 |
| hazmat | and then possibly a separate front end script for the jenkins runner, that setups the keys and environments.yaml as needed | 13:16 |
| gary_poster | hazmat, python callable. Nice easy story is "set up your .juju config with a special environment named foo, configured to run tests whereever makes sense for you". Another nice part of that would be "configure that environment to not ask questions when you bootstrap". A less nice story is "and also modify your .ssh/config to ignore keys from AWS" | 13:16 |
| hazmat | so your saying its okay to modify ~/.juju for a user but not okay to modify ~/.ssh ? | 13:16 |
| hazmat | just have the script set $HOME | 13:16 |
| mgz | gary_poster: what exactly are you wanting to pass? just suppression of the known_hosts prompt or other bits too? | 13:17 |
| gary_poster | hazmat, ok to modify .juju and not .ssh: yes, because in .juju you can constrain everything to a specially named environment. .ssh is blanket for (for instance) AWS. | 13:18 |
| mgz | because juju should probably just integrate smoser's cloud-init method of verifying host keys | 13:18 |
| gary_poster | mgz: yes, just -o 'StrictHostKeyChecking=no' -o 'UserKnownHostsFile=/dev/null' or similar | 13:19 |
| hazmat | gary_poster, this is confusing in terms of the original question.. but the delivery script/callable can set $HOME and setup a private .ssh and .juju as needed | 13:19 |
| mgz | okay, so there's a better way of doing that, +/- the amazon delay on supplying console output | 13:20 |
| mgz | which is to acutally verify the key reported by the cloud-init during instance boot and insert it in ~/.ssh/known_hosts | 13:20 |
| gary_poster | hazmat, ack. Yes, confusing, I said deploy initially and meant bootstrap and took everything down the wrong rabbit hole, I'm sorry. | 13:21 |
| gary_poster | mgz, that sounds very nice | 13:21 |
| mgz | I use this for fully automated deployment of launchpad on canonistack | 13:21 |
| hazmat | mgz, so console output scraping for the host fingerprint? | 13:22 |
| gary_poster | hazmat, ack that $HOME would work. I'll propose it. mgz, that would have to happen in juju though, right? We, as juju clients, can't do that? | 13:22 |
| mgz | hazmat: yup, works really nicely with openstack, may be less good against aws | 13:22 |
| hazmat | kees also suggested it at uds.. the only issue with that on ec2 is the delay can be a bit atrocious.. | 13:23 |
| gary_poster | (The $HOME is not as easy to do as what I proposed, so as a user I'd still prefer the kind of automation mgz suggests) | 13:23 |
| mgz | so I heard 4 minute intervals on output update? and I guess sshd is generally up and waiting before the first 4 minutes | 13:23 |
| gary_poster | well before, often | 13:24 |
| hazmat | mgz, it can take longer, its quite variable | 13:24 |
| mgz | so adding wait time after boot wouldn't be great | 13:24 |
| hazmat | mgz, re output availability, i've hit 10m b4 | 13:24 |
| mgz | can poll at 5s intervals with openstack and get nearly live reporting :) | 13:25 |
| mgz | hazmat: I guess the alternative is a juju config option, to 1) prompt/die as currently 2) accept any host key blndly per gary's args 3) use smoser's trick | 13:26 |
| mgz | and leave which one up to the user | 13:26 |
| mgz | the current behaviour when connecting to a new host on an ip you've seen before is obnoxious | 13:27 |
| gary_poster | agreed | 13:27 |
| smoser | mgz, regarding ssh keys. one thing that would be completely possible for juju is what kirkland's cloud-sandbox does. | 13:27 |
| smoser | it generates ssh public/private key on the local system. passes those through user-data to cloud-init. | 13:28 |
| gary_poster | I think I'd be ok with waiting for this use case, fwiw | 13:28 |
| smoser | connects to the host expecting those credentials, and then destroys them and creates new ones. | 13:29 |
| smoser | so the ones that were sent insecurely via user-data are usable for less than a minute. | 13:29 |
| mgz | right, was going to say, putting the private key in user-data isn't ideal | 13:30 |
| smoser | but its fine if you destroy it right away. | 13:30 |
| smoser | and for everything but the bootstrap node, they can then post the public keys to the bootstrap node. | 13:30 |
| smoser | and your client can aske the bootstrap node for the new nodes public keys. | 13:30 |
| smoser | so you only need the dance once. | 13:30 |
| gary_poster | by which I mean, for this use case, I'd be ok with what you'd need to do for mgz's solution: waiting longer than usual for a trustable machine without having user interaction. For other usecases, less so. | 13:31 |
| jml | is it possible to tell juju to look at a different environments.yaml file? | 13:50 |
| hazmat | jml, not without $HOME manipulation.. juju does respect JUJU_ENV for named env usage | 13:54 |
| jml | hazmat: thanks. | 13:54 |
| jml | fwiw, we're experimenting with using juju to fire up our services for integration tests | 13:54 |
| jml | how can I tell if a local environment is bootstrapped? | 14:12 |
| jml | never mind. destroyed the wrong one :( | 14:13 |
| === zyga is now known as zyga-food | ||
| === al-maisan is now known as almaisan-away | ||
| === almaisan-away is now known as al-maisan | ||
| james_w | if you deploy a charm as serviceA, and then deploy it as serviceB, does it does the same copy of the charm? | 14:59 |
| james_w | i.e. if you modify the charm between the two deploys, would the second need -u to use the changes? | 14:59 |
| jml | is there a valid charm name regex? | 15:04 |
| === objectiveous_ is now known as objectiveous | ||
| mgz | what's the juju equivalent of: | 15:15 |
| mgz | trace.warning("some message to the user about something breaking") | 15:16 |
| mgz | trace.log_exception_quietly() | 15:16 |
| mgz | basically giving a nice clean error without traceback for common failures, but preserves the traceback in log/verbose mode | 15:17 |
| jml | can you use jitsu to watch for destroy-service to complete? | 15:26 |
| === zyga-food is now known as zyga | ||
| === salgado is now known as salgado-lunch | ||
| === al-maisan is now known as almaisan-away | ||
| === almaisan-away is now known as al-maisan | ||
| === al-maisan is now known as almaisan-away | ||
| imbrandon | http://i.imgur.com/x7zO0.jpg , that is all ... | 16:22 |
| hazmat | mgz log.warning, log.debug | 17:12 |
| hazmat | mgz with traceback module formatted exception | 17:12 |
| hazmat | into the debug | 17:13 |
| imbrandon | SpamapS: btw https://launchpad.net/~webstack/+archive/php/+packages | 17:19 |
| SpamapS | imbrandon: cool | 17:20 |
| imbrandon | i still like the idea of a charmers ppa, but thats for now | 17:21 |
| imbrandon | 386 built, amd64 still in queue | 17:21 |
| === salgado-lunch is now known as salgado | ||
| === zyga is now known as zyga-afk | ||
| SpamapS | hrm | 19:59 |
| SpamapS | there's something very broken with debug-hooks | 19:59 |
| SpamapS | sometimes debug-hooks just sits there, and no more hooks execute | 19:59 |
| imbrandon | yea and if you exit go luck ever running it again | 20:00 |
| imbrandon | on that environment | 20:00 |
| SpamapS | exactly | 20:00 |
| SpamapS | hazmat: ^^ debug-hooks seems to have something wrong.. flags not cleared or something | 20:01 |
| SpamapS | ugh, and restarting the unit agent made it worse | 20:09 |
| SpamapS | now no hooks run | 20:09 |
| SpamapS | even when debug-hooks isn't running | 20:09 |
| SpamapS | 2012-07-12 13:09:45,743 INFO Waiting for unit to come up. | 20:10 |
| SpamapS | ah.. you can't relaly restart the unit agent | 20:10 |
| imbrandon | man where is the creat project page | 20:12 |
| imbrandon | grrr | 20:12 |
| SpamapS | front page | 20:12 |
| SpamapS | first thing | 20:12 |
| imbrandon | oh the one place i no look | 20:12 |
| SpamapS | "Register a project" | 20:12 |
| _mup_ | Bug #966617 was filed: supply high level design documents <juju:New> <juju (Ubuntu):Triaged> <juju (Ubuntu Precise):Triaged> <juju (Ubuntu Quantal):Triaged> < https://launchpad.net/bugs/966617 > | 20:25 |
| === pdtpatrick_ is now known as pdtpatrick | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
