[07:17] <hloeung> bdmurray: delete the provider-state and bootstrap-verify files in the control bucket
[07:22] <vila> axw: hi there, still around ? (Replying to your mail while you pong ;)
[07:23] <axw> vila: hiya, I am
[07:25] <vila> axw: almost there
[07:27] <vila> axw: replied for the questions in the first part of the mail, that leaves us with the config you want me to try, let me re-phrase to make sure we're on the same page
[07:28] <axw> sure
[07:28] <vila> no , in fact, I can't ;) So, when you say "Could you
[07:28] <vila> try bootstrapping, then try to manually ssh with the options above and "-v"
[07:28] <vila> to see what keys ssh is trying?", what config and what options do you mean ?
[07:29] <axw> I mean try running the ssh command I pasted:  ssh -v -i ~/.juju/ssh/juju_id_rsa -i ~/.ssh/id_rsa -i ~/.ssh/id_dsa -i ~/.ssh/identity <host>
[07:29] <axw> I realise they're not relevant to you, but that's what juju's doing
[07:29] <vila> axw: I realize the config that was working doesn't make a lot of sense given your explanations, but it was working which blurry what I should try
[07:29] <vila> oh, right
[07:30] <axw> it *should* be working if you've got the private key loaded in ssh-agent
[07:30] <axw> those -i's are just so we try the defaults in case they're not also loaded into an agent
[07:30] <vila> so, restore a config with one key for chinstrap and a different one in authorized-keys-path ?
[07:31] <axw> yes please
[07:31] <vila> ha, loaded in the agent, hmm, hard to say if/when that was the case, let me setup that
[07:31] <axw> I will try the same to see if I can reproduce it
[07:31] <axw> you said in your email it was though?
[07:32] <axw> do you mean it was but maybe is not now?
[07:34] <vila> yeah, there is a grey area during which I suspect I upgraded juju to 1.17.2 while my node was still bootstrapped
[07:34] <vila> running $ juju bootstrap --show-log
[07:35] <vila> damn it it works
[07:36] <axw> doh :)
[07:36] <vila> and so does the ssh command above
[07:37] <axw> if it does fail again, just confirm that the agent has/n't got the key loaded
[07:37] <vila> ssh-add -l lists the key in authorized-keys-pat
[07:37] <vila> h
[07:38] <vila> axw: I will but it's a passwordless key so I think those don't have to be added to the agent, will re-try with a clean one
[07:39] <axw> vila: it does if you want ssh to try it. otherwise it needs to be specified with -i, or in ~/.ssh/config
[07:39] <vila> axw: well, it is specified in my ~/.ssh/config in the section that goes through chinstrap
[07:40] <axw> the authorized-keys-path one?
[07:40] <vila> yes
[07:41] <vila> re-trying after 'ssh-add -D'
[07:42] <vila> queried for my lp key
[07:43] <vila> still working
[07:43] <vila> queried while bootstrap I mean, trying your ssh command
[07:43] <vila> Host key verification failed.
[07:43]  * vila fixes known_hosts
[07:44] <axw> vila: just pass -o StrictHostKeyChecking=no
[07:44] <vila> ssh works too
[07:44] <axw> you'll get that a lot otherwise
[07:44] <vila> huh, OMG:
[07:45] <vila> apt-cache policy juju-core -> 1.16.3-0ubuntu0.13.10.1~ctools
[07:45] <vila> wth
[07:45] <vila> doh
[07:45] <axw> doh :)
[07:45] <vila> I'm on the machine-0 host ;)
[07:45] <vila> right, on my desktop: 1.17.2-0ubuntu1~ubuntu13.10.1~juju1
[07:46] <vila> from http://ppa.launchpad.net/juju/devel/ubuntu/ saucy/main amd64 Packages
[07:47] <axw> so if that's working, I'm really not sure why it wasn't working before
[07:49] <vila> axw: gee, me neither but it sure wasn't, there was an error initially (using one key in ~/.ssh/config and a different one in envs.yaml) but I did align and reproduced the error (without ssh-add -D though but I can't see the link). Then I used the same key and it worked
[07:50] <axw> I'll play around with my keys and see if I can reproduce it
[07:50] <vila> axw: let's call it unreproducible for now
[07:50] <vila> axw:  is there some place I can check for logs about some ssh -vv juju may have internally ?
[07:51] <vila> axw: or some way to pass -vv to juju and get some output ?
[07:51] <axw> vila: nope, stupidly I did not add logging for that bit
[07:51] <vila> axw: bah, I guess, using the ssh command (with nohostchecks) is close enough right ?
[07:52] <axw> yes, and the "-i" args
[07:52] <axw> they're important because they change behaviour of how default keys get picked up and so forth
[07:52] <vila> axw: sure, thanks for that one
[07:53] <vila> axw: yup, the commit msgs were very good to point me in that direction
[07:54] <vila> axw: I'm adding automated tests around bootstrap anyway, so I'll know if I run into that again
[07:54] <axw> great, thanks
[07:54] <vila> axw: thanks to you, enjoy your evening !
[07:55] <axw> cheers, and have a nice day :)
[08:37] <cargill> hi, after having some problems with an existing vagrant env, I removed it (removing Vagrantfile, ./vagrant, ~/.vagrant.d and ~/Virtualbox VMs), then set it up again and I cannot do juju ssh or debug-log anymore (in the freshly created environment)
[08:47] <cargill> this makes juju slightly useless and me stuck because I have no idea whatsoever how to find out what went wrong (if it is #1202682, I do not understand how sinzui divined that it should be an ssh issue, so I cannot say I'm suffering from that)
[08:47] <_mup_> Bug #1202682: debug-log doesn't work with lxc provider <cts-cloud-review> <debug-log> <local-provider> <papercut> <ssh> <ui> <juju-core:In Progress by themue> <https://launchpad.net/bugs/1202682>
[09:01] <cargill> oh, and /var/log/juju-setup.log logs a few different errors/warnings but I can still deploy (some) charms, so they might be inconsequential
[09:03] <TheMue> cargill: never worked with vagrant, the mentioned issue is when using the local provider
[09:03] <TheMue> cargill: but you could paste your setup log to have a look
[09:04] <cargill> TheMue: which the vagrant image sets up (it's a virtualbox machine that has juju+local set up)
[09:04] <cargill> documented here https://juju.ubuntu.com/docs/config-vagrant.html
[09:08] <TheMue> cargill: ah, based on lxc, so at least the ssh/debug-log problem could be the same. the local provider doesn't uses an explicit container for state and api. so ssh doesn't work here.
[09:10] <TheMue> cargill: but the log exists at $JUJU_HOME/<ENVNAME>/log/all-machines.log
[09:18] <cargill> if that's the case, that would explain debug-log not being able to run, but would not explain the failure to ssh to other containers
[09:21] <cargill> this is juju-setup.log, this time for a precise vagrant box http://pastebin.com/zJ88WR6T
[09:34] <cargill> TheMue: there is no all-machines.log file anywhere on the box
[09:36] <TheMue> cargill: hmm, so it is more different from local provider than thought
[09:37] <cargill> if I understand the documentation, it should use a vanilla local provider, so maybe something is horribly broken, just not showing in the juju-setup.log?
[09:39] <cargill> yet nothing matching /error/i in the syslog
[09:40] <cargill> or anywhere relevant in /var/log
[09:43] <cargill> must be something wrong with my setup (I wish juju just worked on Debian directly without an extra level of virtualization)
[09:58] <cargill> Any thoughts on how to find out what's wrong/how to work around this?
[10:11] <cargill> funny, I can "juju ssh postgresql/0" but not "juju ssh 2" where 2 is the machine that postgresql/0 uses
[11:18] <cargill> why does postgres node see connections from another node as coming from 1.0.3.1 instead of the node's IP (10.0.3.86)?
[11:31] <dimitern> cargill, it seems you're using the local provider - its ip addresses are like 10.0.3.x, so this could be the host machine's ip in the lxc bridge
[11:32] <cargill> yes it is
[11:32] <dimitern> cargill, so that's why - the host's ip is also the default route on the lxc subnet
[11:35] <cargill> dimitern: this makes relations to postgres not work, since only the other node's IP gets whitelisted, not the host's IP
[11:35] <cargill> is this a bug?
[11:46] <dimitern> cargill, container networking is still not implemented, so you might expect issues like this
[11:46] <cargill> ah, ok, so a known missing feature and need to work around that, fine with me, is there a list? :)
[11:47] <dimitern> cargill, and manual steps might be needed for charms to work, like setting config options etc. from the command line
[11:48] <dimitern> cargill, juju-dev@lists.ubuntu.com is the place to raise questions or the #juju-dev channel here on freenode, but reporting bugs is always welcome - just please make sure you do a search first to see if what you're seeing is not already reported
[11:49] <cargill> I mean a list of quirks to watch for in a "local" deployment
[11:51] <dimitern> there's a list of known bugs - https://bugs.launchpad.net/juju-core/+bugs?field.tag=local-provider
[12:07] <cargill> dimitern: thanks
[13:28] <melmoth> juju people: what happen in a charm when a "juju destroy-unit" is called ?
[13:33] <marcoceppi> melmoth: faily certain that the unit is marked as dying, then any relations that exist go through the -relation-departed/broken cycle, then stop hook is called, then it's removed from topology
[15:06] <rick_h_> marcoceppi: I'm trying to run two local envs and I see it noting storage ports and such but I'm getting an error ERROR cannot use 37017 as state port, already in use
[15:06] <rick_h_> when I try to bring up the second local env
[15:07] <rick_h_> is there a config bit for the env.yaml for that? I don't see a reference in the docs
[15:07] <marcoceppi> rick_h_: that's annoying, what version juju?
[15:07] <rick_h_> 1.17.2-trusty-amd64
[15:07] <marcoceppi> rick_h_: there's like three things you need to do, iirc, and they're not really well documented anywhere
[15:07] <marcoceppi> thumper: what do you need to run multiple local envs?
[15:08] <rick_h_> marcoceppi: ok, good to know it's not just me missing the boat then.
[15:08] <rick_h_> marcoceppi: he'll be afk, I can check with him tonight. I need to get some instructions written up for demos for sales folks
[15:08] <marcoceppi> rick_h_: yeah, let me go poke at it real quick see if I can't jog my memory
[15:08] <marcoceppi> rick_h_: if you do, please pass those notes on to me so I can get them in the docs
[15:08] <rick_h_> marcoceppi: k, appreciate it if you know, but no rush. I've got a couple of days to get things written out/tested
[15:09] <rick_h_> marcoceppi: rgr
[18:35] <stokachu> is there a decent application for testing websockets in the console?
[18:36] <stokachu> like i want to test the connection to the juju api with json parameters
[19:03] <marcoceppi> Does anyone know if you can deploy a subordinate to a subordinate