[02:06] <thumper> babbageclunk: ping
[02:07] <thumper> anastasiamac: ping
[02:08] <anastasiamac> thumper: ?
[02:09] <thumper> anastasiamac: quick review? https://github.com/juju/bundlechanges/pull/38
[02:09] <anastasiamac> thumper: sure, can i trade u for this one https://github.com/juju/juju/pull/8692?
[02:09] <thumper> sure
[02:10] <thumper> approved
[02:11]  * anastasiamac still looking  :)
[02:11] <babbageclunk> thumper: oops looks like I'm too late?
[02:11] <thumper> babbageclunk: slacker
[02:11] <thumper> babbageclunk: yeah, I think anastasiamac has it in hand
[02:12] <thumper> babbageclunk: although, if you like, you can do the next one: https://github.com/juju/juju/pull/8692#pullrequestreview-119300874
[02:12] <anastasiamac> thumper: approved too :) m sure it was result of overthingking ;) and anticipating
[02:12] <thumper> no, that's not it
[02:12] <thumper> babbageclunk: https://github.com/juju/juju/pull/8693
[02:12] <babbageclunk> ok, I'll look after this one from kelvinliu_
[02:13] <thumper> anastasiamac: yeah, while testing the pr that babbageclunk is going to look at, I realised the mistake I made
[02:13] <thumper> so going back to fix the mistake,
[02:13] <thumper> then update the juju PR with the new bundlechanges
[02:14] <anastasiamac> :) yeah, that was the difficulty - to get all the permutations right: u have the machines and user specified placement; u have machines but user did not specify placement (or specified different placement); and the equivalent for it when u don't have machines... :)
[02:15] <anastasiamac> thumper: thnx for doing the hard work - thinking ;)
[02:15] <thumper> :)
[02:15] <thumper> it makes a change...
[02:27] <babbageclunk> thumper: approved
[02:47] <anastasiamac> veebers: trying to merge a PR and it failed with godeps? http://ci.jujucharms.com/job/github-merge-juju/435/console
[02:47] <anastasiamac> shall i just re-try?
[02:47]  * veebers looks
[02:47] <anastasiamac> veebers: fwiw, it was just a helpdoc text, so no deps were touched...
[02:48] <veebers> anastasiamac: odd failed n "fatal: unable to access 'https://gopkg.in/retry.v1/': Could not resolve host: gopkg.in" let me have a look
[02:49] <anastasiamac> veebers: k, while u look, i'll re-try :)
[02:49] <veebers> anastasiamac: don't retry
[02:49] <veebers> anastasiamac: I'm looking why it can't hit that host :-)
[02:49] <anastasiamac> veebers: k
[02:49] <anastasiamac> veebers: the check suceeded tho, only merge failed on that PR...
[02:50] <veebers> anastasiamac: yep, the host that runs the merge/check is having trouble hitting that url, so any other attempt will fail
[02:51] <anastasiamac> veebers: ack
[02:51]  * anastasiamac waits for green light from veebers before hitting the big red button
[03:03] <veebers> anastasiamac: huh maybe it can hit it at the moment, might have been a hiccup at the time? not sure will follow up with IS.
[03:04] <veebers> anastasiamac: I did take a momement to fix the script so it checks the error code properly, so we should see "2 unknown command" in the output now ^_^
[03:04] <veebers> anastasiamac: feel free to re-try a run
[03:20] <anastasiamac> veebers: ta
[03:41] <thumper> babbageclunk: thanks... but I have found another problem
[03:41]  * thumper sighs
[04:01] <veebers> anastasiamac: any joy?
[09:10] <magicaltrout> hello folks
[09:10] <magicaltrout> juju manual deployment
[09:10] <magicaltrout> we've juju add-user'd
[09:10] <magicaltrout> but juju ssh fails
[09:10] <magicaltrout> with permission denied
[09:10] <magicaltrout> even though keys are added and
[09:10] <magicaltrout> ssh ubuntu@<server> works
[09:15] <magicaltrout> https://asciinema.org/a/n3TN4kTmH59dZQ8fWMKA2Yncx <- kjackal admcleod_ any ideas?
[09:17] <kjackal> magicaltrout: could it be the shell of the user you are adding is not set to bash?
[09:18] <magicaltrout> what are these words you are typing?
[09:18] <kjackal> can you run any other command via ssh?
[09:18] <magicaltrout> eh?
[09:18] <kjackal> looking for docs
[09:19] <kjackal> magicaltrout: ssh user1@server1 date
[09:19] <kjackal> would this work?
[09:19] <kjackal> actually I am not sure if that should work
[09:20] <magicaltrout> well when you add-user whod does it try and authenticate as?
[09:20] <magicaltrout> I assume it still tries to login to ubuntu@
[09:22] <kjackal> can you make sure that in /etc/passwd the last field of the user you are trying to connect with is not /bin/nologin or /bin/false
[09:22] <kjackal> looks like this: https://stackoverflow.com/questions/608533/ssh-to-debian-server-instantly-logs-out?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa
[09:23] <kjackal> magicaltrout: ^
[09:23] <magicaltrout> eh, you can see from the screenrecording that you can ssh into the box successfully
[09:23] <magicaltrout> as ubuntu user
[09:23] <magicaltrout> and from there I can do whatever I want
[09:23] <magicaltrout> sudo -i etc
[09:24] <magicaltrout> the ssh key for ubuntu@ is also my default ssh key so I don't really need to send the key i'm just making mirror what I tried with juju ssh
[09:26] <kjackal> ah sorry did not read the "$ logout"
[09:27] <kjackal> magicaltrout: I think there is a juju ssh-add-key or something... let me search for it
[09:27] <magicaltrout> the juju ssh debug though tells me nothing about the ssh command its actually trying to run
[09:27] <magicaltrout> yeah kjackal thats how our keys ended up in /home/ubuntu/.ssh/authorized_keys
[09:27] <magicaltrout> thats all working
[09:27] <magicaltrout> but when it tries to ssh its doing something wrong
[09:27] <magicaltrout> but debug shows you f all
[09:28] <magicaltrout> juju ssh 0 -i ~/key
[09:28] <magicaltrout> should in theory at least pump in the correct key
[09:29] <kjackal> the "juju ssh-keys" seems right to you?
[09:29] <magicaltrout> yeah works fine
[09:29] <magicaltrout> both mine and my colleagues keys got added to the ubuntu user
[09:29] <magicaltrout> which is why we can manually ssh into any box in the juju cluster
[09:30] <magicaltrout> juju ssh isn't really the problem
[09:30] <magicaltrout> we can't
[09:30] <magicaltrout> juju run stuff
[09:30] <magicaltrout> and thats a pain in the balls
[09:31] <kjackal> magicaltrout: so you got something deployed to the machine and you cannot juju run
[09:32] <magicaltrout> I can't
[09:32] <magicaltrout> nor can I ssh
[09:32] <magicaltrout> or anything that requires any form of ssh access
[09:32] <magicaltrout> https://github.com/juju/juju/blob/develop/cmd/juju/commands/ssh.go looking at this
[09:32] <magicaltrout> you'd have thought
[09:32] <magicaltrout> juju ssh ubuntu@0 -i ~/.ssh/id_rsa.pub
[09:32] <magicaltrout> would work
[09:32] <magicaltrout> becuase i've both told it the user and the key I want to use
[09:33] <magicaltrout> which is no different to
[09:33] <magicaltrout> ssh ubuntu@<ip> -i ~/.ssh/id_rsa.pub
[09:33] <kjackal> juju run --all --debug -- hostname -f
[09:33] <magicaltrout> but that still tells me permission denied
[09:33] <magicaltrout> ERROR permission denied (unauthorized access)
[09:33] <magicaltrout> 10:33:30 DEBUG cmd supercommand.go:459 error stack:
[09:33] <magicaltrout> permission denied (unauthorized access)
[09:33] <magicaltrout> github.com/juju/juju/rpc/client.go:149:
[09:33] <magicaltrout> github.com/juju/juju/api/apiclient.go:924:
[09:44] <magicaltrout> and clearly when i say .pub i'm lying
[09:57] <magicaltrout> alos
[09:57] <magicaltrout> also
[09:58] <magicaltrout> if I look in the target servers auth.log
[09:58] <magicaltrout> it doesn't even log a failed login attempt
[09:58] <magicaltrout> amazeballs
[10:36] <magicaltrout> just did the samething with a lxd local deploy kjackal
[10:36] <magicaltrout> same shit happens
[10:37] <magicaltrout> I was pondering whether tthe permission error is actually the fact the snap can't see the key?
[10:37] <kjackal> which snap?
[10:37] <kjackal> juju?
[10:38] <magicaltrout> yeah
[10:38] <kjackal> I think snaps cannot see hidden directories (starting with .)
[10:38] <magicaltrout> jesus h christ
[10:39] <magicaltrout> hmm
[10:39] <magicaltrout> doesn't seem to make a difference
[10:39] <magicaltrout> copying the key out
[10:40] <magicaltrout> basically
[10:40] <magicaltrout> juju ssh works for the admin user with the embedded key
[10:40] <magicaltrout> but doesn't work for any add-user
[11:52] <rick_h_> magicaltrout: so you added the user and then added their keys?
[11:52] <rick_h_> magicaltrout: juju [add,import]-ssh-key
[11:54] <magicaltrout> I did rick_h_
[11:54] <magicaltrout> you can replicate this on lxd local
[11:55] <magicaltrout> as 1 user bootstrap
[11:55] <magicaltrout> then add another local user as a juju user
[11:55] <magicaltrout> and then try and ssh post add-key as that other user
[11:55] <magicaltrout> the admin ssh works
[11:55] <magicaltrout> the other user doesn't
[11:56] <rick_h_> magicaltrout: huh...ok. I was just using that the other day. I'll wonder if something broke
[11:56] <magicaltrout> of course.... I'll caveat all of this with, it might just be my misunderstanding
[11:56] <magicaltrout> rick_h_: for us its been broken for 6 months
[11:56] <magicaltrout> which is why it might just be me :)
[11:56] <rick_h_> magicaltrout: and the user has been granted access to the model with admin access?
[11:56] <magicaltrout> yeah
[11:56] <magicaltrout> juju status works
[11:56] <magicaltrout> juju ssh/run doesn't
[11:56] <rick_h_> magicaltrout: what's juju show-model have?
[11:56] <magicaltrout> but ssh ubuntu@<machine> for the added user does work
[11:57] <rick_h_> magicaltrout: and can you check the .ssh/authorized_keys to see if the key made it?
[11:57] <rick_h_> magicaltrout: I guess and is this with import-ssh-key or add-ssh-key? (/me used import recently)
[11:57] <magicaltrout> we know it made it in rick_h_
[11:57] <magicaltrout> because you can manually ssh using the key you added
[11:58] <rick_h_> oh...so the key made it to the machines?
[11:58] <magicaltrout> yeah
[11:58] <magicaltrout> https://pastebin.com/rZEfsuAs
[12:00] <magicaltrout> we have the same issue on 2 laptops rick_h_ one Xenial one Bionic both with the snapped juju installed
[12:00] <rick_h_> magicaltrout: k, I'm looking at the email you sent
[12:01] <magicaltrout> ignore the fact i tried to pump in my public key ;)
[12:05] <rick_h_> magicaltrout: is there a proxy or anything involved?
[12:05] <magicaltrout> nope
[12:05] <rick_h_> magicaltrout: the code from your tracebacks come out into code checking proxy details before any ssh stuff is run
[12:08] <magicaltrout> rick_h_: the fact the same thing happens on lxd local suggests proxies are a red herring
[12:08] <rick_h_> magicaltrout: yea, but looking at what's giving the permission denied error in that trace is https://github.com/juju/juju/blob/b47854c63ad6345ffd4642e3b22520b43068ef56/cmd/juju/commands/ssh_common.go#L256
[12:10] <rick_h_> magicaltrout: k, will have to basically go the bug route and figure out wtf. I'm not sure at this point.
[12:10] <magicaltrout> just checking/recording lxd local again as a reproduction path
[12:11] <rick_h_> magicaltrout: check pm
[12:14] <magicaltrout> rick_h_: https://asciinema.org/a/JIlCubAF7LBdg6cc86ZcKoBLU
[12:14] <magicaltrout> there's a full LXD example
[12:14] <rick_h_> wheeee
[12:17]  * rick_h_ is watching the movie 
[12:17] <rick_h_> oooohhhhhh
[12:17] <rick_h_> superuser != model level access with all machines
[12:17] <rick_h_> that's why even as superuser if you want to see all models you have to use --all
[12:18] <rick_h_> try granting admin on the model directly
[12:18] <rick_h_> magicaltrout: ^
[12:19] <magicaltrout> oooh
[12:19] <magicaltrout> there we go
[12:19] <rick_h_> magicaltrout: yea, so superuser is kind of special. Just because you can do anything doesn't mean you want your keys on every machine in the controller and such
[12:19] <rick_h_> magicaltrout: just like you don't want to see every model from every user on the controller
[12:19] <rick_h_> magicaltrout: even though technically, you can gain access to them all as it's YOUR controller
[12:20] <magicaltrout> brain melting... so superusers, whilst superusers don't actually posess any model level superpowers
[12:21] <rick_h_> magicaltrout: it's kind of like sudo, technically you can `sudo dosomething` at any time
[12:21] <rick_h_> magicaltrout: but as you run around the system you get access denied until you sudo
[12:21] <magicaltrout> thanks rick_h_
[12:21] <rick_h_> magicaltrout: it's a bit of a corner world, but kind of done for the controller admin's sanity
[12:22] <rick_h_> magicaltrout: ty for the video, that helped
[12:22] <magicaltrout> no probs rick_h_ thanks for that, hours of confusion solved in moments ;)
[12:25] <rick_h_> magicaltrout: think on it and if you think Juju is doing the wrong thing let's chat.
[12:25] <rick_h_> magicaltrout: clearly we can error better at the minimum
[12:26] <rick_h_> but I'm nervous about making it "just work" though
[12:27] <magicaltrout> no i get the ethos behind it, there could possibly do with some more info in the --help output or something because its not obvious
[14:01] <bobeo> o/
[14:01] <bobeo> hey magicaltrout Im having issues registering to a controller, can you provide me a sample command for registering to one?
[14:02] <bobeo> i tried juju register <controllerIP> as well as juju register <controllerName>
[14:09] <rick_h_> bobeo: so there's a command that gets created you have to use that has some auth/security bits baked into it
[14:10]  * rick_h_ wonders if we ever got around to that bug that you couldn't re-get the register command
[14:11] <TheAbsentOne> is the postgresql charm author in this irc by any chance?
[14:14] <rick_h_> TheAbsentOne: sometimes, but he's in APAC region so timezones is a bit off usually
[14:14] <rick_h_> TheAbsentOne: best thing is to file bugs or something. What are you looking for?
[14:14] <rick_h_> or maybe the mailing list, async coms ftw
[14:14] <TheAbsentOne> ah I see, you know his name? rick_h_
[14:15] <rick_h_> stub: is our pgsql charm hero
[14:16] <TheAbsentOne> rick_h_ not really a bug. You see I created a charm that acts as a proxy and all data (connection details for postgres) are shared nicely BUT it is my proxy charm that is the one who requests this database. So in the auth file of postrges he is allowed but my real "consumer" isn't. I'm looking for a way to make sure both my consumer and proxy charm are both inserted in the auth file
[14:17] <rick_h_> TheAbsentOne: hmmm, that's sticky. you might have to set config or something and set the config for the extra_hbaconf or whatever that config value is
[14:17] <TheAbsentOne> the pgsql has a set_roles function which I'm looking into now but if that doesn't work out I have a big problem
[14:18] <rick_h_> there's a config I know I've used to manually add in entries
[14:18] <bobeo> rick_h_: can you share this magical command? o.O
[14:18] <TheAbsentOne> rick_h_ could you explain it a bit more what you mean? I kinda don't want to edit the pgsql charm or interface
[14:18] <rick_h_> TheAbsentOne: https://jujucharms.com/postgresql/#charm-config-extra_pg_conf
[14:19] <rick_h_> bobeo: so when you run juju add-user it spits out a register command with some hashed bits around the ip, a security code, etc.
[14:19] <rick_h_> bobeo: the command is meant to be sent to the user for them to use to register
[14:19] <TheAbsentOne> rick_h_: hey that might work thanks man
[14:19] <bobeo> rick_h_: how do I generate that? Also, I have a user that exists on the controller already, I need to access the controller itself
[14:19] <rick_h_> bobeo: and prevents just general brute force hacking against the register call since you'll never have the right bits
[14:19] <rick_h_> bobeo: run juju add-user xxxx
[14:20] <rick_h_> bobeo: right, so I tend to create users when I connect from a diff machine to be honest. We've not setup a good way to dump/import the config bits into the .local/share/juju/...
[14:21] <TheAbsentOne> argh I don't think it will work rick_h_ it's only for postgres specific options, not for the host authentication
[14:22] <rick_h_> TheAbsentOne: right, but this config allows you to add the user/ip address allowed to connect
[14:22] <TheAbsentOne> but there is another one: extra_pg_auth <-- this might work
[14:22] <rick_h_> TheAbsentOne: that's the one I linked to
[14:22] <rick_h_> oh, sorry
[14:22] <bobeo> rick_h_:  just to confrim, to grant privs, its juju grant <username> --controller <controllername> super-user correct?
[14:22] <rick_h_> TheAbsentOne: actually my bad, that's the one I MEANT to link to
[14:23] <TheAbsentOne> rick_h_: my bad in not looking but thanks man ^^
[14:23] <rick_h_> bobeo: correct
[14:23] <bobeo> rick_h_: its not working x...x
[14:23] <rick_h_> bobeo: oh sorry, superuser
[14:23] <rick_h_> bobeo: one work, no dash
[14:23] <rick_h_> one word
[14:23] <bobeo> rick_h_: ok, so i am losing my mind
[14:23] <bobeo> rick_h_: its taking so long to load :(
[14:24] <rick_h_> bobeo: sorry, I'm jumping around too much.
[14:24] <bobeo> process? whats te correct term there? does it load, or "process" a command query?
[14:24] <rick_h_> bobeo: what's taking long to load? the grant command?
[14:24] <bobeo> because technically you load it ot the controlelr before it runs
[14:24] <bobeo> yea the grantocmmand rick_h_
[14:24] <bobeo> it keeps hanging
[14:24] <rick_h_> bobeo: k, try with --debug to see if you can get a hint on what's hanging
[14:25] <rick_h_> bobeo: usually that's pretty darn fast as long as the client can reach the controller IP
[14:25] <bobeo> rick_h_: its a lan communication
[14:25] <rick_h_> bobeo: https://docs.jujucharms.com/2.3/en/tut-users is the short tutorial on the sharing/access stuff if you've not peeked yet
[14:26] <bobeo> rick_h_: im an idiot. forgive me, for surely my caffiene hasnt made it to work yet
[14:26] <bobeo> was trying to send it to a non-existant controller
[14:27] <rick_h_> bobeo: ooooohhhh, yea that'll wait a while and make itself a sandwhich waiting for a timeout
[14:27] <TheAbsentOne> rick_h_ is it okay to fill an option from within a charm? Is there a best-practice for this, how do you actually do that?
[14:27] <bobeo> rick_h_: so I have access to the correct controller now, with login privs, but I dont see the models
[14:27] <rick_h_> TheAbsentOne: heh not usually because the charms don't normally get to change config on other charms
[14:27] <rick_h_> bobeo: right, so because you're superuser you don't see all models by default, only those shared with you directly
[14:28] <rick_h_> bobeo: run `juju models --all`
[14:28] <rick_h_> bobeo: and if you want to have them be "my models" you can grant yourself admin privs on them
[14:28] <rick_h_> bobeo: so that admins don't get a super long list of all models from all users on the controller all the time, and can mange their own list of stuff they work on directly
[14:28] <TheAbsentOne> richt thought so x) welp
[14:28] <TheAbsentOne> right*
[14:33] <manadart> I have borked something. "lxc list" hangs and Juju can not contact my LXD controllers.
[14:33] <rick_h_> magicaltrout: try lxc list with -v
[14:33] <rick_h_> err manadart ^
[14:33]  * rick_h_ can't type today evidently...when did monday get here
[14:34] <manadart> rick_h_: Also hanging with no output...
[14:36] <aceng> Hello anyone know if I used the same physical server to install nova controller and keystone why the keystone configuration get overwritten when the nova relations is complete.  Do I have to install with separate containers to prevent this?
[14:37] <rick_h_> break all the things! manadart so searching around finds various bugs/debugging tracks with lxc-monitord and such causing hangs that involve a bunch of killing of things and getting them to restart.
[14:37] <bdx> aceng: try keystone in a container, nova-compute on the host to work around this when using a single machine
[14:38] <manadart> There's a beer in my fridge whispering "Leave it till Monday" seductively to me :)
[14:38] <aceng> okay I guess there is no workaround to preserve the apache and haproxy config when each relation is updating.
[14:38] <rick_h_> manadart: lol, sounds like good advice
[14:38] <rick_h_> manadart: listen to the beer
[14:40] <manadart> rick_h_: You're the boss.
[14:40] <rick_h_> manadart: :) have a good weekend
[15:05] <stub> TheAbsentOne: You might want to look at the pgbouncer charm, which is not nice, but does something similar. It uses an administrative connection to PostgreSQL so it can create its own users.
[15:05] <stub> TheAbsentOne: The trick is that if your proxy uses the db-admin relation to PostgreSQL, it is allowed to connect to any database as any user.
[15:08] <stub> TheAbsentOne: Its the only way I can see it working with the current charm; there is no protocol for a client charm to get the PostgreSQL service to do complex config changes like insert entries into pg_hba.conf
[15:10] <stub> Using the extra_ charm config options on the PostgreSQL charm would work, but is a poor UI since the operator has to set the config options to the magic values rather than the client charm handle it automatically.
[15:15] <TheAbsentOne> hi stub, thanks for the feedback. Currently I used the db instead of db-admin relation actually, maybe I should try first but I don't think it will make a lot of difference. The point is that my proxy itself doesn't care to connect to postgresql it's the consumer of my proxy that needs to do this.
[15:15] <TheAbsentOne> I will take a look at pgbouncer!
[15:16] <TheAbsentOne> But it's idd a problem that I need to perform this manual option config step
[15:18] <stub> If is just a broker rather than an actual proxy, you may be stuck with the PostgreSQL charm config.
[15:18] <stub> Because as you have found, the IP address will be denied access and you can't request it to be opened.
[15:19] <stub> The feature would need to be added.
[15:20] <stub> (It used to be possible to lie about your IP address, but the newer networking stuff thankfully removed that hack)
[15:20] <TheAbsentOne> yeah it needs the entry in the pg_hba.conf file to connect properly then I can connect with adminer idd stub
[15:21] <TheAbsentOne> stub: wouldn't it be a possibility to add an optional parameter to the set_database function that takes a list of hosts that also gain access to the database?
[15:21] <TheAbsentOne> I think if that feature got implemented the problem is solved
[15:23] <stub> Yes, it would be possible to add that feature. A comma separated list of CIDRs. It wouldn't work cross-model, since the client won't know what IP address the server will see.
[15:25] <stub> It needs a good hard think, to be sure it doesn't open any security escalation issues. Gut feeling says it is ok.
[15:25] <TheAbsentOne> not sure if I understand why it wouldn't work cross-model, but if works and is fine within the same model it would already be a huge and interesting thing, give it a thought stub ;)
[15:26] <TheAbsentOne> I can only pray hehe, it kinda sucks I have to request it though, it was kinda a requirement for me to use existing charms without modification.
[15:27] <stub> It would probably work out of the box if you connected to pgbouncer instead of postgresql. pgbouncer handles its authorization differently, and doesn't restrict by IP address
[15:28] <stub> And you want pgbouncer in any non-trivial production deploys anyway
[15:29] <TheAbsentOne> allright interesting, I will look into that stub
[17:14] <roadmr> hey juju friends
[17:15] <roadmr> I have a sick juju env, this appears in logs for all services : INFO juju.worker.dependency engine.go:352 "uniter" manifold worker stopped: failed to initialize uniter for "$SOME_UNIT" : cannot create relations: tomb: dying. How can I fix this?
[17:36] <bobeo_> o/
[19:11] <bobeo_> o/
[19:12] <bobeo_> anyone on got any experience with graylog? im digging through it right now and having issues.
[21:40] <kwmonroe> hey bobeo_, i have experience with the graylog charm. what's going on?