/srv/irclogs.ubuntu.com/2018/05/11/#juju.txt

thumperbabbageclunk: ping02:06
thumperanastasiamac: ping02:07
anastasiamacthumper: ?02:08
thumperanastasiamac: quick review? https://github.com/juju/bundlechanges/pull/3802:09
anastasiamacthumper: sure, can i trade u for this one https://github.com/juju/juju/pull/8692?02:09
thumpersure02:09
thumperapproved02:10
* anastasiamac still looking :)02:11
babbageclunkthumper: oops looks like I'm too late?02:11
thumperbabbageclunk: slacker02:11
thumperbabbageclunk: yeah, I think anastasiamac has it in hand02:11
thumperbabbageclunk: although, if you like, you can do the next one: https://github.com/juju/juju/pull/8692#pullrequestreview-11930087402:12
anastasiamacthumper: approved too :) m sure it was result of overthingking ;) and anticipating02:12
thumperno, that's not it02:12
thumperbabbageclunk: https://github.com/juju/juju/pull/869302:12
babbageclunkok, I'll look after this one from kelvinliu_02:12
thumperanastasiamac: yeah, while testing the pr that babbageclunk is going to look at, I realised the mistake I made02:13
thumperso going back to fix the mistake,02:13
thumperthen update the juju PR with the new bundlechanges02:13
anastasiamac:) yeah, that was the difficulty - to get all the permutations right: u have the machines and user specified placement; u have machines but user did not specify placement (or specified different placement); and the equivalent for it when u don't have machines... :)02:14
anastasiamacthumper: thnx for doing the hard work - thinking ;)02:15
thumper:)02:15
thumperit makes a change...02:15
babbageclunkthumper: approved02:27
anastasiamacveebers: trying to merge a PR and it failed with godeps? http://ci.jujucharms.com/job/github-merge-juju/435/console02:47
anastasiamacshall i just re-try?02:47
* veebers looks02:47
anastasiamacveebers: fwiw, it was just a helpdoc text, so no deps were touched...02:47
veebersanastasiamac: odd failed n "fatal: unable to access 'https://gopkg.in/retry.v1/': Could not resolve host: gopkg.in" let me have a look02:48
anastasiamacveebers: k, while u look, i'll re-try :)02:49
veebersanastasiamac: don't retry02:49
veebersanastasiamac: I'm looking why it can't hit that host :-)02:49
anastasiamacveebers: k02:49
anastasiamacveebers: the check suceeded tho, only merge failed on that PR...02:49
veebersanastasiamac: yep, the host that runs the merge/check is having trouble hitting that url, so any other attempt will fail02:50
anastasiamacveebers: ack02:51
* anastasiamac waits for green light from veebers before hitting the big red button02:51
=== ec0 is now known as jhebden
veebersanastasiamac: huh maybe it can hit it at the moment, might have been a hiccup at the time? not sure will follow up with IS.03:03
veebersanastasiamac: I did take a momement to fix the script so it checks the error code properly, so we should see "2 unknown command" in the output now ^_^03:04
veebersanastasiamac: feel free to re-try a run03:04
anastasiamacveebers: ta03:20
thumperbabbageclunk: thanks... but I have found another problem03:41
* thumper sighs03:41
veebersanastasiamac: any joy?04:01
=== frankban|afk is now known as frankban
magicaltrouthello folks09:10
magicaltroutjuju manual deployment09:10
magicaltroutwe've juju add-user'd09:10
magicaltroutbut juju ssh fails09:10
magicaltroutwith permission denied09:10
magicaltrouteven though keys are added and09:10
magicaltroutssh ubuntu@<server> works09:10
magicaltrouthttps://asciinema.org/a/n3TN4kTmH59dZQ8fWMKA2Yncx <- kjackal admcleod_ any ideas?09:15
kjackalmagicaltrout: could it be the shell of the user you are adding is not set to bash?09:17
magicaltroutwhat are these words you are typing?09:18
kjackalcan you run any other command via ssh?09:18
magicaltrouteh?09:18
kjackallooking for docs09:18
kjackalmagicaltrout: ssh user1@server1 date09:19
kjackalwould this work?09:19
kjackalactually I am not sure if that should work09:19
magicaltroutwell when you add-user whod does it try and authenticate as?09:20
magicaltroutI assume it still tries to login to ubuntu@09:20
kjackalcan you make sure that in /etc/passwd the last field of the user you are trying to connect with is not /bin/nologin or /bin/false09:22
kjackallooks like this: https://stackoverflow.com/questions/608533/ssh-to-debian-server-instantly-logs-out?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa09:22
kjackalmagicaltrout: ^09:23
magicaltrouteh, you can see from the screenrecording that you can ssh into the box successfully09:23
magicaltroutas ubuntu user09:23
magicaltroutand from there I can do whatever I want09:23
magicaltroutsudo -i etc09:23
magicaltroutthe ssh key for ubuntu@ is also my default ssh key so I don't really need to send the key i'm just making mirror what I tried with juju ssh09:24
kjackalah sorry did not read the "$ logout"09:26
kjackalmagicaltrout: I think there is a juju ssh-add-key or something... let me search for it09:27
magicaltroutthe juju ssh debug though tells me nothing about the ssh command its actually trying to run09:27
magicaltroutyeah kjackal thats how our keys ended up in /home/ubuntu/.ssh/authorized_keys09:27
magicaltroutthats all working09:27
magicaltroutbut when it tries to ssh its doing something wrong09:27
magicaltroutbut debug shows you f all09:27
magicaltroutjuju ssh 0 -i ~/key09:28
magicaltroutshould in theory at least pump in the correct key09:28
kjackalthe "juju ssh-keys" seems right to you?09:29
magicaltroutyeah works fine09:29
magicaltroutboth mine and my colleagues keys got added to the ubuntu user09:29
magicaltroutwhich is why we can manually ssh into any box in the juju cluster09:29
magicaltroutjuju ssh isn't really the problem09:30
magicaltroutwe can't09:30
magicaltroutjuju run stuff09:30
magicaltroutand thats a pain in the balls09:30
kjackalmagicaltrout: so you got something deployed to the machine and you cannot juju run09:31
magicaltroutI can't09:32
magicaltroutnor can I ssh09:32
magicaltroutor anything that requires any form of ssh access09:32
magicaltrouthttps://github.com/juju/juju/blob/develop/cmd/juju/commands/ssh.go looking at this09:32
magicaltroutyou'd have thought09:32
magicaltroutjuju ssh ubuntu@0 -i ~/.ssh/id_rsa.pub09:32
magicaltroutwould work09:32
magicaltroutbecuase i've both told it the user and the key I want to use09:32
magicaltroutwhich is no different to09:33
magicaltroutssh ubuntu@<ip> -i ~/.ssh/id_rsa.pub09:33
kjackaljuju run --all --debug -- hostname -f09:33
magicaltroutbut that still tells me permission denied09:33
magicaltroutERROR permission denied (unauthorized access)09:33
magicaltrout10:33:30 DEBUG cmd supercommand.go:459 error stack:09:33
magicaltroutpermission denied (unauthorized access)09:33
magicaltroutgithub.com/juju/juju/rpc/client.go:149:09:33
magicaltroutgithub.com/juju/juju/api/apiclient.go:924:09:33
magicaltroutand clearly when i say .pub i'm lying09:44
magicaltroutalos09:57
magicaltroutalso09:57
magicaltroutif I look in the target servers auth.log09:58
magicaltroutit doesn't even log a failed login attempt09:58
magicaltroutamazeballs09:58
magicaltroutjust did the samething with a lxd local deploy kjackal10:36
magicaltroutsame shit happens10:36
magicaltroutI was pondering whether tthe permission error is actually the fact the snap can't see the key?10:37
kjackalwhich snap?10:37
kjackaljuju?10:37
magicaltroutyeah10:38
kjackalI think snaps cannot see hidden directories (starting with .)10:38
magicaltroutjesus h christ10:38
magicaltrouthmm10:39
magicaltroutdoesn't seem to make a difference10:39
magicaltroutcopying the key out10:39
magicaltroutbasically10:40
magicaltroutjuju ssh works for the admin user with the embedded key10:40
magicaltroutbut doesn't work for any add-user10:40
rick_h_magicaltrout: so you added the user and then added their keys?11:52
rick_h_magicaltrout: juju [add,import]-ssh-key11:52
magicaltroutI did rick_h_11:54
magicaltroutyou can replicate this on lxd local11:54
magicaltroutas 1 user bootstrap11:55
magicaltroutthen add another local user as a juju user11:55
magicaltroutand then try and ssh post add-key as that other user11:55
magicaltroutthe admin ssh works11:55
magicaltroutthe other user doesn't11:55
rick_h_magicaltrout: huh...ok. I was just using that the other day. I'll wonder if something broke11:56
magicaltroutof course.... I'll caveat all of this with, it might just be my misunderstanding11:56
magicaltroutrick_h_: for us its been broken for 6 months11:56
magicaltroutwhich is why it might just be me :)11:56
rick_h_magicaltrout: and the user has been granted access to the model with admin access?11:56
magicaltroutyeah11:56
magicaltroutjuju status works11:56
magicaltroutjuju ssh/run doesn't11:56
rick_h_magicaltrout: what's juju show-model have?11:56
magicaltroutbut ssh ubuntu@<machine> for the added user does work11:56
rick_h_magicaltrout: and can you check the .ssh/authorized_keys to see if the key made it?11:57
rick_h_magicaltrout: I guess and is this with import-ssh-key or add-ssh-key? (/me used import recently)11:57
magicaltroutwe know it made it in rick_h_11:57
magicaltroutbecause you can manually ssh using the key you added11:57
rick_h_oh...so the key made it to the machines?11:58
magicaltroutyeah11:58
magicaltrouthttps://pastebin.com/rZEfsuAs11:58
magicaltroutwe have the same issue on 2 laptops rick_h_ one Xenial one Bionic both with the snapped juju installed12:00
rick_h_magicaltrout: k, I'm looking at the email you sent12:00
magicaltroutignore the fact i tried to pump in my public key ;)12:01
rick_h_magicaltrout: is there a proxy or anything involved?12:05
magicaltroutnope12:05
rick_h_magicaltrout: the code from your tracebacks come out into code checking proxy details before any ssh stuff is run12:05
magicaltroutrick_h_: the fact the same thing happens on lxd local suggests proxies are a red herring12:08
rick_h_magicaltrout: yea, but looking at what's giving the permission denied error in that trace is https://github.com/juju/juju/blob/b47854c63ad6345ffd4642e3b22520b43068ef56/cmd/juju/commands/ssh_common.go#L25612:08
rick_h_magicaltrout: k, will have to basically go the bug route and figure out wtf. I'm not sure at this point.12:10
magicaltroutjust checking/recording lxd local again as a reproduction path12:10
rick_h_magicaltrout: check pm12:11
magicaltroutrick_h_: https://asciinema.org/a/JIlCubAF7LBdg6cc86ZcKoBLU12:14
magicaltroutthere's a full LXD example12:14
rick_h_wheeee12:14
* rick_h_ is watching the movie 12:17
rick_h_oooohhhhhh12:17
rick_h_superuser != model level access with all machines12:17
rick_h_that's why even as superuser if you want to see all models you have to use --all12:17
rick_h_try granting admin on the model directly12:18
rick_h_magicaltrout: ^12:18
magicaltroutoooh12:19
magicaltroutthere we go12:19
rick_h_magicaltrout: yea, so superuser is kind of special. Just because you can do anything doesn't mean you want your keys on every machine in the controller and such12:19
rick_h_magicaltrout: just like you don't want to see every model from every user on the controller12:19
rick_h_magicaltrout: even though technically, you can gain access to them all as it's YOUR controller12:19
magicaltroutbrain melting... so superusers, whilst superusers don't actually posess any model level superpowers12:20
rick_h_magicaltrout: it's kind of like sudo, technically you can `sudo dosomething` at any time12:21
rick_h_magicaltrout: but as you run around the system you get access denied until you sudo12:21
magicaltroutthanks rick_h_12:21
rick_h_magicaltrout: it's a bit of a corner world, but kind of done for the controller admin's sanity12:21
rick_h_magicaltrout: ty for the video, that helped12:22
magicaltroutno probs rick_h_ thanks for that, hours of confusion solved in moments ;)12:22
rick_h_magicaltrout: think on it and if you think Juju is doing the wrong thing let's chat.12:25
rick_h_magicaltrout: clearly we can error better at the minimum12:25
rick_h_but I'm nervous about making it "just work" though12:26
magicaltroutno i get the ethos behind it, there could possibly do with some more info in the --help output or something because its not obvious12:27
bobeoo/14:01
bobeohey magicaltrout Im having issues registering to a controller, can you provide me a sample command for registering to one?14:01
bobeoi tried juju register <controllerIP> as well as juju register <controllerName>14:02
rick_h_bobeo: so there's a command that gets created you have to use that has some auth/security bits baked into it14:09
* rick_h_ wonders if we ever got around to that bug that you couldn't re-get the register command14:10
TheAbsentOneis the postgresql charm author in this irc by any chance?14:11
rick_h_TheAbsentOne: sometimes, but he's in APAC region so timezones is a bit off usually14:14
rick_h_TheAbsentOne: best thing is to file bugs or something. What are you looking for?14:14
rick_h_or maybe the mailing list, async coms ftw14:14
TheAbsentOneah I see, you know his name? rick_h_14:14
rick_h_stub: is our pgsql charm hero14:15
TheAbsentOnerick_h_ not really a bug. You see I created a charm that acts as a proxy and all data (connection details for postgres) are shared nicely BUT it is my proxy charm that is the one who requests this database. So in the auth file of postrges he is allowed but my real "consumer" isn't. I'm looking for a way to make sure both my consumer and proxy charm are both inserted in the auth file14:16
rick_h_TheAbsentOne: hmmm, that's sticky. you might have to set config or something and set the config for the extra_hbaconf or whatever that config value is14:17
TheAbsentOnethe pgsql has a set_roles function which I'm looking into now but if that doesn't work out I have a big problem14:17
rick_h_there's a config I know I've used to manually add in entries14:18
bobeorick_h_: can you share this magical command? o.O14:18
TheAbsentOnerick_h_ could you explain it a bit more what you mean? I kinda don't want to edit the pgsql charm or interface14:18
rick_h_TheAbsentOne: https://jujucharms.com/postgresql/#charm-config-extra_pg_conf14:18
rick_h_bobeo: so when you run juju add-user it spits out a register command with some hashed bits around the ip, a security code, etc.14:19
rick_h_bobeo: the command is meant to be sent to the user for them to use to register14:19
TheAbsentOnerick_h_: hey that might work thanks man14:19
bobeorick_h_: how do I generate that? Also, I have a user that exists on the controller already, I need to access the controller itself14:19
rick_h_bobeo: and prevents just general brute force hacking against the register call since you'll never have the right bits14:19
rick_h_bobeo: run juju add-user xxxx14:19
rick_h_bobeo: right, so I tend to create users when I connect from a diff machine to be honest. We've not setup a good way to dump/import the config bits into the .local/share/juju/...14:20
TheAbsentOneargh I don't think it will work rick_h_ it's only for postgres specific options, not for the host authentication14:21
rick_h_TheAbsentOne: right, but this config allows you to add the user/ip address allowed to connect14:22
TheAbsentOnebut there is another one: extra_pg_auth <-- this might work14:22
rick_h_TheAbsentOne: that's the one I linked to14:22
rick_h_oh, sorry14:22
bobeorick_h_:  just to confrim, to grant privs, its juju grant <username> --controller <controllername> super-user correct?14:22
rick_h_TheAbsentOne: actually my bad, that's the one I MEANT to link to14:22
TheAbsentOnerick_h_: my bad in not looking but thanks man ^^14:23
rick_h_bobeo: correct14:23
bobeorick_h_: its not working x...x14:23
rick_h_bobeo: oh sorry, superuser14:23
rick_h_bobeo: one work, no dash14:23
rick_h_one word14:23
bobeorick_h_: ok, so i am losing my mind14:23
bobeorick_h_: its taking so long to load :(14:23
rick_h_bobeo: sorry, I'm jumping around too much.14:24
bobeoprocess? whats te correct term there? does it load, or "process" a command query?14:24
rick_h_bobeo: what's taking long to load? the grant command?14:24
bobeobecause technically you load it ot the controlelr before it runs14:24
bobeoyea the grantocmmand rick_h_14:24
bobeoit keeps hanging14:24
rick_h_bobeo: k, try with --debug to see if you can get a hint on what's hanging14:24
rick_h_bobeo: usually that's pretty darn fast as long as the client can reach the controller IP14:25
bobeorick_h_: its a lan communication14:25
rick_h_bobeo: https://docs.jujucharms.com/2.3/en/tut-users is the short tutorial on the sharing/access stuff if you've not peeked yet14:25
bobeorick_h_: im an idiot. forgive me, for surely my caffiene hasnt made it to work yet14:26
bobeowas trying to send it to a non-existant controller14:26
rick_h_bobeo: ooooohhhh, yea that'll wait a while and make itself a sandwhich waiting for a timeout14:27
TheAbsentOnerick_h_ is it okay to fill an option from within a charm? Is there a best-practice for this, how do you actually do that?14:27
bobeorick_h_: so I have access to the correct controller now, with login privs, but I dont see the models14:27
rick_h_TheAbsentOne: heh not usually because the charms don't normally get to change config on other charms14:27
rick_h_bobeo: right, so because you're superuser you don't see all models by default, only those shared with you directly14:27
rick_h_bobeo: run `juju models --all`14:28
rick_h_bobeo: and if you want to have them be "my models" you can grant yourself admin privs on them14:28
rick_h_bobeo: so that admins don't get a super long list of all models from all users on the controller all the time, and can mange their own list of stuff they work on directly14:28
TheAbsentOnericht thought so x) welp14:28
TheAbsentOneright*14:28
manadartI have borked something. "lxc list" hangs and Juju can not contact my LXD controllers.14:33
rick_h_magicaltrout: try lxc list with -v14:33
rick_h_err manadart ^14:33
* rick_h_ can't type today evidently...when did monday get here14:33
manadartrick_h_: Also hanging with no output...14:34
acengHello anyone know if I used the same physical server to install nova controller and keystone why the keystone configuration get overwritten when the nova relations is complete.  Do I have to install with separate containers to prevent this?14:36
rick_h_break all the things! manadart so searching around finds various bugs/debugging tracks with lxc-monitord and such causing hangs that involve a bunch of killing of things and getting them to restart.14:37
bdxaceng: try keystone in a container, nova-compute on the host to work around this when using a single machine14:37
manadartThere's a beer in my fridge whispering "Leave it till Monday" seductively to me :)14:38
acengokay I guess there is no workaround to preserve the apache and haproxy config when each relation is updating.14:38
rick_h_manadart: lol, sounds like good advice14:38
rick_h_manadart: listen to the beer14:38
manadartrick_h_: You're the boss.14:40
rick_h_manadart: :) have a good weekend14:40
stubTheAbsentOne: You might want to look at the pgbouncer charm, which is not nice, but does something similar. It uses an administrative connection to PostgreSQL so it can create its own users.15:05
stubTheAbsentOne: The trick is that if your proxy uses the db-admin relation to PostgreSQL, it is allowed to connect to any database as any user.15:05
stubTheAbsentOne: Its the only way I can see it working with the current charm; there is no protocol for a client charm to get the PostgreSQL service to do complex config changes like insert entries into pg_hba.conf15:08
stubUsing the extra_ charm config options on the PostgreSQL charm would work, but is a poor UI since the operator has to set the config options to the magic values rather than the client charm handle it automatically.15:10
TheAbsentOnehi stub, thanks for the feedback. Currently I used the db instead of db-admin relation actually, maybe I should try first but I don't think it will make a lot of difference. The point is that my proxy itself doesn't care to connect to postgresql it's the consumer of my proxy that needs to do this.15:15
TheAbsentOneI will take a look at pgbouncer!15:15
TheAbsentOneBut it's idd a problem that I need to perform this manual option config step15:16
stubIf is just a broker rather than an actual proxy, you may be stuck with the PostgreSQL charm config.15:18
stubBecause as you have found, the IP address will be denied access and you can't request it to be opened.15:18
stubThe feature would need to be added.15:19
stub(It used to be possible to lie about your IP address, but the newer networking stuff thankfully removed that hack)15:20
TheAbsentOneyeah it needs the entry in the pg_hba.conf file to connect properly then I can connect with adminer idd stub15:20
TheAbsentOnestub: wouldn't it be a possibility to add an optional parameter to the set_database function that takes a list of hosts that also gain access to the database?15:21
TheAbsentOneI think if that feature got implemented the problem is solved15:21
stubYes, it would be possible to add that feature. A comma separated list of CIDRs. It wouldn't work cross-model, since the client won't know what IP address the server will see.15:23
stubIt needs a good hard think, to be sure it doesn't open any security escalation issues. Gut feeling says it is ok.15:25
TheAbsentOnenot sure if I understand why it wouldn't work cross-model, but if works and is fine within the same model it would already be a huge and interesting thing, give it a thought stub ;)15:25
TheAbsentOneI can only pray hehe, it kinda sucks I have to request it though, it was kinda a requirement for me to use existing charms without modification.15:26
stubIt would probably work out of the box if you connected to pgbouncer instead of postgresql. pgbouncer handles its authorization differently, and doesn't restrict by IP address15:27
stubAnd you want pgbouncer in any non-trivial production deploys anyway15:28
TheAbsentOneallright interesting, I will look into that stub15:29
=== Guest6945 is now known as zeus
=== frankban is now known as frankban|afk
roadmrhey juju friends17:14
roadmrI have a sick juju env, this appears in logs for all services : INFO juju.worker.dependency engine.go:352 "uniter" manifold worker stopped: failed to initialize uniter for "$SOME_UNIT" : cannot create relations: tomb: dying. How can I fix this?17:15
bobeo_o/17:36
bobeo_o/19:11
bobeo_anyone on got any experience with graylog? im digging through it right now and having issues.19:12
kwmonroehey bobeo_, i have experience with the graylog charm. what's going on?21:40

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!