[11:08] <gsimondon> Hi, is there a way to NOT use ncurses based UI during Canonical K8s installation?
[11:15] <magicaltrout> gsimondon: depends where you're trying to install it
[11:15] <magicaltrout> if you're doing a local LXD install you'd have some fun
[11:16] <magicaltrout> if you're doing a cloud based install, you can  juju bootstrap and then just juju deploy canonical-kubernetes
[11:16] <magicaltrout> you don't have to use conjure up
[11:19] <gsimondon> doing bare metal
[11:20] <gsimondon> @magicaltrout all localhost, bare metal, multi node proof of concept installation
[11:20] <gsimondon> @magicaltrout but I don't want any UIs and I want to automate the process... without hacking around
[11:55] <magicaltrout> sorry gsimondon yeah in that case you can do what I do all the time
[11:56] <magicaltrout> and do a juju manual cloud bootstrap
[11:56] <magicaltrout> and then just place workloads on units
[11:56] <magicaltrout> https://jujucharms.com/docs/2.2/clouds-manual
[11:56] <magicaltrout> that one
[11:57] <magicaltrout> then i usually just look at https://api.jujucharms.com/charmstore/v5/canonical-kubernetes/archive/bundle.yaml
[11:57] <magicaltrout> and "juju deploy" the required services and then "juju add-relation" to join them up
[11:57] <magicaltrout> so if you have 5 baremetal servers you could do
[11:57] <magicaltrout> juju deploy easyrsa --to 1
[11:58] <magicaltrout> juju deploy etcd --to 1
[11:58] <magicaltrout> juju deploy kubernetes-master --to 2
[11:58] <magicaltrout> and so on
[11:58] <magicaltrout> then
[11:58] <magicaltrout> juju add-relation kubernetes-master etcd
[11:58] <magicaltrout> etc
[12:12] <rick_h> magicaltrout: gsimondon to be fair, with the 2.3 work you can update the bundle machine numbers and use --use-existing
[12:13] <rick_h> to help automate it into a single bundle deploy now
[12:15] <magicaltrout> ah yeah
[12:15] <magicaltrout> forgot about that
[12:17] <rick_h> life-easier++
[15:30] <gsimondon> rick_h: magicaltrout, thanks a lot.
[16:38] <kwmonroe> magicaltrout: there's also conjure-up in headless mode.. if you specify the spell plus a cloud, c-u will do it's thing with all defaults selected.  it will try to bootstrap, but you can also specify a pre-existing controller... so if i have a bootstrapped controller called aws-e, this will do a headless deploy of k8s-core: conjure-up kubernetes-core aws/us-east-1 aws-e
[16:38] <kwmonroe> i know gsimondon already left, so that's just an fyi for you magicaltrout.  you're welcome.
[16:39] <kwmonroe> aaaand docs, https://docs.ubuntu.com/conjure-up/2.2.0/en/usage#running-in-headless-mode
[16:39] <rick_h> kwmonroe: oooh, more learnings
[16:39] <rick_h> lol
[16:40] <kwmonroe> rick_h: put that in your tips & tricks doc ;)
[16:40] <kwmonroe> jk, i know you don't have one.
[16:40] <rick_h> kwmonroe: :P
[17:03] <kwmonroe> hey cory_fu, what's a good convention for setting a new key with data_changed('foo', bar)?  i'm trying to avoid name collision.  do most people prefix the key with the layer name?  or does data_changed jam some uuid in front of the key already?
[17:08] <kwmonroe> nm cory_fu, i see https://github.com/juju-solutions/charms.reactive/blob/master/charms/reactive/helpers.py#L144.  i will make my own uniqueness.
[17:09] <kwmonroe> though it's nice to know it's somewhat isolated with reactive.data_chagned.
[17:23] <cory_fu> kwmonroe: Yeah, there's not really a way it could reliably do a uuid, so it's up to you, but it does have the built-in prefix and I'd recommend using the layer name as well
[17:24] <kwmonroe> +1 cory_fu
[19:57] <arosales> manual provider question on juju 2.3
[19:57] <arosales> I can `ssh ubuntu@<ip>` from my client machine
[19:58] <arosales> but from that same machine when I try to bootstrap the manual provider I get "Permissioned denied (publickey)"
[19:58] <arosales> any tips in juju 2.3 and manual provider I should be aware of ?
[20:04] <thumper> o/ arosales
[20:04] <thumper> arosales: I only got the last two lines of your post... what is the issue?
[20:11] <arosales> thumper: hello :-)
[20:11] <arosales> thumper: https://paste.ubuntu.com/26531789/
[20:12] <thumper> arosales: um... did you try to bootstrap manual from within the machine itself?
[20:13] <arosales> nope, client is on my laptop where ssh keys are
[20:13] <arosales> bootstrap node is on DigitalOcean
[20:13] <arosales> I also sshed into the target node and imported my ssh keys just to be sure
[20:13] <arosales> confusing part is I can ssh just fine from the same node I am trying to bootstrap from
[20:15] <thumper> arosales: when adding the ipaddress for the cloud
[20:15] <thumper> try ubuntu@...
[20:15] <yosefrow> arosales, maybe its using the keys in ~/.local/share/juju/ssh
[20:18] <arosales> from --debug I saw juju grabbing my key from ~/.ssh/id_rsa.pub
[20:18] <arosales> but perhaps it is not actually using it . .  .
[20:18] <arosales> I can try to put my key into ~/.local/share/juju/ssh
[20:19] <yosefrow> @arosales, I wouldnt suggest overriding the key in ~/.local/share/juju/ssh because juju may have installed it elsewhere. Instead, you can try adding ~/.local/share/juju/ssh public key to the node you are trying to manually bootstrap
[20:19] <yosefrow> in the remote nodes authorized_keys file
[20:20] <arosales> to confirm my understanding though if I can `ssh ubuntu@<ip>` then juju should be able to ssh to the same node, correct?
[20:24] <yosefrow> @arosales, this part isnt clear to me. But as i said if you are trying to troubleshoot. You can *try* to add the pubkey in ~/.local/share/juju/ssh to the authorized_keys file in the remote node. I honestly dont know why you cant ssh there
[20:24] <yosefrow> via juju
[20:27] <magicaltrout> i ended up in a weird loop back ssh thing arosales a few months ago
[20:27] <magicaltrout> where the remote machine and my local machine had the same key
[20:27] <magicaltrout> and it got very confused
[20:28] <arosales> yosefrow: the target node will not have ~/.local/share/juju/ssh initially
[20:29] <arosales> it will just be stock ubuntu with my ssh key on it
[20:29] <arosales> magicaltrout: what did you do to resolve?
[20:30] <thumper> arosales: it is probably trying to use your name for the SSH key source on the target machine
[20:30] <yosefrow> @arosales, i meant to try to put the public key from your juju client machine folder ~/.local/share/juju/ssh into ~/.ssh/authorized_keys on the remote mahcine. but its just a shot in the dark
[20:30] <thumper> you only entered the ip address of the machine (which admittingly is all it asked for)
[20:30] <thumper> but I think you need to say ubuntu@59.65.74.150
[20:31] <thumper> ubuntu@159.65.74.150
[20:31] <arosales> thumper: ahhh, let me try that
[20:34] <yosefrow> @arosales, did you manually edit your target nodes ~/.ssh/authorized_keys?
[20:34] <yosefrow> nevermind i saw earlier that you did
[20:34] <magicaltrout> just juggled my keys around arosales
[20:35] <arosales> thumper: https://paste.ubuntu.com/26531916/
[20:35] <arosales> thumper: looking for ubuntu on my laptop now . .  .
[20:35] <arosales> magicaltrout: gotcha
[20:35] <yosefrow> @arosales, the issue was you were trying to use a hostname instead of ip?
[20:36] <arosales> I was only using IP
[20:36] <yosefrow> aaah
[20:36] <arosales> thumper suggested I append a user name
[20:37] <yosefrow> @thumper, arosales , this line is misleading: Enter the controller's hostname or IP address: ubuntu@159.65.74.150
[20:37] <yosefrow> it should probably say ssh login or user@hostname
[20:37] <yosefrow> seems like a kind of bug to me
[20:38] <arosales> well putting ubuntu@<IP> didn't work for me
[20:38] <arosales> looks for ubuntu on my local machine, which doesn't exist
[20:39] <yosefrow> @arosales, it looks like earlier you just tried ip and it failed, then you tried ubuntu@ip and then it worked. Is this correct?
[20:39] <arosales> negative
[20:40] <arosales> did _not_ work with ubuntu@ip
[20:43] <yosefrow> @arosales, try adding the user ubuntu
[20:43] <yosefrow> to the machine you are trying to add
[20:43] <arosales> its there on the target controller
[20:44] <arosales> I can `ssh ubuntu@<IP>`
[20:44] <yosefrow> from your current user?
[20:44] <yosefrow> the same user you are using to bootstrap juju?
[20:44] <arosales> correct
[20:47] <yosefrow> @arosales, juju bootstrap manual/192.168.1.128 mycloud
[20:47] <yosefrow> https://jujucharms.com/docs/2.3/clouds-manual
[20:47] <yosefrow> so in your case juju bootstrap manual/159.65.74.150 po1
[20:49] <arosales> yosefrow: also tried that, and didn't work --- note you have to drop the last arg. In your command "po1"
[20:49] <yosefrow> why would you drop the last arg ?
[20:49] <yosefrow> wiki says to include it, doesnt it?
[20:50] <arosales> yosefrow: https://paste.ubuntu.com/26531977/
[20:50] <arosales> the docs do, but alas
[20:50] <yosefrow> what version of juju are you using?
[20:50] <arosales> 2.3.1-xenial-amd64
[20:52] <yosefrow> docs for 2.3 say Usage: juju bootstrap [options] [<cloud name>[/region] [<controller name>]]
[20:52] <yosefrow> ,
[20:52] <arosales> looks like a 2.3.2 may be out there. I'll see if that has the same results
[20:52] <yosefrow> @arosales, not $ juju bootstrap 159.65.74.150 manual/159.65.74.150 do
[20:52] <yosefrow> instead do $ juju bootstrap manual/159.65.74.150 do
[20:53] <arosales> yosefrow: did you see my pastbin?
[20:53] <yosefrow> yes you have an extra ip
[20:53] <yosefrow> in there
[20:54] <arosales> ah yes, my syntax error
[20:54] <arosales> same perm error
[20:54] <yosefrow> patebin?
[20:54] <arosales> https://paste.ubuntu.com/26532007/
[21:00] <arosales> same error with 2.3.2 and manually adding ~/.local/share/juju/ssh/juju_id_rsa.pub juju-client-key to the authorized_keys file for both root and ubuntu users on the target bootstrap node
[21:01] <yosefrow> hmmm
[21:01] <yosefrow> i did this before
[21:02] <yosefrow> and it was easy. thats why this is stumping me now
[21:02] <yosefrow> i wish i had documented it
[21:02] <yosefrow> let me examine the error again
[21:02] <arosales> no worries, yosefrow I appreciate the help
[21:03] <arosales> I can add anyone's ssh key to this server if they want to try and bootstrap
[21:03] <arosales> right now its just a test machine
[21:03] <arosales> in theory juju just need ssh key access to the machine, which I thought it would get from ~/.ssh/
[21:04] <arosales> obviously I am missing something
[21:04] <yosefrow> look in ~/.local/share/juju for a file called environments
[21:04] <yosefrow> environments.yaml
[21:05] <arosales> yosefrow: doesn't exist on my machine
[21:11] <yosefrow> @arosales, i see an archaic usage of bootstrap-user probably not used anymore in 2.3
[21:12]  * arosales nods
[21:12] <yosefrow> whats your juju client user?
[21:13] <yosefrow> arosales?
[21:14] <arosales> I believe so, how do I confirm?
[21:15] <yosefrow> `whoami`
[21:15] <arosales> oh, just on the local machine -- yes "arosales"
[21:17] <yosefrow> mv ~/.juju to ~/.juju.bak, remove ubuntu user from the remote machine
[21:17] <yosefrow> lets start over
[21:18] <yosefrow> remove ubuntu user in remote machine if not needed
[21:19] <yosefrow> @arosales,
[21:19] <yosefrow> let me know when you done it
[21:20] <arosales> done, I'll start with a fresh DO instance as well
[21:20] <yosefrow> sorry meant to say mv ~/.local/share/juju
[21:20] <yosefrow> to ~/.local/share/juju.bak
[21:21] <yosefrow> mv ~/.cache/juju to ~/.cache/juju.bak as well
[21:22] <arosales> k, done
[21:24] <yosefrow> create a new manual cloud specify just the ip of the node u want to be controller
[21:26] <arosales> k, done
[21:26] <yosefrow> ssh-copy-id user@host-ip
[21:26] <yosefrow> to add ur key to new bootstrap node
[21:26] <yosefrow> then ssh user@host-ip
[21:28] <yosefrow> @arosales,
[21:28] <yosefrow> which user are you using to login to the remote node?
[21:29] <arosales> root is the only user on the target controller node
[21:29] <yosefrow> you added the key for root user?
[21:30] <arosales> no I just added my ssh key when making the instance
[21:30] <arosales> that got added to the authorized_keys on the root account on the target machine
[21:30] <arosales> so I can ssh root@<ip>
[21:31] <yosefrow> i have an idea
[21:31] <kwmonroe> ;
[21:33] <yosefrow> @arosales,from root@remote-host$ adduser ubuntu; adduser ubuntu sudo; passwd ubuntu (set some passwd),. Then from juju client create a new cloud, add a new cloud, this time use ubuntu@159.65.74.150 instead of just the ip, ok then finally try to bootstrap to the second cloud
[21:34] <yosefrow> when it bootstraps it will login, then ask for a passwd
[21:34] <yosefrow> enter the passwd you created in the previous step
[21:34] <yosefrow> ill brb to see how it goes
[21:34] <arosales> I did that exact thing with https://paste.ubuntu.com/26531789/
[21:35] <arosales> I then tried ubuntu@<ip> and that also failed per thumpers suggestion
[21:36] <arosales> ubuntu disables login via username/password by default. It seems juju ends up looking for a Ubuntu user locally to sudo to and fails
[21:47] <yosefrow> @arosales, show me the output from trying to login to the second cloud i said you should make, where the host is not just the ip but ubuntu@ip
[21:48] <yosefrow> Enter the controller's hostname or IP address: ubuntu@159.65.74.150
[21:48] <yosefrow> instead of Enter the controller's hostname or IP address: 159.65.74.150
[21:50] <yosefrow> @arosales, last time your output was https://paste.ubuntu.com/26531916/. this time you should get the same output, except now, you should enter the password for ubuntu that you created in the previous step
[21:50] <yosefrow> it appears that juju is logging in successfully via ssh and then attempting to run a command on the remote host with sudo
[21:51] <arosales> yosefrow: https://paste.ubuntu.com/26531916/
[21:51] <yosefrow> great
[21:51] <yosefrow> enter the password you created
[21:51] <yosefrow> on the remote host
[21:52] <kwmonroe> arosales: when you're done messing around, use root@<do-ip> as your cloud.  https://paste.ubuntu.com/26532245/
[21:54] <arosales> thanks kwmonroe that worked
[21:54] <arosales> yosefrow: complained about Ubuntu not being in sudo file, but I think I could have added it and it may have worked
[21:55] <arosales> root worked with out having to mess with Ubuntu
[21:55] <kwmonroe> i'm sure adding the ubuntu user first and doing key mgmnt will probably get ya there, but since DO gives you root and jams your key in there, i say use it!  with metldown and spectre, logging in as root across the internet is the least of your worries ;)  now where's my $5 arosales?
[21:56] <arosales> kwmonroe: put it on my tab
[21:56] <kwmonroe> ha!
[21:56] <yosefrow> @arosales, ubuntu complained because you probably skipped `adduser ubuntu command"
[21:57] <yosefrow> @kwmonroe, has a good workaround, but personally i prefer not directly ssh as root
[21:57] <kwmonroe> yeah yosefrow, i didn't mean to interrupt there, but you guys were messing up my backscroll.
[21:57] <magicaltrout> i'll mess up your backscroll
[21:58] <yosefrow> lol
[21:58] <kwmonroe> gah!
[21:58] <yosefrow> kwmonroe, i thought the point of this chat was to demonstrate how to solve juju related issues and other juju related chat
[21:58] <arosales> yosefrow: thanks or sticking with it for me
[21:58] <yosefrow> should i have answered him in private messages?
[21:58] <arosales> yosefrow: kwmonroe  can tell you I am a tough customer
[21:58] <arosales> next charmer summit I'll buy beers
[21:58] <yosefrow> @arosales, I still think you should do it the way I suggested if you want to follow best practices
[21:58] <arosales> were you guys at config mgmt camp?
[21:59] <yosefrow> but if you want a quick solution then kwmonroe is 100% correct
[21:59] <yosefrow> @arosales, wasnt there
[21:59] <kwmonroe> yosefrow: i was totally messing around -- and no, you don't want to get arosales in a PM.  what you guys were walking through was all good.
[21:59] <yosefrow> kwmonroe, ah ok xD
[22:00] <yosefrow> @kwmonroe, i might write a blog post about this chat soon
[22:00] <yosefrow> we'll see
[22:00] <arosales> yosefrow: I can see the light at that tunnel, the odd thing was that I had created the ubuntu user and added "ubuntu" to the sudoers via `usermod -aG sudo ubuntu`
[22:00] <arosales> but not dice
[22:00] <kwmonroe> yosefrow: +1, and if you do, arosales will send you $5
[22:00] <arosales> but most likely fat fingered it.
[22:00] <arosales> kwmonroe: lol
[22:01] <arosales> in IOUB
[22:01] <yosefrow> @arosales, thats quite strange
[22:02] <yosefrow> @arosales, ok well in that case nevermind. the cake goes to @kwmonroe
[22:02] <yosefrow> i was certain that should work
[22:02] <yosefrow> @kwmonroe, save me a slice of cake
[22:04] <kwmonroe> so it looks like the manual provisioner really wants to add the ubuntu user:  https://github.com/juju/juju/blob/develop/provider/manual/provider.go#L30
[22:04] <kwmonroe> that calls out to https://github.com/juju/juju/blob/develop/environs/manual/sshprovisioner/sshprovisioner.go#L31, which lists the preconditions which will skip that init
[22:05] <kwmonroe> yosefrow: arosales, so i think the key takeaway might be that arosales' ubuntu user needed to have passwordless sudo access.
[22:05] <kwmonroe> anyway, root4life
[22:05] <yosefrow> @kwmonroe, rootkits4life :P
[22:05] <kwmonroe> lol
[22:05] <arosales> man kwmonroe reading Go code now ;-)
[22:06] <kwmonroe> nah, i just have thumper in a side channel telling me what to type
[22:06] <arosales> you must have stopped drinking early PM brews
[22:06] <kwmonroe> them's fightin words
[22:06] <arosales> rofl
[22:06] <yosefrow> @arosales, if you want to do with ubuntu user take kwmonroe suggestion and `visudo`. then https://serverfault.com/questions/160581/how-to-setup-passwordless-sudo-on-linux
[22:06] <yosefrow> i suspected as much as well
[22:07] <yosefrow> i honestly forgot how i got it to work last time, but probably like that
[22:07] <yosefrow> @kwmonroe, did thumper get pissed off with me banging my head against the wall trying to help @arosales ?
[22:08] <arosales> that makes sense since I could sudo apt get with the ubuntu user, but failed when Juju tried
[22:09] <yosefrow> @kwmonroe, arosales anyway that was interesting. Glad we found a solution. laters
[22:11] <kwmonroe> yosefrow: no, thumper doesn't get pissed.  he writes go.  good time today, have a good one!
[22:13] <arosales> thanks yosefrow, kwmonroe, magicaltrout, and thumper for the help
[22:17]  * thumper was on calls, soryr
[22:18] <yosefrow> wb thumper
[22:19] <cory_fu> stub: Hey, could you update juju-wait on pypi to match the snap, please, when you get a chance?