[01:29] <fallenour> what api runs on port 17070?
=== frankban|afk is now known as frankban
[14:12] <sri_> I am exploring the possibility of setting k8 cluster using juju/conjure-up on aws. could someone help me find a working solution to set kubernetes cloud-provider as aws?
[14:13] <sri_> I tried below commands and reboot the containers but containers does not come online anymore
[14:13] <sri_> juju ssh MACHINE-NUMBER "sudo snap set kube-apiserver cloud-provider=aws"; juju ssh MACHINE-NUMBER "sudo snap set kube-controller-manager cloud-provider=aws"; juju ssh MACHINE-NUMBER "sudo snap set kubelet cloud-provider=aws"
[14:18] <tvansteenburgh> sri_: if you deploy to aws with conjure-up, the cloud provider is set for you
[14:19] <tvansteenburgh> sri_: in other words, you don't have to set it manually
[14:43] <sri_> tvansteenburgh, i tried juju ssh MACHINE-NUMBER "sudo snap get kube-apiserver"; which returns nothing
[14:55] <tvansteenburgh> sri_: cat /var/snap/kube-apiserver/current/args
[15:12] <sri_> tvansteenburgh: https://ghostbin.com/paste/xq63r is the output for juju status
[15:15] <sri_> tvansteenburgh: added the output of juju ssh 1 'cat /var/snap/kube-apiserver/current/args' also in the same link
[15:16] <sri_> tvansteenburgh: sorry, pastebin is down hence using ghostbin, not sure if it is the best alternate :-)
[15:25] <ryebot> sri_: Can you share the output of `kubectl get po --all-namespaces` ?
[15:27] <sri_> ryebot: Actually i just pasted very detailed log of yaml files, pod description etc in the same link
[15:28] <ryebot> sri_: excellent, taking a look
[15:29] <sri_> ryebot: just added the output of kubectl get po --all-namespaces at the end of the doc...please help me here
[15:29] <ryebot> sri_: any idea what's going on with "postgres_pv"?
[15:30] <sri_> ryebot: i created the volume vol-0445bc1b6acae0622 in aws and referred to it here
[15:31] <sri_> ryebot: Is it not correct?
[15:31] <ryebot> sri_: I'm not sure. It looks like it's failing to mount, doesn't it?
[15:32] <ryebot> namely, "...vol-0445bc1b6acae0622 does not exist"
[15:33] <sri_> ryebot: thats exactly where i could not trace down to root cause
[15:34] <sri_> ryebot: when i deploy postgres using ebs via juju, it works fine...it does not work only through kubernetes
[15:34] <ryebot> sri_: alright, let me look at this a moment
[15:36] <sri_> ryebot: Thanks a lot...A quick google search results points out that juju is not setting cloud-provider as aws w.r.t https://github.com/juju-solutions/bundle-canonical-kubernetes/issues/346
[15:36] <ryebot> sri_: I see, that's why you're trying to add the cloud-provider flag
[15:36] <sri_> ryebot: but that did not help me :-(
[15:37] <sri_> ryebot: After i set this flag manually, reboot of containers fails and they never successfully start
[15:38] <sri_> ryebot: Not sure if there is a good way to reboot the cluster using juju
[15:38] <ryebot> hmm
[15:38] <ryebot> were you actually able to set the flag?
[15:39] <sri_> yes
[15:39] <ryebot> sri_: where do you see that it has been set?
[15:40] <sri_> ryebot: let me paste those results
[15:40] <ryebot> sri_: thanks
[15:44] <sri_> ryebot: pasted those commands and output at the end of the doc
[15:44] <ryebot> sri_: thanks, looking
[15:46] <ryebot> sri_: How about kubelet?
[15:46] <sri_> as per https://github.com/juju-solutions/bundle-canonical-kubernetes/issues/346, i need to reboot those machines after setting these flags but if i do, i could not start anymore
[15:46] <sri_> I did that for kubelet also in machine 0
[15:46] <ryebot> alright, hmm
[15:47] <sri_> ryebot: is there a good way to restart juju machines?
[15:47] <ryebot> sri_: This is a bit confusing, because if you deployed with the latest conjure-up (e.g., http://blog.astokes.org/conjure-up-dev-summary-aws-cloud-native-integration-and-vsphere-3/), this should all just work for you.
[15:48] <ryebot> sri_: you should be able to juju ssh in and `sudo reboot`, at the very least
[15:49] <sri_> ryebot: Let me do that but i am sure it fails to start...but at-least i can show you the error it gives
[15:50] <ryebot> sri_: that'd be good to see, thanks.
[15:55] <sri_> now, it stuck at `kubernetes-master/0*  waiting   executing  1        13.58.179.90    6443/tcp        (update-status) Waiting to retry addon deployment`
[15:55] <sri_> ryebot: now, it stuck at `kubernetes-master/0*  waiting   executing  1        13.58.179.90    6443/tcp        (update-status) Waiting to retry addon deployment`
[15:55] <sri_> ryebot: This happens only after i add cloud-provider=aws and reboot the machine
[15:56] <ryebot> okay
[15:56] <ryebot> one moment
[15:56] <sri_> ryebot: I am not sure which addon it is trying to deploy...shall i add new juju status, juju debug-log to the pastebin?
[15:57] <ryebot> sri_: juju debug-log would be great
[15:57] <ryebot> thank you
[15:57] <sri_> ryebot: juju debug-log only shows that it cannot connect to kubernetes-master
[15:58] <sri_> ryebot: could be because it is not ready
[15:58] <sri_> ryebot: any other juju commands to get detailed logs per machine and per app?
[15:59] <ryebot> sri_: you can try ssh'ing into the relevant machine and looking in /var/log/juju
[15:59] <ryebot> that should bypass any erroring agents
[16:01] <sri_> ryebot: kindly check https://ghostbin.com/paste/xq63r now
[16:02]  * ryebot clicks
[16:04] <ryebot> sri_: one moment
[16:05] <ryebot> sri_: Can you check the logs for apiserver?
[16:05] <sri_> ryebot: Sure take your time... I see failure logs at `  File "/snap/cdk-addons/169/apply", line 93, in <module>`
[16:05] <sri_> sure
[16:07] <ryebot> sri_: on the maste node, `journalctl -u snap.kube-apiserver.daemon`
[16:07] <ryebot> master*
[16:09] <sri_> ryebot: Pasted new log at the bottom...i think we have the root cause there...
[16:09] <sri_> Oct 19 15:51:27 ip-172-31-25-174 kube-apiserver.daemon[4246]: error setting the external host value: "aws" cloud provider could not be initialized: could not init cloud provider "aws": error fi
[16:10] <ryebot> sri_: can you do that with `journalctl -xn --no-pager`
[16:11] <skay> I'm not seeing a flag I expect from a charm that provides the rabbitmq interface. from reading the src, anything that provides rabbitmq should be setting myname.connected, but I'm not seeing it
[16:12] <skay> https://github.com/openstack/charm-interface-rabbitmq/blob/master/requires.py#L38
[16:12] <sri_> ryebot: output added at the bottom
[16:13] <ryebot> sri_: thanks
[16:14] <ryebot> dangit, journalctl, I just want untruncated output.
[16:15] <ryebot> okay, let's try journalctl -o cat -xn -u snap.kube-apiserver.daemon
[16:15] <ryebot> sri_: or if you know a better way, by all means
[16:16] <sri_> okay, let me add it
[16:17] <sri_> its actually truncating and giving only last few lines :-(
[16:19] <sri_> I redirected output to a tmpfile but its huge...
[16:20] <sri_> ryebot: untracked error message seems to be: `Oct 19 16:00:38 ip-172-31-25-174 kube-apiserver.daemon[12360]: error setting the external host value: "aws" cloud provider could not be initialized: could not init cloud provider "aws": error finding instance i-08e7af80c7f9dc876: "error listing AWS instances: \"NoCredentialProviders: no valid providers in chain. Deprecated. \\n\\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors
[16:20] <ryebot> sri_: okay, that's gross
[16:20]  * ryebot thinks
[16:22] <ryebot> sri_: Did you deploy with conjure-up, as is done here? http://blog.astokes.org/conjure-up-dev-summary-aws-cloud-native-integration-and-vsphere-3/
[16:22] <sri_> ryebot: i-08e7af80c7f9dc876 is actually on us-east-2b and logs shows as us-east-2
[16:22] <sri_> is that an issue?
[16:23] <ryebot> sri_: yeah that's weird
[16:23]  * ryebot thinks
[16:23] <sri_> I am on macOS and i got conjure-up via brew
[16:23] <ryebot> ahhh okay one sec
[16:24] <sri_> ryebot: I deployed using just "conjure-up" command but the UI does not look same as the one shown in the link
[16:24] <sri_> ryebot: I dont think i see addons support
[16:24] <stokachu> sri_: https://github.com/conjure-up/conjure-up/issues/1195 is that you?
[16:26] <sri_> stokachu: yes
[16:26] <stokachu> sri_: ok let me look at your files you attached
[16:27] <sri_> ryebot: I am on mac and my version is: conjure-up 2.3.1
[16:27] <ryebot> sri_: ack, thanks
[16:27] <sri_> ryebot: not sure it has addons and aws native integration in it or not
[16:29] <ryebot> Looks like it errored out looking for juju wait?
[16:31] <sri_> ryebot: sorry, i didn't get you
[16:32] <ryebot> sri_: nvm, just thinking out loud
[16:32] <stokachu> sri_: give me a few minutes im working on something then ill look at your issue
=== frankban is now known as frankban|afk
[16:33] <sri_> Sure thanks alot for your time ryebot, stokachu
[16:50] <sri_> ryebot: You want me to send full journal file in some way?
[17:08] <ryebot> sri_ I think that's enough for now
[17:09] <sri_> ryebot: Okay, do you know any workaround for me to use PVs?
[17:17] <ryebot> sri_: Well, it looks to me like you're on the right track. The problem appears to be the aws credentials got mixed up, and also that you're somehow not getting the native cloud integrations. I suspect stokachu will be able to help with that when he takes a look at your issue.
[17:21] <sri_> ryebot: Thanks. BTW, what do you mean by `aws credentials got mixed up`? anyway I can dig further?
[18:19] <ryebot> sri_: Yeah, it seems strange to me that your instances are coming up in different regions. It makes me think there's a couple sets of credentials pointing to different regions.
[18:22] <sri_> ryebot: But when i do conjure-up, I only give one set of credentials
[18:22] <sri_> BTW, conjure-up only let us choose us-east-2 or us-east-1
[18:23] <ryebot> sri_: Ah dang, I'm sorry, I misread that - -2b and -2 are different AZs, not two regions. I think that should be fine, actually.
[18:23] <sri_> but aws, create instances with an alphabet at the end right
[18:24] <sri_> all my machines are in us-east-2 but with us-east-2a as one us-east-2b as another
[18:25] <ryebot> sri_: Yeah, I think that should be okay.
[18:25] <ryebot> Still, the error breaking us looks like some credentials got mucked up somewhere.
[18:26] <sri_> is there a way i can cleanup all the conjure-up history?
[18:26] <sri_> ryebot: is there a way i can cleanup all the conjure-up history?
[18:27] <sri_> Even though i delete .cache folder, it still remember somehow
[18:27]  * ryebot thinks
[18:28] <ryebot> sri_: I think you want to `snap remove conjure-up`, `rm -rf ~/snap/conjure-up`, and `rm ~/.local/share/juju` if it's there, at the least
[18:28] <ryebot> ah sheesh, you're not using snaps, sorry
[18:28] <sri_> ryebot: i am on mac so snap wont work right
[18:29] <ryebot> I'm not sure what the brew cleanup looks like, tbh. stokachu might know. I'd also look for ~/.local/share/juju and wipe it (if you're not otherwise using it)
[18:29] <sri_> okay
[18:30] <sri_> ryebot: if i use conjure-up edge, cloud-provider config should be aws right
[18:31] <ryebot> I would expect edge to have it, though I don't know why stable wouldn't.
[18:32] <sri_> ryebot: I mean, If i follow the version from http://blog.astokes.org/conjure-up-dev-summary-aws-cloud-native-integration-and-vsphere-3/ and setup cluster and run `juju ssh 1 "sudo snap get kube-apiserver cloud-provider"` it should return aws?
[18:33] <ryebot> sri_: I think that's correct, though I'd replace `1` with `kubernetes-master/0`
[18:34] <sri_> Yes got you...thanks
[20:10] <ryebot> Does juju create new subnets when it's deployed to a vpc?
[21:05] <wpk> No, it uses the ones that are in the VPC
=== thumper is now known as thumper-afk