=== frankban|afk is now known as frankban
[08:40] <NotDavid> Hi. I'm having an issue bootstrapping juju on a private openstack environment. It seems to be unable to initiate the mongodb replicaset. Anyone had this problem before?
[08:43] <cnf> morning
[08:58] <NotDavid> Anyone know where I can find juju's bootstrap logs? Bootstrapping keeps failing.
[09:06] <admcleod> NotDavid: hrm, well juju debug-log may show something, but otherwise you can --debug and --verbose the bootstrap?
[09:06] <NotDavid> I've tried --verbose, trying --debug now
[09:14] <NotDavid> Thanks!
[09:14] <NotDavid> I've found the command that gives me an error
[09:14] <NotDavid> It seems to fail on this action: juju.worker.peergrouper initiate.go:42 Initiating mongo replicaset; dialInfo &mgo.DialInfo{Addrs:[]string{"localhost:37017"}, Direct:false, Timeout:300000000000, FailFast:false, Database:"", ReplicaSetName:"", Source:"", Service:"", ServiceHost:"", Mechanism:"", Username:"", Password:"", PoolLimit:0, DialServer:(func(*mgo.ServerAddr) (net.Conn, error))(0xae73a0), Dial:(func(net.Addr) (net.Conn, error))(nil)}; memberHostport
[09:14] <NotDavid> "10.0.0.12:37017"; user ""; password ""}
[09:15] <NotDavid> Resulting in this: 2017-07-12 09:13:05 INFO juju.worker.peergrouper initiate.go:64 finished InitiateMongoServer
[09:15] <NotDavid> ERROR failed to start mongo: cannot initiate replica set: cannot dial mongo to initiate replicaset: no reachable servers
[09:19] <admcleod> NotDavid: hmm. well, im going to assume its a networking related thing
[09:20] <admcleod> NotDavid: do the instances automatically get external ips? and, can they automatically connect to the internet? (maybe to download a package or something)
[09:22] <NotDavid> My instances can communicate with the internet. Openstack NAT's them. They're not accessible from outside the cloud, but my bootstrapping machine is in the same openstack project, so it's able to reach all the instances it creates.
[09:23] <NotDavid> If they do need to be externally accessible they'd need to have floating ip's assigned to them and juju would then have to know about them. I'm not sure how I would do that
[09:24] <admcleod> NotDavid: shouldnt need that no..
[09:24] <NotDavid> Right, especially since juju's already ssh'ed into the new controller instance.
[09:24] <admcleod> NotDavid: and does 10.0.0.12 check out?
[09:25] <NotDavid> Yeah, it creates itself an instance. That instance is called 10.0.0.12
[09:25] <admcleod> NotDavid: are you able to ssh into it? maybe see if mongo is running and bound to that ip?
[09:25] <NotDavid> It is fully able to update and install packages
[09:26] <NotDavid> I would log into it, but the instance gets destroyed when the bootstrap fails
[09:27] <NotDavid> Could it be a timeout issue. Mongo can take a while to initialize.
[09:27] <admcleod> hrrrrm.....it could be, but it shouldnt be or it would be more frequently reported. what version of juju?
[09:28] <NotDavid> 2.2.1-xenial-amd64
[09:33] <NotDavid> I'm going to re-check my secgroups, might be some blockages there. Shouldn't be though
[09:34] <NotDavid> The correct ports should be open. But i've added ingress rules for all TCP and UDP traffic.
[09:35] <NotDavid> I'm gonna rerun the bootstrap now
[09:35] <admcleod> nod
[09:44] <NotDavid> Nope ... same error
[09:44] <NotDavid> So it's definitely not a network or firewalling issue
[09:46] <NotDavid> well ... I'm stumped. Can't see why it shouldn't work
[09:47] <admcleod> jamespage: tinwood any ideas ^^?
[09:52] <admcleod> NotDavid: well. i guess you could try bootstrap-timeout
[09:53] <admcleod> NotDavid: https://jujucharms.com/docs/2.0/controllers-creating
[09:53] <ak_dev> Hello all, I am developing a charm for kubernetes with OVN using this documentation : https://github.com/openvswitch/ovn-kubernetes
[09:53] <ak_dev> The charm I have developed is : cs:~aakashkt/ovn-2
[09:53] <ak_dev> This acts as a subordinate charm to the already existing kubernetes-master and kubernetes-worker, much like flannel
[09:53] <ak_dev> The problem is everything installs correctly, but in the end, the master stalls at "waiting for kube-system pods to start"
[09:53] <ak_dev> What I could figure out is that the worker is not registering with the master for some reason
[09:53] <admcleod> NotDavid: bottom of the page
[09:53] <NotDavid> found it
[09:53] <NotDavid> trying it now
[09:53] <ak_dev> Here is the bundle file for the deployment https://usercontent.irccloud-cdn.com/file/U1pzdevl/bundle.yaml
[09:53] <admcleod> kjackal___: you around?
[09:54] <NotDavid> I've set the timeout to ten minutes and rerunning it
[09:55] <admcleod> ak_dev: i think kjackal___ may be able to help or point you in the right direction but not sure if he's around
[09:57] <ak_dev> admcleod: Hey, thanks for the reply
[09:57] <ak_dev> will wait for him to come around :-)
[09:57] <admcleod> ak_dev: there are other kubes guys here but they're US timezone
[10:00] <ak_dev> admcleod: oh, will have to be around at night for them then, thanks, need to fix this soon
[10:01] <admcleod> SaMnCo: around?
[10:01] <NotDavid> admcledo: Getting the same error after the same time duration with a longer timeout. So it's not a timeout issue
[10:02] <NotDavid> adcleod*
[10:02] <admcleod> NotDavid: can you pastebin the full bootstrap output?
[10:02] <NotDavid> admleod*
[10:02] <admcleod> haha
[10:02] <NotDavid> Tiny keyboard, giant hands, major issues
[10:03] <admcleod> lol
[10:04] <kjackal___> admcleod: hi, give me 2 mins
[10:06]  * tinwood reading scrollback
[10:07] <kjackal___> hi admcleod, ak_dev let me see
[10:09] <ak_dev> kjackal___:  Hey, thanks, one more thing, I talked to Guru from ovn-kubernetes, he told me that I need to pass the correct options to kubelet, kube-apiserver, kube-scheduler and kube-controller manager
[10:09] <tinwood> admcleod, NotDavid it looks like you're looking at everything I would have checked.  Anything else in the logs?
[10:10] <admcleod> NotDavid: oh yeah, pastebin?
[10:11] <NotDavid> Yeah, rerunning it with a longer scrollback. Give me a sec
[10:11] <NotDavid> The problems of not working on your normal workstation
[10:13] <kjackal___> ak_dev: this looks great! Deploying your bundle on aws now
[10:14] <kjackal___> ak_dev: a couple of questions why would you deploy 1.5 kubernetes? We just released 1.7
[10:15] <ak_dev> yes, that was because the recommended version to use with this was 1.5, told to me by the openvswitch guy
[10:15] <ak_dev> glad you liked the charm :-)
[10:18] <ak_dev> oh, you will have to give access to all ports in aws firewall rules, i had to do that in GCE
[10:20] <kjackal___> ak_dev: waitup... kubernete-master and worker are patched locally by you so I cannoot deploy (/home/aakash/Desktop/k8s-official-charms/xenial/kubernetes-master)
[10:20] <NotDavid> admcleod: Here's the pastebin log: https://pastebin.com/YFFT0Aub
[10:20] <kjackal___> So... ak_dev we will have to do this remote ly
[10:21] <ak_dev> oh, sorry for that, forgot to change them
[10:21] <ak_dev> cs:~containers/kubernetes-master-35
[10:21] <ak_dev> cs:~containers/kubernetes-worker-40
[10:21] <ak_dev> sorry for the mistake
[10:21] <kjackal___> ak_dev: you can juju ssh on the master and kubectl get no to see why/if the nodes are up
[10:22] <kjackal___> updating he bundle now
[10:23] <admcleod> NotDavid: can you try (long shot) --config use-default-secgroup=true
[10:23] <admcleod> NotDavid: and perhaps use-floating-ip=true (after)
[10:24] <NotDavid> now that i'm looking at it ... I notice it's trying to connect to mongo on port 37017. Isn't 27017 the default monogdb port
[10:25] <kjackal___> ak_dev: I need to give access to which ports on which nodes?
[10:25] <stub> tinwood: Do you know the status of moving charm-helpers to github?
[10:26] <ak_dev> kjackal___: 6641, 6642 on master, 8080 on both master and worker
[10:26] <stub> or anyone really (I think it was alex?)
[10:26] <tinwood> stub, I don't sorry.  I know the work has started (jamespage) but it got hung up on transferring the history and email addresses.  I don't know the status though.
[10:26] <ak_dev> i think that should be enough, I actually allowed all ports so can't say exactly which
[10:27] <kjackal___> admcleod:
[10:27] <stub> tinwood: I'm not sure why historic data needs to match to current email addresses, but I'll land this on LP then
[10:28] <tinwood> stub, jamespage will know more.  what's 'this'?  the nrpe change?
[10:28] <admcleod> NotDavid: pretty sre 37017 is ok
[10:29] <stub> https://code.launchpad.net/~peter-sabaini/charm-helpers/bcache-helpers/+merge/323616 , which just became ready
[10:29] <jamespage> tinwood, stub: just target the bzr branch on LP for now; the export/import takes seconds to complete so we'll do that right before switchover
[10:29] <NotDavid> admcleod: Trying your suggestion now
[10:30] <kjackal___> ak_dev: there is the juju run --application kubernetes-master open-port 6641/tcp that we might be able to use
[10:30] <tinwood> jamespage, kk.
[10:31] <ak_dev> kjackal___:  oh, I was not aware of that
[10:33] <kjackal___> ak_dev: it _might_ do what we want, not sure for now. Still deploying
[10:34] <NotDavid> admcleod: No luck, i'm afraid.
[10:35] <NotDavid> amdcleod: Anyway, i've got a meeting now. I'll pick this back up tomorrow
[10:36] <NotDavid> amdcleod: If you have any sudden realisations please let me know
[10:36] <admcleod> NotDavid: hmm ok sure, will do
[10:43] <kjackal___> ak_dev: if you look at the /var/log/syslog on workers, do you see any thing suspicius?
[10:44] <ak_dev> kjackal___: hm, I don't know what to look for to find anything suspicious though
[10:45] <ak_dev> i am going to redeploy in some time, made some changes
[10:45] <kjackal___> ak_dev: I have an error on the juju unit of ovn
[10:46] <kjackal___> http://pastebin.ubuntu.com/25074590/
[10:46] <kjackal___> ak_dev: ^
[10:47] <ak_dev> kjackal___: oh sorry i should have mentioned before, there is an config option in the OVN unit
[10:47] <ak_dev> gateway-physical-interface
[10:47] <ak_dev> you will have to set it to the default interface the machine uses
[10:47] <ak_dev> in GCE, it is ens4
[10:48] <ak_dev> ip route | grep default shoud give u the interface name I suppose
[10:49] <ak_dev> kjackal___: my sincere apologies, I know its a pain to redeploy everything again and again
[10:53] <magicaltrout>   /join #maas
[10:53] <magicaltrout> meh
[11:01] <admcleod> hugs
[11:03] <kjackal___> ak_dev: redeployed only ovn and got http://pastebin.ubuntu.com/25074659/
[11:04] <kjackal___> should be an easy fix
[11:04] <ak_dev> that mkdir error is okay
[11:04] <ak_dev> no problem with that
[11:07] <kjackal___> ak_dev: the node did not register
[11:07] <ak_dev> yes, that was the problem I was facing, don't know why it isn't registering
[11:09] <ak_dev> is the master stuck on "waiting for kube-system pods to start" ?
[11:12] <ak_dev> the network OVN creates is fine, the guy from openvswitch tested it
[11:13] <ak_dev> I have no idea about why the node isn't registering
[11:14] <ak_dev> kjackal___:  ^
[11:14] <kjackal___> ak_dev: do we need to start the node services using extra params?
[11:15] <kjackal___> Cynerva: and ryebot may be interested in this ovn work
[11:15] <ak_dev> yes, there are some he has mentioned in the readme, which I couldn't see passed in the charm code
[11:15] <kjackal___> ak_dev: ah, the README! Let me read!
[11:15] <ak_dev> I am trying that out now
[11:16] <ak_dev> yes just a sec
[11:16] <ak_dev> https://github.com/openvswitch/ovn-kubernetes
[11:16] <ak_dev> this is the overall readme
[11:16] <ak_dev> https://github.com/openvswitch/ovn-kubernetes/blob/master/docs/INSTALL.K8S.md
[11:16] <ak_dev> this is for setting up k8s
[11:17] <ak_dev> kjackal___:  ^ (sorry keep forgetting to tag you)
[11:28] <kjackal___> ak_dev: on the master if you go to /var/snap/kube-apiserver/current/args you can add the params manually and restart the respective service like  systemctl restart snap.kube-apiserver.daemon
[11:29] <kjackal___> the same goes for kube-controller-manager and kube-scheduler on the master
[11:30] <ak_dev> kjackal___: oh thanks, that will help a lot! Will try and set the proper args and see if it works
[11:30] <ak_dev> so in args, how do i specify the arguments?
[11:31] <ak_dev> like how we normally do in command line or is there any other format?
[11:32] <kjackal___> ak_dev: for example --master "http://127.0.0.1:8080"
[11:32] <ak_dev> kjackal___: ah okay, that way then
[11:33] <kjackal___> ak_dev: as soon as you have the right params in place please go ahead and submit a PR so we add them on the charms. There is also a PR to add a config option for custom params...
[11:34] <ak_dev> kjackal___: oh, that config option thing will be great
[11:34] <ak_dev> i will once I am able to get this up and running
[11:35] <kjackal___> ak_dev: What you are doing is really interesting. If you get into trouble and you do not find anyone in this room open an issue here: https://github.com/juju-solutions/bundle-canonical-kubernetes so we follow up
[11:36] <ak_dev> kjackal___: sure will do, thanks for helping out :-)
[11:37] <kjackal___> Usualy eu timezone I am around and later us hours you can ping Cynerva or ryebot
[11:40] <SaMnCo> admcleod: yes back
[11:40] <SaMnCo> what's up?
[11:43] <ak_dev> kjackal___: will do
[11:55] <iguez> Hi all. Juju has the feature of setting an APT mirror URL in the model config, which affects sources.list in LXC containers (juju model-config apt-mirror="URL"). Does anyone here know how to set a custom apt gpg key along with this mirror URL?
[12:06] <ak_dev> kjackal___: i put in my args for kube-apiserver, and restarted the daemon, but it quits just after restart
[12:07] <ak_dev> any reason why this is happening?
[12:07] <ak_dev> everything else restarted correctly
[12:07] <kjackal___> there is nothing in the service logs?
[12:08] <kjackal___> ak_dev: /var/log/syslog ? anything interesting there?
[12:09] <ak_dev> kjackal___: looks like apparmor denied starting service
[12:10] <ak_dev> https://www.irccloud.com/pastebin/blb6zoKf/
[12:11] <kjackal___> ak_dev: can you try restarting again?
[12:13] <ak_dev> https://www.irccloud.com/pastebin/SNV9V6MA/
[12:13] <ak_dev> kjackal___: ^
[12:18] <kjackal_> ak_dev: we need to look into this in more detail. Can you start kubeapi-server from the commandline: like /snap/kube-apiserver/77/kube-apiserver with --v 9
[12:18] <kjackal_> ak_dev: and all your args
[12:20] <ak_dev> kjackal_: oh okay, will do that now
[12:22] <ak_dev> kjackal_: okay i think it started
[12:22] <ak_dev> will do the same for worker and see if it connectws
[12:22] <kjackal> ak_dev: nice
[12:23] <ak_dev> kjackal: curious, what is the --v 9 for?
[12:23] <ak_dev> in my args, he has specified --v 2
[12:28] <ak_dev> kjackal: okay, so i restarted with my args
[12:28] <ak_dev> and kubectl get pods returned the worker
[12:28] <ak_dev> looks like it is running then?
[12:29] <ak_dev> kjackal: 'kubectl get nodes' i mean
[12:30] <ak_dev> this is a confirmation of sorts, that it runs ! all i need to do is now modify kubernetes-master and kubernetes-worker code with my args i guess
=== mpontillo_ is now known as mpontillo
=== icey_ is now known as icey
=== nottrobin_ is now known as nottrobin
=== mup_ is now known as mup
[12:47] <ak_dev> kjackal: kjackal___: i need to figure out how to add parameters to the charms, how do i do that?
[12:48] <ak_dev> I am not sure what i did differently when i restarted
[12:48] <ak_dev> except that i restarted without the SSL
[12:55] <kjackal__> ak_dev: nice to hear you got it working!
[12:55] <kjackal__> for args... let me show you....
[12:58] <kjackal__> ak_dev: https://github.com/kubernetes/kubernetes/blob/master/cluster/juju/layers/kubernetes-master/reactive/kubernetes_master.py#L833 here is where we set the params for some services
[12:59] <kjackal__> Since you are adding a new cni that replaces flannel it might make sence to start a discussion on how we do this the right way...
[13:01] <kjackal__> ak_dev: I will bring it up to the rest of the team. We should discuss this. I think the best place for this would be an issue on the bundle
[13:02] <ak_dev> kjackal__: I did add some things at the place you mentioned, but I couldnt see my args when it actually ran
[13:03] <ak_dev> kjackal__: yes, we can discuss this in an issue
[13:03] <ak_dev> great idea :-)
[13:03] <ak_dev> just for reference, I am working with Narinder Gupta from canonical, my mentor for the OPNFV project
[13:03] <ak_dev> under JOID
[14:24] <kjackal> ak_dev: would you be able to open an issue describing exactly what params you need available on which services? It would also be nice to have some pointers on the charms you have including the source so we can see exaxtly what is happening. For example opening specific ports should be easy.
[14:24] <ak_dev> kjackal: yes, I am writing it up right now
[14:24] <ak_dev> will ping you once done :)
[14:27] <kjackal> thank you
[15:01] <ak_dev> kjackal: kjackal__ : done
[15:02] <ak_dev> here is the link : https://github.com/juju-solutions/bundle-canonical-kubernetes/issues/350
[15:02] <kjackal> ak_dev: awesome thank you
[15:02] <ak_dev> do tell me if you require me to edit anything, and thank you the most :-)
[15:02] <ak_dev> you helped a lot
[15:07] <kjackal> Anytime ak_dev. We have some more things to do. Thank you for your work
[15:11] <ak_dev> :-)
=== frankban is now known as frankban|afk
[18:35] <catbus> Hi, how long does it take for charms to show up on jujucharms.com? I tried both charm and bzr, still don't see the charm on jujucharms.com
[18:41] <rick_h> catbus: the charmstore no longer uses ingestion to pull from bzr. The charm command is used to push things up. Once up you need to publish them and set the permissions such that everyone can see them
[18:41] <rick_h> catbus: let me find the docs page real quick
[18:42] <rick_h> catbus: https://jujucharms.com/docs/stable/authors-charm-store#submitting-a-new-charm
[18:43] <catbus> rick_h: ok, I will try it with 'charm' again.
[18:43] <catbus> thanks.
[18:56] <catbus> rick_h: it works now, thanks.
[18:56] <rick_h> catbus: <3 good to hear