/srv/irclogs.ubuntu.com/2017/10/13/#juju.txt

=== salmankhan1 is now known as salmankhan
wpkIs it possible to deploy Kubernetes with conjure-up to a non-default VPC on EC2?11:35
mark-dickieHello all! I'm quite new to juju and am writing a charm which utilises layer:snap but it never seems to actually install the snap. Is there anyone here who might know what I've done wrong.12:37
boolmanI'm having issues with a charm, https://github.com/MartinHell/charm-collectd/blob/6338fe9d99d8c8c4f510cff28cf617aebdd6f901/reactive/collectd.py#L220  "AttributeError: module 'charmhelpers.fetch' has no attribute 'archiveurl'"12:43
boolmannvm i fixed it13:13
EdShello :) I've brought up canonical-kubernetes using juju, after having conjure up fail. I think I was left with a kubernetes setup that has lots of the settings as per the defaults with conjure up. Would someone be able to advise, for example, how I'd repeat this process to get the kubernetes "external" IPs to be in a subnet of my choosing?13:29
EdSIf it makes any difference, we're hosting this ourselves and it's all provisioned through MAAS13:30
kjackal_hi EdS, I can give it a try13:38
EdShi kjackal :) thank you!13:39
kjackal_EdS: you are deploying canonical-kubernetes13:39
kjackal_what do you mean by "external" Ips?13:40
EdSok, sorry for my terminology. I mean the IP addresses assigned to services that I expose.13:40
kjackal_ok how do you expose the services? nodeport?13:41
EdSthe kubernetes cluster can "expose" a service and it is then assigned an "external ip"13:41
EdShowever, I've never seen anywhere where I can define the CIDR for these addresses13:41
EdSConjure up appeared to allow me to set the desired properties of kubernetes, but did not work.13:43
EdSJuju has worked really smoothly, but I missed out on all the tweaking that would make this new kubernetes cluster usable to us!13:44
EdSyes, I have exposed the first test service with nodeport13:44
EdSand I have ended up with a seemingly random IP 10.1.63.1013:44
kjackal_the 10.1.63.10 is one of the kubernetes nodes, right?13:45
EdSno, the nodes are on 10.10.10.0/1613:46
EdSah, sorry, that's the IP of the pod13:46
kjackal_k8s has a service-cidr config variable13:46
EdSok, brilliant, that sounds like the right thing.13:47
kjackal_can you show me a juju config kubernetes-master13:47
EdSthe exposed service, if I read this right, is 10.152.183.9713:47
kjackal_that sounds better because the service-cidr has a default value of: 10.152.183.0/2413:48
EdSaha ok13:48
EdSso, I think the question is now much simpler. Do you know how to set that? :p13:49
kjackal_but you cannot change the service-cidr after the initial deployment13:49
EdSok, that's fine13:49
kjackal_you will need to redeploy k8s13:49
EdShow would I set it, at all?13:49
kjackal_give me a sec looking for the documenttion page13:49
kjackal_aaah it will be faster if I just tell you13:50
EdSI think, with juju it was so smooth it felt like magic (ok, so it's in the name) that important things like this were missed (at least for me) because of my half-success with conjure up perhaps leaving config around? If that's even possible to happen? IDK13:50
kjackal_thats a good suggestion13:51
kjackal_so what we will do is to grab the bundle from the store change the config variable and deploy it13:52
EdSok, I have that already as I had to tweak constraints13:53
kjackal_can you do a "charm pull canonical-kubernetes"13:53
EdS:)13:53
kjackal_awesome13:53
kjackal_so you go under the kubernetes-master service and you set the service-cidr to what you need13:54
kjackal_let me do this here so I tell you exactly how this looks13:54
cory_fuEdS: When you say that conjure-up failed, can you give me more info?  I don't know much about the k8s side, but I'd like to sory out any issues with conjure-up at least.13:54
kjackal_EdS: it should look like this: http://pastebin.ubuntu.com/25732311/13:56
EdSoh wow :)13:57
EdSok will give that a shot.13:57
EdSCory, two seconds. :)13:57
EdScory_fu: I have a feeling that I was running into several things at once. I'm hunting a few tickets13:59
EdSfirst one; too many machines used, so it ran out of machines to provision14:01
EdSlike this: https://github.com/conjure-up/spells/issues/6714:03
EdSexcept our scenario was less extreme than 4->1814:03
cory_fuWe just had a discussion yesterday about having Juju do better about verifying MAAS / cloud limits / availability early on.  :/14:03
EdSthanks so much for your help kjackal, that had eluded me for ages14:03
EdSlol yeah, might help me out. I unpacked a lot of extra machines trying to get around this14:04
EdSbut got it going in the end.14:05
cory_fuEdS: Odd.  I thought that the "too many machines" bug was resolved already.  Any chance you still have the ~/.cache/conjure-up/conjure-up.log file?14:06
EdSwhile you're here... can you satisfy an enquiring mind? did my conjure-up attempts store configuration that was used in a subsequent attempt with juju and a bundle file I specified myself? Or am I over thinking this?14:06
EdSThis wasn't exactly in the last few days. I can go digging and see if I have it.14:07
EdSooh lots of evidence :/14:07
EdSshall I pastebin?14:08
cory_fuEdS: Not currently.  If you don't go past the "Configure Applications" screen and click the "Deploy All" (or every individual deploy) button, nothing will get saved14:08
cory_fuWell, technically, we were planning on having a resume feature, so we might persist choices into a sqlite db in that ~/.cache/conjure-up directory, but they're never read in again14:09
cory_fuEdS: Yeah, pastebin of the log would be helpful.14:09
EdSok, thanks, that clears up a few doubts14:09
cory_fujam: Hey, can you confirm if a unit's IP address changes due to DHCP whether Juju would trigger a config-changed hook?14:15
EdSmy conjure-up log... sorry about many times I tried this... http://pastebin.ubuntu.com/25732388/14:15
jamcory_fu: so we trigger config-changed on startup anyway, but I'm not 100% sure about where we ended up from auto-populating private-address with new values because of charms that override the value. (openstack charms used to set the VIP instead of their personal addresses)14:16
jamthat said, if a live machine changes its IP address, I think we'll notice within 10 minutes or so, I'm not sure if that immediately triggers a config-changed.14:16
cory_fuEdS: From that log, it looks like you might have had several successful runs.  Did any of those actually succeed or did they get stuck?14:30
EdSIt always got stuck, but that may have been because of various external things.14:31
EdScory_fu: I was setting up MAAS, juju and reading lots.14:31
cory_fuEdS: Odd.  If it got stuck deploying, I would have expected to see log messages about 00-deploy_done failing14:35
EdScory_fu & kjackal: :D thanks so much - that has straightened a lot out in my head!14:36
EdSit's entirely possible I have cleared out the log of the failed runs, but it never felt like I truly succeeded with conjure up14:37
cory_fuEdS: I do see some failures in there related to the connection to the controller failing.  That seems plausible if the machines were provisioned and not released.14:40
cory_fuEdS: You asked about it saving info; as I mentioned, there shouldn't be any persistent effects if you stop before the deploy, but from the log, it looks like you went that far a few times.  Obviously, you'd have to clean up any provisioned machines or anything else that Juju or conjure-up claimed in MAAS14:42
EdScory_fu: yeah. I managed those bits. I think the difference between doing it with conjure-up and juju tricked me into thinking I'd get a similar opportunity to tweak the settings. When juju + maas worked it was all up, but now I realise with defaults, not any leftover config.14:46
EdScory_fu: I think as I'm at the early stages of this setup, I'll tear it all down and try to get the settings I wanted :)14:47
cory_fuEdS: Ok.  If you end up trying conjure-up again with any MAAS issues sorted out and have any issues again, let me or stokachu know.  We're travelling, so might not respond right away, but we'd like to sort out any bugs you might run in to.15:12
cory_fuBut Juju direct is also entirely viable and should be just as configurable, even if it might not be presented as nicely.  (At the end of the day, conjure-up is just calling out to Juju, after all.)15:13
EdScory_fu: superb, thanks you. I'm just setting off from the start again with juju + the bundle. I think personally, the yaml is fine for me. Enjoy your travels.15:14
BarDwellerHiya.. I know I had this working before.. but then I wiped that box & started again.. I'm trying to have my kubernetes (loaded via conjure-up) to use my docker registry (running on the host that did conjure up) .. I thought I used juju run-action registry to make this work before, but that seems to be for secured registries, and mine is unsecured..17:57
BarDwellerI found https://insights.ubuntu.com/2017/10/11/private-docker-registries-and-the-canonical-distribution-of-kubernetes/  which hints I need to set a config key .. which I think is now 'docker-opts' not 'docker-config' as in the article.17:57
EdShow's the config here look? https://insights.ubuntu.com/2017/10/11/private-docker-registries-and-the-canonical-distribution-of-kubernetes/17:57
EdSTim passed me the link here the other day :)17:58
BarDwelleryeah.. thats the link I just pasted right ? ;p17:58
EdSoh haha sorry17:58
EdSdocker-opts sounds familiar from recent docker versions17:59
BarDwelleranyways.. I've done "juju config kubernetes-worker docker-opts="--insecure-registry 192.168.1.xx:2375" .. do I also need to do the juju run-action registry step ?17:59
BarDweller(because atm, if I have an image: tag in my yml for 192.168.1.xx/myimage:latest it complains getsockopt connection refused)18:00
EdSnot if you've already got the registry, it sounds like you have.18:00
BarDwellerI have a registry running at 192.168.1.xx :2375 that I can talk to, push images to, run containers on etc18:00
BarDwellerseems tho that my worker node can't talk to it.. I'm missing something.18:01
EdSyeah. don't deploy a registry with juju run action then :)18:01
EdSis your registry in the same subnet as nodes?18:02
tvansteenburghBarDweller: i would juju ssh to the node and try a docker pull from there, and see what that tells you18:03
tvansteenburghsounds like a networking issue18:03
BarDwellergood plan..18:03
BarDwellerjuju ssh kubernetes-worker/0    .. and then `docker images` is showing me a different docker registry..  but from that env I can ping my other one ok.. lemme see if I change DOCKER_HOST if I can talk to my other reg from that shell18:04
BarDwelleryep18:05
BarDwellerso the kubernetes-worker/0 is capable of reaching my docker registry, and can talk to it.. but seems configured to use a different registry18:05
BarDwellerhmm.. do I need to do something after the juju config that tells the worker to use my registry? (restart the worker or sommat?)18:06
tvansteenburghjuju config kubernetes-worker - do the docker-opts have the correct registry in that output?18:06
tvansteenburghBarDweller: when you set it via config, the charm should do everything for you18:07
tvansteenburghif it's not, that's a bug18:07
BarDwelleryes, juju config kubernetes-worker shows the options I put in docker-opts (--insecure-registry 192.168.1.xx:2375)18:09
BarDwelleranyway I can kick it to tell it to read it ?18:10
BarDwellerhmm.. wait up18:10
BarDwellervagrant ssh in the kubernetes-worker/0 then docker info shows my registry listed in there.. digging further18:11
EdShmm. I have torn down my canonical-kubernetes setup to rebuild with a different service-cidr. This is now stuck waiting with flannel blocked :/18:19
EdSI think I will try again, I have noticed the 1.7->1.8 version bump.18:19
BarDwelleryeah.. there's something odd here.. it's not a network issue, it's a docker config issue.. I'm trying variants atm18:26
BarDwellermebbe it'd be easier if I just started using the registry from the juju charm ?18:27
tvansteenburghBarDweller: if you're just playing around that's fine but it's not a production setup18:28
BarDwelleryeah.. this isn't for prod, it's for local dev18:28
BarDwellerI just need a way to push custom images that I can load into the kube =)18:29
EdSfor production systems, used only internally within our company, would you consider it ok to run a registry pod/service with images stored in an nfs PV?18:32
EdSI'm not sure we need or want to give docker-registry a server of it's own. It seems overkill for us.18:33
tvansteenburghBarDweller: gotcha. i'm still keen to figure out what the issue is so we can fix it if we need to18:37
tvansteenburghEds: yes - the helm chart in that blog post is great for that18:37
BarDwelleryeah.. I'm still digging.. I'm not entirely sure I've got everything lined up right18:38
EdStvansteenburgh: thank you :)18:39
BarDwellerI know if I do `juju ssh kubernetes-worker/0` and then do `export DOCKER_HOST=192.168.1.xx:2375` and then do `docker images` that I can see my expected images18:40
BarDwellerso I know my docker is up, and reachable by the worker node.18:40
tvansteenburghBarDweller: okay, that's good feedback - we can try to reproduce18:41
BarDwellerso then I do `unset DOCKER_HOST` and then `docker info` and I note at the bottom it lists "Insecure Registries: 192.168.1.xx:2375"18:41
tvansteenburghhmm18:42
BarDwellerand then I try 'docker pull 192.168.1.xx:2375/my-image:latest` and it says error image not found.18:42
BarDwellerwhich is an improvement from before.. where it was saying getsockopts conn refused.18:43
tvansteenburghBarDweller: would you mind filing a bug with this info here: https://github.com/juju-solutions/bundle-canonical-kubernetes/issues/new18:43
BarDwellerI had this working a few weeks back.. I have the kube yamls that say so.. but I wiped the host I'd done the magic to add the registry on.. and failed to add what I did to my vagrantfile =)18:44
BarDwellermebbe my registry isn't the right thing18:45
knobbyI assume your docker images when you set the DOCKER_HOST shows my-image.18:45
knobbyI use an insecure registry hosted outside k8s and all I had to do was add that option18:46
BarDwellerI'm seeing people saying they can do things like http://ip:port/v2/_catalog to see images.. mine doesn't seem to like that just gives back "{"message":"page not found"}18:46
knobbydocker pull 192.168.1.xx/image_name just works18:46
knobbyare you running a version 1 registry?18:46
BarDwelleryes, if I set my DOCKER_HOST to be 192.168.1.xx then docker images will show my-image18:46
BarDwellerchecking..18:46
BarDwellerapparently I'm running 17.09.0-ce, api version 1.32 (min ver 1.12) build date sep 26 201718:47
BarDweller(from docker version while docker host is set)18:48
BarDwellerI wonder if I don't have a docker registry, I just have a docker server.. #noobquestion is there a difference ?18:48
knobbyBarDweller: how are you running it? the docker registry is a docker container named registry18:49
knobbyI'm running registry:2 for example18:49
knobbywith DOCKER_HOST working it sounds like you're using a docker daemon instead of a registry18:50
BarDwellerloosely .. apt-get install -y docker-ce socat .. then update dockerd options to pass -H tcp://0.0.0.0:237518:50
BarDwelleryes, that's the realisation I'm coming to (re daemon vs registry)18:50
knobbyah, yep. a registry is typically on port 5000 and is run via something like `docker run -p 5000:5000 registry:218:51
BarDwellerok.. so should I change my original question to.. is there a way to have my kube-worker pull an image from my docker-daemon ?18:51
knobbyBarDweller: I would think it would be easier to crank up a registry myself.18:53
knobbyBarDweller: docker run -p 5000:5000 -e REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/var/lib/registry -v /my/registry/volume:/var/lib/registry registry:218:54
knobbyBarDweller: something like that would do it18:54
BarDwellerhehe.. sounds like an idea.. I'll have a bash18:55
BarDwelleralthough you'd kinda think by this point I should just use the juju docker-registry charm18:56
knobbyBarDweller: if you have a machine for juju to snap up for it, sure. For me, I'm using bare metal and didn't want to waste resources on something that is used so infrequently. I also was able to put it on the nfs server, so file io was local18:57
BarDwellerI have all this inside a vagrant vm.. so it really doesn't make too much difference.. at the mo the vm is running the dockerd .. I'll try deploying a registry first, because that might integrate easier18:58
knobbyBarDweller: sounds like a good idea18:58
BarDwellerouch.. I think I figured this out =)19:14
BarDwellerso to have docker client talk to an insecure registry, you add the --insecure-registry option to the dockerd, (or use daemon.json) however, if the docker you are using is remote, you do it to _that_ docker .. which is awesome in my case, because it means the clients of my vm wont need to care19:17
BarDwellercool.. my image came up finally inside kube =) thanks for the assist =)19:29
knobbyBarDweller: glad to hear you go it going!19:36
BarDwelleryep.. I think before I had used the juju charm to deploy a registry.. but it's not clear to me how I ever had that working, because I never configured anything beyond 'domain'  and set ingress =true .. I never had all the insecure-registry stuff before19:39
BarDwelleranyways.. updated vagrantfile to not do that, and instead use juju config to add the insecure registry bit for the registry launched onto the docker daemon as part of the provisioning19:40
skayI'm seeing "Too many arguments." during config-changed, and I can't figure out where it's coming from19:53
skayI've grepped all of the juju code base19:53
EdSis that bash's too many arguments?19:56
EdSis something being expanded to a long list, eg ls * in a folder with many thousands of files will trigger that, IIRC19:57
skayhmm, I'll have to dig around to see if anything like that is happening20:00
EdSthat was off the top of my head, sorry if I'm way off the mark!20:05
skay:)20:09
=== frankban is now known as frankban|afk

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!