/srv/irclogs.ubuntu.com/2017/08/31/#juju.txt

=== axw_ is now known as axw
=== frankban|afk is now known as frankban
parlosGood Morning, I've got a question wrt. Landscape (standalone) and MAAS. My aim is to use autopilot to deploy OpenStack. In my initial MAAS node commisoned nodes, i only had single nics. Landscape/Autopilot complained, so I hooked up one more network, recommissioned that node. However, Landscape/Autopilot did not detect the change. So I then removed the node, and started it from scratch, and commissioned it.. MAAS detected the new network automatically, b07:50
gaurangt-hi, is it mandatory to specify the network spaces while deploying the applications into LXD?09:22
=== rumble is now known as grumble
orf__has anyone here actually ever successfully deployed Juju to a vsphere host?13:42
orf__it apparently needs a direct connection to the vsphere host, as well as the API13:43
orf__something which isn't documented anywhere.13:43
=== pmatulis_ is now known as pmatulis
rick_hgaurangt-: basically if you use spaces somewhere in the model then you have to do it everywhere to make sure it's clear. If there's no spaces in the model then it should just work sans spaces.13:59
rick_horf__: I've not, but some folks have as they've tested the documentation and stokachu had some updates about conjure-up working better with vsphere recently14:00
rick_horf__: http://blog.astokes.org/conjure-up-dev-summary-aws-cloud-native-integration-and-vsphere-3/14:00
stokachuorf__: yea juju needs to actually talk to the api14:00
stokachuorf__: im not sure how else it would work14:00
stokachuas for the host access im not entirely sure on that14:01
rahworkxHello all, Is there a way to search all controllers/models for a aws instance-id?14:12
gaurangt-rick_h, thanks.. that's what I have observed too.14:13
orf__stokachu: sure, but it tries to contact the vsphere *host*14:13
orf__which is firewalled off, as it should be14:13
orf__`juju.cmd.juju.commands bootstrap.go:492 failed to bootstrap model: cannot start bootstrap instance: failed to create instance in any availability zone: uploading ubuntu-xenial-16.04-cloudimg.vmdk to https://10.32.252.51/nfc/52774700-37f1-4a46-cc1f-de20c50f94e5/disk-0.vmdk: Post https://10.32.252.51/nfc/52774700-37f1-4a46-cc1f-de20c50f94e5/disk-0.vmdk: Service Unavailable`14:13
orf__that IP is the host, the API is accessible14:13
orf__our vsphere guy says it should upload it to the datastore, then create a VM from that vmdk in the datastore14:13
orf__it shouldn't be uploading anything to 10.32.252.51 as far as I can tell14:14
stokachuorf__: ok, sec14:14
orf__thanks for the link rick_h :)14:14
stokachuorf__: can you add your input to https://bugs.launchpad.net/juju/+bug/171101914:16
mupBug #1711019: vsphere: cache VMDKs in datastore to avoid repeated downloads <juju:Triaged> <https://launchpad.net/bugs/1711019>14:16
stokachuit's about repeated downloads but also applies to your issue14:16
stokachuorf__: ill make sure it gets on the radar14:17
orf__thank you :)14:19
stokachuorf__: anytime, sorry about the hiccup14:19
orf__done, no problem stokachu :)14:34
stokachuorf__: awesome ty!14:34
orf__I've been shaving yaks with this setup. Going to see if conjur-up dev channel is better14:34
stokachuyea edge is much better14:34
stormmoremorning juju world o/15:43
rick_hmorning stormmore15:44
=== frankban is now known as frankban|afk
Dwellrstill playing with juju kubernetes-core / canonical-kubernetes .. I can see that once I bring up the world, and deploy microbot as per https://jujucharms.com/kubernetes-core/ that I _can_ reach my service if I access it via the kubernetes-worker/0 machine ip.. but that machine ip is 10.102.82.* and not reachable via my machines adapter address of 10.0.2.15, nor via it's other adapter address of 192.168.1.* .. I feel I'm missing16:38
Dwellrsomething obvious..16:38
Dwellrlike in the example url, when it does kubectl get ingress, it has a reply come back with 172.31.26.109 as an address, where as when I do the same, that field is blank.16:38
Dwellrhmm.. looks like this might be relevant https://github.com/kubernetes/kubernetes/issues/4961416:53
tvansteenburghDwellr: interesting, are you gonna try that fix?17:10
tvansteenburghmaybe our ingress controller needs to be updated17:10
DwellrI tried deploying the rbac ingress, but it wouldn't let me create the roles..17:11
DwellrError from server (Forbidden): error when creating "https://raw.githubusercontent.com/kubernetes/ingress/master/examples/rbac/nginx/nginx-ingress-controller-rbac.yml": roles.rbac.authorization.k8s.io "nginx-ingress-role" is forbidden: attempt to grant extra privileges: [.... long list of privileges ... ]17:11
tvansteenburghDwellr: yeah, rbac is not on by default17:11
Dwellrwell I'm just looking for the simplest way to make this work.. should I figure out how to enable rbac? or figure out how to run a newer ingress that isn't rbac ?17:13
tvansteenburghDwellr: we have a test bundle with rbac enabled by default if you want to try that17:15
Dwellrsure.. how ? =)17:15
Dwellr(do I need to start fresh? I'm in a virtualbox pc, so pretty each to spin up a new one..  or is this something I can magically switch to from a non-rbac enabled conjure-up kubernetes-core install)17:16
tvansteenburghyou'd need to redeploy. this is something we're working on but isn't released yet17:17
tvansteenburghor you could try updating to a newer ingress that's not rbac enabled17:18
tvansteenburghif there is one17:18
Dwellrlets try that first =)17:18
Dwellrof course, I already blew away my ingress-controller replication controller thing.. else mebbe I could have just altered that ;p17:19
Dwellryeah.. found this too.. https://github.com/juju-solutions/bundle-canonical-kubernetes/issues/27917:24
tvansteenburghDwellr: good find, i'd like to know if that actually fixes your problem17:27
Dwellrhmm.. well.. I'm running the new one, and I can still get to the service via it's 10.102.82.39 address, but not via my 192.168.1.* or via 127.0.0.1 from the host etc17:34
Dwellrmakes little sense to me.. dont understand how other ppl are routing any traffic into their conjured up kubes.. since they seem to live on their own network range, disconnected from the connectivity of the host17:47
Dwellrhmm... lxc network attach interface-name kubernetes17:55
Dwellr(from a comment on https://stgraber.org/2017/01/13/kubernetes-inside-lxd/)17:55
Dwellralthough lxc doesn't seem to have a network arg17:56
Dwellroookie.. I'm on lxc 2.0.1017:57
Dwellrsounds like 2.3 changes a lotta stuff17:57
stokachuDwellr: yea cli arguments changed/updated18:00
DwellrI used conjure up to deploy to lxd ..18:00
Dwellrprobly explains why my `sudo lxc list` comes back empty when running kube inside lxd ?18:00
stokachunah we bundled lxd with conjure-up18:00
stokachuconjure-up.lxc list18:00
Dwellroooh.. now there's an idea18:00
stokachuwhich is changing in the next release18:00
stokachub/c bundling lxd didnt help us like we thought18:00
Dwellrand that gives me version 2.1418:00
stokachuyea18:00
stokachuthat'll have the network commands18:00
Dwellrand ... I can see the worker node is connected to my eth0 when I need it connected to eth118:00
Dwellrthis might be what I'm looking for =)18:00
Dwellractually scratch that18:00
Dwellreth0 is the lxd's eth0 not mine =)18:01
Dwellrso the worker node has docker0, eth0, cni0, and flannel.1 network interfaces.. and the eth0 has the address that I have to use at the mo to access the worker with the ingress on it..18:03
Dwellris the conjureup networking documented somewhere so I can figure out what it's trying to do ?18:08
Dwellreg, if I do `conjure-up.lxc network list` I can see it built 2 bridge interfaces.. etc..18:09
Dwellrnot too sure why18:09
stokachuDwellr: unfortunately, no, the reason for the additional bridge was for openstack due to neturon needing an additional network18:11
stokachuDwellr: this has all been fixed, and i'm prepping a candidate now which you probably should use18:11
Dwellrhehe =) just shout when it's good to go =)18:11
Dwellralthough I'm still learning a load by digging around18:11
stokachuDwellr: thanks, it's building now shouldnt be to much longer18:11
stokachuDwellr: lxd will be the snap lxd which is version 2.1718:12
Dwellrlike it's great to have seen the lxc list =) .. I tried adding my physical adapter to the worker container via   conjure-up.lxc network attach enp0s8 juju-d81eff-1 eth1  .. which returned ok, but  conjure-up.lxc list  doesn't show it18:13
stokachuwhat about conjure-up.lxc info juju-d81eff-118:13
Dwellrdoes not list an eth118:14
Dwellrand no address in the Ips: section matches the current ip for enp0s818:14
stokachuhmm18:15
stokachuyou can edit the profile which should match the model18:15
stokachuso `juju models`18:15
stokachuthe conjure-up.lxc profile list18:15
stokachubut thats for all containers using that profile18:16
stokachunot sure why the network attach on the single container didnt update itself with it18:16
DwellrI've not messed with lxc/lxd before =) only docker/virtualbox/vagrant/etc18:17
Dwellrso this is all kinda interesting.. more tools to figure out18:17
stokachucool, https://discuss.linuxcontainers.org/ is a great forum to visit18:17
stokachufor more help18:17
Dwellraye, tho then they kinda want me to understand what the current stuff is trying to do ;p which I'm still figuring out18:17
stokachu:)18:18
Dwellrinteresting.. ok.. I think mebbe adding it to a profile might work, can I change the profile for a running container? hmm.. think I can..18:20
Dwellrlet me try lxc profile copy to clone the current one used by the worker, then assign the worker to the clone18:21
stokachuyea you can change it for running container18:22
stokachuit'll update it18:22
Dwellrwell.. the profile switcharoo worked, but the container still has no eth1 .. even if I exec into it and check with ifconfig18:25
Dwellrmebbe the container needs to restart?18:26
* Dwellr hits the container with the lxc restart hammer.18:27
Dwellrthing is, if I ask lxc network list .. it says the enp0s8 device is used by 1 container18:28
Dwellrand if I do lxc network show enp0s8, I can see it's in use by the worker container18:29
ybaumygod i love vmware support. they recommand to use vsphere client 6.0 u3 for resizing a lun on vsphere 6.5. that went well. we just lost 13TB of data18:33
ybaumyim so happy right now i could die18:33
Dwellr13tb.. ouch18:34
Dwellryou has backups.. right ?18:34
ybaumywe have backups but they are from last night. and its a sql server where the customer migratates big data into it the whole day .. so basically we lost a whole day18:35
ybaumythe good thing is the log backups didnt work18:36
ybaumy:D18:36
ybaumyand nobody cared18:36
ybaumyim not vmware team just storage and linux/unix. so its not my business to check18:37
ybaumyso customer looses a day + restore time18:38
ybaumythank god im already at home and there is beer18:42
Dwellrstokachu: ahh.. mebbe I can't add a physical device directly to a profile .. mebbe it has to be a bridge..18:44
stokachuah18:44
stokachuyea18:44
xarsesHi, I'm having problems getting a bootstrap done to a private openstack cloud, I've generated the image meta-data, and either locally, or http hosted, it fails for "index file has no data for cloud"18:45
Dwellrthis is gonna my my head hurt =) I've got enp0s8 on this system that's a physical interface as far as it knows, but is actually a bridge to my real lan (because I'm in virtualbox, with the network set to bridged) .. so I now need to get that interface into my worker container so I can open ports on it..18:46
tvansteenburghDwellr: https://www.youtube.com/watch?v=3f57PovdY4418:47
Dwellrta =)18:47
Dwellraha.. type:nic ... supports nictype:physical18:50
Dwellrand this is why I play in vagrant.. ended up somehow messing up my network so that lxc thought my physical adapter (that's actually my bridge to my lan via virtualbox) was now actually a bridge, which somehow caused it to move the real adapter to be eth1, which then conflicted with other stuff in lxc, and eventually it wouldnt let me delete that network because it was 'in use'.. yay..19:13
Dwellrvagrant destroy && vagrant up =)19:13
Dwellrooh.. I found this.. =) https://github.com/evanhempel/lxc-portforward19:49
magicaltrouthello folks19:59
magicaltrouti have another CDK question I'm trying to answer before it gets asked again since the first time we tested CDK19:59
magicaltrout"I was wondering if it is possible to support OpenStack Cinder and NFS StorageClass for testing for now." does that mean anything to anyone?! ;)20:00
tvansteenburghmagicaltrout: sure, cdk supports everything that upstream does20:01
magicaltroutah yeah that "its the same as upstream" sales pitch ;)20:01
magicaltroutokay20:01
tvansteenburghmagicaltrout: are you asking for how to do it?20:01
magicaltrouthehe, no just getting an answer20:02
magicaltrouti can fiddle around to figure it out20:02
xarsesany around that can help with getting bootstrap going on openstack?20:10
rick_hhml: have a few min to help out xarses ? or beisner is someone around that might know the process a bit better?20:15
hmlsure20:15
hmlxaras: how can I help?20:15
hmlxarses ^^20:16
xarsestrying to get going. generated metadata, either passed as `--config image-metadata-url` and a webserver, or via `--metadata-source /path/to/local`  I always get "skipping index ... because of missing information: index file has no data for cloud"20:17
hmlxarsas: that shoulds like the path provided isn’t enough for juju to find it.  if you do the bootstrap with —debug, the path juju is searching at will be shown -20:18
hmlxarasa: you can then change the part of the path you’re providing to20:18
xarsesit find the index when i have the stream data hosted on the webserver, and implies the same over file20:19
xarsesit just refuses to find my cloud name in the index20:19
xarsesthe generated data doesn't explicity have a cloud name in it20:20
xarsesI'm guessing its looking for some pattern match, but no clue what pattern its looking for20:20
hmlxarsas: can you provide a pastebin of the bootstrap output please?20:21
xarsesI'd have to redact a bit of data, but sure20:21
hmlxarsas: that should be okay20:25
ybaumygreat we are restoring 13Tb with less then 3Gbit bandwidth..life is good20:29
xarseshml: https://gist.github.com/xarses/307a07d290fcc9f48008b3ae1d192f0520:31
kwmonroehahahaha... i know what rick_h did:  https://github.com/juju/charmstore-client/issues/14320:36
hmlxarses: juju is looking for the openstack endpoint and region provided with the openstack cloud config within the index.json… and can’t find it.20:36
rick_hkwmonroe: :)20:37
rick_hkwmonroe: 3 times now...20:37
magicaltrouti've done that a bunch of times :'(20:37
magicaltroutits the saddest thing ever20:38
hmlxarses: the path to the index.json file listed is correct yes?  there are some files not found messages above20:38
xarsesya, one is found20:38
kwmonroeso, fwiw rick_h, if you would "charm proof" before you "charm push", you'd see some bizaro (albeit informational) output.  that would tell ya not to push :)20:38
xarseshml: ya, that's exactly what I suspect, however the directions for generating the metadata don't have any context for providing the cloud only the region is reflected in the index.json file20:39
rick_hkwmonroe: but I'm happy. my interface updates work, charm is working, woot woot20:39
rick_hjust have to find a path through code review now he20:39
rick_hheh20:39
hmlxarses: the cloud is defined by the endpoint in the metadata20:40
xarseswell, then the endpoints match20:40
hmlxarses: i’m thinking the error messages aren’t good.20:41
hmlxarses: does this file exisit:  http://somelocalhost:8000/images/streams/v1/index.json20:41
hmlat that exact location?20:42
xarseshml: https://gist.github.com/xarses/307a07d290fcc9f48008b3ae1d192f05#file-gistfile2-txt20:44
* hml lookin20:45
magicaltroutxarses reminds me of xerces which makes me real sad because those Java libraries are a right PITA......20:45
xarsesjava is a right PITA....20:45
xarses=)20:45
magicaltroutas a java developer, i am okay with it, some old shit is the worst though :)20:46
magicaltroutof course the other pun with that nick is you could say Java is a right Pain In The xarses .......20:47
magicaltroutits been a long day20:47
xarseshml, I also just posted the metadata generate-image cmd and output20:48
kwmonroewell, it'd have to be "Pain In The xArses" because that's how acronyms work magicaltrout.20:48
xarsesI've partly followed https://jujucharms.com/docs/stable/howto-privatecloud, I haven't done any of the switf nonsense since I dont have an object store, I'm just using python -m SimpleHTTPServer on the folder20:50
hmlxarses: found the updates - trying to find what’s going on here… not jumping out at me20:50
xarsesI guess I should add this random endpoint that they added to the catalog though20:50
hmlxarses: the endpoint added for product-streams assumes that swift etc is used20:51
xarsesits a http get source at that point, adding it shouldn't matter20:51
xarsesbut ya, thats what I initially thought20:52
xarsesbut this output is useless for triaging this issue20:52
xarsesI was hoping that ya'll would have a better idea of what's up20:53
hmlxarses: the usual problem is when the front piece of the path for the metadata doesn’t match what juju is expecting and it can’t find the file20:53
hmlxarses: i’m concerned about the file not found messages in the output20:54
xarseswell, generate-image didn't make any of those20:55
xarsesshould I change the cloudname from custom?20:55
hmlxarses: no - mine says the same20:56
xarsesuh, I jut regenerated it a bunch more times with out the endpoint. it looks like I may have had a problem with the region name I passed to generate-image21:01
xarsesurgh, yep looked back in the data I redacted, the region name was slightly transposed21:02
hmlxarses:  what would do it.21:03
* xarses with no hair left to pull out, pulls out random stubble 21:03
xarsesok, so now it doesn't respect the zone I passed21:04
xarsesso how do I control the availability zone passed?21:04
hmlxarses: yes, openstack is the hardest to bootstrap21:04
xarseslol, looks like it went through every az and finally used the one that worked with the network I passed21:05
hmlxarses: yes, it will do that - though there are some bugs there…21:05
xarsesalthough its still not the az I wanted21:05
xarseszone appears to be valid in the models21:05
xarsesis there an option that bootstrap will take?21:06
hmlxarses: if the network AZ name doesn’t match the AZ for the compute nodes…  so you might have gotten luckily21:06
hmlxarses: looking for the option21:06
xarsesno, we don't have a version of openstack that has a working version of both21:06
xarsesnetwork az don't really do anything useful in mitaka21:06
xarsesand we have routed provider networks, but the code that make provisioning work with out forcing both network and az is only present in oakta21:07
xarsestever, if the instance will come up then I can image it and re-launch it where I need21:08
xarseshmm, it seems to be waiting on "sudo: unable to resolve host juju-e290f0-controller-0"21:09
hmlxarses: not sure i’ve seen that one?21:12
hmlxarses: sometimes the connection take a bit though21:12
xarseswe don't have a dns service21:12
xarsesit looks like it set up a new security group21:12
xarsesthat doesn't accept icmp21:12
hmlxarses: that should be fine… i’m not running it either21:12
hmlxarses: yes it does setup a new sec group21:13
xarsesah, yep doesn't accept icmp21:14
xarsesbut does accept 2221:14
xarsesof course it sent the wrong key by default, but network is good21:14
xarsesits just sitting here doing nothing then21:14
xarsesjust before it tried to login to the ip, then went to fetch agent tools21:15
xarsesthen this sudo unable to resolve21:16
xarseshmm21:16
xarsesits logged into the thing21:16
xarseshmm21:21
hmlxarses: juju bootstrap --to zone=nova - to specify the AZ21:21
xarseshml, oh nice thanks21:21
xarsesit looks like its built the instance ok, I've logged into it21:21
xarseshowever its stuck downloading https://streams.canonical.com/juju/tools/agent/2.2.2/juju-2.2.2-ubuntu-amd64.tgz21:22
hmlxarses: so that’s the intance for the controller21:22
xarsesI was able to wget it and it only took like a 30sec21:22
xarsesya, I'm snooping the ps tree on the contoller21:22
hmlxarses: new toy?  :-)21:23
xarses2^19 pieces. Assembly required. For ages 9+. CAUTION: Contains complex parts may cause brain hemorrhaging and lack of cognitive reasoning21:25
xarsesits still stuck here ...21:28
xarsesnot sure what to do21:29
hmlxarses: hrm…21:30
xarsesahh, figured out the sudo message21:32
hmlxarses: that one i’m not sure on… the bootstrap does have a timeout on it.  it doesn’t ctrl-c well.21:32
xarsesits just a stderr message because the hostname isn't resolvable, otherwise its happy21:32
xarsesstrace of the curl command that stuck pulling its socket21:34
hmlxarses:  did you bootstrap with use-floating-ips?21:35
xarsesnope21:35
hmlxarses: can the instance get to the outside word21:35
hmlworld21:35
xarsesyea21:35
xarsesI was able to download the file fine with wget on the controller21:35
hmlwallyworld: have you seen where bootstrap gets stuck downloading the tools to the new controller instance…. but you can download them fine by hand to that instance?21:37
xarsesits downloading the file very slowly with this curl command21:37
xarsesbut then it like gets stuck21:37
wallyworldi haven't seen that, i've seen where the bootstrap instance is firewalled and can't download at all21:38
xarseswell neat21:40
xarsescurl is broken21:40
xarses0 20.8M    0 32768    0     0    633      0  9:36:08  0:00:51  9:35:17  289621:41
xarses0 20.8M    0 32768    0     0    498      0 12:12:18  0:01:05 12:11:13     021:41
xarsesuhg, something on the network here must be blocking it21:50
xarsesI can't fetch the file at all now21:50
* xarses continues to bang head against desk21:50
hmlxarses: can the instance get things from a local box?  you can provide both images and tools with the metadata flag  - though i haven’t tried the tools part.21:57
xarsesI was looking though bugs that implied that both can't be passed as args21:58
xarsesits supposed to be able to get things, but my box running the command can't fetch the file currently either21:59
hmlxares: if you have the images and tools in the same directory structure - it would work.21:59
xarsescan I generate the metadata for this too? I can get the file from much futher parts in the network21:59
* xarses tries to get off this marry-go-round22:00
hmlhml: i think so… looking for how it works.22:01
hmlxarses: ^^^ I can’t always type :-)22:01
hmlxarses: i just had to put the tools in a specific directory relative to where i put the images… will gather a pastebin for you -22:06
xarsesthx22:06
hmlxarses: https://paste.ubuntu.com/25441295/22:12
hmlxarses: i’m not sure what will happen if you try the images and tools in different locations on the cli22:12
hmlxarses: i do have a the product-streams service configured too22:12
hmlxarses: i downloaded the juju-2.2.2-ubuntu-amd64.tgz from streams.canonical.com - just get the on which matches you’re version of juju and the machine type22:13
xarsesya, 2.2.222:14
xarsesI have the url that the controller is trying to use22:14
hmlxarses: that’s what i used22:15
xarsessigh, it finally died trying on gui22:40
xarsesand on the re-run, its just sitting around waiting for connect22:42
xarses DEBUG juju.provider.common bootstrap.go:497 connection attempt for ... failed: ssh: connect to host ... port 22: Connection refused22:43
xarsesrepeated several times, don't have the tools copy set up yet22:43
xarsesyay, slowly getting further every time23:01

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!