=== axw_ is now known as axw === frankban|afk is now known as frankban [07:50] Good Morning, I've got a question wrt. Landscape (standalone) and MAAS. My aim is to use autopilot to deploy OpenStack. In my initial MAAS node commisoned nodes, i only had single nics. Landscape/Autopilot complained, so I hooked up one more network, recommissioned that node. However, Landscape/Autopilot did not detect the change. So I then removed the node, and started it from scratch, and commissioned it.. MAAS detected the new network automatically, b [09:22] hi, is it mandatory to specify the network spaces while deploying the applications into LXD? === rumble is now known as grumble [13:42] has anyone here actually ever successfully deployed Juju to a vsphere host? [13:43] it apparently needs a direct connection to the vsphere host, as well as the API [13:43] something which isn't documented anywhere. === pmatulis_ is now known as pmatulis [13:59] gaurangt-: basically if you use spaces somewhere in the model then you have to do it everywhere to make sure it's clear. If there's no spaces in the model then it should just work sans spaces. [14:00] orf__: I've not, but some folks have as they've tested the documentation and stokachu had some updates about conjure-up working better with vsphere recently [14:00] orf__: http://blog.astokes.org/conjure-up-dev-summary-aws-cloud-native-integration-and-vsphere-3/ [14:00] orf__: yea juju needs to actually talk to the api [14:00] orf__: im not sure how else it would work [14:01] as for the host access im not entirely sure on that [14:12] Hello all, Is there a way to search all controllers/models for a aws instance-id? [14:13] rick_h, thanks.. that's what I have observed too. [14:13] stokachu: sure, but it tries to contact the vsphere *host* [14:13] which is firewalled off, as it should be [14:13] `juju.cmd.juju.commands bootstrap.go:492 failed to bootstrap model: cannot start bootstrap instance: failed to create instance in any availability zone: uploading ubuntu-xenial-16.04-cloudimg.vmdk to https://10.32.252.51/nfc/52774700-37f1-4a46-cc1f-de20c50f94e5/disk-0.vmdk: Post https://10.32.252.51/nfc/52774700-37f1-4a46-cc1f-de20c50f94e5/disk-0.vmdk: Service Unavailable` [14:13] that IP is the host, the API is accessible [14:13] our vsphere guy says it should upload it to the datastore, then create a VM from that vmdk in the datastore [14:14] it shouldn't be uploading anything to 10.32.252.51 as far as I can tell [14:14] orf__: ok, sec [14:14] thanks for the link rick_h :) [14:16] orf__: can you add your input to https://bugs.launchpad.net/juju/+bug/1711019 [14:16] Bug #1711019: vsphere: cache VMDKs in datastore to avoid repeated downloads [14:16] it's about repeated downloads but also applies to your issue [14:17] orf__: ill make sure it gets on the radar [14:19] thank you :) [14:19] orf__: anytime, sorry about the hiccup [14:34] done, no problem stokachu :) [14:34] orf__: awesome ty! [14:34] I've been shaving yaks with this setup. Going to see if conjur-up dev channel is better [14:34] yea edge is much better [15:43] morning juju world o/ [15:44] morning stormmore === frankban is now known as frankban|afk [16:38] still playing with juju kubernetes-core / canonical-kubernetes .. I can see that once I bring up the world, and deploy microbot as per https://jujucharms.com/kubernetes-core/ that I _can_ reach my service if I access it via the kubernetes-worker/0 machine ip.. but that machine ip is 10.102.82.* and not reachable via my machines adapter address of 10.0.2.15, nor via it's other adapter address of 192.168.1.* .. I feel I'm missing [16:38] something obvious.. [16:38] like in the example url, when it does kubectl get ingress, it has a reply come back with 172.31.26.109 as an address, where as when I do the same, that field is blank. [16:53] hmm.. looks like this might be relevant https://github.com/kubernetes/kubernetes/issues/49614 [17:10] Dwellr: interesting, are you gonna try that fix? [17:10] maybe our ingress controller needs to be updated [17:11] I tried deploying the rbac ingress, but it wouldn't let me create the roles.. [17:11] Error from server (Forbidden): error when creating "https://raw.githubusercontent.com/kubernetes/ingress/master/examples/rbac/nginx/nginx-ingress-controller-rbac.yml": roles.rbac.authorization.k8s.io "nginx-ingress-role" is forbidden: attempt to grant extra privileges: [.... long list of privileges ... ] [17:11] Dwellr: yeah, rbac is not on by default [17:13] well I'm just looking for the simplest way to make this work.. should I figure out how to enable rbac? or figure out how to run a newer ingress that isn't rbac ? [17:15] Dwellr: we have a test bundle with rbac enabled by default if you want to try that [17:15] sure.. how ? =) [17:16] (do I need to start fresh? I'm in a virtualbox pc, so pretty each to spin up a new one.. or is this something I can magically switch to from a non-rbac enabled conjure-up kubernetes-core install) [17:17] you'd need to redeploy. this is something we're working on but isn't released yet [17:18] or you could try updating to a newer ingress that's not rbac enabled [17:18] if there is one [17:18] lets try that first =) [17:19] of course, I already blew away my ingress-controller replication controller thing.. else mebbe I could have just altered that ;p [17:24] yeah.. found this too.. https://github.com/juju-solutions/bundle-canonical-kubernetes/issues/279 [17:27] Dwellr: good find, i'd like to know if that actually fixes your problem [17:34] hmm.. well.. I'm running the new one, and I can still get to the service via it's 10.102.82.39 address, but not via my 192.168.1.* or via 127.0.0.1 from the host etc [17:47] makes little sense to me.. dont understand how other ppl are routing any traffic into their conjured up kubes.. since they seem to live on their own network range, disconnected from the connectivity of the host [17:55] hmm... lxc network attach interface-name kubernetes [17:55] (from a comment on https://stgraber.org/2017/01/13/kubernetes-inside-lxd/) [17:56] although lxc doesn't seem to have a network arg [17:57] oookie.. I'm on lxc 2.0.10 [17:57] sounds like 2.3 changes a lotta stuff [18:00] Dwellr: yea cli arguments changed/updated [18:00] I used conjure up to deploy to lxd .. [18:00] probly explains why my `sudo lxc list` comes back empty when running kube inside lxd ? [18:00] nah we bundled lxd with conjure-up [18:00] conjure-up.lxc list [18:00] oooh.. now there's an idea [18:00] which is changing in the next release [18:00] b/c bundling lxd didnt help us like we thought [18:00] and that gives me version 2.14 [18:00] yea [18:00] that'll have the network commands [18:00] and ... I can see the worker node is connected to my eth0 when I need it connected to eth1 [18:00] this might be what I'm looking for =) [18:00] actually scratch that [18:01] eth0 is the lxd's eth0 not mine =) [18:03] so the worker node has docker0, eth0, cni0, and flannel.1 network interfaces.. and the eth0 has the address that I have to use at the mo to access the worker with the ingress on it.. [18:08] is the conjureup networking documented somewhere so I can figure out what it's trying to do ? [18:09] eg, if I do `conjure-up.lxc network list` I can see it built 2 bridge interfaces.. etc.. [18:09] not too sure why [18:11] Dwellr: unfortunately, no, the reason for the additional bridge was for openstack due to neturon needing an additional network [18:11] Dwellr: this has all been fixed, and i'm prepping a candidate now which you probably should use [18:11] hehe =) just shout when it's good to go =) [18:11] although I'm still learning a load by digging around [18:11] Dwellr: thanks, it's building now shouldnt be to much longer [18:12] Dwellr: lxd will be the snap lxd which is version 2.17 [18:13] like it's great to have seen the lxc list =) .. I tried adding my physical adapter to the worker container via conjure-up.lxc network attach enp0s8 juju-d81eff-1 eth1 .. which returned ok, but conjure-up.lxc list doesn't show it [18:13] what about conjure-up.lxc info juju-d81eff-1 [18:14] does not list an eth1 [18:14] and no address in the Ips: section matches the current ip for enp0s8 [18:15] hmm [18:15] you can edit the profile which should match the model [18:15] so `juju models` [18:15] the conjure-up.lxc profile list [18:16] but thats for all containers using that profile [18:16] not sure why the network attach on the single container didnt update itself with it [18:17] I've not messed with lxc/lxd before =) only docker/virtualbox/vagrant/etc [18:17] so this is all kinda interesting.. more tools to figure out [18:17] cool, https://discuss.linuxcontainers.org/ is a great forum to visit [18:17] for more help [18:17] aye, tho then they kinda want me to understand what the current stuff is trying to do ;p which I'm still figuring out [18:18] :) [18:20] interesting.. ok.. I think mebbe adding it to a profile might work, can I change the profile for a running container? hmm.. think I can.. [18:21] let me try lxc profile copy to clone the current one used by the worker, then assign the worker to the clone [18:22] yea you can change it for running container [18:22] it'll update it [18:25] well.. the profile switcharoo worked, but the container still has no eth1 .. even if I exec into it and check with ifconfig [18:26] mebbe the container needs to restart? [18:27] * Dwellr hits the container with the lxc restart hammer. [18:28] thing is, if I ask lxc network list .. it says the enp0s8 device is used by 1 container [18:29] and if I do lxc network show enp0s8, I can see it's in use by the worker container [18:33] god i love vmware support. they recommand to use vsphere client 6.0 u3 for resizing a lun on vsphere 6.5. that went well. we just lost 13TB of data [18:33] im so happy right now i could die [18:34] 13tb.. ouch [18:34] you has backups.. right ? [18:35] we have backups but they are from last night. and its a sql server where the customer migratates big data into it the whole day .. so basically we lost a whole day [18:36] the good thing is the log backups didnt work [18:36] :D [18:36] and nobody cared [18:37] im not vmware team just storage and linux/unix. so its not my business to check [18:38] so customer looses a day + restore time [18:42] thank god im already at home and there is beer [18:44] stokachu: ahh.. mebbe I can't add a physical device directly to a profile .. mebbe it has to be a bridge.. [18:44] ah [18:44] yea [18:45] Hi, I'm having problems getting a bootstrap done to a private openstack cloud, I've generated the image meta-data, and either locally, or http hosted, it fails for "index file has no data for cloud" [18:46] this is gonna my my head hurt =) I've got enp0s8 on this system that's a physical interface as far as it knows, but is actually a bridge to my real lan (because I'm in virtualbox, with the network set to bridged) .. so I now need to get that interface into my worker container so I can open ports on it.. [18:47] Dwellr: https://www.youtube.com/watch?v=3f57PovdY44 [18:47] ta =) [18:50] aha.. type:nic ... supports nictype:physical [19:13] and this is why I play in vagrant.. ended up somehow messing up my network so that lxc thought my physical adapter (that's actually my bridge to my lan via virtualbox) was now actually a bridge, which somehow caused it to move the real adapter to be eth1, which then conflicted with other stuff in lxc, and eventually it wouldnt let me delete that network because it was 'in use'.. yay.. [19:13] vagrant destroy && vagrant up =) [19:49] ooh.. I found this.. =) https://github.com/evanhempel/lxc-portforward [19:59] hello folks [19:59] i have another CDK question I'm trying to answer before it gets asked again since the first time we tested CDK [20:00] "I was wondering if it is possible to support OpenStack Cinder and NFS StorageClass for testing for now." does that mean anything to anyone?! ;) [20:01] magicaltrout: sure, cdk supports everything that upstream does [20:01] ah yeah that "its the same as upstream" sales pitch ;) [20:01] okay [20:01] magicaltrout: are you asking for how to do it? [20:02] hehe, no just getting an answer [20:02] i can fiddle around to figure it out [20:10] any around that can help with getting bootstrap going on openstack? [20:15] hml: have a few min to help out xarses ? or beisner is someone around that might know the process a bit better? [20:15] sure [20:15] xaras: how can I help? [20:16] xarses ^^ [20:17] trying to get going. generated metadata, either passed as `--config image-metadata-url` and a webserver, or via `--metadata-source /path/to/local` I always get "skipping index ... because of missing information: index file has no data for cloud" [20:18] xarsas: that shoulds like the path provided isn’t enough for juju to find it. if you do the bootstrap with —debug, the path juju is searching at will be shown - [20:18] xarasa: you can then change the part of the path you’re providing to [20:19] it find the index when i have the stream data hosted on the webserver, and implies the same over file [20:19] it just refuses to find my cloud name in the index [20:20] the generated data doesn't explicity have a cloud name in it [20:20] I'm guessing its looking for some pattern match, but no clue what pattern its looking for [20:21] xarsas: can you provide a pastebin of the bootstrap output please? [20:21] I'd have to redact a bit of data, but sure [20:25] xarsas: that should be okay [20:29] great we are restoring 13Tb with less then 3Gbit bandwidth..life is good [20:31] hml: https://gist.github.com/xarses/307a07d290fcc9f48008b3ae1d192f05 [20:36] hahahaha... i know what rick_h did: https://github.com/juju/charmstore-client/issues/143 [20:36] xarses: juju is looking for the openstack endpoint and region provided with the openstack cloud config within the index.json… and can’t find it. [20:37] kwmonroe: :) [20:37] kwmonroe: 3 times now... [20:37] i've done that a bunch of times :'( [20:38] its the saddest thing ever [20:38] xarses: the path to the index.json file listed is correct yes? there are some files not found messages above [20:38] ya, one is found [20:38] so, fwiw rick_h, if you would "charm proof" before you "charm push", you'd see some bizaro (albeit informational) output. that would tell ya not to push :) [20:39] hml: ya, that's exactly what I suspect, however the directions for generating the metadata don't have any context for providing the cloud only the region is reflected in the index.json file [20:39] kwmonroe: but I'm happy. my interface updates work, charm is working, woot woot [20:39] just have to find a path through code review now he [20:39] heh [20:40] xarses: the cloud is defined by the endpoint in the metadata [20:40] well, then the endpoints match [20:41] xarses: i’m thinking the error messages aren’t good. [20:41] xarses: does this file exisit: http://somelocalhost:8000/images/streams/v1/index.json [20:42] at that exact location? [20:44] hml: https://gist.github.com/xarses/307a07d290fcc9f48008b3ae1d192f05#file-gistfile2-txt [20:45] * hml lookin [20:45] xarses reminds me of xerces which makes me real sad because those Java libraries are a right PITA...... [20:45] java is a right PITA.... [20:45] =) [20:46] as a java developer, i am okay with it, some old shit is the worst though :) [20:47] of course the other pun with that nick is you could say Java is a right Pain In The xarses ....... [20:47] its been a long day [20:48] hml, I also just posted the metadata generate-image cmd and output [20:48] well, it'd have to be "Pain In The xArses" because that's how acronyms work magicaltrout. [20:50] I've partly followed https://jujucharms.com/docs/stable/howto-privatecloud, I haven't done any of the switf nonsense since I dont have an object store, I'm just using python -m SimpleHTTPServer on the folder [20:50] xarses: found the updates - trying to find what’s going on here… not jumping out at me [20:50] I guess I should add this random endpoint that they added to the catalog though [20:51] xarses: the endpoint added for product-streams assumes that swift etc is used [20:51] its a http get source at that point, adding it shouldn't matter [20:52] but ya, thats what I initially thought [20:52] but this output is useless for triaging this issue [20:53] I was hoping that ya'll would have a better idea of what's up [20:53] xarses: the usual problem is when the front piece of the path for the metadata doesn’t match what juju is expecting and it can’t find the file [20:54] xarses: i’m concerned about the file not found messages in the output [20:55] well, generate-image didn't make any of those [20:55] should I change the cloudname from custom? [20:56] xarses: no - mine says the same [21:01] uh, I jut regenerated it a bunch more times with out the endpoint. it looks like I may have had a problem with the region name I passed to generate-image [21:02] urgh, yep looked back in the data I redacted, the region name was slightly transposed [21:03] xarses: what would do it. [21:03] * xarses with no hair left to pull out, pulls out random stubble [21:04] ok, so now it doesn't respect the zone I passed [21:04] so how do I control the availability zone passed? [21:04] xarses: yes, openstack is the hardest to bootstrap [21:05] lol, looks like it went through every az and finally used the one that worked with the network I passed [21:05] xarses: yes, it will do that - though there are some bugs there… [21:05] although its still not the az I wanted [21:05] zone appears to be valid in the models [21:06] is there an option that bootstrap will take? [21:06] xarses: if the network AZ name doesn’t match the AZ for the compute nodes… so you might have gotten luckily [21:06] xarses: looking for the option [21:06] no, we don't have a version of openstack that has a working version of both [21:06] network az don't really do anything useful in mitaka [21:07] and we have routed provider networks, but the code that make provisioning work with out forcing both network and az is only present in oakta [21:08] tever, if the instance will come up then I can image it and re-launch it where I need [21:09] hmm, it seems to be waiting on "sudo: unable to resolve host juju-e290f0-controller-0" [21:12] xarses: not sure i’ve seen that one? [21:12] xarses: sometimes the connection take a bit though [21:12] we don't have a dns service [21:12] it looks like it set up a new security group [21:12] that doesn't accept icmp [21:12] xarses: that should be fine… i’m not running it either [21:13] xarses: yes it does setup a new sec group [21:14] ah, yep doesn't accept icmp [21:14] but does accept 22 [21:14] of course it sent the wrong key by default, but network is good [21:14] its just sitting here doing nothing then [21:15] just before it tried to login to the ip, then went to fetch agent tools [21:16] then this sudo unable to resolve [21:16] hmm [21:16] its logged into the thing [21:21] hmm [21:21] xarses: juju bootstrap --to zone=nova - to specify the AZ [21:21] hml, oh nice thanks [21:21] it looks like its built the instance ok, I've logged into it [21:22] however its stuck downloading https://streams.canonical.com/juju/tools/agent/2.2.2/juju-2.2.2-ubuntu-amd64.tgz [21:22] xarses: so that’s the intance for the controller [21:22] I was able to wget it and it only took like a 30sec [21:22] ya, I'm snooping the ps tree on the contoller [21:23] xarses: new toy? :-) [21:25] 2^19 pieces. Assembly required. For ages 9+. CAUTION: Contains complex parts may cause brain hemorrhaging and lack of cognitive reasoning [21:28] its still stuck here ... [21:29] not sure what to do [21:30] xarses: hrm… [21:32] ahh, figured out the sudo message [21:32] xarses: that one i’m not sure on… the bootstrap does have a timeout on it. it doesn’t ctrl-c well. [21:32] its just a stderr message because the hostname isn't resolvable, otherwise its happy [21:34] strace of the curl command that stuck pulling its socket [21:35] xarses: did you bootstrap with use-floating-ips? [21:35] nope [21:35] xarses: can the instance get to the outside word [21:35] world [21:35] yea [21:35] I was able to download the file fine with wget on the controller [21:37] wallyworld: have you seen where bootstrap gets stuck downloading the tools to the new controller instance…. but you can download them fine by hand to that instance? [21:37] its downloading the file very slowly with this curl command [21:37] but then it like gets stuck [21:38] i haven't seen that, i've seen where the bootstrap instance is firewalled and can't download at all [21:40] well neat [21:40] curl is broken [21:41] 0 20.8M 0 32768 0 0 633 0 9:36:08 0:00:51 9:35:17 2896 [21:41] 0 20.8M 0 32768 0 0 498 0 12:12:18 0:01:05 12:11:13 0 [21:50] uhg, something on the network here must be blocking it [21:50] I can't fetch the file at all now [21:50] * xarses continues to bang head against desk [21:57] xarses: can the instance get things from a local box? you can provide both images and tools with the metadata flag - though i haven’t tried the tools part. [21:58] I was looking though bugs that implied that both can't be passed as args [21:59] its supposed to be able to get things, but my box running the command can't fetch the file currently either [21:59] xares: if you have the images and tools in the same directory structure - it would work. [21:59] can I generate the metadata for this too? I can get the file from much futher parts in the network [22:00] * xarses tries to get off this marry-go-round [22:01] hml: i think so… looking for how it works. [22:01] xarses: ^^^ I can’t always type :-) [22:06] xarses: i just had to put the tools in a specific directory relative to where i put the images… will gather a pastebin for you - [22:06] thx [22:12] xarses: https://paste.ubuntu.com/25441295/ [22:12] xarses: i’m not sure what will happen if you try the images and tools in different locations on the cli [22:12] xarses: i do have a the product-streams service configured too [22:13] xarses: i downloaded the juju-2.2.2-ubuntu-amd64.tgz from streams.canonical.com - just get the on which matches you’re version of juju and the machine type [22:14] ya, 2.2.2 [22:14] I have the url that the controller is trying to use [22:15] xarses: that’s what i used [22:40] sigh, it finally died trying on gui [22:42] and on the re-run, its just sitting around waiting for connect [22:43] DEBUG juju.provider.common bootstrap.go:497 connection attempt for ... failed: ssh: connect to host ... port 22: Connection refused [22:43] repeated several times, don't have the tools copy set up yet [23:01] yay, slowly getting further every time