/srv/irclogs.ubuntu.com/2017/03/23/#juju.txt

Budgie^Smorethat is what I am saying hatch ... didn't complain about anything unusual creating the model from the CLI but try see it int the web UI it doesn't exist00:00
hatchthat's....odd00:00
hatchlet me give it a try here00:01
Budgie^Smoreoh and the --refresh command is doing http://paste.ubuntu.com/24231722/00:01
hatchhmm ok let me see what I can find out for you00:03
hatchgimme a few minutes00:03
Budgie^Smoresure but I feel I am going to have to use a hosted juju controller to get around this for now00:04
Budgie^Smorenot a huge deal was just using Jaas to avoid having to have a separate juju controller instance00:04
Budgie^Smoreoh that is ashame that you can update the pastebin paste00:09
Budgie^Smoreok nevermind about the --refresh problem, I see what it is... it is trying to refresh a controller that is powered off00:10
hatchyeah, I'm trying to figure out why the models aren't showing up still :)00:11
Budgie^Smorealthough I would suggest that that is technical is a bug since it should really timeout on controllers it fails to access rather than trying indefinitely00:12
hatchoh definitely00:12
Budgie^Smorelooks like it is set to retry every 7 secs too00:13
Budgie^Smorecorrection 2mins 7secs-ish00:13
Budgie^Smorehatch yeah I would kinda expect to see a blank model in the web ui after issuing the add-model command and before doing any deploy command00:15
Budgie^Smorehmmm ok now it is has shown up00:16
Budgie^SmoreI am hitting the replication buffer00:16
Budgie^Smore2 mysteries solved, 1 to go00:18
Budgie^Smorehatch ^00:19
hatch:D00:19
hatchwhich one is to go still?00:20
Budgie^Smorewhy it can't create the instance00:20
Budgie^Smoreand I know what that is just don't think I can prove it easily and more importantly safely... I suspect it is the default vpc issue for the model since I am using 'juju deploy etcd --constraints "instance-type=m3.medium spaces=private"' to deploy it and the "private" space is in a different VPC that what jaas can currently handle00:22
rick_hBudgie^Smore: that's perfectly possible. A bug with what you're trying to do would be great as early beta feedback so we can investigate and see what we need to do.00:23
rick_hBudgie^Smore: as much copy/paste as possible.00:23
Budgie^SmoreI think it is more of a feature request of having the ability to assign a model to a vpc-id some how but that is very cloud specific so I don't know what priority it would be worth giving it00:25
Budgie^Smorefrom a security stand point how it currently is working makes sense too, juju shouldn't step outside the default VPC unless given the ID to do  so00:26
Budgie^Smorehatch ^ make sense?00:28
hatchdefinitely00:28
Budgie^Smorethe bug I will file though is the one lack of a timeout on --refresh when you have controllers that are inaccessible at the point in time00:28
hatchthanks00:29
Budgie^Smoreunfortunately that does mean the end of my "experiment" of JaaS for now00:31
hatchsorry Budgie^Smore00:32
hatchI'll talk with the team tomorrow and see what I can find out about the VPC issue00:32
Budgie^Smoredon't be, it was fun and love to see how far that has come so far... AWS is only temporary for me anyway00:32
hatch:)00:33
hatchI have to run, have a good night Budgie^Smore00:33
Budgie^SmoreI want to see the design spec for when you can use JaaS with say a private MaaS environment00:33
rick_hBudgie^Smore: patience, one step at a time :)00:36
Budgie^Smorerick_h oh I have that, in the meantime I have to use the "workaround", it's all good :)00:41
Budgie^Smorerick_h that comment was more a vision of the potential it has in my "use-case"00:44
=== lifeless_ is now known as lifeless
Budgie^Smorehmmm looks like the example I am trying to "follow" is using a future feature01:42
Budgie^Smore--to "subnet=subnet-bb1ab2dc"01:42
Budgie^Smorehttps://bugs.launchpad.net/juju/+bug/165964001:49
mupBug #1659640: Juju needs to support specifying subnets for controller <juju:Fix Committed by jameinel> <https://launchpad.net/bugs/1659640>01:49
rick_hBudgie^Smore: yea, but that was landed recently. It's in the alpha out now02:09
Budgie^Smorerick yeah but hasn't hit any ppa repo yet, right?02:10
rick_hBudgie^Smore: it's in the juju devel ppa02:10
rick_hBudgie^Smore: or use the snap :)02:10
rick_hBudgie^Smore: then you can keep things in apart nicely02:10
Budgie^Smorerick_h I found a page that said the devel was on 2.1.2!02:12
Budgie^Smorerick_h of course I have "lost it" again02:13
rick_hBudgie^Smore: yea, I think you're right. Maybe it's only in the snap now? /me checks around02:13
rick_hhttps://launchpad.net/~juju/+archive/ubuntu/daily has 2.2 alpha02:14
rick_hand here: https://launchpad.net/~juju/+archive/ubuntu/devel02:15
rick_hthat has 2.202:15
Budgie^Smoreyes I see that but I didn't look at the repo... not sure if I want to run an alpha build though and I had another "workaround" that worked too :) remove a subnet temporarily and use --to zone=us-west-1a02:16
rick_hBudgie^Smore: and http://paste.ubuntu.com/24232393/02:16
rick_hBudgie^Smore: yea, that's why I like the snap way. You get a new /snap/bin/juju to work with02:16
rick_hBudgie^Smore: but yea, it's always the way. The feature you need/want is in the NEXT release :)02:16
Budgie^Smoreyeah I should probably use snap more02:16
Budgie^Smoreoh hang on a min... edge is at 2.2 beta and beta is at 2.2 alpha?02:17
Budgie^Smorelol ok that is mildly amusing02:17
rick_hBudgie^Smore: so edge is a daily build that tracks "the next release"02:17
rick_hBudgie^Smore: so the non-released, but next target (e.g. what the source code says it is currently if you build from source...) is beta02:18
rick_hlate night for me02:18
* rick_h goes to take out the trash and calm down before bed02:18
Budgie^Smoreoh it is going to be a late one for me02:18
Budgie^Smorejust seems my day to hit weird and wonderful roadblocks02:48
=== mup_ is now known as mup
=== mup_ is now known as mup
=== mup_ is now known as mup
=== mwhudson_ is now known as mwhudson
kjackalGood morning Juju world!08:00
=== frankban|afk is now known as frankban
ybaumymoin08:55
=== sparkieg` is now known as sparkiegeek
ybaumyi forgot to set VIP for mysql or hacluster09:19
ybaumyhow do i do that after deployment09:20
ybaumyah i got it09:24
ybaumyjuju config mysql vip=IP09:24
ybaumyjuju resolved mysql/0 .. 1 .. 209:24
cnfmorning09:54
cnflets try this again09:54
BlackDexhello there. I'm using juju 2.1.2 and spaces. I Have a private and a public space. But if i want to deploy mediawiki for instance, i'm not able to let it use the two spaces. It will only bind to one.10:01
BlackDexIs there a setting i'm missing?10:01
BlackDexthx for jamespage on the other channel. using --constrains "spaces=public,private" does the job10:12
SimonKLBdoes anyone know if it's possible to run the collect-metrics script in the layer-basic venv?11:43
SimonKLBideally run it in the reactive framework somehow11:44
kjackalSimonKLB: you mean it like a feature request?12:08
SimonKLBkjackal: more like if it was already possible12:21
tvansteenburghSimonKLB: i thought collect-metrics is a hook12:25
tvansteenburghhmm, maybe that was just unfortunate naming in the docs12:27
kjackalSimonKLB: there is a separete layer https://github.com/CanonicalLtd/layer-metrics/blob/master/hooks/collect-metrics and here is an example on how to use it https://github.com/juju-solutions/bigtop/tree/master/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager12:28
SimonKLBkjackal: i know how metrics work normally12:29
SimonKLBi was wondering if you could run it in the reactive framework somehow, so that you could access the virtualenv from layer basic easily12:29
SimonKLBtvansteenburgh: i might be mistaken, but i think the metrics works a bit different12:30
mattywcollect-metrics is just a hook. so you could use it with just layer basic by doing @hook.Hook("collect-metrics")12:30
SimonKLBmattyw: ah that's neat!12:30
SimonKLBso now i just need to get charmhelpers somehow :D12:31
mattyw^^ that decorator is from memory so it might not be totally copy/paste accurate12:31
mattywSimonKLB, I think charmhelpers comes with layers right?12:31
SimonKLBcurrently, im accessing charmhelpers from the virtualenv created by layer-basic12:31
SimonKLBmattyw: yea12:31
mattywSimonKLB, depending on what you're doing there's nothing to stop you just having an executable bash script called collect-metrics in the hook directory12:32
mattywSimonKLB, the main difference between the collect-metrics hook and other hooks is that errors in the collect-metrics hook won't put the charm in an error state12:33
SimonKLBmattyw: so normally when you use the reactive framework it creates the hooks for you so that they all execute the reactive main function12:35
SimonKLBim not sure what the best practice is here when using metrics12:36
SimonKLBsince that has its own hook which then executes commands depending on which metric it is collecting12:36
mattywSimonKLB, ah yes I see, I'm sure there was a recent version that included it, if not there should be12:37
mattywSimonKLB, is there any reason you don't want to use the metrics layer?12:37
SimonKLBmattyw: i'd like to access the relations if possible so that i can fetch data from a database-charm when generating the metrics12:45
SimonKLBthat would be very easy if i could run it in the reactive loop12:45
rick_hcnf: morning, how goes things out east?12:46
kjackalSimonKLB: for what is worth, if you want to activate the venv of the basic layer you can do so like this:  https://github.com/juju-solutions/layer-cwr/blob/master/actions/cwr-charm-pr#L712:48
SimonKLBkjackal: yea i know, but i have to go through some unecessary hoops to import layer basic if im going to do it myself instead of letting reactive do it for me :)12:49
mattywSimonKLB, is the information you want stored in charmhelpers.core.unitdata?12:49
mattywSimonKLB, or where you hoping to be able to call relation-get?12:50
cnfrick_h: nice weather! :P12:51
cnfrick_h: good morning :P12:51
cnfjust got a fresh red bull12:51
SimonKLBmattyw: relation-get hopefully :) i want the ip of the relational charm so that i can access the database-api12:51
mattywSimonKLB, unfortunately relation-get insn't available in the collect-metrics hook - that's the other way they're slightly different to normal hooks12:52
mattywSimonKLB, but in your reactive framework when those values are set from the relation the usual approach is to set them in the unitdata key value store. and then make use of them in the collect-metrics hook12:53
SimonKLBmattyw: the problem is that the data is static, im basically keeping track of how many units are running of a perticualr application12:57
SimonKLBmattyw: i could probably use libjuju for this though if it's not possible to get the relation data12:57
mattywSimonKLB, so you want to be able to access the database you're related to?13:02
mattywSimonKLB, but you what you really want to know is the number of units that are running?13:03
SimonKLBmattyw: yea but it could probably be solved in other ways also13:03
SimonKLBmattyw: well yea, or specifically the machinespecs the units are running on13:03
mattywSimonKLB, and I guess just counting cpus/ram in the collect-metric hook is not an acceptable solution?13:05
SimonKLBmattyw: well the problem is that im not counting the cpu/ram on the machine the collect metric is running on, i want to get the cpu/ram on other machines that is running a different charm13:06
SimonKLB(this is for the charmscaler btw)13:06
mattywok - I see13:06
rick_hcnf: send some of that weather over here. Still below freezing each morning here.13:10
cnfouch :P13:11
cnfwe are a country of cold and wet, so we like our sunshine :P13:11
tvansteenburghrick_h: is it possible to remove a charm from the charm store (in a personal namespace)?13:11
cnfrick_h: so, are you available to help me some today?13:11
rick_htvansteenburgh: yes, the charm delete api has been wired up but it's not yet through the CLI to end users. So it's possible but still requires some manual api foo atm13:11
magicaltroutmake it so, my namespace is full of $hit I just hide from users! :)13:12
rick_hcnf: yes, I wanted to kind of reset and ask if you've got notes/doc on what your setup looks like. How many machines, special things about them, what the spaces layout looks like, the bundle you're working to use.13:12
tvansteenburghrick_h: okay, so it's coming soon to the cli?13:12
cnfsure13:12
rick_htvansteenburgh: it's not on the "on fire" list so it's coming but I don't know it's going to be out in the next couple of weeks13:12
cnfrick_h: you want to do that here, or in pm, or somewhere else?13:12
tvansteenburghrick_h: ack13:13
rick_hcnf: we can do here. Folks can ignore us or if you'd prefer PM feel free.13:13
cnfok, so I have a VM running the MAAS controller13:13
cnfand a kvm VM on the maas controller running the juju controller13:14
cnfi have 4 physical HP machines if various layouts, at least 16 cores and 24G in the smallest one13:14
cnfi set up a copper network for PXE boot and maas networking, because i could not figure out howto pxe boot of a LACP link13:14
cnfand all machines have a 10G LACP with vlans for various networking13:15
cnfwell, all but the smallest, that has just a 1G trunk for now13:15
rick_hk13:15
cnfi'm using a slightly modified bundle yaml i got from jamespage13:16
cnfhttp://termbin.com/pjqj13:16
cnfpast few weeks, i have run in a fair amount of bugs and problems with both maas and juju13:17
cnfright now, my state is http://termbin.com/2yke13:17
rick_hcnf: k, so the machines look like they've got two devices? or are there more with the bonds?13:17
cnfuhm, they have a ton of devices :P but active it's 1 for PXE / MAAS, and one bond with 2 or 3 vlan tags13:18
cnfconfigured in maas13:18
rick_hcnf: k13:18
cnfhttps://www.dropbox.com/s/tupaiwe049g8rzk/Screenshot%202017-03-23%2014.18.32.png?dl=0 for example13:18
rick_hcool yea I see machine 0 with 4 addresses in the machine status output13:19
cnfso what stands out to me, atm (i'm hammering the nails as i see them...)13:21
cnfmessage: 'can''t get info for image ''juju/xenial/amd64'': not found'13:21
cnfrick_h: i should add that only the public network on machine 0 can access the internet directly13:21
cnfand i have http-proxy and https-proxy set13:21
cnfwhich needs to go over the maas network13:22
rick_hcnf: ok, yea I see all the lxd on machine 0 is in pending and that is lxd reaching out for the container images13:22
cnfright, so i think it's not respecting proxy settings?13:22
rick_hcnf: yea, I'm looking. So lxd supports these per https://github.com/lxc/lxd/issues/214713:23
rick_hcnf: and https://bugs.launchpad.net/juju/+bug/1594720 says the fix is release ...13:24
mupBug #1594720: lxd containers not using configured proxy for downloading images <2.0> <addressability> <lxd> <network> <proxy> <sts> <juju:Fix Released by thumper> <https://launchpad.net/bugs/1594720>13:24
cnfhmm13:24
rick_hcnf: so looking at the LP bug there's some commands to run that get the LXC config to see if the proxy is set and we can then check if it's set correctly or not.13:25
rick_hcnf: https://bugs.launchpad.net/juju/+bug/1594720/comments/1513:25
mupBug #1594720: lxd containers not using configured proxy for downloading images <2.0> <addressability> <lxd> <network> <proxy> <sts> <juju:Fix Released by thumper> <https://launchpad.net/bugs/1594720>13:25
cnflooking for the lxc command to show that13:27
cnfrick_h: $ sudo lxc config get core.proxy_http13:29
cnfhttp://172.20.20.1:800013:29
rick_hcnf: and that's the correct value for the proxy?13:29
cnfyep13:29
rick_hcnf: what about https?13:30
cnfand it is reachable from the host13:30
cnfset as well13:30
rick_hdoes the proxy also run for https?13:30
cnfyep13:30
rick_hcnf: ok, so then we should try to find in the lxc/lxd logs to see where the actual urls being hit for the images is and see if that lines up correctly or if it's going somewhere crazy13:30
cnf$ curl https://cloud-images.ubuntu.com/releases returns a 30113:31
cnfhttps://bpaste.net/show/a39688f501a913:32
rick_hcnf: do you have the actual error message about the image? I'm not sure on the 404 lines in that log.13:33
cnfrick_h: it literally says lvl=info msg="Sending top level 404" t=2017-03-22T18:11:16+0000 url=/1.0/images/juju/xenial/amd6413:33
rick_hcnf: that's the lxd images right?13:34
rick_hcnf: sorry the lxd logs13:34
* rick_h gets more coffee in him13:34
cnfthat's from /var/log/lxd/lxd.log13:34
cnf:P13:34
cnfi'm on my 2nd red bull :P13:34
rick_hcnf: but in the juju end was there a log that stated anything ?13:34
rick_hcnf: like in the machine-0 log or something?13:34
cnfyes, in http://termbin.com/2yke13:34
cnfcontainers:13:34
cnf      2/lxd/0:13:34
cnf       machine-status:13:35
cnf          current: provisioning error13:35
cnf          message: 'can''t get info for image ''juju/xenial/amd64'': not found'13:35
rick_hright, thanks. Sorry I ended up searching for "xenial" but on the bundle pastebin13:36
cnf:P13:36
cnfi don't know where the path /1.0/images/juju/xenial/amd64 should exist13:37
rick_hcnf: yea, I'm just looking to see if anyone from lxd is around that can shed more light on that.13:38
cnfright13:39
rick_hcnf: I've not used lxd in a proxy setup and so I'm not sure if that's normal "Oh, I don't have an image so I'll go get one" or something failed and that's "404 I don't have and could not get that image"13:39
cnfuhu13:39
cnfi have not used lxd at all13:39
cnfwell, not directly13:39
jamrick_h: cnf: IIRC ivoks had set up something with offline support for cloud-images and found the images were in the wrong directory13:43
cnfhmm13:43
jam(he had done something like an rsync of the cloud-images directory, but turns out on the host machine they are in the wrong path and an apache rule puts them in the right one, IIRC)13:44
cnfuhm13:44
cnfyou lost me, there13:44
cnfapache rule?13:45
cnfapache where?13:45
jamcnf: so, the URL we care about is likely to be http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson13:45
jamcnf: on that machine you should be able to run13:45
jamlxc image list ubunt:13:45
jamsorry13:45
jamlxc image list ubuntu:13:45
jamwhich reads the above URL and http://cloud-images.ubuntu.com/releases/streams/v1/downloads.sjson13:46
jamto return a big list of officially released images13:46
cnfthat's empty13:46
cnfhttps://bpaste.net/show/be8d401c7f1c13:46
jamsorry http://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:download.json13:46
jamcnf: I think you need a :13:46
jam"lxc image list ubunt:"13:46
jamubuntu:13:46
jamwow, that last u doesn't like me13:47
cnferror: Get https://cloud-images.ubuntu.com/releases/streams/v1/index.json: Unable to connect to: cloud-images.ubuntu.com:44313:47
jamotherwise you're searching for a local image named 'ubuntu' vs listing the images in the 'ubuntu:' remote13:47
jamcnf: if you have http-proxy set, do you have https-proxy set?13:47
jamcause LXC only uses https-proxy it doesn't fall back to http-proxy for 443 traffic13:47
cnfjam: yes13:47
cnfcurl https://cloud-images.ubuntu.com works fine13:47
jamand:  sudo lxc config get core.proxy_https13:48
cnf$ sudo lxc config get core.proxy_https13:48
cnfhttp://172.20.20.1:800013:48
jamcnf: if "lxc image list ubuntu:" isn't listing any images13:48
jamthat's why Juju can't find them13:48
cnfjam: right... but why? i can curl it just fine with those proxy settings?13:49
jamcnf: echo $https_proxy and echo $http_proxy ?13:49
cnfshow the same values13:49
cnf$ echo $https_proxy13:50
cnfhttp://172.20.20.1:800013:50
cnfit's all set by juju13:50
jamcnf: well, that doesn't mean we can't get something wrong. The only other quick thing I can think of is to try bouncing the lxd agent13:50
jamsudo service restart lxd13:51
jamor is it "sudo service lxd restart"13:51
jamI usually get that wrong the first time13:51
jamanyway, have to go pick up my wife13:51
cnfi went for sudo service lxd restart :P13:51
jamyou're at a point where I'd have to drop into #lxd and ask stgraber13:51
cnfhmz13:51
rick_hjam: cnf yea, where I was going to head next for the lxd specifics on that.13:52
cnfjam: #lxd is pretty much empty13:52
cnfis it #lxc?13:52
cnfit's #lxcontainers13:54
rick_h#lxcontainers13:54
rick_hyea13:54
cnfi wonder what timezone they are on...13:55
rick_ha mix, I know the lead is in my timezone ish.13:56
cnfright13:56
cnfi need a vacation :/13:59
cnfrick_h: why do i always run into the weird bugs? :P14:15
rick_hcnf: heh, because you're doing the fun complicated real world install14:17
cnfyeah :P14:17
cnfthe "fun" part is wearing off though14:17
cnf3 weeks, and still nothing running is getting frustrating14:18
rick_hcnf: it's hard to setup a lab for this to put into automated testing, but it's this stuff that has to work to go out there into production. Sorry you're hitting it. Looking at bugs/etc thought this should work ok14:18
cnfuhu14:18
magicaltroutreminds me of when SQL had about 100 different dialects that automatic SQL generators had to support. All seemed fine until you tried something tricky! ;)14:21
magicaltroutlots of moving parts in LXD, Juju and MAAS14:21
cnfuhu14:22
cnfwhen i'm working from home14:22
cnfi have zelda to comfort me in the waiting periods :P14:23
cnfhere, no such luck14:23
magicaltroutsounds a bit lewd! ;)14:29
cory_fujamespage: Any objection to me re-promulgating ganglia and ganglia-node under a focused ganglia-team (that I'm going to create) which is owned by openstack-charmers?14:30
jamespagecory_fu: works for me14:31
jamespagecory_fu: might have one already one sec14:31
cory_fuCool.  Also, of note, I'm looking at merging the ganglia-node review14:31
jamespagecory_fu: https://code.launchpad.net/~ganglia-charmers14:31
cory_fujamespage: Ok, cool.  I'll get that sorted out, then14:32
jamcnf: are you able to curl the full url http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson14:32
jamI'm wondering if your proxy is doing caching, and maybe has cached something poorly14:32
cnfjam: yes14:32
cnfjam: curl works just fine14:33
rick_hjam: so looking at #lxdcontainers it works from lxd, but something in how juju is setting things up causes the proxy config to not work.14:39
jamrick_h: you stopped the load test, right? I tried to see the progress but just got a 40414:42
rick_hjam: yes, had a machine issue here that I needed to kill it for unfortunately14:42
rick_hI should setup to be able to start/run it from the maas machine vs my desktop14:42
jamrick_h: np, I feel we showed what we wanted to14:42
cory_fujamespage: Did you intend for the gate-xenial in ganglia and ganglia-node to not be executable and thus be skipped?14:42
rick_hjam: rgr, got over 400 models so yay14:43
jamespagecory_fu: yes - we need to push up the current proposed ones first14:43
jamespagecory_fu: then we can enable the xenial gate tests14:43
cnfhmm14:43
cory_fujamespage: Ok, so only push them to trusty?14:43
cnfrick_h: any more ideas? :P14:43
jamespagecory_fu: series in metadata - its a circular depends issues14:43
cory_fujamespage: Ah, gotcha14:43
jamespageganglia <-> ganglia-node14:43
jamespagecory_fu: I'll push another rev to edge once we have this set done14:44
rick_hcnf: no :( we're beyond my scope. I know jam is EOD but he's the next best person that might be able to help or get someone to help. Honestly, the answer is probably going to be to get as much of this material as possible with the lxd "it works from here" added to the notes into a bug for the engineering team in juju to get at.14:44
cory_fujamespage: Ok14:44
cnfhmm14:44
rick_hcnf: I'm sorry, I was hoping we'd be able to walk through a hole in the setup, but it's setup right. From your end everything *should work*14:44
cnfit should, indeed14:44
cnfrick_h: i appreciate your time none the less14:45
rick_hproxy is set, lxd does work, but somehow juju isn't doing the right thing14:45
cnfit at least validates i didn't do something stupid14:45
cnf(it has been know to happen)14:45
rick_hcnf: completely, I appreciate you working through that with the lxdcontainers folks to make sure the bug isn't there. That's really helpful and one less layer to try to go through14:45
cnfok, lets hope jam becomes available soon14:46
rick_hcnf: so he's UTC+4 so that's why I think for today the best thing is to get the bug filed with all the notes from today14:47
rick_hcnf: and then we can ask nicely for him to find someone to peek at it in the morning.14:47
cnfhmz14:47
cnfthese timezones are killing me :(14:47
rick_hcnf: yea, I guess we've gotten used to it over time so it seems natural but it's definitely not helpful in this situation.14:48
cnfrick_h: what is grating on me, is that i have been doing this for several bugs for weeks now14:48
cnfand getting them fixed just takes so long14:48
cnfget 1 baby step, then wait for the next day14:48
cnfrick_h: and i don't mean to invalidate your effort14:49
cnfjust gets frustrating on my end :P14:49
rick_hcnf: I totally understand. /me will look away while you go kick something14:49
cnf:P14:49
magicaltrouton the flip side cnf if you ever want to job swap, you'll be straight into to a Juju Consultancy role! ;)14:52
cnfha!14:52
cnfmagicaltrout: i actually am a freelance IT consultant14:52
kwmonroetvansteenburgh: amulet issue when tearing down the class:  http://paste.ubuntu.com/24235292/14:53
magicaltroutif you're paid by the hour, surely broken Juju is actually a bonus? ;)14:53
cnfmagicaltrout: it's a long term project, so i get paid either way14:53
cnfmagicaltrout: but i would like something to WORK eventuakky14:53
magicaltroutthat is a valid requirement I guess14:54
magicaltroutmaybe pitch it as a long term goal ;)14:54
cnffor my sanity, if nothing else14:54
magicaltroutaye i know the feeling14:54
cory_fujamespage: What repo are the new ganglia and ganglia-node revisions coming from?  The "Submit a Bug" links point back to lp:charms/ganglia[-node] but those don't have any commits newer than 2014.14:55
magicaltrouttry talking to kwmonroe about big data14:55
magicaltroutit'll soo grind you down14:55
magicaltrout+n14:55
cnfmagicaltrout: i don't like waking up and thinging "ugh, not more of this, again"14:55
jamcnf: rick_h: so it seems that things like "lxc list" are driven by the client, not the lxd agent, which means 'jujud' has to find cloud-images itself. I seem to remember menn0 fixing a bug recently about Juju and http_proxy not getting set early enough in the lifetime of jujud14:56
kwmonroemagicaltrout: shouldn't you be spinning up drill bits?14:56
jamcnf: the other thing that you could try is setting things like http_proxy= in /etc/environment and then restarting agents14:57
magicaltroutsorry pink face14:57
jamespagecory_fu: if you're ok with the charms I can setup all of the metadata - but basically the bug reporting is on launchpad14:57
jamespageand the git repos are on github.com/ganglia-charms14:58
jamespagecory_fu: https://bugs.launchpad.net/charm-ganglia and https://bugs.launchpad.net/charm-ganglia-node14:58
jamespagecory_fu: and then I'll go delete the old branches in bzr14:59
cory_fujamespage: Ok.  I'm getting the "no module named yaml" from ganglia, so it needs the tests.yaml and requirements.txt changes that you did in ganglia-node14:59
cory_fuBut otherwise they seem good14:59
jamespagecory_fu: odd I thought I did that14:59
jamespagecoreycb: lemme check again14:59
cory_fujamespage: I'm over here. ;)14:59
cnfhttps://bugs.launchpad.net/juju/+bug/1675440 for those that want to track it14:59
mupBug #1675440: juju unable to get LXD images <juju:New> <https://launchpad.net/bugs/1675440>14:59
jamespagecory_fu: doh14:59
jamespageI wish irccloud would follow a converstion with nics15:00
jamespagecory_fu: -6 imported into reviews.j.c15:02
cory_fujamespage: Hrm.  Yeah, the review does seem to have those bits.  Wait, I think it's my fault15:02
jamespagecory_fu: no it was missing the requirements.txt15:02
cnfjam, rick_h how can I tell juju to retry an lxd?15:02
cnfresolved is for units, and retry-provisioning doesn't work on containers15:02
cory_fujamespage: Ah, ok15:03
cory_fuThanks15:03
cnfjam: i edited /etc/environment, and restarted lxd and jujud-machine-215:03
jamcnf: there is "juju retry-provisioning" but I don't think it supports containers15:06
cnfjam: it doesn't15:06
cnfhmm15:08
cnfrick_h: any suggestions?15:09
cnfbeside a reboot?15:09
cnfhp servers take forever to reboot :P15:09
rick_hcnf: no, just to manually do the commands vs using the bundle15:09
cnfrick_h: ok, what commands?15:09
rick_hcnf: e.g. deploy or add-unit with a placement directive into a container e.g. "juju add-unit XXXX --to 0:lxd"15:10
rick_hcnf: or just test juju deploy ubuntu --to 0:lxd15:10
cnfrick_h: so i have to remove them, and add them again?15:10
cnfERROR cannot add application "ubuntu": placement scope: invalid model UUID "0"15:11
jamcnf: lxd:015:12
jam'0' is the machine ID15:12
jamin your case it would be15:13
jamyou probably want "juju add-unit ubuntu --to lxd:2"15:13
jamgiven your earlier messages about being on machine 215:13
jamah sorry, your bug says machine 115:13
cnfallocating15:16
cnfoh, bah15:17
cnf message: 'unable to setup network: no obvious space for container "2/lxd/1",15:17
tvansteenburghkwmonroe: the amulet test failure is repeatable?15:19
tvansteenburghkwmonroe: i don't know why you'd get that behavior unless your ~/.local/share/juju/store-usso-token disappeared15:21
kwmonroetvansteenburgh: the same charm (spark-21) failed in all my clouds: http://bigtop.charm.qa/cwr_bundle_spark_processing/4/report.html.  i'll pull it into a simpler test case with the ubuntu charm to make sure.15:21
kwmonroetvansteenburgh: my question is why does "juju remove-application" need to talk to the store at all?15:21
cnfrick_h: i think the new one says " container started"15:22
cnfbut status still shows pending15:22
cnfnot sure is this is expected15:22
cnfit has an instance-id15:22
rick_hcnf: hmm, so what does juju show-machine show for that?15:22
rick_hcnf: sorry, I'm in a meeting so a little distracted atm15:22
cnfrick_h: machine-status: running, juju-status: pending15:23
cnfso it got farther than the other ones15:23
rick_hcnf: hmm, ok15:23
tvansteenburghkwmonroe: it has to auth to the controller15:23
cnfare we sure juju/xenial/amd64 is a valid image?15:23
cory_fujamespage: ganglia and ganglia-node are GTG.  So, you're going to push them to ~ganglia-charmers and re-promulgate?15:24
jamespagecory_fu: I can do that15:24
kwmonroeoooohhh.. yeah tvansteenburgh.  that makes sense.  let me double check my login status on the machine where it's failing.  this is probably just pilot error.15:24
cory_fujamespage: Ok, let me know if you need anything from me, then.15:24
cory_fujamespage: I'm happy to help with any parts of the process15:25
jamespagecory_fu: no worries - thankyou for the reviews :-)15:25
cnfrick_h: maybe it's a broken charm, or something15:25
cnfthough that would be weird, jamespage would have run into this, then15:25
hatchBudgie^Smore `juju add-model --config vpc-id=<id>`15:26
cory_fujamespage: Should I go ahead and close said reviews now, or wait for confirmation that they're re-promulgated?15:26
cnfhmz15:27
jamespagecory_fu: please go ahead - I'll wrap up my meeting and move things around as discussed15:28
cnfuhm15:29
cnfwth15:29
Budgie^Smorehatch I will give the add-model --config vpc-id a whirl, I broke the model again last night so bad at the controller level :)15:33
hatchoops! :)15:34
rick_hcnf: what's it doing now? It was pending but no reason given?15:34
cnfjuju-status is pending on the ubuntu one15:34
rick_hcnf: this is from just "juju deploy ubuntu"15:34
cnfmachine-status is deployed15:34
rick_hcnf: right, and if you show-machine 0 there's no more verbose output there?15:34
cnfno15:35
rick_hcnf: can you ssh to the new machine then? Does it have an IP addr and such?15:35
cnfjuju-status:15:35
cnf          current: pending15:35
cnf          since: 23 Mar 2017 16:19:26+01:0015:35
cnf        instance-id: juju-857117-2-lxd-215:35
cnfrick_h: uhm, not directly15:35
cnfit didn't get an ip in the maas/juju network15:35
ThiagoCMCHey guys, I'm very familiar with OpenStack on Ubuntu (via apt-get and manual setup), however, from now on, I want to start using Juju but, I never used it before...15:36
ThiagoCMCIs there any way to install OpenStack via Juju, without MaaS, just everything in one box?15:37
Budgie^Smoreit probably wasn't truly broken but my Kubernetes master wouldn't start waiting for pods to start and I couldn't be too bother troubleshooting that on a clean model15:37
ThiagoCMCI'm seeing that conjure-up can help with that but, what about going straigh with Juju instead?15:37
rick_hcnf: is there anything useful in the debug-log or the machine-0 log on the host machine?15:37
cnfbut i can ssh to it15:38
admcleodThiagoCMC: you could try https://docs.openstack.org/developer/charm-guide/openstack-on-lxd.html15:38
jamespagecory_fu: https://jujucharms.com/ganglia/ and https://jujucharms.com/ganglia-node all done15:38
ThiagoCMCadmcleod, thank you!15:38
jamespageThiagoCMC: alternatively try with conjure-up15:38
jamespageThiagoCMC: http://conjure-up.io/15:38
ThiagoCMCjamespage, you think that Conjure-up Next PPA is a better starting point?15:38
cnfnothing interesting in the machine-log15:38
jamespageThiagoCMC: use the snap as detailed in that link - I think thats the right route these days stokachu ?15:39
ThiagoCMCOh, I see...15:39
cnfrick_h: /var/log/juju is empty inside the container15:39
rick_hcnf: right, but on the host machine it should have any notes.15:41
cnfnothing of interest15:41
ThiagoCMCDoes Juju OpenStack supports: OpenStack Ocata with Senlin, AODH and new Ceilometer with Gnocchi?15:42
cnfrick_h: i guess 2017-03-23 15:19:39 INFO juju.tools.lxdclient client_image.go:170 found image from https://cloud-images.ubuntu.com/releases for juju/xenial/amd64 = 2cab90c0c342346ea154bc2e8cacdae752a70747a755ce1f2970c9a9ebb5fe8c15:42
cnfso it should bloody well work15:43
magicaltroutThiagoCMC: you need to prepend your questions with jamespage :15:44
magicaltrout:)15:44
ThiagoCMC^_^15:45
Budgie^Smoreoh I remember why I destroyed the controller / model hatch, I hit a bug where juju wouldn't ssh proxy to private systems either by setting proxy-ssh config option or using --proxy flag but there had already a bug filed and triaged for it15:45
jamespageETOMANYCHANNELS15:45
* magicaltrout leaves norwich before he's hunted down15:45
jamespageThiagoCMC: Ocata: yes, Aodh: yes, Senlin: no, Gnocchi: no15:45
hatchBudgie^Smore ahhh ok thanks for looking at the bugs though - appreciate any new issues too :)15:45
ThiagoCMC jamespage, Hmm... That's sad... But okay... Thanks!15:46
Budgie^Smorehatch I didn't delibrately hunt for bugs but they crop up when I am searching for what I am doing wrong ;-)15:46
hatchhaha, hey, whatever works :)15:47
jamespageThiagoCMC: I'm hoping to put Gnocchi on the roadmap for the openstack charms but it won't be in the short term15:47
jamespageThiagoCMC: if someone would like to work on a senlin charm I'm all ears :-) but don't have time with the existing pool of resources working across the charm set (33 of them)15:47
cnfrick_h: so uhm15:48
cnfi removed those units15:48
cnfand added them agin15:48
jamespagemagicaltrout: we should do a beer soon15:48
cnfnow it seems the containers came up15:48
cnfi didn't actually _change_ anything!15:48
magicaltroutindeed now the grayness has left15:48
rick_hcnf: ? so what changed? Did jam have you do something?15:48
cnfno!15:48
cnfnothing was changed15:48
cnf>,<15:48
ThiagoCMCjamespage, that's okay! I have those things working now but, I'm deploying everything using my own Ansible playbooks, workd great! I have to start working with Juju to help you guys!   :-D15:48
cnfshit still isn't working, but the containers at least came up15:49
cnfwth15:50
cnfrick_h: so all my containers are running now15:50
cnfbut none of them have juju come up15:50
cnfwth is going on :(15:52
cnfk, i'm going to destroy my model, and deploy it again15:53
cnfand go for a 20 minute stroll15:53
admcleodcnf: i wonder if it as something to do with one dns resolution attempt working, and one failing - maybe you have a broken dns server somewhere in a pool?15:55
cnfadmcleod: it seems to work for everything else15:56
admcleodcnf: perhaps everything else handles such a failure gracefully? just guessing. ive just tried to reproduce with local lxd provider and proxy and i cant16:00
Budgie^Smoreso my first weird question of the day, how is jaas going to be handle machines that are behind a nat router? (btw hatch that problem with my model not showing up in the UI has something to do with the initial load (after login) of the page)16:09
rick_hBudgie^Smore: so that's why it only works in public cloud atm16:09
Budgie^Smorerick_h but even in a public cloud I can have instances behind a nat router :P16:10
Budgie^Smore(tbh it is my understanding of how a VPC in AWS actually works16:10
rick_hBudgie^Smore: right, but the agents do most of the work to call out to the controller and the two exceptions I can think is juju ssh and juju actions so will have to see how they behave behind a vpc16:11
Budgie^Smorerick_h I am "testing" right now, so we will see, as it stands I don't think ssh (and by extension exec) will work. probably need a proxy system in the environment and juju having knowledge of it16:13
Budgie^Smorerick_h I am thinking something like a poor man's VPN connection using sshuttle16:14
rick_hBudgie^Smore: hmm, yea, will have to add this to the "test and figure out wtf" list16:14
bdxBudgie^Smore: yo16:15
Budgie^Smoregive me an hour or so and I will have an initial test done16:15
Budgie^Smorebdx: well howdy16:15
bdxBudgie^Smore: hey, just reading some of the backlog here16:16
bdxBudgie^Smore: I have plenty of envs that are running in private address spaces (behind a nat) using JAAS .... I've yet to find anything that doesn't work as I would expect ... minus ssh16:18
bdxBudgie^Smore: sshtunnel and (depending on if you have a publicly available endpoing) openvpn are great remedies for access in that use case16:21
Budgie^Smorebdx that is what I expect too ssh is always proablematic in those envs... I have used sshuttle in the past too, as long as I have 1 ssh accessible system in the env that is16:22
bdxBudgie^Smore: are you using AWS provider?16:23
Budgie^Smorebdx in this case, yes16:23
bdxBudgie^Smore: have you configured nat, and igw routing tables, and configured your subnets to use the routing tables that point to the respective nat or igw?16:26
Budgie^Smorebdx: yup 2 x public subnets, 2 x private subnets, 1 x igw and 1 x nat within a vpc. actually found a cloundformation yaml for it too16:28
bdxBudgie^Smore: niceee, sounds like you are on the right track ... did you pipe that through to juju then?16:28
Budgie^Smorebdx yeah I am actually deploying charms right now too it (using the CDK as a guide but customizing to take into account the environment)16:29
bdxBudgie&Smore: nice16:30
cnfrick_h: so the containers started now16:34
bdxBudgie^Smore: I have configured most of my apps have both subnets configured as juju spaces such that only put internet facing proxies land on the igw subnets, and everything else behind nat gateway subnets (the way it should be)16:34
bdxhttp://paste.ubuntu.com/24235685/16:34
cnfrick_h: model isn't working, but for some reason the containers started16:34
Budgie^Smorebdx pretty much doing the same :)16:34
cnfrick_h, jamespage http://termbin.com/ae8116:35
cnfsuggestions?16:35
jamrick_h: I'm really past EOD. my gut says we're failing to give an error when requesting a container that doesn't have a defined space on a machine with multiple network interfaces16:35
Budgie^Smoreok "1 unit, xenial, 8x26.88GHz, 32.00GB, 8.00GB" seems wrong... I don't know of any CPU that has 26GHz cloak!!!16:36
jamand then starting the container but not requesting an IP because we don't know where to put it16:36
jamcnf: do you have bindings specified for your applications?16:36
jamI'm out again16:36
ybaumythe export button does not work for me in juju gui16:36
bdxBudgie^Smore: sweet! yeah, so I just stash my openvpn on a igw subnet in my 'common-infrastructure' model in the same vpc - this is how I'm accessing my instances on the nat subnets16:37
cnfjam: yes, i used jamespage bundle16:37
cnfok :(16:37
Budgie^Smorebdx yeah I will do that if I have to, I will be honest and say I prefer site-to-site when it comes to VPN16:37
cnfis http://termbin.com/023aq not right?16:37
bdxBudgie^Smore: not sure what your mileage is with pfsense .... you can create a custom ami16:39
cnfjamespage: can i get you to look at this?16:39
cnfrick_h: you still here?16:40
Budgie^Smorebdx for now I am just going to use one of those proxies and use sshuttle to tunnel the IP range of the VPC16:40
rick_hjam sorry thought you had gone for EOD.16:40
bdxBudgie^Smore: can you link me to your cloudformation yaml for that16:40
rick_hcnf: sorry, in and out with the schedule ATM.16:40
cnfok :/16:40
Budgie^Smorebdx https://github.com/madeden/blogposts/blob/master/k8s-existing-env/src/cloudformation/NetworkLayout.json16:41
Budgie^Smoreactually it is JSON but same diff16:41
bdxBudgie^Smore: sweet, thanks!16:41
cnfi'm close to throwing in the towel, btw jamespage, rick_h16:41
cnfhmz, how do i look at an application?16:46
cnfor a unit16:46
Budgie^Smoreok so almost complete with this model's initial deploy :)16:49
admcleodcnf: what do you mean, 'look at'?16:49
cnfadmcleod: inspect? show what it wants?16:49
cnfstatus says  Incomplete relation: monitor16:49
cnfwhat does it expect? it not not defined? is it not working?16:50
admcleodcnf: wlel you can 'juju ssh unit/0' - oh, well in that case its waiting for one of the things its related to (defined in that bundle) to provide 'monitor'16:51
cnfadmcleod: everything is waiting on evetything atm it seems16:51
cnfi just don't know why16:51
cnfor what the root problem is16:51
admcleodcnf: ceph-ods is waiting for ceph-mon16:51
admcleodcnf: it looks like all the containers are still 'pending'?16:51
cnfyes, great, and why isn't ceph-mon not coming up?16:51
admcleods/ods/osd16:52
cnfyes, and they'll stay pending forever16:52
cnfbut i don't know why16:53
jamespagecnf: sorry been otp16:54
* jamespage reads backscroll16:54
jamespagecnf: ok so looks like you have16:55
jamespagea number of LXD machines that are marked as running but don't yet have a juju status16:55
jamespagehttps://www.irccloud.com/pastebin/gDuay0BK/16:56
cnfyes, all of them16:56
jamespagecnf: which is the cause of16:57
jamespagehttps://www.irccloud.com/pastebin/oagEPOlJ/16:57
cnfyeah16:58
cnfjamespage: so i think? the containers expect to be in the maas / juju network?16:58
jamespagejam, rick_h: this looks like issues between the jujud machine agents on the LXD containers and the controller machine16:58
cnfmaybe?16:58
jamespagecnf: yeah pondering that16:58
cnfbut if so, why the fuck isn't juju doing this?16:58
cnfand please do pardon the language16:58
cnfi'm quite frustrated at this point16:58
* jamespage puts his hands on his ears16:58
jamespagecnf: I don't know but lets see if we can confirm this16:59
jamespagecnf: please can you "juju ssh 0"17:00
jamespagecnf:  and then sudo to root17:00
jamespagecnf: do a 'lxc list'17:00
cnf(as a side note, and something i'll address later, juju is assigning about 10 public addresses for this o,O)17:00
jamespageand then do a 'lxc exec <one of the container> bash'17:00
jamespagethen we can peek into /var/log/juju17:00
jamespageor even /var/log/cloud-init.log17:00
cnfjamespage: /var/log/juju is empty17:01
cnfin the container17:01
jamespagecnf: ok so the issue is earlier in lifecycle17:01
jamespagecnf: how about /var/log/cloud-init*.log17:01
cnfbunch of tracebacks17:02
cnftries to install stuff, but can't17:02
jamespagecnf: can you pastebin them please17:02
cnfmakes sense, it has NO connectivity17:02
jamespagecnf: as in there are no network interfaces for the container17:02
cnfthere is 1, but that can reach neither internet nor proxy17:03
jamespagecnf: or the juju controller I suspect17:03
jamespagecnf: that's my hunch at least17:04
cnfso i need to manually tell juju to add the maas/juju interface to each container?17:04
jamespagecnf: you should not have todo that17:04
cnfright17:04
jamespagecnf: 'ts why I pinged rich_h and jam17:04
jamespagecnf: which network space is the controller listening on?17:04
cnfspace-maas17:05
cnfpasted screenshot of my maas networking in pm17:05
jamespagecnf: gotcah17:05
Budgie^SmoreOK lazyPower I now I have my "new" k8s cluster up :)17:06
cnfeverything in juju and maas has pushed me, quite aggressively, to put juju/maas mgmt on a separate fabric17:06
lazyPowerBudgie^Smore: :tada:17:06
jamespagecnf: I'll ask you to trying something using the juju cli, and then we'll see if we can work that into the bundle17:06
Budgie^SmorelazyPower right!17:06
jamespagecnf: juju add-machine --to lxd:0 --constraints="spaces=space-maas"17:06
Budgie^SmorelazyPower did we find out what happend to juju-crashdump?17:07
cnfflag provided but not defined: --to17:07
jamespagehuh17:07
jamespagelemme check that17:07
jamespagejuju add-machine lxd:0 --constraints="spaces=space-maas"17:08
jamespagecnf:  ^^17:08
cnfcreated container 0/lxd/917:08
jamespagecnf: ok but the important bit is that the juju status for the machine goes active17:09
jamespagenot pending17:09
jamespagecnf: if this works, we'll need to add a "constraints: spaces=space-maas" to each of the services that needs to run in a LXD17:10
jamespagecnf: 100% agree that juju should just dtrt here and its not afaict17:10
rick_hjamespage: cnf so the containers need some way to phone home to the controller. If it can't reach it via some network then yes you'll need the management network available to the containers.17:11
rick_hjamespage: cnf we'd have to check with jam on what the containers do if they don't have a direct device on the juju controller network but it's supposed to be routable in some way to the controller through the network the container is on17:12
kwmonroetvansteenburgh: it was totally a missing sso-token on my part.  sorry for the noise, tearDownClass is doing just fine :)17:22
ThiagoCMChey guys, how to use "conjure-up openstack" to deploy Ocata? I'm seeing that Newton is the default...17:22
jamespageThiagoCMC: ah its possible that the bundles for conjure-up have not been updated just yet - it lags the main charm/openstack release cycle a little17:22
jamespagestokachu: ^^ care to comment?17:22
ThiagoCMCOh, okay...17:22
cnfrick_h: i can understand that17:23
kwmonroecory_fu:  your layer-basic bits didn't break lint.  my cwr unit was dirty.  sorry for cursing at you so much this morning.  may, i can't seem to blame people as good as i used to.17:23
cnfrick_h: but i'd expect juju to know this :P17:24
jamjamespage: cnf: Juju doesn't know what your route tables look like. if you ask it to deploy a container that is only on 172.20.20.0/24 then that is what it does17:24
jamcnf: if you want it somewhere else, ask it to put it there17:24
rick_hcnf: yes, that get's back to jam's comment earlier about that.17:25
jamjuju deploy app --bind foo --constraints spaces=other17:25
jamcnf: it is entirely plausible that you have a route from 172.20.30.0/24 to 172.20.20.1 or wherever you are hosting Juju Controller17:25
rick_h<jam> John Arbash Meinel rick_h: I'm really past EOD. my gut says we're failing to give an error when requesting a container that doesn't have a defined space on a machine with multiple network interfaces17:25
jamthough I would expect the containers to get *an* IP, if they aren't getting anything at all, that is surprising17:26
cnfthey get an ip17:27
cnfjust not one that works to get to the juju controller17:27
jamcnf: so if one of the subnets is one that containers *have* to be on (to route to internet, etc)17:27
jamthen add it17:27
jamjuju deploy foo --bind bar17:27
jamtells us to put a container with access *only* to bar17:27
stokachuThiagoCMC: i think the description says newton and that needs to be updated17:27
cnfhmm, so i have to manually bind _every_ container to the juju space17:27
cnfo,O17:27
stokachuThiagoCMC: it should be whatever is in th elatest charm store17:27
jamcnf: yes17:27
ThiagoCMCstokachu, that's nice!17:27
ThiagoCMCI'm about to try it now17:28
jamcnf: cause if you're doing something like PIC it may be that you have a firewall between everytihng and the juju controllers17:28
cnfhmm17:28
jamcnf: and don't want your Apache software17:28
stokachuThiagoCMC: ill update the descriptions now but they wont show up until i do another conjure-up release17:28
jamrunning on the same subnet as your Juju controller17:28
jamas your Postgres DB17:28
hatchBudgie^Smore did the vpc config option work for you?17:28
Budgie^Smorehatch yes it did :)17:28
ThiagoCMCstokachu, BTW, if the package "software-properties-common" is not installed, "sudo snap install conjure-up --classic" fails!17:29
Budgie^Smorehatch cluster is setup, just solving how I am going to ssh to the instances right now17:29
cnfjam: i don't want apache on the same subnet as juju, obviously :P17:29
stokachuThiagoCMC: really?17:29
ThiagoCMCyep17:29
stokachuThiagoCMC: AH!17:29
jamcnf: then create a router that can route between them17:29
hatchBudgie^Smore excellent, I've triaged adding that feature into the GUI release after the upcoming one.17:29
stokachuThiagoCMC: nice find!!17:29
cnfuhm17:29
stokachuill fix that now too17:29
cnfhuh?17:29
cnfjam: you lost me17:29
Budgie^Smorehatch awesome :)17:29
ThiagoCMC=)17:30
jamcnf: the issue is that you ask us to only put the container on 172.x.y and we did, but you don't have a route in your network from 172.x.y to the Controller17:30
Budgie^Smorefor some reason sshuttle isn't behaving today :-/17:30
stokachuThiagoCMC: it's for the lxd ppa17:30
stokachuThiagoCMC: totally missed that17:30
ThiagoCMCyep17:30
cnfhmm17:31
ThiagoCMCI always start things with Ubuntu ISO and bare-minimum setup, so I can catch those things.. ha!   =P17:31
cnfi'm not liking these containers much17:31
jamcnf: if you don't have routes, then you can ask us to also put the containers in other subnets additionally17:31
jamvia the "--constraints spaces=" request17:31
jamcnf: and in bundles that looks like a "constraints" section on either the applications generally or the machines specifically17:32
stokachuThiagoCMC: glad you did, thanks again17:32
ThiagoCMCstokachu, another problem, after "conjure-up 2.1.2 from 'canonical' installed", "conjure-up openstack" returns: "-bash: conjure-up: command not found"17:32
ThiagoCMC:-(17:32
cnfwell, then i have to add the spaces=space-maas on _everything_17:32
stokachuThiagoCMC: yea that one is hard to fix17:32
ThiagoCMC=/17:32
stokachuThiagoCMC: you'll just need to exec $SHELL17:32
stokachuor logout of the terminal and back in17:32
ThiagoCMCok17:32
ThiagoCMCdamn... hehe... after logout and login: "$ conjure-up openstack" returns: "snap-confine has elevated permissions and is not confined but should be. Refusing to continue to avoid permission escalation attacks"17:33
ThiagoCMCany clue?17:34
cnfjam: so i need a constraint, but not a binding17:34
jamcnf: first, check if that works, second you could set up routes in whatever is actually managing those networks to allow 172.20* to talk to 172.10*17:34
stokachuThiagoCMC: what does `snap list` show?17:34
Budgie^Smoreand solved my sshultte issue :) so now I can ssh to the "private" instances for troubleshooting :)17:34
jamcnf: binding is where you want the application to talk, constraint is extra information about the machines you want17:34
jamcnf: the containers are likely set up to say "172.20.20.1 can you please send this packet to Juju controller at 172.20.10.1" and they're saying "sorry I can't do that"17:35
ThiagoCMCstokachu, "snap list" shows: "conjure-up  2.1.2 140 ... core        16-2     1441"17:35
stokachuok that should be fine17:35
stokachuhmm17:35
ThiagoCMC=/17:35
cnfwell, they are17:35
stokachuThiagoCMC: asking in #snappy17:35
jamcnf: now, choosing to firewall in that fashion is something people can chose to do, but if they do that, then they need to give us some other way to talk to the controller17:35
ThiagoCMCok17:35
cnfjam: my point was i expected juju to sort that out17:35
cnfor at least communicate it clearly17:35
jamcnf: the particular issue here, is that when the container comes up, it is like a separate machine to us, but because we can't reach the controller we can't tell it that we can't reach it17:36
jamcnf: consider if this were isolated machines (not containers)17:37
stokachujamespage: i pull the readme from https://jujucharms.com/openstack-base/ but it still says newton is the latest17:37
jamif we ask maas to bring up a machine that can't talk to us, there isn't much we can do. The Controller couldn't 'ssh' in to find out why, cause we can't talk to it.17:37
jamcnf: there is some stuff that we could special case for containers, cause we can poke at them from the host machine, but it is pretty fiddly and isn't something that we've gotten to yet17:38
cnfjam: maybe something like "this is the juju network, always be in it"17:38
jamcnf: again, its perfectly reasonable to have *routers* rather than need to be in the network17:38
jamthat's what gateways *do*17:39
cnfjam: yeah, i don't _want_ my storage network routed17:39
jamcnf: my home router knows how to get to 8.8.8.8, it doesn't live in 8.8.8.8 network17:39
beisnerstokachu, i believe we have ocata stable charm store bundle in flight.  with this cycle being short, and the first offset cycle, we kept newton as current stable in that published bundle, pending further verification.17:39
cnfjam: yes, but your home network isn't a security domain17:39
jamcnf: so, absolutely a choice you can make. and one you're choosing. but you can set up only routes to the Juju controller network17:39
jamnot to all possible public IPs17:40
jamfor example17:40
jamor you can make your containers on the Juju network17:40
jamwhich likely *is* routable17:40
jamcnf: cause the Juju controller needs access usually to the outside world17:40
cnf...17:40
jamcnf: so it depends what you mean by "I don't want my storage network to be routable"17:40
stokachubeisner: ok cool17:40
jamcnf: if every machine that is on the storage network is also on the externally-routable network, that sounds less secure17:40
jamcnf: than setting up just a route from 172.STORAGE to 172.JUJUCONTROLLER17:41
cnfit should be more clear on why shit isn't working, either way17:41
cory_fukwmonroe: :)17:42
jamcnf: I absolutely hear you on that, and honestly, shit is hard to get right and hard to convey, and hard to probe and know what's going on.17:43
stokachuThiagoCMC: re: openstack version, it is Newton, but Ocata is pending17:43
jamcnf: we are making improvements here, but we don't do everything everywhere yet17:43
jamanyway, need to go take my son to bed17:44
jambbaiab17:44
cnfcannot deploy bundle: cannot deploy application "cinder-ceph": subordinate application must be deployed without constraints17:45
cnfffs17:45
cnfok, lets see how this behaves17:46
cnfhmm, juju doesn't add interfaces to existing containers, it seems17:47
cnfk, destroying, and deploying again17:48
cnfand then off home17:48
admcleodThiagoCMC: did you want to deploy this on 'some machines' or just on a machine with lxd for testing/dev/playing around?17:52
Budgie^SmorelazyPower is there a reason why after an a fresh install there is 3 heapster replica sets?17:57
ThiagoCMCadmcleod, at first, I want OpenStack in one box. So, yes, for testing... I want to go Juju only but, looks like that conjure-up is required to start this thing more easily...17:57
ThiagoCMCLater, I'll go ahead and try to deploy it in my MaaS environment...17:58
ThiagoCMCI already have MaaS up and running, with 20 bare-metal servers...17:58
ThiagoCMCBut I need to fully understand Juju first.17:58
admcleodThiagoCMC: well, if you wanna do it on one box with lxd and ocata, the openstack-on-lxd link will work for that17:58
ThiagoCMCAnd conjure-up "masks" it...17:58
ThiagoCMCadmcleod, that's nice!17:59
ThiagoCMCdo you have some link?17:59
ThiagoCMCwith docs or quick and dirty guide?  =P17:59
admcleodThiagoCMC: https://docs.openstack.org/developer/charm-guide/openstack-on-lxd.html17:59
beisnerocata is also WIP there, but there's a branch18:00
ThiagoCMCnice!18:00
admcleodThiagoCMC: the doc refers to a git repo with bundles, the ocata bundle is in a branch called 'add-ocata' (for now, we will merge that soon)18:00
admcleodwhat beisner said18:01
ThiagoCMCI'm really interested in Ocata, Senlin, Aodh, Ceilometer and Gnocchi! I already have those things working but, I manually installed everything via "apt-get".18:01
ThiagoCMCJuju might be the way to go! I believe...18:01
ThiagoCMCI even have senlin-dashboard! https://launchpad.net/~sandvine/+archive/ubuntu/cloud-archive-ocata  :-D18:02
ThiagoCMCBut it is too complicated to maintain a cloud with just a bunch of ansible playbooks and/or manual steps...18:02
lazyPowerBudgie^Smore: that seems like a bug18:05
lazyPowerBudgie^Smore: and i haven't seen that prior. can you paste me a kubectl get po,svc,rc --all-namespaces so i can see precisely what you're referring to?18:05
Budgie^Smoreyeah I am filing a big against CDK right now and you guys can funnel it the right places :)18:05
Budgie^Smoreoh and yes I can grab that out18:06
lazyPowerta18:06
Budgie^SmorelazyPower I will say it doesn look like it increases the number of pods just replica sets18:06
lazyPowerBudgie^Smore: ok, was this a fresh install? as in no upgrade, just a bare juju deploy cdk and this was the result?18:08
Budgie^SmorelazyPower yes, the only changes I made were to constraint services to subnets and instance type18:08
Budgie^SmorelazyPower interesting it doesn't seem to show up in the cli output18:12
=== frankban is now known as frankban|afk
Budgie^SmorelazyPower: https://github.com/juju-solutions/bundle-canonical-kubernetes/issues/24118:15
ThiagoCMCWell, "conjure-up openstack", after staring the OpenStack deployment, failed again: https://paste.ubuntu.com/24236284/18:16
ThiagoCMCI gave up...   :-(18:16
lazyPowerBudgie^Smore: so CLI doesn't list the additional RC's that is ee, but it showing up in the dashboard?18:16
lazyPowerthats fun18:17
Budgie^SmorelazyPower yeah!18:17
lazyPowerBudgie^Smore: can you kubectl describe those RC's that are ghosted?18:17
lazyPowerkubectl describe rc heapster-v1.2.0.1-907310421  for example18:17
Budgie^Smoreyeah definitely something to do with the dashboard and not kubectl!18:20
lazyPowerBudgie^Smore: ok, i see the follow up there too18:20
lazyPoweri wonder, if you refresh do they go away?18:20
lazyPowerthe dashboard isn't socket based, so you dont get real time updates, its entirely possible that you loaded the dashboard while operations were happening and they weren't fully cleaned up during that page load18:21
Budgie^SmorelazyPower nope18:21
lazyPoweroh well that nukes that idea out of the water18:21
lazyPowerthats a weird one18:21
Budgie^Smoreeven did a ctrl+f5 to make sure it was a fresh pull18:21
lazyPowerif we can reproduce i'd say we need to file this against the dashboard project upstream, but barring that, you've found a fun one-off18:21
lazyPowerBudgie^Smore: when i'm done validating the etcd work i have in the pipeline i'll context swap and try to reproduce18:22
Budgie^SmorelazyPower I vaguely remember seeing it in the last cluster... I am not sure if it is the dashboard project or the where the initial deployment happens18:22
Budgie^SmorelazyPower no rush, it is justa weird one, probably could be low priority as it is just an oddity (unless I am missing any risks to leaving those there)18:23
lazyPowerBudgie^Smore: nah, if the api doesn't konw anything about it, you're proably fine18:23
lazyPowerBudgie^Smore: just for fun, kubectl get rs doesn't show them either right?18:23
lazyPowerreplicaset vs replication controller18:24
lazyPowerfor clarity, here's the upstream definition of the difference18:25
lazyPowerReplicaSet is the next-generation Replication Controller. The only difference between a ReplicaSet and a Replication Controller right now is the selector support. ReplicaSet supports the new set-based selector requirements as described in the labels user guide whereas a Replication Controller only supports equality-based selector requirements.18:25
Budgie^SmorelazyPower damn it we found them... oh yeah I like ReplicaSet / Deployment model :)18:25
lazyPowerok, so i had suspected that might be the case too18:25
lazyPowerdescribe them and lets see just dubya tee fig18:26
Budgie^SmorelazyPower, kubectl describe rs <rs name>?18:27
lazyPoweryep18:27
ThiagoCMCstokachu,  "conjure-up openstack", after staring the OpenStack deployment, failed again: https://paste.ubuntu.com/24236284/18:29
Budgie^SmorelazyPower yeah now I feel like I am being visited by Schrodinger's cat here!18:30
lazyPowerit both exists and doesnt exist at the same time eh?18:30
Budgie^SmorelazyPower pretty much!18:30
lazyPowerheisenbug18:31
Budgie^SmorelazyPower not sure if it has reach heisenbug level though18:32
lazyPowerBudgie^Smore: ok, i'll give this a reproduction attempt though18:37
lazyPowerthere has to be a reasonable explanation as to why there are replicasets and replication controllers in there that i dont see when testing18:38
lazyPowerthis might be related to our latest release, it might be some funkyness in the templates, it might also be a legit race bug where the addons are getting created and recreated18:38
Budgie^SmorelazyPower I suspect the later18:39
kklimondacan I disable juju unit/application for debugging, so it doesn't interfere with what I'm doing on the server?18:40
ThiagoCMCadmcleod, the following instructions also doesn't work: https://docs.openstack.org/developer/charm-guide/openstack-on-lxd.html18:42
ThiagoCMCThe "Juju Profile Update" is incomplete18:42
cnfwow18:42
cnfif i define constraints on my applications like jam said18:42
ThiagoCMCI can't find those "lxd-profile.yaml" and "config.yaml" files...18:42
cnfi get no hosts from MAAS18:42
cnfhmm18:43
beisnerhi ThiagoCMC - i'18:51
beisnerhi ThiagoCMC - i've run through openstack-on-lxd verbatim a number of times w/o issue.  at what point are you hitting the issue?18:51
ThiagoCMCbeisner, at "Juju Profile Update", where that "lxd-profile.yaml" comes from?18:52
ThiagoCMCAlso, on next step, I can't find that "juju config.yaml" either...18:52
beisnerThiagoCMC, did you complete the Host Setup section? https://docs.openstack.org/developer/charm-guide/openstack-on-lxd.html#host-setup   (ie. clone the repo)18:53
ThiagoCMCThere it is!!!  lol18:53
beisnerwoot ThiagoCMC18:54
ThiagoCMC^_^18:54
bdxdoes libjuju support `juju attach`?19:38
marcoceppibdx: good question, not sure, if it doesn't it should20:42
Budgie^SmorelazyPower and I am back at getting AWS to behave :-/21:52
kwmonroehey petevg, is this matrixy thing of concern to you? http://paste.ubuntu.com/24237456/22:38
kwmonroepetevg: specifically the "ha" bits.  we already talked about the "Could not run reboot" bits.22:38
petevgkwmonroe: yep. That's a bug, and a bad one, too. Please file a ticket.22:39
kwmonroepetevg: i'm asking you first because i'd love to blame cory_fu (https://github.com/juju-solutions/matrix/commit/7329196f6706e84c29b4cffa6cd64feb7c19f55e), but he already got the better of me when i tried to blame a layer-basic thng on his this mroning.22:39
kwmonroe*morning even22:39
petevgkwmonroe: no, that's my PR. That should be context.config.ha. Not sure why the tests didn't catch it :-/22:41
kwmonroelet's still blame cory.22:41
petevg... it might just be a missing case. Bleh. I'll fix it asap, in any case.22:42
kwmonroepetevg: mucho thanko: https://github.com/juju-solutions/matrix/issues/10822:43
petevgThis is my bad. He didn't catch it, though :-)22:43
petevgkwmonroe: fix here https://github.com/juju-solutions/matrix/pull/10922:52
petevgkwmonroe: it was an entertaining bug. The unit tests didn't catch it, because I was mocking out the context (unit tests don't test your assumptions!)22:52
petevg... and the functional tests didn't catch it, because it causes a failure only when the test has failed.22:52
petevgSo my "make sure this fails" tests were failing as expected.22:53
petevgI should make the functional tests check for failure reasons at some point. For now, the above PR should fix my bad.22:53

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!