[00:00] <Budgie^Smore> that is what I am saying hatch ... didn't complain about anything unusual creating the model from the CLI but try see it int the web UI it doesn't exist
[00:00] <hatch> that's....odd
[00:01] <hatch> let me give it a try here
[00:01] <Budgie^Smore> oh and the --refresh command is doing http://paste.ubuntu.com/24231722/
[00:03] <hatch> hmm ok let me see what I can find out for you
[00:03] <hatch> gimme a few minutes
[00:04] <Budgie^Smore> sure but I feel I am going to have to use a hosted juju controller to get around this for now
[00:04] <Budgie^Smore> not a huge deal was just using Jaas to avoid having to have a separate juju controller instance
[00:09] <Budgie^Smore> oh that is ashame that you can update the pastebin paste
[00:10] <Budgie^Smore> ok nevermind about the --refresh problem, I see what it is... it is trying to refresh a controller that is powered off
[00:11] <hatch> yeah, I'm trying to figure out why the models aren't showing up still :)
[00:12] <Budgie^Smore> although I would suggest that that is technical is a bug since it should really timeout on controllers it fails to access rather than trying indefinitely
[00:12] <hatch> oh definitely
[00:13] <Budgie^Smore> looks like it is set to retry every 7 secs too
[00:13] <Budgie^Smore> correction 2mins 7secs-ish
[00:15] <Budgie^Smore> hatch yeah I would kinda expect to see a blank model in the web ui after issuing the add-model command and before doing any deploy command
[00:16] <Budgie^Smore> hmmm ok now it is has shown up
[00:16] <Budgie^Smore> I am hitting the replication buffer
[00:18] <Budgie^Smore> 2 mysteries solved, 1 to go
[00:19] <Budgie^Smore> hatch ^
[00:19] <hatch> :D
[00:20] <hatch> which one is to go still?
[00:20] <Budgie^Smore> why it can't create the instance
[00:22] <Budgie^Smore> and I know what that is just don't think I can prove it easily and more importantly safely... I suspect it is the default vpc issue for the model since I am using 'juju deploy etcd --constraints "instance-type=m3.medium spaces=private"' to deploy it and the "private" space is in a different VPC that what jaas can currently handle
[00:23] <rick_h> Budgie^Smore: that's perfectly possible. A bug with what you're trying to do would be great as early beta feedback so we can investigate and see what we need to do.
[00:23] <rick_h> Budgie^Smore: as much copy/paste as possible.
[00:25] <Budgie^Smore> I think it is more of a feature request of having the ability to assign a model to a vpc-id some how but that is very cloud specific so I don't know what priority it would be worth giving it
[00:26] <Budgie^Smore> from a security stand point how it currently is working makes sense too, juju shouldn't step outside the default VPC unless given the ID to do  so
[00:28] <Budgie^Smore> hatch ^ make sense?
[00:28] <hatch> definitely
[00:28] <Budgie^Smore> the bug I will file though is the one lack of a timeout on --refresh when you have controllers that are inaccessible at the point in time
[00:29] <hatch> thanks
[00:31] <Budgie^Smore> unfortunately that does mean the end of my "experiment" of JaaS for now
[00:32] <hatch> sorry Budgie^Smore
[00:32] <hatch> I'll talk with the team tomorrow and see what I can find out about the VPC issue
[00:32] <Budgie^Smore> don't be, it was fun and love to see how far that has come so far... AWS is only temporary for me anyway
[00:33] <hatch> :)
[00:33] <hatch> I have to run, have a good night Budgie^Smore
[00:33] <Budgie^Smore> I want to see the design spec for when you can use JaaS with say a private MaaS environment
[00:36] <rick_h> Budgie^Smore: patience, one step at a time :)
[00:41] <Budgie^Smore> rick_h oh I have that, in the meantime I have to use the "workaround", it's all good :)
[00:44] <Budgie^Smore> rick_h that comment was more a vision of the potential it has in my "use-case"
[01:42] <Budgie^Smore> hmmm looks like the example I am trying to "follow" is using a future feature
[01:42] <Budgie^Smore> --to "subnet=subnet-bb1ab2dc"
[01:49] <Budgie^Smore> https://bugs.launchpad.net/juju/+bug/1659640
[01:49] <mup> Bug #1659640: Juju needs to support specifying subnets for controller <juju:Fix Committed by jameinel> <https://launchpad.net/bugs/1659640>
[02:09] <rick_h> Budgie^Smore: yea, but that was landed recently. It's in the alpha out now
[02:10] <Budgie^Smore> rick yeah but hasn't hit any ppa repo yet, right?
[02:10] <rick_h> Budgie^Smore: it's in the juju devel ppa
[02:10] <rick_h> Budgie^Smore: or use the snap :)
[02:10] <rick_h> Budgie^Smore: then you can keep things in apart nicely
[02:12] <Budgie^Smore> rick_h I found a page that said the devel was on 2.1.2!
[02:13] <Budgie^Smore> rick_h of course I have "lost it" again
[02:13] <rick_h> Budgie^Smore: yea, I think you're right. Maybe it's only in the snap now? /me checks around
[02:14] <rick_h> https://launchpad.net/~juju/+archive/ubuntu/daily has 2.2 alpha
[02:15] <rick_h> and here: https://launchpad.net/~juju/+archive/ubuntu/devel
[02:15] <rick_h> that has 2.2
[02:16] <Budgie^Smore> yes I see that but I didn't look at the repo... not sure if I want to run an alpha build though and I had another "workaround" that worked too :) remove a subnet temporarily and use --to zone=us-west-1a
[02:16] <rick_h> Budgie^Smore: and http://paste.ubuntu.com/24232393/
[02:16] <rick_h> Budgie^Smore: yea, that's why I like the snap way. You get a new /snap/bin/juju to work with
[02:16] <rick_h> Budgie^Smore: but yea, it's always the way. The feature you need/want is in the NEXT release :)
[02:16] <Budgie^Smore> yeah I should probably use snap more
[02:17] <Budgie^Smore> oh hang on a min... edge is at 2.2 beta and beta is at 2.2 alpha?
[02:17] <Budgie^Smore> lol ok that is mildly amusing
[02:17] <rick_h> Budgie^Smore: so edge is a daily build that tracks "the next release"
[02:18] <rick_h> Budgie^Smore: so the non-released, but next target (e.g. what the source code says it is currently if you build from source...) is beta
[02:18] <rick_h> late night for me
[02:18]  * rick_h goes to take out the trash and calm down before bed
[02:18] <Budgie^Smore> oh it is going to be a late one for me
[02:48] <Budgie^Smore> just seems my day to hit weird and wonderful roadblocks
[08:00] <kjackal> Good morning Juju world!
[08:55] <ybaumy> moin
[09:19] <ybaumy> i forgot to set VIP for mysql or hacluster
[09:20] <ybaumy> how do i do that after deployment
[09:24] <ybaumy> ah i got it
[09:24] <ybaumy> juju config mysql vip=IP
[09:24] <ybaumy> juju resolved mysql/0 .. 1 .. 2
[09:54] <cnf> morning
[09:54] <cnf> lets try this again
[10:01] <BlackDex> hello there. I'm using juju 2.1.2 and spaces. I Have a private and a public space. But if i want to deploy mediawiki for instance, i'm not able to let it use the two spaces. It will only bind to one.
[10:01] <BlackDex> Is there a setting i'm missing?
[10:12] <BlackDex> thx for jamespage on the other channel. using --constrains "spaces=public,private" does the job
[11:43] <SimonKLB> does anyone know if it's possible to run the collect-metrics script in the layer-basic venv?
[11:44] <SimonKLB> ideally run it in the reactive framework somehow
[12:08] <kjackal> SimonKLB: you mean it like a feature request?
[12:21] <SimonKLB> kjackal: more like if it was already possible
[12:25] <tvansteenburgh> SimonKLB: i thought collect-metrics is a hook
[12:27] <tvansteenburgh> hmm, maybe that was just unfortunate naming in the docs
[12:28] <kjackal> SimonKLB: there is a separete layer https://github.com/CanonicalLtd/layer-metrics/blob/master/hooks/collect-metrics and here is an example on how to use it https://github.com/juju-solutions/bigtop/tree/master/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager
[12:29] <SimonKLB> kjackal: i know how metrics work normally
[12:29] <SimonKLB> i was wondering if you could run it in the reactive framework somehow, so that you could access the virtualenv from layer basic easily
[12:30] <SimonKLB> tvansteenburgh: i might be mistaken, but i think the metrics works a bit different
[12:30] <mattyw> collect-metrics is just a hook. so you could use it with just layer basic by doing @hook.Hook("collect-metrics")
[12:30] <SimonKLB> mattyw: ah that's neat!
[12:31] <SimonKLB> so now i just need to get charmhelpers somehow :D
[12:31] <mattyw> ^^ that decorator is from memory so it might not be totally copy/paste accurate
[12:31] <mattyw> SimonKLB, I think charmhelpers comes with layers right?
[12:31] <SimonKLB> currently, im accessing charmhelpers from the virtualenv created by layer-basic
[12:31] <SimonKLB> mattyw: yea
[12:32] <mattyw> SimonKLB, depending on what you're doing there's nothing to stop you just having an executable bash script called collect-metrics in the hook directory
[12:33] <mattyw> SimonKLB, the main difference between the collect-metrics hook and other hooks is that errors in the collect-metrics hook won't put the charm in an error state
[12:35] <SimonKLB> mattyw: so normally when you use the reactive framework it creates the hooks for you so that they all execute the reactive main function
[12:36] <SimonKLB> im not sure what the best practice is here when using metrics
[12:36] <SimonKLB> since that has its own hook which then executes commands depending on which metric it is collecting
[12:37] <mattyw> SimonKLB, ah yes I see, I'm sure there was a recent version that included it, if not there should be
[12:37] <mattyw> SimonKLB, is there any reason you don't want to use the metrics layer?
[12:45] <SimonKLB> mattyw: i'd like to access the relations if possible so that i can fetch data from a database-charm when generating the metrics
[12:45] <SimonKLB> that would be very easy if i could run it in the reactive loop
[12:46] <rick_h> cnf: morning, how goes things out east?
[12:48] <kjackal> SimonKLB: for what is worth, if you want to activate the venv of the basic layer you can do so like this:  https://github.com/juju-solutions/layer-cwr/blob/master/actions/cwr-charm-pr#L7
[12:49] <SimonKLB> kjackal: yea i know, but i have to go through some unecessary hoops to import layer basic if im going to do it myself instead of letting reactive do it for me :)
[12:49] <mattyw> SimonKLB, is the information you want stored in charmhelpers.core.unitdata?
[12:50] <mattyw> SimonKLB, or where you hoping to be able to call relation-get?
[12:51] <cnf> rick_h: nice weather! :P
[12:51] <cnf> rick_h: good morning :P
[12:51] <cnf> just got a fresh red bull
[12:51] <SimonKLB> mattyw: relation-get hopefully :) i want the ip of the relational charm so that i can access the database-api
[12:52] <mattyw> SimonKLB, unfortunately relation-get insn't available in the collect-metrics hook - that's the other way they're slightly different to normal hooks
[12:53] <mattyw> SimonKLB, but in your reactive framework when those values are set from the relation the usual approach is to set them in the unitdata key value store. and then make use of them in the collect-metrics hook
[12:57] <SimonKLB> mattyw: the problem is that the data is static, im basically keeping track of how many units are running of a perticualr application
[12:57] <SimonKLB> mattyw: i could probably use libjuju for this though if it's not possible to get the relation data
[13:02] <mattyw> SimonKLB, so you want to be able to access the database you're related to?
[13:03] <mattyw> SimonKLB, but you what you really want to know is the number of units that are running?
[13:03] <SimonKLB> mattyw: yea but it could probably be solved in other ways also
[13:03] <SimonKLB> mattyw: well yea, or specifically the machinespecs the units are running on
[13:05] <mattyw> SimonKLB, and I guess just counting cpus/ram in the collect-metric hook is not an acceptable solution?
[13:06] <SimonKLB> mattyw: well the problem is that im not counting the cpu/ram on the machine the collect metric is running on, i want to get the cpu/ram on other machines that is running a different charm
[13:06] <SimonKLB> (this is for the charmscaler btw)
[13:06] <mattyw> ok - I see
[13:10] <rick_h> cnf: send some of that weather over here. Still below freezing each morning here.
[13:11] <cnf> ouch :P
[13:11] <cnf> we are a country of cold and wet, so we like our sunshine :P
[13:11] <tvansteenburgh> rick_h: is it possible to remove a charm from the charm store (in a personal namespace)?
[13:11] <cnf> rick_h: so, are you available to help me some today?
[13:11] <rick_h> tvansteenburgh: yes, the charm delete api has been wired up but it's not yet through the CLI to end users. So it's possible but still requires some manual api foo atm
[13:12] <magicaltrout> make it so, my namespace is full of $hit I just hide from users! :)
[13:12] <rick_h> cnf: yes, I wanted to kind of reset and ask if you've got notes/doc on what your setup looks like. How many machines, special things about them, what the spaces layout looks like, the bundle you're working to use.
[13:12] <tvansteenburgh> rick_h: okay, so it's coming soon to the cli?
[13:12] <cnf> sure
[13:12] <rick_h> tvansteenburgh: it's not on the "on fire" list so it's coming but I don't know it's going to be out in the next couple of weeks
[13:12] <cnf> rick_h: you want to do that here, or in pm, or somewhere else?
[13:13] <tvansteenburgh> rick_h: ack
[13:13] <rick_h> cnf: we can do here. Folks can ignore us or if you'd prefer PM feel free.
[13:13] <cnf> ok, so I have a VM running the MAAS controller
[13:14] <cnf> and a kvm VM on the maas controller running the juju controller
[13:14] <cnf> i have 4 physical HP machines if various layouts, at least 16 cores and 24G in the smallest one
[13:14] <cnf> i set up a copper network for PXE boot and maas networking, because i could not figure out howto pxe boot of a LACP link
[13:15] <cnf> and all machines have a 10G LACP with vlans for various networking
[13:15] <cnf> well, all but the smallest, that has just a 1G trunk for now
[13:15] <rick_h> k
[13:16] <cnf> i'm using a slightly modified bundle yaml i got from jamespage
[13:16] <cnf> http://termbin.com/pjqj
[13:17] <cnf> past few weeks, i have run in a fair amount of bugs and problems with both maas and juju
[13:17] <cnf> right now, my state is http://termbin.com/2yke
[13:17] <rick_h> cnf: k, so the machines look like they've got two devices? or are there more with the bonds?
[13:18] <cnf> uhm, they have a ton of devices :P but active it's 1 for PXE / MAAS, and one bond with 2 or 3 vlan tags
[13:18] <cnf> configured in maas
[13:18] <rick_h> cnf: k
[13:18] <cnf> https://www.dropbox.com/s/tupaiwe049g8rzk/Screenshot%202017-03-23%2014.18.32.png?dl=0 for example
[13:19] <rick_h> cool yea I see machine 0 with 4 addresses in the machine status output
[13:21] <cnf> so what stands out to me, atm (i'm hammering the nails as i see them...)
[13:21] <cnf> message: 'can''t get info for image ''juju/xenial/amd64'': not found'
[13:21] <cnf> rick_h: i should add that only the public network on machine 0 can access the internet directly
[13:21] <cnf> and i have http-proxy and https-proxy set
[13:22] <cnf> which needs to go over the maas network
[13:22] <rick_h> cnf: ok, yea I see all the lxd on machine 0 is in pending and that is lxd reaching out for the container images
[13:22] <cnf> right, so i think it's not respecting proxy settings?
[13:23] <rick_h> cnf: yea, I'm looking. So lxd supports these per https://github.com/lxc/lxd/issues/2147
[13:24] <rick_h> cnf: and https://bugs.launchpad.net/juju/+bug/1594720 says the fix is release ...
[13:24] <mup> Bug #1594720: lxd containers not using configured proxy for downloading images <2.0> <addressability> <lxd> <network> <proxy> <sts> <juju:Fix Released by thumper> <https://launchpad.net/bugs/1594720>
[13:24] <cnf> hmm
[13:25] <rick_h> cnf: so looking at the LP bug there's some commands to run that get the LXC config to see if the proxy is set and we can then check if it's set correctly or not.
[13:25] <rick_h> cnf: https://bugs.launchpad.net/juju/+bug/1594720/comments/15
[13:25] <mup> Bug #1594720: lxd containers not using configured proxy for downloading images <2.0> <addressability> <lxd> <network> <proxy> <sts> <juju:Fix Released by thumper> <https://launchpad.net/bugs/1594720>
[13:27] <cnf> looking for the lxc command to show that
[13:29] <cnf> rick_h: $ sudo lxc config get core.proxy_http
[13:29] <cnf> http://172.20.20.1:8000
[13:29] <rick_h> cnf: and that's the correct value for the proxy?
[13:29] <cnf> yep
[13:30] <rick_h> cnf: what about https?
[13:30] <cnf> and it is reachable from the host
[13:30] <cnf> set as well
[13:30] <rick_h> does the proxy also run for https?
[13:30] <cnf> yep
[13:30] <rick_h> cnf: ok, so then we should try to find in the lxc/lxd logs to see where the actual urls being hit for the images is and see if that lines up correctly or if it's going somewhere crazy
[13:31] <cnf> $ curl https://cloud-images.ubuntu.com/releases returns a 301
[13:32] <cnf> https://bpaste.net/show/a39688f501a9
[13:33] <rick_h> cnf: do you have the actual error message about the image? I'm not sure on the 404 lines in that log.
[13:33] <cnf> rick_h: it literally says lvl=info msg="Sending top level 404" t=2017-03-22T18:11:16+0000 url=/1.0/images/juju/xenial/amd64
[13:34] <rick_h> cnf: that's the lxd images right?
[13:34] <rick_h> cnf: sorry the lxd logs
[13:34]  * rick_h gets more coffee in him
[13:34] <cnf> that's from /var/log/lxd/lxd.log
[13:34] <cnf> :P
[13:34] <cnf> i'm on my 2nd red bull :P
[13:34] <rick_h> cnf: but in the juju end was there a log that stated anything ?
[13:34] <rick_h> cnf: like in the machine-0 log or something?
[13:34] <cnf> yes, in http://termbin.com/2yke
[13:34] <cnf> containers:
[13:34] <cnf>       2/lxd/0:
[13:35] <cnf>        machine-status:
[13:35] <cnf>           current: provisioning error
[13:35] <cnf>           message: 'can''t get info for image ''juju/xenial/amd64'': not found'
[13:36] <rick_h> right, thanks. Sorry I ended up searching for "xenial" but on the bundle pastebin
[13:36] <cnf> :P
[13:37] <cnf> i don't know where the path /1.0/images/juju/xenial/amd64 should exist
[13:38] <rick_h> cnf: yea, I'm just looking to see if anyone from lxd is around that can shed more light on that.
[13:39] <cnf> right
[13:39] <rick_h> cnf: I've not used lxd in a proxy setup and so I'm not sure if that's normal "Oh, I don't have an image so I'll go get one" or something failed and that's "404 I don't have and could not get that image"
[13:39] <cnf> uhu
[13:39] <cnf> i have not used lxd at all
[13:39] <cnf> well, not directly
[13:43] <jam> rick_h: cnf: IIRC ivoks had set up something with offline support for cloud-images and found the images were in the wrong directory
[13:43] <cnf> hmm
[13:44] <jam> (he had done something like an rsync of the cloud-images directory, but turns out on the host machine they are in the wrong path and an apache rule puts them in the right one, IIRC)
[13:44] <cnf> uhm
[13:44] <cnf> you lost me, there
[13:45] <cnf> apache rule?
[13:45] <cnf> apache where?
[13:45] <jam> cnf: so, the URL we care about is likely to be http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson
[13:45] <jam> cnf: on that machine you should be able to run
[13:45] <jam> lxc image list ubunt:
[13:45] <jam> sorry
[13:45] <jam> lxc image list ubuntu:
[13:46] <jam> which reads the above URL and http://cloud-images.ubuntu.com/releases/streams/v1/downloads.sjson
[13:46] <jam> to return a big list of officially released images
[13:46] <cnf> that's empty
[13:46] <cnf> https://bpaste.net/show/be8d401c7f1c
[13:46] <jam> sorry http://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:download.json
[13:46] <jam> cnf: I think you need a :
[13:46] <jam> "lxc image list ubunt:"
[13:46] <jam> ubuntu:
[13:47] <jam> wow, that last u doesn't like me
[13:47] <cnf> error: Get https://cloud-images.ubuntu.com/releases/streams/v1/index.json: Unable to connect to: cloud-images.ubuntu.com:443
[13:47] <jam> otherwise you're searching for a local image named 'ubuntu' vs listing the images in the 'ubuntu:' remote
[13:47] <jam> cnf: if you have http-proxy set, do you have https-proxy set?
[13:47] <jam> cause LXC only uses https-proxy it doesn't fall back to http-proxy for 443 traffic
[13:47] <cnf> jam: yes
[13:47] <cnf> curl https://cloud-images.ubuntu.com works fine
[13:48] <jam> and:  sudo lxc config get core.proxy_https
[13:48] <cnf> $ sudo lxc config get core.proxy_https
[13:48] <cnf> http://172.20.20.1:8000
[13:48] <jam> cnf: if "lxc image list ubuntu:" isn't listing any images
[13:48] <jam> that's why Juju can't find them
[13:49] <cnf> jam: right... but why? i can curl it just fine with those proxy settings?
[13:49] <jam> cnf: echo $https_proxy and echo $http_proxy ?
[13:49] <cnf> show the same values
[13:50] <cnf> $ echo $https_proxy
[13:50] <cnf> http://172.20.20.1:8000
[13:50] <cnf> it's all set by juju
[13:50] <jam> cnf: well, that doesn't mean we can't get something wrong. The only other quick thing I can think of is to try bouncing the lxd agent
[13:51] <jam> sudo service restart lxd
[13:51] <jam> or is it "sudo service lxd restart"
[13:51] <jam> I usually get that wrong the first time
[13:51] <jam> anyway, have to go pick up my wife
[13:51] <cnf> i went for sudo service lxd restart :P
[13:51] <jam> you're at a point where I'd have to drop into #lxd and ask stgraber
[13:51] <cnf> hmz
[13:52] <rick_h> jam: cnf yea, where I was going to head next for the lxd specifics on that.
[13:52] <cnf> jam: #lxd is pretty much empty
[13:52] <cnf> is it #lxc?
[13:54] <cnf> it's #lxcontainers
[13:54] <rick_h> #lxcontainers
[13:54] <rick_h> yea
[13:55] <cnf> i wonder what timezone they are on...
[13:56] <rick_h> a mix, I know the lead is in my timezone ish.
[13:56] <cnf> right
[13:59] <cnf> i need a vacation :/
[14:15] <cnf> rick_h: why do i always run into the weird bugs? :P
[14:17] <rick_h> cnf: heh, because you're doing the fun complicated real world install
[14:17] <cnf> yeah :P
[14:17] <cnf> the "fun" part is wearing off though
[14:18] <cnf> 3 weeks, and still nothing running is getting frustrating
[14:18] <rick_h> cnf: it's hard to setup a lab for this to put into automated testing, but it's this stuff that has to work to go out there into production. Sorry you're hitting it. Looking at bugs/etc thought this should work ok
[14:18] <cnf> uhu
[14:21] <magicaltrout> reminds me of when SQL had about 100 different dialects that automatic SQL generators had to support. All seemed fine until you tried something tricky! ;)
[14:21] <magicaltrout> lots of moving parts in LXD, Juju and MAAS
[14:22] <cnf> uhu
[14:22] <cnf> when i'm working from home
[14:23] <cnf> i have zelda to comfort me in the waiting periods :P
[14:23] <cnf> here, no such luck
[14:29] <magicaltrout> sounds a bit lewd! ;)
[14:30] <cory_fu> jamespage: Any objection to me re-promulgating ganglia and ganglia-node under a focused ganglia-team (that I'm going to create) which is owned by openstack-charmers?
[14:31] <jamespage> cory_fu: works for me
[14:31] <jamespage> cory_fu: might have one already one sec
[14:31] <cory_fu> Cool.  Also, of note, I'm looking at merging the ganglia-node review
[14:31] <jamespage> cory_fu: https://code.launchpad.net/~ganglia-charmers
[14:32] <cory_fu> jamespage: Ok, cool.  I'll get that sorted out, then
[14:32] <jam> cnf: are you able to curl the full url http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson
[14:32] <jam> I'm wondering if your proxy is doing caching, and maybe has cached something poorly
[14:32] <cnf> jam: yes
[14:33] <cnf> jam: curl works just fine
[14:39] <rick_h> jam: so looking at #lxdcontainers it works from lxd, but something in how juju is setting things up causes the proxy config to not work.
[14:42] <jam> rick_h: you stopped the load test, right? I tried to see the progress but just got a 404
[14:42] <rick_h> jam: yes, had a machine issue here that I needed to kill it for unfortunately
[14:42] <rick_h> I should setup to be able to start/run it from the maas machine vs my desktop
[14:42] <jam> rick_h: np, I feel we showed what we wanted to
[14:42] <cory_fu> jamespage: Did you intend for the gate-xenial in ganglia and ganglia-node to not be executable and thus be skipped?
[14:43] <rick_h> jam: rgr, got over 400 models so yay
[14:43] <jamespage> cory_fu: yes - we need to push up the current proposed ones first
[14:43] <jamespage> cory_fu: then we can enable the xenial gate tests
[14:43] <cnf> hmm
[14:43] <cory_fu> jamespage: Ok, so only push them to trusty?
[14:43] <cnf> rick_h: any more ideas? :P
[14:43] <jamespage> cory_fu: series in metadata - its a circular depends issues
[14:43] <cory_fu> jamespage: Ah, gotcha
[14:43] <jamespage> ganglia <-> ganglia-node
[14:44] <jamespage> cory_fu: I'll push another rev to edge once we have this set done
[14:44] <rick_h> cnf: no :( we're beyond my scope. I know jam is EOD but he's the next best person that might be able to help or get someone to help. Honestly, the answer is probably going to be to get as much of this material as possible with the lxd "it works from here" added to the notes into a bug for the engineering team in juju to get at.
[14:44] <cory_fu> jamespage: Ok
[14:44] <cnf> hmm
[14:44] <rick_h> cnf: I'm sorry, I was hoping we'd be able to walk through a hole in the setup, but it's setup right. From your end everything *should work*
[14:44] <cnf> it should, indeed
[14:45] <cnf> rick_h: i appreciate your time none the less
[14:45] <rick_h> proxy is set, lxd does work, but somehow juju isn't doing the right thing
[14:45] <cnf> it at least validates i didn't do something stupid
[14:45] <cnf> (it has been know to happen)
[14:45] <rick_h> cnf: completely, I appreciate you working through that with the lxdcontainers folks to make sure the bug isn't there. That's really helpful and one less layer to try to go through
[14:46] <cnf> ok, lets hope jam becomes available soon
[14:47] <rick_h> cnf: so he's UTC+4 so that's why I think for today the best thing is to get the bug filed with all the notes from today
[14:47] <rick_h> cnf: and then we can ask nicely for him to find someone to peek at it in the morning.
[14:47] <cnf> hmz
[14:47] <cnf> these timezones are killing me :(
[14:48] <rick_h> cnf: yea, I guess we've gotten used to it over time so it seems natural but it's definitely not helpful in this situation.
[14:48] <cnf> rick_h: what is grating on me, is that i have been doing this for several bugs for weeks now
[14:48] <cnf> and getting them fixed just takes so long
[14:48] <cnf> get 1 baby step, then wait for the next day
[14:49] <cnf> rick_h: and i don't mean to invalidate your effort
[14:49] <cnf> just gets frustrating on my end :P
[14:49] <rick_h> cnf: I totally understand. /me will look away while you go kick something
[14:49] <cnf> :P
[14:52] <magicaltrout> on the flip side cnf if you ever want to job swap, you'll be straight into to a Juju Consultancy role! ;)
[14:52] <cnf> ha!
[14:52] <cnf> magicaltrout: i actually am a freelance IT consultant
[14:53] <kwmonroe> tvansteenburgh: amulet issue when tearing down the class:  http://paste.ubuntu.com/24235292/
[14:53] <magicaltrout> if you're paid by the hour, surely broken Juju is actually a bonus? ;)
[14:53] <cnf> magicaltrout: it's a long term project, so i get paid either way
[14:53] <cnf> magicaltrout: but i would like something to WORK eventuakky
[14:54] <magicaltrout> that is a valid requirement I guess
[14:54] <magicaltrout> maybe pitch it as a long term goal ;)
[14:54] <cnf> for my sanity, if nothing else
[14:54] <magicaltrout> aye i know the feeling
[14:55] <cory_fu> jamespage: What repo are the new ganglia and ganglia-node revisions coming from?  The "Submit a Bug" links point back to lp:charms/ganglia[-node] but those don't have any commits newer than 2014.
[14:55] <magicaltrout> try talking to kwmonroe about big data
[14:55] <magicaltrout> it'll soo grind you down
[14:55] <magicaltrout> +n
[14:55] <cnf> magicaltrout: i don't like waking up and thinging "ugh, not more of this, again"
[14:56] <jam> cnf: rick_h: so it seems that things like "lxc list" are driven by the client, not the lxd agent, which means 'jujud' has to find cloud-images itself. I seem to remember menn0 fixing a bug recently about Juju and http_proxy not getting set early enough in the lifetime of jujud
[14:56] <kwmonroe> magicaltrout: shouldn't you be spinning up drill bits?
[14:57] <jam> cnf: the other thing that you could try is setting things like http_proxy= in /etc/environment and then restarting agents
[14:57] <magicaltrout> sorry pink face
[14:57] <jamespage> cory_fu: if you're ok with the charms I can setup all of the metadata - but basically the bug reporting is on launchpad
[14:58] <jamespage> and the git repos are on github.com/ganglia-charms
[14:58] <jamespage> cory_fu: https://bugs.launchpad.net/charm-ganglia and https://bugs.launchpad.net/charm-ganglia-node
[14:59] <jamespage> cory_fu: and then I'll go delete the old branches in bzr
[14:59] <cory_fu> jamespage: Ok.  I'm getting the "no module named yaml" from ganglia, so it needs the tests.yaml and requirements.txt changes that you did in ganglia-node
[14:59] <cory_fu> But otherwise they seem good
[14:59] <jamespage> cory_fu: odd I thought I did that
[14:59] <jamespage> coreycb: lemme check again
[14:59] <cory_fu> jamespage: I'm over here. ;)
[14:59] <cnf> https://bugs.launchpad.net/juju/+bug/1675440 for those that want to track it
[14:59] <mup> Bug #1675440: juju unable to get LXD images <juju:New> <https://launchpad.net/bugs/1675440>
[14:59] <jamespage> cory_fu: doh
[15:00] <jamespage> I wish irccloud would follow a converstion with nics
[15:02] <jamespage> cory_fu: -6 imported into reviews.j.c
[15:02] <cory_fu> jamespage: Hrm.  Yeah, the review does seem to have those bits.  Wait, I think it's my fault
[15:02] <jamespage> cory_fu: no it was missing the requirements.txt
[15:02] <cnf> jam, rick_h how can I tell juju to retry an lxd?
[15:02] <cnf> resolved is for units, and retry-provisioning doesn't work on containers
[15:03] <cory_fu> jamespage: Ah, ok
[15:03] <cory_fu> Thanks
[15:03] <cnf> jam: i edited /etc/environment, and restarted lxd and jujud-machine-2
[15:06] <jam> cnf: there is "juju retry-provisioning" but I don't think it supports containers
[15:06] <cnf> jam: it doesn't
[15:08] <cnf> hmm
[15:09] <cnf> rick_h: any suggestions?
[15:09] <cnf> beside a reboot?
[15:09] <cnf> hp servers take forever to reboot :P
[15:09] <rick_h> cnf: no, just to manually do the commands vs using the bundle
[15:09] <cnf> rick_h: ok, what commands?
[15:10] <rick_h> cnf: e.g. deploy or add-unit with a placement directive into a container e.g. "juju add-unit XXXX --to 0:lxd"
[15:10] <rick_h> cnf: or just test juju deploy ubuntu --to 0:lxd
[15:10] <cnf> rick_h: so i have to remove them, and add them again?
[15:11] <cnf> ERROR cannot add application "ubuntu": placement scope: invalid model UUID "0"
[15:12] <jam> cnf: lxd:0
[15:12] <jam> '0' is the machine ID
[15:13] <jam> in your case it would be
[15:13] <jam> you probably want "juju add-unit ubuntu --to lxd:2"
[15:13] <jam> given your earlier messages about being on machine 2
[15:13] <jam> ah sorry, your bug says machine 1
[15:16] <cnf> allocating
[15:17] <cnf> oh, bah
[15:17] <cnf>  message: 'unable to setup network: no obvious space for container "2/lxd/1",
[15:19] <tvansteenburgh> kwmonroe: the amulet test failure is repeatable?
[15:21] <tvansteenburgh> kwmonroe: i don't know why you'd get that behavior unless your ~/.local/share/juju/store-usso-token disappeared
[15:21] <kwmonroe> tvansteenburgh: the same charm (spark-21) failed in all my clouds: http://bigtop.charm.qa/cwr_bundle_spark_processing/4/report.html.  i'll pull it into a simpler test case with the ubuntu charm to make sure.
[15:21] <kwmonroe> tvansteenburgh: my question is why does "juju remove-application" need to talk to the store at all?
[15:22] <cnf> rick_h: i think the new one says " container started"
[15:22] <cnf> but status still shows pending
[15:22] <cnf> not sure is this is expected
[15:22] <cnf> it has an instance-id
[15:22] <rick_h> cnf: hmm, so what does juju show-machine show for that?
[15:22] <rick_h> cnf: sorry, I'm in a meeting so a little distracted atm
[15:23] <cnf> rick_h: machine-status: running, juju-status: pending
[15:23] <cnf> so it got farther than the other ones
[15:23] <rick_h> cnf: hmm, ok
[15:23] <tvansteenburgh> kwmonroe: it has to auth to the controller
[15:23] <cnf> are we sure juju/xenial/amd64 is a valid image?
[15:24] <cory_fu> jamespage: ganglia and ganglia-node are GTG.  So, you're going to push them to ~ganglia-charmers and re-promulgate?
[15:24] <jamespage> cory_fu: I can do that
[15:24] <kwmonroe> oooohhh.. yeah tvansteenburgh.  that makes sense.  let me double check my login status on the machine where it's failing.  this is probably just pilot error.
[15:24] <cory_fu> jamespage: Ok, let me know if you need anything from me, then.
[15:25] <cory_fu> jamespage: I'm happy to help with any parts of the process
[15:25] <jamespage> cory_fu: no worries - thankyou for the reviews :-)
[15:25] <cnf> rick_h: maybe it's a broken charm, or something
[15:25] <cnf> though that would be weird, jamespage would have run into this, then
[15:26] <hatch> Budgie^Smore `juju add-model --config vpc-id=<id>`
[15:26] <cory_fu> jamespage: Should I go ahead and close said reviews now, or wait for confirmation that they're re-promulgated?
[15:27] <cnf> hmz
[15:28] <jamespage> cory_fu: please go ahead - I'll wrap up my meeting and move things around as discussed
[15:29] <cnf> uhm
[15:29] <cnf> wth
[15:33] <Budgie^Smore> hatch I will give the add-model --config vpc-id a whirl, I broke the model again last night so bad at the controller level :)
[15:34] <hatch> oops! :)
[15:34] <rick_h> cnf: what's it doing now? It was pending but no reason given?
[15:34] <cnf> juju-status is pending on the ubuntu one
[15:34] <rick_h> cnf: this is from just "juju deploy ubuntu"
[15:34] <cnf> machine-status is deployed
[15:34] <rick_h> cnf: right, and if you show-machine 0 there's no more verbose output there?
[15:35] <cnf> no
[15:35] <rick_h> cnf: can you ssh to the new machine then? Does it have an IP addr and such?
[15:35] <cnf> juju-status:
[15:35] <cnf>           current: pending
[15:35] <cnf>           since: 23 Mar 2017 16:19:26+01:00
[15:35] <cnf>         instance-id: juju-857117-2-lxd-2
[15:35] <cnf> rick_h: uhm, not directly
[15:35] <cnf> it didn't get an ip in the maas/juju network
[15:36] <ThiagoCMC> Hey guys, I'm very familiar with OpenStack on Ubuntu (via apt-get and manual setup), however, from now on, I want to start using Juju but, I never used it before...
[15:37] <ThiagoCMC> Is there any way to install OpenStack via Juju, without MaaS, just everything in one box?
[15:37] <Budgie^Smore> it probably wasn't truly broken but my Kubernetes master wouldn't start waiting for pods to start and I couldn't be too bother troubleshooting that on a clean model
[15:37] <ThiagoCMC> I'm seeing that conjure-up can help with that but, what about going straigh with Juju instead?
[15:37] <rick_h> cnf: is there anything useful in the debug-log or the machine-0 log on the host machine?
[15:38] <cnf> but i can ssh to it
[15:38] <admcleod> ThiagoCMC: you could try https://docs.openstack.org/developer/charm-guide/openstack-on-lxd.html
[15:38] <jamespage> cory_fu: https://jujucharms.com/ganglia/ and https://jujucharms.com/ganglia-node all done
[15:38] <ThiagoCMC> admcleod, thank you!
[15:38] <jamespage> ThiagoCMC: alternatively try with conjure-up
[15:38] <jamespage> ThiagoCMC: http://conjure-up.io/
[15:38] <ThiagoCMC> jamespage, you think that Conjure-up Next PPA is a better starting point?
[15:38] <cnf> nothing interesting in the machine-log
[15:39] <jamespage> ThiagoCMC: use the snap as detailed in that link - I think thats the right route these days stokachu ?
[15:39] <ThiagoCMC> Oh, I see...
[15:39] <cnf> rick_h: /var/log/juju is empty inside the container
[15:41] <rick_h> cnf: right, but on the host machine it should have any notes.
[15:41] <cnf> nothing of interest
[15:42] <ThiagoCMC> Does Juju OpenStack supports: OpenStack Ocata with Senlin, AODH and new Ceilometer with Gnocchi?
[15:42] <cnf> rick_h: i guess 2017-03-23 15:19:39 INFO juju.tools.lxdclient client_image.go:170 found image from https://cloud-images.ubuntu.com/releases for juju/xenial/amd64 = 2cab90c0c342346ea154bc2e8cacdae752a70747a755ce1f2970c9a9ebb5fe8c
[15:43] <cnf> so it should bloody well work
[15:44] <magicaltrout> ThiagoCMC: you need to prepend your questions with jamespage :
[15:44] <magicaltrout> :)
[15:45] <ThiagoCMC> ^_^
[15:45] <Budgie^Smore> oh I remember why I destroyed the controller / model hatch, I hit a bug where juju wouldn't ssh proxy to private systems either by setting proxy-ssh config option or using --proxy flag but there had already a bug filed and triaged for it
[15:45] <jamespage> ETOMANYCHANNELS
[15:45]  * magicaltrout leaves norwich before he's hunted down
[15:45] <jamespage> ThiagoCMC: Ocata: yes, Aodh: yes, Senlin: no, Gnocchi: no
[15:45] <hatch> Budgie^Smore ahhh ok thanks for looking at the bugs though - appreciate any new issues too :)
[15:46] <ThiagoCMC>  jamespage, Hmm... That's sad... But okay... Thanks!
[15:46] <Budgie^Smore> hatch I didn't delibrately hunt for bugs but they crop up when I am searching for what I am doing wrong ;-)
[15:47] <hatch> haha, hey, whatever works :)
[15:47] <jamespage> ThiagoCMC: I'm hoping to put Gnocchi on the roadmap for the openstack charms but it won't be in the short term
[15:47] <jamespage> ThiagoCMC: if someone would like to work on a senlin charm I'm all ears :-) but don't have time with the existing pool of resources working across the charm set (33 of them)
[15:48] <cnf> rick_h: so uhm
[15:48] <cnf> i removed those units
[15:48] <cnf> and added them agin
[15:48] <jamespage> magicaltrout: we should do a beer soon
[15:48] <cnf> now it seems the containers came up
[15:48] <cnf> i didn't actually _change_ anything!
[15:48] <magicaltrout> indeed now the grayness has left
[15:48] <rick_h> cnf: ? so what changed? Did jam have you do something?
[15:48] <cnf> no!
[15:48] <cnf> nothing was changed
[15:48] <cnf> >,<
[15:48] <ThiagoCMC> jamespage, that's okay! I have those things working now but, I'm deploying everything using my own Ansible playbooks, workd great! I have to start working with Juju to help you guys!   :-D
[15:49] <cnf> shit still isn't working, but the containers at least came up
[15:50] <cnf> wth
[15:50] <cnf> rick_h: so all my containers are running now
[15:50] <cnf> but none of them have juju come up
[15:52] <cnf> wth is going on :(
[15:53] <cnf> k, i'm going to destroy my model, and deploy it again
[15:53] <cnf> and go for a 20 minute stroll
[15:55] <admcleod> cnf: i wonder if it as something to do with one dns resolution attempt working, and one failing - maybe you have a broken dns server somewhere in a pool?
[15:56] <cnf> admcleod: it seems to work for everything else
[16:00] <admcleod> cnf: perhaps everything else handles such a failure gracefully? just guessing. ive just tried to reproduce with local lxd provider and proxy and i cant
[16:09] <Budgie^Smore> so my first weird question of the day, how is jaas going to be handle machines that are behind a nat router? (btw hatch that problem with my model not showing up in the UI has something to do with the initial load (after login) of the page)
[16:09] <rick_h> Budgie^Smore: so that's why it only works in public cloud atm
[16:10] <Budgie^Smore> rick_h but even in a public cloud I can have instances behind a nat router :P
[16:10] <Budgie^Smore> (tbh it is my understanding of how a VPC in AWS actually works
[16:11] <rick_h> Budgie^Smore: right, but the agents do most of the work to call out to the controller and the two exceptions I can think is juju ssh and juju actions so will have to see how they behave behind a vpc
[16:13] <Budgie^Smore> rick_h I am "testing" right now, so we will see, as it stands I don't think ssh (and by extension exec) will work. probably need a proxy system in the environment and juju having knowledge of it
[16:14] <Budgie^Smore> rick_h I am thinking something like a poor man's VPN connection using sshuttle
[16:14] <rick_h> Budgie^Smore: hmm, yea, will have to add this to the "test and figure out wtf" list
[16:15] <bdx> Budgie^Smore: yo
[16:15] <Budgie^Smore> give me an hour or so and I will have an initial test done
[16:15] <Budgie^Smore> bdx: well howdy
[16:16] <bdx> Budgie^Smore: hey, just reading some of the backlog here
[16:18] <bdx> Budgie^Smore: I have plenty of envs that are running in private address spaces (behind a nat) using JAAS .... I've yet to find anything that doesn't work as I would expect ... minus ssh
[16:21] <bdx> Budgie^Smore: sshtunnel and (depending on if you have a publicly available endpoing) openvpn are great remedies for access in that use case
[16:22] <Budgie^Smore> bdx that is what I expect too ssh is always proablematic in those envs... I have used sshuttle in the past too, as long as I have 1 ssh accessible system in the env that is
[16:23] <bdx> Budgie^Smore: are you using AWS provider?
[16:23] <Budgie^Smore> bdx in this case, yes
[16:26] <bdx> Budgie^Smore: have you configured nat, and igw routing tables, and configured your subnets to use the routing tables that point to the respective nat or igw?
[16:28] <Budgie^Smore> bdx: yup 2 x public subnets, 2 x private subnets, 1 x igw and 1 x nat within a vpc. actually found a cloundformation yaml for it too
[16:28] <bdx> Budgie^Smore: niceee, sounds like you are on the right track ... did you pipe that through to juju then?
[16:29] <Budgie^Smore> bdx yeah I am actually deploying charms right now too it (using the CDK as a guide but customizing to take into account the environment)
[16:30] <bdx> Budgie&Smore: nice
[16:34] <cnf> rick_h: so the containers started now
[16:34] <bdx> Budgie^Smore: I have configured most of my apps have both subnets configured as juju spaces such that only put internet facing proxies land on the igw subnets, and everything else behind nat gateway subnets (the way it should be)
[16:34] <bdx> http://paste.ubuntu.com/24235685/
[16:34] <cnf> rick_h: model isn't working, but for some reason the containers started
[16:34] <Budgie^Smore> bdx pretty much doing the same :)
[16:35] <cnf> rick_h, jamespage http://termbin.com/ae81
[16:35] <cnf> suggestions?
[16:35] <jam> rick_h: I'm really past EOD. my gut says we're failing to give an error when requesting a container that doesn't have a defined space on a machine with multiple network interfaces
[16:36] <Budgie^Smore> ok "1 unit, xenial, 8x26.88GHz, 32.00GB, 8.00GB" seems wrong... I don't know of any CPU that has 26GHz cloak!!!
[16:36] <jam> and then starting the container but not requesting an IP because we don't know where to put it
[16:36] <jam> cnf: do you have bindings specified for your applications?
[16:36] <jam> I'm out again
[16:36] <ybaumy> the export button does not work for me in juju gui
[16:37] <bdx> Budgie^Smore: sweet! yeah, so I just stash my openvpn on a igw subnet in my 'common-infrastructure' model in the same vpc - this is how I'm accessing my instances on the nat subnets
[16:37] <cnf> jam: yes, i used jamespage bundle
[16:37] <cnf> ok :(
[16:37] <Budgie^Smore> bdx yeah I will do that if I have to, I will be honest and say I prefer site-to-site when it comes to VPN
[16:37] <cnf> is http://termbin.com/023aq not right?
[16:39] <bdx> Budgie^Smore: not sure what your mileage is with pfsense .... you can create a custom ami
[16:39] <cnf> jamespage: can i get you to look at this?
[16:40] <cnf> rick_h: you still here?
[16:40] <Budgie^Smore> bdx for now I am just going to use one of those proxies and use sshuttle to tunnel the IP range of the VPC
[16:40] <rick_h> jam sorry thought you had gone for EOD.
[16:40] <bdx> Budgie^Smore: can you link me to your cloudformation yaml for that
[16:40] <rick_h> cnf: sorry, in and out with the schedule ATM.
[16:40] <cnf> ok :/
[16:41] <Budgie^Smore> bdx https://github.com/madeden/blogposts/blob/master/k8s-existing-env/src/cloudformation/NetworkLayout.json
[16:41] <Budgie^Smore> actually it is JSON but same diff
[16:41] <bdx> Budgie^Smore: sweet, thanks!
[16:41] <cnf> i'm close to throwing in the towel, btw jamespage, rick_h
[16:46] <cnf> hmz, how do i look at an application?
[16:46] <cnf> or a unit
[16:49] <Budgie^Smore> ok so almost complete with this model's initial deploy :)
[16:49] <admcleod> cnf: what do you mean, 'look at'?
[16:49] <cnf> admcleod: inspect? show what it wants?
[16:49] <cnf> status says  Incomplete relation: monitor
[16:50] <cnf> what does it expect? it not not defined? is it not working?
[16:51] <admcleod> cnf: wlel you can 'juju ssh unit/0' - oh, well in that case its waiting for one of the things its related to (defined in that bundle) to provide 'monitor'
[16:51] <cnf> admcleod: everything is waiting on evetything atm it seems
[16:51] <cnf> i just don't know why
[16:51] <cnf> or what the root problem is
[16:51] <admcleod> cnf: ceph-ods is waiting for ceph-mon
[16:51] <admcleod> cnf: it looks like all the containers are still 'pending'?
[16:51] <cnf> yes, great, and why isn't ceph-mon not coming up?
[16:52] <admcleod> s/ods/osd
[16:52] <cnf> yes, and they'll stay pending forever
[16:53] <cnf> but i don't know why
[16:54] <jamespage> cnf: sorry been otp
[16:54]  * jamespage reads backscroll
[16:55] <jamespage> cnf: ok so looks like you have
[16:55] <jamespage> a number of LXD machines that are marked as running but don't yet have a juju status
[16:56] <jamespage> https://www.irccloud.com/pastebin/gDuay0BK/
[16:56] <cnf> yes, all of them
[16:57] <jamespage> cnf: which is the cause of
[16:57] <jamespage> https://www.irccloud.com/pastebin/oagEPOlJ/
[16:58] <cnf> yeah
[16:58] <cnf> jamespage: so i think? the containers expect to be in the maas / juju network?
[16:58] <jamespage> jam, rick_h: this looks like issues between the jujud machine agents on the LXD containers and the controller machine
[16:58] <cnf> maybe?
[16:58] <jamespage> cnf: yeah pondering that
[16:58] <cnf> but if so, why the fuck isn't juju doing this?
[16:58] <cnf> and please do pardon the language
[16:58] <cnf> i'm quite frustrated at this point
[16:58]  * jamespage puts his hands on his ears
[16:59] <jamespage> cnf: I don't know but lets see if we can confirm this
[17:00] <jamespage> cnf: please can you "juju ssh 0"
[17:00] <jamespage> cnf:  and then sudo to root
[17:00] <jamespage> cnf: do a 'lxc list'
[17:00] <cnf> (as a side note, and something i'll address later, juju is assigning about 10 public addresses for this o,O)
[17:00] <jamespage> and then do a 'lxc exec <one of the container> bash'
[17:00] <jamespage> then we can peek into /var/log/juju
[17:00] <jamespage> or even /var/log/cloud-init.log
[17:01] <cnf> jamespage: /var/log/juju is empty
[17:01] <cnf> in the container
[17:01] <jamespage> cnf: ok so the issue is earlier in lifecycle
[17:01] <jamespage> cnf: how about /var/log/cloud-init*.log
[17:02] <cnf> bunch of tracebacks
[17:02] <cnf> tries to install stuff, but can't
[17:02] <jamespage> cnf: can you pastebin them please
[17:02] <cnf> makes sense, it has NO connectivity
[17:02] <jamespage> cnf: as in there are no network interfaces for the container
[17:03] <cnf> there is 1, but that can reach neither internet nor proxy
[17:03] <jamespage> cnf: or the juju controller I suspect
[17:04] <jamespage> cnf: that's my hunch at least
[17:04] <cnf> so i need to manually tell juju to add the maas/juju interface to each container?
[17:04] <jamespage> cnf: you should not have todo that
[17:04] <cnf> right
[17:04] <jamespage> cnf: 'ts why I pinged rich_h and jam
[17:04] <jamespage> cnf: which network space is the controller listening on?
[17:05] <cnf> space-maas
[17:05] <cnf> pasted screenshot of my maas networking in pm
[17:05] <jamespage> cnf: gotcah
[17:06] <Budgie^Smore> OK lazyPower I now I have my "new" k8s cluster up :)
[17:06] <cnf> everything in juju and maas has pushed me, quite aggressively, to put juju/maas mgmt on a separate fabric
[17:06] <lazyPower> Budgie^Smore: :tada:
[17:06] <jamespage> cnf: I'll ask you to trying something using the juju cli, and then we'll see if we can work that into the bundle
[17:06] <Budgie^Smore> lazyPower right!
[17:06] <jamespage> cnf: juju add-machine --to lxd:0 --constraints="spaces=space-maas"
[17:07] <Budgie^Smore> lazyPower did we find out what happend to juju-crashdump?
[17:07] <cnf> flag provided but not defined: --to
[17:07] <jamespage> huh
[17:07] <jamespage> lemme check that
[17:08] <jamespage> juju add-machine lxd:0 --constraints="spaces=space-maas"
[17:08] <jamespage> cnf:  ^^
[17:08] <cnf> created container 0/lxd/9
[17:09] <jamespage> cnf: ok but the important bit is that the juju status for the machine goes active
[17:09] <jamespage> not pending
[17:10] <jamespage> cnf: if this works, we'll need to add a "constraints: spaces=space-maas" to each of the services that needs to run in a LXD
[17:10] <jamespage> cnf: 100% agree that juju should just dtrt here and its not afaict
[17:11] <rick_h> jamespage: cnf so the containers need some way to phone home to the controller. If it can't reach it via some network then yes you'll need the management network available to the containers.
[17:12] <rick_h> jamespage: cnf we'd have to check with jam on what the containers do if they don't have a direct device on the juju controller network but it's supposed to be routable in some way to the controller through the network the container is on
[17:22] <kwmonroe> tvansteenburgh: it was totally a missing sso-token on my part.  sorry for the noise, tearDownClass is doing just fine :)
[17:22] <ThiagoCMC> hey guys, how to use "conjure-up openstack" to deploy Ocata? I'm seeing that Newton is the default...
[17:22] <jamespage> ThiagoCMC: ah its possible that the bundles for conjure-up have not been updated just yet - it lags the main charm/openstack release cycle a little
[17:22] <jamespage> stokachu: ^^ care to comment?
[17:22] <ThiagoCMC> Oh, okay...
[17:23] <cnf> rick_h: i can understand that
[17:23] <kwmonroe> cory_fu:  your layer-basic bits didn't break lint.  my cwr unit was dirty.  sorry for cursing at you so much this morning.  may, i can't seem to blame people as good as i used to.
[17:24] <cnf> rick_h: but i'd expect juju to know this :P
[17:24] <jam> jamespage: cnf: Juju doesn't know what your route tables look like. if you ask it to deploy a container that is only on 172.20.20.0/24 then that is what it does
[17:24] <jam> cnf: if you want it somewhere else, ask it to put it there
[17:25] <rick_h> cnf: yes, that get's back to jam's comment earlier about that.
[17:25] <jam> juju deploy app --bind foo --constraints spaces=other
[17:25] <jam> cnf: it is entirely plausible that you have a route from 172.20.30.0/24 to 172.20.20.1 or wherever you are hosting Juju Controller
 John Arbash Meinel rick_h: I'm really past EOD. my gut says we're failing to give an error when requesting a container that doesn't have a defined space on a machine with multiple network interfaces
[17:26] <jam> though I would expect the containers to get *an* IP, if they aren't getting anything at all, that is surprising
[17:27] <cnf> they get an ip
[17:27] <cnf> just not one that works to get to the juju controller
[17:27] <jam> cnf: so if one of the subnets is one that containers *have* to be on (to route to internet, etc)
[17:27] <jam> then add it
[17:27] <jam> juju deploy foo --bind bar
[17:27] <jam> tells us to put a container with access *only* to bar
[17:27] <stokachu> ThiagoCMC: i think the description says newton and that needs to be updated
[17:27] <cnf> hmm, so i have to manually bind _every_ container to the juju space
[17:27] <cnf> o,O
[17:27] <stokachu> ThiagoCMC: it should be whatever is in th elatest charm store
[17:27] <jam> cnf: yes
[17:27] <ThiagoCMC> stokachu, that's nice!
[17:28] <ThiagoCMC> I'm about to try it now
[17:28] <jam> cnf: cause if you're doing something like PIC it may be that you have a firewall between everytihng and the juju controllers
[17:28] <cnf> hmm
[17:28] <jam> cnf: and don't want your Apache software
[17:28] <stokachu> ThiagoCMC: ill update the descriptions now but they wont show up until i do another conjure-up release
[17:28] <jam> running on the same subnet as your Juju controller
[17:28] <jam> as your Postgres DB
[17:28] <hatch> Budgie^Smore did the vpc config option work for you?
[17:28] <Budgie^Smore> hatch yes it did :)
[17:29] <ThiagoCMC> stokachu, BTW, if the package "software-properties-common" is not installed, "sudo snap install conjure-up --classic" fails!
[17:29] <Budgie^Smore> hatch cluster is setup, just solving how I am going to ssh to the instances right now
[17:29] <cnf> jam: i don't want apache on the same subnet as juju, obviously :P
[17:29] <stokachu> ThiagoCMC: really?
[17:29] <ThiagoCMC> yep
[17:29] <stokachu> ThiagoCMC: AH!
[17:29] <jam> cnf: then create a router that can route between them
[17:29] <hatch> Budgie^Smore excellent, I've triaged adding that feature into the GUI release after the upcoming one.
[17:29] <stokachu> ThiagoCMC: nice find!!
[17:29] <cnf> uhm
[17:29] <stokachu> ill fix that now too
[17:29] <cnf> huh?
[17:29] <cnf> jam: you lost me
[17:29] <Budgie^Smore> hatch awesome :)
[17:30] <ThiagoCMC> =)
[17:30] <jam> cnf: the issue is that you ask us to only put the container on 172.x.y and we did, but you don't have a route in your network from 172.x.y to the Controller
[17:30] <Budgie^Smore> for some reason sshuttle isn't behaving today :-/
[17:30] <stokachu> ThiagoCMC: it's for the lxd ppa
[17:30] <stokachu> ThiagoCMC: totally missed that
[17:30] <ThiagoCMC> yep
[17:31] <cnf> hmm
[17:31] <ThiagoCMC> I always start things with Ubuntu ISO and bare-minimum setup, so I can catch those things.. ha!   =P
[17:31] <cnf> i'm not liking these containers much
[17:31] <jam> cnf: if you don't have routes, then you can ask us to also put the containers in other subnets additionally
[17:31] <jam> via the "--constraints spaces=" request
[17:32] <jam> cnf: and in bundles that looks like a "constraints" section on either the applications generally or the machines specifically
[17:32] <stokachu> ThiagoCMC: glad you did, thanks again
[17:32] <ThiagoCMC> stokachu, another problem, after "conjure-up 2.1.2 from 'canonical' installed", "conjure-up openstack" returns: "-bash: conjure-up: command not found"
[17:32] <ThiagoCMC> :-(
[17:32] <cnf> well, then i have to add the spaces=space-maas on _everything_
[17:32] <stokachu> ThiagoCMC: yea that one is hard to fix
[17:32] <ThiagoCMC> =/
[17:32] <stokachu> ThiagoCMC: you'll just need to exec $SHELL
[17:32] <stokachu> or logout of the terminal and back in
[17:32] <ThiagoCMC> ok
[17:33] <ThiagoCMC> damn... hehe... after logout and login: "$ conjure-up openstack" returns: "snap-confine has elevated permissions and is not confined but should be. Refusing to continue to avoid permission escalation attacks"
[17:34] <ThiagoCMC> any clue?
[17:34] <cnf> jam: so i need a constraint, but not a binding
[17:34] <jam> cnf: first, check if that works, second you could set up routes in whatever is actually managing those networks to allow 172.20* to talk to 172.10*
[17:34] <stokachu> ThiagoCMC: what does `snap list` show?
[17:34] <Budgie^Smore> and solved my sshultte issue :) so now I can ssh to the "private" instances for troubleshooting :)
[17:34] <jam> cnf: binding is where you want the application to talk, constraint is extra information about the machines you want
[17:35] <jam> cnf: the containers are likely set up to say "172.20.20.1 can you please send this packet to Juju controller at 172.20.10.1" and they're saying "sorry I can't do that"
[17:35] <ThiagoCMC> stokachu, "snap list" shows: "conjure-up  2.1.2 140 ... core        16-2     1441"
[17:35] <stokachu> ok that should be fine
[17:35] <stokachu> hmm
[17:35] <ThiagoCMC> =/
[17:35] <cnf> well, they are
[17:35] <stokachu> ThiagoCMC: asking in #snappy
[17:35] <jam> cnf: now, choosing to firewall in that fashion is something people can chose to do, but if they do that, then they need to give us some other way to talk to the controller
[17:35] <ThiagoCMC> ok
[17:35] <cnf> jam: my point was i expected juju to sort that out
[17:35] <cnf> or at least communicate it clearly
[17:36] <jam> cnf: the particular issue here, is that when the container comes up, it is like a separate machine to us, but because we can't reach the controller we can't tell it that we can't reach it
[17:37] <jam> cnf: consider if this were isolated machines (not containers)
[17:37] <stokachu> jamespage: i pull the readme from https://jujucharms.com/openstack-base/ but it still says newton is the latest
[17:37] <jam> if we ask maas to bring up a machine that can't talk to us, there isn't much we can do. The Controller couldn't 'ssh' in to find out why, cause we can't talk to it.
[17:38] <jam> cnf: there is some stuff that we could special case for containers, cause we can poke at them from the host machine, but it is pretty fiddly and isn't something that we've gotten to yet
[17:38] <cnf> jam: maybe something like "this is the juju network, always be in it"
[17:38] <jam> cnf: again, its perfectly reasonable to have *routers* rather than need to be in the network
[17:39] <jam> that's what gateways *do*
[17:39] <cnf> jam: yeah, i don't _want_ my storage network routed
[17:39] <jam> cnf: my home router knows how to get to 8.8.8.8, it doesn't live in 8.8.8.8 network
[17:39] <beisner> stokachu, i believe we have ocata stable charm store bundle in flight.  with this cycle being short, and the first offset cycle, we kept newton as current stable in that published bundle, pending further verification.
[17:39] <cnf> jam: yes, but your home network isn't a security domain
[17:39] <jam> cnf: so, absolutely a choice you can make. and one you're choosing. but you can set up only routes to the Juju controller network
[17:40] <jam> not to all possible public IPs
[17:40] <jam> for example
[17:40] <jam> or you can make your containers on the Juju network
[17:40] <jam> which likely *is* routable
[17:40] <jam> cnf: cause the Juju controller needs access usually to the outside world
[17:40] <cnf> ...
[17:40] <jam> cnf: so it depends what you mean by "I don't want my storage network to be routable"
[17:40] <stokachu> beisner: ok cool
[17:40] <jam> cnf: if every machine that is on the storage network is also on the externally-routable network, that sounds less secure
[17:41] <jam> cnf: than setting up just a route from 172.STORAGE to 172.JUJUCONTROLLER
[17:41] <cnf> it should be more clear on why shit isn't working, either way
[17:42] <cory_fu> kwmonroe: :)
[17:43] <jam> cnf: I absolutely hear you on that, and honestly, shit is hard to get right and hard to convey, and hard to probe and know what's going on.
[17:43] <stokachu> ThiagoCMC: re: openstack version, it is Newton, but Ocata is pending
[17:43] <jam> cnf: we are making improvements here, but we don't do everything everywhere yet
[17:44] <jam> anyway, need to go take my son to bed
[17:44] <jam> bbaiab
[17:45] <cnf> cannot deploy bundle: cannot deploy application "cinder-ceph": subordinate application must be deployed without constraints
[17:45] <cnf> ffs
[17:46] <cnf> ok, lets see how this behaves
[17:47] <cnf> hmm, juju doesn't add interfaces to existing containers, it seems
[17:48] <cnf> k, destroying, and deploying again
[17:48] <cnf> and then off home
[17:52] <admcleod> ThiagoCMC: did you want to deploy this on 'some machines' or just on a machine with lxd for testing/dev/playing around?
[17:57] <Budgie^Smore> lazyPower is there a reason why after an a fresh install there is 3 heapster replica sets?
[17:57] <ThiagoCMC> admcleod, at first, I want OpenStack in one box. So, yes, for testing... I want to go Juju only but, looks like that conjure-up is required to start this thing more easily...
[17:58] <ThiagoCMC> Later, I'll go ahead and try to deploy it in my MaaS environment...
[17:58] <ThiagoCMC> I already have MaaS up and running, with 20 bare-metal servers...
[17:58] <ThiagoCMC> But I need to fully understand Juju first.
[17:58] <admcleod> ThiagoCMC: well, if you wanna do it on one box with lxd and ocata, the openstack-on-lxd link will work for that
[17:58] <ThiagoCMC> And conjure-up "masks" it...
[17:59] <ThiagoCMC> admcleod, that's nice!
[17:59] <ThiagoCMC> do you have some link?
[17:59] <ThiagoCMC> with docs or quick and dirty guide?  =P
[17:59] <admcleod> ThiagoCMC: https://docs.openstack.org/developer/charm-guide/openstack-on-lxd.html
[18:00] <beisner> ocata is also WIP there, but there's a branch
[18:00] <ThiagoCMC> nice!
[18:00] <admcleod> ThiagoCMC: the doc refers to a git repo with bundles, the ocata bundle is in a branch called 'add-ocata' (for now, we will merge that soon)
[18:01] <admcleod> what beisner said
[18:01] <ThiagoCMC> I'm really interested in Ocata, Senlin, Aodh, Ceilometer and Gnocchi! I already have those things working but, I manually installed everything via "apt-get".
[18:01] <ThiagoCMC> Juju might be the way to go! I believe...
[18:02] <ThiagoCMC> I even have senlin-dashboard! https://launchpad.net/~sandvine/+archive/ubuntu/cloud-archive-ocata  :-D
[18:02] <ThiagoCMC> But it is too complicated to maintain a cloud with just a bunch of ansible playbooks and/or manual steps...
[18:05] <lazyPower> Budgie^Smore: that seems like a bug
[18:05] <lazyPower> Budgie^Smore: and i haven't seen that prior. can you paste me a kubectl get po,svc,rc --all-namespaces so i can see precisely what you're referring to?
[18:05] <Budgie^Smore> yeah I am filing a big against CDK right now and you guys can funnel it the right places :)
[18:06] <Budgie^Smore> oh and yes I can grab that out
[18:06] <lazyPower> ta
[18:06] <Budgie^Smore> lazyPower I will say it doesn look like it increases the number of pods just replica sets
[18:08] <lazyPower> Budgie^Smore: ok, was this a fresh install? as in no upgrade, just a bare juju deploy cdk and this was the result?
[18:08] <Budgie^Smore> lazyPower yes, the only changes I made were to constraint services to subnets and instance type
[18:12] <Budgie^Smore> lazyPower interesting it doesn't seem to show up in the cli output
[18:15] <Budgie^Smore> lazyPower: https://github.com/juju-solutions/bundle-canonical-kubernetes/issues/241
[18:16] <ThiagoCMC> Well, "conjure-up openstack", after staring the OpenStack deployment, failed again: https://paste.ubuntu.com/24236284/
[18:16] <ThiagoCMC> I gave up...   :-(
[18:16] <lazyPower> Budgie^Smore: so CLI doesn't list the additional RC's that is ee, but it showing up in the dashboard?
[18:17] <lazyPower> thats fun
[18:17] <Budgie^Smore> lazyPower yeah!
[18:17] <lazyPower> Budgie^Smore: can you kubectl describe those RC's that are ghosted?
[18:17] <lazyPower> kubectl describe rc heapster-v1.2.0.1-907310421  for example
[18:20] <Budgie^Smore> yeah definitely something to do with the dashboard and not kubectl!
[18:20] <lazyPower> Budgie^Smore: ok, i see the follow up there too
[18:20] <lazyPower> i wonder, if you refresh do they go away?
[18:21] <lazyPower> the dashboard isn't socket based, so you dont get real time updates, its entirely possible that you loaded the dashboard while operations were happening and they weren't fully cleaned up during that page load
[18:21] <Budgie^Smore> lazyPower nope
[18:21] <lazyPower> oh well that nukes that idea out of the water
[18:21] <lazyPower> thats a weird one
[18:21] <Budgie^Smore> even did a ctrl+f5 to make sure it was a fresh pull
[18:21] <lazyPower> if we can reproduce i'd say we need to file this against the dashboard project upstream, but barring that, you've found a fun one-off
[18:22] <lazyPower> Budgie^Smore: when i'm done validating the etcd work i have in the pipeline i'll context swap and try to reproduce
[18:22] <Budgie^Smore> lazyPower I vaguely remember seeing it in the last cluster... I am not sure if it is the dashboard project or the where the initial deployment happens
[18:23] <Budgie^Smore> lazyPower no rush, it is justa weird one, probably could be low priority as it is just an oddity (unless I am missing any risks to leaving those there)
[18:23] <lazyPower> Budgie^Smore: nah, if the api doesn't konw anything about it, you're proably fine
[18:23] <lazyPower> Budgie^Smore: just for fun, kubectl get rs doesn't show them either right?
[18:24] <lazyPower> replicaset vs replication controller
[18:25] <lazyPower> for clarity, here's the upstream definition of the difference
[18:25] <lazyPower> ReplicaSet is the next-generation Replication Controller. The only difference between a ReplicaSet and a Replication Controller right now is the selector support. ReplicaSet supports the new set-based selector requirements as described in the labels user guide whereas a Replication Controller only supports equality-based selector requirements.
[18:25] <Budgie^Smore> lazyPower damn it we found them... oh yeah I like ReplicaSet / Deployment model :)
[18:25] <lazyPower> ok, so i had suspected that might be the case too
[18:26] <lazyPower> describe them and lets see just dubya tee fig
[18:27] <Budgie^Smore> lazyPower, kubectl describe rs <rs name>?
[18:27] <lazyPower> yep
[18:29] <ThiagoCMC> stokachu,  "conjure-up openstack", after staring the OpenStack deployment, failed again: https://paste.ubuntu.com/24236284/
[18:30] <Budgie^Smore> lazyPower yeah now I feel like I am being visited by Schrodinger's cat here!
[18:30] <lazyPower> it both exists and doesnt exist at the same time eh?
[18:30] <Budgie^Smore> lazyPower pretty much!
[18:31] <lazyPower> heisenbug
[18:32] <Budgie^Smore> lazyPower not sure if it has reach heisenbug level though
[18:37] <lazyPower> Budgie^Smore: ok, i'll give this a reproduction attempt though
[18:38] <lazyPower> there has to be a reasonable explanation as to why there are replicasets and replication controllers in there that i dont see when testing
[18:38] <lazyPower> this might be related to our latest release, it might be some funkyness in the templates, it might also be a legit race bug where the addons are getting created and recreated
[18:39] <Budgie^Smore> lazyPower I suspect the later
[18:40] <kklimonda> can I disable juju unit/application for debugging, so it doesn't interfere with what I'm doing on the server?
[18:42] <ThiagoCMC> admcleod, the following instructions also doesn't work: https://docs.openstack.org/developer/charm-guide/openstack-on-lxd.html
[18:42] <ThiagoCMC> The "Juju Profile Update" is incomplete
[18:42] <cnf> wow
[18:42] <cnf> if i define constraints on my applications like jam said
[18:42] <ThiagoCMC> I can't find those "lxd-profile.yaml" and "config.yaml" files...
[18:42] <cnf> i get no hosts from MAAS
[18:43] <cnf> hmm
[18:51] <beisner> hi ThiagoCMC - i'
[18:51] <beisner> hi ThiagoCMC - i've run through openstack-on-lxd verbatim a number of times w/o issue.  at what point are you hitting the issue?
[18:52] <ThiagoCMC> beisner, at "Juju Profile Update", where that "lxd-profile.yaml" comes from?
[18:52] <ThiagoCMC> Also, on next step, I can't find that "juju config.yaml" either...
[18:53] <beisner> ThiagoCMC, did you complete the Host Setup section? https://docs.openstack.org/developer/charm-guide/openstack-on-lxd.html#host-setup   (ie. clone the repo)
[18:53] <ThiagoCMC> There it is!!!  lol
[18:54] <beisner> woot ThiagoCMC
[18:54] <ThiagoCMC> ^_^
[19:38] <bdx> does libjuju support `juju attach`?
[20:42] <marcoceppi> bdx: good question, not sure, if it doesn't it should
[21:52] <Budgie^Smore> lazyPower and I am back at getting AWS to behave :-/
[22:38] <kwmonroe> hey petevg, is this matrixy thing of concern to you? http://paste.ubuntu.com/24237456/
[22:38] <kwmonroe> petevg: specifically the "ha" bits.  we already talked about the "Could not run reboot" bits.
[22:39] <petevg> kwmonroe: yep. That's a bug, and a bad one, too. Please file a ticket.
[22:39] <kwmonroe> petevg: i'm asking you first because i'd love to blame cory_fu (https://github.com/juju-solutions/matrix/commit/7329196f6706e84c29b4cffa6cd64feb7c19f55e), but he already got the better of me when i tried to blame a layer-basic thng on his this mroning.
[22:39] <kwmonroe> *morning even
[22:41] <petevg> kwmonroe: no, that's my PR. That should be context.config.ha. Not sure why the tests didn't catch it :-/
[22:41] <kwmonroe> let's still blame cory.
[22:42] <petevg> ... it might just be a missing case. Bleh. I'll fix it asap, in any case.
[22:43] <kwmonroe> petevg: mucho thanko: https://github.com/juju-solutions/matrix/issues/108
[22:43] <petevg> This is my bad. He didn't catch it, though :-)
[22:52] <petevg> kwmonroe: fix here https://github.com/juju-solutions/matrix/pull/109
[22:52] <petevg> kwmonroe: it was an entertaining bug. The unit tests didn't catch it, because I was mocking out the context (unit tests don't test your assumptions!)
[22:52] <petevg> ... and the functional tests didn't catch it, because it causes a failure only when the test has failed.
[22:53] <petevg> So my "make sure this fails" tests were failing as expected.
[22:53] <petevg> I should make the functional tests check for failure reasons at some point. For now, the above PR should fix my bad.