[10:36] <Odd_Bloke> Hello all; I'm trying to add servicegroup support to the ubuntu-repository-charm, using the latest charm-helpers.
[10:37] <Odd_Bloke> I've added the config option and set it (for a new deployment), but none of the services show up in Nagios as having a servicegroup.
[10:40] <Odd_Bloke> Is there some configuration I need to do in Nagios?
[10:41] <Odd_Bloke> Here are some (potentially) relevant bits of config on the hosts: http://paste.ubuntu.com/11689441/
[10:49] <jamespage> lukasa, around? I have a query re use of etcd on the neutron-api charm charms for Calico
[10:51] <lukasa> jamespage: Howdy
[10:51] <lukasa> Shoot
[10:52] <jamespage> lukasa, just working through the current set of proposed changes
[10:52] <jamespage> lukasa, and I note the introduction of etcd into the neutron-api charm
[10:52] <lukasa> Yup
[10:52] <jamespage> lukasa, is that used by Calico to maintain state?
[10:52] <lukasa> It is indeed
[10:53] <jamespage> I'm trying to understand how it needs to scale - clouds may need to scale the neutron-api service quite big
[10:53] <jamespage> does that make sense for the etcd bits?
[10:53] <lukasa> Yes. =)
[10:53] <lukasa> So, etcd will scale separately from the neutron-api side
[10:54] <jamespage> lukasa, right now it will match the number of neutron-api units
[10:54] <lukasa> Ohhh, I see
[10:54] <lukasa> No, it won't, not quite. =)
[10:54] <lukasa> neutron-api installs etcd on each node, but configures that etcd instance as a proxy
[10:54] <jamespage> lukasa, we also have quite a strong etcd charm coming out of the cloudfoundry work
[10:54] <jamespage> https://jujucharms.com/u/kubernetes/etcd/trusty/2
[10:54] <lukasa> jamespage: Yeah, I'm aware, I have a roadmap item to migrate to that. =D
[10:55] <lukasa> But the etcd proxy instances don't count towards the cluster size
[10:55] <lukasa> They're basically just fancy HTTP proxies
[10:56] <jamespage> lukasa, what are they proxying to?
[10:56] <lukasa> The core etcd install, which is covered by the etcd relation I added
[10:57] <jamespage> lukasa, ah - sorry the use or 'peer' in the interface name confused me for a moment there
[10:57] <lukasa> Yeah, sorry
[10:57] <lukasa> From an etcd configuration perspective they count as peers
[10:57] <lukasa> But they don't participate in raft
[10:58] <jamespage> lukasa, so these don't fit into the relations provided by https://api.jujucharms.com/charmstore/v4/~kubernetes/trusty/etcd-2/archive/metadata.yaml ?
[10:59] <jamespage> i.e. its not etcd (client interface)
[10:59] <lukasa> I'm not sure yet
[10:59] <lukasa> I've had a bit of a chat with Whit and Charles, but I haven't sat down and examined that charm in depth yet
[11:00] <lukasa> And that charm may need an enhancement to work for us (though it may not, they're playing with it for calico-docker)
[11:00] <jamespage> lukasa, ack - OK - I understand enough to review for now
[11:00] <lukasa> Awesome. =) Feel free to ask more questions whenever if you need
[11:12] <jamespage> lukasa, I'm making the assumption I'm going to see etcd proxy use in the neutron-calico charm as well right?
[11:13] <lukasa> jamespage: Correct =)
[11:13] <jamespage> lukasa, do you have a charm-helpers mp pending as well?
[11:13] <lukasa> Already merged, I think
[11:14] <jamespage> lukasa, checking now
[11:14] <jamespage> lukasa, this one - https://code.launchpad.net/~cory-benfield/charm-helpers/trunk/+merge/257260
[11:16] <jamespage> lukasa, fyi gnuoy and I where talking about plugins/neutron-api yesterday a bit; we're going to come up with a suboridnate approach which makes the code separation in the charms between core neutron-api and plugin a bit clearer
[11:16] <lukasa> jamespage: That's the one
[11:16] <jamespage> lukasa, you don;'t need to rework anything, but we'll probably do a tech-debt run through at some point in the future to move all plugins to that model
[11:16] <lukasa> jamespage: Sure thing
[11:16] <lukasa> I'll be around, drop me an email when you do and I'll do my best to help out
[11:16] <jamespage> lukasa, thanks
[11:17] <jamespage> lukasa, well come up with a base template and stuff as well
[11:17] <jamespage> make the whole process easier for all to consume
[11:17] <lukasa> \o/
[11:34] <jamespage> lukasa, do you have a nice bundle to deploy this?
[11:36] <jamespage> lukasa, looking at the peer relation on the neutron-calico charm - if related to bird, is the data passed on that relation superceeded
[11:36] <jamespage> lukasa, peer relations get noisy at scale
[11:37] <jamespage> lukasa, read the code and answered my own question
[11:42] <jamespage> lukasa, sent you a MP for a slight nicer way of parsing calico-origin in the install hook
[11:43] <jamespage> lukasa, I think you could also answer the feedback on the NEW charm bug by making install and config-changed to exactly the same thing
[11:43] <jamespage> or maybe no-opping install althogether
[11:43] <jamespage> the service bringup will do install -> config-changed anyway
[11:45] <jamespage> lukasa, remind me again about that bird bug in trusty?
[11:46] <jamespage> lukasa, shutting up - funny how you raised it and everything :-)
[11:46] <jamespage> https://bugs.launchpad.net/ubuntu/precise/+source/bird/+bug/1342173
[11:46] <mup> Bug #1342173: BIRD 1.4.0-1 fails to start on Ubuntu 14.04 <trusty> <bird (Ubuntu):Fix Released> <bird (Ubuntu Precise):Confirmed> <bird (Ubuntu Trusty):Confirmed> <bird (Ubuntu Utopic):Fix Released> <https://launchpad.net/bugs/1342173>
[12:04] <jamespage> lukasa, I'll get that into the SRU queue today - however its probably OK to use the upstream PPA as well, as the maintainer is the same person as in debian
[12:14] <jamespage> lukasa, fixes in the queue - that bug report will be updated once accepted
[12:32] <lukasa> jamespage: Sorry, was at lunch =)
[12:38] <lazyPower> jamespage: we have an etcd charm that supports large scale already :)
[12:39] <lazyPower> https://jujucharms.com/u/kubernetes/etcd/trusty/2 - i'd be interested in any feedbak you have here.
[12:40] <lazyPower> looks like we need to cut a new release however - https://github.com/whitmo/etcd-charm  v0.0.3 is the latest, and supports bintar deployments + private network bootstrap (vs using etcd to bootstrap etcd)
[13:06] <jamespage> lazyPower, indeed we do - can we adapt it for lukasa's requirements for etcd proxies on local units, related to a clustered, core etcd deployment.
[13:06] <lazyPower> jamespage: im thinking so. I have already started talking to lukasa and crew about it :)
[13:06] <lukasa> \o/
[13:06] <lazyPower> we have pending work to address first, but that was next
[13:06] <jamespage> lazyPower, so i gather
[13:07] <jamespage> lazyPower, awesome - I was having a general run through the charmset inc the openstack charms today
[13:07] <lazyPower> jamespage: i'll try to refrain from jumpin in a convo before i've had my coffee moving forward, i realized about 10 minutes later, there was a whole heap more to that conversation.
[13:07] <jamespage> lazyPower, its all looking pretty close - if we could close on the etcd stuff, then there are only niggles to land
[13:08] <lazyPower> lukasa: what were we pending? just adding the client relation bits and running a test right?
[13:11] <lukasa> lazyPower: Yeah, and me finding time to do that
[13:11] <lukasa> Currently swamped coordinating about 5 or 6 different things at once. ;)
[13:11] <lazyPower> no pressure :D
[13:11] <lukasa> Sometime this week or early next I hope
[13:12] <lazyPower> ack, if you want some support from our side feel free to tap me in lukasa
[13:13] <lazyPower> i'm pretty fmiliar with what our etcd charm is doing since i ran with the upgrades to 0.2.x
[13:14] <lukasa> Yeah, will do. =) I just blew up an attempted deployment to Kilo, but I think I just got the config wrong ;)
[13:14] <lukasa> So I'll see how this goes
[13:21] <Odd_Bloke> Can anyone help me with a Nagios servicegroups question: http://paste.ubuntu.com/11690101/ ?
[13:21] <Odd_Bloke> (Trying again now East Coast people might be around :)
[14:34] <jamespage> beisner, https://code.launchpad.net/~james-page/ubuntu-openstack-ci/neutron-gateway-rename/+merge/261630
[15:31] <mbruzek> marcoceppi: did you have an image generator for bundles?
[15:31] <marcoceppi> mbruzek: svg.juju.solutions
[15:52] <Odd_Bloke> Anyone around to give me pointers on a nagios_servicegroups problem I'm seeing?
[15:52] <Odd_Bloke> See http://paste.ubuntu.com/11690101/ for details.
[16:01] <beisner> jamespage, thanks, merged.  i'll follow up on that by updating amulet tests as i think they may call the old name as well.
[17:14] <beisner> jamespage, gnuoy - also, i'll work up an MP to update neutron name in the mojo specs.
[17:33] <cholcombe> lazyPower, so leader election in juju.  I get that by calling is-leader right?
[17:34] <lazyPower> cholcombe: corret, thats a predicate to determine who is the leader of the service group
[17:34] <cholcombe> are there docs on that?  i'm having trouble finding them
[17:34] <lazyPower> iirc they only exist in the devel doc release notes
[17:34] <cholcombe> ok
[17:34] <lazyPower> it hasnt been officially docuemented/included - and its subject to being reworked.
[17:34] <lazyPower> last i heard they were going to rework the leader-election bits sometime this or next cycle.
[17:34] <cholcombe> i'm on 1.23.3 juju
[17:35] <cholcombe> i'm going to give it a try and see what it returns
[17:35] <Bialogs> Does anyone here have experience with the Cinder charm? I'm getting the following error: losetup: Could not find any loop device. Maybe this kernel does not know about the loop device? (If so, recompile or `modprobe loop'.)
[17:35] <lazyPower> cholcombe: https://jujucharms.com/docs/stable/reference-release-notes
[17:35] <lazyPower> search "is-leader"
[17:35] <cholcombe> sweet
[17:35] <cholcombe> yeah i saw it
[17:36] <cholcombe> i see we have new hooks also. i need to take this into account
[17:36] <lazyPower> Bialogs: that sounds somewhat familar... did you provide the cinder charm a block device?
[17:37] <lazyPower> Bialogs: i beleive thats also covered in the charm readme if you havent taken a look yet - https://jujucharms.com/cinder/trusty/23
[17:38] <Bialogs> Yes I did but I'm not certain that I did that correctly, in my config file: block-device: /var/lib/cinder-sdb.img|10G
[17:38] <lazyPower> beisner: tap tap :)
[17:39] <beisner> lazyPower, who's there?
[17:39] <lazyPower> beisner: any known issues with cinder and loopback storage? the config provided by Bialogs looks correct to me according to the config option.
[17:39] <Bialogs> lazyPower: I should also mention that I'm trying to do this in LXC
[17:39] <lazyPower> oooooooooooooo
[17:40] <lazyPower> i bet its an apparmor issue
[17:40] <beisner> lazyPower, loopback storage is a testing/experimental feature;  i'd add that i've not tried to use it inside lxc.
[17:40] <lazyPower> but we'll see what the wizard has to say
[17:40] <lazyPower> beisner is my lifeline into the openstack wizards circle :)
[17:40] <Bialogs> lazyPower: On the host I can't see the kernel module either though
[17:40] <lazyPower> i'm willing to bet thats the issue
[17:40] <lazyPower> apparmor is preventing you from loading the loopback kernel module
[17:41] <lazyPower> i've run into so many weird edge cases with apparmor being very strict about what you're allowed to do in lxc
[17:41] <cholcombe> lazyPower: is-leader rocks :)
[17:41] <Bialogs> modprobe loop doesn't return anything either
[17:41] <Bialogs> apparmor will prevent that on the host machine too?
[17:41] <lazyPower> not on the host, no
[17:41] <lazyPower> only in the lxc client iirc
[17:42] <Bialogs> Ya so that's strange
[17:42] <lazyPower> you have to whitelist modules in apparmor to get them to load in lxc
[17:42] <lazyPower> if thats the issue that is
[17:42] <lazyPower> mind you i'm making a guess at this, without having evidence thats the problem
[17:44] <Bialogs> I'm going to try and get it loaded on the host first
[17:44] <beisner> lazyPower, sounds reasonable to at least look at apparmor.
[17:44] <ennoble> hi, I'm trying to manually provision a node and I get an error, "ERROR error checking if provisioned: subprocess encountered error code 1" is there any debugging I can turn on to further diagnose the problem?
[17:44] <Bialogs> Assuming I don't get this working what's the alternative to loopback Cinder
[17:45] <beisner> Bialogs, please do let us know if you see issues with that outside a container.
[17:45] <lazyPower> ennoble: can you do 'juju retry-provisioning' with --verbose?
[17:45] <lazyPower> er -v
[17:48] <ennoble> lazyPower: when I try that with the machine name I get "error invalid machine", without it "error: no machine specified." The machine isn't listed in juju status in any state after the failure
[17:48] <ennoble> lazyPower: is there a way to get verbose output from add-machine? -v doesn't seem to do anything
[17:51] <lazyPower> ennoble: you can tail the machine log on your bootstrap node, but that wont give you much output during the provisioning process unfortunately
[17:51] <lazyPower> i'm trying to think if there's a debug method on that to spit out more data like bootstrap --debug -v
[17:51] <mgz_> lazyPower: `juju help logging`
[17:52] <mgz_> turn up the verbosity, then look at the machine log
[17:52] <lazyPower> oh yeah i sent a Pr to the docs about this
[17:52] <lazyPower> mgz_: *hattip*
[17:54] <Destreyf> marcoceppi: Did you have any issues with your Hyper Visor not showing up in the stack?
[17:57] <ennoble> mgz_: I tried that, the help, isn't quite right in 1.23.3 (looks like you need juju set-environment logging-config="juju=TRACE;unit=TRACE"
[17:57] <ennoble> lazyPower: funny enough it worked after setting logging even though I've tried it many times previously... not sure how to debug that more
[17:57] <lazyPower> ennoble: that is indeed odd
[17:58] <lazyPower> perhaps a transient networking error?
[17:58] <ennoble> lazyPower: Possibly, although i've had an ssh session open to the machine the whole time as well.
[17:59] <ennoble> lazyPower: thanks for the input. If it happens again maybe I'll see a pattern
[17:59] <lazyPower> np ennoble, cheers :)
[18:02] <cholcombe> is is-leader does some strange things :).  when my leader-elected hook was called the normal juju context stuff that I gather didn't work
[18:03] <lazyPower> cholcombe: might be worth following up in #juju-dev to see why certain things arent available in leader context
[18:03] <cholcombe> ok
[18:03] <cholcombe> is that on freenode also?
[18:03] <lazyPower> yep
[18:05] <ennoble> lazyPower: Actually my bad, I was wrong, I spelled the node name incorrectly and the machine created didn't exist when I retried. I'm back to the issue.
[18:07] <ennoble> lazyPower: I set logging-config to juju=TRACE;unit=TRACE where is the logging done? I don't see anything on the orchestration node
[18:09] <lazyPower> with the logging set to trace, it will show up on your workstation when you run 'juju debug-log'
[18:09] <lazyPower> but its going to be a firehose of info, so be prepared
[18:12] <ennoble> lazyPower: while I see pings and such from other instances, nothing about the machine I'm trying to add seems to show up
[18:15] <lazyPower> thats weird
[18:15] <lazyPower> you should at the very least be getting something from the action, and anything post cloud-init
[18:20] <Bialogs> lazyPower: Hey, I have some more information. The hook that Juju is running executes a losetup -f command which fails because it can't find the loop module. When I modprobe from the container I get the following error: modprobe: ERROR: ../libkmod/libkmod.c:556 kmod_search_moddep() could not open moddep file '/lib/modules/3.13.0-53-generic/modules.dep.bin
[18:21] <Bialogs> lazyPower: does that point more to app armor?
[18:22] <lazyPower> Bialogs: does the command work after you sudo depmod?
[18:24] <bbaqar> Has anyone done a complete kilo openstack HA deployment?
[18:25] <Bialogs> lazyPower: I did a depmod on the host before I ran this command -- any difference in LXC?
[18:26] <lazyPower> Bialogs: if it made no difference, i would point at apparmor, if it works its ounds like the module.bin was missing the module you needed
[18:26] <lazyPower> you should be getting an error in syslog (i think) about the apparmor denial
[18:26] <lazyPower> if its apparmor
[18:26] <Bialogs> lazyPower: lol depmod fails in the container
[18:26] <lazyPower> ok, yeah
[18:27] <lazyPower> its more than likely the apparmor profile for lxc then, which is unfortunate.
[18:27] <lazyPower> that gets fiddly
[18:28] <Bialogs> The app armor profile is on the host machine, correct?
[18:28] <lazyPower> correct
[18:38] <cholcombe> lazyPower: i think my leader stuff works now :)
[18:38] <Bialogs> lazyPower: don't really see anything that says denied, just a lot of status messages
[18:38] <cholcombe> i'm not seeing a crap ton of duplicate unit messages now
[18:38] <lazyPower> cholcombe: awesome!
[18:38] <cholcombe> :)
[18:38] <lazyPower> Bialogs: :(
[18:38] <Bialogs> Every smile has a frown
[18:38] <lazyPower> Bialogs: thats the pitts, i'm not sure what to recommend if its not giving you a big red shiny signal
[18:39] <lazyPower> Bialogs: what i can however s uggest is emailing the juju list with the issue
[18:39] <lazyPower> someone thats not around now (primarily the UK openstack team) should be able to lend a hand with whats going wrong if theyve encountered this before, otherwise they will point you to file a bug
[18:40] <lazyPower> Bialogs: juju@lists.ubuntu.com - or alternatively you can open a bug first and ping the list with the bug, which may be the better route to go, to raise visibility and get some bug-heat on it.
[18:41] <Bialogs> I'm going to see if I can poke around the app armor config first
[18:41] <Bialogs> Then I'll email the list or post the bug
[18:41] <Bialogs> Thanks!
[18:47] <lazyPower> sorry i wasnt more help Bialogs :( good luck
[19:34] <Bialogs> So this is strange
[19:35] <Bialogs> Tried it again and It's not failing at that part, just can't communicate with RabbitMQ
[19:48] <marcoceppi> mbruzek: sweet blog post!
[19:48] <mbruzek> thanks marcoceppi
[19:49] <mbruzek> http://bruzer.net/2015/06/10/deploy-a-kubernetes-development-cluster-with-juju/
[20:01] <tvansteenburgh> great write-up mbruzek, just read it myself
[20:02] <mbruzek> thanks tim
[20:02] <tvansteenburgh> nice work on kubes lazyPower, mbruzek, and whit; cool stuff!
[20:26] <kwmonroe> cory_fu: do you recall if unitdata is available in charmhelpers render_template?  and if so, does this feel right in a .j2 template? hostname = {{ unitdata.kv['relations.ready']['my-charm'].values()[0]['private-address'] }}
[20:33] <cory_fu> kwmonroe: Do you mean in the bigdata branch, or in trunk?
[20:35] <kwmonroe> either cory_fu, but i got it.. i'm using any_ready_unit instead of relations.ready now.  please disregard :)
[20:35] <cory_fu> kwmonroe: It is available in the templates in the big data branch, yes, but I'd recommend you use {{ any_ready_unit('my-charm')['private-address'] }} instead
[20:35] <kwmonroe> duh
[20:35] <cory_fu> :)
[20:44] <lazyPower> thanks tvansteenburgh :)
[21:46] <Bialogs> lazyPower: I submitted that bug - ended up getting another error eventually...Still super confused. Thanks for your help
[22:59] <Destreyf> guys, i'm running into a weird issue
[22:59] <Destreyf> when i deploy a machine using the manual method
[22:59] <Destreyf> my 2nd machine never starts up the virbr0
[23:00] <Destreyf> and then i get connction timeouts