=== kadams54 is now known as kadams54-away === natefinch-afk is now known as natefinch === brandon is now known as Guest42275 === kadams54 is now known as kadams54-away === kadams54-away is now known as kadams54 === kadams54 is now known as kadams54-away === scuttlemonkey is now known as scuttle|afk === zz_CyberJacob is now known as CyberJacob === CyberJacob is now known as zz_CyberJacob === zz_CyberJacob is now known as CyberJacob [10:36] Hello all; I'm trying to add servicegroup support to the ubuntu-repository-charm, using the latest charm-helpers. [10:37] I've added the config option and set it (for a new deployment), but none of the services show up in Nagios as having a servicegroup. [10:40] Is there some configuration I need to do in Nagios? [10:41] Here are some (potentially) relevant bits of config on the hosts: http://paste.ubuntu.com/11689441/ [10:49] lukasa, around? I have a query re use of etcd on the neutron-api charm charms for Calico [10:51] jamespage: Howdy [10:51] Shoot [10:52] lukasa, just working through the current set of proposed changes [10:52] lukasa, and I note the introduction of etcd into the neutron-api charm [10:52] Yup [10:52] lukasa, is that used by Calico to maintain state? [10:52] It is indeed [10:53] I'm trying to understand how it needs to scale - clouds may need to scale the neutron-api service quite big [10:53] does that make sense for the etcd bits? [10:53] Yes. =) [10:53] So, etcd will scale separately from the neutron-api side [10:54] lukasa, right now it will match the number of neutron-api units [10:54] Ohhh, I see [10:54] No, it won't, not quite. =) [10:54] neutron-api installs etcd on each node, but configures that etcd instance as a proxy [10:54] lukasa, we also have quite a strong etcd charm coming out of the cloudfoundry work [10:54] https://jujucharms.com/u/kubernetes/etcd/trusty/2 [10:54] jamespage: Yeah, I'm aware, I have a roadmap item to migrate to that. =D [10:55] But the etcd proxy instances don't count towards the cluster size [10:55] They're basically just fancy HTTP proxies [10:56] lukasa, what are they proxying to? [10:56] The core etcd install, which is covered by the etcd relation I added [10:57] lukasa, ah - sorry the use or 'peer' in the interface name confused me for a moment there [10:57] Yeah, sorry [10:57] From an etcd configuration perspective they count as peers [10:57] But they don't participate in raft [10:58] lukasa, so these don't fit into the relations provided by https://api.jujucharms.com/charmstore/v4/~kubernetes/trusty/etcd-2/archive/metadata.yaml ? [10:59] i.e. its not etcd (client interface) [10:59] I'm not sure yet [10:59] I've had a bit of a chat with Whit and Charles, but I haven't sat down and examined that charm in depth yet [11:00] And that charm may need an enhancement to work for us (though it may not, they're playing with it for calico-docker) [11:00] lukasa, ack - OK - I understand enough to review for now [11:00] Awesome. =) Feel free to ask more questions whenever if you need [11:12] lukasa, I'm making the assumption I'm going to see etcd proxy use in the neutron-calico charm as well right? [11:13] jamespage: Correct =) [11:13] lukasa, do you have a charm-helpers mp pending as well? [11:13] Already merged, I think [11:14] lukasa, checking now [11:14] lukasa, this one - https://code.launchpad.net/~cory-benfield/charm-helpers/trunk/+merge/257260 [11:16] lukasa, fyi gnuoy and I where talking about plugins/neutron-api yesterday a bit; we're going to come up with a suboridnate approach which makes the code separation in the charms between core neutron-api and plugin a bit clearer [11:16] jamespage: That's the one [11:16] lukasa, you don;'t need to rework anything, but we'll probably do a tech-debt run through at some point in the future to move all plugins to that model [11:16] jamespage: Sure thing [11:16] I'll be around, drop me an email when you do and I'll do my best to help out [11:16] lukasa, thanks [11:17] lukasa, well come up with a base template and stuff as well [11:17] make the whole process easier for all to consume [11:17] \o/ [11:34] lukasa, do you have a nice bundle to deploy this? [11:36] lukasa, looking at the peer relation on the neutron-calico charm - if related to bird, is the data passed on that relation superceeded [11:36] lukasa, peer relations get noisy at scale [11:37] lukasa, read the code and answered my own question [11:42] lukasa, sent you a MP for a slight nicer way of parsing calico-origin in the install hook [11:43] lukasa, I think you could also answer the feedback on the NEW charm bug by making install and config-changed to exactly the same thing [11:43] or maybe no-opping install althogether [11:43] the service bringup will do install -> config-changed anyway [11:45] lukasa, remind me again about that bird bug in trusty? [11:46] lukasa, shutting up - funny how you raised it and everything :-) [11:46] https://bugs.launchpad.net/ubuntu/precise/+source/bird/+bug/1342173 [11:46] Bug #1342173: BIRD 1.4.0-1 fails to start on Ubuntu 14.04 [12:04] lukasa, I'll get that into the SRU queue today - however its probably OK to use the upstream PPA as well, as the maintainer is the same person as in debian [12:14] lukasa, fixes in the queue - that bug report will be updated once accepted [12:32] jamespage: Sorry, was at lunch =) [12:38] jamespage: we have an etcd charm that supports large scale already :) [12:39] https://jujucharms.com/u/kubernetes/etcd/trusty/2 - i'd be interested in any feedbak you have here. [12:40] looks like we need to cut a new release however - https://github.com/whitmo/etcd-charm v0.0.3 is the latest, and supports bintar deployments + private network bootstrap (vs using etcd to bootstrap etcd) === scuttle|afk is now known as scuttlemonkey [13:06] lazyPower, indeed we do - can we adapt it for lukasa's requirements for etcd proxies on local units, related to a clustered, core etcd deployment. [13:06] jamespage: im thinking so. I have already started talking to lukasa and crew about it :) [13:06] \o/ [13:06] we have pending work to address first, but that was next [13:06] lazyPower, so i gather [13:07] lazyPower, awesome - I was having a general run through the charmset inc the openstack charms today [13:07] jamespage: i'll try to refrain from jumpin in a convo before i've had my coffee moving forward, i realized about 10 minutes later, there was a whole heap more to that conversation. [13:07] lazyPower, its all looking pretty close - if we could close on the etcd stuff, then there are only niggles to land [13:08] lukasa: what were we pending? just adding the client relation bits and running a test right? [13:11] lazyPower: Yeah, and me finding time to do that [13:11] Currently swamped coordinating about 5 or 6 different things at once. ;) [13:11] no pressure :D [13:11] Sometime this week or early next I hope [13:12] ack, if you want some support from our side feel free to tap me in lukasa [13:13] i'm pretty fmiliar with what our etcd charm is doing since i ran with the upgrades to 0.2.x [13:14] Yeah, will do. =) I just blew up an attempted deployment to Kilo, but I think I just got the config wrong ;) [13:14] So I'll see how this goes [13:21] Can anyone help me with a Nagios servicegroups question: http://paste.ubuntu.com/11690101/ ? [13:21] (Trying again now East Coast people might be around :) === cmagina_ is now known as cmagina [14:34] beisner, https://code.launchpad.net/~james-page/ubuntu-openstack-ci/neutron-gateway-rename/+merge/261630 [15:31] marcoceppi: did you have an image generator for bundles? [15:31] mbruzek: svg.juju.solutions [15:52] Anyone around to give me pointers on a nagios_servicegroups problem I'm seeing? [15:52] See http://paste.ubuntu.com/11690101/ for details. [16:01] jamespage, thanks, merged. i'll follow up on that by updating amulet tests as i think they may call the old name as well. === ejat is now known as ejat- [17:14] jamespage, gnuoy - also, i'll work up an MP to update neutron name in the mojo specs. [17:33] lazyPower, so leader election in juju. I get that by calling is-leader right? [17:34] cholcombe: corret, thats a predicate to determine who is the leader of the service group [17:34] are there docs on that? i'm having trouble finding them [17:34] iirc they only exist in the devel doc release notes [17:34] ok [17:34] it hasnt been officially docuemented/included - and its subject to being reworked. [17:34] last i heard they were going to rework the leader-election bits sometime this or next cycle. [17:34] i'm on 1.23.3 juju [17:35] i'm going to give it a try and see what it returns [17:35] Does anyone here have experience with the Cinder charm? I'm getting the following error: losetup: Could not find any loop device. Maybe this kernel does not know about the loop device? (If so, recompile or `modprobe loop'.) [17:35] cholcombe: https://jujucharms.com/docs/stable/reference-release-notes [17:35] search "is-leader" [17:35] sweet [17:35] yeah i saw it [17:36] i see we have new hooks also. i need to take this into account [17:36] Bialogs: that sounds somewhat familar... did you provide the cinder charm a block device? [17:37] Bialogs: i beleive thats also covered in the charm readme if you havent taken a look yet - https://jujucharms.com/cinder/trusty/23 [17:38] Yes I did but I'm not certain that I did that correctly, in my config file: block-device: /var/lib/cinder-sdb.img|10G [17:38] beisner: tap tap :) [17:39] lazyPower, who's there? [17:39] beisner: any known issues with cinder and loopback storage? the config provided by Bialogs looks correct to me according to the config option. [17:39] lazyPower: I should also mention that I'm trying to do this in LXC [17:39] oooooooooooooo [17:40] i bet its an apparmor issue [17:40] lazyPower, loopback storage is a testing/experimental feature; i'd add that i've not tried to use it inside lxc. [17:40] but we'll see what the wizard has to say [17:40] beisner is my lifeline into the openstack wizards circle :) [17:40] lazyPower: On the host I can't see the kernel module either though [17:40] i'm willing to bet thats the issue [17:40] apparmor is preventing you from loading the loopback kernel module [17:41] i've run into so many weird edge cases with apparmor being very strict about what you're allowed to do in lxc [17:41] lazyPower: is-leader rocks :) [17:41] modprobe loop doesn't return anything either [17:41] apparmor will prevent that on the host machine too? [17:41] not on the host, no [17:41] only in the lxc client iirc [17:42] Ya so that's strange [17:42] you have to whitelist modules in apparmor to get them to load in lxc [17:42] if thats the issue that is [17:42] mind you i'm making a guess at this, without having evidence thats the problem [17:44] I'm going to try and get it loaded on the host first [17:44] lazyPower, sounds reasonable to at least look at apparmor. [17:44] hi, I'm trying to manually provision a node and I get an error, "ERROR error checking if provisioned: subprocess encountered error code 1" is there any debugging I can turn on to further diagnose the problem? [17:44] Assuming I don't get this working what's the alternative to loopback Cinder [17:45] Bialogs, please do let us know if you see issues with that outside a container. [17:45] ennoble: can you do 'juju retry-provisioning' with --verbose? [17:45] er -v [17:48] lazyPower: when I try that with the machine name I get "error invalid machine", without it "error: no machine specified." The machine isn't listed in juju status in any state after the failure [17:48] lazyPower: is there a way to get verbose output from add-machine? -v doesn't seem to do anything [17:51] ennoble: you can tail the machine log on your bootstrap node, but that wont give you much output during the provisioning process unfortunately [17:51] i'm trying to think if there's a debug method on that to spit out more data like bootstrap --debug -v [17:51] lazyPower: `juju help logging` [17:52] turn up the verbosity, then look at the machine log [17:52] oh yeah i sent a Pr to the docs about this [17:52] mgz_: *hattip* === kadams54_ is now known as kadams54-away [17:54] marcoceppi: Did you have any issues with your Hyper Visor not showing up in the stack? [17:57] mgz_: I tried that, the help, isn't quite right in 1.23.3 (looks like you need juju set-environment logging-config="juju=TRACE;unit=TRACE" [17:57] lazyPower: funny enough it worked after setting logging even though I've tried it many times previously... not sure how to debug that more [17:57] ennoble: that is indeed odd [17:58] perhaps a transient networking error? [17:58] lazyPower: Possibly, although i've had an ssh session open to the machine the whole time as well. [17:59] lazyPower: thanks for the input. If it happens again maybe I'll see a pattern [17:59] np ennoble, cheers :) [18:02] is is-leader does some strange things :). when my leader-elected hook was called the normal juju context stuff that I gather didn't work [18:03] cholcombe: might be worth following up in #juju-dev to see why certain things arent available in leader context [18:03] ok [18:03] is that on freenode also? [18:03] yep [18:05] lazyPower: Actually my bad, I was wrong, I spelled the node name incorrectly and the machine created didn't exist when I retried. I'm back to the issue. [18:07] lazyPower: I set logging-config to juju=TRACE;unit=TRACE where is the logging done? I don't see anything on the orchestration node [18:09] with the logging set to trace, it will show up on your workstation when you run 'juju debug-log' [18:09] but its going to be a firehose of info, so be prepared [18:12] lazyPower: while I see pings and such from other instances, nothing about the machine I'm trying to add seems to show up [18:15] thats weird [18:15] you should at the very least be getting something from the action, and anything post cloud-init [18:20] lazyPower: Hey, I have some more information. The hook that Juju is running executes a losetup -f command which fails because it can't find the loop module. When I modprobe from the container I get the following error: modprobe: ERROR: ../libkmod/libkmod.c:556 kmod_search_moddep() could not open moddep file '/lib/modules/3.13.0-53-generic/modules.dep.bin [18:21] lazyPower: does that point more to app armor? [18:22] Bialogs: does the command work after you sudo depmod? [18:24] Has anyone done a complete kilo openstack HA deployment? [18:25] lazyPower: I did a depmod on the host before I ran this command -- any difference in LXC? [18:26] Bialogs: if it made no difference, i would point at apparmor, if it works its ounds like the module.bin was missing the module you needed [18:26] you should be getting an error in syslog (i think) about the apparmor denial [18:26] if its apparmor [18:26] lazyPower: lol depmod fails in the container [18:26] ok, yeah [18:27] its more than likely the apparmor profile for lxc then, which is unfortunate. [18:27] that gets fiddly [18:28] The app armor profile is on the host machine, correct? [18:28] correct [18:38] lazyPower: i think my leader stuff works now :) [18:38] lazyPower: don't really see anything that says denied, just a lot of status messages [18:38] i'm not seeing a crap ton of duplicate unit messages now [18:38] cholcombe: awesome! [18:38] :) [18:38] Bialogs: :( [18:38] Every smile has a frown [18:38] Bialogs: thats the pitts, i'm not sure what to recommend if its not giving you a big red shiny signal [18:39] Bialogs: what i can however s uggest is emailing the juju list with the issue [18:39] someone thats not around now (primarily the UK openstack team) should be able to lend a hand with whats going wrong if theyve encountered this before, otherwise they will point you to file a bug [18:40] Bialogs: juju@lists.ubuntu.com - or alternatively you can open a bug first and ping the list with the bug, which may be the better route to go, to raise visibility and get some bug-heat on it. [18:41] I'm going to see if I can poke around the app armor config first [18:41] Then I'll email the list or post the bug [18:41] Thanks! [18:47] sorry i wasnt more help Bialogs :( good luck === scuttlemonkey is now known as scuttle|afk === kadams54-away is now known as kadams54_ [19:34] So this is strange [19:35] Tried it again and It's not failing at that part, just can't communicate with RabbitMQ [19:48] mbruzek: sweet blog post! [19:48] thanks marcoceppi [19:49] http://bruzer.net/2015/06/10/deploy-a-kubernetes-development-cluster-with-juju/ [20:01] great write-up mbruzek, just read it myself [20:02] thanks tim [20:02] nice work on kubes lazyPower, mbruzek, and whit; cool stuff! [20:26] cory_fu: do you recall if unitdata is available in charmhelpers render_template? and if so, does this feel right in a .j2 template? hostname = {{ unitdata.kv['relations.ready']['my-charm'].values()[0]['private-address'] }} [20:33] kwmonroe: Do you mean in the bigdata branch, or in trunk? [20:35] either cory_fu, but i got it.. i'm using any_ready_unit instead of relations.ready now. please disregard :) [20:35] kwmonroe: It is available in the templates in the big data branch, yes, but I'd recommend you use {{ any_ready_unit('my-charm')['private-address'] }} instead [20:35] duh [20:35] :) [20:44] thanks tvansteenburgh :) === natefinch is now known as natefinch-afk [21:46] lazyPower: I submitted that bug - ended up getting another error eventually...Still super confused. Thanks for your help === kadams54_ is now known as kadams54-away === JoshStrobl is now known as JoshStrobl|AFK [22:59] guys, i'm running into a weird issue [22:59] when i deploy a machine using the manual method [22:59] my 2nd machine never starts up the virbr0 [23:00] and then i get connction timeouts