| === kadams54 is now known as kadams54-away | ||
| === natefinch-afk is now known as natefinch | ||
| === brandon is now known as Guest42275 | ||
| === kadams54 is now known as kadams54-away | ||
| === kadams54-away is now known as kadams54 | ||
| === kadams54 is now known as kadams54-away | ||
| === scuttlemonkey is now known as scuttle|afk | ||
| === zz_CyberJacob is now known as CyberJacob | ||
| === CyberJacob is now known as zz_CyberJacob | ||
| === zz_CyberJacob is now known as CyberJacob | ||
| Odd_Bloke | Hello all; I'm trying to add servicegroup support to the ubuntu-repository-charm, using the latest charm-helpers. | 10:36 |
|---|---|---|
| Odd_Bloke | I've added the config option and set it (for a new deployment), but none of the services show up in Nagios as having a servicegroup. | 10:37 |
| Odd_Bloke | Is there some configuration I need to do in Nagios? | 10:40 |
| Odd_Bloke | Here are some (potentially) relevant bits of config on the hosts: http://paste.ubuntu.com/11689441/ | 10:41 |
| jamespage | lukasa, around? I have a query re use of etcd on the neutron-api charm charms for Calico | 10:49 |
| lukasa | jamespage: Howdy | 10:51 |
| lukasa | Shoot | 10:51 |
| jamespage | lukasa, just working through the current set of proposed changes | 10:52 |
| jamespage | lukasa, and I note the introduction of etcd into the neutron-api charm | 10:52 |
| lukasa | Yup | 10:52 |
| jamespage | lukasa, is that used by Calico to maintain state? | 10:52 |
| lukasa | It is indeed | 10:52 |
| jamespage | I'm trying to understand how it needs to scale - clouds may need to scale the neutron-api service quite big | 10:53 |
| jamespage | does that make sense for the etcd bits? | 10:53 |
| lukasa | Yes. =) | 10:53 |
| lukasa | So, etcd will scale separately from the neutron-api side | 10:53 |
| jamespage | lukasa, right now it will match the number of neutron-api units | 10:54 |
| lukasa | Ohhh, I see | 10:54 |
| lukasa | No, it won't, not quite. =) | 10:54 |
| lukasa | neutron-api installs etcd on each node, but configures that etcd instance as a proxy | 10:54 |
| jamespage | lukasa, we also have quite a strong etcd charm coming out of the cloudfoundry work | 10:54 |
| jamespage | https://jujucharms.com/u/kubernetes/etcd/trusty/2 | 10:54 |
| lukasa | jamespage: Yeah, I'm aware, I have a roadmap item to migrate to that. =D | 10:54 |
| lukasa | But the etcd proxy instances don't count towards the cluster size | 10:55 |
| lukasa | They're basically just fancy HTTP proxies | 10:55 |
| jamespage | lukasa, what are they proxying to? | 10:56 |
| lukasa | The core etcd install, which is covered by the etcd relation I added | 10:56 |
| jamespage | lukasa, ah - sorry the use or 'peer' in the interface name confused me for a moment there | 10:57 |
| lukasa | Yeah, sorry | 10:57 |
| lukasa | From an etcd configuration perspective they count as peers | 10:57 |
| lukasa | But they don't participate in raft | 10:57 |
| jamespage | lukasa, so these don't fit into the relations provided by https://api.jujucharms.com/charmstore/v4/~kubernetes/trusty/etcd-2/archive/metadata.yaml ? | 10:58 |
| jamespage | i.e. its not etcd (client interface) | 10:59 |
| lukasa | I'm not sure yet | 10:59 |
| lukasa | I've had a bit of a chat with Whit and Charles, but I haven't sat down and examined that charm in depth yet | 10:59 |
| lukasa | And that charm may need an enhancement to work for us (though it may not, they're playing with it for calico-docker) | 11:00 |
| jamespage | lukasa, ack - OK - I understand enough to review for now | 11:00 |
| lukasa | Awesome. =) Feel free to ask more questions whenever if you need | 11:00 |
| jamespage | lukasa, I'm making the assumption I'm going to see etcd proxy use in the neutron-calico charm as well right? | 11:12 |
| lukasa | jamespage: Correct =) | 11:13 |
| jamespage | lukasa, do you have a charm-helpers mp pending as well? | 11:13 |
| lukasa | Already merged, I think | 11:13 |
| jamespage | lukasa, checking now | 11:14 |
| jamespage | lukasa, this one - https://code.launchpad.net/~cory-benfield/charm-helpers/trunk/+merge/257260 | 11:14 |
| jamespage | lukasa, fyi gnuoy and I where talking about plugins/neutron-api yesterday a bit; we're going to come up with a suboridnate approach which makes the code separation in the charms between core neutron-api and plugin a bit clearer | 11:16 |
| lukasa | jamespage: That's the one | 11:16 |
| jamespage | lukasa, you don;'t need to rework anything, but we'll probably do a tech-debt run through at some point in the future to move all plugins to that model | 11:16 |
| lukasa | jamespage: Sure thing | 11:16 |
| lukasa | I'll be around, drop me an email when you do and I'll do my best to help out | 11:16 |
| jamespage | lukasa, thanks | 11:16 |
| jamespage | lukasa, well come up with a base template and stuff as well | 11:17 |
| jamespage | make the whole process easier for all to consume | 11:17 |
| lukasa | \o/ | 11:17 |
| jamespage | lukasa, do you have a nice bundle to deploy this? | 11:34 |
| jamespage | lukasa, looking at the peer relation on the neutron-calico charm - if related to bird, is the data passed on that relation superceeded | 11:36 |
| jamespage | lukasa, peer relations get noisy at scale | 11:36 |
| jamespage | lukasa, read the code and answered my own question | 11:37 |
| jamespage | lukasa, sent you a MP for a slight nicer way of parsing calico-origin in the install hook | 11:42 |
| jamespage | lukasa, I think you could also answer the feedback on the NEW charm bug by making install and config-changed to exactly the same thing | 11:43 |
| jamespage | or maybe no-opping install althogether | 11:43 |
| jamespage | the service bringup will do install -> config-changed anyway | 11:43 |
| jamespage | lukasa, remind me again about that bird bug in trusty? | 11:45 |
| jamespage | lukasa, shutting up - funny how you raised it and everything :-) | 11:46 |
| jamespage | https://bugs.launchpad.net/ubuntu/precise/+source/bird/+bug/1342173 | 11:46 |
| mup | Bug #1342173: BIRD 1.4.0-1 fails to start on Ubuntu 14.04 <trusty> <bird (Ubuntu):Fix Released> <bird (Ubuntu Precise):Confirmed> <bird (Ubuntu Trusty):Confirmed> <bird (Ubuntu Utopic):Fix Released> <https://launchpad.net/bugs/1342173> | 11:46 |
| jamespage | lukasa, I'll get that into the SRU queue today - however its probably OK to use the upstream PPA as well, as the maintainer is the same person as in debian | 12:04 |
| jamespage | lukasa, fixes in the queue - that bug report will be updated once accepted | 12:14 |
| lukasa | jamespage: Sorry, was at lunch =) | 12:32 |
| lazyPower | jamespage: we have an etcd charm that supports large scale already :) | 12:38 |
| lazyPower | https://jujucharms.com/u/kubernetes/etcd/trusty/2 - i'd be interested in any feedbak you have here. | 12:39 |
| lazyPower | looks like we need to cut a new release however - https://github.com/whitmo/etcd-charm v0.0.3 is the latest, and supports bintar deployments + private network bootstrap (vs using etcd to bootstrap etcd) | 12:40 |
| === scuttle|afk is now known as scuttlemonkey | ||
| jamespage | lazyPower, indeed we do - can we adapt it for lukasa's requirements for etcd proxies on local units, related to a clustered, core etcd deployment. | 13:06 |
| lazyPower | jamespage: im thinking so. I have already started talking to lukasa and crew about it :) | 13:06 |
| lukasa | \o/ | 13:06 |
| lazyPower | we have pending work to address first, but that was next | 13:06 |
| jamespage | lazyPower, so i gather | 13:06 |
| jamespage | lazyPower, awesome - I was having a general run through the charmset inc the openstack charms today | 13:07 |
| lazyPower | jamespage: i'll try to refrain from jumpin in a convo before i've had my coffee moving forward, i realized about 10 minutes later, there was a whole heap more to that conversation. | 13:07 |
| jamespage | lazyPower, its all looking pretty close - if we could close on the etcd stuff, then there are only niggles to land | 13:07 |
| lazyPower | lukasa: what were we pending? just adding the client relation bits and running a test right? | 13:08 |
| lukasa | lazyPower: Yeah, and me finding time to do that | 13:11 |
| lukasa | Currently swamped coordinating about 5 or 6 different things at once. ;) | 13:11 |
| lazyPower | no pressure :D | 13:11 |
| lukasa | Sometime this week or early next I hope | 13:11 |
| lazyPower | ack, if you want some support from our side feel free to tap me in lukasa | 13:12 |
| lazyPower | i'm pretty fmiliar with what our etcd charm is doing since i ran with the upgrades to 0.2.x | 13:13 |
| lukasa | Yeah, will do. =) I just blew up an attempted deployment to Kilo, but I think I just got the config wrong ;) | 13:14 |
| lukasa | So I'll see how this goes | 13:14 |
| Odd_Bloke | Can anyone help me with a Nagios servicegroups question: http://paste.ubuntu.com/11690101/ ? | 13:21 |
| Odd_Bloke | (Trying again now East Coast people might be around :) | 13:21 |
| === cmagina_ is now known as cmagina | ||
| jamespage | beisner, https://code.launchpad.net/~james-page/ubuntu-openstack-ci/neutron-gateway-rename/+merge/261630 | 14:34 |
| mbruzek | marcoceppi: did you have an image generator for bundles? | 15:31 |
| marcoceppi | mbruzek: svg.juju.solutions | 15:31 |
| Odd_Bloke | Anyone around to give me pointers on a nagios_servicegroups problem I'm seeing? | 15:52 |
| Odd_Bloke | See http://paste.ubuntu.com/11690101/ for details. | 15:52 |
| beisner | jamespage, thanks, merged. i'll follow up on that by updating amulet tests as i think they may call the old name as well. | 16:01 |
| === ejat is now known as ejat- | ||
| beisner | jamespage, gnuoy - also, i'll work up an MP to update neutron name in the mojo specs. | 17:14 |
| cholcombe | lazyPower, so leader election in juju. I get that by calling is-leader right? | 17:33 |
| lazyPower | cholcombe: corret, thats a predicate to determine who is the leader of the service group | 17:34 |
| cholcombe | are there docs on that? i'm having trouble finding them | 17:34 |
| lazyPower | iirc they only exist in the devel doc release notes | 17:34 |
| cholcombe | ok | 17:34 |
| lazyPower | it hasnt been officially docuemented/included - and its subject to being reworked. | 17:34 |
| lazyPower | last i heard they were going to rework the leader-election bits sometime this or next cycle. | 17:34 |
| cholcombe | i'm on 1.23.3 juju | 17:34 |
| cholcombe | i'm going to give it a try and see what it returns | 17:35 |
| Bialogs | Does anyone here have experience with the Cinder charm? I'm getting the following error: losetup: Could not find any loop device. Maybe this kernel does not know about the loop device? (If so, recompile or `modprobe loop'.) | 17:35 |
| lazyPower | cholcombe: https://jujucharms.com/docs/stable/reference-release-notes | 17:35 |
| lazyPower | search "is-leader" | 17:35 |
| cholcombe | sweet | 17:35 |
| cholcombe | yeah i saw it | 17:35 |
| cholcombe | i see we have new hooks also. i need to take this into account | 17:36 |
| lazyPower | Bialogs: that sounds somewhat familar... did you provide the cinder charm a block device? | 17:36 |
| lazyPower | Bialogs: i beleive thats also covered in the charm readme if you havent taken a look yet - https://jujucharms.com/cinder/trusty/23 | 17:37 |
| Bialogs | Yes I did but I'm not certain that I did that correctly, in my config file: block-device: /var/lib/cinder-sdb.img|10G | 17:38 |
| lazyPower | beisner: tap tap :) | 17:38 |
| beisner | lazyPower, who's there? | 17:39 |
| lazyPower | beisner: any known issues with cinder and loopback storage? the config provided by Bialogs looks correct to me according to the config option. | 17:39 |
| Bialogs | lazyPower: I should also mention that I'm trying to do this in LXC | 17:39 |
| lazyPower | oooooooooooooo | 17:39 |
| lazyPower | i bet its an apparmor issue | 17:40 |
| beisner | lazyPower, loopback storage is a testing/experimental feature; i'd add that i've not tried to use it inside lxc. | 17:40 |
| lazyPower | but we'll see what the wizard has to say | 17:40 |
| lazyPower | beisner is my lifeline into the openstack wizards circle :) | 17:40 |
| Bialogs | lazyPower: On the host I can't see the kernel module either though | 17:40 |
| lazyPower | i'm willing to bet thats the issue | 17:40 |
| lazyPower | apparmor is preventing you from loading the loopback kernel module | 17:40 |
| lazyPower | i've run into so many weird edge cases with apparmor being very strict about what you're allowed to do in lxc | 17:41 |
| cholcombe | lazyPower: is-leader rocks :) | 17:41 |
| Bialogs | modprobe loop doesn't return anything either | 17:41 |
| Bialogs | apparmor will prevent that on the host machine too? | 17:41 |
| lazyPower | not on the host, no | 17:41 |
| lazyPower | only in the lxc client iirc | 17:41 |
| Bialogs | Ya so that's strange | 17:42 |
| lazyPower | you have to whitelist modules in apparmor to get them to load in lxc | 17:42 |
| lazyPower | if thats the issue that is | 17:42 |
| lazyPower | mind you i'm making a guess at this, without having evidence thats the problem | 17:42 |
| Bialogs | I'm going to try and get it loaded on the host first | 17:44 |
| beisner | lazyPower, sounds reasonable to at least look at apparmor. | 17:44 |
| ennoble | hi, I'm trying to manually provision a node and I get an error, "ERROR error checking if provisioned: subprocess encountered error code 1" is there any debugging I can turn on to further diagnose the problem? | 17:44 |
| Bialogs | Assuming I don't get this working what's the alternative to loopback Cinder | 17:44 |
| beisner | Bialogs, please do let us know if you see issues with that outside a container. | 17:45 |
| lazyPower | ennoble: can you do 'juju retry-provisioning' with --verbose? | 17:45 |
| lazyPower | er -v | 17:45 |
| ennoble | lazyPower: when I try that with the machine name I get "error invalid machine", without it "error: no machine specified." The machine isn't listed in juju status in any state after the failure | 17:48 |
| ennoble | lazyPower: is there a way to get verbose output from add-machine? -v doesn't seem to do anything | 17:48 |
| lazyPower | ennoble: you can tail the machine log on your bootstrap node, but that wont give you much output during the provisioning process unfortunately | 17:51 |
| lazyPower | i'm trying to think if there's a debug method on that to spit out more data like bootstrap --debug -v | 17:51 |
| mgz_ | lazyPower: `juju help logging` | 17:51 |
| mgz_ | turn up the verbosity, then look at the machine log | 17:52 |
| lazyPower | oh yeah i sent a Pr to the docs about this | 17:52 |
| lazyPower | mgz_: *hattip* | 17:52 |
| === kadams54_ is now known as kadams54-away | ||
| Destreyf | marcoceppi: Did you have any issues with your Hyper Visor not showing up in the stack? | 17:54 |
| ennoble | mgz_: I tried that, the help, isn't quite right in 1.23.3 (looks like you need juju set-environment logging-config="juju=TRACE;unit=TRACE" | 17:57 |
| ennoble | lazyPower: funny enough it worked after setting logging even though I've tried it many times previously... not sure how to debug that more | 17:57 |
| lazyPower | ennoble: that is indeed odd | 17:57 |
| lazyPower | perhaps a transient networking error? | 17:58 |
| ennoble | lazyPower: Possibly, although i've had an ssh session open to the machine the whole time as well. | 17:58 |
| ennoble | lazyPower: thanks for the input. If it happens again maybe I'll see a pattern | 17:59 |
| lazyPower | np ennoble, cheers :) | 17:59 |
| cholcombe | is is-leader does some strange things :). when my leader-elected hook was called the normal juju context stuff that I gather didn't work | 18:02 |
| lazyPower | cholcombe: might be worth following up in #juju-dev to see why certain things arent available in leader context | 18:03 |
| cholcombe | ok | 18:03 |
| cholcombe | is that on freenode also? | 18:03 |
| lazyPower | yep | 18:03 |
| ennoble | lazyPower: Actually my bad, I was wrong, I spelled the node name incorrectly and the machine created didn't exist when I retried. I'm back to the issue. | 18:05 |
| ennoble | lazyPower: I set logging-config to juju=TRACE;unit=TRACE where is the logging done? I don't see anything on the orchestration node | 18:07 |
| lazyPower | with the logging set to trace, it will show up on your workstation when you run 'juju debug-log' | 18:09 |
| lazyPower | but its going to be a firehose of info, so be prepared | 18:09 |
| ennoble | lazyPower: while I see pings and such from other instances, nothing about the machine I'm trying to add seems to show up | 18:12 |
| lazyPower | thats weird | 18:15 |
| lazyPower | you should at the very least be getting something from the action, and anything post cloud-init | 18:15 |
| Bialogs | lazyPower: Hey, I have some more information. The hook that Juju is running executes a losetup -f command which fails because it can't find the loop module. When I modprobe from the container I get the following error: modprobe: ERROR: ../libkmod/libkmod.c:556 kmod_search_moddep() could not open moddep file '/lib/modules/3.13.0-53-generic/modules.dep.bin | 18:20 |
| Bialogs | lazyPower: does that point more to app armor? | 18:21 |
| lazyPower | Bialogs: does the command work after you sudo depmod? | 18:22 |
| bbaqar | Has anyone done a complete kilo openstack HA deployment? | 18:24 |
| Bialogs | lazyPower: I did a depmod on the host before I ran this command -- any difference in LXC? | 18:25 |
| lazyPower | Bialogs: if it made no difference, i would point at apparmor, if it works its ounds like the module.bin was missing the module you needed | 18:26 |
| lazyPower | you should be getting an error in syslog (i think) about the apparmor denial | 18:26 |
| lazyPower | if its apparmor | 18:26 |
| Bialogs | lazyPower: lol depmod fails in the container | 18:26 |
| lazyPower | ok, yeah | 18:26 |
| lazyPower | its more than likely the apparmor profile for lxc then, which is unfortunate. | 18:27 |
| lazyPower | that gets fiddly | 18:27 |
| Bialogs | The app armor profile is on the host machine, correct? | 18:28 |
| lazyPower | correct | 18:28 |
| cholcombe | lazyPower: i think my leader stuff works now :) | 18:38 |
| Bialogs | lazyPower: don't really see anything that says denied, just a lot of status messages | 18:38 |
| cholcombe | i'm not seeing a crap ton of duplicate unit messages now | 18:38 |
| lazyPower | cholcombe: awesome! | 18:38 |
| cholcombe | :) | 18:38 |
| lazyPower | Bialogs: :( | 18:38 |
| Bialogs | Every smile has a frown | 18:38 |
| lazyPower | Bialogs: thats the pitts, i'm not sure what to recommend if its not giving you a big red shiny signal | 18:38 |
| lazyPower | Bialogs: what i can however s uggest is emailing the juju list with the issue | 18:39 |
| lazyPower | someone thats not around now (primarily the UK openstack team) should be able to lend a hand with whats going wrong if theyve encountered this before, otherwise they will point you to file a bug | 18:39 |
| lazyPower | Bialogs: juju@lists.ubuntu.com - or alternatively you can open a bug first and ping the list with the bug, which may be the better route to go, to raise visibility and get some bug-heat on it. | 18:40 |
| Bialogs | I'm going to see if I can poke around the app armor config first | 18:41 |
| Bialogs | Then I'll email the list or post the bug | 18:41 |
| Bialogs | Thanks! | 18:41 |
| lazyPower | sorry i wasnt more help Bialogs :( good luck | 18:47 |
| === scuttlemonkey is now known as scuttle|afk | ||
| === kadams54-away is now known as kadams54_ | ||
| Bialogs | So this is strange | 19:34 |
| Bialogs | Tried it again and It's not failing at that part, just can't communicate with RabbitMQ | 19:35 |
| marcoceppi | mbruzek: sweet blog post! | 19:48 |
| mbruzek | thanks marcoceppi | 19:48 |
| mbruzek | http://bruzer.net/2015/06/10/deploy-a-kubernetes-development-cluster-with-juju/ | 19:49 |
| tvansteenburgh | great write-up mbruzek, just read it myself | 20:01 |
| mbruzek | thanks tim | 20:02 |
| tvansteenburgh | nice work on kubes lazyPower, mbruzek, and whit; cool stuff! | 20:02 |
| kwmonroe | cory_fu: do you recall if unitdata is available in charmhelpers render_template? and if so, does this feel right in a .j2 template? hostname = {{ unitdata.kv['relations.ready']['my-charm'].values()[0]['private-address'] }} | 20:26 |
| cory_fu | kwmonroe: Do you mean in the bigdata branch, or in trunk? | 20:33 |
| kwmonroe | either cory_fu, but i got it.. i'm using any_ready_unit instead of relations.ready now. please disregard :) | 20:35 |
| cory_fu | kwmonroe: It is available in the templates in the big data branch, yes, but I'd recommend you use {{ any_ready_unit('my-charm')['private-address'] }} instead | 20:35 |
| kwmonroe | duh | 20:35 |
| cory_fu | :) | 20:35 |
| lazyPower | thanks tvansteenburgh :) | 20:44 |
| === natefinch is now known as natefinch-afk | ||
| Bialogs | lazyPower: I submitted that bug - ended up getting another error eventually...Still super confused. Thanks for your help | 21:46 |
| === kadams54_ is now known as kadams54-away | ||
| === JoshStrobl is now known as JoshStrobl|AFK | ||
| Destreyf | guys, i'm running into a weird issue | 22:59 |
| Destreyf | when i deploy a machine using the manual method | 22:59 |
| Destreyf | my 2nd machine never starts up the virbr0 | 22:59 |
| Destreyf | and then i get connction timeouts | 23:00 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!