=== kadams54 is now known as kadams54-away
=== natefinch-afk is now known as natefinch
=== brandon is now known as Guest42275
=== kadams54 is now known as kadams54-away
=== kadams54-away is now known as kadams54
=== kadams54 is now known as kadams54-away
=== scuttlemonkey is now known as scuttle|afk
=== zz_CyberJacob is now known as CyberJacob
=== CyberJacob is now known as zz_CyberJacob
=== zz_CyberJacob is now known as CyberJacob
Odd_BlokeHello all; I'm trying to add servicegroup support to the ubuntu-repository-charm, using the latest charm-helpers.10:36
Odd_BlokeI've added the config option and set it (for a new deployment), but none of the services show up in Nagios as having a servicegroup.10:37
Odd_BlokeIs there some configuration I need to do in Nagios?10:40
Odd_BlokeHere are some (potentially) relevant bits of config on the hosts: http://paste.ubuntu.com/11689441/10:41
jamespagelukasa, around? I have a query re use of etcd on the neutron-api charm charms for Calico10:49
lukasajamespage: Howdy10:51
jamespagelukasa, just working through the current set of proposed changes10:52
jamespagelukasa, and I note the introduction of etcd into the neutron-api charm10:52
jamespagelukasa, is that used by Calico to maintain state?10:52
lukasaIt is indeed10:52
jamespageI'm trying to understand how it needs to scale - clouds may need to scale the neutron-api service quite big10:53
jamespagedoes that make sense for the etcd bits?10:53
lukasaYes. =)10:53
lukasaSo, etcd will scale separately from the neutron-api side10:53
jamespagelukasa, right now it will match the number of neutron-api units10:54
lukasaOhhh, I see10:54
lukasaNo, it won't, not quite. =)10:54
lukasaneutron-api installs etcd on each node, but configures that etcd instance as a proxy10:54
jamespagelukasa, we also have quite a strong etcd charm coming out of the cloudfoundry work10:54
lukasajamespage: Yeah, I'm aware, I have a roadmap item to migrate to that. =D10:54
lukasaBut the etcd proxy instances don't count towards the cluster size10:55
lukasaThey're basically just fancy HTTP proxies10:55
jamespagelukasa, what are they proxying to?10:56
lukasaThe core etcd install, which is covered by the etcd relation I added10:56
jamespagelukasa, ah - sorry the use or 'peer' in the interface name confused me for a moment there10:57
lukasaYeah, sorry10:57
lukasaFrom an etcd configuration perspective they count as peers10:57
lukasaBut they don't participate in raft10:57
jamespagelukasa, so these don't fit into the relations provided by https://api.jujucharms.com/charmstore/v4/~kubernetes/trusty/etcd-2/archive/metadata.yaml ?10:58
jamespagei.e. its not etcd (client interface)10:59
lukasaI'm not sure yet10:59
lukasaI've had a bit of a chat with Whit and Charles, but I haven't sat down and examined that charm in depth yet10:59
lukasaAnd that charm may need an enhancement to work for us (though it may not, they're playing with it for calico-docker)11:00
jamespagelukasa, ack - OK - I understand enough to review for now11:00
lukasaAwesome. =) Feel free to ask more questions whenever if you need11:00
jamespagelukasa, I'm making the assumption I'm going to see etcd proxy use in the neutron-calico charm as well right?11:12
lukasajamespage: Correct =)11:13
jamespagelukasa, do you have a charm-helpers mp pending as well?11:13
lukasaAlready merged, I think11:13
jamespagelukasa, checking now11:14
jamespagelukasa, this one - https://code.launchpad.net/~cory-benfield/charm-helpers/trunk/+merge/25726011:14
jamespagelukasa, fyi gnuoy and I where talking about plugins/neutron-api yesterday a bit; we're going to come up with a suboridnate approach which makes the code separation in the charms between core neutron-api and plugin a bit clearer11:16
lukasajamespage: That's the one11:16
jamespagelukasa, you don;'t need to rework anything, but we'll probably do a tech-debt run through at some point in the future to move all plugins to that model11:16
lukasajamespage: Sure thing11:16
lukasaI'll be around, drop me an email when you do and I'll do my best to help out11:16
jamespagelukasa, thanks11:16
jamespagelukasa, well come up with a base template and stuff as well11:17
jamespagemake the whole process easier for all to consume11:17
jamespagelukasa, do you have a nice bundle to deploy this?11:34
jamespagelukasa, looking at the peer relation on the neutron-calico charm - if related to bird, is the data passed on that relation superceeded11:36
jamespagelukasa, peer relations get noisy at scale11:36
jamespagelukasa, read the code and answered my own question11:37
jamespagelukasa, sent you a MP for a slight nicer way of parsing calico-origin in the install hook11:42
jamespagelukasa, I think you could also answer the feedback on the NEW charm bug by making install and config-changed to exactly the same thing11:43
jamespageor maybe no-opping install althogether11:43
jamespagethe service bringup will do install -> config-changed anyway11:43
jamespagelukasa, remind me again about that bird bug in trusty?11:45
jamespagelukasa, shutting up - funny how you raised it and everything :-)11:46
mupBug #1342173: BIRD 1.4.0-1 fails to start on Ubuntu 14.04 <trusty> <bird (Ubuntu):Fix Released> <bird (Ubuntu Precise):Confirmed> <bird (Ubuntu Trusty):Confirmed> <bird (Ubuntu Utopic):Fix Released> <https://launchpad.net/bugs/1342173>11:46
jamespagelukasa, I'll get that into the SRU queue today - however its probably OK to use the upstream PPA as well, as the maintainer is the same person as in debian12:04
jamespagelukasa, fixes in the queue - that bug report will be updated once accepted12:14
lukasajamespage: Sorry, was at lunch =)12:32
lazyPowerjamespage: we have an etcd charm that supports large scale already :)12:38
lazyPowerhttps://jujucharms.com/u/kubernetes/etcd/trusty/2 - i'd be interested in any feedbak you have here.12:39
lazyPowerlooks like we need to cut a new release however - https://github.com/whitmo/etcd-charm  v0.0.3 is the latest, and supports bintar deployments + private network bootstrap (vs using etcd to bootstrap etcd)12:40
=== scuttle|afk is now known as scuttlemonkey
jamespagelazyPower, indeed we do - can we adapt it for lukasa's requirements for etcd proxies on local units, related to a clustered, core etcd deployment.13:06
lazyPowerjamespage: im thinking so. I have already started talking to lukasa and crew about it :)13:06
lazyPowerwe have pending work to address first, but that was next13:06
jamespagelazyPower, so i gather13:06
jamespagelazyPower, awesome - I was having a general run through the charmset inc the openstack charms today13:07
lazyPowerjamespage: i'll try to refrain from jumpin in a convo before i've had my coffee moving forward, i realized about 10 minutes later, there was a whole heap more to that conversation.13:07
jamespagelazyPower, its all looking pretty close - if we could close on the etcd stuff, then there are only niggles to land13:07
lazyPowerlukasa: what were we pending? just adding the client relation bits and running a test right?13:08
lukasalazyPower: Yeah, and me finding time to do that13:11
lukasaCurrently swamped coordinating about 5 or 6 different things at once. ;)13:11
lazyPowerno pressure :D13:11
lukasaSometime this week or early next I hope13:11
lazyPowerack, if you want some support from our side feel free to tap me in lukasa13:12
lazyPoweri'm pretty fmiliar with what our etcd charm is doing since i ran with the upgrades to 0.2.x13:13
lukasaYeah, will do. =) I just blew up an attempted deployment to Kilo, but I think I just got the config wrong ;)13:14
lukasaSo I'll see how this goes13:14
Odd_BlokeCan anyone help me with a Nagios servicegroups question: http://paste.ubuntu.com/11690101/ ?13:21
Odd_Bloke(Trying again now East Coast people might be around :)13:21
=== cmagina_ is now known as cmagina
jamespagebeisner, https://code.launchpad.net/~james-page/ubuntu-openstack-ci/neutron-gateway-rename/+merge/26163014:34
mbruzekmarcoceppi: did you have an image generator for bundles?15:31
marcoceppimbruzek: svg.juju.solutions15:31
Odd_BlokeAnyone around to give me pointers on a nagios_servicegroups problem I'm seeing?15:52
Odd_BlokeSee http://paste.ubuntu.com/11690101/ for details.15:52
beisnerjamespage, thanks, merged.  i'll follow up on that by updating amulet tests as i think they may call the old name as well.16:01
=== ejat is now known as ejat-
beisnerjamespage, gnuoy - also, i'll work up an MP to update neutron name in the mojo specs.17:14
cholcombelazyPower, so leader election in juju.  I get that by calling is-leader right?17:33
lazyPowercholcombe: corret, thats a predicate to determine who is the leader of the service group17:34
cholcombeare there docs on that?  i'm having trouble finding them17:34
lazyPoweriirc they only exist in the devel doc release notes17:34
lazyPowerit hasnt been officially docuemented/included - and its subject to being reworked.17:34
lazyPowerlast i heard they were going to rework the leader-election bits sometime this or next cycle.17:34
cholcombei'm on 1.23.3 juju17:34
cholcombei'm going to give it a try and see what it returns17:35
BialogsDoes anyone here have experience with the Cinder charm? I'm getting the following error: losetup: Could not find any loop device. Maybe this kernel does not know about the loop device? (If so, recompile or `modprobe loop'.)17:35
lazyPowercholcombe: https://jujucharms.com/docs/stable/reference-release-notes17:35
lazyPowersearch "is-leader"17:35
cholcombeyeah i saw it17:35
cholcombei see we have new hooks also. i need to take this into account17:36
lazyPowerBialogs: that sounds somewhat familar... did you provide the cinder charm a block device?17:36
lazyPowerBialogs: i beleive thats also covered in the charm readme if you havent taken a look yet - https://jujucharms.com/cinder/trusty/2317:37
BialogsYes I did but I'm not certain that I did that correctly, in my config file: block-device: /var/lib/cinder-sdb.img|10G17:38
lazyPowerbeisner: tap tap :)17:38
beisnerlazyPower, who's there?17:39
lazyPowerbeisner: any known issues with cinder and loopback storage? the config provided by Bialogs looks correct to me according to the config option.17:39
BialogslazyPower: I should also mention that I'm trying to do this in LXC17:39
lazyPoweri bet its an apparmor issue17:40
beisnerlazyPower, loopback storage is a testing/experimental feature;  i'd add that i've not tried to use it inside lxc.17:40
lazyPowerbut we'll see what the wizard has to say17:40
lazyPowerbeisner is my lifeline into the openstack wizards circle :)17:40
BialogslazyPower: On the host I can't see the kernel module either though17:40
lazyPoweri'm willing to bet thats the issue17:40
lazyPowerapparmor is preventing you from loading the loopback kernel module17:40
lazyPoweri've run into so many weird edge cases with apparmor being very strict about what you're allowed to do in lxc17:41
cholcombelazyPower: is-leader rocks :)17:41
Bialogsmodprobe loop doesn't return anything either17:41
Bialogsapparmor will prevent that on the host machine too?17:41
lazyPowernot on the host, no17:41
lazyPoweronly in the lxc client iirc17:41
BialogsYa so that's strange17:42
lazyPoweryou have to whitelist modules in apparmor to get them to load in lxc17:42
lazyPowerif thats the issue that is17:42
lazyPowermind you i'm making a guess at this, without having evidence thats the problem17:42
BialogsI'm going to try and get it loaded on the host first17:44
beisnerlazyPower, sounds reasonable to at least look at apparmor.17:44
ennoblehi, I'm trying to manually provision a node and I get an error, "ERROR error checking if provisioned: subprocess encountered error code 1" is there any debugging I can turn on to further diagnose the problem?17:44
BialogsAssuming I don't get this working what's the alternative to loopback Cinder17:44
beisnerBialogs, please do let us know if you see issues with that outside a container.17:45
lazyPowerennoble: can you do 'juju retry-provisioning' with --verbose?17:45
lazyPowerer -v17:45
ennoblelazyPower: when I try that with the machine name I get "error invalid machine", without it "error: no machine specified." The machine isn't listed in juju status in any state after the failure17:48
ennoblelazyPower: is there a way to get verbose output from add-machine? -v doesn't seem to do anything17:48
lazyPowerennoble: you can tail the machine log on your bootstrap node, but that wont give you much output during the provisioning process unfortunately17:51
lazyPoweri'm trying to think if there's a debug method on that to spit out more data like bootstrap --debug -v17:51
mgz_lazyPower: `juju help logging`17:51
mgz_turn up the verbosity, then look at the machine log17:52
lazyPoweroh yeah i sent a Pr to the docs about this17:52
lazyPowermgz_: *hattip*17:52
=== kadams54_ is now known as kadams54-away
Destreyfmarcoceppi: Did you have any issues with your Hyper Visor not showing up in the stack?17:54
ennoblemgz_: I tried that, the help, isn't quite right in 1.23.3 (looks like you need juju set-environment logging-config="juju=TRACE;unit=TRACE"17:57
ennoblelazyPower: funny enough it worked after setting logging even though I've tried it many times previously... not sure how to debug that more17:57
lazyPowerennoble: that is indeed odd17:57
lazyPowerperhaps a transient networking error?17:58
ennoblelazyPower: Possibly, although i've had an ssh session open to the machine the whole time as well.17:58
ennoblelazyPower: thanks for the input. If it happens again maybe I'll see a pattern17:59
lazyPowernp ennoble, cheers :)17:59
cholcombeis is-leader does some strange things :).  when my leader-elected hook was called the normal juju context stuff that I gather didn't work18:02
lazyPowercholcombe: might be worth following up in #juju-dev to see why certain things arent available in leader context18:03
cholcombeis that on freenode also?18:03
ennoblelazyPower: Actually my bad, I was wrong, I spelled the node name incorrectly and the machine created didn't exist when I retried. I'm back to the issue.18:05
ennoblelazyPower: I set logging-config to juju=TRACE;unit=TRACE where is the logging done? I don't see anything on the orchestration node18:07
lazyPowerwith the logging set to trace, it will show up on your workstation when you run 'juju debug-log'18:09
lazyPowerbut its going to be a firehose of info, so be prepared18:09
ennoblelazyPower: while I see pings and such from other instances, nothing about the machine I'm trying to add seems to show up18:12
lazyPowerthats weird18:15
lazyPoweryou should at the very least be getting something from the action, and anything post cloud-init18:15
BialogslazyPower: Hey, I have some more information. The hook that Juju is running executes a losetup -f command which fails because it can't find the loop module. When I modprobe from the container I get the following error: modprobe: ERROR: ../libkmod/libkmod.c:556 kmod_search_moddep() could not open moddep file '/lib/modules/3.13.0-53-generic/modules.dep.bin18:20
BialogslazyPower: does that point more to app armor?18:21
lazyPowerBialogs: does the command work after you sudo depmod?18:22
bbaqarHas anyone done a complete kilo openstack HA deployment?18:24
BialogslazyPower: I did a depmod on the host before I ran this command -- any difference in LXC?18:25
lazyPowerBialogs: if it made no difference, i would point at apparmor, if it works its ounds like the module.bin was missing the module you needed18:26
lazyPoweryou should be getting an error in syslog (i think) about the apparmor denial18:26
lazyPowerif its apparmor18:26
BialogslazyPower: lol depmod fails in the container18:26
lazyPowerok, yeah18:26
lazyPowerits more than likely the apparmor profile for lxc then, which is unfortunate.18:27
lazyPowerthat gets fiddly18:27
BialogsThe app armor profile is on the host machine, correct?18:28
cholcombelazyPower: i think my leader stuff works now :)18:38
BialogslazyPower: don't really see anything that says denied, just a lot of status messages18:38
cholcombei'm not seeing a crap ton of duplicate unit messages now18:38
lazyPowercholcombe: awesome!18:38
lazyPowerBialogs: :(18:38
BialogsEvery smile has a frown18:38
lazyPowerBialogs: thats the pitts, i'm not sure what to recommend if its not giving you a big red shiny signal18:38
lazyPowerBialogs: what i can however s uggest is emailing the juju list with the issue18:39
lazyPowersomeone thats not around now (primarily the UK openstack team) should be able to lend a hand with whats going wrong if theyve encountered this before, otherwise they will point you to file a bug18:39
lazyPowerBialogs: juju@lists.ubuntu.com - or alternatively you can open a bug first and ping the list with the bug, which may be the better route to go, to raise visibility and get some bug-heat on it.18:40
BialogsI'm going to see if I can poke around the app armor config first18:41
BialogsThen I'll email the list or post the bug18:41
lazyPowersorry i wasnt more help Bialogs :( good luck18:47
=== scuttlemonkey is now known as scuttle|afk
=== kadams54-away is now known as kadams54_
BialogsSo this is strange19:34
BialogsTried it again and It's not failing at that part, just can't communicate with RabbitMQ19:35
marcoceppimbruzek: sweet blog post!19:48
mbruzekthanks marcoceppi19:48
tvansteenburghgreat write-up mbruzek, just read it myself20:01
mbruzekthanks tim20:02
tvansteenburghnice work on kubes lazyPower, mbruzek, and whit; cool stuff!20:02
kwmonroecory_fu: do you recall if unitdata is available in charmhelpers render_template?  and if so, does this feel right in a .j2 template? hostname = {{ unitdata.kv['relations.ready']['my-charm'].values()[0]['private-address'] }}20:26
cory_fukwmonroe: Do you mean in the bigdata branch, or in trunk?20:33
kwmonroeeither cory_fu, but i got it.. i'm using any_ready_unit instead of relations.ready now.  please disregard :)20:35
cory_fukwmonroe: It is available in the templates in the big data branch, yes, but I'd recommend you use {{ any_ready_unit('my-charm')['private-address'] }} instead20:35
lazyPowerthanks tvansteenburgh :)20:44
=== natefinch is now known as natefinch-afk
BialogslazyPower: I submitted that bug - ended up getting another error eventually...Still super confused. Thanks for your help21:46
=== kadams54_ is now known as kadams54-away
=== JoshStrobl is now known as JoshStrobl|AFK
Destreyfguys, i'm running into a weird issue22:59
Destreyfwhen i deploy a machine using the manual method22:59
Destreyfmy 2nd machine never starts up the virbr022:59
Destreyfand then i get connction timeouts23:00

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!