/srv/irclogs.ubuntu.com/2015/06/16/#juju.txt

=== scuttlemonkey is now known as scuttle|afk
thomiwhile doing 'juju bootstrap' on the local (lxc) env I get "ERROR there was an issue examining the environment: required environment variable not set for credentials attribute: User"03:46
thomiAny hints as to what that means?03:46
* thumper thinks03:47
thumperthomi: you aren't using lxc03:48
thumperor local03:48
thomithumper: oh wait, I think yeah03:48
thumperthat error comes from the openstack provider03:48
thomithumper: sorry, thinko on my part03:48
thumpernp03:48
thomiforgot I had the env var exported03:48
thomithanks03:48
lazyPowerthumper: good catch03:51
thumpero/ lazyPower03:51
lazyPowerthumper: btw, i *will* get to your django MP's this week, soz its taken me so long to get to them03:51
thumperlazyPower: review poke03:51
thumper:_03:51
thumperheh03:51
lazyPowerhah03:51
lazyPoweralready on your wavelength mate03:51
thumperlazyPower: once that one is in, I'll submit the celery one03:51
lazyPowerI have something you're going to want to take for a spin i think.03:51
thumperI'm using it now03:52
lazyPowerhttp://github.com/chuckbutler/dns-charm03:52
lazyPoweri've been reviving this project from last year quite a bit03:52
lazyPowerhuge feature branch is going to land later this week that includes RT53 as a provider03:52
thumperlazyPower: interesting03:54
lazyPowerI like to think so03:54
lazyPowerI've got a long road ahead of me w/ the unit tests that are failing03:54
lazyPoweri think i've failed to encapsulate my contrib code from charmhelpers somewhere, its failing on the cache file for config() in ci03:55
lazyPowerbut thats future chucks problem (by future i mean tomorrow)03:55
thumper:)03:58
lazyPowerthumper: i'll hit your MP up first thing when i clock into the office tomorrow, that sound good? that gives you 3 days to refactor before I head out for SF if anything needs some touch ups03:59
lazyPowerthen i'll be out until Thurs of next week03:59
thumperlazyPower: sounds god.03:59
thumperlazyPower: should be fine though03:59
lazyPoweraight, you're on the calendar03:59
* thumper crosses fingers03:59
lazyPowerIt more than likely is :)03:59
lazyPoweri have faith in your ability to python04:00
thumpernice04:00
thumperlazyPower: I just remembered another fix that I should submit...04:00
thumperlazyPower: although this one is all docs04:00
lazyPowerYOU WROTE DOCS?04:01
thumperno04:01
lazyPoweroh04:01
thumperthe readme was stomped over04:01
thumperbetween version 6 and 7 of the charm04:01
lazyPowerdude dont get me excited like that04:01
thumperand they no longer reflect reality04:01
thumperhowever...04:01
lazyPoweri dont think my heart could take it04:01
thumperI am going to write docs04:01
thumperaround how to write a payload charm for it04:01
thumperbecause that truly sucked04:01
thumpermessing around working that out04:02
lazyPoweri wouldn't doubt it04:02
lazyPowerpayload charms are tricky to get write04:02
lazyPower*right04:02
thumperI did learn a lot though :)04:02
thumperwould be good to capture that04:02
thumperin a way someone else can learn from it04:02
=== JoshStrobl is now known as JoshStrobl|AFK
jamwallyworld: hey, I just saw your cursor on the resources spec, there was a change to "disk-path" I mentioned.07:20
wallyworldjam: i've been making lots of changes and also responding to rick's comments. i understand the default is needed but also think we need to not hard code it to allow deployers to say their resources go elsewhere, eg onto an ebs volume07:22
wallyworldwe can hard code it if that's the plan, but i think it is a bit limiting? what if the resource won't fit on the root disk?07:23
jamwallyworld: can we fit those resources in gridfs?07:27
wallyworldjam: i had thought'd we'd use a separate mongo db so we can shard etc07:28
wallyworldbut i guess it doesn't matter, we can just hard code07:28
jamwallyworld: so I'd like to leave it in the "do we want to add this" pile07:28
jamwe can decide on it, but I'd rather start simple07:28
wallyworldok07:28
wallyworldit's not that much to support, there' s much harder stuff first up :-)07:29
wallyworldalso, isn't the default root disk size on aws 8GB?07:29
jamwallyworld: I certainly agree it isn't hard, but it is complexity that we may never actually need.07:29
wallyworldthat's rather small07:29
jamwallyworld: it is, but so is the size of the MongoDB that's running the environment.07:29
wallyworldnot if we use a separate  db for resources07:30
jamwallyworld: where does that DB *live* ?07:30
wallyworldwell, fair point07:30
wallyworldthat would be a complication first up07:31
jamwallyworld: I guess if we let you tweak the Juju API server to put the Resource cache onto a different disk07:31
wallyworldsomething lke that. but as you say, we can start simple07:31
jamwallyworld: so I'm happy to have it as a separate logical Database (like we do for presence and logging)07:31
wallyworldyup07:31
jamwallyworld: and especially for the large multiple environments having a way to go in and do some sort of surgery to handle scale will be good07:32
wallyworldyeah, we always planned to use a separate logical db07:32
wallyworldjam: so i'm off to soccer soon, i think i've answered most of rick's questions but i need time away from the doc as it's starting to blur into a mess of works. i'll revisit later and tweak some more. need to add sample output etc. there's still some points needing clarification. hopefully it's getting close07:34
jamwallyworld: np, have a good night07:35
wallyworldty, be back after soccer07:35
Odd_BlokeIf someone could take a look at https://code.launchpad.net/~daniel-thewatkins/charms/trusty/ubuntu-repository-cache/update_charm-helpers/+merge/262072, it would be much appreciated.11:22
Odd_BlokeI failed to add some of the new charmhelpers files, so the ubuntu-repository-cache charm is broken.11:22
Odd_BlokeIt's a very easy code review. :)11:22
=== JoshStrobl|AFK is now known as JoshStrobl
=== anthonyf is now known as Guest89967
lukasalazyPower: ping =)12:59
=== scuttle|afk is now known as scuttlemonkey
lazyPowerlukasa: pong13:50
lazyPowero/13:50
lukasao/13:52
lukasaWanted to get your eyes on this quickly: https://github.com/whitmo/etcd-charm/pull/1013:52
lazyPowerso, as these etcd units are not raft peers they aren't part of the same cluster?13:54
lazyPowerjust independent etcd nodes on each docker host?13:54
lukasalazyPower: Correct13:54
beisnerhi coreycb, a merge/review for you re: ceilometer amulet test updates:  https://code.launchpad.net/~1chb1n/charms/trusty/ceilometer/next-amulet-kilo/+merge/26185013:54
lazyPowerwell, s/docker/service/13:54
lazyPowerok13:55
lukasalazyPower: Eh, you say tomato...13:55
lazyPowerhehe, well the bug mentions calico openstack13:56
lazyPowerbut i bet this is for both13:56
lukasa=P Certainly on OpenStack we deploy etcd proxies everywhere for scale reasons more than anything else13:56
coreycbbeisner, ok I'll look later today probably13:56
lukasaBut also for homogeneity13:56
lukasa(Fun word, glad I got to use it)13:56
lazyPowerok, i'm good with this. would be excelent to see tests here too but i wont block on that13:56
lukasaWell, do you want to hold off a sec?13:56
beisnercoreycb, ack thanks13:56
lazyPowersure13:56
lukasaI'm writing the Calico side of things, and I can quickly sanity check by actually running the damn thing13:56
lukasa=D13:56
lazyPoweri'm +1 for that13:56
lazyPowerwhile i'ev got your attention13:57
lukasaAwesome, so that'll get done today or tomorrow13:57
lazyPoweris the docker merge still blocked on CLA?13:57
lukasaAFAIK, yes, but I'll double check13:57
lazyPowerok let me reach out to my contact and poke them again13:57
lukasalazyPower: Fab, I'm checking on my end as well13:59
lukasalazyPower: Yup, as far as we know we're still waiting on the CLA stuff14:09
lazyPoweri unfortunately had presumed as much, i just poked my contact again. i think they're dragging feet on a confirmation from management to sign it.14:10
lazyPoweri'll run the ropes on this and see if i cant get it expedited14:10
lazyPowerwhen you get some free time i'd like to work through whats there with you, i still haven't gotten a good test from it yet, but thats more than likely pebkac14:10
lukasaHopefully I'll be sitting on a little bit of time this week, assuming this etcd charm change goes off without a hitch14:14
lazyPowerright on14:20
lazyPowerI'm the lone ranger left on my team prepping for dockercon, so our roles have been reversed this week14:21
lazyPowerbut after the conf i should have some time14:21
lukasa=D Nice14:24
lukasaOur docker folks are all heads down atm, so I'm manning the fort on the charms side14:24
lukasalazyPower: Still about?14:54
lazyPowersurely14:55
lazyPowerwhats up14:55
lukasaThe install hook of the etcd charm assumes that easy_install will be present14:59
lukasaBut it's not present on an Ubuntu cloud image as far as I know...14:59
lukasaSo installing the charm explodes =P14:59
lazyPowereasy_install is shipped in cloud images on CPP clouds14:59
lazyPowerwhere are you running these tests?14:59
lukasaOn a MAAS box14:59
lazyPowerhmm14:59
lazyPowerthats bizarre, ok.15:00
lukasaWell, it's not necessarily the most up to date MAAS in the world15:00
lazyPoweri guess we can throw down a quick block fo code to install easy_install.15:00
lukasaIt's easy enough to fix, just need to manually intervene15:00
lukasaWell, you could15:00
lazyPowerbut easy_install has been present on everything i've tested on15:00
lukasaOr you could just skip the middle-man and use get-pip.py directly to install pip ;)15:00
lukasaWhich has the advantage of doing it over a secure connection, unlike easy_install15:01
lazyPowerah, i'm not a fan of doing the wget | bash method15:01
lukasaOh sure, I mean literally bundle get-pip.py15:01
lukasaJust a single file =)15:01
lazyPowerthis all stems from our pip package in archive being busted15:01
lazyPowerinstall requests and the world blows up15:01
lazyPowerstupid python dependencies :|15:01
* lazyPower rages silently against a problem thats been cropping up more and more15:01
lukasa=P This is where I put my hand up as a requests core developer15:01
lukasaSo this is a little bit my fault15:02
* lazyPower instantly un-rages and apologizes15:02
lukasa=D15:02
lukasaIt's totally ok15:02
lukasaThe situation is a mess15:02
lazyPowerit really is15:02
lukasaBut HP are paying dstufft full-time to fix it15:02
lazyPowersystem dependencies not being in a venv make this tricky15:02
lukasaYup15:02
lukasaPresumably the charm could have a virtualenv, though...?15:02
lazyPowerthats tricky. we have venv being prepped in our docker charm - but we haven't really leveraged it15:03
lazyPoweri'm not sure what issues we will crop up with going that route - but i'm game for trying it out15:03
lazyPowerthats a hefty feature branch however, as it effects the entirety of the charm15:03
lukasaYeah, I wouldn't do it now15:03
lazyPowerlets file a bug and explore that at a later date15:03
lukasaFor now I can just do a juju add-machine and hop on and install easy_install15:03
lukasaThen deploy the charm to it directly15:04
lazyPowerok, sorry about the inconvenience, but good to know if we have a substrate thats not shipping with batteries15:04
lukasa=P It's a pretty minor inconvenience15:04
lukasaI think I also have a too-old Juju, so I'm updating that as well while I'm here15:04
lazyPowerbut *handsigns* magic15:04
lazyPowerbe aware that 1.23.x has an issue whend estrying the env it pulls the socket out from underneath you15:04
lazyPower*destroying15:05
lukasaWhat's the net effect of that?15:05
lazyPowerthings like bundletester have random bouts of errors when running multiple test cases15:05
lazyPowerclient connections are terminated and you get a stacktrace while destrying an env, but the env *does* get destroyed.15:05
lazyPowerhttp://juju-ci.vapour.ws:8080/job/charm-bundle-test-aws/173/console15:06
lazyPoweris a good example of the output you'll see15:06
lazyPowerthe "reset" bits that loop for ~ 30 lines15:06
lukasaEh, I'm not scared of stacktraces15:06
lukasaOh, btw, we're dropping a new 'feature' that should make docker demos a bit nicer, which we may want to incorporate into the charm15:07
lazyPoweroh?15:07
lukasaBut basically, on cloud environments we can set up ip-in-ip tunnels between hosts and run the Calico traffic through them15:07
lukasaThis means you don't need a cloud that gives you a proper fabric15:07
lazyPowernice :)15:07
lazyPowerwhen is that expected to land?15:07
lazyPoweri have a work item this week to get SDN in our bundle we're using @ the conf15:08
lukasaIt's already in the latest release of Calico, I think the next calico-docker release will contain it15:08
lukasaWhich I'd expect...today, I think?15:08
lazyPoweroh nice15:08
lazyPoweri'll def. tail the repo and when it lands give it a go15:08
lukasaWe don't plan to call that a productised feature because customers won't deploy Calico in that kind of fabric15:08
lazyPowerright15:08
lukasaBut it's useful for demos and trying it out on clouds15:08
lazyPower+1 to that15:08
lukasaAlso, setting up those tunnels involves typing a series of *super* cryptic 'ip' commands into Linux, so charms are perfect for it. ;)15:09
lazyPowerjuju power activate!15:10
lazyPowercalco will form the network15:10
MrOJHello to everybody. Is this the place where I can share my troubles with Juju? =)15:17
gnuoyjamespage, odl-controller mp https://code.launchpad.net/~sdn-charmers/charms/trusty/odl-controller/odl-cmds/+merge/262095 (no great rush)15:22
MrOJMost of all I have question about juju agents. Is there a way to restore or regenerate agents apipasswords?15:23
lazyPowerMrOJ: agent configurations are all listed /var/lib/juju15:25
lazyPowerlet me get a direct path for you 1 m15:25
lazyPowerMrOJ: so assuming your charm name is 'test'15:26
lazyPowerteh agent config path is /var/lib/juju/agents/unit-test-#/agent.conf15:26
lazyPowerthe .conf file is a yaml formatted key/value store of all the data required to communicate w/ the state server. You can update all the values in there if required, including repointing to a new state server, updating the api password, etc.15:27
MrOJYes I know that. It' a long story but right now I don't have that directory in my system15:28
MrOJSorry my english. I'm from Finland and  its not my main language15:29
lazyPowerno worries MrOJ15:29
MrOJI think that somehow Bug #1464304 might have made this situation15:31
mupBug #1464304: Sending a SIGABRT to jujud process causes jujud to uninstall (wiping /var/lib/juju) <cts> <sts> <juju-core:Triaged> <https://launchpad.net/bugs/1464304>15:31
lazyPoweryikes!15:33
MrOJI've managed to manually restore agents.conf and all other files in /var/lib/juju and jujud start scripts in /etc/init.15:37
MrOJBut if I start jujud-machine-xx it removes /var/lib/juju again in that node.15:39
MrOJIn /var/log/juju/machine-xx.log is mention about "invalid entity name or password" and after that "fatal "api": agent should be terminated "15:44
lazyPowernatefinch: ping15:45
natefinchlazyPower: sup15:45
lazyPowerhave you seen behavior like this? is this due to a stale status left over in the state server terminating the restoration of the unit agent? i'm a bi tout of my depth here15:45
lazyPowerMrOJ: ran into a pretty hefty bug that's terminating a unit out from underneath him15:45
natefinchlazyPower: reading history15:46
natefinchMrOJ, lazyPower:  ouch, that's a gnarly one.15:49
MrOJVersion is 1.23.315:49
natefinchMrOJ: what provider are you using?  (like, amazon, maas, openstack, etc)?15:50
MrOJIt's maas15:50
natefinchMrOJ: Do you need to keep that machine running. or can you just destroy it and recreate it?15:51
MrOJI need to have it running because it's in production.15:52
MrOJI have small Openstack cloud running in our company and machine is part of it.15:53
MrOJOpenstack deployment itself is ok15:54
natefinchMrOJ: tricky. I'm talking to some of the other devs to see if we have a way to get that machine back in working order.15:57
MrOJI've learned basics about mongodb and have recovered most of data straight from there but I can't figure out how I can restore apipassword15:58
MrOJnatefinch: Thank you15:58
=== kadams54 is now known as kadams54-away
natefinchMrOJ: Still doing some tests to try to figure out the best way to get you recovered.16:14
MrOJnatefinch: Thanks again!16:16
=== natefinch_ is now known as natefinch
=== lukasa is now known as lukasa_away
=== lukasa_away is now known as lukasa
perrito666hey MrOJ :)16:32
perrito666let me recap here,  the files in /var/lib/juju where lost and you rebuilt it right?16:32
perrito666and all seems ok excepting for the api password16:33
natefinchMrOJ: I have to run for a bit, so I'm handing you off to the very capable perrito666.16:33
MrOJperrito666: Yes that's right. I forgot to mention statepassword too..16:35
=== scuttlemonkey is now known as scuttle|afk
perrito666MrOJ: currently the status for said service says something?16:43
MrOJperrito666: juju status says "agent-state: down"16:46
perrito666is it the only unit for that service?16:46
MrOJperrito666: no but they all says the same16:47
perrito666oh, so you have multiple machines/containers in that shape?16:48
MrOJyes that is the situation..16:48
perrito666ah, sorry, I had missed that part16:48
MrOJit's ok..16:49
MrOJActually all my machines are in that situation..16:50
MrOJExcept state servers16:50
MrOJI had to restore HA state servers and same time I had dns problem in MAAS.. I didn't know that then. Because of this I bumped to bug I mentioned earlier16:55
=== kadams54-away is now known as kadams54
MrOJAt least, I think this is what happened..16:57
=== scuttle|afk is now known as scuttlemonkey
perrito666MrOJ:  I am thinking, I am sure we can rescue this, but I am thinking which is the best way, either to nuke all the password for one that we can use or something like that16:58
perrito666brb, lunch.16:58
MrOJperrito666: I can restore each unit one by one.. We have only about 50 units so it's not so big job..17:02
MrOJperrito666: Ok. Take your time and have a great lunch =)17:03
cholcombecan i setup a new deployment in each test_case for amulet?  Is that advisable ?17:06
lazyPowercholcombe: Typically when the entire toplogy is undergoing a rapid change it warrants a new test file as the deployment map is defined in __init__()17:11
lazyPowerbut i'm open to seeing a different pattern emerge :)17:11
cholcombeok interesting17:12
lazyPowercholcombe: if you're only adding a unit to the topology, it should be fine to just self.deploy.add_unit() or add a new service. start w/ bare bones and iterate through the test file17:13
lazyPowerit'll cut down the overall test-run time which is a good thing, right now the integration tests are very slow17:13
lazyPowerso its kind of dependent on what you're doing17:13
cholcombewell the issue is gluster has like 10 different volume types and i want to test each one17:13
lazyPoweris this something that needs to be defined at deploy time?17:14
lazyPoweror can you reconfigure the charm w/ the different volume type17:14
cholcombei've been setting the volume type in the charm config and then running deploy17:14
lazyPowermeaning once its stood up and running, is it possible to reconfigure the charm for that volume type17:14
lazyPoweror do you *have* to redeploy to gain that volume type17:14
lazyPoweri'm thinking this is like ceph, that your volumes are defined at deploy, and as its storage you're locked into that volume type for the duration17:15
cholcombepretty much yeah17:15
perrito666MrOJ: oh, didnt catch that and had a medium to bad lunch :p17:15
cholcombeyou set it before you run it and you're locked in17:15
lazyPowerYeah, you'lll need to do a different permutation of the charm then, which would warrant a new test - as afaik there's no way to destroy a service in amulet to date17:16
MrOJperrito666: I know the feeling =)17:16
skaythanks for python-django work17:42
skayI haven't been able to look at it in a while, but definately appreciate the work in the meanwhile17:42
skaydelurking to give props17:43
beisnercholcombe, lazyPower - you might be interested in related in-flight work on the ceph amulet tests...17:47
cholcombeoh?17:47
beisnerthe pivot point is different (ubuntu:openstack release) with the same topology17:47
beisnerWIPs @ https://code.launchpad.net/~1chb1n/charms/trusty/ceph/next-amulet-update/+merge/26201617:48
beisner& https://code.launchpad.net/~1chb1n/charm-helpers/amulet-ceph-cinder-updates17:48
beisnerso that exercises the same ceph topology against precise-icehouse through vivid-kilo17:48
* cholcombe checking17:48
cholcombeyeah that's similar to what i need to do17:49
beisneractively working to update and enable kilo and predictive liberty prep17:49
beisner^ on all os-charms that is.17:49
lazyPowerbeisner: wow thats a huge diff17:49
lazyPowerthe tl;dr is you get bundle permutations mid-flight with this?17:50
cholcombeyeah really17:50
beisneryeah, some refactoring for shared usage by cinder and glance when i get there17:50
lazyPowerhmm17:50
lazyPoweryou should blog about this :)17:50
lazyPowerso i can read the blog intead of the diff17:50
lazyPower<317:50
lazyPower*instead17:50
cholcombelol17:50
beisnerha!17:52
beisnerhow about a pastebin of the test output?  ;-)   trusty-icehouse:  http://paste.ubuntu.com/11726195/17:54
beisneroh i guess that paste includes precise-icehouse too.   just got the kilo stuff working, but no osci results yet.17:55
lazyPowerbeisner: it has no pictures17:57
lazyPoweri need pictures, and a story to go with it17:57
beisneri know i know, needs shine  ;-)17:57
lazyPower:D17:57
lazyPowerso, i'll put you down as writing a blog post next week on this? excellent17:57
lazyPowerjcastro: ^17:57
lazyPoweryou saw it here first, beisner agreed to blog about his awesome osci permutations code17:57
beisneractually, that is on my list o things to do, lazyPower17:57
lazyPowerI'm being deliberately obtuse to rally support for your cause17:58
lazyPowerin the form of giving you work items17:58
lazyPowerthat are totally awesome, and i can tweet about17:58
lazyPowerwhere do you blog currently beisner?17:59
lazyPoweri'd like to add you to my feed17:59
beisnerlazyPower, http://www.beisner.com - it's been mostly idle though as i've been mostly throttled18:01
lazyPowerack, thanks for the link18:02
=== kadams54 is now known as kadams54-away
jcastrohey jose18:22
joseohai18:22
jcastrohey so office hours in 2 days iirc?18:22
josejcastro: yes, want me to host? if so, can we move it up by 2h?18:22
jcastroI would like to confirm the time so marcoceppi doesn't make fun of me18:22
jcastroI can host, can you resend me the creds just in case though?18:23
natefinchMrOJ: looks like we're handing you back to me.  Do you have log files you could share?  all-machines.log on one of the state servers might contain some useful information for figuring out what went wrong.18:23
josejcastro: sure, will do right now, along with some instructions18:24
jcastroexcellent18:24
jcastromy calendar has it for 8pm UTC, is that what you have?18:25
josejcastro: sorry, was having lunch. I do have it at 20 UTC. looks like we're good18:43
=== anthonyf is now known as Guest41199
natefinchMrOJ, ericsnow: we should talk here.  MrOJ, ericsnow is one of the developers that worked on the backup code (along with perrito666).19:49
ericsnownatefinch, MrOJ: note that I'm not nearly as familiar with the restore side of things, but I'll help as much as I can19:50
natefinchMrOJ: do you have a machine log from one of the machines that killed its state server?19:51
MrOJnatefinch, ericsnow: I think I have. just a moment19:59
perrito666I am back20:00
MrOJnatefinch: Yes I have log, but filesezi is almost 20M20:16
natefinchMrOJ: how big is it if you compress it?  It should compress a *lot*20:21
MrOJnatefinch: I'll check20:27
MrOJnatefinch: Ok.. Now I have log from machine-0 and machine-2.20:36
MrOJnatefinch: those files are about 1M compressed20:37
MrOJnatefinch: Can I email those to you or somebody else?20:40
natefinchMrOJ: email to nate.finch@canonical.com please and thank you20:45
MrOJnatefinch: ok.. I'll send those from my work email -> timo.ojala@kl-varaosat.fi20:49
thumperlazyPower: cheers for the python-django review20:53
natefinchMrOJ: btw, you said you were doing a restore while having DNS issues... why were you doing the restore in the first place?20:54
lazyPowerthumper: happy to help :)20:56
=== kadams54-away is now known as kadams54
=== kadams54 is now known as kadams54-away
=== kadams54-away is now known as kadams54
=== kadams54 is now known as kadams54-away
=== anthonyf is now known as Guest21341

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!