/srv/irclogs.ubuntu.com/2016/03/31/#juju.txt

cholcombeis there a list somewhere of the allowed types in actions.yaml?00:18
cholcombei thought there was but i'm having trouble finding it now00:18
cholcombeah ok i see. it's the json schema types.  Nevermind.  http://json-schema.org/latest/json-schema-core.html <-- for future people00:20
arosalesman zeppelin doesn't like zenial or spark stand-alone or both00:23
arosalesdebug continues00:23
arosalesbut dinner calls00:23
arosaless/zenial/xenial :-)00:23
cholcombelazyPower, 2 new interfaces today thanks to icey's help: https://github.com/cholcombe973/juju-interface-ceph https://github.com/cholcombe973/juju-interface-backup00:34
=== natefinch-afk is now known as natefinch
=== marlinc_ is now known as marlinc
=== xnox_ is now known as xnox
=== stokachu_ is now known as stokachu
jaceknmarcoceppi: so I understand what you are saying I just completely disagree with that approach. If you "charm pull-source" for an official, curated charm you should absolutely get official and curated layer. Not one from someone's personal branch who may have decided to add "rm -rf" there just for laugh07:36
jaceknmarcoceppi: defaulting to more secure approach is always good, especially if it's so trivial to do07:37
jamespagegnuoy, before I raise any more reviews08:07
jamespagecan you sanity check one of https://review.openstack.org/#/q/topic:network-spaces08:07
gnuoysure08:07
jamespagethe delta is pretty small08:07
gnuoyjamespage, duff grammer in README.md08:08
jamespagegnuoy, hah08:08
gnuoyjamespage, when you say "deployer" in the README do you mean that syntax is valid for both juju-deployer and juju native deployer and if not maybe its worth being explicit?08:09
jamespagegnuoy, I was intentionally vague08:10
jamespagethe juju v4 bundle format in juju supports it08:10
gnuoyhaha ok :)08:10
gnuoyjamespage, other than that cinder lgtm08:14
jamespagegnuoy, can you comment on the README and I'll update all of them08:14
gnuoyjamespage, done08:18
kjackalhey people, is there a way to tell amulet to "attach" to an existing set of charms instead of deploying them from scratch? I am trying to save up some time while writing tests.08:22
gnuoykjackal, not that I know of but let me know if you find a way!08:25
kjackalok gnuoy08:26
=== SaMnCo-laptop is now known as SaMnCo-desktop
deanmanHi, do you keep logs of this channel somewhere?08:44
magicaltroutdeanman: http://irclogs.ubuntu.com/2016/03/31/%23juju.txt08:46
deanmanmagicaltrout, thanks08:46
kjackalgnuoy, running the same test for a second time will not redeploy the charms so it does more or less an attach08:58
gnuoykjackal, well, you still loose time while it looks at the deployment, I want a way for it to jump straight to the functional tests themselves08:59
kjackalyes, true09:00
Sophie_hello my dear folks09:09
Sophie_I got a trouble situation here...09:10
jaceknwould anybody have time to have a look at my collectd subordinate review? https://bugs.launchpad.net/charms/+bug/153857309:54
mupBug #1538573: New collectd subordinate charm <Juju Charms Collection:New> <https://launchpad.net/bugs/1538573>09:55
magicaltroutjacekn: you had your turn, back of the queue now! ;)09:58
jaceknmagicaltrout: the queue is a lie! If you don't ask here you don't bet review ;)10:00
jacekns/bet/get/10:00
magicaltrouttrue10:01
deanmanmagicaltrout, apart from irc logs retention on the link you gave me is there any other alternative that you can suggest for following discussion here offline ?10:32
magicaltroutyeah, spin up the juju big data quickstart, ingest the log files into hadoop and run a bunch of Map Reduce textual analysis over it.....10:33
neiljerramDid the Ubuntu keyserver have a problem in the last hour?  I was seeing GPG key retrieval fail during bundle deployment; then suddenly it started working again...12:39
marcoceppijacekn: there is no official curated layer. YOUR layer is the official one, we don't store layers, and you're the trusted upstream because we trusted you to put the charm in the store13:10
jaceknmarcoceppi: trust level can change easily, social engineers just wait for holes like this (gain trust, get something into the charmstore and then exploit it by modifying their layer)13:13
marcoceppijacekn: but modifying the layer won't affect the charm in the store13:32
jaceknmarcoceppi: no but it will affect everybody who wants to develop the charm or those who want to compose right before deployment for new hotness13:35
marcoceppijacekn: but that's on /them/ to validate the source they're building13:36
jaceknmarcoceppi: I really can't see why keeping layers in non-personal branches is a bad idea13:36
marcoceppijacekn: we don't own any branches anymore13:36
marcoceppijacekn: launchpad ~charmers as we know it is dead13:36
jaceknmarcoceppi: sorry but realying on people fails sooner or later13:36
jaceknmarcoceppi: oh really? So all new charms in the charmstore are under personal branches?13:36
marcoceppijacekn: there is no "personal branch"13:36
marcoceppithere's no branch13:37
marcoceppiit's just a blob upload13:37
marcoceppipromulgation is a pointer to the namespace it was uploaded to13:37
marcoceppiand we gate allowance of people updating promulgated charms13:37
marcoceppiwe do all the QA we do now, but the person who created the charm "owns" the space13:37
jaceknmarcoceppi: so what happens if I delete my LP account one day?13:37
marcoceppijacekn: who cares? it's gone. The charms will still be in the store it's fine13:38
jaceknmarcoceppi: well those who might want to submit patches will care, if there is no branch there is no way to raise MP13:38
marcoceppijacekn: well, we have a way to change ownership of a promulgated source, and people can pull the charm - as it is compiled - and uncompile it since we have a manifest that tracks the files and what layers they came from13:39
marcoceppiit can be rebuilt somewhere else13:39
marcoceppior just updated in place13:39
marcoceppithe charm store is the single source of truth regardless where upstream development is13:39
marcoceppiwhat happens if an upstream's code goes away one day13:40
marcoceppithe packages in the archive won't suddenly break13:40
jaceknmarcoceppi: if I wipe my LP accont there will be no layer to pull13:40
marcoceppijacekn: I can reconstruct your layer, from the built charm13:40
mbruzekHi guys do we have the key/value "configuration" (in layer.yaml) documented anywhere? Like how do I get those values and what would be a good case to use them?  Asking for a friend.13:40
marcoceppimbruzek: not quite documented much anywhere - yet, but we do have a few examples13:41
jaceknmarcoceppi: wow all this seems super weird and is IMO asknig for trouble. In any case - the error that was noticed is NOT picked up by the latest stable charm-tools13:41
jaceknmarcoceppi: so I can't even test the fix13:41
mbruzekmarcoceppi: is it mostly the big data guys?13:41
mbruzekmarcoceppi: I would like to send my friend an example.13:42
jaceknmarcoceppi: and of course I'm not too happy about fixing something that's clearly not a requirement when I use stable tools13:42
marcoceppimbruzek: https://github.com/juju-solutions/layer-basic/blob/master/lib/charms/layer/__init__.py13:43
marcoceppifrom charms.layer import options13:43
marcoceppioptions('layer-you-want-options-from')13:43
marcoceppimbruzek: how to define options: https://github.com/juju-solutions/layer-basic/blob/master/layer.yaml13:44
marcoceppimbruzek: how to set them: https://github.com/marcoceppi/layer-gitlab/blob/master/layer.yaml13:44
marcoceppijacekn: if you don't want to put a repo key. Don't.13:44
mbruzekmarcoceppi: thank you13:45
jaceknmarcoceppi: well that's apparently waht's blockin my charmstore submission13:45
marcoceppiit's not, it was a suggestion13:45
jaceknmarcoceppi: ok so it's very unclear to me what's going on with my submission. tvansteenburgh reviewed it but the only feedback he provided was a repo key. Does it mean it's approved other than that?13:46
jaceknmarcoceppi: I suspect you can see that I'm getting increasingly upset by the review process ;)13:46
marcoceppijacekn: sure, we're tyring to build an extensive and fruitful ecosystem, but if you don't want to contribute to that by simply point to your upstream layer repo, people just won't know where to contribute to the charm and we'lljust have a bunch of forks of the compiled artifact13:47
marcoceppiwe'll simply do it for you, prior to promulgation13:48
marcoceppithe review process is two fold. One to make sure charms are sane and two - to help educate charmers as the ecosystem evolves13:48
jaceknmarcoceppi: sorry but saying that I don't want to contribute is not fair. I've been trying to get this charm promulgated for the past 2 months and I chased it more times than I can count13:50
marcoceppijacekn: yes, and the review queue is old, outdated and the team reviewing is small and over burdened13:51
marcoceppijacekn: these aren't unique to you, it's something everyone has had to sufferthough and no one is happy about13:51
marcoceppijacekn: we're aggresively building a new review queue and process as the entire way charms are uploaded and submitted to the store is changing13:51
marcoceppijacekn: the goal of this is to stream line the process so that review times go from months and weeks to days and hours13:51
marcoceppijacekn: all while juju 2.0 is being built and a shift of process is happening13:52
marcoceppijacekn: we appreciate your patcience and feel the same strain13:52
marcoceppiif you have an actual problem with the idea of a repo key, I suggest mailing the list so it can be hashed out, if you're just tired of waiting and making a stand we'll just add the key for you13:53
lazyPowermbruzek - we have examples of layer options in the docker layer as well13:53
jaceknmarcoceppi: that layer BTW is still being developed but because CS submissions are so slow I just froze it to get it thorugh the review. One it's in the charmstore there will be updates coming. So every week of delay is bigger delta to review later13:53
marcoceppijacekn: we understand, again, we have the same problems and no one is disagreeing13:53
jaceknmarcoceppi: I don't have real problem with that other than it's something I had no chance of knowing about because stable tools don't care and it's causing yet another delay13:54
marcoceppiin two weeks, if we can get the dev time, we'll be launching a beta of the new review queue which will allow you to iterate in the store outside of the review process13:54
jaceknmarcoceppi: and I can't test it using stable tools so I can't really fix it13:54
marcoceppijacekn: our stable tools are in this same flux, we have a 2.1.2 of charm-tools in the devel ppa and pypi -it's considered stable but we're waiting for them to land in xenial before ocnsidering them released13:55
marcoceppiwelcome to the most awkard time of the ecosystem13:55
marcoceppiif you don't feel comfortable adding it, we'll take care of it13:55
deanmanI have a setup of a single VM running locally and using manual provider to bootstrap it and it works fine when deploying inside lxc but as soon as i move inside a proxied network it gets stuck endlessly trying to setup the agent on the container. Any hints/clues what could be wrong?13:56
deanman(environments.yaml has proper proxy configuration)13:56
jaceknmarcoceppi: cool, thanks. FTR feel free to point at my branch I don't mind I just can't comfortably do it myself because of testing13:57
marcoceppijacekn: understood, we have charm-tools in pypi and ppa:juju/devel if you want to test in a lxd machine or virtualenv in the future13:58
jaceknack13:59
marcoceppijacekn: we hope to have it in xenial and /stable in the next two week13:59
jaceknmarcoceppi: nice!13:59
marcoceppijacekn: where are you doing your layer dev?14:00
jaceknmarcoceppi: in a LTS VM (currently trusty)14:01
marcoceppijacekn: ah, so not pushed anywhere?14:01
marcoceppideanman: what version of Juju?14:02
jaceknmarcoceppi: ah sorry that's what you meant. It is in LP https://code.launchpad.net/~jacekn/canonical-is-charms/collectd-composer14:02
marcoceppijacekn: <3 thanks14:02
marcoceppijacekn: charm LGTM, promulgating14:03
deanmanmarcoceppi, 1.25.314:03
jaceknmarcoceppi: thanks! It's 1.25.3-trusty-amd64 if you want to know (from juju table PPA). I think you can see the trend with "stable" here ;)14:03
marcoceppijacekn: hum, so it should work, but you said manual provider so I just realized I don't think cloud init gets run on manual provider which means the proxy stuff won't be put in place14:03
marcoceppideanman: ^^14:03
marcoceppideanman: can you `grep -ri "proxy" /etc/apt/apt.conf.d` on the manual machine? do you get any hits?14:04
deanmanmarcoceppi, has several entries14:06
jaceknmarcoceppi: sorry I don't get it. What's the problem with the proxy?14:06
marcoceppijacekn deanman: juju passes proxy information with cloud-init which I don't think gets run in a manual provider bootstrap, but I coudl be wrong14:06
tvansteenburghstub: have you used your juju-wait plugin with juju2 at all?14:07
jaceknmarcoceppi: that part I understand but how does it relate to my charm?14:07
deanmanmarcoceppi,14:07
marcoceppijacekn: wrong ping, sorry, meant to ping deanman14:07
deanmanmarcoceppi, "/etc/apt/apt.conf.d/42-juju-proxy-settings:Acquire::http::Proxy "http://......."14:07
marcoceppijacekn: you're charm is fine14:08
jaceknmarcoceppi: hahaha OK!14:08
marcoceppideanman: hum, so that's good. is it failing to bootstrap or failling to set the LXC up?14:08
deanmandeanman, it's failing both to deploy on lxc and on bare machine with --to 0. "Waiting agent to finish".14:09
marcoceppideanman: can you grab the logs?14:10
deanmanmarcoceppi, machine? unit ?14:10
marcoceppideanman: all?14:10
lazyPowermbruzek - have a sec to review a hotfix patch?14:11
mbruzekyes14:11
lazyPowermbruzek - we're broken in master :( - https://github.com/juju-solutions/layer-docker/pull/30/files14:11
lazyPowerI should probably get an amulet test written to stress that compose functionality just moved into the wheelhouse... mbruzek  - do you want that befor eyou do the review? I can bust something out in the next ho ur14:13
mbruzeklazyPower: I didn't see tests, so I am building and will test it manually here14:14
lazyPowerack14:14
=== cos1 is now known as c0s
deanmanmarcoceppi, http://paste.ubuntu.com/15568095/14:30
deanmanmarcoceppi, ls -lh /var/cache/lxc/cloud-trusty/ is empty even several minutes after deploying, not downloading image?14:31
marcoceppideanman: it looks like it's failing to hit itsefl after proxy14:31
marcoceppithat's the EOF14:32
deanmanmarcoceppi, could be a no_proxy env issue then ?14:32
marcoceppideanman: add 127.0.0.1 and 10.0.2.15 and 192.168.11.11 to no_proxy14:32
mbruzeklazyPower: +1 merged the docker layer14:53
lazyPowerta mbruzek , thanks for the priority review14:53
mbruzeklazyPower: thanks for sending me quality code14:54
deanmanmbruzek, first link on README at layer-docker points to non-valid URL.14:55
mbruzekdeanman on it14:55
lazyPoweroh wow it says "composer" in there14:56
lazyPowerthats some old text :D14:56
deanmanyeah, that confused me while trying to understand how to build my layer docker charm. I think i found some other references that need update too, sec...14:57
deanmanhttps://jujucharms.com/docs/1.25/authors-charm-with-docker this also points to wrong "composing documentation" link.14:59
mbruzekhttps://github.com/juju-solutions/layer-docker/pull/3215:14
A-KaserH15:14
A-KaserHi15:14
mbruzekdeanman: Please review this change based on your suggestions.15:14
deanmanmbruzek, done15:19
mbruzekThank you15:19
mbruzekdeanman: thanks for helping us improve15:21
lazyPowerA-Kaser o/15:27
deanmanmarcoceppi, are you still around ?15:32
A-Kaserhave you tips t debug an error when i run charm build15:35
A-KaserI have this error "TypeError: list indices must be integers, not str" but no more information about which file15:36
tvansteenburghA-Kaser: please put the `charm build` output in a pastebin15:37
marcoceppiA-Kaser: also -l debug might help15:37
marcoceppideanman: what's up?15:38
A-Kaserhttp://pastebin.com/c3LrVRi715:38
deanmanmarcoceppi, the no_proxy didn't seem to work out. http://paste.ubuntu.com/15568500/15:38
tvansteenburghA-Kaser: can you paste your layer.yaml too?15:40
marcoceppiA-Kaser: try upgrading to 1.11.215:40
tvansteenburghA-Kaser: actually, paste your metadata.yaml15:42
deanmanlazyPower, the juju deploy local:<series>/<charm-name> works for me, the one you are proposing is also for stable or for 2.0 ?15:42
lazyPowerdeanman - that goes away in juju 2.015:42
A-Kaserok brew charm-tools is only 1.11.1 I'm cloning the source15:42
tvansteenburghA-Kaser: i think you have a malformed metadata.yaml15:43
deanmanlazyPower, ok maybe have both at README because if you land there from stable docs then it might be confusing if it's not working ?15:43
marcoceppiA-Kaser: crud, that's my bad. I'll make sure 1.11.2 is in brew. A-Kaser it may be easier to just install from pip in a virutalenv on your OSX machine15:44
marcoceppiA-Kaser: virtualenv ~/charm-tools; . ~/charm-tools/bin/activate; pip install charm-tools==1.11.215:45
A-Kasermetadata : http://pastebin.com/R2EhNDVg15:45
marcoceppiA-Kaser: 1.11.2 is the last 1.X build, 2.0 and beyond are a bit different and wont' work as expected until they're packaged in brew15:45
marcoceppiA-Kaser: ah, your metadata is also a bit off15:46
A-Kasermarcoceppi: I have installed charm-tools there 2 days15:46
marcoceppiA-Kaser: that requires: packages: - screen should be in layer.yaml not metadata.yaml15:46
marcoceppiA-Kaser: as an example: https://github.com/marcoceppi/layer-gitlab/blob/master/layer.yaml15:47
marcoceppiA-Kaser: that requires: section is for relations, not packages15:47
A-Kaserok I've removed packages lines thx15:47
A-Kasercharm build work , thx15:48
marcoceppiA-Kaser: cheers15:49
marcoceppiA-Kaser: expect a new charm-tools 2.0 build in homebrew in the next week or so15:49
A-KaserI use juju2 but charm-tools 1.x , is it a problem ?15:51
marcoceppiA-Kaser: no, no problem15:54
marcoceppiA-Kaser: charm 2.0 will be much nicer when you go to upload charms to the store, but it's more or less the same build command, etc15:55
jamespagethedac, some comments on https://code.launchpad.net/~thedac/charm-helpers/apparmor/+merge/29009616:47
thedacjamespage: thanks.16:47
jamespageok taking a break - may be back later...16:54
thedacjamespage: pushed up fixes17:42
PrabakaranHelo Team, Hope you all doing great  I was trying to use ibm-base-layer and as confirm by matt it uses juju 2.0 so i was trying to install juju 2.0 in my machine by following the link https://jujucharms.com/docs/devel/introducing-2 and found it has steps for aws setup only and I am not able to install local lxc containers. Can you please advise on how to set up local lxc containers in juju 2.0.17:43
c0sI ingested a bigger dataset from gh-archive this time and ran some analysis on it http://54.183.80.145:9090/#/notebook/2BEZN498517:44
marcoceppic0s: sweet17:49
c0swheee ;)17:49
mbruzekPrabakaran: LXD is the new name for the local lxc provider: https://jujucharms.com/docs/devel/config-LXD17:50
cory_fuc0s: Sorry I haven't been available yet to work on the actions.  RQ and meetings have taken up my whole day17:51
mbruzekPrabakaran: You might be having issues because the local directory changes for Juju 2.0.  From ~/.juju to ~/.local/share/juju17:51
cory_fumbruzek: Is that true?  The ibm-base-layer requires 2.0?17:52
cory_fuIs it using resources?17:52
mbruzeknot yet but very soon, the ibm-base-layer is using the "terms" feature17:53
mbruzekthat is only in 2.017:53
cory_fuAh, right.17:53
c0scory_fu: no worries17:53
c0sthe main thing that I have unblocked you ;)17:53
c0scommits to *juju* repos over a couple of weeks back in January http://54.183.80.145:9090/#/notebook/2BEZN4985/paragraph/20160331-172642_1279078054?asIframe17:54
cory_fuc0s: For the action to submit a job, is there any value to making it able to schedule the job to run repeatedly?17:56
c0sI'd say yes17:57
cory_fuc0s: And would cron be the best way to do that?17:58
c0sNot sure if Zeppelin has a REST call for that... but with spark-submit, spark-shell, and other CLI tools we should be able to17:58
c0syup17:58
c0scron always is the best ;)17:58
cory_fuc0s: I see the Zeppelin REST API has endpoints for cron jobs.  Does that schedule a given notebook to be re-run periodically?18:01
c0sI think yes, that' for the whole thing18:02
c0scory_fu: kwmonroe, this http://54.183.80.145:18080/ doesn't look right18:03
c0sI have ran a bunch of the Z jobs on that cluster yet it doesn't show any18:03
cory_fuc0s: Can you check the log on the Spark unit and see if it has anything?  juju debug-log -i spark/018:10
cory_fujuju debug-log -i spark/0 --replay18:11
bugrumon juju2 what is the proper way of deploying a charm?18:12
c0ssec18:12
rick_h_bugrum: ? juju deploy mysql? what charm are you looking for?18:12
bugrumI got juju up and running (thanks to LiftedKilt) and am now trying to deploying openstack on top of it18:12
bugrumjuju2 says it can't resolve URL "cs:openstack"18:13
c0scory_fu: nothing, it just hanging there18:13
c0sdo I need to run something at the same time?18:13
cory_fuBah.  I think the -i option doesn't work very well with 2.018:14
cory_fuc0s: You might have to just `juju ssh spark/0` and view the log on there (sudo less /var/log/juju/unit-spark-0.log)18:14
c0sthere's a bunch of juju messages (what seems to be from the deployment) cory_fu18:16
kwmonroecory_fu: c0s, fyi, -i with juju 2.0 needs "unit-spark-0" and not "spark/0"18:18
c0sbut looking under /var/log/spark I see that the process is running history server like this18:18
c0s  org.apache.spark.deploy.history.HistoryServer hdfs:///user/ubuntu/directory18:18
cory_fuORLY?18:18
cory_fuGood to know18:18
c0s+1 kwmonroe18:18
kwmonroec0s: what user is running that HistoryServer process?  ubuntu?18:18
c0snotices the different name while browsing the logs manually18:19
c0syes kwmonroe18:19
kwmonroec0s: as ubuntu on the spark/0 unit, can you "hdfs -dfs -ls -R /user/ubuntu/directory"?18:19
c0sinteresting hdfs:///user/ubuntu/directory/application_1459442777673_0001.inprogress18:19
kwmonroeahh18:19
c0sthere's nothing else in this directory, so the history server is correct18:20
kwmonroeso it's writing stuff.. that's good.18:20
c0swell, sparsly18:20
c0sI have ran at least 3 different jobs, they all complete now18:20
c0sthat's the only trace of it18:20
kwmonroei wonder if the spark slaves or zepp is configured to use 54.183.80.145 as the history server18:20
c0sit's not in progress for sure18:20
c0sis there any way to check the configs using juju? Or shall I just manually look at it?18:21
kwmonroec0s: run ~/sparkpi.sh from the spark/0 unit and then check your :18080 URL.  that will tell us if spark-submit jobs are making it to the historyserver18:22
c0sso, /etc/spark/conf/ would be the correct place to check, I presume18:22
c0sok, runnig18:22
kwmonroeyeah c0s, i dunno what (if any) history server config is possible for zepp18:23
c0sand I can see http://54.183.80.145:18080/?page=1&showIncomplete=true new job in there18:23
kwmonroeok, so spark-submit works directly on the unit.  when you ran your z jobs earlier, were they using spark-submit through zeppelin?18:23
c0skwmonroe: I don't know if Zepp has to do with it. These are just spark jobs18:23
c0syeah... so, why Zepp's jobs aren't showing?18:24
c0snits18:24
kwmonroemay need to dig into the /var/log/zeppelin/* on the spark unit18:24
c0snuts18:24
c0slemme check18:24
cory_fukwmonroe: Have you run in to services getting stuck on "Waiting for agent initialization to finish" and having this in the logs over and over: http://pastebin.ubuntu.com/15570047/18:25
kwmonroeyeah cory_fu18:25
kwmonroejuju upgrade-charm <foo>18:25
cory_fuIs there a bug open for this?18:25
c0sjudging by this tail -f zeppelin-ubuntu-plugin-0.out18:26
c0sthe spark jobs are running as needed18:26
c0sI see the output in the file18:26
c0slike ------ Create new SparkContext yarn-client -------18:26
c0sand the progress indicators18:26
c0scrazy18:27
kwmonroecory_fu: you sure it's a bug?  i hit that when i deploy a local charm, hackity hack hack, rebuild and try to deploy the new rev of that local charm... at which point there's a mismatch between my $JR charm and unit-0's charm, which i was ok with.  upgrade-charm to refresh unit-0 seemed like the right thing to require in that case.18:29
c0sand I am quite positive that last night the jobs were displayed in the history server UI18:29
cory_fukwmonroe: This was on a fresh deploy18:30
cory_fuThough I did keep my bootstrap node around and just created a new model18:30
kwmonroec0s: what spark execution mode are you in?  or rather, what is the MASTER envvar set to?18:31
cory_fuStill, I think Juju should definitely handle that better.18:31
c0syarn-client18:31
c0saccording to the Zep's configuration18:32
kwmonroec0s: is it possible spark's job history won't work in yarn-client mode?  juju expose resourcemanager and go take a look at that UI to see if your jobs are listed there.18:35
c0swell, the sparkpi stuff runs in the yarn-client mode too18:35
c0sand it is logged18:36
c0skwmonroe: ^^18:36
kwmonroethat, c0s, is a mighty fine point.18:37
c0s:)18:39
c0sanyhow, perhaps isn't a huge issue, but looks like a minor bug to me. However, I am still using the bigtop-dev bundle, so it might be expected to be a bit flaky I guess18:40
cory_fuWell, we're planning to release those bundles so it shouldn't really be flaky18:41
c0sagree cory_fu18:43
c0swhat I was saying is that this is still in development, so some regressions are possible18:43
c0skwmonroe: is it possible to add say kafka to an already deployed bundle?19:19
kwmonroeyou bet c0s!  "juju deploy cs:~bigdata-dev/trusty/apache-kafka kafka"19:23
kwmonroebut then it'll wait for zookeepers, so you'll need to "juju deploy cs:~bigdata-dev/trusty/apache-zookeeper zk" and "juju add-relation kafka zk"19:23
c0sok, let me try that ;)19:24
c0sthanks!19:24
c0swhat the worst could happen19:24
kwmonroethere is no try c0s.  just do it.19:24
c0sok, kwmonroe - I won't try ;)19:24
kwmonroe:)19:24
=== blr_ is now known as blr
c0skwmonroe: shall I scale up ZK to 3 nodes as well? Or 1 will do ?19:34
admcleod1hello19:39
admcleod1c0s: i dunno if he replied, but 1 zk is fine19:40
c0sit seems to be holding for the POC purposes, yes19:41
c0sthanks!19:41
c0skaffffffka20:08
lazyPowershouted to the tune of "KHAAAAAAN"20:08
c0syes lazyPower20:10
c0salongside with the sound of my sobbing ;(20:10
kwmonroec0s: sobbing because everything went great?  sobs and frownie faces of joy?20:20
arosaleskwmonroe: is the latest code for the hadoop charms still at lp?20:34
arosalesfor example hdfs master I get directed to20:34
arosaleshttps://code.launchpad.net/~bigdata-charmers/charms/trusty/apache-hadoop-hdfs-master/trunk20:34
arosalesfrom https://jujucharms.com/apache-hadoop-hdfs-master/trusty/9 which is the promulgated version20:34
arosaleskwmonroe: that doesn't look like a reactive charm though . . . .20:38
arosalescory_fu: ^20:38
cory_fuarosales: No, all of the new charms are on github20:38
cory_fuAlso, hdfs-master is the oldness.  apache-hadoop-namenode is the newness20:39
deanmananyone using brew charm-tools 1.11.1 ?20:39
arosalescory_fu: ya I also thought all the new charms were in git20:39
arosalescory_fu: but this is the current recommeded charm20:40
arosalescory_fu: it is just the case that we haven't got the git charms to be the recommeded ones?20:40
cory_fuarosales: I'd like to fix that right now.  I'm going to start promulgating charms and bundles20:40
cory_fuAfter this meeting20:40
arosalescory_fu: ya super confusing atm20:40
arosalescory_fu: also as a result of that, if any of the charms at https://jujucharms.com/big-data change please file a bug on https://github.com/CanonicalLtd/jujucharms.com/issues20:41
arosalesdeanman: ah brew still probably at the 1.x versions20:47
arosalesdeanman: hitting any issues?20:47
deanmanarosales, yeah when trying to $charm build20:47
deanmanarosales, and this would be the tailed output when using -l DEBUG http://pastebin.ubuntu.com/15571120/20:51
* arosales looking at pastebin20:53
mbruzekHey deanman the brew release of charm-tools is a bit behind the other ones. I would recommend setting up an venv and pip installing charm tools until the new version is available.20:53
arosalesdeanman: I saw a similar error but on ubuntu I needed to install venv package20:55
deanmanmbruzek, No problem mbruzek, i'm building from within the ubuntu VM (which runs 1.11.2).20:55
arosalesdeanman: your running this on osx?20:55
deanmanarosales, Yes20:55
deanmanon ubuntu with 1.11.2 it works just fine20:55
arosalesok, deanman it looks like the dep is broken in brew20:55
arosalesaisrael: are you still looking after charm-tools in brew?20:55
aisraelarosales: Marco's been doing it20:56
aisraelI think20:56
arosalesaisrael: if so could you take an action to update to the latest, possibly even 2.0 that marcoceppi is going to put into xenial?20:56
arosalesmarcoceppi's brewing for osx :-)  I bet he has been keeping that a secret :-)20:56
lazyPowerdeanman - do you have git installed?20:56
marcoceppiarosales: it's already on my list, it'll be out tomomrrow20:56
arosalesaisrael: could you follow up with marcoceppi though?20:56
lazyPoweri assume yeah but there's only a few paths that end in that error20:56
arosalesmarcoceppi: ok20:57
lazyPowerand its typically due to not having unzip, git, or  one of the other smallish dependencies that we're shelling out to invoke20:57
arosalesdeanman: are you ok to leverage ubuntu in the vm for now?20:57
lazyPowermbruzek - oh i figured out why its tanked, we never got the restructuring merge done20:58
deanmanarosales, Yeah it's fine, just asking if you wanted to raise an issue on github20:58
lazyPowermbruzek https://github.com/juju-solutions/charms.docker/pull/14  -- when you have time <3 its kidn of big but most of it is documentation updates for the charms.docker api20:59
arosalesdeanman: thanks. Sounds like marcoceppi is going to update brew soon20:59
lazyPowerdeanman - that particular issue has been patched. https://github.com/juju/charm-tools/issues/15920:59
lazyPowerdeanman - is there a remote on your git repository? or is it a local git repo only?20:59
deanmandeanman, Yeah i do have a single remote setup21:00
deanmanlazyPower, ....21:00
lazyPowerdeanman do you have bzr installed?21:01
lazyPowerif any of those libs are in a launchpad repo, its gonna die with that error as it tries to bzr branch the remote21:01
lazyPowers/libs/layers21:01
deanmanlazyPower, a private remote repo, not a launchpad repo21:02
lazyPowerdeanman - can i see a pastebin of your layer.yaml?21:02
deanmanlazyPower, if only pastebinit on mac didn't complain about "Bad API request...", gimme a sec21:04
deanmanlazyPower, Single liner -> includes: ['layer:docker']21:04
lazyPowerand thats not in lp21:04
* lazyPower is puzzled now21:04
deanmanlazyPower, fails after copying tox.ini21:05
lazyPowerdeanman - do you have virtualenv installed?21:08
lazyPowerand/or pip21:08
deanmanafter installing virtualenv with pip the build goes through21:09
lazyPower\o/ paydirt21:09
lazyPowerhttps://github.com/juju/charm-tools/blob/v1.11.1/charmtools/build/tactics.py#L622 - was the clue21:09
lazyPowerthats poopy though, i'm pretty sure this error messaging was patched with the future revs21:10
lazyPowerso its more apparent whats happened21:10
deanmanremoved virtualenv and bug reappeared, so definitely installing virtualevn was the key !!21:10
lazyPowerhttps://github.com/juju/charm-tools/blob/v2.1.2/charmtools/build/tactics.py#L636  - doesn't look like it unless we patched it on the invocation of this in utils21:12
LiftedKiltwhat does juju-gui-console-enabled do in the juju-gui charm?21:15
lazyPowerLiftedKilt - it opens my garage door. plz dont toggle that option21:18
jrwrenafaik, nothing. it may have been used in the past.21:20
marcoceppiarosales: was waiting for the packages to settle21:23
LiftedKiltlazyPower: I'm toggling it furiously now21:24
LiftedKiltjrwren: so no plans to re-enable it then?21:24
lazyPowerLiftedKilt: schenanigans!21:25
jrwrenLiftedKilt: none that I have heard.21:25
jrwrenWe probably did not delete it because it would break bundles which specify it, but we could mark it obsolete and document that it is unused and is a no-op21:26
LiftedKiltbut it sounds like a cool feature haha21:28
arosalesmarcoceppi: ya I figured you wanted to updated with what we put into xenial21:32
arosalesmarcoceppi: appreciate all the work you have been putting into charm{tools} and getting it out there for folks21:33
A-Kaserre21:33
=== scuttle|afk is now known as scuttlemonkey
A-Kasermy charm create a new service but no units21:48
A-Kaserjuju debug-log show me nothing, another way to see log ?21:48
lazyPowerA-Kaser - did you deploy it via dragging on the gui or something? there's a scenario where you can stage a service without any allocated units, so you can do things like upload resources, tweak config21:49
A-Kaserjuju deploy local:trusty/spark-notebook21:49
lazyPower*then* you can add units to kind of "flush" the whole operation into the environment, but if thats not what you did, i'm uncertain why you got a service declaration without any units though21:49
admcleod1c0s: with the bigtop puppet deploy, how do you define the role of the machine you're deploying to?21:49
A-Kaserit's my own charm, may be I have missing something21:49
c0sadmcleod1: there are two ways21:50
A-Kasers/missing/missed/21:50
c0syou can just specify FQDNs for master nodes (NN, spark, etc.) and then the rest of the nodes will be worker nodes.21:50
c0sor you can do more complex settings of the roles via hiera21:51
c0sadmcleod1: you can check https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/hieradata/bigtop/cluster.yaml for more info21:54
c0ssearch for roles21:54
c0sthe complexity, mostly, comes from the fact that you need to maintain different hiera configs on different nodes21:55
c0swhere's the simpler method would mandate uniformed configs21:55
admcleod1c0s: right, thanks22:00
admcleod1c0s: ah yeah ok pretty straight forward22:00
c0syeah, we are simple folks ... over there ;)22:01
c0sin our corners of them woods22:01
c0sbloody hell, what damn Kafka doesn't have a thin client?22:02
c0swhat's the point of having all this crap laying around if every Joe and Frank needs to write tons of custom code to handle all that parallelism ;(22:02
c0sdamn hipsters22:03
admcleod1hah22:03
c0skwmonroe: cory_fu, admcleod1, looks like apache-kafka-14 is pulling in22:31
c0s  "url": "https://s3.amazonaws.com/jujubigdata/apache/x86_64/kafka_2.10-0.8.2.1-89ede9a.tgz"22:31
c0swhich has the bug KAFKA-171122:31
c0snot sure if this is because of that, I can not even writing anything into a topic22:31
c0sperhaps not as I can write data to a topic using juju actions22:36
c0sweeeird22:36
arosalescory_fu: how would you suggest to pull in admcleod1 branches into github/juju-solutions for the hadoop HA bits?23:19
kwmonroewhy would you want to do that arosales?23:20
kwmonroeit's not ready to merge into the juju-solutions namespace23:20
kwmonroeor, admcleod1 can push his HA-enabled charms to his namespace, and we can bundle that into a bundle-ha.yaml in the apache-core-batch-processing bundle (which does live in juju-solutions)23:22
arosaleskwmonroe: I guess I just need to understand how the juju-solutions name space is used23:23
arosaleskwmonroe: for example if I wanted to submit a merge request against the new HA bits. I have the following questions23:25
arosales1. how do I find them23:25
arosales2. where do I submit issues23:25
arosales3. where so I submit branches against23:25
cory_fuarosales: github.com/juju-solutions/apache-hadoop-namenode is the upstream.  To work on features, we fork that to our personal namespace, work, and then when it's ready submit a PR against the upstream.23:30
cory_fuOnce HA is ready, there will be a PR and once that's merged, it will be in juju-solutions23:31
arosalescory_fu: seems that doesn't give a lot of opportunity for folks collaborate on the branch before pushing to a final version23:31
arosalesbut completely agree that during early dev personal name space works well23:32
* arosales just wondering if there should be a deb branch off of http://github.com/juju-solutions/apache-hadoop-namenode23:32
cory_fuarosales: You can fork Andrew's repo and submit PRs back to him23:32
arosalesthat has early bits folks want others to test23:32
c0sI am struggling to come up with a good data ingest example for Kafka ;(23:32
arosalescory_fu: and thats one way, but now I have to follow your namespace, kwmonroe, kjackal, admcleod1, and anyone elese for the dev version23:33
cory_fuarosales: We can also have branches on the juju-solutions repo, but the "typical" GitHub workflow is to fork and submit PRs from your personal namespace23:33
arosalescory_fu: +1 on personal name space to hack.23:33
arosalescory_fu: my noob questions is where do I go to collaborate on the dev versions23:33
arosalescory_fu: if the answer is ping here and find out who's personal namespace has the dev branch I can do that23:34
arosalescory_fu: just wanted to be sure that was the correct process before I started filling issues against kjackal  and admcleod1 namespace23:34
cory_fuSo, with this workflow, there isn't really a single dev version.  There are multiple feature branches.  We had that before, but they lived in a single namespace and eventually got merged into bigdata-dev just prior to release.  So bigdata-dev was more like staging23:35
cory_fuarosales: If we think it would be easier, we can do branches within the upstream repo.  I'm not against that23:35
arosalescory_fu: you guys are the experts here :-) I just wanted to play nice with your workflow23:36
cory_fuWe'll discuss it during sync tomorrow23:36
arosalescory_fu: what ever is working for you guys I can follow, I just simple didn't know23:36
arosalescory_fu: sounds good23:36
cory_fuarosales: You can also see all the forks pretty easily on the GH website23:37
cory_fuJust click the number next to "Fork"23:37
LiftedKiltIf I can deploy xenial and trusty machines on maas with juju, and maas is telling me that a wily machine is deployed but my juju controller still shows pending, where would one go to troubleshoot that?23:38
arosalescory_fu: gotcha and that works when there are a few folks to guess who branch is the dev version I should submit issues against, but when more folks get involved (ie start branching) that guess may be a little hard, but perhaps look at that problem when it occurs.23:39
cory_fuarosales: Or you could just submit all issues against the upstream23:40
arosalescory_fu: true, as long as I have found the right dev branch to find the issues, which is the crux of my questions23:41
marcoceppibusy day , time to wind down23:42
arosalesLiftedKilt: I would start with at least "juju debug-log" and see if that gives you any useful pointers23:42
marcoceppijuju kill-controller23:42
marcoceppiWaiting on 5 models, 20 machines, 22 services23:42
marcoceppi$_$23:43
arosalesLiftedKilt: and I am guessing you are using maas 1.9 and juju 2.0?23:43
LiftedKiltarosales: correct23:43
arosalesI *think* the maas team and juju team are working on compatibility between those two versions23:44
LiftedKiltarosales: ERROR verification failed: no macaroons23:44
LiftedKiltarosales: I know they've been working on maas 2 and juju 2 due to the new maas api, but I didn't know they had compatibility problems with 1.9?23:45
arosalesany juju-devs around that can confirm that?23:45
arosalesLiftedKilt: I may very well be wrong23:45
* arosales is going to pop into juju-dev and confirm23:45
LiftedKiltyall got any more of them macaroons?23:46
LiftedKiltjust need to borrow one real quick. I promise not to eat it.23:46
arosalesmacaroons way tastier that cookies23:47
arosaless/that/than/23:47
arosalesc0s: were you looking for a stream set?23:49
c0snot exactly23:50
rick_h_arosales: LiftedKilt we're workiing on juju 2 and maas 2.023:50
rick_h_arosales: LiftedKilt 1.9 should be peachy23:50
arosalesLiftedKilt: juju-core folks say maas 1.9 and juju 2.0 should work together23:50
LiftedKiltrick_h_ arosales: any idea why juju debug-log is complaining about no macaroons?23:50
c0shere's the main dilemma: Kafka is poorly suited to work with a variety of data sources out of the box. It is a message bus, after all, so you need all sorts of sinks, consumers, etc for each particular case23:51
rick_h_LiftedKilt: hmm, that sounds like a bug to be honest.23:51
c0shence, developing a generic enough data flow (ie without writing a bunch of special code) is a challenge23:51
c0sI was thinking may be we should experiment with something like SpringXD to provide a good pipeline building tool. Which is integrated with Kafka pretty well, it seems23:52
arosalesLiftedKilt: can you post what you are seeing at http://paste.ubuntu.com/23:52
LiftedKiltI noticed all my wily charms were failing, so I tried to add-machine a couple wily machines to see if I could independently get them to work23:52
c0sor Gobblin to build ETL-like flows23:52
rick_h_LiftedKilt: they're failing to deploy via deploying wily based charms?23:53
rick_h_LiftedKilt: can you speak to what you're deploying and how?23:53
LiftedKiltarosales: https://paste.ubuntu.com/15572264/23:53
c0sor if we are to focus on collection- or aggregating-like use cases, then Flume+Kafka23:53
c0sbut bare Kafka seems to be too bare to start using is without any preps.23:54
LiftedKiltrick_h_: sure - attempting to deploy the openstack-lxd bundle23:54
arosalesc0s: perhaps prep the data before sending over to kafka23:54
rick_h_LiftedKilt: do me a favor, try juju status --format=yaml23:54
c0sarosales: you still need to connect to data23:54
LiftedKiltrick_h_: when I noticed it was hanging on provisioning the machines, I created a new model and tried adding a machine there manually23:55
LiftedKilthttps://paste.ubuntu.com/15572272/23:55
c0shaving a source isn't enough. You need to have source-specific sink23:55
c0sor producer23:55
arosalesc0s: ya the data format for a gernal case . . .23:55
rick_h_LiftedKilt: gotcha, so can you juju ssh 0 ?23:56
* arosales also looking up SpringXD c0s23:56
rick_h_LiftedKilt: and check out if anything is fishy in the /var/log/juju/xxxxx log files23:56
c0sBTW, out of the box setting up kafka on top of spark-zeppelin bundle doesn't let me to use their producer script in the shell session. Not sure if it's a bug or not23:56
LiftedKiltrick_h_: juju ssh 0 hangs23:56
rick_h_LiftedKilt: hmm, can you manually ssh to the machine IP?23:57
LiftedKiltrick_h_: It would let me it I had the key set up23:58
arosalesc0s: does seem like an issue23:58
rick_h_LiftedKilt: the yaml status says the machine is deployed?23:58
rick_h_LiftedKilt: oh heh23:58
arosalesc0s: got it on the board to investigate23:59
c0sarosales: it is either a differences in the environment between what I get via 'juju ssh kafka/0' and 'juju action do kafka/0...'23:59
c0sor else23:59
c0sok, will add a ticket now23:59
LiftedKiltrick_h_: yeah it's bizarre - it sees that it's up, but yet at the same time doesn't23:59
c0swill dig more into this later... getting to the end of my rope today :/23:59

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!