[04:04] <heckles1000> #luigi
[04:04] <heckles1000> whoops
=== frankban|afk is now known as frankban
=== disposable3 is now known as disposable2
[16:45] <bdx> kwmonroe: I'm creating the issue for the bigtop project on apache jira
[16:45] <bdx> should I tag it as "New Feature"?
[16:49] <bdx> kwmonroe: https://issues.apache.org/jira/browse/BIGTOP-3007
[16:51] <bdx> kwmonroe: let me know if theres anything else you want/don't want to see on there and I'll add/remove it
[16:53] <kwmonroe> perfect bdx!  thank you!!
[16:53] <bdx> kwmonroe: np, thank you
[16:54] <bdx> kwmonroe: would I be able to fork->branch the juju-solutions/bigtop repo and just add those properties to the hive config to get started working on implementing those bits between the charms?
[16:54] <bdx> or is there something else
[16:56] <kwmonroe> bdx: you probably want to fork upstream, right? https://github.com/apache/bigtop
[16:57] <bdx> ahh, but the juju-solutions fork has the layers .....
[16:58] <bdx> hmmm, possibly I need to do a bit more digging to understand how these pieces fit together
[17:01] <kwmonroe> bdx: upstream has the layers too: https://github.com/apache/bigtop/tree/master/bigtop-packages/src/charm
[17:01] <kwmonroe> fwiw, the upstream layers are the source of truth for the charms in the store -- not our juju-soln/bigtop fork
[17:02] <bdx> ahhh gotcha
[17:14] <bdx> kwmonroe: should these properties be conditional in the xml template similar to hbase.zookeeper.quorum?
[17:14] <bdx> im thinking so
[17:15] <bdx> ahh, possibly hive.support.concurrency should default to false
[17:16] <bdx> and hive.zookeeper.quorum shoudl get the conditional
[17:20] <kwmonroe> yeah bdx, i think you're on the right track.. for reference, here's how we added 3 new options in the puppet scripts:  https://github.com/apache/bigtop/pull/238/files
[17:21] <kwmonroe> bdx: you'd do something similar to initialize the vars in hadoop_hive's init.pp: https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/modules/hadoop_hive/manifests/init.pp#L48
[17:21] <kwmonroe> and then conditionally use those vars in the hive-site.xml: https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/modules/hadoop_hive/templates/hive-site.xml
[17:24] <kwmonroe> bdx: as for the default values, i would set concurrency to false by default (since that is the default): https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.support.concurrency
[17:25] <kwmonroe> and make hive.zk.quorum an empty string
[17:25] <kwmonroe> ^^ both of those in hadoop_hive's init.pp common_config class def
[17:26] <kwmonroe> bdx: then write out the concurrency property and conditional zk.quorum property in hadoop_hive's templates/hive-site.xml
[17:26] <bdx> done
[17:27] <bdx> https://github.com/jamesbeedy/bigtop/blob/hive_config_add_zookeeper/bigtop-deploy/puppet/modules/hadoop_hive/manifests/init.pp#L50,L51
[17:27] <bdx> https://github.com/jamesbeedy/bigtop/blob/hive_config_add_zookeeper/bigtop-deploy/puppet/modules/hadoop_hive/templates/hive-site.xml#L45,L57
[17:28] <kwmonroe> perfect
[17:28] <kwmonroe> see?!?!  big data is easy!
[17:28] <kwmonroe> can i get an amen magicaltrout?
=== frankban is now known as frankban|afk
=== chrisp_ is now known as chrisp262
[18:20] <bdx> kwmonroe: https://github.com/jamesbeedy/bigtop/blob/hive_config_add_zookeeper/bigtop-packages/src/charm/hive/layer-hive/lib/charms/layer/bigtop_hive.py
[18:22] <bdx> and https://github.com/jamesbeedy/bigtop/blob/hive_config_add_zookeeper/bigtop-packages/src/charm/hive/layer-hive/reactive/hive.py
[18:29] <boucherv> Some issue with the snaps website and api ? https://snapcraft.io/ - 503 Service Temporarily Unavailable nginx/1.13.7
[18:29] <kwmonroe> looks great bdx!  when you're ready, you can PR that against the bigtop repo.  be sure to follow the PR title guidelines so jira will auto-link your pr to the ticket:  https://github.com/juju-solutions/bigdata-community/wiki/Contributing#pull-request-formatting
[18:30] <bdx> kwmonroe: awesome, thanks
[18:46] <kwmonroe> boucherv: i was seeing 502/503 too, but it seems to be back for me now.
[18:47] <boucherv> kwmonroe: Yes It seems to be back now! ty
[19:07] <bdx> kwmonroe: https://github.com/apache/bigtop/blob/master/bigtop-packages/src/charm/hive/layer-hive/lib/charms/layer/bigtop_hive.py#L61,L63
[19:07] <bdx> (self.dist_config.path('hive_conf') / 'hive-env.sh.template')
[19:07] <bdx> I cant seem to find hive-env.sh, or hive-env.sh.template
[19:07] <bdx> should I create this file?
[19:08] <bdx> I feel like I'm missing something, as the charm should have hit the error I'm seeing long ago if that file didnt exist
[19:09] <bdx> possibly I borked something https://paste.ubuntu.com/p/W9tN3Ygyn9/
[19:30] <kwmonroe> bdx: you are correct... i can't find ref to that template either.  wth!  /me looks deeper down the rabbit hole.
[20:09] <bdx> kwmonroe: https://paste.ubuntu.com/p/CSscBbmwtf/
[20:09] <bdx> when I deploy the upstream hive charm
[20:09] <bdx> I see the files
[20:10] <bdx> and it doesnt error
[20:20] <zeestrat> rick_h: Are requests to subscribe to juju@lists.ubuntu.com moderated or should you get a confirmation mail asap? Trying to change my mail, but getting ain't no confirmation love :(
[20:36] <rick_h> zeestrat: thought it just auto went through.
[20:42] <zeestrat> rick_h: Alright, thanks. I'll give it a day before I start nagging.
[20:55] <magicaltrout> i've changed my email address on the juju list before
[20:55] <magicaltrout> it took about 3 seconds
[20:57] <kwmonroe> aight bdx, i think i know what's up.  first of all, those template files come from the upstream tgz that gets used during the bigtop deb build, specifically here: https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hive/install_hive.sh#L161
[20:57] <bdx> gotcha, was just figuring that out myself
[20:57] <kwmonroe> so bigtop doesn't explicitly have a hive-env.sh like it does for hive-site.xml, but it does get there from the deb.
[20:58] <kwmonroe> bdx: the next part was a bit of a mystery.. i was able to confirm the failure when i used hive-21 (from the candidate channel) and hadoop-plugin-31 (from the stable channel)
[20:58] <kwmonroe> the problem is that the stable channel plugin is still set to use bigtop-1.2.0, whereas the candidate charms are bigtop-1.2.1.
[20:59] <kwmonroe> this is a problem because hive will set up it's puppet heiradata on initial install and get ready to run puppet apply using the config from bigtop-1.2.1.
[20:59] <kwmonroe> then the plugin comes along and overwrites all that config with bigtop-1.2.0
[20:59] <kwmonroe> when the plugin does this, the puppet config is no longer including the "hive" role, and so puppet apply never does an "apt install hive"
[21:00] <bdx> oh man lol
[21:00] <kwmonroe> without installing the pacakge, puppet apply succeeds, but since hive didn't actually get installed, there's no /etc/hive/conf, and hence the failure in the hive install charm method
[21:00] <bdx> nice
[21:00] <kwmonroe> so, to confirm that, would you confirm if you used a stable hadoop-plugin with a candidate (or beta / edge) version of the hive charm?
[21:00] <bdx> that makes sense from what little I know about the process
[21:01] <bdx> yes, I'll give that a try now
[21:01] <bdx> can I just build hive from upstream apache/bigtop and use that?
[21:02] <magicaltrout> you make kwmonroe when you suggest such things
[21:02] <magicaltrout> cry
[21:02] <magicaltrout> missed the important word
[21:02] <magicaltrout> should drink less gin
[21:02] <kwmonroe> you sure can bdx -- but if you do, make sure you use hadoop-plugin-33 (from candidate) so everybody's on bigtop-1.2.1
[21:02] <bdx> ok
[21:02] <kwmonroe> heh magicaltrout.  big data is still easy!  where's my amen?
[21:03] <magicaltrout> amen!
[21:03] <kwmonroe> i am complete
[21:04] <kwmonroe> now to track down the knucklehead that didn't release candidate charms to stable like 2 months ago.
[21:31] <bdx> kwmonroe: you seeing this https://paste.ubuntu.com/p/K75vMzTgrV/
[21:32] <bdx> kwmonroe: http://paste.ubuntu.com/p/Pc9kQF35Zg/
[21:33] <bdx> I dont think we will hit the error on hive because its stuck waiting on hbase
[21:33] <bdx> hbase hits it though
[21:34] <bdx> ohhhh
[21:34] <bdx> I see why
[21:35] <bdx> I needed to deploy condidate hbase too
[21:35] <bdx> my bad
[21:35] <bdx> dah
[21:35] <kwmonroe> yeah, sorry about that bdx, i'm doing a rebuild now to see if anything has changed recently.. should get those up to stable after CI runs through.
[21:35] <bdx> cool
[21:35] <kwmonroe> but yeah, everything needs to be -c candidate for now
[23:17] <beisner> hi all - in case you haven't seen our announcement on the openstack-dev ML:  1802 openstack charms release is rescheduled for Thu Mar 8.  hop over to freenode #openstack-charms if you'd like to interact or track it more closely.  cheers!
[23:20] <magicaltrout> isn't openstack 2010?
[23:20] <magicaltrout> hey beisner hows the kettle?
[23:21] <beisner> push a button, receive hot water
[23:21] <magicaltrout> amazing kit, right? ;)
[23:22] <beisner> it has single-handedly increased my personal throughput more than ... any beer ever has.
[23:22] <jhobbs> zojirushi?
[23:22] <beisner> no, baby steps.  mine's from walmart, with training wheels.  but that zojirushi looks sweet1
[23:23] <beisner> !
[23:23] <jhobbs> hehe
[23:23] <beisner> 2010's openstack isn't one that you'd enjoy much, magicaltrout
[23:24] <magicaltrout> awwww
[23:24] <magicaltrout> I'm forced to run CDK on Openstack at the moment
[23:24] <magicaltrout> Openstack makes magicaltrout sad
[23:24] <beisner> whyfor?
[23:24] <magicaltrout> not for any particular reason
[23:24] <magicaltrout> other than its just not baremetal :)
[23:25] <beisner> ha!  i hear you.
[23:25] <magicaltrout> on the plus side, CDK support for GPUs via openstack seems to work a treat
[23:25] <magicaltrout> which does make me a little less sad
[23:25] <beisner> the world still needs virtual machines.  look!  even you do right now, apparently.
[23:25] <beisner> nice
[23:25] <magicaltrout> only cause I don't get my own way all the time :P
[23:26] <magicaltrout> i'm fixing the world
[23:26] <magicaltrout> one kettle at a time!
[23:26] <beisner> word.
[23:26] <magicaltrout> that said, docker gets on my tits as well
[23:26] <magicaltrout> i'm hard to please ;)
[23:27] <beisner> is that like ruby on rails?  docker on ____ ?
[23:27] <magicaltrout> haha
[23:27] <beisner> you may have something there.
[23:28] <magicaltrout> kwmonroe... do you own a kettle?
[23:29] <beisner> gotta run.  i'll leave you to ""gpu research"" aka ethereum mining, magicaltrout ;-)
[23:29] <magicaltrout> works a treat
[23:30] <magicaltrout> on government GPUs ;)
[23:30] <magicaltrout> have fun
[23:30] <beisner> cheers
[23:31] <kwmonroe> i do not magicaltrout
[23:32] <magicaltrout> its a texan miracle!
[23:50] <bdx> kwmonroe: concerning zookeeper <-> hive, do you have any suggestions on how to go about making the relation between the two optional?
[23:50] <bdx> I have a few ideas I'm playing around with atm
[23:54] <bdx> 1) conditional 'use-zookeeper' or equivalent charm config, 2) call Hive.install() when zookeeper.joined and make a separate handler to get the zookeeper hosts and store them in unitdata so that install_hive() doesn't depend on 'zookeeper.ready'
[23:55] <bdx> I have #1 working
[23:55] <bdx> but the more I think about it, the less I like it (I'm sure you will agree here)
[23:55] <bdx> so, working on #2
[23:56] <kwmonroe> bdx: yeah, boo #1.  i fail to see how the zk relation isn't already optional.  your install handler isn't gated with zookeeper.ready: https://github.com/jamesbeedy/bigtop/blob/hive_config_add_zookeeper/bigtop-packages/src/charm/hive/layer-hive/reactive/hive.py#L66
[23:56] <bdx> ahh , yeah srry, uncommited changes
[23:56] <kwmonroe> heh, well, don't guard it, and zookeeper will be optional :)
[23:56] <bdx> I was not getting the zk properties in hive-site.xml with my add_hive branch
[23:57] <bdx> ok ... I'll keep poking around with what I have in that branch then ... possibly I'm over thinking it
[23:58] <bdx> the fact that install_hive() doesnt depend on 'zookeeper.ready'
[23:58] <kwmonroe> ah, yeah bdx.. so here's the thing.  without the @when zk.ready, you'll enter that install_hive handler over and over... and *only* when the deployment matrix changes will anything actually happen.
[23:58] <bdx> yeah
[23:59] <kwmonroe> so, i would suggest you move this https://github.com/jamesbeedy/bigtop/blob/hive_config_add_zookeeper/bigtop-packages/src/charm/hive/layer-hive/reactive/hive.py#L110,L112 up
[23:59] <kwmonroe> and add hive_install-args to the dep matrix here: https://github.com/jamesbeedy/bigtop/blob/hive_config_add_zookeeper/bigtop-packages/src/charm/hive/layer-hive/reactive/hive.py#L89
[23:59] <kwmonroe> otherwise, you're hitting this: https://github.com/jamesbeedy/bigtop/blob/hive_config_add_zookeeper/bigtop-packages/src/charm/hive/layer-hive/reactive/hive.py#L103
[23:59] <bdx> oh totally
[23:59] <kwmonroe> and returning because the matrix didn't change