| heckles1000 | #luigi | 04:04 |
|---|---|---|
| heckles1000 | whoops | 04:04 |
| === frankban|afk is now known as frankban | ||
| === disposable3 is now known as disposable2 | ||
| bdx | kwmonroe: I'm creating the issue for the bigtop project on apache jira | 16:45 |
| bdx | should I tag it as "New Feature"? | 16:45 |
| bdx | kwmonroe: https://issues.apache.org/jira/browse/BIGTOP-3007 | 16:49 |
| bdx | kwmonroe: let me know if theres anything else you want/don't want to see on there and I'll add/remove it | 16:51 |
| kwmonroe | perfect bdx! thank you!! | 16:53 |
| bdx | kwmonroe: np, thank you | 16:53 |
| bdx | kwmonroe: would I be able to fork->branch the juju-solutions/bigtop repo and just add those properties to the hive config to get started working on implementing those bits between the charms? | 16:54 |
| bdx | or is there something else | 16:54 |
| kwmonroe | bdx: you probably want to fork upstream, right? https://github.com/apache/bigtop | 16:56 |
| bdx | ahh, but the juju-solutions fork has the layers ..... | 16:57 |
| bdx | hmmm, possibly I need to do a bit more digging to understand how these pieces fit together | 16:58 |
| kwmonroe | bdx: upstream has the layers too: https://github.com/apache/bigtop/tree/master/bigtop-packages/src/charm | 17:01 |
| kwmonroe | fwiw, the upstream layers are the source of truth for the charms in the store -- not our juju-soln/bigtop fork | 17:01 |
| bdx | ahhh gotcha | 17:02 |
| bdx | kwmonroe: should these properties be conditional in the xml template similar to hbase.zookeeper.quorum? | 17:14 |
| bdx | im thinking so | 17:14 |
| bdx | ahh, possibly hive.support.concurrency should default to false | 17:15 |
| bdx | and hive.zookeeper.quorum shoudl get the conditional | 17:16 |
| kwmonroe | yeah bdx, i think you're on the right track.. for reference, here's how we added 3 new options in the puppet scripts: https://github.com/apache/bigtop/pull/238/files | 17:20 |
| kwmonroe | bdx: you'd do something similar to initialize the vars in hadoop_hive's init.pp: https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/modules/hadoop_hive/manifests/init.pp#L48 | 17:21 |
| kwmonroe | and then conditionally use those vars in the hive-site.xml: https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/modules/hadoop_hive/templates/hive-site.xml | 17:21 |
| kwmonroe | bdx: as for the default values, i would set concurrency to false by default (since that is the default): https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.support.concurrency | 17:24 |
| kwmonroe | and make hive.zk.quorum an empty string | 17:25 |
| kwmonroe | ^^ both of those in hadoop_hive's init.pp common_config class def | 17:25 |
| kwmonroe | bdx: then write out the concurrency property and conditional zk.quorum property in hadoop_hive's templates/hive-site.xml | 17:26 |
| bdx | done | 17:26 |
| bdx | https://github.com/jamesbeedy/bigtop/blob/hive_config_add_zookeeper/bigtop-deploy/puppet/modules/hadoop_hive/manifests/init.pp#L50,L51 | 17:27 |
| bdx | https://github.com/jamesbeedy/bigtop/blob/hive_config_add_zookeeper/bigtop-deploy/puppet/modules/hadoop_hive/templates/hive-site.xml#L45,L57 | 17:27 |
| kwmonroe | perfect | 17:28 |
| kwmonroe | see?!?! big data is easy! | 17:28 |
| kwmonroe | can i get an amen magicaltrout? | 17:28 |
| === frankban is now known as frankban|afk | ||
| === chrisp_ is now known as chrisp262 | ||
| bdx | kwmonroe: https://github.com/jamesbeedy/bigtop/blob/hive_config_add_zookeeper/bigtop-packages/src/charm/hive/layer-hive/lib/charms/layer/bigtop_hive.py | 18:20 |
| bdx | and https://github.com/jamesbeedy/bigtop/blob/hive_config_add_zookeeper/bigtop-packages/src/charm/hive/layer-hive/reactive/hive.py | 18:22 |
| boucherv | Some issue with the snaps website and api ? https://snapcraft.io/ - 503 Service Temporarily Unavailable nginx/1.13.7 | 18:29 |
| kwmonroe | looks great bdx! when you're ready, you can PR that against the bigtop repo. be sure to follow the PR title guidelines so jira will auto-link your pr to the ticket: https://github.com/juju-solutions/bigdata-community/wiki/Contributing#pull-request-formatting | 18:29 |
| bdx | kwmonroe: awesome, thanks | 18:30 |
| kwmonroe | boucherv: i was seeing 502/503 too, but it seems to be back for me now. | 18:46 |
| boucherv | kwmonroe: Yes It seems to be back now! ty | 18:47 |
| bdx | kwmonroe: https://github.com/apache/bigtop/blob/master/bigtop-packages/src/charm/hive/layer-hive/lib/charms/layer/bigtop_hive.py#L61,L63 | 19:07 |
| bdx | (self.dist_config.path('hive_conf') / 'hive-env.sh.template') | 19:07 |
| bdx | I cant seem to find hive-env.sh, or hive-env.sh.template | 19:07 |
| bdx | should I create this file? | 19:07 |
| bdx | I feel like I'm missing something, as the charm should have hit the error I'm seeing long ago if that file didnt exist | 19:08 |
| bdx | possibly I borked something https://paste.ubuntu.com/p/W9tN3Ygyn9/ | 19:09 |
| kwmonroe | bdx: you are correct... i can't find ref to that template either. wth! /me looks deeper down the rabbit hole. | 19:30 |
| bdx | kwmonroe: https://paste.ubuntu.com/p/CSscBbmwtf/ | 20:09 |
| bdx | when I deploy the upstream hive charm | 20:09 |
| bdx | I see the files | 20:09 |
| bdx | and it doesnt error | 20:10 |
| zeestrat | rick_h: Are requests to subscribe to juju@lists.ubuntu.com moderated or should you get a confirmation mail asap? Trying to change my mail, but getting ain't no confirmation love :( | 20:20 |
| rick_h | zeestrat: thought it just auto went through. | 20:36 |
| zeestrat | rick_h: Alright, thanks. I'll give it a day before I start nagging. | 20:42 |
| magicaltrout | i've changed my email address on the juju list before | 20:55 |
| magicaltrout | it took about 3 seconds | 20:55 |
| kwmonroe | aight bdx, i think i know what's up. first of all, those template files come from the upstream tgz that gets used during the bigtop deb build, specifically here: https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hive/install_hive.sh#L161 | 20:57 |
| bdx | gotcha, was just figuring that out myself | 20:57 |
| kwmonroe | so bigtop doesn't explicitly have a hive-env.sh like it does for hive-site.xml, but it does get there from the deb. | 20:57 |
| kwmonroe | bdx: the next part was a bit of a mystery.. i was able to confirm the failure when i used hive-21 (from the candidate channel) and hadoop-plugin-31 (from the stable channel) | 20:58 |
| kwmonroe | the problem is that the stable channel plugin is still set to use bigtop-1.2.0, whereas the candidate charms are bigtop-1.2.1. | 20:58 |
| kwmonroe | this is a problem because hive will set up it's puppet heiradata on initial install and get ready to run puppet apply using the config from bigtop-1.2.1. | 20:59 |
| kwmonroe | then the plugin comes along and overwrites all that config with bigtop-1.2.0 | 20:59 |
| kwmonroe | when the plugin does this, the puppet config is no longer including the "hive" role, and so puppet apply never does an "apt install hive" | 20:59 |
| bdx | oh man lol | 21:00 |
| kwmonroe | without installing the pacakge, puppet apply succeeds, but since hive didn't actually get installed, there's no /etc/hive/conf, and hence the failure in the hive install charm method | 21:00 |
| bdx | nice | 21:00 |
| kwmonroe | so, to confirm that, would you confirm if you used a stable hadoop-plugin with a candidate (or beta / edge) version of the hive charm? | 21:00 |
| bdx | that makes sense from what little I know about the process | 21:00 |
| bdx | yes, I'll give that a try now | 21:01 |
| bdx | can I just build hive from upstream apache/bigtop and use that? | 21:01 |
| magicaltrout | you make kwmonroe when you suggest such things | 21:02 |
| magicaltrout | cry | 21:02 |
| magicaltrout | missed the important word | 21:02 |
| magicaltrout | should drink less gin | 21:02 |
| kwmonroe | you sure can bdx -- but if you do, make sure you use hadoop-plugin-33 (from candidate) so everybody's on bigtop-1.2.1 | 21:02 |
| bdx | ok | 21:02 |
| kwmonroe | heh magicaltrout. big data is still easy! where's my amen? | 21:02 |
| magicaltrout | amen! | 21:03 |
| kwmonroe | i am complete | 21:03 |
| kwmonroe | now to track down the knucklehead that didn't release candidate charms to stable like 2 months ago. | 21:04 |
| bdx | kwmonroe: you seeing this https://paste.ubuntu.com/p/K75vMzTgrV/ | 21:31 |
| bdx | kwmonroe: http://paste.ubuntu.com/p/Pc9kQF35Zg/ | 21:32 |
| bdx | I dont think we will hit the error on hive because its stuck waiting on hbase | 21:33 |
| bdx | hbase hits it though | 21:33 |
| bdx | ohhhh | 21:34 |
| bdx | I see why | 21:34 |
| bdx | I needed to deploy condidate hbase too | 21:35 |
| bdx | my bad | 21:35 |
| bdx | dah | 21:35 |
| kwmonroe | yeah, sorry about that bdx, i'm doing a rebuild now to see if anything has changed recently.. should get those up to stable after CI runs through. | 21:35 |
| bdx | cool | 21:35 |
| kwmonroe | but yeah, everything needs to be -c candidate for now | 21:35 |
| beisner | hi all - in case you haven't seen our announcement on the openstack-dev ML: 1802 openstack charms release is rescheduled for Thu Mar 8. hop over to freenode #openstack-charms if you'd like to interact or track it more closely. cheers! | 23:17 |
| magicaltrout | isn't openstack 2010? | 23:20 |
| magicaltrout | hey beisner hows the kettle? | 23:20 |
| beisner | push a button, receive hot water | 23:21 |
| magicaltrout | amazing kit, right? ;) | 23:21 |
| beisner | it has single-handedly increased my personal throughput more than ... any beer ever has. | 23:22 |
| jhobbs | zojirushi? | 23:22 |
| beisner | no, baby steps. mine's from walmart, with training wheels. but that zojirushi looks sweet1 | 23:22 |
| beisner | ! | 23:23 |
| jhobbs | hehe | 23:23 |
| beisner | 2010's openstack isn't one that you'd enjoy much, magicaltrout | 23:23 |
| magicaltrout | awwww | 23:24 |
| magicaltrout | I'm forced to run CDK on Openstack at the moment | 23:24 |
| magicaltrout | Openstack makes magicaltrout sad | 23:24 |
| beisner | whyfor? | 23:24 |
| magicaltrout | not for any particular reason | 23:24 |
| magicaltrout | other than its just not baremetal :) | 23:24 |
| beisner | ha! i hear you. | 23:25 |
| magicaltrout | on the plus side, CDK support for GPUs via openstack seems to work a treat | 23:25 |
| magicaltrout | which does make me a little less sad | 23:25 |
| beisner | the world still needs virtual machines. look! even you do right now, apparently. | 23:25 |
| beisner | nice | 23:25 |
| magicaltrout | only cause I don't get my own way all the time :P | 23:25 |
| magicaltrout | i'm fixing the world | 23:26 |
| magicaltrout | one kettle at a time! | 23:26 |
| beisner | word. | 23:26 |
| magicaltrout | that said, docker gets on my tits as well | 23:26 |
| magicaltrout | i'm hard to please ;) | 23:26 |
| beisner | is that like ruby on rails? docker on ____ ? | 23:27 |
| magicaltrout | haha | 23:27 |
| beisner | you may have something there. | 23:27 |
| magicaltrout | kwmonroe... do you own a kettle? | 23:28 |
| beisner | gotta run. i'll leave you to ""gpu research"" aka ethereum mining, magicaltrout ;-) | 23:29 |
| magicaltrout | works a treat | 23:29 |
| magicaltrout | on government GPUs ;) | 23:30 |
| magicaltrout | have fun | 23:30 |
| beisner | cheers | 23:30 |
| kwmonroe | i do not magicaltrout | 23:31 |
| magicaltrout | its a texan miracle! | 23:32 |
| bdx | kwmonroe: concerning zookeeper <-> hive, do you have any suggestions on how to go about making the relation between the two optional? | 23:50 |
| bdx | I have a few ideas I'm playing around with atm | 23:50 |
| bdx | 1) conditional 'use-zookeeper' or equivalent charm config, 2) call Hive.install() when zookeeper.joined and make a separate handler to get the zookeeper hosts and store them in unitdata so that install_hive() doesn't depend on 'zookeeper.ready' | 23:54 |
| bdx | I have #1 working | 23:55 |
| bdx | but the more I think about it, the less I like it (I'm sure you will agree here) | 23:55 |
| bdx | so, working on #2 | 23:55 |
| kwmonroe | bdx: yeah, boo #1. i fail to see how the zk relation isn't already optional. your install handler isn't gated with zookeeper.ready: https://github.com/jamesbeedy/bigtop/blob/hive_config_add_zookeeper/bigtop-packages/src/charm/hive/layer-hive/reactive/hive.py#L66 | 23:56 |
| bdx | ahh , yeah srry, uncommited changes | 23:56 |
| kwmonroe | heh, well, don't guard it, and zookeeper will be optional :) | 23:56 |
| bdx | I was not getting the zk properties in hive-site.xml with my add_hive branch | 23:56 |
| bdx | ok ... I'll keep poking around with what I have in that branch then ... possibly I'm over thinking it | 23:57 |
| bdx | the fact that install_hive() doesnt depend on 'zookeeper.ready' | 23:58 |
| kwmonroe | ah, yeah bdx.. so here's the thing. without the @when zk.ready, you'll enter that install_hive handler over and over... and *only* when the deployment matrix changes will anything actually happen. | 23:58 |
| bdx | yeah | 23:58 |
| kwmonroe | so, i would suggest you move this https://github.com/jamesbeedy/bigtop/blob/hive_config_add_zookeeper/bigtop-packages/src/charm/hive/layer-hive/reactive/hive.py#L110,L112 up | 23:59 |
| kwmonroe | and add hive_install-args to the dep matrix here: https://github.com/jamesbeedy/bigtop/blob/hive_config_add_zookeeper/bigtop-packages/src/charm/hive/layer-hive/reactive/hive.py#L89 | 23:59 |
| kwmonroe | otherwise, you're hitting this: https://github.com/jamesbeedy/bigtop/blob/hive_config_add_zookeeper/bigtop-packages/src/charm/hive/layer-hive/reactive/hive.py#L103 | 23:59 |
| bdx | oh totally | 23:59 |
| kwmonroe | and returning because the matrix didn't change | 23:59 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!