| === salmankhan1 is now known as salmankhan | ||
| elmaciej | Hi! Can someone quickly explain me how to write my charm - I want it to connect to existing spark client, copy jar file to spark master and run a spark job. It has to be a charm actually not a script . Can someone advise me how to start it - I'm reading documentation and honestly it totaly confused me | 14:27 |
|---|---|---|
| kwmonroe | elmaciej: you mentioned a req for being a charm, but the spark charm has a 'spark-submit' action that may do what you want. the readme talks about the spark-submit action here: https://github.com/apache/bigtop/tree/master/bigtop-packages/src/charm/spark/layer-spark#actions | 21:43 |
| kwmonroe | you can pass a jar (local, remote, hdfs -- whatever spark-submit accepts as it's jar param), give it runtime options, and schedule it to run with cron if needed. | 21:45 |
| kwmonroe | elmaciej: if that doesn't work for you, the tengu team has written a subordinate charm called 'spark-job' that sits on a deployed spark charm. it takes a job url and runs that on the spark cluster: https://jujucharms.com/u/tengu-team/spark-job | 21:47 |
| kwmonroe | elmaciej: if that doesn't suite your needs but you still want to take the subordinate charm approach, i suggest modelling your charm off of their source: https://github.com/tengu-team/layer-python-spark-job | 21:48 |
| kwmonroe | elmaciej: and if that's still not the route you'd like to go, the spark interface has a short client snippet for charms that need to "require" spark: https://github.com/juju-solutions/interface-spark | 21:56 |
| kwmonroe | all this is predicated on some knowledge of charm authoring -- if you're just starting out, i'd recommend trying a hello-world with help from https://jujucharms.com/docs/stable/developer-getting-started, and asking specifics here -- we can def help you! | 22:01 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!