Spark: Difference between revisions
Jump to navigation
Jump to search
| Line 63: | Line 63: | ||
* [https://stackoverflow.com/questions/59249135/ Spark » Install Slave as a Daemon] | * [https://stackoverflow.com/questions/59249135/ Spark » Install Slave as a Daemon] | ||
* [https://stackoverflow.com/questions/40166056/ Spark » As a Linux Service] | * [https://stackoverflow.com/questions/40166056/ Spark » As a Linux Service] | ||
* [ | * [https://www.baeldung.com/apache-spark Spark » Introduction] | ||
* [[Machine Learning]] | * [[Machine Learning]] | ||
* [https://hive.apache.org/ Apache Hive] | * [https://hive.apache.org/ Apache Hive] | ||
| Line 72: | Line 72: | ||
* [https://unix.stackexchange.com/questions/388483/ Delay Systemd Service If File Exist] | * [https://unix.stackexchange.com/questions/388483/ Delay Systemd Service If File Exist] | ||
* [https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Conditions%20and%20Asserts System Unit Conditions & Asserts] | * [https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Conditions%20and%20Asserts System Unit Conditions & Asserts] | ||
* [[VS Code on iPad Pro]] | |||
* [[Keycloak]] | * [[Keycloak]] | ||
* [[GraphQL]] | * [[GraphQL]] | ||
Revision as of 11:57, 25 September 2022
export PYSPARK_PYTHON='/usr/bin/python3';\
export SPARK_HOME='/opt/cli/spark-3.3.0-bin-hadoop3';\
export JAVA_HOME='/usr/lib/jvm/java-17-openjdk-amd64';\
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin
spark-shell pyspark
http://localhost:8080/ http://localhost:7077/ http://localhost:4040/
Master Node
sudo apt -qq update;\
export PYSPARK_PYTHON='/usr/bin/python3';\
export SPARK_HOME='/opt/cli/spark-3.3.0-bin-hadoop3';\
export JAVA_HOME='/usr/lib/jvm/java-17-openjdk-arm64';\
bash <(curl -s 'https://cdn.chorke.org/exec/cli/bash/install/apache-spark-master/3.3.0.sh.txt')
sudo systemctl daemon-reload sudo systemctl enable spark-master.service sudo systemctl start spark-master.service sudo systemctl status spark-master.service
Worker Node
sudo apt -qq update;\
export PYSPARK_PYTHON='/usr/bin/python3';\
export SPARK_MASTER='spark://ns12-pc04:7077';\
export SPARK_HOME='/opt/cli/spark-3.3.0-bin-hadoop3';\
export JAVA_HOME='/usr/lib/jvm/java-17-openjdk-amd64';\
bash <(curl -s 'https://cdn.chorke.org/exec/cli/bash/install/apache-spark-slave/3.3.0.sh.txt')
sudo systemctl daemon-reload sudo systemctl enable spark-slave.service sudo systemctl start spark-slave.service sudo systemctl status spark-slave.service
References
|
| ||