Spark: Difference between revisions

From Chorke Wiki
Jump to navigation Jump to search
Line 1: Line 1:
==Spark Master==
==Spark Master==
<source lang="bash">
<source lang="bash">
sudo tee -a /etc/systemd/system/spark-master.service >/dev/null <<EOF
sudo apt -qq update;\
[Unit]
export PYSPARK_PYTHON='/usr/bin/python3';\
Description=Apache Spark Master
export SPARK_HOME='/opt/cli/spark-3.3.0-bin-hadoop3';\
Documentation=https://spark.apache.org/docs/3.3.0
export JAVA_HOME='/usr/lib/jvm/java-17-openjdk-arm64';\
Wants=network-online.target
bash <(curl -s 'https://cdn.chorke.org/exec/cli/bash/install/apache-spark-master/3.3.0.sh.txt')
After=network-online.target
After=systemd-user-sessions.service
 
[Service]
User=spark
Group=spark
Type=forking
EnvironmentFile=${SPARK_HOME}/etc/env
ExecStart=${SPARK_HOME}/sbin/start-master.sh
ExecStop=${SPARK_HOME}/sbin/stop-master.sh
 
[Install]
WantedBy=multi-user.target
EOF
</source>
</source>


Revision as of 11:19, 25 September 2022

Spark Master

sudo apt -qq update;\
export PYSPARK_PYTHON='/usr/bin/python3';\
export SPARK_HOME='/opt/cli/spark-3.3.0-bin-hadoop3';\
export JAVA_HOME='/usr/lib/jvm/java-17-openjdk-arm64';\
bash <(curl -s 'https://cdn.chorke.org/exec/cli/bash/install/apache-spark-master/3.3.0.sh.txt')

Slave Daemon

sudo apt -qq update;\
export PYSPARK_PYTHON='/usr/bin/python3';\
export SPARK_MASTER_NODE='spark://ns12-pc04:7077';\
export SPARK_HOME='/opt/cli/spark-3.3.0-bin-hadoop3';\
export JAVA_HOME='/usr/lib/jvm/java-17-openjdk-amd64';\
bash <(curl -s 'https://cdn.chorke.org/exec/cli/bash/install/apache-spark-slave/3.3.0.sh.txt')

References

Retrieved from "https://cdn.chorke.org/weke/index.php?title=Spark&oldid=8400"