Spark: Difference between revisions
Jump to navigation
Jump to search
No edit summary |
|||
| Line 74: | Line 74: | ||
* [https://github.com/juju-solutions/layer-apache-spark/blob/master/templates/master-systemd.conf Spark » Install Master as a Daemon] | * [https://github.com/juju-solutions/layer-apache-spark/blob/master/templates/master-systemd.conf Spark » Install Master as a Daemon] | ||
* [https://stackoverflow.com/questions/59249135/ Spark » Install Slave as a Daemon] | * [https://stackoverflow.com/questions/59249135/ Spark » Install Slave as a Daemon] | ||
* [https://spark.apache.org/docs/latest/submitting-applications.html Spark » Deploy » Spark Submit] | |||
* [https://stackoverflow.com/questions/40166056/ Spark » As a Linux Service] | * [https://stackoverflow.com/questions/40166056/ Spark » As a Linux Service] | ||
* [https://spark.apache.org/docs/latest/api/python/index.html Spark » API Docs » Python] | |||
* [https://spark.apache.org/docs/latest/api/scala/org/apache/spark/index.html Spark » API Docs » Scala] | |||
* [https://spark.apache.org/docs/latest/api/java/index.html Spark » API Docs » Java] | |||
* [https://spark.apache.org/docs/latest/api/sql/index.html Spark » API Docs » SQL] | |||
* [https://spark.apache.org/docs/latest/api/R/index.html Spark » API Docs » R] | |||
|} | |} | ||
Revision as of 05:03, 25 September 2022
Spark Master
sudo tee -a /etc/systemd/system/spark-master.service >/dev/null <<EOF
[Unit]
Description=Apache Spark Master
Documentation=https://spark.apache.org/docs/3.3.0
Wants=network-online.target
After=network-online.target
After=systemd-user-sessions.service
[Service]
User=spark
Type=forking
EnvironmentFile=${SPARK_HOME}/etc/.env
ExecStart=${SPARK_HOME}/sbin/start-master.sh
ExecStop=${SPARK_HOME}/sbin/stop-master.sh
[Install]
WantedBy=multi-user.target
EOF
Slave Daemon
sudo tee -a /etc/systemd/system/spark-slave.service >/dev/null <<EOF
[Unit]
Description=Apache Spark Slave
Documentation=https://spark.apache.org/docs/3.3.0
Wants=network-online.target
After=network-online.target
[Service]
User=spark
Type=forking
EnvironmentFile=${SPARK_HOME}/etc/.env
WorkingDirectory=${SPARK_HOME}/sbin
ExecStart=${SPARK_HOME}/sbin/start-slave.sh spark://10.20.22.10:7077
ExecStop=${SPARK_HOME}/sbin/stop-slave.sh
Restart=on-failure
RestartSec=10s
[Install]
WantedBy=multi-user.target
EOF