sodcalculator

10 Tech How-To

Install Spark on Ubuntu: Step-by-Step Guide

Guide 10-13115 · Updated · Step by step

To install Apache Spark on Ubuntu, download the latest Spark release from the official Apache website, extract it to a directory, and configure environment variables like SPARK_HOME and PATH. This guide walks through the entire process, including installing Java, verifying the installation, and starting a simple Spark session.

What is Apache Spark?

Apache Spark is an open-source, unified analytics engine designed for large-scale data processing. It provides an interface for programming entire clusters with implicit data parallelism and fault tolerance. Spark is known for its speed—it runs programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk—thanks to its in-memory computing and optimized query execution. It supports multiple languages (Scala, Java, Python, R) and includes libraries for SQL, streaming, machine learning, and graph processing.

Prerequisites

Before you begin, ensure your Ubuntu system meets these requirements:

  • Java 8/11/17 – Spark requires Java; version 8 is the classic choice, but 11 and 17 are supported in recent releases.
  • Python (optional) – Needed if you want to use PySpark. Most Ubuntu installations have Python 3 pre-installed.
  • Internet connection – To download Spark and Java packages.
  • At least 4 GB RAM – Spark is memory-hungry; more is better for real workloads.

Step 1: Install Java

First, update your package list and install OpenJDK. The command below installs OpenJDK 11, which works with Spark 3.x:

  1. Run sudo apt update to refresh package lists.
  2. Install Java with sudo apt install openjdk-11-jdk -y.
  3. Verify the installation: java -version should show the Java version.
  4. Set JAVA_HOME in your shell profile (e.g., ~/.bashrc): export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 (adjust path if needed).

Step 2: Download and Extract Spark

Go to the Apache Spark download page and copy the link for the latest stable release (e.g., spark-3.5.1-bin-hadoop3.tgz). Then in your terminal:

  1. Change to a directory where you want Spark installed, e.g., cd /opt (you may need sudo for permissions).
  2. Download the tarball: sudo wget https://dlcdn.apache.org/spark/spark-3.5.1/spark-3.5.1-bin-hadoop3.tgz (replace with the actual URL).
  3. Extract it: sudo tar -xzf spark-3.5.1-bin-hadoop3.tgz.
  4. Rename for simplicity: sudo mv spark-3.5.1-bin-hadoop3 spark.

Step 3: Configure Environment Variables

Add Spark to your PATH and set SPARK_HOME so you can run Spark commands from anywhere. Edit your ~/.bashrc file:

  1. Open the file: nano ~/.bashrc.
  2. Add these lines at the end:
export SPARK_HOME=/opt/spark
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin
  1. Save and exit, then run source ~/.bashrc to apply changes.
  2. Verify: echo $SPARK_HOME should print /opt/spark.

Step 4: Verify the Installation

Run Spark’s built-in example to ensure everything works:

  1. Start the Spark shell with spark-shell (Scala) or pyspark (Python).
  2. In the shell, run a simple command like sc.parallelize(1 to 10).sum() (Scala) or sc.parallelize(range(1,11)).sum() (Python). You should see the output 55.
  3. Exit with :quit (Scala) or exit() (Python).

If you see the result, Spark is installed and running correctly.

Common Issues and Fixes

  • Java not found – Ensure JAVA_HOME is set correctly; run echo $JAVA_HOME and check the path exists.
  • Permission denied – If you installed Spark in /opt, you may need to change ownership: sudo chown -R $USER:$USER /opt/spark.
  • Memory errors – Spark defaults to 1 GB executor memory; if your machine has less, adjust SPARK_MEM in spark-env.sh.
  • Hadoop native library warnings – These are harmless; they appear because native libraries aren’t installed. You can ignore them for local use.

Frequently Asked Questions

Do I need Hadoop to run Spark?

No. Spark can run standalone without Hadoop. The pre-built package includes Hadoop client libraries, but you don’t need a Hadoop cluster. For local development, Spark runs in local mode using threads.

Can I install Spark via apt or snap?

Ubuntu’s official repositories do not include Spark, and snap packages are not officially maintained. The recommended way is to download the binary from Apache. Some third-party PPAs exist, but they may be outdated or insecure.

How do I start Spark on a cluster?

To run a multi-node cluster, you need to configure workers and masters. Start the master with start-master.sh, then workers with start-worker.sh spark://<master-ip>:7077. For a quick start, use start-all.sh to launch a standalone cluster on the same machine.

What is the difference between Spark and Hadoop MapReduce?

Spark keeps data in memory across operations, while MapReduce writes intermediate results to disk. This makes Spark much faster for iterative algorithms and interactive queries. However, Spark uses more memory, so resource management is important.

How do I uninstall Spark?

Simply delete the Spark directory (e.g., sudo rm -rf /opt/spark) and remove the environment variables you added to ~/.bashrc. No system packages are installed, so uninstalling is clean.

Next Steps

Once Spark is running, explore its capabilities by reading the official documentation, trying the examples in the examples folder, or setting up a notebook environment like Jupyter with PySpark. For production, consider using a cluster manager like YARN or Kubernetes, and monitor resource usage to avoid out-of-memory errors.

Related

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →