sodcalculator

10 Tech How-To

Install Spark on Windows 10: Step-by-Step Guide

Guide 10-20626 · Updated · Step by step

Yes, you can install Apache Spark on Windows 10, and the process takes about 15 minutes once you have the prerequisites. This guide walks you through installing Java, Python, and Spark itself, then configuring environment variables so you can run Spark from the command line or PySpark.

What Apache Spark Is and Why Run It on Windows

Apache Spark is an open-source, distributed computing system used for large-scale data processing, machine learning, and real-time analytics. It runs on clusters of machines, but you can install it on a single Windows 10 PC for learning, development, and small-scale testing. Running Spark locally on Windows is common for data engineers and data scientists who want to develop and debug code before deploying to a Linux cluster or cloud service.

Windows is not Spark’s native environment—it was designed for Unix-like systems—but with a few configuration tweaks it works fine for local development. The main challenges are setting up Hadoop’s Windows binaries (which Spark uses for file access) and ensuring Java and Python are correctly installed and referenced.

Prerequisites: Java, Python, and Hadoop Winutils

Before installing Spark, you need three things:

  • Java 8 or 11 (64-bit) – Spark runs on the Java Virtual Machine. Java 8 is the most widely used with Spark 3.x; Java 11 also works with recent versions. You can check if Java is installed by opening a command prompt and typing java -version.
  • Python 3.8 or later (64-bit) – Required if you want to use PySpark, Spark’s Python API. Most data work uses PySpark, so install Python and add it to your PATH during installation.
  • Hadoop Windows binaries (winutils.exe) – Spark on Windows needs a file called winutils.exe and a matching hadoop.dll to access local files. These are not included with Spark; you download them from a third-party mirror (e.g., the cdh5.6.0 build) and place them in a folder like C:\hadoop\bin.

Also make sure you have at least 4 GB of free disk space and 4 GB of RAM (8 GB recommended) for Spark’s local mode.

Step-by-Step Installation

Follow these steps in order. The exact version numbers change over time, but the process remains the same.

  1. Install Java – Download the OpenJDK or Oracle JDK 8/11 installer, run it, and note the installation path (e.g., C:\Program Files\Java\jdk1.8.0_202).
  2. Install Python – Download Python from python.org, check “Add Python to PATH” during installation, and verify with python --version.
  3. Download Spark – Go to the Apache Spark download page, choose a release (e.g., 3.5.x), and select a package type that includes Hadoop (e.g., “Pre-built for Apache Hadoop 3.3 and later”). Download the .tgz file.
  4. Extract Spark – Use 7-Zip or WinRAR to extract the .tgz file to a simple path like C:\spark. Avoid spaces in the path (e.g., C:\Program Files can cause issues).
  5. Download winutils.exe – Search for “winutils.exe for Spark on Windows” and place it in C:\hadoop\bin. Also put hadoop.dll in the same folder and in C:\Windows\System32.
  6. Set environment variables – Open System Properties → Environment Variables and add:
    • SPARK_HOME = C:\spark
    • HADOOP_HOME = C:\hadoop
    • JAVA_HOME = your Java installation path (e.g., C:\Program Files\Java\jdk1.8.0_202)
    • Add %SPARK_HOME%\bin and %HADOOP_HOME%\bin to the Path variable.
  7. Test the installation – Open a new command prompt, type spark-shell (for Scala) or pyspark (for Python). You should see a welcome message and a scala> or >>> prompt.

Common Problems and Fixes

Most installation issues fall into a few categories:

  • “Error: Unable to access jarfile” – Usually means SPARK_HOME is wrong or the path has spaces. Double-check the environment variable and ensure Spark is extracted to a simple path.
  • “Failed to locate the winutils binary” – Spark can’t find winutils.exe. Confirm HADOOP_HOME points to the folder containing bin\winutils.exe, and that the file is present.
  • “Python not found” – PySpark can’t locate Python. Make sure Python is in your PATH and that the PYSPARK_PYTHON environment variable is set to python if needed.
  • Java version mismatch – Spark 3.x requires Java 8/11/17. If you have a newer Java, install an older JDK and set JAVA_HOME to it.
  • Memory errors during startup – Spark tries to allocate 1 GB by default. If your PC has less RAM, edit spark-defaults.conf (in conf folder) to reduce spark.driver.memory.

Running Your First Spark Job

Once Spark is installed, test it with a simple PySpark script. Create a file called test.py with:

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("Test").getOrCreate()
df = spark.createDataFrame([(1, "a"), (2, "b")], ["id", "letter"])
df.show()
spark.stop()

Run it with python test.py. You should see a table with two rows. This confirms that Spark can start, create DataFrames, and execute jobs locally.

For Scala, open spark-shell and type:

val data = Seq((1, "a"), (2, "b"))
val df = data.toDF("id", "letter")
df.show()

Alternatives to Native Windows Installation

If the manual setup feels fragile, you have other options:

  • Windows Subsystem for Linux (WSL2) – Install Ubuntu under WSL2 and then install Spark inside Linux. This avoids winutils entirely and mirrors a production environment more closely.
  • Docker Desktop – Run a pre-built Spark image (e.g., bitnami/spark) in a container. This is the fastest way to get a clean, isolated Spark environment.
  • Databricks Community Edition – A free cloud service that gives you a Spark cluster without any local installation. Good for learning, but requires an internet connection and account.
  • Use a Linux virtual machine – Tools like VirtualBox can host a Linux VM with Spark pre-installed. More resource-heavy but very reliable.

Frequently Asked Questions

Do I need Hadoop installed to run Spark on Windows?

No, you only need the Hadoop Windows binaries (winutils.exe and hadoop.dll), not the full Hadoop framework. Spark uses these small files to interact with the local file system. You do not need to install HDFS or YARN.

Can I install Spark on Windows 11?

Yes, the same steps work on Windows 11. The process is identical; just ensure your Java and Python are 64-bit and compatible with Spark 3.x.

Why does Spark say “Unable to load native-hadoop library”?

This warning appears when the native Hadoop library is not found. It is not fatal—Spark will still run, but slower for some file operations. Placing hadoop.dll in System32 or setting HADOOP_HOME correctly usually resolves it.

How much RAM does Spark need on Windows?

For local development, 4 GB of RAM is the practical minimum, but 8 GB is recommended. Spark’s default driver memory is 1 GB, so even a modest machine can run simple jobs. Heavy workloads on large datasets will need more.

Is PySpark the same as Spark?

PySpark is Spark’s Python API. The core engine is written in Scala and runs on the JVM, but PySpark lets you write Spark applications in Python. For most beginners, PySpark is the easiest entry point.

How do I uninstall Spark from Windows?

Simply delete the Spark extraction folder, remove the SPARK_HOME, HADOOP_HOME, and JAVA_HOME environment variables (unless you need Java for other tools), and delete the C:\hadoop folder. No registry changes are needed.

Related

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →