sodcalculator

10 Tech How-To

Large NASA Earth Science Datasets on a Laptop

Guide 10-47059 · Updated · Step by step

Working with large NASA Earth science datasets—like MODIS, GPM, or GEDI—on a laptop is possible if you use the right tools and strategies. This guide shows you how to download, subset, and analyze these datasets without needing a supercomputer. You’ll learn to use NASA’s cloud services, efficient file formats, and Python libraries to keep memory usage low and processing fast.

What You’ll Need

  • A laptop with at least 8 GB RAM (16 GB recommended) and 20 GB free disk space.
  • Python 3.8+ installed, along with pip.
  • Earthdata Login account (free) from NASA.
  • Optional: conda or mamba for environment management.

Step-by-Step

  1. Set up your Earthdata Login. Go to urs.earthdata.nasa.gov and register. This is required for downloading most NASA data. After registering, you’ll need to authorize specific data products (e.g., GPM, MODIS) by visiting their landing pages and clicking “Authorize” or “Get Data”.
  2. Install essential Python packages. Open a terminal and run:
    pip install earthaccess h5py netCDF4 xarray rioxarray geopandas matplotlib tqdm

    earthaccess handles NASA authentication and downloads. xarray and rioxarray handle multidimensional data. geopandas is for vector data like GEDI footprints.

  3. Authenticate using earthaccess. In Python, run:
    import earthaccess
    earthaccess.login(strategy="interactive")

    This will prompt for your Earthdata username and password. It stores credentials in a netrc file for future sessions.

  4. Search for data granules. Use earthaccess.search_data() to find granules. For example, to get GPM precipitation data for a specific date and bounding box:
    results = earthaccess.search_data(
        short_name="GPM_3IMERGHH",
        bounding_box=(-120, 30, -110, 40),
        temporal=("2023-06-01", "2023-06-02")
    )

    This returns a list of data links. For large time ranges, use earthaccess.download() to fetch files in parallel.

  5. Download only what you need. Instead of downloading entire global granules, use earthaccess.download() with a subsetting option if available. For example, for GPM, you can request a spatial subset via the GES DISC API. Alternatively, download full granules but choose a coarse resolution product (e.g., 0.1° instead of 0.1° hourly). For MODIS, use the AppEEARS API to extract time series at points.
  6. Use cloud-optimized formats. Many NASA datasets are now available as Cloud Optimized GeoTIFFs (COGs) or Zarr. These allow you to read only the parts you need without downloading the whole file. For example, use xarray.open_zarr() with a remote URL if the data is in S3. For COGs, use rioxarray.open_rasterio() with a range header to read a subset.
  7. Process in chunks. When reading large netCDF or HDF5 files, use xarray.open_dataset() with chunks={} to enable lazy loading. This way, operations like mean() are computed in chunks, not all at once. For example:
    ds = xr.open_dataset("file.nc", chunks={"time": 100})

    This keeps memory usage low.

  8. Use dask for parallel processing. Install dask and distributed. When you open a dataset with chunks, xarray automatically uses dask arrays. You can then use .compute() to trigger computation. For large operations, consider using a local dask cluster:
    from dask.distributed import Client
    client = Client()

    This uses all CPU cores.

  9. Aggregate and reduce data. Instead of loading all time steps, compute daily or monthly means on the fly. For example, to get a monthly mean from hourly GPM data:
    ds_monthly = ds.resample(time="1MS").mean()

    This reduces the data size before you save or plot.

  10. Save processed results in efficient formats. Use to_netcdf() with compression (zlib=True, complevel=4) or save as Parquet for tabular data. This reduces disk usage and speeds up future loads.

Common Mistakes

  • Downloading entire granules when you only need a small region. This wastes bandwidth and disk. Use subsetting APIs or read remote COGs.
  • Loading entire files into memory with load(). Avoid calling .load() on large datasets. Keep them as dask arrays.
  • Ignoring data types. Some datasets use float32 or int16; converting to float64 doubles memory. Keep original dtypes.
  • Not using chunking. Without chunking, xarray reads the whole file at once, causing memory errors.
  • Forgetting to close files. Use with xr.open_dataset(...) as ds: to ensure files are closed.

Tips

  • Use earthaccess to automatically handle authentication and download retries.
  • For repeated analyses, store intermediate results as compressed netCDF or Parquet.
  • Use rioxarray to clip data to a shapefile using clip_box() or clip().
  • When working with GEDI or ICESat-2, use the h5py library to read specific beams and shots without loading the whole file.
  • Consider using NASA’s Harmony API for on-the-fly subsetting and reprojection.

Troubleshooting

  • Authentication fails. Re-run earthaccess.login() and ensure you’ve authorized the specific product on the Earthdata website. Check if your password has special characters that need escaping in the netrc file.
  • MemoryError during processing. Reduce chunk sizes, use dask with a distributed client, or process a smaller time range. Also, close any other applications.
  • Download speed is slow. Use earthaccess.download() with parallel=True (default) and ensure your internet connection is stable. For large files, use the wget command with -c to resume.
  • Cannot open HDF5 files. Install h5py and netCDF4. Some older HDF4 files require the pyhdf package.
  • CRS issues when plotting. Use rioxarray to set the CRS and reproject if needed.

Frequently Asked Questions

  1. Can I work with MODIS data on a laptop? Yes, but use the MODIS products that are tiled (e.g., MOD13Q1) and download only the tiles you need. Use earthaccess to subset by tile ID.
  2. What is the best way to handle GEDI L2A data? GEDI L2A files are large (hundreds of MB). Use h5py to read only the beams and shots you need, and extract the variables of interest (e.g., canopy height) into a smaller DataFrame.
  3. Do I need a GPU? No, most analysis is CPU-bound. GPU is only useful for deep learning on imagery, which is not typical for basic analysis.
  4. Can I use Google Colab for free? Yes, Colab provides a free VM with ~12 GB RAM. You can use the same steps, but you’ll need to upload your Earthdata credentials. Be mindful of session timeouts.
  5. How do I avoid downloading the same data twice? Use a local cache directory and check if the file exists before downloading. earthaccess does not cache by default, so you can implement a simple check.
  6. Is it better to use the cloud (AWS) to process NASA data? For very large analyses, yes. NASA hosts some datasets on AWS, and you can use AWS Lambda or EC2 to process them near the data. But for most laptop-sized analyses, the steps above work fine.

Related

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →