sodcalculator

10 Tech How-To

Run AI Models With Limited GPU Memory

Guide 10-53443 · Updated · Step by step

This guide shows you how to run large AI models on a GPU with limited VRAM by using techniques like quantization, offloading, and memory-efficient loading. You’ll learn practical steps that let you run models that would otherwise exceed your graphics card’s memory.

What You’ll Need

  • A computer with an NVIDIA GPU (CUDA support) or an Apple Silicon Mac (MPS).
  • Python 3.8 or later installed.
  • Pip package manager.
  • Enough system RAM (at least as much as the model size you plan to run).
  • Basic familiarity with the command line.

Step-by-Step

  1. Check your GPU memory. Run nvidia-smi (Windows/Linux) or system_profiler SPDisplaysDataType (macOS) to see how much VRAM you have. This tells you the hard limit you’re working with.
  2. Pick a model that fits. As a rule of thumb, a 7B parameter model in 16-bit precision needs about 14 GB of VRAM. For 8 GB VRAM, look for 3B or smaller models, or use quantization (next step).
  3. Use quantization to shrink the model. Quantization reduces the precision of weights (e.g., from 16-bit to 8-bit or 4-bit). This can cut memory usage by 50–75% with minimal quality loss. Use libraries like bitsandbytes for 4-bit/8-bit loading in PyTorch.
  4. Load the model in 4-bit with bitsandbytes. In your Python script, add:
    from transformers import AutoModelForCausalLM, BitsAndBytesConfig
    quant_config = BitsAndBytesConfig(load_in_4bit=True)
    model = AutoModelForCausalLM.from_pretrained("model-name", quantization_config=quant_config)
  5. Enable CPU offloading. If the model still doesn’t fit, offload some layers to system RAM. In Hugging Face Transformers, set device_map="auto" to let the library place layers on GPU and CPU as needed.
  6. Use gradient checkpointing for training. If you’re fine-tuning, enable model.gradient_checkpointing_enable() to trade compute for memory. This saves activations rather than storing them all.
  7. Reduce batch size. For inference or training, set batch_size=1 or the smallest that works. This drastically cuts memory usage.
  8. Use mixed precision. For training, set torch.cuda.amp.autocast() to use 16-bit floats for forward/backward passes, reducing memory and speeding up computation.
  9. Clear GPU cache. Before loading a model, run torch.cuda.empty_cache() to free up unused memory. Also close other GPU-using applications.
  10. Try a smaller variant. If all else fails, look for distilled or pruned versions of the model. For example, use a 3B model instead of 7B.

Common Mistakes

  • Ignoring system RAM. CPU offloading requires enough system RAM. If you have 8 GB VRAM but only 8 GB system RAM, offloading will fail.
  • Not checking for CUDA availability. Ensure PyTorch is installed with CUDA support (pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118).
  • Using the wrong precision for the task. For inference, 4-bit is fine; for training, 8-bit is safer to avoid quality loss.
  • Forgetting to set device_map. Without it, the model loads entirely on GPU and crashes.
  • Not restarting the kernel after installing libraries. This can cause import errors.

Tips

  • Use transformers with device_map="auto" – it’s the easiest way to get offloading right.
  • For text generation, set max_new_tokens to a reasonable number (e.g., 512) to avoid memory spikes during generation.
  • Monitor memory usage with nvidia-smi -l 1 while running to see if you’re close to the limit.
  • If you’re on a Mac, use device_map="mps" and consider using torch.float16 for better performance.
  • For large models, consider using accelerate library – it simplifies offloading and mixed precision.

Troubleshooting

  • “CUDA out of memory” error. Reduce batch size, enable quantization, or offload more layers to CPU. Also close other GPU apps.
  • Model loads but runs very slowly. This is normal when offloading to CPU. Try to fit more layers on GPU by reducing quantization bits (e.g., from 4-bit to 8-bit if you have space).
  • Import error for bitsandbytes. On Windows, you may need to install a pre-release version: pip install bitsandbytes --pre.
  • “RuntimeError: No CUDA GPUs are available” – Check your PyTorch installation with torch.cuda.is_available(). If False, reinstall with CUDA support.
  • Model gives gibberish output. This can happen if quantization is too aggressive. Try 8-bit instead of 4-bit.

Frequently Asked Questions

Can I run a 7B model on 8 GB VRAM?

Yes, with 4-bit quantization and CPU offloading, a 7B model can run on 8 GB VRAM, but it will be slow. For better speed, use a 3B model.

What is the difference between 4-bit and 8-bit quantization?

4-bit uses less memory (about 4 bits per weight) but can slightly reduce accuracy. 8-bit is more accurate but uses twice the memory.

Does CPU offloading affect output quality?

No, offloading only moves layers to CPU memory; the computation is the same. It just slows down inference.

Can I use this with TensorFlow?

Yes, TensorFlow has similar options: mixed_float16 policy and model quantization via TFLite, but the steps differ. This guide focuses on PyTorch.

How much system RAM do I need?

At least as much as the model size in bytes. For a 7B model in 4-bit (about 4 GB), you need at least 4 GB free system RAM, but more is better.

Will this work on AMD GPUs?

Yes, with ROCm support in PyTorch, but the setup is more complex. The same quantization and offloading principles apply.

Related

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →