Running a large AI model can quickly exhaust GPU memory, causing out-of-memory errors or forcing you to use smaller batch sizes. This guide shows practical, step-by-step methods to reduce VRAM usage, from simple settings to advanced techniques, so you can fit bigger models or larger batches on your existing hardware.
What You’ll Need
- A GPU with at least 4 GB of VRAM (more for very large models).
- Python environment with PyTorch or TensorFlow installed.
- Access to the model’s code or a framework like Hugging Face Transformers.
- Basic familiarity with command line and Python scripts.
Step-by-Step
- Reduce batch size – The simplest and most effective step. Halve the batch size until the model fits. For example, if batch size 8 causes OOM, try 4, then 2. This directly reduces the memory used for activations and gradients.
- Use mixed precision (FP16) – Enable automatic mixed precision (AMP) in PyTorch or TensorFlow. This stores tensors in 16-bit floating point, halving memory for many operations. In PyTorch:
with torch.cuda.amp.autocast():and useGradScalerfor the loss. In TensorFlow, usetf.keras.mixed_precision.set_global_policy('mixed_float16'). - Gradient accumulation – If you need a large effective batch size but lack memory, accumulate gradients over several smaller batches. For example, accumulate over 4 steps of batch size 2 to simulate a batch of 8. This keeps memory low while preserving training dynamics.
- Freeze layers or use LoRA – For fine-tuning, freeze most of the model’s layers and only train a small set of parameters. Alternatively, use Low-Rank Adaptation (LoRA) via libraries like PEFT, which adds tiny trainable matrices, drastically reducing optimizer memory.
- Use gradient checkpointing – Also called activation checkpointing. This recomputes activations during the backward pass instead of storing them, trading compute for memory. In Transformers, set
model.gradient_checkpointing_enable(). This can reduce memory by up to 60%. - Offload to CPU (or NVMe) – For very large models, use
accelerateordeepseedto offload parameters, gradients, and optimizer states to CPU memory or disk. This allows running models that exceed VRAM, though slower. In Hugging Face, usedevice_map="auto"withload_in_8bit=Trueorload_in_4bit=Truefor quantization. - Quantize the model – Load the model in 8-bit or 4-bit precision using bitsandbytes. This reduces memory by 4x or 8x compared to FP32. For inference, use
model = AutoModelForCausalLM.from_pretrained(..., load_in_4bit=True). - Limit sequence length – For transformers, memory scales quadratically with sequence length. Trim input texts or use sliding window attention if supported. Even reducing max length from 512 to 256 can free significant memory.
- Clear cache and monitor – Use
torch.cuda.empty_cache()after each step or before evaluation. Also monitor memory withnvidia-smito see what’s using VRAM. Sometimes memory fragmentation can be fixed by restarting the kernel. - Use a memory-efficient optimizer – Switch from Adam to AdamW with
foreach=Falseor use SGD with momentum. Optimizers like Adafactor store less state. In Transformers, useoptimizer="adafactor"to reduce memory.
Common Mistakes
- Not using mixed precision because of accuracy concerns – For most tasks, FP16 loss scaling preserves accuracy. Test it; the speed and memory benefits usually outweigh minor precision changes.
- Setting batch size too high initially – Always start small and increase until OOM, rather than starting high and crashing.
- Forgetting to set
model.eval()during inference – In eval mode, dropout and batch norm behave differently, which can affect memory and results. - Using gradient checkpointing with a small model – It adds overhead; only use it when memory is a bottleneck.
- Ignoring the data loader – Sometimes the data loader pins memory or loads large batches; set
num_workersappropriately and avoid loading entire datasets into VRAM.
Tips
- Combine techniques: mixed precision + gradient checkpointing + batch size reduction is often enough for most models.
- Use
torch.cuda.set_per_process_memory_fraction(0.9)to limit PyTorch’s memory allocation and avoid OOM from other processes. - When using Hugging Face, try
model.half()for FP16 inference, but be aware that some layers may need float32. - For inference, consider using ONNX Runtime with FP16 optimization, which can be more efficient than PyTorch.
- Keep your CUDA and drivers updated; newer versions often have memory optimizations.
Troubleshooting
- Still OOM after reducing batch size – Try gradient checkpointing and mixed precision together. If it still fails, use CPU offloading or quantization.
- Mixed precision causes NaN loss – Ensure you use a GradScaler in PyTorch. If problems persist, use FP16 only for certain layers (e.g., via
torch.cuda.amp.autocastwithdtype=torch.float16). - Model loads but training is slow – CPU offloading is slow; consider using more GPU memory or a smaller model. Also check if the bottleneck is disk I/O.
- Quantization errors or poor output – Some models are sensitive to 4-bit quantization. Try 8-bit or use quantization-aware training if possible.
- Memory fragmentation – If memory usage grows over time, it may be fragmentation. Restart the kernel or use
torch.cuda.empty_cache()periodically.
Frequently Asked Questions
Will reducing batch size affect model accuracy?
It may, but you can compensate with gradient accumulation to keep the effective batch size the same. In practice, smaller batches can sometimes even improve generalization.
Is mixed precision supported on all GPUs?
No, it works best on NVIDIA GPUs with Tensor Cores (Volta and newer). Older GPUs may not see speed benefits, but memory savings still apply.
How much memory can I save with gradient checkpointing?
Typically 50-70% reduction in activation memory, but it increases compute time by about 20-30%.
Can I run a 70B model on a 24GB GPU?
Yes, with 4-bit quantization and CPU offloading, it’s possible, but inference will be slow. For training, it’s not practical.
What is the difference between FP16 and BF16?
BF16 has a larger range but lower precision. It’s better for training very large models because it avoids overflow issues. Some newer GPUs support BF16 natively.
Why does my memory usage increase during training even with a fixed batch size?
This is often due to accumulating gradients for the optimizer step, or fragmentation. Use torch.cuda.empty_cache() and check if you’re accidentally storing intermediate tensors.
Related
General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.
More in this bed
- PlayStation Overheating: Causes & Fixes
- How to Remove or Insert a SIM Card in Your iPhone
- How to Send a Fax from Android or iOS
- Outlook Stuck on Loading Profile? Fix It
- MacBook Trackpad Protection: A Complete Guide
- Error 0x803f7001 Windows: Fixes & Meaning