Choosing the right Ollama model is the difference between a fast, responsive local AI and one that crawls or crashes. This guide walks you through matching a model’s size and quantization to your system’s RAM and GPU VRAM, so you get the best quality your hardware can handle. You’ll learn how to read model names, estimate memory usage, and pick a model that runs smoothly.
What You’ll Need
- Your computer’s total RAM and GPU VRAM (check in Task Manager on Windows, Activity Monitor on macOS, or
nvidia-smion Linux). - Ollama installed and working (download from ollama.com).
- Basic familiarity with the command line (you’ll run
ollama pullandollama run).
Step-by-Step
- Check your hardware.
Open Task Manager (Windows) or Activity Monitor (macOS) and note your total RAM. For GPU, use
nvidia-smi(NVIDIA) or check System Information on macOS. Write down both numbers—you’ll need them for every decision below. - Understand model sizes and quantization.
Ollama models are named like
llama3.1:8borllama3.1:8b-q4_K_M. The number (e.g., 8b) is the parameter count in billions. The suffix (e.g., q4_K_M) is the quantization—how many bits per weight. Lower quantization (q4, q5) uses less memory but slightly reduces quality; higher (q8, fp16) uses more memory but preserves quality. The default is usually q4_K_M, a good balance. - Estimate memory usage.
A rough rule: a model needs about 0.5–0.7 GB of RAM/VRAM per billion parameters at q4 quantization. So an 8B model needs ~4–6 GB, a 13B needs ~7–9 GB, a 70B needs ~35–50 GB. For fp16 (full precision), double those numbers. Use this as a starting point, then check the model’s page on Ollama’s library for exact requirements.
- Match to your GPU VRAM first.
If you have a GPU, the ideal is to fit the entire model in VRAM. That gives the fastest speeds. For example, with 8 GB VRAM, an 8B q4 model (≈5 GB) fits comfortably. With 12 GB, you can try a 13B q4 (≈8 GB). If the model fits in VRAM, you’re done—it will run entirely on GPU.
- If it doesn’t fit, use CPU + GPU offloading.
Ollama automatically offloads layers to CPU when VRAM is insufficient. This works, but it’s slower. As long as the model’s total memory (VRAM + RAM) fits within your system’s combined memory, it will run. For example, a 70B q4 model needs ~40 GB total. If you have 16 GB VRAM and 64 GB RAM, that’s 80 GB total—it will run, but with some CPU offloading.
- Choose a model size based on your total memory.
If you have no GPU or very little VRAM, rely on system RAM. A 7B–8B model runs fine on 16 GB RAM. A 13B model needs ~16 GB RAM just for the model, so 32 GB is safer. For 70B models, you need at least 64 GB RAM to run them at q4. Always leave at least 2–4 GB of RAM for your OS and other apps.
- Pick a quantization that fits.
If the default q4 model is too big, try a smaller quantization like
q3_K_Sorq2_K. These use less memory but may degrade quality noticeably. If you have plenty of VRAM, tryq5_K_Morq6_Kfor better quality. Check the model’s library page for available quantizations. - Test with a quick run.
Run
ollama run <model>and ask a simple question. Watch your system monitor: if memory usage is near 100%, the model may be swapping to disk—that’s a sign to choose a smaller model or quantization. If it responds quickly, you’re good. - Consider task-specific models.
Smaller models (3B–8B) are great for code completion, summarization, and simple chat. For complex reasoning or creative writing, you might need 13B–70B. Don’t just pick the biggest—pick the smallest that does the job well.
Common Mistakes
- Ignoring quantization. Two models with the same parameter count can have very different memory footprints. Always check the suffix.
- Assuming GPU is always faster. If your GPU is old or has low VRAM, CPU-only might be more stable. Don’t force a model into VRAM if it causes crashes.
- Not leaving headroom. Your OS and other programs use memory too. If you have 16 GB RAM, don’t try a model that needs 15 GB—you’ll end up with swapping and poor performance.
- Choosing the largest model you can fit. Bigger isn’t always better. A 70B model on a CPU-only system can be painfully slow; a 13B might be more useful.
Tips
- Use
ollama show <model>to see the exact file size and parameters before pulling. - Start with a small model (like
llama3.2:3b) to verify your setup, then scale up. - If you have multiple GPUs, Ollama can split the model across them—just make sure the total VRAM is enough.
- For CPU-only systems, consider models with lower quantization (q4 or q3) to speed up inference.
- Keep your Ollama version updated—newer versions often have better memory management.
Troubleshooting
- Model runs out of memory (OOM) or crashes. This means the model is too big for your combined RAM+VRAM. Choose a smaller model or a lower quantization. If you’re on Windows, make sure you have enough page file space.
- Extremely slow responses. Likely the model is offloading to CPU or swapping to disk. Check your system monitor. If CPU is at 100%, consider a smaller model or lower quantization. If disk is active, you need more RAM.
- GPU not being used. Run
ollama runand check if the GPU shows activity. If not, your GPU may not be supported (e.g., older AMD cards). Try settingOLLAMA_GPU_LAYERS=1as a test, or check Ollama’s docs for GPU support. - Model pulls but won’t run. Try pulling a specific quantization tag, e.g.,
ollama pull llama3.1:8b-q4_0. Sometimes the default tag is a larger version. - Incorrect or garbled output. This can happen with very low quantizations (q2). Try a higher quantization or a different model.
Frequently Asked Questions
How much RAM do I need for a 7B model?
At q4 quantization, a 7B model needs about 4–5 GB of RAM. With your OS and other apps, 8 GB total RAM is the minimum, but 16 GB is recommended for smooth operation.
Can I run a 70B model on 32 GB RAM?
Yes, but only with heavy quantization (q2 or q3) and it will be slow. You’ll also need to close all other applications. A 70B q4 model needs ~35–40 GB, so 64 GB RAM is more realistic.
What does ‘q4_K_M’ mean?
It’s a quantization method. ‘q4’ means 4-bit weights, ‘K’ is a specific format, and ‘M’ indicates the size (S, M, L). Lower numbers use less memory but reduce quality slightly.
Is it better to have more VRAM or more RAM?
For speed, VRAM is better—the entire model can run on the GPU. But RAM is more flexible and cheaper. For large models, you’ll need both: VRAM for speed, RAM for capacity.
How do I know if a model is using my GPU?
While running a model, open Task Manager (Windows) or Activity Monitor (macOS) and check GPU utilization. If it’s near 100%, the model is running on GPU. If it’s 0%, it’s using CPU.
Can I run multiple models at once?
Yes, but each model uses its own memory. If you have 32 GB RAM, you can run two 8B models simultaneously, but they’ll compete for resources. It’s usually better to run one at a time.
Related
General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.
More in this bed
- Pair Phone to Pioneer Stereo After Bluetooth Reset
- Fix Windows Not Saving Remote Desktop Credentials
- Fix Error 0x8007025d in Windows 10
- Fix Alexa App Stuck on Setup
- Android Camera for 7 Office Tasks: Step-by-Step
- Essential Tips for macOS Users Switching to Windows