How to Choose a GPU for Large GGUF Models

Running large GGUF models locally means loading quantized LLMs like Llama 3.1 70B or Mixtral into your own hardware. The GPU you pick determines which models fit in VRAM, how fast tokens stream out, and whether you need to offload layers to system RAM. This guide walks through the exact specs to check, how to match them to your target model, and where people usually get it wrong.

What You’ll Need

The GGUF model you want to run (or a list of candidates), including its parameter count and quantization level (e.g., Q4_K_M, Q5_K_M).
The model’s file size in GB, which you can find on its Hugging Face page or by downloading the file.
The context length you plan to use (e.g., 4096, 8192 tokens). Longer contexts eat more VRAM.
A rough budget and your power supply’s PCIe power connectors.

Step-by-Step

Find the model’s VRAM footprint. Look up the GGUF file size for your desired quantization. For example, a 70B model at Q4_K_M is roughly 40 GB. Add 1-2 GB for compute buffers and CUDA overhead, plus about 1 GB per 4096 tokens of context (more for larger models). That total is the minimum VRAM you need.
Decide on quantization first. Lower quantizations (Q2, Q3) shrink files but hurt quality; Q4_K_M is the sweet spot for most people. If you can’t fit Q4, consider a smaller model rather than dropping to Q2.
Match VRAM to the model. For a 7B model at Q4, 6-8 GB VRAM works. For 13B, 10-12 GB. For 30B, 20-24 GB. For 70B, you need 40+ GB, which means an RTX 4090 (24 GB) won’t fit—you’ll need dual GPUs or a used RTX 3090 with 24 GB each, or a professional card like the RTX 6000 Ada.
Check memory bandwidth, not just capacity. Token generation speed scales with memory bandwidth. A 4090 has ~1 TB/s, a 3090 ~936 GB/s, a 4080 ~717 GB/s. If you plan to offload layers to system RAM, your RAM speed (DDR4 3200 vs DDR5 6000) becomes the bottleneck and generation will drop to 1-3 tokens/sec.
Consider PCIe lanes and multi-GPU. If you need two GPUs, ensure your motherboard has enough PCIe x16 slots and that the cards can communicate via NVLink (for 3090s) or just PCIe (which works but is slower). For dual 3090s, you get 48 GB total, enough for 70B Q4 with a small context.
Check power and physical fit. A 4090 draws 450W, a 3090 350W. Verify your PSU has enough wattage and the right 12VHPWR or 8-pin connectors. Measure your case clearance—these cards are long and thick.
Look at used market for value. RTX 3090s are often the best price-to-VRAM ratio for large models. Newer cards like the 4070 Ti Super (16 GB) or 4080 Super (16 GB) are good for 13B-30B but can’t touch 70B.
Test with llama.cpp or Ollama. Before buying, use a cloud GPU or a friend’s machine to run the exact model and quantization you want. Measure tokens/sec and VRAM usage with the --verbose flag or nvidia-smi.

Common Mistakes

Ignoring context length. A 70B Q4 model might be 40 GB, but a 32K context can add 10+ GB. Always add overhead.
Buying a 16 GB card for 70B. It won’t fit, period. You’ll end up offloading to RAM and getting 0.5 tokens/sec.
Focusing on CUDA cores or TFLOPS. For LLM inference, memory bandwidth matters more than compute. A 4060 Ti (8 GB) is slower than a used 3090 despite being newer.
Forgetting that GGUF uses CPU offload. If you run out of VRAM, llama.cpp will offload layers to system RAM, which is much slower. You should aim to fit everything in VRAM for decent speed.
Assuming all 24 GB cards are equal. The 3090, 4090, and 7900 XTX all have 24 GB, but their bandwidth and software support differ. For GGUF, NVIDIA is generally better supported with CUDA.
Not checking the PSU’s PCIe cables. Many older PSUs lack the 12VHPWR connector for 40-series cards, or don’t have enough 8-pin cables for dual GPUs.

Tips

Use the GGUF My Repo tool to see file sizes for different quantizations.
If you’re on a budget, look for used RTX 3090s—they often sell for under $800 and give 24 GB of fast GDDR6X.
For 70B models, consider dual 3090s. With llama.cpp, you can split layers across GPUs using --split-mode layer.
Enable --flash-attn in llama.cpp to reduce memory usage and speed up inference.
If you only run smaller models (7B-13B), a 12 GB card like the RTX 3060 is a cost-effective choice.
Monitor VRAM usage with nvidia-smi -l 1 while running to see if you’re close to the limit.

Troubleshooting

Model doesn’t load, says “out of memory”. Lower the context length, switch to a lower quantization (e.g., Q4 to Q3), or reduce the number of layers offloaded to GPU (use --n-gpu-layers).
Very slow generation (under 1 token/sec). Check if layers are being offloaded to CPU. Reduce --n-gpu-layers until it fits entirely in VRAM, or upgrade RAM speed.
GPU not detected. Update drivers, ensure the card is seated properly, and check that your PSU provides enough power. For dual GPUs, make sure the motherboard supports the slot configuration.
Two GPUs not working together. In llama.cpp, specify --main-gpu 0 and --tensor-split to balance layers. Also ensure both cards are in PCIe x16 slots (x8 is okay but slower).
System crashes under load. Check temperatures—GPUs throttle at 83°C. Improve case airflow or undervolt the card.

Frequently Asked Questions

Can I run a 70B GGUF model on a 24 GB GPU?

Only if you use a very low quantization like Q2_K (about 20 GB) and a short context, but quality suffers. For Q4_K_M, you need ~40 GB, so dual 24 GB cards or a 48 GB professional card.

Is more VRAM always better?

Yes, for fitting larger models. But speed also depends on memory bandwidth. A 4090 (24 GB) is faster than a 3090 (24 GB) because of higher bandwidth, but both can fit the same models.

Does the GPU brand matter for GGUF?

NVIDIA is generally better supported with CUDA, which llama.cpp and Ollama use. AMD cards work but may require extra setup and can be slower due to less optimized ROCm support.

How much does context length affect VRAM?

For each 4096 tokens of context, add roughly 1-2 GB for a 7B model, 2-3 GB for a 13B, and 4-6 GB for a 70B. It’s not linear, but it’s significant.

Can I use a gaming GPU for this?

Yes, most people use gaming GPUs like the RTX 3090 or 4090. They work fine for inference; you don’t need a professional card unless you need more than 24 GB VRAM.

Should I wait for the next generation of GPUs?

If you’re not in a hurry, newer cards may offer better price-to-performance, but the used market for 3090s is stable. For 70B models, dual 3090s remain the most cost-effective solution.

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →