How to Choose a GPU for Local LLMs

Choosing a GPU for running local large language models (LLMs) means balancing memory capacity, bandwidth, and compute speed against your budget and use case. This guide walks you through the key specs, how to match them to the models you want to run, and common pitfalls to avoid.

What You’ll Need

• A list of the LLMs you plan to run (e.g., Llama 3, Mistral, Qwen) and their parameter sizes (7B, 13B, 70B).
• Knowledge of your current PC’s power supply wattage and available PCIe slots.
• A rough budget range.

Step-by-Step

1. Determine the model size you need. Start with the largest model you realistically want to run. LLMs are measured in billions of parameters (B). A 7B model needs about 4–6 GB of VRAM in 4-bit quantization, 13B needs 8–10 GB, and 70B needs 40–48 GB. If you want to run a 70B model comfortably, you’re looking at a 48 GB card like the RTX A6000 or a dual-GPU setup.
2. Check VRAM capacity first. VRAM is the most critical spec. It determines which models can load entirely into memory. Running a model that doesn’t fit in VRAM forces it to spill to system RAM, which slows inference dramatically. Aim for at least 16 GB for 13B models, 24 GB for 30B models, and 48 GB+ for 70B models.
3. Look at memory bandwidth. Bandwidth (GB/s) affects token generation speed. Higher bandwidth means faster responses. Consumer cards like the RTX 4090 have ~1 TB/s, while server cards like the H100 exceed 3 TB/s. For most hobbyists, 500 GB/s or more is good; for serious multitasking or large models, aim higher.
4. Consider compute performance (TFLOPS). This matters for training and for very long context windows. For pure inference, VRAM and bandwidth matter more. But if you plan to fine-tune, look for cards with high FP16/BF16 throughput.
5. Check software compatibility. Most local LLM tools (llama.cpp, Ollama, text-generation-webui) support NVIDIA CUDA best. AMD cards work via ROCm but can have setup friction. Intel Arc is improving but still has limited support. If you want a plug-and-play experience, choose NVIDIA.
6. Verify power and physical fit. High-end GPUs draw 300–450 W. Check your PSU’s wattage and available PCIe power connectors. Also measure your case clearance—many 4090s are huge.
7. Decide between new, used, or cloud. Used enterprise cards (like the Tesla P40 with 24 GB) can be cheap but lack modern features. Cloud GPU rental is an alternative if you don’t need 24/7 local access.

Common Mistakes

• Ignoring VRAM limits: Buying a fast card with 8 GB VRAM and then trying to run a 13B model will result in swapping and terrible performance.
• Confusing system RAM with VRAM: System RAM is not a substitute for VRAM; models run much slower when offloaded to CPU.
• Overlooking bandwidth: Two cards with the same VRAM can have very different speeds. A 3090 (936 GB/s) is much faster than a 4060 Ti 16GB (288 GB/s) for LLMs.
• Buying a gaming card for server workloads: Cards like the RTX 4090 are great but lack ECC memory and can throttle under sustained load.
• Not accounting for quantization: You can run larger models with quantization (e.g., 4-bit), but quality drops. Don’t assume you can run a 70B on a 24 GB card without significant quality loss.

Tips

• Use tools like llama.cpp to test model memory requirements before buying.
• If you’re on a budget, consider used RTX 3090s—they offer 24 GB VRAM and good bandwidth for a fraction of the cost of new 4090s.
• For multi-GPU setups, ensure your motherboard supports PCIe bifurcation and that your PSU can handle the combined draw.
• Keep an eye on thermals; LLM inference can push GPUs to high utilization for long periods. Good case airflow is essential.

Troubleshooting

• Model won’t load (out of memory): Reduce context length, use a lower quantization (e.g., Q4_K_M instead of Q8), or switch to a smaller model.
• Very slow generation: Check if the model is partially offloaded to CPU. Use the GPU’s VRAM fully, and close other GPU-using applications.
• CUDA errors: Ensure you have the latest NVIDIA drivers and that your software is compiled with CUDA support. Reinstall CUDA toolkit if needed.
• GPU not detected: Check PCIe slot seating, power connections, and BIOS settings. Try a different slot if possible.
• Thermal throttling: Monitor temperatures with GPU-Z. If throttling, improve case airflow, undervolt, or lower power limit.

Frequently Asked Questions

1. Can I run LLMs on a gaming GPU like the RTX 4060? Yes, but only small models (7B or less) due to limited VRAM and bandwidth. For larger models, you’ll need 16 GB+ VRAM.
2. Is more VRAM always better? Generally yes, but only up to the point where you can fit your largest model. After that, bandwidth and compute matter more.
3. Can I use multiple GPUs to run one large model? Yes, tools like llama.cpp support multi-GPU splitting. However, scaling isn’t linear, and you need enough PCIe bandwidth between cards.
4. Do AMD GPUs work for local LLMs? They can, but setup is more complex due to ROCm compatibility. Some models have better support than others; check the specific tool’s documentation.
5. What is quantization and why does it matter? Quantization reduces the precision of model weights (e.g., from 16-bit to 4-bit) to save VRAM. It allows running larger models on smaller GPUs with a slight quality trade-off.
6. Should I buy a workstation card like the RTX A6000? Only if you need 48 GB+ VRAM or ECC memory. For most hobbyists, a used RTX 3090 offers better value.

Related

• Show the Developer Tab in Word: Step-by-Step
• How to Share Your Location on Android
• How to Add Line Numbers in Microsoft Word