sodcalculator

10 Tech How-To

How to Choose a GPU and VRAM for Local LLM Inference

Guide 10-36981 · Updated · Step by step

This guide walks you through picking a graphics card and the right amount of VRAM for running large language models on your own hardware. You’ll learn how to estimate VRAM needs based on model size and quantization, what GPU specs matter most, and how to match a card to your budget and use case.

What You’ll Need

  • An idea of which LLM(s) you want to run (e.g., Llama 3, Mistral, Phi-3)
  • Basic familiarity with model parameters (7B, 13B, 70B) and quantization (4-bit, 8-bit)
  • A budget range for the GPU purchase
  • Your current PC’s power supply wattage and case dimensions (to ensure compatibility)

Step-by-Step

  1. Identify the models you plan to run. Start with the largest model you realistically want to use. For example, if you want to run Llama 3 8B, that’s a 8-billion-parameter model. If you want to run a 70B model, that’s a much larger VRAM requirement.
  2. Determine the quantization level. Quantization reduces model size by using fewer bits per weight. Common levels: 4-bit (most compact), 8-bit (good balance), 16-bit (full precision). For local inference, 4-bit is popular because it cuts VRAM needs by roughly 75% compared to 16-bit.
  3. Estimate VRAM per model. Use this rule of thumb: VRAM ≈ (parameters in billions) × (bits per weight) / 8, then add overhead for context and compute buffers. For a 7B model at 4-bit: 7 × 4 / 8 = 3.5 GB, plus ~1-2 GB overhead → ~5 GB. At 8-bit: 7 × 8 / 8 = 7 GB, plus overhead → ~9 GB. For 13B at 4-bit: ~6.5 GB + overhead → ~8 GB. For 70B at 4-bit: ~35 GB + overhead → ~40 GB. Always add 20-30% headroom for context length and CUDA overhead.
  4. Check your context length needs. Longer context (e.g., 32K tokens) increases memory usage. A rough estimate: each token of context uses about 0.5-1 MB of VRAM depending on the model and precision. For 8K context, add 4-8 GB. If you need long context, factor that into your VRAM budget.
  5. Decide on a GPU tier. Based on your VRAM estimate:
    • Under 8 GB: Entry-level cards like NVIDIA RTX 3060 (12 GB) or RTX 4060 (8 GB) can run 7B models at 4-bit with short context.
    • 8-12 GB: RTX 4070 (12 GB) or RTX 4070 Ti (12 GB) handle 7B-13B models comfortably at 4-bit.
    • 16-24 GB: RTX 4080 (16 GB) or RTX 4090 (24 GB) support 13B-30B models at 4-bit, or 7B at 8-bit.
    • 24 GB+: For 70B models, you’d need 48 GB+ (e.g., two RTX 3090/4090 in NVLink or a used RTX A6000). Alternatively, consider cloud or CPU offloading.
  6. Consider memory bandwidth and compute. VRAM capacity isn’t everything. Memory bandwidth (GB/s) determines token generation speed. A 4090 has ~1 TB/s, while a 3060 has ~360 GB/s. Higher bandwidth means faster responses. Compute (TFLOPS) matters for prompt processing. For inference, bandwidth is more critical than raw compute.
  7. Check software compatibility. Most local LLM tools (llama.cpp, Ollama, LM Studio) support NVIDIA CUDA and AMD ROCm. NVIDIA is generally better supported and has more mature libraries. If you use AMD, ensure your card is on the ROCm supported list.
  8. Verify power and physical fit. High-end GPUs draw 300-450W. Check your PSU wattage and available PCIe power connectors. Also measure your case length—many cards are 300+ mm long and may not fit in small cases.
  9. Buy used if budget-constrained. Cards like the RTX 3090 (24 GB) can be found used at reasonable prices, offering great VRAM per dollar. Ensure the card hasn’t been mined on heavily (check thermals and fan noise).
  10. Test with a small model first. After installing, run a 1-2B model to verify setup, then scale up. This helps isolate issues before committing to a large download.

Common Mistakes

  • Underestimating VRAM overhead. Many people calculate only the model size and forget context, batch size, and CUDA buffers. Always add 20-30% headroom.
  • Ignoring memory bandwidth. A card with 24 GB but low bandwidth (e.g., some older cards) will be painfully slow. Balance capacity with speed.
  • Buying a card with too little VRAM for the intended model. If you want to run a 13B at 8-bit, an 8 GB card won’t cut it. Check the numbers before purchasing.
  • Forgetting about power supply. A 4090 needs a 1000W PSU with a 12VHPWR connector. Not having that can cause shutdowns or require an adapter.
  • Assuming all GPUs are equal. NVIDIA cards are generally easier to set up with CUDA. AMD cards may work but require ROCm, which has limited support on Windows.

Tips

  • Use tools like llama.cpp or Ollama to automatically offload layers to GPU and CPU. This allows you to run models slightly larger than your VRAM, at a speed cost.
  • Consider using GGUF quantized models (e.g., Q4_K_M) which are optimized for CPU/GPU hybrid inference.
  • If you’re on a tight budget, look for used RTX 3060 12GB or RTX 3080 10GB—both offer good value for 7B models.
  • For multi-GPU setups, ensure your motherboard supports PCIe bifurcation or use NVLink (for older cards) to combine VRAM.
  • Monitor VRAM usage with tools like nvidia-smi to see actual consumption and adjust model settings accordingly.

Troubleshooting

  • Model fails to load with “CUDA out of memory”. Reduce context length, lower quantization (e.g., from 8-bit to 4-bit), or use CPU offloading. Try a smaller model.
  • Very slow token generation. Check if the model is running on CPU instead of GPU. In llama.cpp, ensure you set -ngl (number of GPU layers) high enough. Also, close other GPU-heavy applications.
  • GPU not detected. Update drivers, ensure the card is properly seated, and check if your motherboard’s PCIe slot is functional. For AMD, verify ROCm installation.
  • System crashes under load. This often indicates insufficient PSU wattage or overheating. Check temps and power draw. Reduce GPU power limit slightly in software like MSI Afterburner.
  • Model outputs gibberish. This can happen if the model is corrupted or the quantization is too aggressive. Re-download the model or try a different quant.

Frequently Asked Questions

Can I run an LLM on a laptop GPU? Yes, but laptop GPUs have lower power limits and less VRAM. A laptop RTX 4060 (8 GB) can run 7B models at 4-bit with short context, but expect slower speeds.

Is more VRAM always better? Not necessarily—if you only run small models, extra VRAM won’t help. But if you plan to scale up, more VRAM future-proofs your setup.

What is the minimum VRAM for a 7B model? At 4-bit, you need about 5-6 GB. At 8-bit, about 9-10 GB. For comfortable use with 8K context, aim for 8 GB or more.

Can I use an AMD GPU for LLM inference? Yes, but support is more limited. On Linux, ROCm works well for many cards. On Windows, support is spotty; consider using Linux or NVIDIA.

How much does context length affect VRAM? Significantly. For a 7B model, each token of context can use ~1 MB. A 32K context adds ~32 GB, which is huge. Use sliding window attention if possible to reduce this.

Should I buy a new or used GPU? Used can save money, especially for high-VRAM cards like the RTX 3090. Just test it thoroughly and check for signs of mining wear.

Related

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →