sodcalculator

10 Tech How-To

How to Choose a Server for NVIDIA GPU AI Workloads

Guide 10-54822 · Updated · Step by step

Choosing a server for NVIDIA GPU-accelerated AI workloads means matching hardware to your specific training, inference, and budget constraints. This guide walks through the key decisions: GPU selection, CPU and memory balance, storage, power, cooling, and form factor, so you can spec a machine that actually runs your models without overspending.

What You’ll Need

  • Your AI workload profile: training vs. inference, model size, batch size, and expected concurrency
  • A budget range for the server itself (not just GPUs)
  • Knowledge of your data center or office power (voltage, amperage, and available circuits)
  • Physical space and cooling capacity (rack units, BTU output)
  • Software stack requirements (CUDA version, PyTorch/TensorFlow, containerization needs)

Step-by-Step

  1. Define your workload first. Training large transformer models needs high VRAM and fast inter-GPU bandwidth (NVLink). Inference with many concurrent users needs more GPUs with lower VRAM but higher throughput. Write down your model size, typical input batch, and how many requests per second you expect.
  2. Pick the GPU generation and count. For most new builds, choose NVIDIA H100, A100, or L40S for training; for inference, L4 or A10 may suffice. Decide how many GPUs: 1-2 for prototyping, 4-8 for serious training. Check that the GPUs support NVLink if you need faster multi-GPU communication.
  3. Match CPU to GPU count. A single GPU can be fed by a 16-core CPU. For 4+ GPUs, use a dual-socket motherboard with 32-64 cores total. AMD EPYC or Intel Xeon are standard. Do not overspend on CPU for pure GPU-bound workloads.
  4. Size the RAM and storage. System RAM should be at least 2x the total GPU VRAM (e.g., 4x 80GB GPUs = 320GB VRAM, so 512GB-1TB RAM). For storage, use NVMe SSDs for datasets and checkpoints; 2-4TB is a starting point. Add a separate slower drive for logs or backups.
  5. Choose the right form factor. For 1-2 GPUs, a tower or 4U workstation works. For 4+ GPUs, use a 4U rack server with GPU riser cables. 2U servers often have poor airflow for GPUs; avoid them unless the vendor explicitly supports GPU cooling.
  6. Check power supply and connectors. Each H100 can draw 700W. For 4 GPUs, that’s 2800W plus CPU and rest, so you need a 3000W+ PSU (or dual PSUs). Verify you have the right 8-pin or 12VHPWR cables. For rack servers, ensure the PSUs match your facility’s voltage (208V or 240V preferred).
  7. Plan cooling. GPUs exhaust hot air; ensure front-to-back airflow in the chassis. For dense setups, consider liquid cooling or ensure the room has adequate AC. Check the GPU’s operating temperature limit (typically 85°C) and that the case can keep it below that under sustained load.
  8. Verify PCIe lanes and topology. Each GPU needs x16 PCIe Gen4 or Gen5 lanes. A dual-socket CPU motherboard with 128 lanes can handle 4 GPUs at x16. For 8 GPUs, you need a server board with a PCIe switch. Check the motherboard manual for lane distribution.
  9. Select the OS and drivers. Ubuntu 22.04 LTS is the safest choice for NVIDIA drivers and CUDA. Install the NVIDIA driver and CUDA toolkit, then verify with nvidia-smi. For multi-GPU, enable NVLink and check with nvidia-smi topo -m.
  10. Test before you buy. If possible, run a small benchmark (like a ResNet training loop) on the exact hardware you plan to purchase. This catches thermal throttling or PCIe bottlenecks early.

Common Mistakes

  • Underpowering the PSU. GPUs can spike above their TDP. Always add 20-30% headroom beyond total theoretical draw.
  • Ignoring PCIe lane limits. Consumer motherboards often have only 24 lanes; putting 4 GPUs on them drops them to x8 or x4, hurting performance.
  • Choosing a 2U chassis for 4 GPUs. The airflow is usually inadequate, leading to thermal throttling within minutes.
  • Buying more VRAM than needed. If your model fits in 24GB, an 80GB GPU is wasted money. Size to your largest expected model plus a margin for batch size growth.
  • Forgetting about NVLink for training. Without NVLink, multi-GPU training over PCIe is much slower for models that need frequent gradient sync.

Tips

  • For inference-only workloads, consider GPUs with lower VRAM but higher tensor core throughput, like the L4, which is cheaper per request.
  • Use NVIDIA’s official compatibility matrix to ensure your CPU, motherboard, and GPU combination is validated.
  • Buy a server with redundant PSUs (2+1) if uptime matters; a single PSU failure takes the whole node down.
  • Plan for future expansion: get a chassis with extra PCIe slots and PSU headroom even if you only populate 2 GPUs now.
  • Use NVMe drives in RAID 1 for the OS and a separate NVMe for datasets; this avoids I/O bottlenecks during training.

Troubleshooting

  • GPU not detected: Check PCIe seating, power cables, and BIOS settings (enable Above 4G Decoding and Resizable BAR). Run lspci | grep -i nvidia.
  • Thermal throttling: Monitor with nvidia-smi dmon. If temps hit 85°C, increase fan speed, improve case airflow, or reduce ambient temperature.
  • Slow training despite good GPUs: Check PCIe link speed with nvidia-smi -q -d PCI. If it shows x8 instead of x16, reseat the GPU or change the slot.
  • System won’t POST with multiple GPUs: Remove all but one GPU, update BIOS, then add GPUs one by one. Ensure the PSU can handle the inrush current.
  • CUDA out of memory: Reduce batch size, use gradient accumulation, or switch to mixed precision (FP16/BF16) to halve VRAM usage.

Frequently Asked Questions

Q: How much VRAM do I need for a 7B parameter model?
A: For inference in FP16, you need about 14GB plus overhead, so 24GB is safe. For training, you need roughly 4x that (56GB+), so 80GB GPUs are recommended.

Q: Can I use gaming GPUs like the RTX 4090 for AI?
A: Yes for small workloads, but they lack NVLink, have lower memory bandwidth, and are not designed for 24/7 data center use. They also void warranty in servers.

Q: What’s the difference between H100 and A100?
A: H100 is newer with higher FP8 throughput and faster memory (HBM3), but costs more. A100 is still capable for most models and often cheaper on the used market.

Q: Do I need a server-grade motherboard?
A: For 1-2 GPUs, a high-end workstation board works. For 4+, yes, because consumer boards lack PCIe lanes and proper power delivery.

Q: How much power will my server draw?
A: A 4x H100 server can draw 3.5-4kW under load. Check your circuit breaker; you may need a dedicated 30A 240V circuit.

Q: Should I buy a prebuilt or build my own?
A: Prebuilt servers from Dell, Supermicro, or Lenovo come with validated cooling and power. Building your own saves money but requires careful component matching.

Related

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →