GPU Capacity Planning for AI Training & Serving

Planning GPU capacity for an AI model means estimating how many GPUs you need to train the model within an acceptable time and to serve inference requests without excessive latency or cost. This guide walks you through the key calculations and decisions, from profiling your workload to choosing between on-premises and cloud GPUs. You’ll need basic familiarity with your model’s architecture, expected traffic, and latency targets.

What You’ll Need

• Model details: parameter count, architecture (transformer, CNN, etc.), input sequence length, batch size
• Training data size and desired number of epochs
• Inference traffic estimates: requests per second (RPS), peak vs. average, latency SLA (e.g., p95 under 100 ms)
• GPU specifications: memory (VRAM), compute (TFLOPS), memory bandwidth, interconnect (NVLink, PCIe)
• Budget and timeline constraints

Step-by-Step

1. Estimate training memory footprint. For a model with P parameters, memory for weights is roughly 2 bytes per parameter for FP16 (or 4 bytes for FP32). Add optimizer states (e.g., Adam uses 8 bytes per parameter), gradients (2 bytes), and activations (depends on batch size and sequence length). A rough rule: total VRAM per GPU ≈ (weights + gradients + optimizer) * 1.2 for overhead. For example, a 7B parameter model in mixed precision: weights 14 GB, gradients 14 GB, optimizer 56 GB → ~84 GB, so you’d need at least 2×80 GB GPUs just for training state, plus activation memory.
2. Determine training throughput per GPU. Measure or estimate tokens/second per GPU. For a given GPU (e.g., A100 80GB), a 7B model might process ~2,000-5,000 tokens/sec with batch size 32. Multiply by number of GPUs to get total throughput. Then compute total tokens in your dataset (e.g., 10B tokens) and divide by throughput to get training time. For example, 10B tokens / (3,000 tokens/sec * 8 GPUs) ≈ 48 hours.
3. Scale out with data parallelism. If one GPU doesn’t have enough memory, use model parallelism (shard layers across GPUs) or pipeline parallelism. For large models, use a combination: data parallel across nodes, tensor parallel within a node. The number of GPUs needed for training = max(memory requirement per GPU, throughput target).
4. Estimate inference memory. For serving, you need VRAM for the model weights (same as training weights) plus KV cache for transformers. KV cache size = 2 (K and V) * layers * hidden_size * sequence_length * batch_size * bytes_per_element (2 for FP16). For a 7B model with 32 layers, hidden size 4096, sequence length 2048, batch size 32: 2 * 32 * 4096 * 2048 * 32 * 2 ≈ 34 GB. Add to weights (14 GB) → ~48 GB, so one 80 GB GPU can handle that batch.
5. Calculate inference compute. Each token generation requires roughly 2 * P FLOPs (for forward pass). For a 7B model, that’s ~14 GFLOPs per token. With a target of 100 tokens/sec per request, that’s 1.4 TFLOPs per request. If a GPU delivers 312 TFLOPS (FP16 dense), it can handle ~220 requests/sec theoretically, but in practice you’ll get 20-30% of that due to memory bandwidth and overhead, so maybe 50-100 RPS per GPU.
6. Account for batching and concurrency. Inference servers batch requests to improve utilization. Determine max batch size that fits in memory (step 4) and then compute RPS per GPU = (batch_size * tokens_per_second_per_batch) / (average_tokens_per_request). For example, batch 32, each request 200 tokens, GPU processes 3,000 tokens/sec → 15 requests/sec per GPU.
7. Plan for peak traffic. Multiply average RPS by a peak factor (e.g., 2-3× for daily spikes). Add redundancy for failover (e.g., 20% headroom). For example, if average is 100 RPS, peak 300 RPS, and each GPU handles 15 RPS, you need 20 GPUs for peak, plus 4 spare = 24 GPUs.
8. Choose between on-premises and cloud. For cloud, you can scale dynamically with auto-scaling groups. For on-premises, you must purchase upfront. Consider GPU types: A100, H100, L4, T4 etc. H100 is ~2-3× faster than A100 for training but costs more. Use cloud for bursty workloads, on-prem for steady state.
9. Validate with a pilot. Run a small-scale training job and a load test on a single GPU to measure actual throughput and memory. Use profiling tools like NVIDIA Nsight Systems, PyTorch Profiler, or TensorBoard. Adjust your estimates based on real numbers.
10. Iterate and monitor. After deployment, monitor GPU utilization, memory, latency, and queue length. Use these metrics to right-size your cluster. For training, track loss convergence to ensure you’re not over/under-provisioning.

Common Mistakes

• Ignoring activation memory. Activations can dominate memory for long sequences. Use activation checkpointing to trade compute for memory.
• Assuming linear scaling. Adding GPUs doesn’t always double throughput due to communication overhead. Use scaling tests to find the sweet spot.
• Overestimating inference throughput. Real-world RPS is much lower than theoretical FLOPs due to memory bandwidth, kernel launch overhead, and variable request sizes.
• Not accounting for KV cache. For long sequences or large batches, KV cache can exceed model weights. Always calculate it.
• Forgetting about CPU and memory. GPUs need fast CPU, enough RAM, and high-bandwidth networking (InfiniBand or 100GbE) to feed data.

Tips

• Use mixed precision (FP16/BF16) to halve memory and double throughput.
• For serving, consider quantization (INT8) to reduce memory and increase speed, but watch for accuracy loss.
• Use model parallelism for models larger than a single GPU’s memory, but keep tensor parallelism within a node to avoid slow inter-node communication.
• For cloud, use spot instances for training (if fault-tolerant) and reserved instances for steady serving.
• Use a serving framework like TensorRT-LLM or vLLM to optimize inference (they handle batching and KV cache efficiently).
• Start with a conservative estimate and scale up based on actual metrics.

Troubleshooting

• Out of memory (OOM) during training. Reduce batch size, use gradient accumulation, enable activation checkpointing, or use a larger GPU.
• GPU utilization low (e.g., <50%). Check for data loading bottlenecks (use DataLoader with num_workers), communication overhead (use gradient accumulation to reduce sync frequency), or small batch sizes.
• Inference latency too high. Increase batch size (if memory allows), use dynamic batching, optimize model with TensorRT, or move to a faster GPU.
• Scaling not linear. Profile communication time. If it’s high, switch to a faster interconnect or use pipeline parallelism.
• Model doesn’t fit on one GPU. Use model parallelism or offload to CPU (slow). Consider using a smaller model or quantization.
• Cloud costs exploding. Use auto-scaling with a minimum and maximum, and consider spot instances for non-critical workloads.

Frequently Asked Questions

How do I estimate the number of GPUs for training a 70B parameter model?

For a 70B model in FP16, you need at least 140 GB for weights, 140 GB for gradients, and 280 GB for optimizer (Adam) → 560 GB total, so you’d need at least 8×80 GB GPUs (or 4×140 GB). In practice, you’ll need more for activations, so plan for 16×80 GB or use ZeRO/FSDP to shard optimizer states.

What’s the difference between training and serving GPU requirements?

Training requires high compute throughput and large memory for gradients and optimizer states. Serving requires lower compute but must handle latency-sensitive requests with batching. Serving GPUs can be smaller (e.g., T4) if the model is quantized, but training typically needs high-end GPUs like A100/H100.

Can I use the same GPUs for training and serving?

Yes, but it’s not ideal. Training is bursty and long-running, while serving is continuous and latency-sensitive. Mixing them can cause interference. Use separate clusters or schedule them at different times. Cloud makes this easy with separate node pools.

How do I calculate VRAM for inference with a transformer model?

VRAM = model weights (parameters × bytes per param) + KV cache (2 × layers × hidden_size × seq_len × batch_size × bytes) + overhead. For a 7B model with FP16, weights = 14 GB. For a batch of 32, seq_len 2048, 32 layers, hidden 4096, KV cache = 2 × 32 × 4096 × 2048 × 32 × 2 = 34 GB. Total ≈ 48 GB.

What is the rule of thumb for GPU memory per billion parameters?

For FP16 training, roughly 2 GB per billion parameters for weights, plus 2 GB for gradients, plus 8 GB for optimizer (Adam) = 12 GB per billion parameters. For inference, only weights: 2 GB per billion. So a 7B model needs ~84 GB for training and ~14 GB for inference.

How do I plan for future scaling?

Design your system to be modular: use Kubernetes with GPU node pools that can scale horizontally. Monitor usage trends and add capacity before you hit limits. For cloud, set up auto-scaling with a buffer of 20-30% above projected peak.

Related

• Why Python’s ‘is’ Works for Some Integers but Fails for Others
• How to Set Up Apple Cash Card on iPhone
• Assist Senior Using Computer: Step-by-Step Guide