This guide walks you through measuring GPU utilization and pinpointing bottlenecks in enterprise AI workloads. You’ll learn which metrics matter, which tools to use, and how to interpret the data to improve training and inference performance.
What You’ll Need
- Access to the GPU servers (SSH or a monitoring dashboard)
- NVIDIA GPUs (most common in enterprise AI) – the tools below are NVIDIA-specific; for AMD or Intel, use their equivalents
- NVIDIA driver installed (version 450+ recommended for full metrics)
- Basic command-line familiarity
- Optional: Prometheus + Grafana for historical tracking
Step-by-Step
- Start with nvidia-smi for a quick snapshot. Run
nvidia-smito see current GPU utilization, memory usage, temperature, and power draw. The “Util” column shows the percentage of time during the sampling window that the GPU had at least one kernel executing. This is your first clue, but it’s coarse – a GPU can show 100% utilization while being memory-bound or stalled. - Use nvidia-smi dmon for real-time per-GPU metrics. Run
nvidia-smi dmon -s pucvmetto get a rolling display of power, utilization, clock speeds, memory, temperature, and more. Watch for patterns: if utilization is high but power is low, the GPU may be stalled on memory or synchronization. - Install and run nvidia-smi’s full metrics via NVML. For scripted monitoring, use the NVML Python bindings (
pynvml) or thenvidia-smi --query-gpu=...command. Query key fields:utilization.gpu,utilization.memory,clocks.sm,clocks.mem,power.draw,temperature.gpu, andpcie.link.gen.current. - Profile your workload with NVIDIA Nsight Systems. Run
nsys profile --trace=cuda,nvtx,osrt -o my_profile python train.py. This captures kernel execution times, memory transfers, and CPU activity. Open the resulting .nsys-rep file in Nsight Systems GUI to see a timeline. Look for gaps between kernels (idle GPU) and long memory copies. - Drill into kernel-level details with Nsight Compute. For a specific kernel that appears slow, run
ncu --set full -o kernel_profile python train.py. This gives metrics like compute throughput, memory throughput, and stall reasons. The “Speed of Light” section shows how close you are to theoretical peak for compute and memory. - Check PCIe bandwidth utilization. If you see high memory usage but low GPU utilization, the bottleneck might be data transfer over PCIe. Use
nvidia-smi pmon -c 1to see per-process memory and GPU usage, and checknvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.width.current --format=csvto ensure you’re running at the expected PCIe generation and width. - Monitor CPU and system memory. Often the GPU is waiting on the CPU to feed it data. Use
toporhtopto see if CPU cores are pegged. If you have many CPU threads busy but GPU utilization is low, your data pipeline is the bottleneck. Check disk I/O withiostat -x 1as well. - Set up continuous monitoring. For enterprise workloads, install DCGM (NVIDIA Data Center GPU Manager) and export metrics to Prometheus. Use Grafana dashboards to track utilization, power, temperature, and errors over time. This helps spot intermittent bottlenecks that don’t show up in a single snapshot.
- Correlate with training logs. If you’re using a framework like PyTorch or TensorFlow, log loss and throughput (samples/sec) per step. A sudden drop in throughput alongside low GPU utilization indicates a bottleneck (e.g., network sync, data loading).
- Interpret the data. Here’s a quick cheat sheet:
- GPU util > 90%, power near TDP, memory high → healthy compute-bound workload.
- GPU util high but power low (<50% TDP) → memory-bound or latency-bound (small kernels, dependencies).
- GPU util low, CPU busy → data pipeline or preprocessing bottleneck.
- GPU util low, CPU idle → waiting on I/O, network, or synchronization.
- Memory util high but GPU util low → memory bandwidth saturation or excessive data transfers.
Common Mistakes
- Relying solely on the GPU utilization percentage. It’s a coarse measure; a GPU can be 100% utilized but still underperforming due to memory stalls.
- Ignoring power draw. Low power with high utilization is a red flag for stalls.
- Forgetting to check CPU and disk I/O. The GPU is often waiting on data.
- Using default batch sizes that are too small, causing kernel launch overhead to dominate.
- Not accounting for multi-GPU synchronization overhead in distributed training.
- Sampling too infrequently – a 1-second sample can miss short-lived bottlenecks.
Tips
- Use
nvidia-smi --query-gpu=utilization.gpu,memory.used,power.draw --format=csv -l 1to log metrics every second to a file for later analysis. - Increase batch size to improve GPU utilization, but watch memory limits.
- Use mixed precision (FP16) to reduce memory bandwidth pressure and speed up compute.
- Ensure your data loading uses multiple workers and prefetching to keep the GPU fed.
- For inference, use batching and model optimization (TensorRT) to reduce latency and improve utilization.
- Check if your GPU is running at the correct PCIe generation and width; a x8 slot can bottleneck data transfer.
Troubleshooting
- nvidia-smi shows no GPUs. Check if the driver is loaded (
lsmod | grep nvidia) and if the GPU is visible inlspci | grep -i nvidia. Reinstall the driver if needed. - GPU utilization stays at 0%. The process might be using CPU only, or the GPU is in a low-power state. Run
nvidia-smito see if any process is using the GPU. If not, check your code for CUDA errors or ensure the device is set correctly. - Utilization fluctuates wildly. This is often due to data loading or network synchronization. Profile with Nsight Systems to identify gaps.
- Memory utilization is always near 100%. You may be hitting memory capacity limits, causing swapping or OOM. Reduce batch size or use gradient accumulation.
- Power draw is much lower than TDP. Check if the GPU is in a power-saving mode. Use
nvidia-smi -q -d POWERto see power limits. You can increase the limit withnvidia-smi -pl 300(if allowed). - Profiling with Nsight Compute fails. Ensure you have the correct permissions and that the GPU is not in use by another process. Run with
sudoif necessary.
Frequently Asked Questions
What is considered good GPU utilization?
For compute-bound training, 90%+ is excellent. For inference, it depends on latency requirements; you might intentionally keep utilization lower to minimize response time.
Why is my GPU utilization low but memory usage high?
This often indicates memory bandwidth saturation or that the model is too large for the GPU, causing memory swaps. It could also be that your kernels are memory-bound (e.g., large matrix multiplications with low arithmetic intensity).
How do I measure GPU utilization in a container?
Use nvidia-smi inside the container if you have the NVIDIA runtime. Alternatively, use nvidia-smi on the host and filter by process ID (the container’s process is visible on the host).
What’s the difference between GPU utilization and memory utilization?
GPU utilization is the percentage of time the GPU’s compute units are busy. Memory utilization is the percentage of memory bandwidth used, not capacity. High memory utilization means the GPU is moving data, which can be a bottleneck.
Can I use these tools on AMD or Intel GPUs?
No, nvidia-smi, Nsight, and DCGM are NVIDIA-specific. For AMD, use ROCm tools like rocm-smi and rocprof. For Intel, use intel_gpu_top and VTune.
How often should I sample utilization?
For real-time debugging, 1-second intervals are fine. For long-term monitoring, 15-30 second intervals are typical. For profiling, use Nsight’s event-based capture, which is more precise.
Related
General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.
More in this bed
- Fix Text Cursor Turning White in Windows 11
- Force Quit Non-Responding Apps on Mac: 6 Ways
- Chromebook Not Charging? Fix It Fast
- iPhone Won’t Make Calls but Texts Work? Fixes
- Android to iPhone Hotspot: Connect & Fix Issues
- Fix d3dx9 38.dll Missing or Not Designed to Run on Windows