Measure AI Infrastructure Efficiency Beyond GPU Utilization

This guide explains how to measure the true efficiency of an AI infrastructure once GPU utilization alone stops telling the full story. You will learn which metrics to track, how to collect them, and how to interpret them to find where your system actually wastes time, memory, or energy. No special tools are required beyond standard monitoring software and access to your cluster’s metrics endpoint.

What You’ll Need

Access to your AI infrastructure’s monitoring system (e.g., Prometheus, Grafana, Datadog, or cloud provider’s built-in metrics).
Ability to run basic commands on a server or pod (e.g., nvidia-smi, top, or kubectl top).
A representative workload (training or inference) that you can run repeatedly for baseline measurements.
A spreadsheet or notebook for logging metrics over time.
Optional: a profiling tool like TensorBoard, PyTorch Profiler, or NVIDIA Nsight Systems for deeper analysis.

Step-by-Step

Start with a single workload baseline. Run your representative workload once while logging all the metrics below. Record the wall-clock time from start to finish, the number of samples processed (or steps completed), and the total energy consumed if your hardware exposes it (e.g., via IPMI or NVIDIA’s power readings).
Measure memory utilization per device. For each GPU or accelerator, record the peak memory usage and the average memory usage during the run. Use nvidia-smi --query-gpu=memory.used --format=csv in a loop, or set up a Prometheus exporter. Low memory utilization (e.g., under 60%) often indicates the model is too small for the hardware or the batch size is too conservative.
Track data loading time. Instrument your training loop to log the time spent waiting for data (e.g., time.sleep in the data loader, or the time between when a batch is requested and when it arrives). In PyTorch, you can use profiler to see the DataLoader time. High data loading time (more than 20% of total step time) means your storage or network is the bottleneck.
Calculate effective throughput. Divide the number of samples processed by the total wall-clock time. This gives you samples per second. Then divide by the number of GPUs to get per-GPU throughput. This is a more meaningful number than raw GPU utilization because it reflects real work done.
Compute the utilization-to-throughput ratio. For each GPU, take the average GPU utilization (from step 1) and divide it by the per-GPU throughput. A high ratio (e.g., 90% utilization but low throughput) suggests the GPU is busy but not doing useful work—often due to memory stalls or inefficient kernels. A low ratio with high throughput is ideal.
Measure energy efficiency. If your infrastructure reports power draw (in watts), record the average power during the run. Calculate energy per sample as (average watts × total time) / total samples. This gives you joules per sample, which is a direct measure of efficiency that combines time and energy.
Profile kernel-level efficiency. Use a profiler like PyTorch Profiler or Nsight Systems to identify the top 10 most time-consuming kernels. Look for kernels that take more than 10% of total time but have low occupancy (below 50%). This indicates algorithmic inefficiency, not just hardware utilization.
Check for stragglers in distributed training. If you use multiple GPUs or nodes, record the time each GPU takes to complete a step. If one GPU is consistently slower (by more than 5%), you have a load imbalance or network bottleneck. Use nccl-tests to measure inter-GPU bandwidth.
Run a stress test with different batch sizes. Increase the batch size until you hit an out-of-memory error, then back off by 10%. Run the workload with this optimal batch size and compare the metrics. If throughput doesn’t increase proportionally, you may be memory-bandwidth bound, not compute bound.
Compare against a theoretical peak. Look up your GPU’s peak FLOPS (e.g., for A100, ~19.5 TFLOPS FP32). Calculate your achieved FLOPS as (total operations per sample × samples per second) / number of GPUs. Divide by peak FLOPS to get a percentage. Anything above 50% is excellent; below 20% suggests major inefficiencies.
Monitor network and I/O separately. Track network throughput (e.g., nload) and disk I/O (e.g., iostat) during the run. If network is near 100% of its link speed, your communication is the bottleneck. If disk I/O is high, consider preloading data into RAM.

Common Mistakes

Relying solely on GPU utilization percentage. A GPU can show 95% utilization while spending most cycles waiting for memory access. Always pair utilization with throughput.
Measuring only peak memory, not average. Peak memory might be fine, but if average is much lower, you’re wasting memory capacity that could be used for larger batches.
Ignoring data loading time. Many teams optimize GPU code while the real bottleneck is a slow data pipeline. Always profile data loading separately.
Using wall-clock time without accounting for queuing. If your workload shares the cluster, wall-clock time includes waiting for resources. Use pure compute time for efficiency metrics.
Not normalizing for model size. A larger model will naturally have lower utilization. Compare efficiency only between runs of the same model, or use a metric like samples per second per parameter.

Tips

Automate metric collection. Set up a script that logs all metrics to a CSV file every few seconds. This makes it easy to compare runs.
Use a consistent workload. Run the same model and dataset each time you test changes. This isolates the effect of your optimization.
Look for memory-bound patterns. If increasing batch size doesn’t improve throughput, your kernel is likely memory-bound. Try using mixed precision or enabling tensor cores.
Consider using a profiling dashboard. Tools like TensorBoard can show GPU utilization, memory, and data loading time in one place.
Check for CPU bottlenecks. If CPU utilization is near 100% while GPU is low, your data preprocessing is too heavy. Move it to the GPU or use a faster library.

Troubleshooting

GPU utilization is high but throughput is low. This usually means memory stalls. Use nvidia-smi -q -d MEMORY to check memory clock speed and see if it’s throttled. Try reducing the number of threads per block or using shared memory more effectively.
Data loading time is high. First, check if your storage is SSD. If not, move data to SSD. Then, increase the number of data loader workers and enable pin_memory=True in PyTorch. If still slow, try using a larger prefetch factor.
One GPU is consistently slower. Check network latency between nodes using ping and bandwidth with iperf. If it’s a single node, check for thermal throttling—use nvidia-smi -q -d TEMPERATURE. Also, verify that the PCIe link is running at the correct speed (e.g., x16).
Energy efficiency numbers seem off. Ensure you’re measuring power at the GPU level, not the entire server. Use nvidia-smi --query-gpu=power.draw. Also, make sure the workload runs long enough to average out spikes.
Profiler shows low occupancy but you can’t fix it. This may be due to register pressure. Try using __launch_bounds__ to limit registers and increase occupancy. If the kernel is from a library, consider using a different implementation (e.g., cuDNN vs. native).

Frequently Asked Questions

What is a good GPU utilization number?

Above 80% is generally good for compute-bound workloads, but it’s not the whole story. A model that uses 80% utilization but only achieves 30% of peak FLOPS is still inefficient. Always pair utilization with throughput.

How do I measure FLOPS achieved?

You need to know the number of floating-point operations per sample (e.g., from the model’s forward pass). Multiply by samples per second to get FLOPS. Then divide by the GPU’s peak FLOPS (from the spec sheet).

Should I measure efficiency for training or inference?

Both, but the metrics differ. For training, focus on throughput and convergence time. For inference, focus on latency and throughput under a given latency budget. The same principles apply, but inference often has more overhead from input/output processing.

Is energy efficiency more important than time efficiency?

It depends on your goals. If you’re running a large training run, energy cost might be a significant factor. If you’re serving inference, latency is usually more critical. Measure both and decide based on your priorities.

Can I use these metrics in the cloud?

Yes, most cloud providers offer metrics like GPU utilization, memory, and network. Some also provide power usage if you enable the option. Use their monitoring tools or export to your own system.

What if my workload is not compute-heavy, like data preprocessing?

For CPU-bound workloads, GPU utilization will be low, which is fine. Instead, measure CPU utilization, memory bandwidth, and I/O. The same principle applies: look for bottlenecks in the resource that is actually used.

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →