Monitor AI Inference: Latency, Throughput & GPU

This guide walks you through setting up a practical monitoring stack for AI inference services, covering latency, throughput, and GPU utilization. You’ll learn what to measure, which tools to use, and how to interpret the numbers to keep your model serving fast and efficient.

What You’ll Need

• A running AI inference server (e.g., TensorFlow Serving, PyTorch, Triton, or a custom FastAPI endpoint)
• Command-line access to the server (SSH or local terminal)
• Python 3.7+ installed (for writing a small test script)
• GPU drivers and CUDA installed (if you’re monitoring GPU utilization)
• Optional: Prometheus and Grafana for persistent dashboards

Step-by-Step

1. Define your metrics. Latency is the time from request to response, typically measured in milliseconds (ms) at percentiles like p50, p95, and p99. Throughput is the number of requests processed per second (RPS). GPU utilization is the percentage of time the GPU’s compute units are busy, plus memory usage.
2. Instrument your inference server. If you’re using a framework like Triton or TensorFlow Serving, they often expose metrics endpoints (e.g., /metrics in Prometheus format). For a custom app, add timing around the inference call and expose a counter for requests and a histogram for latency. Use a library like prometheus_client in Python.
3. Set up a monitoring agent. Install Prometheus on a separate VM or container. Configure it to scrape your inference server’s metrics endpoint every 5–15 seconds. Add a scrape config in prometheus.yml with the target URL and a job name like “inference”.
4. Collect GPU metrics. Use nvidia-smi for a quick snapshot: run nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv to see current values. For continuous monitoring, install the NVIDIA DCGM exporter and add it as another Prometheus target. It exposes GPU utilization, memory, temperature, and power.
5. Create a Grafana dashboard. Connect Grafana to your Prometheus data source. Build panels: a graph of latency percentiles (using histogram_quantile(0.95, sum(rate(latency_seconds_bucket[5m])) by (le))), a throughput graph (rate of request counter), and GPU utilization (from DCGM metric DCGM_FI_DEV_GPU_UTIL).
6. Run a load test. Write a simple Python script using requests or aiohttp to send a batch of requests (e.g., 1000) to your endpoint. Record the start time, send the request, and measure the elapsed time. Save the results to a file or send them to a test endpoint.
7. Analyze the results. From your load test, compute average latency, p95, and p99. Divide the number of requests by the total test duration to get throughput. Compare these with the GPU utilization during the test. If latency is high but GPU is under 50%, you may have a bottleneck elsewhere (CPU, I/O, network).
8. Set alerts. In Grafana, create alert rules for latency exceeding a threshold (e.g., p95 > 200ms) or GPU utilization staying at 100% for more than 5 minutes (indicating saturation). Send alerts to email, Slack, or PagerDuty.

Common Mistakes

• Measuring latency from the client only. Client-side latency includes network overhead. For server-side metrics, use the server’s own timing or add a X-Process-Time header.
• Ignoring batch size. Throughput and latency are highly dependent on batch size. A larger batch increases throughput but may increase latency per request. Always report batch size alongside your metrics.
• Using average latency as the only metric. Averages hide outliers. Always use percentiles (p95, p99) to understand tail latency.
• Not monitoring GPU memory. GPU utilization can be low while memory is nearly full, causing out-of-memory errors. Monitor both.
• Scraping too frequently. Scraping every second can overload the inference server. Keep the scrape interval at 5–15 seconds.

Tips

• Use a dedicated GPU for inference if possible; sharing with training can cause unpredictable latency.
• Consider using a queuing system (like RabbitMQ) to decouple requests from inference, but measure the queue latency separately.
• Profile your model with nsys or nvprof to find GPU kernel bottlenecks.
• For real-time monitoring, use a tool like nvidia-smi dmon to watch GPU metrics in a live terminal.
• Store historical metrics for at least 30 days to spot trends and plan capacity.

Troubleshooting

• Metrics not showing in Prometheus: Check the target health in Prometheus UI (Status > Targets). Ensure the endpoint is reachable and returns valid metrics. Try curl http://localhost:8000/metrics on the server.
• GPU utilization stays at 0%: Your process may not be using the GPU. Run nvidia-smi to see if your process appears. If not, check that CUDA is installed and your inference framework is compiled with GPU support.
• Latency spikes during load test: This could be due to CPU bottlenecks, network limits, or GPU thermal throttling. Check CPU usage and GPU temperature during the test.
• Grafana shows “No data”: Verify the Prometheus data source URL and that the metric names match. Use the Prometheus query explorer to test the query.
• Throughput is lower than expected: Check if the model is using dynamic batching (if supported). Enable it to increase throughput at the cost of latency.

Frequently Asked Questions

What is a good p99 latency for inference?

It depends on your use case. Real-time applications like chatbots often target p99 under 500ms, while offline batch processing can tolerate seconds.

How do I measure throughput accurately?

Run a load test with a fixed number of requests and divide the count by the total time. Use a tool like wrk or locust for more realistic loads.

Should I monitor GPU utilization per process?

Yes, if you have multiple models on one GPU. Use nvidia-smi --query-compute-apps=pid,used_memory --format=csv to see per-process usage.

Can I use cloud monitoring tools instead?

Yes, AWS CloudWatch, Azure Monitor, and GCP Monitoring provide GPU metrics for their managed inference services, but they may have less granularity than local tools.

How often should I scrape metrics?

Every 5–15 seconds is typical. More frequent scraping adds overhead. For high-frequency trading, you might need sub-second, but that’s rare.

What if my GPU utilization is 100% but latency is fine?

That’s actually ideal—you’re using the GPU efficiently. If latency is acceptable, you’re at full capacity; consider scaling out if you need more throughput.

Related

• How to Install Windows After Replacing Motherboard
• Microsoft Store Error 0x80131505: Fixes That Work
• How to Lock Text Messages on a Samsung Galaxy Phone