sodcalculator

10 Tech How-To

Monitor GPU Cloud Reliability, Capacity & Job Failures

Guide 10-48663 · Updated · Step by step

This guide walks you through setting up practical monitoring for GPU cloud instances, covering reliability (uptime and health), capacity (utilization and quotas), and job failures (crash detection and retries). You’ll learn what metrics to track, which tools to use, and how to build a simple alerting pipeline that works across major cloud providers.

What You’ll Need

  • Access to your cloud provider’s console (AWS, Azure, GCP, or similar) with permissions to view metrics and create alarms.
  • Command-line access to your GPU instances (SSH) for installing monitoring agents.
  • Basic familiarity with shell commands and JSON/YAML configuration files.
  • An optional alerting destination: email, Slack webhook, or PagerDuty endpoint.

Step-by-Step

  1. Identify your critical GPU instances. List all instances that run training or inference jobs. Note their instance IDs, regions, and the jobs they run. This becomes your monitoring inventory.
  2. Enable cloud-level monitoring. In your provider’s console, turn on detailed monitoring for each GPU instance. For AWS, this means enabling “Detailed monitoring” in EC2; for GCP, ensure “Cloud Monitoring” agent is installed; for Azure, enable “Diagnostic settings” for your VM. This gives you 1-minute or 5-minute granularity for CPU, network, and disk metrics.
  3. Install a GPU metrics exporter. On each instance, install the NVIDIA Data Center GPU Manager (DCGM) or the `nvidia-smi` based exporter (e.g., `dcgm-exporter` for Prometheus). This exposes GPU-specific metrics like utilization, memory usage, temperature, and power draw. For a quick start, run `nvidia-smi –query-gpu=utilization.gpu,memory.used –format=csv` to verify it works.
  4. Set up a metrics collection endpoint. Deploy Prometheus (or a managed service like Amazon Managed Service for Prometheus, Azure Monitor, or Google Cloud Managed Service for Prometheus) to scrape the exporters. Configure a scrape job targeting each GPU instance’s IP:port (default for DCGM exporter is 9400). Test with `curl http://:9400/metrics` to see GPU metrics.
  5. Create dashboards for capacity. In Grafana (or your provider’s dashboard), build a view showing: GPU utilization (%), memory used vs. total, temperature, and power draw. Add a panel for “GPU memory available” to spot capacity bottlenecks. Set a threshold line at 80% utilization to flag near-saturation.
  6. Define reliability alerts. Create alerts for: instance status checks failing (e.g., EC2 status check failed), GPU temperature above 85°C, GPU memory errors (from `nvidia-smi` ECC counters), and unexpected instance reboots. Use cloud alarms (CloudWatch, Azure Monitor, etc.) or Prometheus alert rules. Example Prometheus rule: `alert: GPUHighTemp` if `nvidia_gpu_temperature > 85` for 5 minutes.
  7. Monitor job failures via logs. Ensure your job runner (Slurm, Kubernetes, or custom script) writes structured logs to stdout/stderr. Forward these logs to a central log aggregator (CloudWatch Logs, Stackdriver, or Loki). Create a log-based alert that triggers on patterns like “ERROR”, “CUDA error”, “OutOfMemory”, or “Traceback”.
  8. Implement job retry and dead-letter queues. For batch jobs, configure retries with exponential backoff (e.g., 3 retries). If a job fails permanently, send a notification to your alerting channel. In Kubernetes, use a Job with `backoffLimit`; in cloud batch services, set max retries.
  9. Track capacity quotas. Check your cloud provider’s GPU quota limits (e.g., number of P100 instances allowed). Set up a budget alert when you use 80% of your quota. For AWS, use Service Quotas; for GCP, use IAM quotas; for Azure, use subscription limits.
  10. Set up a central alerting destination. Configure all alerts to send to a single channel (email, Slack, or PagerDuty). Include the instance ID, metric value, and a link to the dashboard in each alert. Test one alert by intentionally stopping a test instance.
  11. Create a weekly reliability report. Automate a summary of: total GPU hours, average utilization, number of job failures, and downtime incidents. Use Grafana reports or a cron job that queries metrics and emails a PDF.

Common Mistakes

  • Ignoring GPU memory errors. ECC errors can indicate failing hardware; monitor them and replace instances early.
  • Only monitoring utilization, not memory. A GPU can be 0% utilized but have full memory, blocking new jobs.
  • Alert fatigue. Setting thresholds too low (e.g., 50% utilization) generates constant false alarms. Use 80-90% for capacity alerts.
  • Forgetting to monitor the monitoring system. If your Prometheus server dies, you lose visibility. Set a heartbeat alert.
  • Not testing alerts. Alerts that never fire are useless; simulate failures periodically.

Tips

  • Use labels wisely. Tag instances with role (training, inference), project, and owner. This makes filtering and alerting much easier.
  • Start with a few key metrics. Don’t over-engineer. Begin with GPU utilization, memory, temperature, and job exit status. Add more as needed.
  • Leverage managed services. If you don’t want to run Prometheus, use cloud-native monitoring (CloudWatch, Azure Monitor) – they integrate with auto-scaling and IAM.
  • Automate remediation. For transient failures, use auto-restart policies (e.g., EC2 Auto Recovery) to restart unhealthy instances automatically.
  • Document your runbooks. Write down what to do when an alert fires: who to contact, where to look, and how to escalate.

Troubleshooting

  • No metrics appearing in Prometheus. Check that the exporter is running (`systemctl status nvidia-dcgm-exporter`), the port is open (firewall rules), and the scrape target is correct in `prometheus.yml`.
  • Alerts not firing. Verify the alert rule expression matches the metric name exactly (use `curl` to query Prometheus). Also check the `for` duration – if the condition lasts less than that, no alert.
  • Logs not showing in aggregator. Ensure the log agent (e.g., CloudWatch agent) is configured to read the correct log file path and has permissions. Test by writing a test log line.
  • GPU metrics show N/A. This often means the exporter can’t access the GPU. Run `nvidia-smi` to see if the driver is loaded. If not, reinstall the NVIDIA driver.
  • Quota alerts inaccurate. Cloud quotas are region-specific. Make sure you’re checking the right region and that you’ve requested quota increases if needed.

Frequently Asked Questions

What is the best tool for GPU monitoring?

For most teams, Prometheus + Grafana with DCGM exporter is the standard open-source stack. Cloud-native alternatives like CloudWatch or Azure Monitor are easier to set up but less flexible.

How often should I check GPU utilization?

For real-time capacity planning, scrape every 15-30 seconds. For historical trends, 1-minute granularity is sufficient. Avoid scraping more often than every 10 seconds to reduce overhead.

Can I monitor GPU jobs without changing my code?

Yes. Use external monitoring: `nvidia-smi` for GPU stats, and your job scheduler’s logs for failures. No code changes needed.

How do I detect a GPU failure before it causes job crashes?

Watch for ECC errors, rising temperatures, and sudden drops in utilization. Set alerts on these metrics with a threshold that triggers before the GPU becomes unstable.

What should I do when a job fails due to GPU OOM?

First, check if the GPU memory was fully used. If so, reduce batch size or use gradient accumulation. If it’s a memory leak, monitor memory over time and restart the job periodically.

How do I set up alerts for multiple GPU clusters?

Use a central Prometheus server that scrapes all clusters, or use federation between Prometheus instances. Tag each cluster with a `cluster` label to filter alerts.

Related

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →