sodcalculator

10 Tech How-To

Securely Expose NVIDIA GPU Metrics for Remote Monitoring

Guide 10-44314 · Updated · Step by step

This guide walks you through setting up a secure pipeline to expose NVIDIA GPU metrics (utilization, temperature, memory, power) for remote monitoring using Prometheus and node_exporter with textfile collector, or a dedicated exporter, all behind authentication and TLS. By the end, you’ll have a metrics endpoint that only authorized clients can access, protecting your GPU data from unauthorized snooping.

What You’ll Need

  • A Linux server with an NVIDIA GPU (driver installed, nvidia-smi works)
  • Root or sudo access on that server
  • A remote monitoring system (e.g., Prometheus, Grafana) that can scrape HTTP endpoints
  • Basic familiarity with the command line and systemd

Step-by-Step

  1. Install the NVIDIA DCGM exporter (recommended) or use the textfile collector. The DCGM exporter provides rich GPU metrics and is the standard for NVIDIA GPUs. Download the latest release from the NVIDIA/dcgm-exporter GitHub repository. Alternatively, if you prefer a lightweight setup, you can use the node_exporter textfile collector with a script that runs nvidia-smi and formats the output. For this guide, we’ll assume the DCGM exporter.
  2. Run the DCGM exporter as a systemd service. Create a systemd unit file at /etc/systemd/system/dcgm-exporter.service with the following content:
    [Unit]
    Description=DCGM Exporter
    After=network.target
    
    [Service]
    ExecStart=/usr/local/bin/dcgm-exporter
    Restart=always
    User=nobody
    Group=nogroup
    
    [Install]
    WantedBy=multi-user.target

    Adjust the path to the binary. Enable and start the service: sudo systemctl enable --now dcgm-exporter.

  3. Verify the exporter is running. Check the service status and test the metrics endpoint locally: curl http://localhost:9400/metrics. You should see output starting with # HELP and # TYPE lines, plus metrics like DCGM_FI_DEV_GPU_UTIL.
  4. Secure the endpoint with basic authentication. Since the exporter itself doesn’t support auth, put a reverse proxy in front of it. Install nginx or Apache. For nginx, create a config file at /etc/nginx/sites-available/dcgm with:
    server {
        listen 9401 ssl;
        server_name _;
    
        ssl_certificate /etc/nginx/ssl/your.crt;
        ssl_certificate_key /etc/nginx/ssl/your.key;
    
        auth_basic "Restricted";
        auth_basic_user_file /etc/nginx/.htpasswd;
    
        location / {
            proxy_pass http://127.0.0.1:9400;
            proxy_set_header Host $host;
        }
    }

    Generate a self-signed certificate or use a proper CA-signed one. Create the password file with sudo htpasswd -c /etc/nginx/.htpasswd monitoring and set a strong password.

  5. Enable the nginx site and reload. Symlink the config to sites-enabled and run sudo nginx -t then sudo systemctl reload nginx.
  6. Configure your remote Prometheus to scrape the secured endpoint. In your Prometheus prometheus.yml, add a scrape config with basic auth and TLS settings:
    scrape_configs:
      - job_name: 'nvidia_gpu'
        scheme: https
        basic_auth:
          username: monitoring
          password: 'your_password'
        tls_config:
          insecure_skip_verify: true  # only if self-signed
        static_configs:
          - targets: ['your-server-ip:9401']

    Replace the password with the one you set. If using a proper CA, remove the insecure_skip_verify line.

  7. Test the scraping. Restart Prometheus and check the targets page for the job. It should show UP. Query a metric like DCGM_FI_DEV_GPU_UTIL in Prometheus to confirm data is flowing.
  8. Optionally, add firewall rules. Restrict access to port 9401 to only your monitoring server’s IP using ufw or iptables. For example: sudo ufw allow from 192.168.1.100 to any port 9401.

Common Mistakes

  • Exposing the exporter directly on port 9400 without auth. This allows anyone on the network to see your GPU metrics, which may reveal workload patterns.
  • Using a weak password or leaving the htpasswd file with default permissions. Always set a strong password and ensure the file is readable only by the nginx user.
  • Forgetting to enable the nginx site or not reloading after config changes. This results in a connection refused error.
  • Using HTTP instead of HTTPS. Basic auth sends credentials in base64, which is easily decoded if intercepted. Always use TLS.
  • Not setting the insecure_skip_verify flag when using self-signed certs. Prometheus will reject the certificate and the scrape will fail.

Tips

  • Use a dedicated monitoring user with minimal permissions, not root, for the exporter service.
  • Set up alerting rules in Prometheus for GPU temperature or utilization thresholds to get proactive notifications.
  • Regularly update the DCGM exporter and nginx to get security patches.
  • If you have multiple GPUs, the DCGM exporter automatically exposes metrics for all of them, labeled by GPU index.
  • Consider using a VPN or SSH tunnel if you want an extra layer of security without exposing the port publicly.

Troubleshooting

  • Connection refused on port 9400. Check that the DCGM exporter service is running (systemctl status dcgm-exporter) and listening (ss -tlnp | grep 9400).
  • 401 Unauthorized when scraping. Verify the username and password in Prometheus match the htpasswd file. Test with curl: curl -u monitoring:password https://your-server:9401/metrics.
  • Certificate errors in Prometheus. If using self-signed, ensure insecure_skip_verify: true is set. If not, check that the certificate is valid for the hostname you’re using.
  • Metrics are empty or missing. The exporter may not have permission to access the GPU. Check the service logs with journalctl -u dcgm-exporter for errors like “Failed to initialize DCGM”.
  • Nginx returns 502 Bad Gateway. The proxy_pass target may be wrong or the exporter is not running. Check that curl http://127.0.0.1:9400/metrics works locally.

Frequently Asked Questions

Can I use the node_exporter textfile collector instead?

Yes, but it’s less efficient and requires a cron job to run nvidia-smi and write to a file. The DCGM exporter is the recommended approach for production.

How do I add GPU metrics to Grafana?

Add Prometheus as a data source, then import a dashboard like the NVIDIA DCGM Exporter dashboard (ID 12239) from Grafana.com.

Is basic auth enough for security?

Basic auth over HTTPS is acceptable for most internal monitoring setups. For higher security, use mTLS or a VPN.

Can I expose metrics on a different port?

Yes, change the nginx listen port and the Prometheus target accordingly.

What if I have multiple servers with GPUs?

Set up the same exporter on each server and add multiple targets in Prometheus. Use labels to differentiate them.

Do I need to open ports in the firewall?

Yes, if your monitoring server is remote, allow access to the port you chose (9401) from that server’s IP only.

Related

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →