This guide walks you through setting up a secure pipeline to expose NVIDIA GPU metrics (utilization, temperature, memory, power) for remote monitoring using Prometheus and node_exporter with textfile collector, or a dedicated exporter, all behind authentication and TLS. By the end, you’ll have a metrics endpoint that only authorized clients can access, protecting your GPU data from unauthorized snooping.
What You’ll Need
- A Linux server with an NVIDIA GPU (driver installed,
nvidia-smiworks) - Root or sudo access on that server
- A remote monitoring system (e.g., Prometheus, Grafana) that can scrape HTTP endpoints
- Basic familiarity with the command line and systemd
Step-by-Step
- Install the NVIDIA DCGM exporter (recommended) or use the textfile collector. The DCGM exporter provides rich GPU metrics and is the standard for NVIDIA GPUs. Download the latest release from the NVIDIA/dcgm-exporter GitHub repository. Alternatively, if you prefer a lightweight setup, you can use the node_exporter textfile collector with a script that runs
nvidia-smiand formats the output. For this guide, we’ll assume the DCGM exporter. - Run the DCGM exporter as a systemd service. Create a systemd unit file at
/etc/systemd/system/dcgm-exporter.servicewith the following content:[Unit] Description=DCGM Exporter After=network.target [Service] ExecStart=/usr/local/bin/dcgm-exporter Restart=always User=nobody Group=nogroup [Install] WantedBy=multi-user.targetAdjust the path to the binary. Enable and start the service:
sudo systemctl enable --now dcgm-exporter. - Verify the exporter is running. Check the service status and test the metrics endpoint locally:
curl http://localhost:9400/metrics. You should see output starting with# HELPand# TYPElines, plus metrics likeDCGM_FI_DEV_GPU_UTIL. - Secure the endpoint with basic authentication. Since the exporter itself doesn’t support auth, put a reverse proxy in front of it. Install nginx or Apache. For nginx, create a config file at
/etc/nginx/sites-available/dcgmwith:server { listen 9401 ssl; server_name _; ssl_certificate /etc/nginx/ssl/your.crt; ssl_certificate_key /etc/nginx/ssl/your.key; auth_basic "Restricted"; auth_basic_user_file /etc/nginx/.htpasswd; location / { proxy_pass http://127.0.0.1:9400; proxy_set_header Host $host; } }Generate a self-signed certificate or use a proper CA-signed one. Create the password file with
sudo htpasswd -c /etc/nginx/.htpasswd monitoringand set a strong password. - Enable the nginx site and reload. Symlink the config to
sites-enabledand runsudo nginx -tthensudo systemctl reload nginx. - Configure your remote Prometheus to scrape the secured endpoint. In your Prometheus
prometheus.yml, add a scrape config with basic auth and TLS settings:scrape_configs: - job_name: 'nvidia_gpu' scheme: https basic_auth: username: monitoring password: 'your_password' tls_config: insecure_skip_verify: true # only if self-signed static_configs: - targets: ['your-server-ip:9401']Replace the password with the one you set. If using a proper CA, remove the
insecure_skip_verifyline. - Test the scraping. Restart Prometheus and check the targets page for the job. It should show UP. Query a metric like
DCGM_FI_DEV_GPU_UTILin Prometheus to confirm data is flowing. - Optionally, add firewall rules. Restrict access to port 9401 to only your monitoring server’s IP using ufw or iptables. For example:
sudo ufw allow from 192.168.1.100 to any port 9401.
Common Mistakes
- Exposing the exporter directly on port 9400 without auth. This allows anyone on the network to see your GPU metrics, which may reveal workload patterns.
- Using a weak password or leaving the htpasswd file with default permissions. Always set a strong password and ensure the file is readable only by the nginx user.
- Forgetting to enable the nginx site or not reloading after config changes. This results in a connection refused error.
- Using HTTP instead of HTTPS. Basic auth sends credentials in base64, which is easily decoded if intercepted. Always use TLS.
- Not setting the
insecure_skip_verifyflag when using self-signed certs. Prometheus will reject the certificate and the scrape will fail.
Tips
- Use a dedicated monitoring user with minimal permissions, not root, for the exporter service.
- Set up alerting rules in Prometheus for GPU temperature or utilization thresholds to get proactive notifications.
- Regularly update the DCGM exporter and nginx to get security patches.
- If you have multiple GPUs, the DCGM exporter automatically exposes metrics for all of them, labeled by GPU index.
- Consider using a VPN or SSH tunnel if you want an extra layer of security without exposing the port publicly.
Troubleshooting
- Connection refused on port 9400. Check that the DCGM exporter service is running (
systemctl status dcgm-exporter) and listening (ss -tlnp | grep 9400). - 401 Unauthorized when scraping. Verify the username and password in Prometheus match the htpasswd file. Test with curl:
curl -u monitoring:password https://your-server:9401/metrics. - Certificate errors in Prometheus. If using self-signed, ensure
insecure_skip_verify: trueis set. If not, check that the certificate is valid for the hostname you’re using. - Metrics are empty or missing. The exporter may not have permission to access the GPU. Check the service logs with
journalctl -u dcgm-exporterfor errors like “Failed to initialize DCGM”. - Nginx returns 502 Bad Gateway. The proxy_pass target may be wrong or the exporter is not running. Check that
curl http://127.0.0.1:9400/metricsworks locally.
Frequently Asked Questions
Can I use the node_exporter textfile collector instead?
Yes, but it’s less efficient and requires a cron job to run nvidia-smi and write to a file. The DCGM exporter is the recommended approach for production.
How do I add GPU metrics to Grafana?
Add Prometheus as a data source, then import a dashboard like the NVIDIA DCGM Exporter dashboard (ID 12239) from Grafana.com.
Is basic auth enough for security?
Basic auth over HTTPS is acceptable for most internal monitoring setups. For higher security, use mTLS or a VPN.
Can I expose metrics on a different port?
Yes, change the nginx listen port and the Prometheus target accordingly.
What if I have multiple servers with GPUs?
Set up the same exporter on each server and add multiple targets in Prometheus. Use labels to differentiate them.
Do I need to open ports in the firewall?
Yes, if your monitoring server is remote, allow access to the port you chose (9401) from that server’s IP only.
Related
General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.
More in this bed
- How to Create App Shortcuts in Windows 10 Start Menu
- How to Rename a SQL Server Database
- Fix Google Calendar Not Syncing on iPhone
- Fix Skype Microphone Not Working on Windows
- Google Docs Landscape: Change Page Orientation
- Recover Deleted Admin Account in Windows 11