sodcalculator

10 Tech How-To

Detect & Fix Unauthorized Access in GPU Cluster

Guide 10-57771 · Updated · Step by step

This guide helps you identify and stop unauthorized access on a GPU cluster—a shared resource that is a prime target for crypto-mining, data theft, and resource abuse. You’ll learn how to spot the warning signs, isolate the breach, and harden your cluster against repeat intrusions. Before starting, ensure you have administrative (root or sudo) access to the cluster’s head node and compute nodes, plus the ability to inspect running processes and network connections.

What You’ll Need

  • Administrative (root/sudo) access to the cluster’s head node and at least one compute node.
  • SSH access to the cluster’s login node and compute nodes.
  • Familiarity with basic Linux command-line tools (ps, netstat, lsof, grep).
  • Access to the cluster’s job scheduler (SLURM, PBS, or similar) to view job history and running jobs.
  • A log management tool or at least the ability to read system logs (journalctl, /var/log/auth.log, etc.).

Step-by-Step

  1. Check for unexpected GPU usage. Run nvidia-smi on each compute node. Look for processes that are not tied to a legitimate job. A common sign is a process running as a system user (like www-data or nobody) consuming high GPU memory or compute.
  2. Review running processes. Use ps aux --sort=-%cpu on the head node and compute nodes. Look for unfamiliar process names, especially those running from /tmp, /dev/shm, or a user’s home directory that shouldn’t be there. Note the PID and the user.
  3. Check for suspicious network connections. Run netstat -tulpn or ss -tulpn to see open ports and established connections. Look for connections to known mining pool addresses (e.g., ports 3333, 4444, 8888) or to unusual external IPs. Also check for outbound connections from processes you don’t recognize.
  4. Inspect the job scheduler. Use squeue (for SLURM) or qstat (for PBS) to list currently running jobs. Compare against expected jobs. If you see a job running under a user who isn’t supposed to be active, or a job that has been running for an unusually long time, investigate immediately.
  5. Examine authentication logs. Look at /var/log/auth.log or journalctl -u sshd for failed login attempts, successful logins at odd hours, or logins from unexpected IP addresses. Pay special attention to repeated password guessing or any successful login from a foreign IP.
  6. Check for unauthorized SSH keys. Inspect ~/.ssh/authorized_keys for every user on the cluster. Look for keys that you don’t recognize. An attacker often adds their own key to maintain access.
  7. Look for modified system files. Run rpm -Va (on RHEL/CentOS) or dpkg -V (on Debian/Ubuntu) to check for altered system binaries. Also check for cron jobs (crontab -l for each user, and /etc/cron*) that might be running malicious scripts.
  8. Isolate the affected node. Once you’ve identified a compromised node, immediately isolate it from the network. You can do this by disconnecting it from the cluster’s network (e.g., by disabling its network interface) or by using a firewall rule to block all traffic to and from that node.
  9. Kill the malicious processes. Use kill -9 <PID> to terminate any process you’ve identified as unauthorized. Be careful not to kill legitimate jobs. If you’re unsure, check with the job’s owner first.
  10. Remove the attacker’s foothold. Delete any unauthorized SSH keys, cron jobs, or startup scripts you found. Remove any suspicious files from /tmp, /dev/shm, and user home directories.
  11. Change all passwords and keys. Force a password change for all users, and rotate SSH host keys on all nodes. This ensures that any stolen credentials are no longer valid.
  12. Patch and harden. Update all software to the latest versions, especially the operating system, GPU drivers, and the job scheduler. Disable password-based SSH login and use key-based authentication only. Restrict SSH access to specific IP ranges if possible.
  13. Monitor going forward. Set up monitoring tools like nvidia-smi dmon (a built-in monitoring tool) or a more comprehensive solution like Prometheus + Grafana to track GPU usage, network connections, and process activity. Set up alerts for unusual activity.

Common Mistakes

  • Ignoring the login node. Attackers often target the login node because it’s the entry point. Monitor it as closely as compute nodes.
  • Only checking GPU usage. Unauthorized access might not always be about GPU mining. An attacker could be using the cluster for data exfiltration or as a launchpad for other attacks.
  • Not checking all nodes. A compromise might be isolated to one node. Always check every compute node, not just the one where you first noticed the issue.
  • Deleting evidence too quickly. Before killing processes or deleting files, take a snapshot of the running processes, network connections, and any suspicious files. This evidence is crucial for understanding the attack and preventing future ones.
  • Not changing all credentials. If you only change the password of the compromised user, the attacker might still have access through another user’s account or a stolen SSH key.

Tips

  • Use a centralized logging system (e.g., ELK stack) to collect logs from all nodes in one place. This makes it easier to spot patterns.
  • Implement a strict job scheduling policy that requires all jobs to be submitted through the scheduler. Disable interactive shells on compute nodes if possible.
  • Use nvidia-smi --query-compute-apps=pid,used_memory --format=csv to get a clean list of GPU processes that can be scripted and monitored.
  • Set up a firewall (e.g., iptables or firewalld) to block outbound connections from compute nodes to the internet, except for necessary services.
  • Regularly audit user accounts and remove inactive ones. The fewer accounts, the smaller the attack surface.

Troubleshooting

I can’t find any suspicious processes but GPU usage is high. This could be a legitimate job that is misusing resources. Check the job scheduler for any job that is running longer than expected or using more resources than requested. Also, check for kernel-level rootkits that hide processes. Use chkrootkit or rkhunter to scan for these.

I killed the process, but it comes back. The attacker likely has a persistence mechanism, such as a cron job, a systemd service, or a startup script. Check /etc/rc.local, /etc/systemd/system, and user crontabs. Also, check for a modified ld.so.preload file that could be loading a malicious library.

I’m not sure if a specific process is malicious. Look up the process name and its command line. If it’s running from a suspicious location like /tmp or /dev/shm, it’s almost certainly malicious. You can also check the file’s checksum against a known good version if it’s a system binary.

The attacker is still able to log in after I changed passwords. They might have a backdoor in the SSH daemon or a modified authorized_keys file that you missed. Reinstall the SSH service from a trusted source and thoroughly inspect all authorized_keys files again.

I’ve isolated the node, but the attack is spreading. This suggests lateral movement. Disconnect the entire cluster from the network if necessary, and perform a forensic analysis on each node. Consider reinstalling the operating system on compromised nodes.

Frequently Asked Questions

Q: How can I detect unauthorized access if I don’t have GPU monitoring?
A: Use system tools like ps and netstat to spot unusual processes and connections. Also, check CPU usage; crypto miners often cause high CPU usage as well.

Q: What are the most common signs of a GPU cluster being hacked?
A: Unexpected GPU utilization, processes running as system users, outbound connections to mining pools, and failed SSH login attempts are the most common indicators.

Q: Should I immediately shut down the cluster when I suspect a breach?
A: Not necessarily. First, isolate the affected node(s) to prevent further spread. Then, gather evidence before shutting down, as a shutdown can destroy volatile data like running processes.

Q: Can I prevent unauthorized access with a firewall?
A: Yes, a firewall can block inbound and outbound connections. Restrict SSH access to trusted IPs and block outbound connections to known malicious IPs. However, a firewall is not a substitute for strong authentication and regular monitoring.

Q: How often should I audit my cluster for unauthorized access?
A: At minimum, perform a monthly audit of user accounts, SSH keys, and running processes. If you have the resources, continuous monitoring with alerts is highly recommended.

Q: What should I do if I find a miner running as root?
A: This is a critical situation. Immediately isolate the node, kill the process, and change the root password. You should also assume that the entire cluster is compromised and reinstall the operating system on that node.

Related

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →