sodcalculator

10 Tech How-To

Check If Canceled Agent Stopped GPU Job

Guide 10-49969 · Updated · Step by step

When you cancel an agent that was running a GPU job, the job may not always stop immediately—or at all. This guide shows you how to confirm whether the GPU process actually terminated, how to find orphaned processes, and what to do if the job is still running.

What You’ll Need

  • Access to the machine where the agent ran (SSH, console, or remote desktop).
  • Command-line access (terminal or PowerShell).
  • Basic knowledge of process management commands (ps, kill, tasklist, etc.).
  • If the GPU is shared, check with other users before killing processes.

Step-by-Step

  1. Check the agent’s status. Look at the agent’s dashboard or logs to see if it reports a canceled state. If the agent process itself is gone, that’s a good sign, but the GPU job it spawned might still be running.
  2. List GPU processes. Use nvidia-smi on Linux/Windows with NVIDIA GPUs, or rocm-smi for AMD. Run nvidia-smi and look for processes under the “Processes” section. If your job’s PID or command name appears, it’s still running.
  3. Find the process tree. If you see a process, identify its parent PID (PPID). Use ps -ef (Linux) or tasklist /v (Windows) to trace back to the agent. If the parent is the agent and the agent is gone, the process is orphaned.
  4. Check for child processes. Even if the main job process is gone, it may have spawned children (e.g., data loaders, subprocesses). Use pgrep -P <PPID> on Linux or wmic process where (ParentProcessId=PPID) get ProcessId,CommandLine on Windows.
  5. Verify GPU memory usage. A job that’s still running will hold GPU memory. Compare nvidia-smi memory usage before and after cancellation. If memory is still allocated, the job hasn’t fully stopped.
  6. Check logs for termination messages. Look at the job’s stdout/stderr logs. If cancellation was successful, you should see a graceful shutdown message or a kill signal notice. If not, the job may have ignored the signal.
  7. If the job is still running, force it to stop. Use kill <PID> (SIGTERM) first, wait a few seconds, then kill -9 <PID> (SIGKILL) if needed. On Windows, use taskkill /PID <PID> /F.
  8. Clean up orphaned processes. If you find child processes, kill them as well, usually with pkill -P <PPID> or manually.
  9. Verify again. Run nvidia-smi again to confirm no processes remain and GPU memory is freed.

Common Mistakes

  • Assuming the agent cancellation kills the job. Many agents only send a signal; the job must handle it. If the job doesn’t trap the signal, it keeps running.
  • Only checking the agent’s UI. The UI might show “canceled” even if the underlying process is still alive.
  • Ignoring child processes. The main process might die, but its children continue to use the GPU.
  • Using kill without checking the PID. Make sure you’re killing the right process; killing a system process can cause instability.

Tips

  • Use nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv for a clean list of GPU processes.
  • Set a timeout in your job script to automatically kill itself if the parent agent disappears.
  • Use strace -p <PID> on Linux to see if the process is still doing work or stuck.
  • Document the expected behavior of your agent and job so you know what “stopped” looks like.

Troubleshooting

If the job is still running after you kill it: Try sending SIGKILL directly. If it’s unkillable (D state on Linux), you may need to reboot the machine, but first check if it’s a zombie process (already dead but not reaped).

If nvidia-smi shows no processes but GPU memory is still high: The memory might be leaked by a process that exited but didn’t release memory. Restart the GPU driver or reboot if necessary.

If you can’t find the PID: Use fuser -v /dev/nvidia* to see which processes have the GPU device files open.

If the agent won’t cancel at all: Check if the agent is hung. You may need to kill the agent process itself, then clean up the job processes.

Frequently Asked Questions

How long should I wait after canceling before checking?

Give it a few seconds for the signal to be processed. If the job handles SIGTERM gracefully, it might take longer to clean up. Check after 10-30 seconds.

What if the job is running on a remote GPU server?

You need SSH access to that server. The steps are the same, but you’ll run commands over SSH.

Can I prevent orphaned GPU jobs in the future?

Yes, use a job wrapper that monitors the parent PID and kills itself if the parent dies. Many job schedulers (Slurm, Kubernetes) handle this automatically.

Is it safe to kill a GPU process with kill -9?

It’s generally safe for your job, but it can leave GPU memory in an inconsistent state. The driver usually recovers, but a reboot might be needed if you see errors.

How do I check GPU usage on Windows?

Use nvidia-smi in Command Prompt, or Task Manager’s Performance tab. For process-level detail, use nvidia-smi or PowerShell with Get-Process.

What does “D state” mean in ps output?

It means the process is in uninterruptible sleep, usually waiting on I/O. It cannot be killed until the I/O completes, which might require a reboot.

Related

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →