Fix Kubernetes Pods Stuck Waiting for a GPU

This guide walks you through diagnosing and resolving Kubernetes pods that remain in a Pending state because they cannot be scheduled onto a node with an available GPU. You will learn how to verify GPU availability, check scheduler events, and correct common configuration issues that block GPU allocation.

What You’ll Need

• Access to the Kubernetes cluster via kubectl with permissions to view nodes, pods, and events.
• NVIDIA GPU drivers installed on the worker nodes (for NVIDIA GPUs).
• The NVIDIA device plugin daemonset deployed (or the appropriate vendor plugin for your GPU type).
• Basic familiarity with Kubernetes YAML manifests and kubectl commands.

Step-by-Step

1. Check the pod status and events
   Run kubectl describe pod <pod-name> -n <namespace>. Look at the Events section for scheduling errors such as 0/3 nodes are available: 3 Insufficient nvidia.com/gpu. This tells you no node has enough GPU resources.
2. Verify GPU resources on nodes
   Run kubectl get nodes -o json | jq '.items[].status.allocatable' and look for nvidia.com/gpu. If the key is missing, the device plugin is not running or not registered.
3. Check the device plugin daemonset
   Run kubectl get daemonset -n kube-system | grep nvidia. If no daemonset exists, deploy the NVIDIA device plugin. If it exists, check its pods: kubectl get pods -n kube-system -l name=nvidia-device-plugin. Ensure they are Running and not in CrashLoopBackOff.
4. Inspect device plugin logs
   For a failing plugin pod, run kubectl logs <plugin-pod> -n kube-system. Common errors include missing drivers, incorrect library paths, or inability to access /dev/nvidia* devices.
5. Verify GPU drivers on the node
   SSH into a node and run nvidia-smi. If it fails, the driver is not installed or not loaded. Reinstall the driver and reboot the node.
6. Check if the pod requests GPUs correctly
   Inspect your pod spec: kubectl get pod <pod-name> -n <namespace> -o yaml. Look for resources.limits and resources.requests containing nvidia.com/gpu: 1. If you only set limits without requests, Kubernetes may not schedule correctly—set both to the same value.
7. Check for node selectors and taints
   Run kubectl describe node <node-name> to see taints. If the node has a taint like nvidia.com/gpu=present:NoSchedule, your pod must have a matching toleration. Also verify your pod’s nodeSelector or affinity matches the GPU node labels.
8. Check if GPUs are already allocated
   Run kubectl describe node <node-name> and look at Allocatable vs Allocated resources. If all GPUs are used by other pods, you need to scale down or add nodes.
9. Check for extended resource registration
   If the device plugin is running but the node does not show nvidia.com/gpu, the plugin may not have registered. Restart the plugin daemonset: kubectl rollout restart daemonset/nvidia-device-plugin -n kube-system.
10. Verify the container runtime supports GPUs
    If using Docker, ensure nvidia-container-runtime is configured. For containerd, the nvidia-container-runtime must be set as the default runtime in /etc/containerd/config.toml. Misconfiguration here prevents the plugin from seeing GPUs.

Common Mistakes

• Forgetting to set resources.requests for GPUs—only setting limits can cause scheduling issues.
• Using the wrong resource name, e.g., nvidia.com/gpu vs nvidia.com/gpu-memory (the latter is for MIG).
• Not adding tolerations for GPU node taints.
• Assuming all nodes have GPUs—verify with kubectl get nodes -o wide.
• Running the device plugin on nodes without drivers, causing crash loops.

Tips

• Use kubectl get events --sort-by=.metadata.creationTimestamp to see recent cluster-wide events that might indicate node issues.
• Label GPU nodes clearly, e.g., kubectl label node <node> gpu=present, and use nodeSelector in your pod spec to target them.
• For clusters with mixed GPU types, use device plugin’s nvidia.com/gpu.product label to differentiate.
• Test GPU scheduling with a simple pod like nvidia/cuda:11.0-base running nvidia-smi to isolate the issue.
• Keep the device plugin version aligned with your Kubernetes version and driver version.

Troubleshooting

If the pod remains stuck after following the steps:

• Re-check the device plugin logs—a common error is Failed to initialize NVML: Insufficient Permissions. This means the plugin container lacks the necessary device privileges. Ensure your daemonset has the required securityContext with privileged: true and mounts /dev and /var/run correctly.
• Check if the node is in a NotReady state—run kubectl get nodes. If a node is NotReady, it won’t be scheduled. Investigate kubelet logs on that node.
• Verify that the GPU resource is exposed as an extended resource—run kubectl describe node <node> and look for nvidia.com/gpu in the Capacity section. If it’s missing, the plugin hasn’t registered.
• Check if the pod is using a GPU resource that is not available—for example, requesting nvidia.com/gpu-memory when the plugin only exposes nvidia.com/gpu.
• Look at the scheduler logs—if you have access, check the kube-scheduler logs for detailed reasons why the pod cannot be placed.

Frequently Asked Questions

Why does my pod stay in Pending even though the node shows GPU capacity?

This usually means the GPU resource is already fully allocated to other pods, or the node has a taint that your pod doesn’t tolerate. Check the node’s allocated resources and taints.

Do I need to install the device plugin manually?

Yes, unless your Kubernetes distribution includes it. The NVIDIA device plugin is a daemonset that must be deployed to expose GPUs as schedulable resources.

Can I use GPUs without requesting them in the pod spec?

No, you must explicitly request nvidia.com/gpu in the pod’s resources. The device plugin only allocates GPUs to pods that ask for them.

What does the error ‘Insufficient nvidia.com/gpu’ mean?

It means no node has enough free GPUs to satisfy the pod’s request. Either add more GPU nodes, reduce the number of GPUs requested, or free up GPUs by removing other pods.

How do I know if my GPU driver is working?

SSH into the node and run nvidia-smi. If it shows GPU information, the driver is working. If it returns an error, the driver is not installed or not properly loaded.

What if I use AMD or Intel GPUs?

You need the respective device plugin (e.g., AMD GPU device plugin). The steps are similar, but the resource name and plugin deployment differ.

Related

• Fix Bluetooth Icon Missing in Windows 11
• Blank No Name Twitter: Meaning & Other Tips
• Install Ubuntu Touch on Nexus: Full Guide