Fix Kubernetes Pods Stuck in Pending (CPU/Memory)

When a Kubernetes pod stays in Pending state, it usually means the scheduler cannot find a node that satisfies the pod’s resource requests. This guide walks you through diagnosing and fixing CPU or memory shortages, from checking events to adjusting requests and cluster capacity.

What You’ll Need

kubectl configured to access your cluster
Permissions to view pods, nodes, and events (typically cluster-admin or equivalent)
Permissions to edit deployments, statefulsets, or pods (if you plan to change resource requests)
Optional: access to the cluster’s autoscaler configuration if you use one

Step-by-Step

Identify the stuck pod. Run kubectl get pods and note the pod name and namespace. Look for pods with STATUS = Pending.
Check pod events. Run kubectl describe pod <pod-name> -n <namespace>. In the Events section, look for messages like 0/4 nodes are available: 1 Insufficient cpu, 3 Insufficient memory. This tells you exactly which resource is short.
Check node resource usage. Run kubectl top nodes to see current CPU and memory usage. Also run kubectl describe nodes to see the total allocatable resources and any taints or conditions that might block scheduling.
Review the pod’s resource requests. Run kubectl get pod <pod-name> -n <namespace> -o yaml and look under spec.containers[].resources.requests. Note the cpu and memory values. These are the amounts the scheduler must find free on a node.
Decide on a fix. Choose one or more of the following based on your situation:
- Reduce requests if the requests are set higher than the workload actually needs. Edit the deployment or pod spec and lower the requests.cpu and/or requests.memory.
- Add more nodes to the cluster if the total cluster capacity is insufficient. This could be manual or via a cluster autoscaler.
- Free up resources by deleting or scaling down other workloads that are using too much CPU or memory.
- Remove taints that prevent scheduling if the node has a taint that the pod doesn’t tolerate. Check kubectl describe nodes for taints, and if they are not needed, remove them with kubectl taint nodes <node-name> <key>-.
Apply the change. If you edited a deployment, run kubectl rollout restart deployment <deployment-name> or kubectl apply -f <file> to update. For a standalone pod, you may need to delete and recreate it.
Verify the fix. Run kubectl get pods again and check that the pod is no longer Pending. If it’s still pending, repeat the describe command to see new events.

Common Mistakes

Confusing requests and limits. Requests are what the scheduler uses for placement. Limits are the maximum allowed. Setting a high limit without a request does not cause pending, but setting a high request does.
Ignoring init containers. Init containers also have resource requests that count toward the pod total. Check them in the pod spec.
Forgetting about node taints. Even if resources are available, a taint without a matching toleration will keep the pod pending.
Looking only at current usage. The scheduler looks at requests, not actual usage. A node might show low CPU usage but still be unable to schedule a pod because requests already exceed capacity.
Not checking for node pressure conditions. MemoryPressure or DiskPressure conditions on a node can prevent scheduling even if there is nominally enough memory.

Tips

Use kubectl get events --sort-by='.lastTimestamp' to see cluster-wide events that might indicate resource issues.
Set resource requests realistically based on actual usage. Use kubectl top pods to see per-pod usage and adjust requests to be slightly above typical usage.
If you have a cluster autoscaler, make sure it’s configured to add nodes when pods are pending. Check the autoscaler logs if nodes are not being added.
For memory, remember that Kubernetes counts memory in bytes (e.g., 512Mi = 512 mebibytes). CPU is measured in cores or millicores (100m = 0.1 core).
Use kubectl describe node <node> to see the Allocatable resources, which are what the scheduler actually uses (after subtracting system-reserved and kube-reserved).

Troubleshooting

If the pod is still pending after following the steps above, try these additional checks:

Check for persistent volume issues. A pod can be pending if it needs a PVC that is not bound. Look for events like PersistentVolumeClaim is not bound. If so, fix the PVC first.
Check for node selectors and affinity. If the pod has a nodeSelector or affinity rules, the scheduler might not find a matching node. Review the pod spec for these constraints.
Check if the scheduler is running. In a control-plane failure, the scheduler may not be scheduling at all. Check the kube-scheduler pod status in the kube-system namespace.
Look at the scheduler logs. If you have access, check the logs of the scheduler pod for detailed reasons why scheduling failed.
Consider resource fragmentation. Even if a node has enough total resources, the scheduler might not be able to fit the pod due to how requests are distributed. For example, a node might have 1 CPU free, but the pod requests 2 CPU. This is normal; you may need to consolidate workloads or add nodes.

Frequently Asked Questions

What exactly does “Insufficient cpu” mean in pod events?

It means that on every node the scheduler checked, the sum of CPU requests from existing pods plus the new pod’s request exceeds the node’s allocatable CPU capacity. The node has no free CPU “slots” to accommodate the pod.

Can a pod be pending due to memory even if the node has free memory?

Yes. The scheduler uses requests, not actual usage. If the sum of memory requests on a node is already at or above the allocatable memory, a new pod with a memory request will not fit, even if actual usage is low.

How do I reduce resource requests without affecting performance?

Monitor your pod’s actual usage with kubectl top pods. Set requests to a value slightly above the average usage, but below the limit. For CPU, you can often use a value like 100m or 250m if your app is not CPU-intensive. For memory, look at the peak usage and add a buffer of 10-20%.

What is the difference between requests and limits in this context?

Requests are what the scheduler uses to place the pod. Limits are the maximum amount the container can use. A pod can be scheduled even if limits are high, as long as requests fit. But if limits are set too low, the container might be throttled or killed.

Should I set resource requests on all my pods?

Yes, it’s best practice. Without requests, the scheduler assumes the pod needs zero resources, which can lead to overcommitment and poor performance. Setting requests helps the scheduler make better decisions and prevents one pod from starving others.

Can I force a pod to schedule on a specific node to avoid pending?

You can use nodeSelector or nodeName in the pod spec, but this is generally discouraged because it reduces flexibility. It’s better to fix the resource issue or let the scheduler choose the best node. If the node has enough resources, forcing placement might work, but it won’t solve the underlying capacity problem.

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →