Schedule GPU Workloads on Kubernetes: Resource Limits

This guide shows you how to schedule GPU workloads on Kubernetes by defining resource requests and limits, and by using ResourceQuotas to control GPU usage across namespaces. You’ll learn how to configure your pods, set quotas, and avoid common pitfalls that lead to failed scheduling or wasted GPU capacity.

What You’ll Need

A Kubernetes cluster with GPU nodes (NVIDIA, AMD, or other) and the corresponding device plugin installed (e.g., NVIDIA device plugin for NVIDIA GPUs).
kubectl configured to access your cluster.
Cluster-admin or namespace-admin permissions to create ResourceQuotas and LimitRanges.
Basic understanding of Kubernetes pods, deployments, and namespaces.

Step-by-Step

Verify GPU availability on nodes: Run kubectl get nodes -o json | jq '.items[].status.allocatable' and look for nvidia.com/gpu (or your vendor’s resource name). If absent, install the device plugin for your GPU vendor.
Create a dedicated namespace (optional but recommended): Run kubectl create namespace gpu-workloads to isolate GPU workloads and apply quotas cleanly.

Define a ResourceQuota for GPU resources: Create a YAML file (e.g., gpu-quota.yaml) with the following content:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota
  namespace: gpu-workloads
spec:
  hard:
    requests.nvidia.com/gpu: 4
    limits.nvidia.com/gpu: 4
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi

Adjust the numbers to your cluster capacity. Apply it with kubectl apply -f gpu-quota.yaml.

Set a LimitRange for default GPU requests (optional): Create gpu-limitrange.yaml:
```
apiVersion: v1
kind: LimitRange
metadata:
  name: gpu-limitrange
  namespace: gpu-workloads
spec:
  limits:
  - default:
      nvidia.com/gpu: 1
    defaultRequest:
      nvidia.com/gpu: 1
    type: Container
```
This ensures every container in the namespace gets a default GPU request if not specified. Apply it with kubectl apply -f gpu-limitrange.yaml.
Write a pod spec with GPU requests and limits: In your deployment or pod YAML, under spec.containers.resources, specify both requests and limits for the GPU resource. For example:
```
resources:
  requests:
    nvidia.com/gpu: 1
    cpu: "2"
    memory: 4Gi
  limits:
    nvidia.com/gpu: 1
    cpu: "4"
    memory: 8Gi
```
Using the same value for request and limit is common for GPUs to avoid overcommitting.
Deploy your workload: Use kubectl apply -f your-deployment.yaml. The scheduler will place the pod on a node with available GPU capacity that also satisfies CPU/memory requests.
Verify scheduling: Run kubectl get pods -n gpu-workloads -o wide to see which node the pod landed on. Check kubectl describe pod <pod-name> -n gpu-workloads for events like Successfully assigned.
Monitor GPU usage: Use kubectl exec -it <pod> -- nvidia-smi (if using NVIDIA) to confirm the GPU is visible and allocated.

Common Mistakes

Setting only limits, not requests: If you set only limits, Kubernetes may not reserve the GPU for your pod, leading to overcommit and potential OOM or scheduling issues. Always set both.
Using the wrong resource name: For NVIDIA GPUs, the resource name is nvidia.com/gpu. For AMD, it’s amd.com/gpu. Using the wrong name results in the pod never being scheduled.
Forgetting the device plugin: Without the device plugin, the GPU resource is not advertised on nodes, and the scheduler will not see any GPU capacity.
Setting quota limits too low: If your quota is lower than the sum of requests from your pods, new pods will be rejected with a QuotaExceeded error. Double-check your quota against your actual needs.
Not accounting for whole-number GPU allocation: GPUs are allocated in whole numbers. You cannot request 0.5 GPUs. If you need fractional GPUs, consider using time-slicing or MIG (Multi-Instance GPU) if your hardware supports it.

Tips

Use node selectors or taints/tolerations: To ensure GPU workloads land on GPU nodes, add nodeSelector: {gpu: "true"} or apply a taint to GPU nodes and a toleration in your pod spec. This prevents non-GPU workloads from occupying GPU nodes.
Set resource quotas per namespace: This prevents a single team from consuming all GPU resources. Use ResourceQuota to cap total GPU requests and limits.
Consider using PriorityClasses: Assign higher priority to critical GPU jobs so they preempt lower-priority ones when resources are scarce.
Monitor with metrics: Use Prometheus and Grafana to track GPU utilization and identify idle GPUs. This helps in right-sizing requests.
Use the resources.requests field for autoscaling: If using HPA, base scaling on GPU utilization or custom metrics, not just CPU.

Troubleshooting

Pod stuck in Pending: Run kubectl describe pod <pod>. If you see 0/1 nodes available: 1 Insufficient nvidia.com/gpu, it means no node has enough free GPUs. Check node capacity with kubectl describe node and free up GPUs or reduce requests.
QuotaExceeded error: Check the ResourceQuota status: kubectl get resourcequota -n <namespace> -o yaml. See which resource is over the limit and either increase the quota or reduce pod requests.
GPU not visible inside container: Ensure the device plugin is running and the pod has the right resource request. Also check that the container has the necessary drivers installed (e.g., CUDA libraries) for the GPU to be usable.
Node with GPU but pod not scheduled there: The scheduler might be avoiding the node due to CPU/memory constraints or taints. Check node conditions and taints with kubectl describe node.
ResourceQuota not taking effect: Ensure the quota is applied to the same namespace as the pod. Verify with kubectl get quota -n <namespace>.

Frequently Asked Questions

Can I request a fraction of a GPU? No, standard Kubernetes GPU resources are allocated as whole numbers. For fractional GPUs, you need features like NVIDIA MIG or time-slicing, which require additional setup.
What happens if I set a limit higher than the request? Kubernetes will try to schedule the pod based on the request, but the limit may cause the pod to be evicted if the node cannot provide the limit. For GPUs, it’s best to set them equal.
Do I need to specify GPU requests and limits for every container in a pod? If you have multiple containers, each that needs a GPU must specify its own request/limit. The total for the pod is the sum across containers.
How do I check how many GPUs are available in my cluster? Use kubectl get nodes -o json | jq '.items[].status.allocatable' and look for the GPU resource name. Alternatively, kubectl describe node shows allocatable resources.
Can I use a ResourceQuota to limit GPU usage per team? Yes, create a ResourceQuota in each team’s namespace and set the GPU request/limit caps. This is a common practice for multi-tenant clusters.
What if my GPU workload needs more memory than the node has? The scheduler will not place the pod if the node lacks sufficient memory. You may need to use a different node or reduce memory requests. GPU nodes often have limited memory, so plan accordingly.

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →