🚨 Kubernetes Error Guide: Pod Status 'SchedulingGated' — Fix Pods That Never Schedule

Overview

A pod shows SchedulingGated and never moves to Pending or Running. This is not a failure — it is a deliberate hold. The pod carries one or more spec.schedulingGates, and the scheduler is designed to completely ignore any pod that still has a gate. Until every gate is removed (by whatever controller added it), the scheduler will not even consider placing the pod.

You will see it in the status column:

NAME                     READY   STATUS            RESTARTS   AGE
batch-worker-0           0/1     SchedulingGated   0          6m

Scheduling gates were added so an external controller can hold a pod before scheduling — for example to wait for a quota grant, a network to be provisioned, or a custom admission workflow. The gate is a named marker in spec.schedulingGates. If the controller that was supposed to clear it never runs (or was uninstalled), the pod sits in SchedulingGated forever. Unlike Pending, the scheduler emits no FailedScheduling events — because it is not trying to schedule the pod at all.

Symptoms

• Pod status is SchedulingGated, not Pending.
• No FailedScheduling events and no scheduler activity for the pod.
• kubectl get pod -o yaml shows a non-empty spec.schedulingGates list.
• The pod never gets a nodeName assigned; it simply waits.

kubectl get pod batch-worker-0

NAME             READY   STATUS            RESTARTS   AGE
batch-worker-0   0/1     SchedulingGated   0          6m

kubectl get pod batch-worker-0 -o jsonpath='{.spec.schedulingGates}'

[{"name":"quota.example.com/pending-approval"}]

Common Root Causes

1. A gate controller that never cleared the gate

The most common cause: a controller (quota manager, capacity reservation, custom scheduler front-end) adds a gate and is supposed to remove it once a condition is met, but the controller is not running, crashed, or was uninstalled.

kubectl get pod batch-worker-0 -o jsonpath='{.spec.schedulingGates[*].name}'

quota.example.com/pending-approval

Check whether the controller that owns that gate name is deployed and healthy. If it is gone, the gate will never be cleared automatically.

2. A gate added by a mutating webhook / operator you forgot about

An admission webhook or operator injected a gate on pod creation to enforce an ordering guarantee (e.g. wait for a dependency). If the reconcile loop that removes it is broken, pods pile up gated.

kubectl get mutatingwebhookconfigurations
kubectl get pod batch-worker-0 -o yaml | grep -A3 schedulingGates

3. A gate you set manually and never removed

Gates can be set directly in the pod spec (for testing a hold-then-release workflow). If you added schedulingGates in the manifest and no process removes it, the pod stays gated.

4. The clearing controller lacks RBAC to patch pods

The controller is running but cannot remove the gate because its ServiceAccount lacks patch on pods. It logs permission errors while pods stay gated.

kubectl -n gatekeeper-system logs deploy/quota-controller | grep -i forbidden

pods "batch-worker-0" is forbidden: User "system:serviceaccount:..." cannot patch resource "pods"

Diagnostic Workflow

Step 1: Confirm it is gated, not pending

kubectl get pod <POD> -o wide

SchedulingGated (not Pending) tells you the scheduler is intentionally skipping it.

Step 2: List the gates

kubectl get pod <POD> -o jsonpath='{.spec.schedulingGates[*].name}'

The gate name(s) identify which controller is responsible for clearing them.

Step 3: Find and check the owning controller

kubectl get pods -A | grep -i <controller>
kubectl logs deploy/<controller> -n <ns> --tail=50

Verify the controller is running and reconciling. Look for crashes, RBAC forbidden errors, or a stuck condition.

Step 4: Verify why the gate is not being cleared

kubectl describe pod <POD>
kubectl get events -n <ns> --sort-by='.lastTimestamp' | grep <POD>

The gate stays until its condition (quota approved, dependency ready) is satisfied. Confirm that condition.

Step 5: Clear the gate if it is orphaned

If the owning controller is gone and the gate is safe to remove, patch it out. Gates can only be removed, never added, after creation, and removing all of them releases the pod to the scheduler.

kubectl patch pod <POD> --type merge -p '{"spec":{"schedulingGates":[]}}'

Example Root Cause Analysis

A batch platform uses a home-grown quota controller that gates pods until capacity is reserved. After a cluster upgrade, new jobs stall:

kubectl get pods -l app=batch-worker

NAME             READY   STATUS            RESTARTS   AGE
batch-worker-0   0/1     SchedulingGated   0          12m
batch-worker-1   0/1     SchedulingGated   0          12m

The gate points at the quota controller:

kubectl get pod batch-worker-0 -o jsonpath='{.spec.schedulingGates[*].name}'

quota.example.com/pending-approval

Checking the controller reveals it never restarted after the upgrade:

kubectl get deploy quota-controller -n batch-system

NAME               READY   UP-TO-DATE   AVAILABLE   AGE
quota-controller   0/1     1            0           12m

The controller pod is CrashLoopBackOff due to a removed API version, so it never clears the gate. The real fix is to repair the controller. As an immediate unblock for a known-safe workload, the gate can be removed manually:

kubectl patch pod batch-worker-0 --type merge -p '{"spec":{"schedulingGates":[]}}'

The pod immediately moves to Pending, then Running once the scheduler places it.

Prevention Best Practices

• Monitor the health of any controller that owns scheduling gates; if it goes down, gated pods accumulate silently with no scheduler events to alert on.
• Alert on pods in SchedulingGated beyond a threshold age — because there are no FailedScheduling events, standard scheduling alerts miss them.
• Give gate-clearing controllers the exact RBAC they need (patch on pods) and test it, since a permission gap looks identical to a stuck condition.
• Document every gate name your platform uses so on-call can immediately map a gate to its owning controller.
• Avoid leaving manually-set gates in manifests destined for production. See more in Kubernetes & Helm guides.

Quick Command Reference

# Confirm gated vs pending
kubectl get pod <POD> -o wide

# List the gate names
kubectl get pod <POD> -o jsonpath='{.spec.schedulingGates[*].name}'

# Inspect the owning controller
kubectl get pods -A | grep <controller>
kubectl logs deploy/<controller> -n <ns> --tail=50

# Check for RBAC failures clearing the gate
kubectl logs deploy/<controller> -n <ns> | grep -i forbidden

# Manually release an orphaned gate (only if safe)
kubectl patch pod <POD> --type merge -p '{"spec":{"schedulingGates":[]}}'

Conclusion

SchedulingGated means the scheduler is intentionally ignoring the pod because it still carries one or more spec.schedulingGates. The usual causes:

1. The controller responsible for clearing the gate is not running or crashed.
2. A webhook/operator injected a gate whose reconcile loop is broken.
3. A manually-set gate was never removed.
4. The clearing controller lacks RBAC patch on pods.

List the gate name, find its owning controller, and fix that controller — or, for a known-safe orphaned gate, remove it with a merge patch. For ad-hoc triage, the free incident assistant can map a gated pod to the controller that should have released it.