🚨 Kubernetes Error Guide: 'Evicted: The node was low on resource' Pod Eviction

Exact Error Message

NAME                      READY   STATUS    RESTARTS   AGE
log-shipper-6c8f9-2pq4r   0/1     Evicted   0          7m

# kubectl describe pod log-shipper-6c8f9-2pq4r
Status:   Failed
Reason:   Evicted
Message:  The node was low on resource: ephemeral-storage.
          Threshold quantity: 2Gi, available: 1487Mi.
          Container shipper was using 6Gi, request is 0,
          which exceeds its request of 0.

What the Error Means

Evicted with The node was low on resource is a node-pressure eviction. The kubelet continuously monitors node resources — memory, ephemeral (local) disk, inodes, and PID count. When usage crosses a configured eviction threshold, the kubelet reclaims the node by terminating pods, marking them Evicted with Reason: Failed. The evicted pod’s containers are stopped, and the pod object remains in Evicted status as a tombstone until cleaned up.

The kubelet ranks pods for eviction by their resource usage relative to their requests and by QoS class. Pods exceeding their requests (or with no requests at all, BestEffort) are evicted first. The message names the exact resource (ephemeral-storage, memory, inodes, or pid) and the threshold that was crossed.

Common Causes

• Ephemeral-storage pressure — logs, temp files, image layers, or emptyDir volumes filling the node’s disk past the eviction threshold.
• Memory pressure — total container memory usage on the node exceeding the kubelet’s memory eviction threshold (distinct from per-container OOMKilled).
• No resource requests set, making pods BestEffort and the first eviction targets.
• A noisy-neighbor pod writing large files or consuming memory and triggering eviction of better-behaved pods on the same node.
• Inode exhaustion from many small files even when raw disk space looks available.
• PID exhaustion from a runaway process spawning threads/processes.

How to Reproduce the Error

Run a pod with no requests that writes large files to ephemeral storage until the node crosses its threshold:

apiVersion: v1
kind: Pod
metadata:
  name: disk-hog
spec:
  containers:
    - name: hog
      image: busybox:1.36
      command: ["sh","-c","dd if=/dev/zero of=/tmp/fill bs=1M count=20000; sleep 3600"]

kubectl apply -f disk-hog.yaml
# After the node crosses its ephemeral-storage threshold:
kubectl get pods | grep Evicted

Diagnostic Commands

# Find evicted pods and the reason
kubectl get pods -A --field-selector status.phase=Failed
kubectl describe pod <POD> | grep -A3 -iE 'evicted|low on resource'

# Which resource and node? Check node conditions
kubectl get nodes
kubectl describe node <NODE> | grep -A8 -iE 'conditions|pressure'

# Node resource usage and which pods consume the most
kubectl top node <NODE>
kubectl top pods -A --sort-by=memory
kubectl top pods -A --sort-by=cpu

# Inspect kubelet eviction thresholds and node capacity
kubectl describe node <NODE> | grep -A6 -iE 'capacity|allocatable'

# On the node: actual disk and inode usage
df -h /var/lib/kubelet
df -i /var/lib/kubelet

# Recent eviction events on the node
kubectl get events -A --field-selector reason=Evicted --sort-by=.lastTimestamp

Step-by-Step Resolution

1. Identify the pressured resource and node. The describe message names it; kubectl describe node confirms which condition is True (MemoryPressure, DiskPressure, PIDPressure):

kubectl describe node <NODE> | grep -iE 'pressure'

2. Clean up the evicted pod tombstones. Evicted pods linger and clutter listings; remove them once you have read the cause:

kubectl get pods -A --field-selector status.phase=Failed -o name | \
  xargs -r kubectl delete

3. Reclaim the pressured resource. For disk pressure, prune unused images and clear oversized logs/emptyDir usage:

crictl rmi --prune
df -h /var/lib/kubelet

For memory pressure, reschedule or scale down the heavy consumers identified by kubectl top.

4. Set requests and limits so pods are protected. Pods with no requests are evicted first. Give workloads explicit requests/limits to raise their QoS class:

resources:
  requests: { cpu: "100m", memory: "256Mi", ephemeral-storage: "1Gi" }
  limits:   { cpu: "500m", memory: "512Mi", ephemeral-storage: "2Gi" }

With an ephemeral-storage limit, the kubelet evicts the offending container before it starves the node.

5. Constrain noisy workloads. Cap log volume, mount large scratch space on a dedicated PVC instead of node disk, and use sizeLimit on emptyDir:

volumes:
  - name: scratch
    emptyDir:
      sizeLimit: 1Gi

6. Scale the cluster if pressure is chronic. If nodes are routinely pressured, add nodes or move to larger instances rather than fighting evictions.

Prevention and Best Practices

• Always set memory and ephemeral-storage requests and limits so the kubelet can rank and bound pods correctly.
• Cap emptyDir with sizeLimit and route bulk/scratch data to dedicated volumes, not node-local disk.
• Configure log rotation and ship logs off-node so application logs do not fill ephemeral storage.
• Monitor kubelet eviction thresholds and node disk/memory/inode usage with alerts before pressure builds.
• Use Guaranteed QoS (requests == limits) for critical pods so they are the last to be evicted.
• Right-size nodes and enable cluster autoscaling so capacity grows with demand. See more in Kubernetes & Helm guides.

Related Errors

• OOMKilled — a single container exceeded its own memory limit (cgroup kill), different from node-level memory eviction.
• FailedScheduling: Insufficient cpu/memory — there is no node with enough room to place the pod.
• DiskPressure / MemoryPressure node conditions — the underlying signals that drive evictions.
• NodeNotReady — sustained pressure can ultimately render a node unready.

Frequently Asked Questions

What is the difference between Evicted and OOMKilled? OOMKilled is a per-container kill: the container exceeded its own memory limit and the kernel killed that process (exit 137). Evicted is a node-level action: the kubelet proactively terminates whole pods to relieve node pressure, even pods that are within their own limits.

Why do evicted pods stay in the list forever? Evicted pods are kept as tombstones so you can inspect the reason. They are not automatically garbage-collected by default. Delete them once reviewed; consider a controller or cron to prune status.phase=Failed pods.

Why was my pod evicted when it was barely using memory? Eviction ranks by usage relative to requests and by QoS. A BestEffort pod with no requests is evicted before a Guaranteed pod, even if the BestEffort pod uses less in absolute terms. Set requests to protect it.

Can I stop ephemeral-storage evictions entirely? Not entirely, but you can prevent them in practice: set ephemeral-storage limits on pods, cap emptyDir sizes, rotate logs, and keep enough headroom on node disks so the threshold is rarely approached.