☸️ Kubernetes Error Guide: 'Usage of EmptyDir volume exceeds the limit' Ephemeral Storage Eviction

Overview

Status:   Failed
Reason:   Evicted
Message:  Usage of EmptyDir volume "cache" exceeds the limit "512Mi".

or, at the node level:

Warning  Evicted  kubelet
  Pod ephemeral local storage usage exceeds the total limit of containers 1Gi.

Kubernetes treats ephemeral storage — the writable container layer, emptyDir volumes, and container logs — as a schedulable, limited resource. The kubelet continuously measures how much local disk a pod uses. When a pod exceeds its ephemeral-storage limit (or an emptyDir sizeLimit), the kubelet evicts the pod to protect the node. Separately, when the node itself runs low on disk, the kubelet enters DiskPressure and evicts pods by priority to reclaim space.

The eviction is deliberate resource enforcement, not a crash. The pod is killed, marked Evicted, and (if managed by a controller) rescheduled — often onto a node where it fills the disk again unless you fix the storage sizing or the runaway writes.

Symptoms

• Pods show Status: Evicted with a message naming ephemeral-storage or an EmptyDir volume ... exceeds the limit.
• kubectl get pods lists many Evicted pods accumulating over time.
• Nodes report the DiskPressure condition as True.
• Application logs, temp files, or emptyDir caches grow unbounded until eviction.
• Restarted pods fill the disk again within minutes or hours.

Common Root Causes

1. An emptyDir sizeLimit is too small for the workload

The pod writes more cache/temp data than the emptyDir sizeLimit allows, so the kubelet evicts it the moment usage crosses the threshold.

2. The pod’s ephemeral-storage limit is too low

resources.limits.ephemeral-storage is set below what the container legitimately needs for logs plus scratch space.

3. Runaway logging to the container filesystem

An app logs verbosely to stdout or to files inside the container; container logs count toward ephemeral storage and fill the node.

4. Node disk genuinely full (DiskPressure)

Image layers, dead containers, and orphaned volumes fill /var/lib/kubelet or /var/lib/containerd, triggering node-level eviction independent of any single pod’s limit.

5. No limits at all, so one pod starves the node

Without ephemeral-storage requests/limits, a single pod can consume the whole disk and take neighbors down with it.

Diagnostic Workflow

Step 1: Confirm the eviction reason

kubectl get pods -A --field-selector status.phase=Failed
kubectl describe pod <pod> -n <namespace> | grep -A3 -i "evicted\|ephemeral\|exceeds the limit"

Step 2: Check node disk pressure

kubectl describe node <node> | grep -A6 Conditions
kubectl describe node <node> | grep -i "DiskPressure\|ephemeral-storage"

Step 3: See what is consuming disk on the node

df -h /var/lib/kubelet /var/lib/containerd
sudo du -sh /var/lib/kubelet/pods/* 2>/dev/null | sort -h | tail

Step 4: Inspect image and container disk usage via crictl

sudo crictl imagefsinfo
sudo crictl images | wc -l

Step 5: Review the pod’s storage requests and limits

kubectl get pod <pod> -n <namespace> \
  -o jsonpath='{.spec.containers[*].resources}{"\n"}'

Step-by-Step Resolution

1. Clear out evicted pods so they stop cluttering the namespace (the underlying cause still needs fixing):

   kubectl get pods -A --field-selector status.phase=Failed -o name \
     | xargs -r kubectl delete

2. Right-size the ephemeral-storage request and limit to the container’s real usage plus headroom:

   resources:
     requests:
       ephemeral-storage: "1Gi"
     limits:
       ephemeral-storage: "2Gi"

3. Raise or remove the emptyDir sizeLimit if the cache legitimately needs more, or move it to memory-backed storage for speed:

   volumes:
     - name: cache
       emptyDir:
         sizeLimit: 2Gi

4. Tame runaway logs. Cap application log verbosity, rotate file logs, and rely on node-level log rotation. For huge datasets, mount a real PersistentVolume instead of writing to ephemeral storage.

5. Reclaim node disk if the node is under DiskPressure — prune unused images and dead containers:

   sudo crictl rmi --prune
   df -h /var/lib/containerd

6. Redeploy and verify the pod stays healthy and the node clears pressure:

   kubectl apply -f deployment.yaml
   kubectl describe node <node> | grep DiskPressure

Prevention

• Always set ephemeral-storage requests and limits so the scheduler accounts for disk and one pod cannot starve the node.
• Size emptyDir sizeLimit from measured usage; use a PersistentVolume for anything that grows unbounded.
• Enforce log rotation and sane verbosity; treat logs as ephemeral storage consumers.
• Monitor node disk and the DiskPressure condition, and tune kubelet eviction thresholds (--eviction-hard) for headroom.
• Schedule image garbage collection and prune dead containers so /var/lib/containerd does not creep toward full.

Related Errors

• The node was low on resource: ephemeral-storage — the node-level eviction message for the same class of problem.
• Pod ephemeral local storage usage exceeds the total limit of containers — the sum-of-containers limit rather than a single emptyDir.
• node.kubernetes.io/disk-pressure taint — applied when a node is under disk pressure, blocking new pods.
• Evicted: The node had condition: [MemoryPressure] — a memory-driven eviction, a different resource but the same eviction mechanism.

Frequently Asked Questions

What counts as ephemeral storage? The container’s writable layer, emptyDir volumes (not memory-backed), and container logs on the node. PersistentVolumes do not count. The kubelet sums these against the pod’s ephemeral-storage limit.

Why do evicted pods keep piling up? Eviction marks the pod Failed but does not delete it, so records accumulate. Delete them with a field selector, but fix the storage sizing or runaway writes or the replacements evict too.

How is a pod limit different from node DiskPressure? A pod limit evicts one offending pod that exceeds its own quota. DiskPressure is node-wide: when the disk runs low, the kubelet evicts pods by priority to reclaim space regardless of individual limits.

Should I use emptyDir or a PersistentVolume? Use emptyDir for small, transient scratch data with a sizeLimit. For large or growing data, mount a PersistentVolume so it never counts against ephemeral storage. Generate a right-sizing plan with the DevOps AI prompt library.

Where can I learn more? See the Kubernetes & Helm guides.