🚨 Kubernetes Error Guide: 'DiskPressure True' Node Condition and Eviction Taint

Exact Error Message

When a node’s disk drops below the kubelet’s eviction threshold, the node’s DiskPressure condition flips to True, a taint is applied, and pods are evicted:

$ kubectl describe node worker-3
Conditions:
  Type             Status  Reason                  Message
  ----             ------  ------                  -------
  DiskPressure     True    KubeletHasDiskPressure  kubelet has disk pressure
Taints:            node.kubernetes.io/disk-pressure:NoSchedule

Events:
  Type     Reason                 Age   From     Message
  ----     ------                 ----  ----     -------
  Warning  EvictionThresholdMet   2m    kubelet  Attempting to reclaim ephemeral-storage
  Warning  Evicted                118s  kubelet  The node was low on resource: ephemeral-storage.

The evicted pod shows the corresponding status:

$ kubectl get pod web-7d9 -o wide
NAME      READY   STATUS    RESTARTS   AGE   NODE
web-7d9   0/1     Evicted   0          3m    worker-3

What the Error Means

The kubelet continuously monitors two filesystems: nodefs (the root filesystem holding logs, emptyDir volumes, and pod ephemeral storage) and imagefs (where the container runtime stores images and writable layers). When free space or free inodes fall below the configured eviction thresholds — by default nodefs.available<10%, nodefs.inodesFree<5%, and imagefs.available<15% — the kubelet sets DiskPressure=True.

Two things then happen: the kubelet taints the node node.kubernetes.io/disk-pressure:NoSchedule so no new pods land there, and it begins reclaiming disk — first by garbage-collecting dead containers and unused images, then by evicting pods ranked by their ephemeral-storage usage relative to requests.

Common Causes

• Runaway container logs — a chatty app fills /var/log/containers (nodefs) faster than rotation can trim it.
• Image bloat on imagefs — many large images pulled over time exhaust the image filesystem before GC kicks in.
• emptyDir / ephemeral volumes — a pod writing scratch data to an emptyDir with no sizeLimit consumes nodefs.
• Inode exhaustion — millions of tiny files trip inodesFree even when bytes look fine.
• Undersized disks — small root volumes on dense nodes hit the threshold under normal load.
• Stuck/orphaned volumes — unreleased volumes or dead-container layers the runtime never reclaimed.

How to Reproduce the Error

On a test node, write a large file into a pod’s ephemeral storage until nodefs crosses the threshold:

apiVersion: v1
kind: Pod
metadata:
  name: disk-filler
spec:
  containers:
    - name: fill
      image: busybox
      command: ["sh", "-c", "dd if=/dev/zero of=/data/big bs=1M count=200000; sleep 3600"]
      volumeMounts:
        - { name: scratch, mountPath: /data }
  volumes:
    - name: scratch
      emptyDir: {}

kubectl apply -f disk-filler.yaml
kubectl describe node <NODE> | grep -A2 DiskPressure

As the emptyDir fills nodefs, the kubelet flips DiskPressure=True, taints the node, and evicts the filler pod with The node was low on resource: ephemeral-storage.

Diagnostic Commands

# Confirm the condition and the taint
kubectl describe node <NODE> | grep -A2 'DiskPressure\|Taints'

# Find evicted pods cluster-wide
kubectl get pods -A --field-selector=status.phase=Failed -o wide | grep -i evict

# See which filesystem is full and whether it is bytes or inodes (read-only)
kubectl debug node/<NODE> -it --image=busybox -- df -h
kubectl debug node/<NODE> -it --image=busybox -- df -i

# Kubelet eviction events
kubectl get events -A --field-selector reason=Evicted --sort-by=.lastTimestamp

# Inspect kubelet eviction thresholds in effect
kubectl get --raw "/api/v1/nodes/<NODE>/proxy/configz" | grep -o 'eviction[^,]*'

df -h vs df -i is the key distinction: a node can show plenty of free bytes yet still report DiskPressure because it ran out of inodes.

Step-by-Step Resolution

1. Identify nodefs vs imagefs and bytes vs inodes. Run df -h and df -i on the node. This tells you whether to clean logs/emptyDir (nodefs) or images (imagefs), and whether the constraint is space or inodes.

2. Reclaim image space. The kubelet runs image GC, but you can confirm what is consuming imagefs and prune unused images via the runtime. Reducing image count and size on imagefs is the fastest relief when imagefs is the culprit.

3. Cap log and ephemeral usage. Add a sizeLimit to emptyDir volumes and set ephemeral-storage requests/limits so a single pod cannot fill the node:

resources:
  limits:
    ephemeral-storage: "2Gi"
volumes:
  - name: scratch
    emptyDir: { sizeLimit: "1Gi" }

4. Wait for the taint to clear. Once free disk rises back above the threshold (eviction thresholds use a soft/hard margin), the kubelet clears DiskPressure and removes the node.kubernetes.io/disk-pressure taint automatically; new pods schedule again.

5. Grow the disk if undersized. If the node legitimately needs more storage, expand the root or imagefs volume, or move to larger nodes.

6. Re-run evicted workloads. Evicted pods are not rescheduled automatically when bare; controllers (Deployments/StatefulSets) recreate them. Delete leftover Evicted pod objects to clean up.

Prevention and Best Practices

• Set ephemeral-storage requests and limits on every workload so the scheduler accounts for disk and a noisy pod is capped, not the node.
• Always give emptyDir volumes a sizeLimit.
• Centralize and rotate logs off-node so container logs never accumulate on nodefs.
• Monitor both node_filesystem_avail_bytes and node_filesystem_files_free (inodes) and alert before the 10–15% thresholds.
• Tune image GC (imageGCHighThresholdPercent/LowThreshold) and keep base images lean to reduce imagefs churn.
• Size node disks for peak image + log + ephemeral footprint, not steady state. More in our Kubernetes & Helm guides.

Related Errors

• The node was low on resource: ephemeral-storage (Evicted) — the eviction this condition triggers.
• MemoryPressure True — the memory-side sibling condition.
• FailedScheduling — new pods refused by the disk-pressure taint.

Frequently Asked Questions

Why is DiskPressure True when df shows free space? Almost always inodes. The kubelet evicts on nodefs.inodesFree and imagefs.inodesFree too. Run df -i: a filesystem with millions of tiny files can be out of inodes while bytes look healthy.

Does the disk-pressure taint block all pods? It is a NoSchedule taint, so it prevents new pods from scheduling onto the node but does not by itself evict running pods — the kubelet’s eviction logic does that separately. Pods that tolerate the taint can still be placed, which is why critical DaemonSets keep running.

My pod was evicted but its disk usage was small — why it and not another? The kubelet ranks eviction candidates by usage above requests. A pod with no ephemeral-storage request that uses even a little is ranked above a pod using less than its request. Set requests so well-behaved pods are protected.

How long until the taint clears? As soon as reclaimed space pushes free disk back over the hard threshold (plus the eviction minimum-reclaim margin), the kubelet flips DiskPressure back to False and removes the taint, typically within a minute or two of GC/eviction freeing space.