💾 Disk Pressure, Image GC, and Why the Kubelet Evicted Your Pods

Memory pressure gets all the attention, but in my experience the more common 3 a.m. page is disk pressure. A node fills up — usually from container logs, image layers, or an emptyDir someone forgot was unbounded — and the kubelet starts evicting pods to save itself. Suddenly healthy workloads are Evicted, the scheduler refuses to place new pods on that node, and nobody changed a single line of application code. The kubelet was protecting the node exactly as designed; we just hadn’t tuned the thresholds or understood image garbage collection.

This guide explains how the kubelet decides a node is under disk pressure, how image GC works, and how to keep nodes healthy without surprise evictions.

How the kubelet measures disk pressure

The kubelet watches two filesystems: nodefs (where emptyDir volumes, container logs, and the kubelet’s own state live) and imagefs (where the container runtime stores image layers and writable container layers). When the available space or inodes on either drops below a configured threshold, the kubelet sets the DiskPressure node condition and begins acting.

You can see the current state directly:

kubectl describe node <node> | grep -A5 Conditions
# Look for: DiskPressure   True   ...
kubectl get nodes -o json | jq '.items[].status.conditions[] |
  select(.type=="DiskPressure")'

The thresholds come in two flavors. Soft eviction thresholds trigger after a grace period and let pods terminate gracefully. Hard eviction thresholds trigger immediately with no grace period — the kubelet kills pods now to save the node. They’re set in the kubelet config:

# kubelet config (KubeletConfiguration)
evictionHard:
  nodefs.available: "10%"
  imagefs.available: "15%"
  nodefs.inodesFree: "5%"
evictionSoft:
  nodefs.available: "15%"
  imagefs.available: "20%"
evictionSoftGracePeriod:
  nodefs.available: "2m"
  imagefs.available: "2m"
evictionMaxPodGracePeriod: 60

When disk pressure hits, the kubelet ranks pods for eviction by their disk usage relative to requests and their QoS class — BestEffort and Burstable pods that exceed their (often unset) ephemeral-storage requests go first.

Image garbage collection runs first

Before the kubelet starts evicting pods, it tries to reclaim space by deleting unused images. Image GC is governed by two percentages on imagefs:

imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80

When imagefs usage climbs above the high threshold (85%), the kubelet deletes unused images — oldest and least-recently-used first — until usage falls back below the low threshold (80%). Images currently used by running containers are never collected. This is the cluster quietly cleaning up after itself, and most of the time it works invisibly. You only notice it when it can’t keep up — for instance, when a CI node pulls dozens of large images faster than GC reclaims them, or when the writable container layers (not images) are what’s filling the disk.

Pro Tip: Image GC only reclaims images, not the writable layers of running or exited containers, and not container logs. If your disk is filling from a chatty container writing gigabytes to stdout, no amount of image GC saves you — that’s nodefs, governed by log rotation (containerLogMaxSize, containerLogMaxFiles) and your runtime’s log config, not image cleanup. Diagnose which filesystem is full before you assume GC is the answer.

Set ephemeral-storage requests and limits

The single most effective defense against disk-pressure evictions is treating ephemeral storage like the finite resource it is. Pods can declare it just like CPU and memory:

spec:
  containers:
    - name: worker
      image: registry.internal/worker:1.4.2
      resources:
        requests:
          ephemeral-storage: "1Gi"
        limits:
          ephemeral-storage: "2Gi"

The requests value lets the scheduler account for disk when placing the pod, so it won’t cram ten storage-hungry pods onto one node. The limits value lets the kubelet evict only the offending pod when it exceeds 2Gi — instead of the node hitting global pressure and evicting innocent neighbors. A pod that blows its ephemeral-storage limit gets evicted with a clear reason, which is a vastly better failure mode than a node-wide pressure event.

Diagnosing a node that’s already full

When you’re paged, the question is always “what’s eating the disk?” Walk the candidates:

# Overall node filesystem usage
kubectl debug node/<node> -it --image=busybox -- df -h

# Biggest consumers under the kubelet/runtime roots
du -xh /var/lib/containerd /var/log/pods 2>/dev/null | sort -rh | head

# Container logs specifically
du -sh /var/log/pods/* | sort -rh | head

Usually it’s one of three things: runaway container logs, an emptyDir that’s grown unbounded, or accumulated image layers on a node that pulls a lot. Each has a different fix — log rotation, an emptyDir.sizeLimit, or more aggressive image GC respectively — so identifying the culprit is the whole job.

Where an AI assistant speeds this up

This is mostly a diagnosis-and-tuning problem, and that’s a good fit for an AI assistant working from text. I’ll paste the output of kubectl describe node, the du results, and the current kubelet config, then ask the model to identify which filesystem is under pressure, name the likely culprit, and propose threshold or log-rotation changes. It’s quick at correlating “imagefs at 92%, image GC high threshold at 85%, but pull rate exceeds GC rate” into a clear recommendation.

The model is a fast junior engineer here: it reads telemetry and drafts config, but it does not drive the cluster. Kubelet config and eviction thresholds change node-level behavior — get them wrong and you can make a node evict everything or never protect itself — so this stays human-in-the-loop. I apply changes through my config-management pipeline, not by letting a model near a node. And the assistant never receives a kubeconfig, SSH access, or any production credential; it gets command output I’ve already captured. For walking through an active incident there’s a structured flow in the incident response dashboard, continuous signals in the monitoring alerts dashboard, and reusable starting points in the prompt library.

A diagnostic prompt I lean on:

Here is `kubectl describe node`, `df -h`, `du` of /var/log/pods and
/var/lib/containerd, and the kubelet evictionHard/imageGC config. Tell me:
which filesystem is pressured, the most likely root cause, and a concrete
config or log-rotation change to prevent recurrence. Analysis only — no
commands run against the cluster.

Preventing the next page

Tuning thresholds treats symptoms; prevention is about caps. Set ephemeral-storage requests and limits on every workload, put sizeLimit on every emptyDir, configure container log rotation cluster-wide, and right-size your nodes’ disks for the image churn they actually see (CI and build nodes need far more imagefs than steady-state app nodes). A little headroom and a few limits turn disk pressure from a recurring incident into a non-event.

Wrapping up

The kubelet evicts pods to keep nodes alive, and disk pressure is the trigger people forget to plan for. Understand that image GC reclaims images but not logs or writable layers, set ephemeral-storage limits so a greedy pod hurts only itself, and diagnose which filesystem is full before you act. Let an AI assistant correlate telemetry and draft config while you keep your hands on the nodes and your credentials to yourself. More node-level operations guidance lives in the Kubernetes & Helm guides.