🚨 GCP Error Guide: 'The node was low on resource: ephemeral-storage' — Fix GKE DiskPressure Evictions

Overview

The GKE kubelet evicts pods when a node’s ephemeral storage crosses its eviction threshold, marking the node DiskPressure. The evicted pod’s status shows the literal message:

The node was low on resource: ephemeral-storage. Threshold quantity: 10120387530, available: 9843210240.

You will also see the node condition and the kubelet’s reclaim attempt:

Node condition DiskPressure is now: True
attempting to reclaim ephemeral-storage

Ephemeral storage is the node’s boot disk shared by the container runtime images, writable container layers, emptyDir volumes, and — most often — container logs. When it fills, the kubelet evicts pods to protect the node, which looks like random pod restarts until you connect them to DiskPressure.

Symptoms

• Pods in Evicted status with the ephemeral-storage low-resource message.
• Nodes flapping to DiskPressure=True under load, then recovering after eviction.
• Image pulls failing with no space left on device on the node.
• Pods rescheduled repeatedly, sometimes onto another node that then also fills.
• kubectl describe node shows DiskPressure and low ephemeral-storage capacity.

Common Root Causes

• Unbounded container logs — a chatty container writing gigabytes of stdout that fill the node’s log partition.
• Large emptyDir volumes — pods writing temp data to emptyDir, which is backed by the node’s ephemeral storage.
• Big writable container layers — apps writing into the container filesystem instead of a mounted volume.
• Image accumulation — many large images pulled onto a node without garbage collection keeping up.
• Boot disk too small — default small node boot disks on nodes running storage-heavy workloads.
• No ephemeral-storage requests/limits — pods with no ephemeral-storage accounting, so the scheduler over-packs the node.

Diagnostic Workflow

Confirm the eviction reason and which node is under pressure:

kubectl get events --field-selector reason=Evicted -A
kubectl describe node NODE | grep -A3 -i 'DiskPressure\|ephemeral'

See how much ephemeral storage the node actually has and how full it is (SSH via the node’s underlying VM or use a debug pod):

kubectl get node NODE -o jsonpath='{.status.capacity.ephemeral-storage}{"\n"}'
kubectl get node NODE -o jsonpath='{.status.allocatable.ephemeral-storage}{"\n"}'

Find what’s consuming disk on the node — logs, emptyDir, and image cache are the usual suspects:

# Run a privileged debug pod / node shell, then:
sudo du -sh /var/log/pods/* 2>/dev/null | sort -rh | head
sudo du -sh /var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/* 2>/dev/null | sort -rh | head

Check the node pool’s boot disk size to see if it’s simply undersized:

gcloud container node-pools describe POOL --cluster=CLUSTER --region=REGION \
  --format='value(config.diskSizeGb, config.diskType)'

Example Root Cause Analysis

A GKE workload started seeing pods Evicted with node was low on resource: ephemeral-storage every few hours, always on the busiest nodes.

Diagnosis: kubectl describe node confirmed DiskPressure=True. A node shell showed /var/log/pods at over 30 GB on a 100 GB boot disk, dominated by one deployment logging verbose debug output to stdout at high volume. The pods had no ephemeral-storage requests, so the scheduler packed several replicas of the chatty workload onto the same node, compounding the log growth.

Root cause: unbounded stdout logging filled the node’s ephemeral storage, and the absence of ephemeral-storage requests let the scheduler over-pack the noisy workload onto single nodes.

Fix: reduce the application log verbosity and set container log rotation limits, add ephemeral-storage requests/limits to the pods so the scheduler accounts for their disk footprint and spreads them, and enlarge the node pool boot disk for headroom. DiskPressure evictions stopped.

Prevention Best Practices

• Set ephemeral-storage requests and limits on pods so the scheduler accounts for disk and doesn’t over-pack nodes.
• Cap and rotate container logs; keep application log verbosity sane in production and ship logs to Cloud Logging rather than hoarding on-node.
• Use mounted volumes (PVCs) for large temp data instead of emptyDir, which consumes node ephemeral storage.
• Size node boot disks for storage-heavy workloads; the default may be too small for image-plus-log-plus-emptyDir demand.
• Let image garbage collection do its job and avoid pulling many huge images onto the same nodes.
• Alert on node ephemeral-storage utilization and DiskPressure conditions before eviction, not after.

Quick Command Reference

# Find evicted pods and the reason
kubectl get events --field-selector reason=Evicted -A

# Check node disk-pressure condition and ephemeral capacity
kubectl describe node NODE | grep -A3 -i 'DiskPressure\|ephemeral'

# Show node pool boot disk size
gcloud container node-pools describe POOL --cluster=CLUSTER --region=REGION \
  --format='value(config.diskSizeGb, config.diskType)'

# Inspect on-node disk usage (from a node shell/debug pod)
sudo du -sh /var/log/pods/* | sort -rh | head

Conclusion

The node was low on resource: ephemeral-storage is a GKE DiskPressure eviction: the node’s boot disk — shared by logs, emptyDir, writable layers, and images — filled up and the kubelet evicted pods to protect the node. The usual culprit is unbounded container logs, amplified by pods that carry no ephemeral-storage requests so the scheduler over-packs them. Fix it by capping logs, setting ephemeral-storage requests/limits, moving big temp data to PVCs, and sizing node boot disks with headroom — then alert on disk utilization so you act before the evictions, not after.