☸️ Kubernetes Error Guide: 'failed to garbage collect required amount of images' Kubelet Disk Pressure

Overview

When a node’s image filesystem fills up, the kubelet tries to reclaim space by deleting unused images. If it cannot free enough, it logs:

E0716 11:47:03.882145   1 image_gc_manager.go:214] "Failed to garbage collect required amount of images.
Wanted to free 5368709120 bytes, but freed 1073741824 bytes"

and the node reports DiskPressure. The kubelet runs image garbage collection based on two thresholds: --image-gc-high-threshold (default 85%) and --image-gc-low-threshold (default 80%). When imagefs usage crosses the high threshold, the kubelet deletes unused images, oldest first, until it drops to the low threshold. This error means it tried but could not free the required amount, usually because most of the disk is consumed by running containers, logs, or ephemeral storage rather than by removable images.

Once the node is under DiskPressure it stops pulling new images, may evict pods, and taints itself so the scheduler avoids it. The goal is to reclaim real disk space and keep imagefs below the eviction thresholds.

Symptoms

• kubectl describe node shows DiskPressure=True and the image_gc_manager error in kubelet logs.
• Node carries the node.kubernetes.io/disk-pressure:NoSchedule taint.
• Pods on the node are Evicted with The node was low on resource: ephemeral-storage or imagefs.
• New pods stuck in ContainerCreating with Failed to pull image ... no space left on device.
• df -h on the node shows the container/image partition near 100%.

Common Root Causes

1. imagefs genuinely full of removable images

Many large or unused images accumulated (frequent deploys, big base images). GC can help here, but only if enough images are actually unused.

2. Space consumed by things GC cannot remove

Image GC only deletes unused images. If the disk is full of running-container writable layers, container logs, or emptyDir/ephemeral data, GC frees little and the error persists.

3. Undersized node disk

The node’s disk is simply too small for the images and workloads it runs, so it lives permanently near the threshold.

4. Log or ephemeral-storage growth

Runaway container logs (/var/log/pods) or pods writing large ephemeral files push imagefs over the threshold independent of images.

5. Container runtime not reclaiming layers

Orphaned snapshots/layers in containerd that were never cleaned up occupy space GC does not target.

Diagnostic Workflow

Step 1: Confirm DiskPressure and read the kubelet error

kubectl describe node <node-name> | grep -A2 DiskPressure
sudo journalctl -u kubelet --no-pager | grep -i "garbage collect" | tail

Step 2: See what is actually using the disk

df -h /var/lib/containerd /var/lib/kubelet
sudo du -xh /var/lib/containerd --max-depth=1 | sort -rh | head

Step 3: List images and their sizes via crictl

sudo crictl images
sudo crictl images --digests | wc -l

Step 4: Check container log and ephemeral usage

sudo du -sh /var/log/pods/* 2>/dev/null | sort -rh | head
sudo du -sh /var/lib/kubelet/pods/* 2>/dev/null | sort -rh | head

Step-by-Step Resolution

1. Identify whether images are even the problem. If du shows logs or ephemeral pod data dominating, image GC will never succeed — fix that source instead.

2. Prune unused images with crictl. This mirrors what the kubelet tries to do, and clears images not held by any container:

sudo crictl rmi --prune

3. Remove specific large unused images if prune is too conservative:

sudo crictl images
sudo crictl rmi <image-id>

4. Truncate or rotate oversized container logs (do not delete active log files out from under the kubelet; rotate them):

sudo du -sh /var/log/pods/* | sort -rh | head
sudo find /var/log/pods -name "*.log" -size +500M -exec ls -lh {} \;

5. Clean up stopped containers and orphaned sandboxes holding writable layers:

sudo crictl ps -a --state Exited
sudo crictl rm $(sudo crictl ps -a -q --state Exited)

6. Verify the node recovers. DiskPressure clears once imagefs drops below the low threshold, and the taint is removed automatically:

kubectl get nodes
kubectl describe node <node-name> | grep -A2 DiskPressure

7. If the disk is chronically full, grow it or lower thresholds. Resize the volume, or tune GC/eviction so reclamation starts sooner:

grep -E "imageGCHigh|imageGCLow|evictionHard" /var/lib/kubelet/config.yaml

Prevention

• Size node disks for peak image + ephemeral usage, not just steady state, and alert on imagefs percentage.
• Configure container log rotation (containerLogMaxSize, containerLogMaxFiles) so logs cannot fill imagefs.
• Set ephemeral-storage requests/limits on pods so a single workload cannot exhaust the node.
• Keep base images small and prune registries of stale tags to reduce what lands on nodes.
• Tune --image-gc-high-threshold/--image-gc-low-threshold to start reclamation before eviction triggers.

Related Errors

• The node was low on resource: ephemeral-storage — the eviction that follows unrelieved DiskPressure.
• Failed to pull image ... no space left on device — image pulls blocked by a full imagefs.
• attempting to reclaim ephemeral-storage in kubelet logs — the eviction manager acting on the same pressure.
• disk-pressure:NoSchedule taint keeping pods Pending — the scheduler avoiding the stressed node.

Frequently Asked Questions

What does failed to garbage collect required amount of images actually mean? The kubelet tried to delete unused images to get imagefs below its threshold but could not free enough space. Often the disk is full of running containers, logs, or ephemeral data that image GC cannot remove.

How do I free space immediately? Run sudo crictl rmi --prune to remove unused images, rotate oversized container logs, and remove exited containers. Then confirm DiskPressure clears with kubectl describe node.

Why does GC fail even though I have many images? Image GC only deletes images not referenced by any container. If most images are in use, or the space is consumed by logs/ephemeral storage, there is little for GC to reclaim. The DevOps AI prompt library has prompts for auditing node disk consumers.

How do I stop it recurring? Right-size node disks, enable container log rotation, set ephemeral-storage limits on pods, and keep images small. Alert on imagefs usage before it crosses the GC threshold.

Will the node recover on its own? It can, once enough space frees up and usage drops below the low threshold — the kubelet removes the DiskPressure taint automatically. But if the disk is chronically full you must reclaim space or grow the volume. For more, see the Kubernetes & Helm guides.