☸️ Kubernetes Error Guide: 'invalid capacity 0 on image filesystem' Kubelet Log

Overview

invalid capacity 0 on image filesystem is a kubelet log message that means the kubelet could not read a valid size for the image filesystem (imagefs) from cAdvisor and the container runtime’s stats. Because the reported capacity is zero, the kubelet cannot compute disk usage, which breaks image garbage collection and disk-pressure eviction thresholds.

The message usually appears repeatedly in the kubelet journal on nodes running containerd or CRI-O when the CRI stats provider is not returning filesystem info yet, or is misconfigured. It is frequently benign and transient right after the kubelet or runtime starts, but if it persists it signals a real integration problem between the kubelet, cAdvisor, and the runtime’s overlay/snapshotter storage.

Left unresolved, a node with a broken imagefs reading will never garbage-collect unused images and may fail to react to disk pressure, eventually filling the disk.

Symptoms

kubelet[1123]: W0716 10:22:41.334  eviction_manager.go:  invalid capacity 0 on image filesystem
kubelet[1123]: I0716 10:22:41.334  container_manager_linux.go:  Failed to get imagefs stats

You may also see garbage collection warnings such as failed to garbage collect required amount of images and ImageGCFailed events. kubectl describe node may show ImageGCFailed under events, and disk usage climbs without cleanup even though old images exist.

Common Root Causes

1. CRI stats provider not yet ready after start

Immediately after the kubelet or containerd restarts, cAdvisor may query the imagefs before the runtime has populated filesystem stats, yielding a momentary capacity of zero.

2. Containerd snapshotter or root path misconfigured

If containerd’s root directory or snapshotter (overlayfs) is misconfigured, or points at a filesystem cAdvisor cannot stat, the runtime returns no usable filesystem info.

3. Kubelet and runtime disagree on the stats source

With --container-runtime-endpoint set but the CRI stats path wrong, or when using an outdated cAdvisor stats provider against a CRI-only runtime, the imagefs lookup fails.

4. Overlay/graph directory on an unsupported or separate mount

If the image store lives on a filesystem type cAdvisor does not recognize, or on a bind mount without proper stats, capacity reads back as zero.

Diagnostic Workflow

Step 1: Confirm the message frequency

On the node:

journalctl -u kubelet --since "10 min ago" | grep -i "invalid capacity"

A handful of lines right after a restart is transient. A steady stream every few seconds is a real fault.

Step 2: Check the runtime and its storage root

crictl info | grep -iE "root|snapshotter"
containerd config dump | grep -iE "root|snapshotter"

Confirm the root path exists and the snapshotter is overlayfs.

Step 3: Ask the CRI for filesystem stats directly

crictl imagefsinfo

If this returns empty or an error, the runtime itself is not reporting imagefs, which is the true source.

Step 4: Verify the underlying filesystem

df -h /var/lib/containerd
stat -f /var/lib/containerd

Confirm the image store is on a real, mounted, supported filesystem.

Step-by-Step Resolution

1. If the messages are only at startup and stop within a minute, treat them as transient and move on. Confirm image GC is actually working:

   kubectl describe node <node> | grep -i imagegc

2. If crictl imagefsinfo returns nothing, fix the runtime. Ensure containerd’s config points its root at a valid path and uses the overlayfs snapshotter:

   sudo vi /etc/containerd/config.toml
   # confirm: root = "/var/lib/containerd"
   # confirm: [plugins."io.containerd.grpc.v1.cri".containerd] snapshotter = "overlayfs"

3. Restart the runtime, then the kubelet, so stats are re-collected:

   sudo systemctl restart containerd
   sudo systemctl restart kubelet

4. Verify the kubelet’s runtime endpoint matches the running runtime:

   sudo cat /var/lib/kubelet/kubeadm-flags.env | grep container-runtime-endpoint

   It should point at unix:///run/containerd/containerd.sock.

5. Confirm imagefs now reports a nonzero capacity:

   crictl imagefsinfo

   {"status":{"timestamp":..., "fsId":{"mountpoint":"/var/lib/containerd/..."},
    "usedBytes":{"value":"5123456789"}, "capacityBytes":{"value":"107374182400"}}}

6. Watch the kubelet log to confirm the warning has cleared:

   journalctl -u kubelet -f | grep -i capacity

Prevention

• Keep the container runtime and kubelet versions aligned; mismatches are a leading cause of stats-provider gaps.
• Provision the image store (/var/lib/containerd) on a standard, supported filesystem, not an exotic or unmounted path.
• Order service startup so containerd is fully up before the kubelet queries it; use After=containerd.service in the kubelet unit if you manage it manually.
• Monitor ImageGCFailed node events in your alerting so a persistent zero-capacity read is caught before the disk fills.
• Validate node images in a canary pool after any runtime upgrade. The DevOps AI prompt library includes prompts for auditing kubelet and containerd configuration.

Related Errors

• ImageGCFailed: failed to garbage collect required amount of images — the downstream effect of a broken imagefs reading.
• Failed to get imagefs stats — the companion kubelet line describing the same lookup failure.
• no space left on device — what eventually happens if GC never runs because capacity reads zero.
• failed to get rootfs info — a related stats failure for the node root filesystem rather than imagefs.

Frequently Asked Questions

Is ‘invalid capacity 0 on image filesystem’ dangerous? It is harmless if it only appears briefly at startup. If it persists, image garbage collection and disk-pressure eviction stop working, which can fill the node’s disk.

What does image filesystem (imagefs) mean here? It is the filesystem the container runtime uses to store image layers and container writable layers, typically /var/lib/containerd for containerd. The kubelet tracks its capacity for GC and eviction decisions.

How do I know if it is just a startup glitch? Grep the kubelet journal after the node has been up for several minutes. A few lines right after a restart is normal; a continuous stream is a genuine runtime stats problem.

Why does crictl imagefsinfo matter? It queries the CRI directly. If it returns empty or errors, the runtime is not reporting filesystem stats, which is the real root cause rather than the kubelet.

Can this cause pods to be evicted? Not directly, but a broken imagefs reading disables proper disk-pressure detection, so the node may run out of space and then evict pods abruptly. For more node-level troubleshooting, see the Kubernetes & Helm guides.