☸️ Kubernetes Error Guide: 'too old resource version' Watch and Informer Failures

Overview

The too old resource version error appears when a watch or informer tries to resume from a resource version that the API server can no longer serve. The message looks like this:

W0716 09:14:22.118374   1 reflector.go:539] k8s.io/client-go/informers/factory.go:159:
watch of *v1.Pod ended with: too old resource version: 12345 (67890)

Every object in Kubernetes carries a resourceVersion, and that value maps to a revision in etcd. Watches are resumable: a client says “send me everything that changed after revision X.” etcd periodically compacts old revisions to reclaim space, and once revision X has been compacted, the API server can no longer produce the delta from that point. It rejects the watch with too old resource version, and the client is expected to perform a fresh list (“relist”) to get a current baseline before watching again.

This is usually a benign, self-healing event. It becomes a real problem when it happens constantly, because each relist is an expensive full read against etcd. Storms of these errors point at controllers or CRD clients that watch inefficiently or fall behind.

Symptoms

• Controller, operator, or kubectl logs repeatedly printing too old resource version: <old> (<new>).
• reflector.go warnings and Watch close - watch chan error messages in client-go based components.
• Informer-backed controllers briefly stop reacting to changes, then resume after a relist.
• Elevated API server latency and etcd read load correlated with the warnings.
• Custom clients using a hardcoded or persisted resourceVersion failing immediately on start.

Common Root Causes

1. Normal etcd compaction

etcd compacts historical revisions on a schedule (default every 5 minutes via --etcd-compaction-interval). A watch that pauses longer than the retained history resumes past the compaction point and is legitimately too old. A single such message after a network blip is expected.

2. A slow or stalled watch consumer

If the client processes events slower than they arrive, its internal buffer fills, the watch is dropped, and the resume attempt lands on a compacted revision. Under-resourced controllers or blocking event handlers cause this.

3. Clients passing a stale or fixed resourceVersion

Custom code (or scripts) that persist a resourceVersion and later start a watch from it will fail once that revision is compacted. Watches should generally start from the resource version returned by a fresh list, not a stored value.

4. etcd under memory/mvcc pressure

Aggressive compaction or a defrag cycle shortens the effective history window, so even short pauses exceed retained revisions.

Diagnostic Workflow

Step 1: Confirm which client is complaining

Grep the offending component’s logs for the reflector warning and note the watched type:

kubectl logs -n kube-system deploy/my-controller | grep "too old resource version"

Step 2: Check API server and etcd health

kubectl get --raw='/readyz?verbose'
kubectl get --raw='/metrics' | grep etcd_request_duration_seconds_count

Step 3: Inspect etcd compaction settings

On a control-plane node, check the flag the kube-apiserver passes to etcd:

grep -E "etcd-compaction-interval|etcd-servers" /etc/kubernetes/manifests/kube-apiserver.yaml

Step 4: Measure how often it fires

If it is a handful of times per hour, it is normal. If it is many times per minute, treat it as a client-efficiency or etcd-pressure bug:

kubectl logs -n kube-system deploy/my-controller --since=10m \
  | grep -c "too old resource version"

Step-by-Step Resolution

1. Confirm it is actually causing harm. Occasional warnings are expected and require no action. Only act when the frequency is high or a controller is visibly lagging.

2. Ensure clients relist correctly. With client-go informers, this is automatic: on too old resource version the reflector performs a ListAndWatch from a fresh list. Upgrade any component pinned to an old client-go that does not relist cleanly.

3. Never start a watch from a persisted resourceVersion. In custom clients, always list first and start the watch from the list’s metadata.resourceVersion:

# Get the current collection resource version to anchor a fresh watch
kubectl get pods -o jsonpath='{.metadata.resourceVersion}'

4. Speed up slow consumers. Give lagging controllers more CPU/memory and remove blocking work from event handlers so watches are not dropped for buffer overflow.

5. Reduce watch volume with label selectors or fewer replicas. Watching all pods cluster-wide from many replicas multiplies etcd load. Scope watches with selectors where possible.

6. Tune etcd compaction only as a last resort. If pressure is genuine, a longer --etcd-compaction-interval retains more history but costs disk. Verify etcd headroom before changing it:

sudo ETCDCTL_API=3 etcdctl endpoint status --write-out=table \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Prevention

• Rely on the informer/reflector machinery in client-go rather than hand-rolling watch loops; it handles relist-on-compaction for you.
• Keep controllers adequately resourced so they never fall behind the event stream.
• Scope watches with label/field selectors and avoid redundant cluster-wide watches from many replicas.
• Monitor etcd DB size, defrag cadence, and compaction interval; keep etcd off a saturated disk.
• Alert on a rate of too old resource version messages, not a single occurrence, to avoid noise.

Related Errors

• etcdserver: mvcc: required revision has been compacted — the etcd-side equivalent of the same compaction race.
• etcdserver: too many requests — overload throttling that often accompanies watch storms.
• context deadline exceeded on list/watch — the client timing out against a stressed API server.
• The resourceVersion for the provided watch is too old — the aggregated-API-server variant of the same condition.

Frequently Asked Questions

Is too old resource version an error I need to fix? Usually not. An occasional warning is normal etcd compaction and the client relists automatically. Only investigate if it fires constantly or a controller is falling behind.

Why does etcd compaction cause this? etcd discards old revisions to reclaim space. A watch that resumes from a revision that was already compacted cannot be served, so the API server rejects it and asks the client to relist.

How should custom clients handle it? Do a fresh list and start the watch from that list’s metadata.resourceVersion. Never persist an old resourceVersion and reuse it later. The DevOps AI prompt library has ready-made prompts for auditing custom controller watch logic.

Will increasing the compaction interval fix it? It can retain more history and reduce the frequency, but it costs etcd disk and memory. Fix slow or misbehaving clients first; only tune compaction if etcd genuinely has headroom.

Does this indicate data loss? No. Compaction only removes historical revisions used for resuming watches. Current object state is intact; the client simply relists to rebuild its cache. For more patterns, see the Kubernetes & Helm guides.