☸️ Kubernetes Error Guide: 'Failed to watch *v1.Pod: unexpected EOF' Informer Watch Drops

Exact Error Message

W0716 10:42:18.113456       1 reflector.go:539] k8s.io/client-go/informers/factory.go:150:
watch of *v1.Pod ended with: an error on the server ("unexpected EOF") has prevented
the request from succeeding

E0716 10:42:18.114001       1 reflector.go:147] Failed to watch *v1.Pod: failed to list *v1.Pod:
Get "https://10.96.0.1:443/api/v1/pods?...&watch=true": unexpected EOF

The *v1.Pod part varies by whatever the controller watches — *v1.Endpoints, *v1.ConfigMap, *v1.Node, or a custom kind.

Overview

Controllers, operators, and kubelets keep long-lived HTTP watch streams open to the API server so they receive object changes in real time. A client-go reflector opens the stream and reads events until it is told the connection ended. unexpected EOF means the underlying TCP connection was closed mid-stream without a clean HTTP end — the reflector was mid-read when the socket dropped.

Individually these are warnings, not failures. client-go re-establishes the watch automatically with backoff, so a handful per hour is normal churn. The message becomes a real problem when it is frequent: a load balancer with a short idle timeout, an overloaded or restarting API server, or a proxy that caps connection lifetime will sever watches so often that controllers spend their time re-listing instead of reconciling, adding latency and API-server load.

Symptoms

• Controller or operator logs repeat Failed to watch *v1.<Kind>: ... unexpected EOF on a regular cadence.
• Reconciliation lags: changes take longer than usual to be acted on.
• API-server audit logs show a burst of LIST/WATCH requests as informers resync.
• The interval between EOFs closely matches a load balancer or proxy idle-timeout setting (for example every 60s or 300s).

Common Root Causes

1. Load balancer or proxy idle timeout

A cloud LB or reverse proxy in front of the API server closes idle connections after a fixed timeout, cutting watches that are quiet between events.

2. API server restart or rollout

During a control-plane upgrade or a kube-apiserver crash/restart, every open watch is dropped at once, producing a synchronized burst of EOFs.

3. API server overload

A saturated apiserver (high inflight requests, apf throttling, or memory pressure) drops or resets connections under load.

4. Network path instability

Packet loss, MTU mismatch, or a flapping CNI between the controller and the API server tears down TCP connections mid-stream.

Diagnostic Workflow

Step 1: Measure the frequency and cadence

kubectl logs -n kube-system deploy/my-controller --since=1h | grep -c "unexpected EOF"

A steady interval (every 60s, every 300s) strongly implies an idle timeout rather than random loss.

Step 2: Check whether the API server restarted

kubectl get pods -n kube-system -l component=kube-apiserver
kubectl get events -n kube-system --field-selector reason=Killing

Step 3: Look for API-server overload signals

kubectl get --raw='/metrics' | grep apiserver_current_inflight_requests
kubectl get --raw='/metrics' | grep apiserver_flowcontrol_rejected_requests_total

Step 4: Test raw watch stability past the suspected timeout

kubectl get pods --watch --request-timeout=0 -v=6

If the stream dies at a suspiciously round interval, an intermediary is closing it.

Step-by-Step Resolution

1. Confirm the cadence and correlate it with any LB or proxy idle-timeout configuration in front of the API server.

2. If a load balancer is severing idle watches, raise its idle timeout above the API server’s default watch lifetime (commonly extend to 300s or more), so quiet watches survive.

3. If the API server is overloaded, check and, if appropriate, tune API Priority and Fairness so the controller’s watches are not being shed:

kubectl get flowschemas
kubectl get prioritylevelconfigurations

4. Confirm the control plane is healthy and not crash-looping:

kubectl get componentstatuses
kubectl logs -n kube-system -l component=kube-apiserver --tail=100 | grep -i panic

5. Ensure client-side timeouts are sane. If a custom controller sets an aggressive --kube-api-timeout, lengthen it so normal watches are not cut client-side.

6. Verify recovery: after adjusting the timeout or relieving load, the EOF rate should fall to occasional churn:

kubectl logs -n kube-system deploy/my-controller --since=15m | grep -c "unexpected EOF"

For generating controller health checks and watch-resilience patterns, the DevOps AI prompt library has reusable prompts.

Prevention

• Set load balancer and proxy idle timeouts in front of the API server well above the expected quiet interval of watches.
• Keep controllers on a current client-go, which handles watch re-establishment and bookmarks efficiently.
• Enable and rely on watch bookmarks so re-established watches resume cheaply instead of full re-lists.
• Size the control plane for the number of watchers; many operators multiply open watches quickly.
• Alert on a high rate of reflector EOFs rather than treating any single one as a failure.
• Stagger controller restarts so they do not all re-list against the API server at the same moment after a control-plane blip.

Related Errors

• watch of *v1.<Kind> ended with: too old resource version — the watch fell behind the etcd compaction window, a different resync cause.
• the server has received too many requests and has asked us to try again — API Priority and Fairness throttling.
• http2: client connection lost — an HTTP/2 transport drop, often the same root cause as EOF.
• context deadline exceeded — a client- or server-side timeout rather than a mid-stream socket close.

Frequently Asked Questions

Is unexpected EOF on a watch always a problem? No. client-go re-establishes watches automatically, so occasional EOFs are normal churn. It becomes a problem only when the rate is high enough to delay reconciliation or spike API-server load.

Why do the EOFs happen at a regular interval? A fixed cadence — every 60s or 300s — almost always points at an idle-timeout on a load balancer or proxy between the controller and the API server, which closes connections that are quiet between events.

How do I tell an overloaded apiserver from a network drop? Check apiserver_current_inflight_requests and apiserver_flowcontrol_rejected_requests_total. Elevated values point at overload/throttling; if those are calm but EOFs persist on a timer, suspect an intermediary or the network path.

Do I need to restart the controller to fix it? Usually not. The reflector recovers on its own. Fix the cause — LB idle timeout, apiserver load, or network stability — and the EOF rate falls without restarting the controller.

For more control-plane and controller fixes, see the Kubernetes & Helm guides.