🚨 Kubernetes Error Guide: 'Client.Timeout exceeded while awaiting headers' kubectl Timeouts

Exact Error Message

A client (kubectl, a controller, or any client-go program) gives up waiting for the API server to send response headers and cancels the request:

Unable to connect to the server: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

The same failure shows up inside controllers and operators wrapping a list/watch:

E0628 15:31:09.774120  1 reflector.go:147] k8s.io/client-go/informers: Failed to watch *v1.Pod: failed to list *v1.Pod: Get "https://10.0.0.1:443/api/v1/pods?limit=500": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

The defining phrase is Client.Timeout exceeded while awaiting headers: the TCP connection was established, but the server did not return the response headers within the client’s timeout, so the Go HTTP client aborted the request.

What the Error Means

This error is generated client-side, by Go’s net/http. The client opened a connection and sent the request, then waited for the server’s response headers. When no headers arrived within the configured timeout — kubectl --request-timeout, a client-go Timeout, or a transport deadline — the client itself canceled the request. “Awaiting headers” specifically means the wait failed before any response body started, i.e. the server had not even begun replying.

Crucially, the connection itself usually succeeded (otherwise you would see connection refused or i/o timeout on dial). The bottleneck is time to first byte from the apiserver. That points to one of: an overloaded or slow API server, a slow etcd backend behind it, an expensive unpaginated query, or a load balancer / proxy between you and the apiserver that is adding latency or holding the connection.

So this is a latency symptom, not a connectivity symptom. The fix is to find what is slow in the path — apiserver, etcd, the request itself, or the network/LB — rather than assuming the cluster is unreachable.

Common Causes

• Overloaded API server — high QPS, CPU saturation, or API Priority and Fairness throttling delays the response.
• Slow etcd backend — apiserver waits on etcd reads (slow disk, election storms), so headers are late.
• Expensive query — a large unpaginated LIST (all pods/events cluster-wide) takes longer than the timeout.
• Load balancer latency / idle timeouts — an LB or proxy in front of the apiserver adds delay or resets long calls.
• Too-low --request-timeout — the client deadline is shorter than a legitimately slow but valid request.
• Network latency / packet loss — high RTT between client and apiserver endpoint.
• apiserver restarting — rolling control-plane update leaves a brief window with no responder.

How to Reproduce the Error

Force the timeout by setting an absurdly short client deadline against a normal apiserver:

# A 1ms request timeout will almost always trip the awaiting-headers cancel
kubectl get pods --all-namespaces --request-timeout=1ms

Unable to connect to the server: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

In production the realistic trigger is a heavy query against a busy apiserver:

# Large unpaginated list against a loaded control plane can exceed a short timeout
kubectl get events --all-namespaces --request-timeout=5s

The connection succeeds; the apiserver just does not produce headers fast enough within the deadline.

Diagnostic Commands

# Is it slow or unreachable? Time a trivial call with verbose round-trip detail
time kubectl get --raw='/readyz?verbose' --request-timeout=30s

# Measure apiserver health and latency directly
kubectl get --raw='/healthz' --request-timeout=10s
kubectl get --raw='/livez?verbose' --request-timeout=10s

# apiserver request latency and throttling from its own metrics
kubectl get --raw='/metrics' | grep -E 'apiserver_request_duration_seconds|apiserver_flowcontrol_rejected_requests_total'

# Is the slowness in etcd? Check backend latency the apiserver depends on
curl -s http://127.0.0.1:2381/metrics | grep -E 'etcd_disk_backend_commit_duration|etcd_server_slow'

# Control-plane / LB reachability and round-trip time
journalctl -u kube-apiserver --no-pager 2>/dev/null | grep -iE 'Timeout|too slow|trace'

Compare a trivial /healthz call (fast = apiserver fine, the problem is your heavy query or client timeout) against a real list. Rising apiserver_request_duration_seconds p99 or nonzero apiserver_flowcontrol_rejected_requests_total points at apiserver overload/throttling; high etcd_disk_backend_commit_duration points at etcd.

Step-by-Step Resolution

1. Separate latency from connectivity. A short raw /healthz that succeeds means the path is reachable — you have a latency problem, not an outage. If even /healthz times out, suspect the LB or a down apiserver.

2. Time a baseline call. time kubectl get --raw=/readyz. Sub-second is healthy; multi-second means the control plane or path is slow.

3. Fix the query, not just the timeout. If a specific LIST is slow, paginate or scope it (--chunk-size, label/field selectors, a single namespace) instead of pulling everything. Unbounded event/pod lists are a frequent culprit.

4. Raise --request-timeout for legitimately long calls. If the request is valid but slow, increase the client deadline rather than failing. For controllers, raise the client-go Timeout/QPS/Burst appropriately.

5. Relieve apiserver/etcd load. If apiserver latency or APF rejections are high, scale the control plane, identify abusive clients (a hot controller, a watch storm), and ensure etcd has fast disk. Apiserver waits on etcd, so etcd latency surfaces here.

6. Inspect the load balancer. Confirm the LB’s idle/response timeout exceeds your longest valid request and that it is not resetting long-lived watches. Check RTT and packet loss to the apiserver VIP.

Prevention and Best Practices

• Always paginate large reads (--chunk-size, server-side pagination) so a single client cannot issue a multi-second unbounded LIST.
• Alert on apiserver_request_duration_seconds p99 and APF rejection counters to catch overload before clients time out.
• Keep etcd on fast disks and monitor backend commit latency — apiserver response time inherits it.
• Set LB response/idle timeouts longer than the apiserver’s own request timeout, and exclude watches from aggressive idle resets.
• Tune controller client-go QPS/Burst/Timeout for the cluster size; defaults throttle large clusters.
• Use realistic --request-timeout values in automation so transient slowness retries rather than hard-failing. More in our Kubernetes & Helm guides.

Related Errors

• context deadline exceeded — the server-side deadline cousin (apiserver-to-etcd), versus this client-side cancel.
• etcd request timed out — slow etcd that makes the apiserver slow to respond.
• etcdserver: leader changed — election storms that spike apiserver latency.

Frequently Asked Questions

Does this mean my cluster is down? Usually not. The phrase “awaiting headers” means the connection succeeded but the server was slow to start replying. A down or unreachable apiserver fails differently — connection refused or a dial i/o timeout. This is a latency problem.

Why does kubectl get pods time out but kubectl get --raw=/healthz works? /healthz is a trivial, cheap endpoint, so it returns instantly even when the apiserver is busy. A full pod list hits etcd and serializes many objects; under load it can exceed your client timeout while the lightweight health check still passes.

Is the timeout coming from kubectl or the server? From the client. Client.Timeout is the Go HTTP client’s own deadline (--request-timeout for kubectl, the client-go Timeout for controllers). The server did not impose it — your client gave up waiting.

Will raising --request-timeout fix it? It fixes the symptom for genuinely slow-but-valid requests. But if the apiserver or etcd is overloaded, a longer timeout just waits longer. Address the underlying latency (paginate, scale, fix etcd) for a durable fix.

My controller logs this on watches — is that a problem? Occasional watch re-establishment after a timeout is normal; client-go reconnects automatically. A flood of these means the apiserver is overloaded or an LB is killing long-lived connections — investigate apiserver latency and LB idle timeouts.