☸️ Kubernetes Error Guide: 'etcdserver: too many requests' API Server Overload

Overview

The etcdserver: too many requests error surfaces when the API server forwards a request to etcd and etcd rejects it because too many operations are already in flight. A typical failure looks like this:

Error from server: etcdserver: too many requests

or, in a controller log:

E0716 10:02:41.559210   1 status.go:71] apiserver received an error that is not an
metav1.Status: rpctypes.EtcdError{code:0xe, desc:"etcdserver: too many requests"}

etcd is the single source of truth for all cluster state, and it protects itself with an inflight-request limit and a bounded mvcc (multi-version concurrency control) transaction pipeline. When writes and reads pile up faster than etcd can commit them — because of a slow disk, a large database, a defrag, or a flood of expensive list/write calls — etcd returns too many requests and the API server propagates it to the caller.

This is a capacity and back-pressure signal, not a corruption. The fix is to reduce the load etcd is under, speed up etcd’s storage, or both.

Symptoms

• kubectl commands intermittently failing with etcdserver: too many requests.
• Controllers and operators logging the same error and backing off/retrying.
• API server latency spikes; apiserver_request_total showing 429/500 responses.
• etcd logs showing apply request took too long, slow fdatasync, or took too long ... to execute.
• Symptoms worsening during large kubectl get ... --all-namespaces calls or bulk applies.

Common Root Causes

1. Slow etcd disk (fsync latency)

etcd fsyncs every write to the WAL. If backing storage has high fsync/fdatasync latency (network disks, throttled IOPS, noisy neighbors), commits queue up and the inflight limit is hit.

2. A large etcd database and mvcc pressure

A big keyspace, many object revisions, or a deferred defrag inflate the DB. mvcc bookkeeping grows, transactions slow, and back-pressure kicks in. Large numbers of Events, Secrets, or CRs are common culprits.

3. Expensive or high-frequency requests

Full cluster-wide LIST calls (especially unpaginated, non-cached), tight reconcile loops, or a misbehaving client hammering the API server multiply etcd load.

4. Defragmentation or compaction in progress

etcdctl defrag briefly blocks the member it runs against; if run on all members at once, the cluster rejects requests.

5. Undersized control plane

Too few etcd members, CPU-starved control-plane nodes, or a single-node etcd carrying a large cluster cannot keep up.

Diagnostic Workflow

Step 1: Confirm etcd is the bottleneck

kubectl get --raw='/readyz?verbose' | grep -i etcd
kubectl get componentstatuses

Step 2: Check etcd endpoint status and DB size

sudo ETCDCTL_API=3 etcdctl endpoint status --write-out=table \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Step 3: Look for slow-disk warnings in etcd logs

kubectl -n kube-system logs etcd-$(hostname) | grep -Ei "slow|took too long|fdatasync"

Step 4: Find what is generating load

kubectl get --raw='/metrics' | grep -E "etcd_disk_wal_fsync_duration|apiserver_request_total" | head
kubectl get events --all-namespaces | wc -l

Step-by-Step Resolution

1. Verify cluster health first. Confirm all etcd members are up and one is not lagging or down, which forces the rest to absorb extra load.

2. Compact and defragment etcd (one member at a time). A bloated DB is the most common cause. Compact to the current revision, then defrag each member individually:

REV=$(sudo ETCDCTL_API=3 etcdctl endpoint status --write-out=json \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key | grep -o '"revision":[0-9]*' | head -1 | cut -d: -f2)
sudo ETCDCTL_API=3 etcdctl compact $REV --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key
sudo ETCDCTL_API=3 etcdctl defrag --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

3. Move etcd to faster storage. If wal_fsync_duration is high, put etcd on low-latency local SSD/NVMe with dedicated IOPS. This is the single biggest lever.

4. Reduce request load. Find and fix clients doing frequent full LISTs; ensure controllers use informers/watch caches and paginate large reads.

5. Trim churny objects. Shorten Event TTL and reduce high-frequency object updates that inflate the keyspace:

grep event-ttl /etc/kubernetes/manifests/kube-apiserver.yaml

6. Scale the control plane. Give etcd/apiserver nodes more CPU and memory, and run a 3- or 5-member etcd cluster on separate fast disks.

7. Re-test after each change. Watch the error rate drop before moving on:

kubectl get --raw='/metrics' | grep 'apiserver_request_total.*429'

Prevention

• Run etcd on dedicated low-latency SSD/NVMe and monitor etcd_disk_wal_fsync_duration_seconds.
• Schedule regular compaction and staggered defrag (never all members at once).
• Keep the etcd DB well under its --quota-backend-bytes limit and alert on growth.
• Audit controllers for full-collection LISTs; require watch/informer caches and pagination.
• Cap Event volume with a sane --event-ttl and avoid hot-looping updates on the same objects.

Related Errors

• etcdserver: mvcc: database space exceeded — etcd hit its backend quota; needs compaction/defrag and an alarm reset.
• etcdserver: request timed out — a write did not commit in time, often the same disk pressure.
• too old resource version — watch churn that adds to etcd load.
• rpc error: code = Unavailable desc = etcdserver: leader changed — a leader election under stress.

Frequently Asked Questions

Does etcdserver: too many requests mean etcd is corrupted? No. It is a back-pressure signal meaning etcd has too many in-flight operations. Data is intact; etcd is protecting itself from overload.

What is the fastest single fix? Put etcd on faster, low-latency disk. High WAL fsync latency is the most common root cause, and better storage relieves it immediately.

Is it safe to run etcdctl defrag? Yes, but run it on one member at a time. Defrag briefly blocks the member it targets, so defragging all members simultaneously will make the cluster reject requests. The DevOps AI prompt library includes prompts for building a safe rolling-defrag runbook.

Why do large kubectl get --all-namespaces calls trigger it? Big unpaginated LISTs are expensive reads against etcd. Under existing pressure they push it over the inflight limit. Paginate and prefer cached reads.

How do I stop it recurring? Fix slow disks, compact/defrag regularly, reduce churny objects like Events, and make controllers use watch caches instead of repeated full LISTs. For more, see the Kubernetes & Helm guides.