🚨 Filebeat Error Guide: '429 Too Many Requests' — Relieve Bulk Backpressure

Overview

When Elasticsearch’s write thread pool is saturated, it rejects bulk items with HTTP 429. Filebeat treats 429 as retryable backpressure, resending the rejected items with backoff, and logs:

Failed to perform any bulk index operations: 429 Too Many Requests: {"error":{"type":"es_rejected_execution_exception","reason":"rejected execution of coordinating operation [...] queue capacity exceeded"}}

429 is Elasticsearch’s flow-control signal: the write thread pool’s queue is full, so it refuses additional bulk operations rather than falling over. This is healthy backpressure — Filebeat retries and no data is lost — but sustained 429s mean the cluster cannot keep up with the offered indexing rate. The fix is to reduce how hard Filebeat pushes and/or increase how much the cluster can absorb, not to ignore the signal.

Symptoms

• Recurring bulk 429 Too Many Requests with es_rejected_execution_exception in the Filebeat log.
• Ingestion throughput plateaus and sawtooths as retries back off and resume.
• _cat/thread_pool/write shows a climbing rejected counter and queue near capacity.
• Latency on searches rises because data nodes are pinned on indexing/merges.
• Bursts of 429 align with traffic spikes or scheduled log floods.

Common Root Causes

• Offered rate exceeds capacity — more events per second than the hot data nodes can index.
• Too many workers / oversized bulks — high worker count and large bulk_max_size overwhelm the write queue.
• Too few hot nodes or shards — indexing concentrated on a small number of primaries.
• Merge / refresh pressure — a low refresh_interval forces frequent segment work, starving the write pool.
• Slow disks — I/O-bound data nodes cannot flush fast enough.
• Thundering herd — many Filebeat instances flush in synchronized waves.

Diagnostic Workflow

Confirm the rejections are on the write thread pool and growing:

curl -sk 'https://es01:9200/_cat/thread_pool/write?v&h=node_name,active,queue,rejected,completed' \
  -u elastic:$ES_PASS

Watch Filebeat retry the same batch:

journalctl -u filebeat -f | grep -iE '429|too many requests|retry'

Check indexing pressure and hot threads on the busy nodes:

curl -sk 'https://es01:9200/_nodes/stats/indexing_pressure?pretty' -u elastic:$ES_PASS \
  | grep -A5 primary
curl -sk 'https://es01:9200/_nodes/hot_threads' -u elastic:$ES_PASS | head -40

Review the target index’s shard spread and refresh interval:

curl -sk 'https://es01:9200/_cat/shards/filebeat-*?v&h=index,shard,prirep,node,docs' -u elastic:$ES_PASS
curl -sk 'https://es01:9200/filebeat-*/_settings/index.refresh_interval?pretty' -u elastic:$ES_PASS

Example Root Cause Analysis

During a nightly batch job, dozens of Filebeat agents flooded the cluster and every one logged 429 Too Many Requests. _cat/thread_pool/write showed rejected jumping by thousands on the two hot nodes, with queue pinned at 10000. Two contributing factors emerged: Filebeat used worker: 4 with bulk_max_size: 4096, and the target index had only two primary shards, so all writes hit two nodes.

The team tuned Filebeat to push less aggressively and let backoff spread the load:

output.elasticsearch:
  worker: 1
  bulk_max_size: 1000
  backoff.init: 2s
  backoff.max: 120s

They also raised the write index to four primaries and lengthened refresh_interval to 30s during the batch window to cut refresh overhead. Afterward, rejected stopped incrementing and the 429s cleared — the offered rate now fit within the cluster’s absorbed rate.

Prevention Best Practices

• Size shards and hot nodes for the sustained event rate so the write queue is not chronically near capacity.
• Keep bulk_max_size moderate and worker low enough that many agents do not overwhelm the cluster in unison.
• Raise refresh_interval on high-ingest indices (e.g. 30s) to reduce refresh/merge pressure.
• Monitor _cat/thread_pool/write rejected and indexing pressure; alert on sustained rejections, not one-off spikes.
• Stagger or buffer bulk log floods (Kafka/Logstash) so batch jobs do not create synchronized write waves.

Quick Command Reference

curl -sk 'https://es01:9200/_cat/thread_pool/write?v&h=node_name,active,queue,rejected' -u elastic:$ES_PASS
curl -sk 'https://es01:9200/_nodes/stats/indexing_pressure?pretty' -u elastic:$ES_PASS
curl -sk 'https://es01:9200/_cat/shards/filebeat-*?v' -u elastic:$ES_PASS
journalctl -u filebeat -f | grep -i 429

Conclusion

429 Too Many Requests is Elasticsearch applying backpressure because its write queue is full — Filebeat’s retries mean no data is lost, but sustained 429s signal a capacity gap. Verify the rejections on the write thread pool, then push less from Filebeat (smaller bulks, fewer workers, longer backoff) and absorb more in the cluster (more shards/nodes, longer refresh). Balance the two and the throttling clears. More output tuning is in the Filebeat guides.