🚨 Logstash Error Guide: 'retrying failed action with response code: 429' — Handle Elasticsearch Backpressure

Overview

The Logstash elasticsearch output sends events in bulk requests. When Elasticsearch’s write thread pool is saturated it rejects the bulk with HTTP 429, and Logstash logs the retry:

[INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or
were rejected by the previous bulk request. {:count=>250}
[ERROR][logstash.outputs.elasticsearch] Encountered a retryable error (will retry with
exponential backoff) {:code=>429, :url=>"http://es:9200/_bulk",
:content_length=>1048576, :body=>"{\"error\":{\"type\":\"es_rejected_execution_exception\"...}"}

429 (too_many_requests / es_rejected_execution_exception) is Elasticsearch applying backpressure: it cannot index as fast as Logstash is pushing. Logstash retries with exponential backoff, so no data is lost — but the pipeline stalls and lag grows until the cluster catches up.

Symptoms

• Repeated retrying failed action ... response code: 429 lines in logstash-plain.log.
• Ingest lag climbs; documents appear in Kibana minutes or hours late.
• _cat/thread_pool/write shows a high rejected counter and a full queue on data nodes.
• Logstash CPU and throughput drop as workers spend time backing off.
• Upstream inputs apply backpressure — Beats clients slow down, or the persistent queue fills.

Common Root Causes

• Undersized or overloaded cluster — too few data nodes, slow disks, or hot shards cannot absorb the write rate.
• Write thread-pool queue exhausted — the write pool’s bounded queue (default 10000) fills and further bulks are rejected.
• Bulk requests too large or too frequent — huge pipeline.batch.size × many elasticsearch output workers overwhelm ES.
• Expensive indexing — many fields, heavy mappings, refresh_interval: 1s, or ingest pipelines make each document costly.
• Shard hotspotting — all writes hitting one shard/node instead of spreading across the cluster.
• Force-merge or heavy search load competing with indexing for the same threads.

Diagnostic Workflow

Confirm the rejections are on the Elasticsearch side and see which nodes are hot:

curl -s 'http://es:9200/_cat/thread_pool/write?v&h=node_name,active,queue,rejected,completed'
curl -s 'http://es:9200/_nodes/stats/thread_pool?filter_path=**.write' | \
  grep -A5 write

Check cluster and indexing health to see whether ES is genuinely overloaded:

curl -s 'http://es:9200/_cluster/health?pretty'
curl -s 'http://es:9200/_cat/indices/logstash-*?v&h=index,pri,rep,docs.count,store.size&s=store.size:desc'
curl -s 'http://es:9200/_nodes/hot_threads?threads=3'

Look at the Logstash side — the output’s retry and bulk behavior:

curl -s localhost:9600/_node/stats/pipelines?pretty | \
  grep -A8 'elasticsearch'

Tune the pipeline to send smaller, less frequent bulks. In pipelines.yml:

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/main.conf"
  pipeline.workers: 4
  pipeline.batch.size: 500      # smaller bulk per flush relieves the ES write queue

And in the output block, cap concurrency and enlarge backoff:

output {
  elasticsearch {
    hosts => ["http://es1:9200","http://es2:9200"]
    index => "logstash-%{+YYYY.MM.dd}"
    retry_on_conflict => 3
    # Fewer output workers -> fewer concurrent bulks hitting ES
  }
}

On the Elasticsearch side, relax the index for bulk ingest:

curl -s -X PUT 'http://es:9200/logstash-*/_settings' -H 'Content-Type: application/json' -d'
{ "index": { "refresh_interval": "30s", "number_of_replicas": 1 } }'

Example Root Cause Analysis

During a marketing event, log volume tripled. Logstash began logging retrying failed action ... 429 continuously and Kibana fell 40 minutes behind. _cat/thread_pool/write showed queue=10000 rejected=1.4M concentrated on two of five data nodes — a hotspot.

The index logstash-2026.07.10 had only 1 primary shard, so every write funneled to a single node. Combined with refresh_interval: 1s and a pipeline.batch.size of 3000 across 8 output workers (24k events per flush cycle), the two involved nodes could not drain their write queues.

The remediation: increase primaries to 5 (via the index template, applied on the next daily index) to spread writes, raise refresh_interval to 30s during peak, and drop pipeline.batch.size to 500. Rejections fell to zero within minutes and lag drained. Because Logstash retries 429s with backoff, no events were lost during the incident — only delayed.

Prevention Best Practices

• Size primary shards so writes spread across all data nodes; avoid single-shard daily indices under high ingest.
• Raise refresh_interval (e.g. 30s) on ingest-heavy indices; the 1s default costs indexing throughput for little benefit on log data.
• Keep pipeline.batch.size modest and scale ingest with more Logstash pipelines/nodes rather than giant bulks.
• Monitor _cat/thread_pool/write rejected and alert on a rising rate — it is the leading indicator of 429s.
• Use an index template with sensible mappings; fewer analyzed fields and no unnecessary keyword+text doubling lowers per-doc cost.
• Provision headroom: 429s during predictable peaks mean the cluster is undersized for the burst, not just misconfigured.

Quick Command Reference

# Are rejections happening, and where?
curl -s 'http://es:9200/_cat/thread_pool/write?v&h=node_name,active,queue,rejected'

# Cluster and hot-shard view
curl -s 'http://es:9200/_cluster/health?pretty'
curl -s 'http://es:9200/_nodes/hot_threads?threads=3'

# Logstash ES output stats
curl -s localhost:9600/_node/stats/pipelines?pretty | grep -A8 elasticsearch

# Relieve ingest pressure on the index
curl -s -X PUT 'http://es:9200/logstash-*/_settings' -H 'Content-Type: application/json' \
  -d '{"index":{"refresh_interval":"30s"}}'

Conclusion

A 429 from the Logstash elasticsearch output is Elasticsearch throttling ingest because its write thread pool is saturated — not data loss, since Logstash retries with exponential backoff. Fix it on both ends: spread writes across shards/nodes, raise refresh_interval, and lighten mappings on the ES side; shrink pipeline.batch.size and output concurrency on the Logstash side. Alert on the write pool’s rejected counter so you catch backpressure before it becomes hours of ingest lag.