Logstash Error: 'Failed to flush outgoing items'

Overview

The elasticsearch output sends events to ES in bulk batches. When a bulk flush throws an exception — instead of returning any HTTP response — the output logs Failed to flush outgoing items with the underlying exception attached:

[ERROR][logstash.outputs.elasticsearch][main] Failed to flush outgoing items {:outgoing_count=>1, :exception=>"Manticore::SocketTimeout", :backtrace=>["...", "..."]}

The key detail is that the request threw rather than replied. A Manticore::SocketTimeout (or a connection reset) means Logstash sent the bulk and never got an answer back within the HTTP timeout. The output retries this indefinitely — which is safe for data, but it blocks the pipeline while it retries, so backpressure builds and upstream inputs slow or stop. This is distinct from Elasticsearch Unreachable (a transport failure where the connection itself is dead) and from a 429 rejection (ES answered, but told you to slow down).

Symptoms

Failed to flush outgoing items with Manticore::SocketTimeout or a connection-reset exception in logstash-plain.log.
The same batch retries repeatedly; the pipeline stalls and the queue or DLQ grows.
Upstream inputs experience backpressure — Beats connections stall, file tailing lags.
Elasticsearch is up and reachable but slow: high write-thread-pool queue, long GC pauses, or heavy load.
It clusters around ingest spikes, large batches, or periods when ES nodes are under pressure.

Common Root Causes

Elasticsearch overloaded or in long GC — the node is too busy to answer the bulk within the timeout, so the socket times out.
Network drops between Logstash and ES — a reset or lost connection kills the in-flight bulk.
HTTP timeout too low — the default 60 is not enough for large or slow bulks, so Logstash gives up before ES finishes.
Batch too large — a big pipeline.batch.size produces bulks that take longer than the timeout to process.
Insufficient ES write capacity — too few write threads / nodes for the ingest rate, so the write thread pool backs up.

How to diagnose

Check whether Elasticsearch is the bottleneck first. Look at the write thread pool for a growing queue or rejections, and at node health for GC and load:

# Write thread pool: queue and rejected counts
curl -s 'https://es.internal:9200/_cat/thread_pool/write?v&h=node_name,active,queue,rejected'

# Node load, heap, and GC pressure
curl -s 'https://es.internal:9200/_cat/nodes?v&h=name,heap.percent,ram.percent,cpu,load_1m'

Watch the Logstash node stats to see the output stalling and events backing up:

# Output and pipeline stats — look at retries and queue depth
curl -s localhost:9600/_node/stats/pipelines?pretty | grep -A15 elasticsearch

Review the output’s timeout and the pipeline batch size, since both govern how long a bulk takes and how long Logstash waits. The HTTP timeout lives on the output (Logstash .conf uses a Ruby-like DSL, shown as ruby):

output {
  elasticsearch {
    hosts   => ["https://es.internal:9200"]
    index   => "app-logs-%{+YYYY.MM.dd}"
    timeout => 60          # default; too low for large/slow bulks
  }
}

The batch size lives in logstash.yml, shown as yaml:

pipeline.batch.size: 1000   # large batches produce slow bulks

Fixes

Give slow bulks more time and make each bulk smaller so it completes inside the timeout. Raise the output timeout:

output {
  elasticsearch {
    hosts   => ["https://es.internal:9200"]
    index   => "app-logs-%{+YYYY.MM.dd}"
    timeout => 90          # allow slow-but-healthy bulks to finish
  }
}

Reduce the batch size so each bulk is lighter on ES:

pipeline.batch.size: 500    # smaller, faster-to-process bulks

Then address the ES side, which is usually the real cause: if _cat/thread_pool/write shows a growing queue or non-zero rejected, the cluster needs more write capacity — add nodes, tune shard counts, or reduce competing load. Restart Logstash after changing logstash.yml, and confirm recovery:

sudo systemctl restart logstash
sudo tail -f /var/log/logstash/logstash-plain.log | grep -Ei 'flush|SocketTimeout'

If the failure is genuinely transient — a brief GC pause or a momentary network blip — the output’s built-in retry recovers on its own once ES catches up; the events are not lost.

What to watch out for

The output retries indefinitely, so events are not dropped — but the pipeline blocks while it retries, which is why upstream backpressure is the real symptom to watch.
Distinguish the three failure modes: Failed to flush / SocketTimeout (ES too slow to answer), Elasticsearch Unreachable (transport dead), and 429 (ES answered, rate-limiting you). The fixes differ.
Raising timeout masks a chronically overloaded cluster — if the write thread pool is constantly rejecting, add ES capacity rather than just waiting longer.
Smaller batches reduce per-bulk latency but increase request overhead; tune pipeline.batch.size against your throughput, not to an extreme.
Monitor the DLQ and node stats so a persistent flush failure is caught before the queue fills and backpressure reaches your inputs.

Logstash Error: 'Failed to flush outgoing items' — Cause, Fix, and Troubleshooting Guide

Fixing errors like this? Get 500 free DevOps AI prompts