🚨 Logstash Error Guide: 'Pipeline is blocked' — Diagnose and Clear Output Backpressure

Overview

Logstash pushes events forward only as fast as its slowest output accepts them. When an output stalls, the pipeline blocks and the stall propagates back to the inputs. Logstash surfaces this as a stalled-worker warning:

[WARN ][org.logstash.execution.ShutdownWatcherExt] Received shutdown signal, but
pipeline is still waiting for in-flight events to be processed. {"inflight_count"=>500,
"stalling_threads_info"=>{"other"=>[{"thread_id"=>27,
"name"=>"[main]>worker3", "current_call"=>
"[...]/logstash-output-elasticsearch/.../common.rb:...:in `safe_bulk'"}]}}

The current_call pointing into an output plugin (here safe_bulk in the Elasticsearch output) is the tell: the pipeline is blocked on that output. Inputs stop reading, upstream queues fill, and end-to-end lag grows until the output drains.

Symptoms

• pipeline is still waiting for in-flight events with stalling_threads_info naming an output plugin.
• Ingest lag climbs; the persistent queue grows toward queue.max_bytes.
• Beats/TCP inputs apply backpressure and upstream clients slow or buffer.
• _node/stats shows output duration_in_millis dominating pipeline time.
• Shutdown hangs — systemctl stop logstash takes a long time because in-flight events cannot flush.
• CPU is low even though throughput is low: workers are waiting, not computing.

Common Root Causes

• Slow or overloaded output — Elasticsearch returning 429s, a saturated Kafka broker, a laggy HTTP endpoint, or slow disk on a file output.
• Too few workers or too small a batch for a high-latency output — round-trip latency starves throughput.
• A blocking filter — a ruby filter making a synchronous network call, or grok with catastrophic backtracking, stalling workers before the output.
• Persistent queue on slow disk — queue.type: persisted fsync latency limits throughput.
• Network issues — packet loss or high RTT to the output endpoint.
• Downstream backpressure by design — the output deliberately rate-limits and Logstash correctly blocks rather than dropping data.

Diagnostic Workflow

Identify where the time goes. The monitoring API breaks down duration per plugin:

curl -s localhost:9600/_node/stats/pipelines?pretty | \
  grep -E '"id"|duration_in_millis|"in"|"out"'

Find stalling threads directly with a thread dump — look for workers parked in an output:

PID=$(pgrep -f org.logstash.Logstash)
jstack "$PID" | grep -A5 '\[main\]>worker'

Check the queue depth — a growing persistent queue confirms downstream backpressure:

curl -s localhost:9600/_node/stats/pipelines?pretty | grep -A6 '"queue"'
du -sh /var/lib/logstash/queue/main/

Tune pipeline parallelism for a high-latency output in pipelines.yml:

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/main.conf"
  pipeline.workers: 8            # more workers hide per-request latency
  pipeline.batch.size: 250
  queue.type: persisted          # spill to disk instead of stalling inputs
  queue.max_bytes: 8gb

If Elasticsearch is the blocked output, confirm it is the bottleneck:

curl -s 'http://es:9200/_cat/thread_pool/write?v&h=node_name,active,queue,rejected'
curl -s 'http://es:9200/_cluster/health?pretty' | grep status

Test the output endpoint’s latency directly to rule out the network:

time curl -s -o /dev/null 'http://es:9200/_bulk' -H 'Content-Type: application/x-ndjson' \
  --data-binary $'{"index":{"_index":"probe"}}\n{"t":"x"}\n'

Example Root Cause Analysis

A pipeline forwarding to a remote Elasticsearch cluster over a WAN link (60 ms RTT) ran with the default pipeline.workers: 4 and pipeline.batch.size: 125. Throughput capped at ~8k events/s and the persistent queue steadily grew, with shutdown watcher warnings pointing at safe_bulk.

A jstack showed all four workers parked inside the Elasticsearch output, each waiting on a bulk round-trip. With 60 ms latency and only 4 concurrent bulks of 125 events, the pipeline was latency-bound, not CPU-bound — ES itself showed queue=0 rejected=0, so the cluster was not the problem; the link RTT was.

The fix was to raise pipeline.workers to 12 and pipeline.batch.size to 500, increasing in-flight bulk concurrency to hide the WAN latency. Throughput rose to ~35k events/s, the queue drained, and the stall warnings stopped. Because the queue was persistent, no data was lost while the backlog cleared.

Prevention Best Practices

• Always read stalling_threads_info / a jstack to identify which output is blocking before tuning blindly.
• For high-latency outputs, increase pipeline.workers and pipeline.batch.size to raise in-flight concurrency; for CPU-bound filters, match workers to cores.
• Use a persistent queue with generous queue.max_bytes so transient output slowness spills to disk instead of stalling inputs and dropping upstream data.
• Never make synchronous network calls inside a ruby filter; move enrichment to purpose-built async filters.
• Monitor per-plugin duration_in_millis and queue depth; alert when the queue trends toward queue.max_bytes.
• Right-size or scale the downstream (Elasticsearch shards, Kafka partitions) so it can absorb Logstash’s throughput.

Quick Command Reference

# Per-plugin timing (find the slow stage)
curl -s localhost:9600/_node/stats/pipelines?pretty | grep -E 'duration_in_millis|"id"'

# Which workers are stalled and where?
jstack $(pgrep -f org.logstash.Logstash) | grep -A5 '\[main\]>worker'

# Queue depth (backpressure indicator)
du -sh /var/lib/logstash/queue/main/
curl -s localhost:9600/_node/stats/pipelines?pretty | grep -A6 '"queue"'

# Is Elasticsearch the bottleneck?
curl -s 'http://es:9200/_cat/thread_pool/write?v&h=node_name,active,queue,rejected'

Conclusion

A blocked Logstash pipeline is a backpressure symptom: the slowest output cannot accept events fast enough, and the stall propagates back through the workers to the inputs. Diagnose it precisely with stalling_threads_info and a jstack to name the offending output, then tune deliberately — more workers and larger batches for latency-bound outputs, more capacity for saturated ones. Front the pipeline with a persistent queue so transient slowness spills to disk instead of dropping upstream data, and alert on queue depth so you intervene before the backlog becomes visible lag.