🚨 Logstash Error Guide: 'the pipeline is blocked, temporary refusing new connection' — Relieve Beats Input Backpressure

Overview

The beats input is the endpoint Filebeat, Metricbeat, and other Beats agents connect to over the Lumberjack protocol. It is a listener: it accepts TCP connections, reads batches, and hands them into the pipeline’s filter/output stages. When those downstream stages cannot keep up, the batch the input is holding cannot be released, and the input stops accepting new connections until it drains. At that moment it logs:

[2026-07-10T15:04:12,778][INFO ][org.logstash.beats.BeatsHandler][main][a1b2c3] [local: 10.0.4.7:5044, remote: 10.0.9.31:51423] Handling exception: the pipeline is blocked, temporary refusing new connection. (caused by: the pipeline is blocked, temporary refusing new connection.)

You will also see the mirror-image line from the input as it declines connections, and — importantly — matching errors on the Filebeat side:

[2026-07-10T15:04:12,779][INFO ][org.logstash.beats.Server][main] Refusing new connection because the pipeline is blocked

# On the Filebeat host:
{"log.level":"error","message":"Failed to connect to backoff(async(tcp://logstash:5044)): read tcp ... i/o timeout","service.name":"filebeat"}
{"log.level":"info","message":"Attempting to reconnect to backoff(async(tcp://logstash:5044)) with 3 reconnect attempt(s)","service.name":"filebeat"}

What it means: this is backpressure surfacing at the input. The beats input is deliberately refusing new Filebeat connections because it cannot push the data it already holds into a busy pipeline. It is a symptom, not a fault — the beats input is doing exactly what it should (protecting itself and applying flow control back to the shippers). The visible pain is on the ingest edge: Filebeat connections are refused, time out, back off, and retry, so log delivery stalls and lags. This guide focuses on that input-side symptom — why Filebeat is being refused and how to relieve the pressure — rather than on deep persistent-queue or output-store internals.

Symptoms

• Repeated the pipeline is blocked, temporary refusing new connection lines in /var/log/logstash/logstash-plain.log, often in bursts.
• Filebeat hosts log i/o timeout, connection reset by peer, or Failed to connect to backoff(async(tcp://logstash:5044)), followed by reconnect/backoff messages.
• Filebeat’s own registry lags: harvester offsets stop advancing and the shipper’s publish queue grows.
• Ingest latency climbs — events appear in Kibana minutes (or longer) after they were written.
• Under load, only new connections are refused; already-established Beats connections may keep trickling data slowly.
• The Logstash node’s pipeline shows a high proportion of time in the filter/output stage in _node/stats.

Common Root Causes

• A slow or stalled output — the downstream Elasticsearch is red/yellow, rejecting bulk requests (HTTP 429), or simply undersized. The output blocks, the whole pipeline blocks, and the beats input refuses connections. This is the most frequent underlying cause.
• Too few pipeline workers or too small a batch — pipeline.workers and pipeline.batch.size set too low for the event rate means the filter stage cannot drain the input fast enough.
• Expensive filters — a heavy grok with catastrophic backtracking, a dns filter doing synchronous lookups, or a ruby filter blocking on each event serializes throughput and stalls the pipeline.
• Thundering herd of Beats clients — hundreds of Filebeat agents connecting to a single beats input; the input’s executor/thread pool saturates and refuses new sockets even when the pipeline itself is only moderately busy.
• Under-provisioned Logstash node — insufficient CPU or a starved JVM heap causing long GC pauses that freeze the pipeline periodically.
• No load balancing — all Beats pointed at one Logstash node instead of a list, concentrating all backpressure on a single input.
• A blocked in-memory queue — the default memory queue between input and workers is bounded; when workers stall, it fills, and the input has nowhere to put the next batch.

Diagnostic Workflow

First, rule out a config error — a broken filter or output is a common reason the pipeline stalls. Validate before restarting anything:

# Syntax-check the full configured pipeline
sudo -u logstash /usr/share/logstash/bin/logstash --config.test_and_exit \
  --path.settings /etc/logstash

# Check one file
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/beats-in.conf --config.test_and_exit

Then look at where the pipeline is spending its time. The node stats API exposes per-plugin duration — a filter or output with a huge duration_in_millis relative to events is your bottleneck:

curl -s 'localhost:9600/_node/stats/pipelines?pretty' | \
  grep -A4 '"duration_in_millis"' | head -40

# Queue backpressure and reload state
curl -s 'localhost:9600/_node/stats/pipelines?pretty' | grep -i 'queue\|backpressure'

Here is a realistic beats input pipeline. The important knobs for this symptom are the input’s executor_threads/client_inactivity_timeout and the pipeline-level worker/batch settings that let the filter stage keep up:

input {
  beats {
    port                     => 5044
    # Threads that read from Beats sockets — raise when many agents connect
    executor_threads         => 8
    # Drop idle Beats connections so the pool is not held by dead sockets
    client_inactivity_timeout => 60
  }
}

filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:ts} %{LOGLEVEL:level} %{GREEDYDATA:msg}" }
    # Bail fast on non-matching lines instead of backtracking forever
    timeout_millis => 500
  }
  date { match => [ "ts", "ISO8601" ] }
}

output {
  elasticsearch {
    hosts => ["http://es01:9200","http://es02:9200"]
    index => "app-logs-%{+YYYY.MM.dd}"
  }
}

Pipeline throughput lives in /etc/logstash/pipelines.yml (or logstash.yml) — raise workers and batch size so the filter stage drains the beats input:

- pipeline.id: beats
  path.config: "/etc/logstash/conf.d/beats-in.conf"
  pipeline.workers: 8          # default = number of CPU cores
  pipeline.batch.size: 250     # larger batches = fewer, fatter bulk writes
  pipeline.batch.delay: 50

Confirm the output is not the real blocker — check Elasticsearch health and watch for 429s:

curl -s 'localhost:9200/_cluster/health?pretty' | grep status
grep -i 'retrying failed action\|429\|bulk' /var/log/logstash/logstash-plain.log

On the Filebeat side, spread agents across multiple Logstash nodes and enable load balancing so no single input is a choke point:

output.logstash:
  hosts: ["logstash01:5044", "logstash02:5044"]
  loadbalance: true
  ttl: 30s          # cycle connections so they rebalance

Example Root Cause Analysis

A platform team ran a single Logstash node fronting about 220 Filebeat agents. During each morning traffic ramp, Filebeat hosts began logging i/o timeout and Logstash filled with the pipeline is blocked, temporary refusing new connection. Ingest lag grew to 15 minutes.

The instinct was to blame the beats input, so they raised executor_threads — which helped only marginally. The _node/stats/pipelines API told the real story: the elasticsearch output’s duration_in_millis dwarfed every other stage, and logstash-plain.log showed periodic retrying failed action with response code: 429. The Elasticsearch cluster was rejecting bulk requests because its write thread pool was saturated — a two-data-node cluster taking the full 220-agent firehose. The output blocked, which blocked the pipeline, which forced the beats input to refuse Filebeat connections.

Two changes fixed it. First, they scaled the ingest tier: a second Logstash node plus loadbalance: true on Filebeat spread connections across both inputs, halving the per-node pressure. Second, they raised pipeline.workers to match the (now larger) core count and increased pipeline.batch.size to 250 so each bulk write to Elasticsearch was fatter and fewer. They also added a third Elasticsearch data node to absorb the bulk load. The temporary refusing new connection lines disappeared, Filebeat stopped backing off, and lag returned to seconds. The lesson: the refusal is an input-side symptom of downstream backpressure — fix what the pipeline is waiting on, then give the input room to breathe.

Prevention Best Practices

• Never point a fleet at a single beats input. Run at least two Logstash nodes and set loadbalance: true with a ttl on the Filebeat output.logstash so connections rebalance.
• Size pipeline.workers to CPU cores and tune pipeline.batch.size so the filter/output stage can drain the input at your real event rate.
• Watch the output first. Most “input blocked” incidents are actually a slow or 429-throttled Elasticsearch — monitor cluster health and bulk rejections.
• Guard expensive filters. Add timeout_millis to grok, avoid synchronous dns/ruby filters on the hot path, and anchor patterns to prevent catastrophic backtracking.
• Raise executor_threads on the beats input when hundreds of agents connect, and set client_inactivity_timeout so dead sockets release their slot.
• Give Logstash headroom — adequate CPU and a right-sized JVM heap to avoid GC pauses that periodically freeze the pipeline.
• Alert on the refusal line in logstash-plain.log; a rising rate is an early warning that ingest capacity is running out before Filebeat data starts dropping.

Quick Command Reference

# Validate pipeline syntax before restarting
bin/logstash --config.test_and_exit --path.settings /etc/logstash

# Find the slow stage (highest duration_in_millis)
curl -s 'localhost:9600/_node/stats/pipelines?pretty' | grep -B2 -A4 duration_in_millis

# Watch for the refusal and output-throttling messages
grep -i 'refusing new connection\|pipeline is blocked\|429\|retrying failed action' \
  /var/log/logstash/logstash-plain.log

# Is Elasticsearch the real bottleneck?
curl -s 'localhost:9200/_cluster/health?pretty' | grep status
curl -s 'localhost:9200/_nodes/stats/thread_pool/write?pretty' | grep -A3 rejected

# Check what Filebeat sees (on the shipper host)
journalctl -u filebeat --since '10 min ago' | grep -i 'backoff\|i/o timeout\|reconnect'

# Confirm the beats port is listening
ss -ltnp | grep 5044

Conclusion

the pipeline is blocked, temporary refusing new connection is the beats input applying flow control: it refuses new Filebeat connections because it cannot push the data it holds into a stalled pipeline. Treat it as a symptom on the ingest edge, not a fault in the input. The refusal almost always traces back to a slow output (a 429-throttled or undersized Elasticsearch), an under-tuned filter stage, or a single node absorbing too many agents. Use the _node/stats/pipelines API to find the slow stage, validate config with --config.test_and_exit, tune pipeline.workers/pipeline.batch.size so the filter stage keeps up, and spread Beats across multiple nodes with loadbalance: true. Relieve what the pipeline is waiting on and the input stops turning Filebeat away. For more ingest and pipeline fixes, browse the Logstash category.