🚨 Logstash Error Guide: Beats Input 'Connection reset by peer' — Stabilize Filebeat

Overview

The Logstash beats input (port 5044 by default) speaks the Lumberjack protocol to Filebeat, Metricbeat, and the other Beats shippers. When that TCP connection is torn down mid-flight, Filebeat logs a connection reset by peer error, stops shipping for a moment, and retries. On the Logstash side you often see the mirror image — a client that disappeared or a pipeline that could not keep up.

The Filebeat error looks like this:

ERROR   [publisher_pipeline_output] pipeline/output.go   failed to publish events:
  write tcp 10.0.1.20:52344->10.0.1.9:5044: write: connection reset by peer
INFO    [publisher_pipeline_output] pipeline/output.go   Connecting to backoff(async(tcp://logstash:5044))

On the Logstash node, logstash-plain.log frequently shows the input logging the peer going away, or the pipeline signalling back-pressure:

[WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired,
  io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
[INFO ][org.logstash.beats.BeatsHandler] [local: 10.0.1.9:5044, remote: 10.0.1.20:52344]
  Handling exception: Connection reset by peer

Connection reset by peer means one side sent a TCP RST — an abrupt teardown rather than a graceful FIN. The important insight is that this is almost never a network cabling problem. In a Beats-to-Logstash setup, the reset is usually a symptom of back-pressure: Logstash’s pipeline is full, the beats input stops reading from the socket, an idle or timeout threshold trips, and the connection is killed. Fixing it means finding what is stalling the pipeline, not blaming the wire.

Symptoms

• Filebeat logs repeated write: connection reset by peer errors followed by Connecting to backoff(...) reconnection attempts.
• Log delivery is bursty or laggy — events arrive in Elasticsearch in clumps with gaps, and Filebeat’s libbeat.output.events.dropped/acked metrics show stalls.
• Logstash’s logstash-plain.log shows Netty Connection reset by peer exceptions from org.logstash.beats.BeatsHandler or io.netty on the input.
• The Logstash node shows sustained high CPU, a full pipeline queue, or GC pauses at the moments Filebeat resets.
• The problem worsens under load — a log burst (deploy, incident, batch job) triggers a wave of resets, then it settles when volume drops.
• TLS-enabled setups fail immediately and consistently at connection time rather than intermittently under load (a different, configuration-based variant).

Common Root Causes

• Pipeline back-pressure — the most common cause. Logstash’s downstream output (usually Elasticsearch) is slow or rejecting bulk requests, the internal queue fills, the beats input stops reading, and the connection times out and resets. The reset is a downstream problem surfacing at the input.
• Filebeat client_inactivity_timeout too low — the beats input closes connections idle longer than this timeout (default 60s). If Filebeat’s keep_alive/publish cadence is slower than the timeout, Logstash resets an otherwise-healthy connection.
• Congestion vs. slow_start on large batches — very large bulk_max_size from Filebeat combined with slow Elasticsearch causes the connection to stall past timeouts.
• TLS/SSL mismatch — one side has TLS enabled and the other does not, or certificates/CA do not match, producing immediate resets at handshake time.
• Logstash restart or reload — during a config reload or restart, existing beats connections are dropped with an RST; Filebeat logs the reset and reconnects (benign but noisy).
• JVM heap pressure / long GC pauses — a Logstash node starved of heap pauses long enough that the beats input becomes unresponsive and connections reset.
• Network middleboxes — a load balancer or firewall with an aggressive idle-connection timeout silently drops long-lived Lumberjack connections, appearing as resets to both ends.

Diagnostic Workflow

Always begin by confirming the Logstash config is valid, because a half-broken pipeline reload is itself a source of resets:

# Validate the beats pipeline before restarting anything
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/beats.conf --config.test_and_exit

# Shorthand
/usr/share/logstash/bin/logstash -t -f /etc/logstash/conf.d/beats.conf

Here is a realistic, robust beats input pipeline. The commented settings are the ones that matter for reset problems:

# /etc/logstash/conf.d/beats.conf
input {
  beats {
    port => 5044
    # Raise this if Filebeat resets appear on idle/low-traffic connections.
    # Must be comfortably larger than Filebeat's publish interval.
    client_inactivity_timeout => 120

    # Enable TLS ONLY if Filebeat also has ssl.enabled: true with a matching CA.
    # ssl => true
    # ssl_certificate => "/etc/logstash/certs/logstash.crt"
    # ssl_key => "/etc/logstash/certs/logstash.key"
  }
}

filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:ts} %{LOGLEVEL:level} %{GREEDYDATA:msg}" }
  }
}

output {
  elasticsearch {
    hosts => ["http://es01:9200", "http://es02:9200"]
    # A slow/rejecting ES here is the usual root of back-pressure resets.
  }
}

Confirm the beats input is actually listening and accepting connections:

# Is Logstash listening on 5044?
sudo ss -ltnp 'sport = :5044'

# Can the Filebeat host reach it? (run from the Filebeat node)
nc -zv logstash 5044

Now check for back-pressure, the number-one cause. Query Logstash’s monitoring API for pipeline and queue health:

# Overall pipeline events, plus flow rates
curl -s localhost:9600/_node/stats/pipelines?pretty | grep -A6 '"events"'

# Look for a full queue and rising back-pressure
curl -s localhost:9600/_node/stats?pretty | grep -iE 'queue|backpressure|inflight'

If the output plugin (Elasticsearch) is the bottleneck, you will see the input’s event rate throttled and Elasticsearch bulk rejections. Correlate the timestamps of Filebeat resets with Logstash pipeline stalls and JVM GC pauses:

# Correlate resets with Logstash-side events and GC
sudo grep -iE 'connection reset|BeatsHandler|GC ' /var/log/logstash/logstash-plain.log | tail -50
sudo journalctl -u logstash --since '15 min ago' | grep -iE 'reset|pipeline|OutOfMemory'

# On the Filebeat host, watch the reset/reconnect cycle
sudo journalctl -u filebeat -f | grep -iE 'reset|backoff|publish'

For a persistent-queue setup, verify the queue is draining and not wedged:

curl -s localhost:9600/_node/stats/pipelines?pretty | grep -A8 '"queue"'

Example Root Cause Analysis

During a nightly batch job, a fleet of 40 Filebeat agents suddenly logs write: connection reset by peer in unison, and log delivery to Kibana stalls for several minutes before catching up. The network team confirms no packet loss between the hosts and Logstash, so attention turns to the pipeline.

Checking localhost:9600/_node/stats on the Logstash node during the incident shows the pipeline inflight count pinned at its maximum and the Elasticsearch output logging bulk rejections:

[WARN ][logstash.outputs.elasticsearch] Received a 429 (Too Many Requests) from Elasticsearch,
  will retry. {:batch_size=>500}

The chain is now clear. The batch job produced a burst of logs. Elasticsearch’s bulk queue filled and started returning HTTP 429. Logstash’s Elasticsearch output retried, its internal pipeline queue filled, and the beats input stopped reading from the sockets. With the input no longer reading, Filebeat’s writes stalled past client_inactivity_timeout, Logstash reset the idle connections, and all 40 agents reconnected at once — amplifying the problem.

The fix is applied in two layers. First, absorb bursts on the Logstash side by enabling a persistent queue so the input keeps reading even when the output is slow:

# /etc/logstash/logstash.yml
queue.type: persisted
queue.max_bytes: 4gb

Second, raise the idle timeout so momentary output slowness does not sever healthy connections, and let Filebeat’s built-in backoff smooth reconnections:

input {
  beats {
    port => 5044
    client_inactivity_timeout => 240
  }
}

# filebeat.yml — congestion-aware output
output.logstash:
  hosts: ["logstash:5044"]
  bulk_max_size: 1024
  ttl: 0                 # keep connections persistent
  slow_start: true       # ramp batch size up gradually after (re)connect

After the change, the next batch run produces no resets: the persistent queue buffers the burst, Elasticsearch drains it steadily, and connections stay up. The root cause was never the network — it was downstream back-pressure surfacing as connection resets at the beats input.

Prevention Best Practices

• Treat resets as a back-pressure signal. Before touching timeouts, check whether the Elasticsearch (or other) output is the true bottleneck via the _node/stats API and ES bulk-rejection logs.
• Enable a persistent queue (queue.type: persisted) so the beats input keeps reading during output slowdowns and absorbs bursts instead of resetting connections.
• Set client_inactivity_timeout comfortably above Filebeat’s publish cadence so healthy, low-traffic connections are never reaped as idle.
• Use Filebeat slow_start: true and a sane bulk_max_size so large batches ramp up rather than overwhelming a recovering pipeline.
• Keep TLS symmetric — if the beats input has ssl => true, every Filebeat must set ssl.enabled: true with a matching CA; otherwise connections reset at handshake.
• Right-size and monitor JVM heap so GC pauses never make the beats input unresponsive; alert on sustained high heap and long GC times.
• Check middlebox idle timeouts. If a load balancer sits between Filebeat and Logstash, set its TCP idle timeout longer than client_inactivity_timeout.
• Expect resets during Logstash restarts/reloads and treat a brief burst of them as benign — Filebeat’s at-least-once delivery and backoff will recover without data loss.

Quick Command Reference

# Validate the beats pipeline config
bin/logstash -t -f /etc/logstash/conf.d/beats.conf

# Confirm Logstash is listening on the beats port
sudo ss -ltnp 'sport = :5044'

# Reachability from the Filebeat host
nc -zv logstash 5044

# Inspect pipeline / queue / back-pressure
curl -s localhost:9600/_node/stats/pipelines?pretty
curl -s localhost:9600/_node/stats?pretty | grep -iE 'queue|inflight|backpressure'

# Correlate resets on both ends
sudo grep -iE 'connection reset|BeatsHandler' /var/log/logstash/logstash-plain.log | tail
sudo journalctl -u filebeat -f | grep -iE 'reset|backoff'

# Validate Filebeat's own config
sudo filebeat test config
sudo filebeat test output

Conclusion

Connection reset by peer between Filebeat and the Logstash beats input almost always tells you something downstream is stalling, not that the network is broken. The reset is the last link in a chain: a slow or rejecting output fills Logstash’s queue, the beats input stops reading, idle timeouts trip, and connections are severed. The durable fix is to relieve the pressure — enable a persistent queue to absorb bursts, right-size the Elasticsearch output, set client_inactivity_timeout above Filebeat’s publish interval, and let Filebeat’s slow_start and backoff smooth reconnections. Verify TLS is symmetric on both ends, watch JVM heap and middlebox idle timeouts, and use the _node/stats API to prove where the bottleneck really is before you change a single setting. Do that, and the resets stop — because you have fixed the cause, not the symptom.