🚨 Filebeat Error Guide: 'connection reset by peer' to Logstash — Stabilize the Beats Output

Overview

When Filebeat ships to a logstash output, it opens a persistent Lumberjack (Beats protocol) connection. If Logstash forcibly closes that socket, Filebeat logs a reset and reconnects with backoff:

Failed to connect to backoff(async(tcp://logstash01:5044)): read tcp 10.0.4.8:52344->10.0.4.20:5044: read: connection reset by peer

connection reset by peer is a TCP RST from Logstash’s side — the remote closed the socket abruptly instead of a graceful shutdown. Unlike connection refused (nothing listening), the connection was established and then torn down: the Beats input restarted, TLS was misconfigured, a proxy idle-timeout fired, or Logstash’s pipeline stalled under backpressure and dropped the client. Filebeat retries and preserves offsets, so events are safe, but throughput suffers while the channel flaps.

Symptoms

• Repeated connection reset by peer (or read tcp ...: i/o timeout) against the Logstash 5044 output.
• Filebeat cycles through connect / publish / reset / reconnect in the log.
• Logstash logs Beats input warnings, TLS handshake errors, or pipeline backpressure around the same time.
• Ingest throughput is bursty; events arrive in Elasticsearch in irregular batches.
• Resets increase during traffic spikes or right after a Logstash config reload.

Common Root Causes

• Logstash pipeline backpressure — a slow filter/output stalls the pipeline, so the Beats input stops reading and the connection is reset.
• Beats input restart — a Logstash config reload restarts the beats input, dropping active connections.
• TLS mismatch — one side expects TLS and the other does not, or client-auth (ssl_verify_mode) settings disagree.
• Proxy / LB idle timeout — an intermediary resets long-lived Lumberjack connections after an idle period.
• Version/protocol skew — incompatible Beats protocol versions between Filebeat and the Logstash beats input.
• Resource exhaustion — Logstash hitting file-descriptor or heap limits closes connections.

Diagnostic Workflow

Test the Filebeat output to confirm it targets the right host, port, and TLS mode:

filebeat test output
grep -A8 'output.logstash' /etc/filebeat/filebeat.yml

Verify raw reachability and that Logstash is listening on 5044:

nc -vz logstash01 5044
ss -ltnp | grep 5044        # run on the Logstash host

Inspect Logstash for backpressure and Beats input errors:

journalctl -u logstash --since '15 min ago' | grep -iE 'beats|reset|backpressure|pipeline'
curl -s localhost:9600/_node/stats/pipelines?pretty | grep -A4 '"events"'

If TLS is involved, confirm both ends agree; a plaintext client to a TLS input resets immediately:

openssl s_client -connect logstash01:5044 </dev/null 2>&1 | head

Example Root Cause Analysis

Filebeat agents intermittently logged connection reset by peer to Logstash, worst during peak hours. curl localhost:9600/_node/stats/pipelines on Logstash showed the pipeline’s queue events climbing and the beats input pausing — a downstream Elasticsearch output was slow, creating backpressure that propagated back to the Beats input, which then dropped clients to shed load.

Because Filebeat safely retries, the durable fix was to relieve the downstream stall and add congestion tolerance rather than chase the resets. They increased the persistent queue and Elasticsearch output workers in Logstash, and on the Filebeat side spread load across multiple Logstash hosts with load balancing:

output.logstash:
  hosts: ["logstash01:5044", "logstash02:5044"]
  loadbalance: true
  ttl: 30s          # recycle connections so no single host is pinned

After giving the Logstash pipeline headroom and load-balancing the Beats connections, the resets dropped to near zero and throughput smoothed out.

Prevention Best Practices

• Give Logstash a persistent queue and enough output workers so pipeline backpressure does not force the Beats input to drop clients.
• Load-balance Filebeat across multiple Logstash hosts (loadbalance: true) and set a ttl so connections recycle.
• Keep TLS settings symmetric — if the Beats input uses ssl, Filebeat must use ssl.enabled: true with the matching CA.
• Raise idle timeouts on any proxy/LB carrying Lumberjack traffic, or bypass the proxy for long-lived Beats connections.
• Monitor Logstash pipeline stats (/_node/stats/pipelines) and Filebeat reconnect metrics to catch flapping early.

Quick Command Reference

filebeat test output
nc -vz logstash01 5044
ss -ltnp | grep 5044                                  # on Logstash host
journalctl -u logstash --since '15 min ago' | grep -i beats
curl -s localhost:9600/_node/stats/pipelines?pretty   # Logstash backpressure
openssl s_client -connect logstash01:5044 </dev/null  # TLS sanity

Conclusion

connection reset by peer to Logstash means the connection was made and then torn down — most often Logstash pipeline backpressure, a config-reload restart, or a TLS/proxy mismatch. Confirm reachability, check Logstash pipeline stats for stalls, and relieve backpressure while load-balancing Filebeat across multiple Logstash hosts. Filebeat’s retries keep data safe; your job is to stop the channel from flapping. More output fixes are in the Filebeat guides.