🚨 Filebeat Error: 'Failed to publish events caused by: read tcp ... i/o timeout' — Cause, Fix, and Troubleshooting Guide

Overview

This timeout happens after the connection is established: Filebeat sent a bulk request and then waited longer than the output timeout for the response before the read gave up:

Failed to publish events caused by: read tcp 10.0.4.8:52210->10.0.3.14:9200: i/o timeout

It is distinct from a dial tcp ... i/o timeout, which fails during connection setup. Here the TCP connection is up and the request was delivered — Elasticsearch (or a proxy/load balancer in between) is just too slow to answer within the deadline. The usual causes are an overloaded cluster, oversized bulk requests that take too long to process, or an idle/timeout on an intermediary. Filebeat re-queues the batch and retries, so events are not lost, but throughput stalls.

Symptoms

• Failed to publish events caused by: read tcp ...->...:9200: i/o timeout.
• The connection succeeds (no dial error) but responses time out.
• libbeat.output.events.retry climbs; throughput sawtooths.
• Elasticsearch is slow: high write-queue, GC pauses, or hot data nodes at the same timestamps.

Common Root Causes

• Output timeout too low for a busy cluster (default 90s, but proxies may cut it shorter).
• Oversized bulk_max_size — large requests take longer than the deadline to process.
• Overloaded Elasticsearch — write thread pool saturated, slow to respond.
• Load balancer / proxy idle timeout shorter than the bulk processing time, cutting the read.
• GC or merge pauses on hot data nodes stalling responses.

How to diagnose

Correlate Filebeat retries with cluster load:

journalctl -u filebeat -f | grep -i 'read tcp\|i/o timeout\|retry'
curl -sk 'https://es01:9200/_cat/thread_pool/write?v&h=node_name,active,queue,rejected' \
  -u elastic:$ES_PASS
curl -sk 'https://es01:9200/_nodes/stats/jvm?filter_path=nodes.*.jvm.gc' -u elastic:$ES_PASS

Check for a proxy/LB between Filebeat and ES with a short idle timeout:

grep -A4 'output.elasticsearch' /etc/filebeat/filebeat.yml

Fixes

Give slow responses more room and send smaller bulk requests so each completes faster:

output.elasticsearch:
  hosts: ["https://es01:9200"]
  timeout: 120              # seconds to wait for a bulk response (default 90)
  bulk_max_size: 800        # smaller requests process faster, beat the deadline
  worker: 1
  backoff.init: 1s
  backoff.max: 60s

If a load balancer sits in the path, raise its idle/read timeout above the expected bulk processing time so it does not sever the connection. The durable fix, though, is cluster capacity — relieve the write thread pool and GC pressure by adding hot nodes or reducing shard/merge load. Restart Filebeat after tuning and confirm retries stop climbing.

What to watch out for

• read tcp ... i/o timeout is a response timeout, not a connection failure — don’t chase DNS/firewall.
• Raising timeout without shrinking bulk_max_size just delays the same failure under load.
• A proxy with a short idle timeout can cause this even when ES itself is healthy.
• Events are retried, not dropped; persistent timeouts mean genuine cluster slowness.

Related

• Filebeat Error Guide: ‘dial tcp: i/o timeout’
• Filebeat Error Guide: ‘temporary bulk send failure’
• Filebeat Error Guide: ‘429 Too Many Requests’