🚨 Filebeat Error Guide: 'dial tcp: i/o timeout' — Fix a Silently Dropped Output

Overview

Filebeat cannot complete the TCP handshake to its Elasticsearch output within the dial timeout, so the connection attempt times out and the publisher backs off:

Failed to connect to backoff(elasticsearch(https://es01:9200)): Get "https://es01:9200": dial tcp 10.0.3.14:9200: i/o timeout

An i/o timeout is fundamentally different from connection refused. Refused means a host actively rejected the packet (RST); a timeout means the SYN went out and nothing came back — the packets are being silently dropped. That points at a firewall/security-group DROP rule, a black-holed route, a wrong IP, or an overloaded/unreachable node, not at Elasticsearch answering “no.” Filebeat keeps events queued and retries, so data is safe, but ingestion halts until the path is restored.

Symptoms

• Repeated dial tcp ...: i/o timeout in the Filebeat log with growing backoff.
• filebeat test output hangs for several seconds, then fails at the connection... step.
• nc -vz / curl to the ES host also hang and time out (rather than failing instantly like “refused”).
• No documents index; the queue fills over time.
• Timeouts appear only from certain hosts/subnets, implicating network policy rather than the cluster.

Common Root Causes

• Firewall / security group DROP — a rule silently discards packets to the ES port from the Filebeat subnet.
• Wrong or stale IP — DNS resolves to an address that no longer hosts Elasticsearch, so packets go nowhere.
• Routing black hole — a missing route or NAT misconfiguration between subnets/VPCs.
• Overloaded node — an ES node so saturated it cannot complete the handshake in time.
• MTU / MSS mismatch — path MTU issues on VPN/overlay networks cause handshakes to stall.
• Down node behind a VIP — the load balancer forwards to a dead backend that never answers.

Diagnostic Workflow

Confirm the timeout is at connect time and time it:

filebeat test output
time nc -vz es01 9200          # hangs then fails on a drop; instant on a refuse

Resolve the name and try the raw TCP path with an explicit timeout:

getent hosts es01
timeout 5 bash -c 'cat < /dev/null > /dev/tcp/es01/9200' && echo open || echo blocked

Curl with a connect timeout to see where it stalls:

curl -v --connect-timeout 5 https://es01:9200 -u elastic:$ES_PASS

Trace the path to spot where packets die, and check MTU on overlay links:

traceroute -T -p 9200 es01
ping -M do -s 1472 es01        # DF-bit ping to detect MTU black holes

From the ES side, confirm it is listening and not simply overwhelmed:

ss -ltnp | grep 9200
curl -s localhost:9200/_cluster/health?pretty

Example Root Cause Analysis

After a VPC network policy change, one subnet of Filebeat agents began logging dial tcp 10.0.3.14:9200: i/o timeout, while agents in another subnet indexed fine. The instant-vs-hang distinction was the clue: nc -vz hung for the full timeout rather than failing immediately, meaning packets were dropped, not refused.

traceroute -T -p 9200 es01 from the affected subnet died after the last internal hop, and the ES node’s ss -ltnp confirmed it was still listening and healthy — so the cluster was fine. A new security-group rule had omitted the affected subnet’s CIDR from the allow list for port 9200. Adding 10.0.3.0/24 to the security group restored the path; nc -vz then connected instantly and Filebeat drained its queue via backoff retries. No Filebeat config changed — this was purely a network-policy fix.

Prevention Best Practices

• Distinguish timeout (dropped) from refused (rejected) early — it immediately tells you firewall/routing vs. wrong-port/down-service.
• Keep security-group and firewall allow lists in configuration management so subnet additions never miss the ES port.
• Use stable DNS or a load balancer for the output so a node IP change does not black-hole traffic.
• Monitor path reachability (synthetic nc/curl checks) from each Filebeat subnet to the cluster.
• On overlay/VPN networks, standardize MTU and test with DF-bit pings to avoid handshake black holes.

Quick Command Reference

filebeat test output
time nc -vz es01 9200                       # hang=drop, instant fail=refuse
curl -v --connect-timeout 5 https://es01:9200 -u elastic:$ES_PASS
traceroute -T -p 9200 es01
ping -M do -s 1472 es01                      # MTU black-hole check
ss -ltnp | grep 9200                         # on the ES node

Conclusion

dial tcp: i/o timeout means Filebeat’s connection packets vanished with no reply — the hallmark of a firewall DROP, a routing black hole, a stale IP, or an unreachable node, not a service that said no. Use the hang-vs-instant behavior of nc and a -T traceroute to locate where packets die, fix the network path, and Filebeat’s retries flush the backlog. More connectivity fixes are in the Filebeat guides.