🚨 Logstash Error Guide: 'Elasticsearch Unreachable' — Restore the Output Connection

Overview

The Logstash elasticsearch output pools connections to each configured host and health-checks them. When it cannot reach a node, it marks the connection dead and logs a resurrection attempt:

[WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES
instance, but got an error {:url=>"http://es1:9200/",
:exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError,
:message=>"Elasticsearch Unreachable: [http://es1:9200/][Manticore::SocketException]
Connection refused (Connection refused)"}

Until at least one host resurrects, the output cannot flush. Events back up in the queue and the whole pipeline blocks. “Unreachable” is a transport-layer failure — DNS, TCP, TLS, or the node being down — not a rejection of the data itself.

Symptoms

• Repeated Attempted to resurrect connection to dead ES instance / Elasticsearch Unreachable warnings.
• Connection refused, Connection timed out, No route to host, or Name or service not known in the exception.
• Ingest stops entirely (single host) or degrades (some hosts dead); the queue grows.
• After ES restarts, a cluster upgrade, or a network blip, Logstash keeps retrying until the node returns.
• With TLS, the error is instead an SSL handshake failure (a related but distinct problem).

Common Root Causes

• Elasticsearch is down or restarting — a node crash, rolling upgrade, or OOM took the target offline.
• Wrong host/port — a typo in hosts, the wrong port (9300 transport vs 9200 HTTP), or an internal name that does not resolve.
• DNS failure — the ES hostname does not resolve from the Logstash host (Name or service not known).
• Firewall / security group — port 9200 blocked between Logstash and Elasticsearch.
• Authentication failure surfacing as unreachable — with security enabled, missing/expired credentials can prevent the healthcheck from succeeding.
• Cluster overloaded — ES is up but so busy it drops the healthcheck connection.

Diagnostic Workflow

Prove basic reachability from the Logstash host itself — this isolates network from Logstash config:

getent hosts es1                          # DNS resolves?
nc -vz es1 9200                           # TCP port open?
curl -s -o /dev/null -w '%{http_code}\n' http://es1:9200/   # HTTP responds?

Hit the Elasticsearch health endpoint with the same credentials Logstash uses:

curl -s -u logstash_writer:$PW 'http://es1:9200/_cluster/health?pretty'
curl -s 'http://es1:9200/_cat/nodes?v'

Confirm the output’s configured hosts match reality:

output {
  elasticsearch {
    hosts => ["http://es1:9200","http://es2:9200","http://es3:9200"]
    user  => "logstash_writer"
    password => "${ES_PW}"
    # sniffing => false when hosts are behind a load balancer / NAT
  }
}

Check the Logstash monitoring API for which output connections are alive:

curl -s localhost:9600/_node/stats/pipelines?pretty | grep -A10 elasticsearch

Verify the credentials env var is actually set for the service:

systemctl show logstash -p Environment | tr ' ' '\n' | grep ES_PW
sudo -u logstash bash -c '[ -n "$ES_PW" ] && echo set || echo MISSING'

Watch the log for recovery after fixing the root cause:

tail -f /var/log/logstash/logstash-plain.log | grep -Ei 'unreachable|resurrect|Restored'

You want to see the Restored connection to ES instance line, which confirms the output is healthy again.

Example Root Cause Analysis

After migrating Elasticsearch behind an internal load balancer, Logstash began logging Elasticsearch Unreachable: Connection refused for http://es1:9200 continuously, even though the cluster was healthy. nc -vz es1 9200 from the Logstash host failed, but nc -vz es-lb 9200 succeeded — the old per-node hostname es1 still pointed at a decommissioned IP.

The output still listed the individual old nodes in hosts and had sniffing => true, so even the load-balancer address was being replaced by sniffed (now-dead) node addresses. The fix was to point hosts at the load balancer only and set sniffing => false (sniffing bypasses the LB by discovering direct node addresses). After a SIGHUP reload, the log showed Restored connection to ES instance and the backed-up queue drained.

Prevention Best Practices

• List multiple hosts for redundancy so one dead node does not stop ingest; the output round-robins over the live ones.
• Behind a load balancer or NAT, set sniffing => false so Logstash talks to the LB instead of discovering unreachable direct node addresses.
• Use stable DNS names or service discovery for ES endpoints and validate resolution from the Logstash host in monitoring.
• Supply credentials via environment variables/secret stores and assert they are non-empty at startup.
• Alert on the Elasticsearch Unreachable / resurrect log pattern and on queue growth so a downstream outage is noticed immediately.
• Confirm firewall/security-group rules allow Logstash → ES on 9200 (HTTP) — not 9300 (transport) — in every environment.

Quick Command Reference

# Layered reachability check from the Logstash host
getent hosts es1
nc -vz es1 9200
curl -s -o /dev/null -w '%{http_code}\n' http://es1:9200/

# Cluster health with Logstash's credentials
curl -s -u logstash_writer:$ES_PW 'http://es1:9200/_cluster/health?pretty'

# Which ES connections does the output see?
curl -s localhost:9600/_node/stats/pipelines?pretty | grep -A10 elasticsearch

# Watch for recovery
tail -f /var/log/logstash/logstash-plain.log | grep -Ei 'unreachable|Restored'

Conclusion

Elasticsearch Unreachable is a transport-layer failure between Logstash and its output — DNS, TCP, TLS, or the node simply being down — not a rejection of your events. Diagnose it from the outside in: resolve the name, test the TCP port, then curl the cluster health with the same credentials Logstash uses. Common fixes are correcting hosts, disabling sniffing behind a load balancer, opening port 9200, and supplying valid credentials. List multiple hosts and alert on the resurrection log pattern so a downstream ES hiccup never quietly stalls your pipeline.