🚨 Filebeat Error Guide: '503 Service Unavailable' — Recover a Blocked Output

Overview

Filebeat logs this when a _bulk request returns HTTP 503 from Elasticsearch or a proxy in front of it. libbeat treats 503 as temporary and retries with backoff, so it repeats while the cluster is unavailable:

Failed to perform any bulk index operations: 503 Service Unavailable: {"error":{"root_cause":[{"type":"cluster_block_exception","reason":"blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];"}]}}

A 503 means Elasticsearch (or a load balancer) accepted the connection but cannot serve the request right now: the cluster state is not recovered, primary shards are unassigned, an index is blocked, or a proxy has no healthy upstream. Unlike a mapping error, this is transient by nature — Filebeat keeps the events queued and retries until the service returns. The risk is a prolonged 503 filling the queue and backing up harvesters.

Symptoms

• Repeated bulk 503 Service Unavailable in the Filebeat log, often with cluster_block_exception or master_not_discovered_exception in the body.
• curl to _cluster/health returns 503 or a red status.
• Ingestion stops entirely across all sources at once (unlike a single-index mapping error).
• Filebeat’s queue fills; you may see queue is full and paused harvesters.
• The event burst coincides with a cluster restart, master election, or proxy failover.

Common Root Causes

• Cluster still recovering — after a restart, state is not recovered / initialized until enough nodes join and shards allocate.
• No elected master — a quorum loss leaves the cluster unable to serve writes (master_not_discovered_exception).
• Red index — unassigned primary shards for the target data stream make writes impossible.
• Index write block — a read_only_allow_delete block from a disk watermark returns 503-style rejections.
• Proxy / load balancer — an LB in front of Elasticsearch returns 503 when its backend pool is empty or failing health checks.
• Rolling upgrade — nodes cycling out temporarily reduce capacity below quorum.

Diagnostic Workflow

Check whether the cluster itself is healthy and has a master:

curl -sk https://es01:9200/_cluster/health?pretty -u elastic:$ES_PASS
curl -sk https://es01:9200/_cat/master?v -u elastic:$ES_PASS

If health is red, find the unassigned shards and why:

curl -sk 'https://es01:9200/_cat/shards?v&h=index,shard,state,unassigned.reason' \
  -u elastic:$ES_PASS | grep -v STARTED
curl -sk 'https://es01:9200/_cluster/allocation/explain?pretty' -u elastic:$ES_PASS

Reproduce the write path exactly as Filebeat does:

curl -sk -o /dev/null -w '%{http_code}\n' -X POST -u elastic:$ES_PASS \
  -H 'Content-Type: application/x-ndjson' 'https://es01:9200/_bulk' \
  --data-binary $'{"index":{"_index":"filebeat-probe"}}\n{"message":"probe"}\n'

Check for a disk-watermark write block if 503 comes with read_only_allow_delete:

curl -sk 'https://es01:9200/_cat/allocation?v' -u elastic:$ES_PASS

Example Root Cause Analysis

Filebeat across a fleet began logging bulk 503 Service Unavailable with cluster_block_exception ... state not recovered / initialized. The team had rebooted the Elasticsearch nodes for kernel patching; two of three master-eligible nodes came back, but a network ACL blocked the third’s transport port, so the cluster could not form a stable quorum and stayed in the not-recovered state.

_cat/master returned empty, confirming no elected master. Restoring the transport port (9300) let the third node rejoin; within a minute _cluster/health went green and the manual _bulk probe returned 200. Filebeat’s backoff retries then drained the queued events automatically — no config change was needed because the offsets were preserved in the registry. The lesson: a fleet-wide 503 is almost always a cluster-state problem, not a Filebeat one.

Prevention Best Practices

• Run three master-eligible nodes and verify quorum settings so a single node loss never blocks writes.
• Monitor _cluster/health and alert on red/yellow and on _cat/master being empty before Filebeat backs up.
• Enforce disk watermarks and alert well before the flood stage so read_only_allow_delete blocks never trigger.
• Size the Filebeat queue and consider a Kafka/Logstash buffer for long cluster outages so harvesters do not stall.
• Perform rolling restarts with allocation disabled/enabled correctly to keep the cluster serving throughout.

Quick Command Reference

curl -sk https://es01:9200/_cluster/health?pretty -u elastic:$ES_PASS
curl -sk https://es01:9200/_cat/master?v -u elastic:$ES_PASS
curl -sk 'https://es01:9200/_cat/shards?v&h=index,shard,state,unassigned.reason' -u elastic:$ES_PASS
curl -sk 'https://es01:9200/_cluster/allocation/explain?pretty' -u elastic:$ES_PASS
journalctl -u filebeat -f | grep -i 503

Conclusion

A bulk 503 Service Unavailable is Elasticsearch telling you it cannot serve writes yet — recovering state, no master, red shards, or a write block. Filebeat is doing the right thing by retrying, so the fix is entirely cluster-side: restore quorum, allocate shards, or clear the write block, and the queued events flush on their own. Keep an eye on cluster health so you catch it before the queue fills. More recovery guides live in the Filebeat guides.