🚨 Filebeat Error Guide: 'temporary bulk send failure' — Stabilize the Bulk Pipeline

Overview

Filebeat emits this when a _bulk request to Elasticsearch does not complete cleanly and the publisher decides the failure is retryable. The events are not dropped — they are re-queued and resent — but the message repeats whenever the output is unhealthy:

Failed to publish events: temporary bulk send failure

This is a deliberately generic wrapper. Filebeat batches events and POSTs them to _bulk; if the whole request errors (connection reset, timeout, 5xx) or the response reports items it could not index for a retryable reason, libbeat logs temporary bulk send failure and schedules a retry with backoff. It means the transport or the cluster hiccuped, not that your event data is malformed. Persistent repetition, however, points at a saturated cluster or a systematic rejection hiding behind the generic text.

Symptoms

• Recurring Failed to publish events: temporary bulk send failure in the Filebeat log, often in bursts.
• Ingestion throughput sawtooths — documents arrive, stall, then catch up as retries succeed.
• Elasticsearch logs show bulk rejections, EsRejectedExecutionException, or thread-pool queue-full warnings around the same timestamps.
• libbeat.output.events.failed and libbeat.output.events.retry metrics climb.
• On the wire, some bulk POSTs return 429, 503, or reset connections mid-request.

Common Root Causes

• Overloaded write thread pool — the cluster rejects bulk items when its write queue is full, which libbeat treats as temporary.
• Bulk batch too large — a huge bulk_max_size produces multi-megabyte requests that time out or trip http.max_content_length.
• Network instability — flaky links or an aggressive load balancer idle-timeout resets long bulk POSTs.
• Slow storage / merge pressure — hot data nodes with saturated disks cannot keep up with indexing.
• Ingest pipeline failures — a per-document pipeline error surfaces as retryable item failures.
• Under-provisioned cluster — too few shards or nodes for the event rate.

Diagnostic Workflow

Watch Filebeat while it retries and note the cadence and burst size:

journalctl -u filebeat -f | grep -i 'bulk send failure\|retry\|publish'

Check the Elasticsearch side for rejections at the same time — this is where the real reason lives:

curl -sk https://es01:9200/_nodes/thread_pool/write?pretty \
  -u elastic:$ES_PASS | grep -A3 '"write"'
curl -sk 'https://es01:9200/_cat/thread_pool/write?v&h=node_name,active,queue,rejected' \
  -u elastic:$ES_PASS

Look at cluster and index health for red/yellow shards or write pressure:

curl -sk https://es01:9200/_cluster/health?pretty -u elastic:$ES_PASS
curl -sk 'https://es01:9200/_cat/indices/filebeat-*?v&s=store.size:desc' -u elastic:$ES_PASS

Reproduce a bulk request manually to see the raw response Filebeat is reacting to:

curl -sk -H 'Content-Type: application/x-ndjson' \
  -u elastic:$ES_PASS 'https://es01:9200/_bulk' --data-binary $'
{"index":{"_index":"filebeat-test"}}
{"message":"probe","@timestamp":"2026-07-10T00:00:00Z"}
' | jq '.errors, .items[0]'

Example Root Cause Analysis

A pipeline pushing ~40k events/second logged temporary bulk send failure in tight bursts every few minutes. The Filebeat log alone was uninformative, but _cat/thread_pool/write showed the rejected counter climbing on two hot data nodes with queue pinned at its limit. The cluster was rejecting bulk items because the write queue overflowed during merge spikes.

Two changes stabilized it. First, Filebeat batched too aggressively, so bulk requests arrived in large synchronized waves:

output.elasticsearch:
  bulk_max_size: 800        # was 3200; smaller, steadier requests
  worker: 2
  backoff.init: 1s
  backoff.max: 60s

Second, the cluster gained a third hot node to spread write load. After the change, rejected stopped incrementing and the temporary bulk send failure messages disappeared — the retries had been masking a genuine capacity ceiling.

Prevention Best Practices

• Right-size bulk_max_size (start near 800–1600) so requests stay under http.max_content_length and the write queue.
• Monitor Elasticsearch _cat/thread_pool/write rejected and queue — rising rejections predict this error before Filebeat logs it.
• Scale hot data nodes or shards to match sustained event rate rather than relying on retries to absorb overload.
• Keep backoff.init/backoff.max sane so a struggling cluster is not hammered, but retries still recover quickly.
• Validate ingest pipelines separately so a document-level pipeline failure is not misread as a transport problem.

Quick Command Reference

journalctl -u filebeat -f | grep -i bulk        # watch retries
curl -sk 'https://es01:9200/_cat/thread_pool/write?v&h=node_name,active,queue,rejected' -u elastic:$ES_PASS
curl -sk https://es01:9200/_cluster/health?pretty -u elastic:$ES_PASS
curl -sk 'https://es01:9200/_cat/indices/filebeat-*?v' -u elastic:$ES_PASS
filebeat test output                            # confirm basic reachability

Conclusion

temporary bulk send failure is Filebeat correctly retrying a bulk request that the cluster or network could not complete. Occasional messages are harmless self-healing; a steady stream means Elasticsearch is rejecting writes. The fix lives on the cluster side — check the write thread pool for rejections, then reduce bulk_max_size and add write capacity. More output tuning lives in the Filebeat guides.