🚨 Filebeat Error: 'cluster_block_exception ... index read-only / allow delete (api)' — Cause, Fix, and Troubleshooting Guide

What this error means

When an Elasticsearch data node crosses the flood-stage disk watermark (95% by default), Elasticsearch sets a read-only-allow-delete block on every index with a shard on that node. Filebeat’s bulk writes are then rejected:

Failed to perform any bulk index operations: 403 Forbidden: {"error":{"type":"cluster_block_exception","reason":"index [filebeat-8.13.0-2026.07.12] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}}

This is a disk-full protection, not an auth problem despite the 403. Once tripped, the block does not clear automatically even after you free space in older versions — you must remove it explicitly (newer versions auto-release once above the high watermark). Filebeat keeps retrying and queuing, so no events are lost, but ingestion halts until disk is freed and the block cleared.

How the agent reports it

• cluster_block_exception with FORBIDDEN/12/index read-only / allow delete (api) in bulk errors.
• A 403 Forbidden that is about disk, not credentials — filebeat test output still authenticates.
• One or more data nodes over ~95% disk in _cat/allocation.
• Ingestion stopped abruptly across many/all indices at once.

Checking the agent configuration

Check disk usage per node and cluster block status:

curl -sk 'https://es01:9200/_cat/allocation?v' -u elastic:$ES_PASS
curl -sk 'https://es01:9200/_cat/nodes?v&h=name,disk.used_percent,disk.avail' \
  -u elastic:$ES_PASS
curl -sk 'https://es01:9200/filebeat-*/_settings?flat_settings=true&pretty' \
  -u elastic:$ES_PASS | grep read_only

Agent configuration causes

• Flood-stage watermark hit — a data node exceeded 95% disk.
• Runaway index growth — no ILM, so indices grow until the disk fills.
• Small disks / large shards on hot nodes.
• Old block persisting after freeing space (needs manual clear on older ES).
• Uneven shard allocation filling one node while others have room.

Step-by-step resolution

First free disk (delete/roll old indices, expand storage). Then explicitly clear the block:

curl -sk -XPUT 'https://es01:9200/_all/_settings' -u elastic:$ES_PASS \
  -H 'Content-Type: application/json' -d '{
    "index.blocks.read_only_allow_delete": null
  }'

Delete or shrink old indices to reclaim space quickly:

curl -sk -XDELETE 'https://es01:9200/filebeat-8.13.0-2026.05.*' -u elastic:$ES_PASS

Prevent recurrence with ILM rollover/delete and, if needed, adjust watermarks temporarily while you add disk:

curl -sk -XPUT 'https://es01:9200/_cluster/settings' -u elastic:$ES_PASS \
  -H 'Content-Type: application/json' -d '{
    "transient": {
      "cluster.routing.allocation.disk.watermark.flood_stage": "97%"
    }
  }'

Filebeat’s queued events flush automatically once the block is gone.

Safer agent defaults

• The 403 is about disk, not credentials — do not chase auth.
• On older ES the block persists after freeing space; you must null the setting.
• Raising watermarks is a stopgap; the real fix is ILM plus adequate disk.
• The block hits every index with a shard on the full node, so impact is broad.

Related agent errors

• Filebeat Error Guide: ‘503 Service Unavailable’
• Filebeat Error Guide: ‘ILM policy not found’
• Filebeat Error Guide: ‘429 Too Many Requests’