🚨 Filebeat Error: 'cluster_block_exception ... index read-only / allow delete (api)' — Cause, Fix, and Troubleshooting Guide

Overview

When an Elasticsearch data node crosses the flood-stage disk watermark (95% by default), Elasticsearch sets a read-only-allow-delete block on every index with a shard on that node. Filebeat’s bulk writes are then rejected:

Failed to perform any bulk index operations: 403 Forbidden: {"error":{"type":"cluster_block_exception","reason":"index [filebeat-8.13.0-2026.07.12] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}}

This is a disk-full protection, not an auth problem despite the 403. Once tripped, the block does not clear automatically even after you free space in older versions — you must remove it explicitly (newer versions auto-release once above the high watermark). Filebeat keeps retrying and queuing, so no events are lost, but ingestion halts until disk is freed and the block cleared.

Symptoms

• cluster_block_exception with FORBIDDEN/12/index read-only / allow delete (api) in bulk errors.
• A 403 Forbidden that is about disk, not credentials — filebeat test output still authenticates.
• One or more data nodes over ~95% disk in _cat/allocation.
• Ingestion stopped abruptly across many/all indices at once.

Common Root Causes

• Flood-stage watermark hit — a data node exceeded 95% disk.
• Runaway index growth — no ILM, so indices grow until the disk fills.
• Small disks / large shards on hot nodes.
• Old block persisting after freeing space (needs manual clear on older ES).
• Uneven shard allocation filling one node while others have room.

How to diagnose

Check disk usage per node and cluster block status:

curl -sk 'https://es01:9200/_cat/allocation?v' -u elastic:$ES_PASS
curl -sk 'https://es01:9200/_cat/nodes?v&h=name,disk.used_percent,disk.avail' \
  -u elastic:$ES_PASS
curl -sk 'https://es01:9200/filebeat-*/_settings?flat_settings=true&pretty' \
  -u elastic:$ES_PASS | grep read_only

Fixes

First free disk (delete/roll old indices, expand storage). Then explicitly clear the block:

curl -sk -XPUT 'https://es01:9200/_all/_settings' -u elastic:$ES_PASS \
  -H 'Content-Type: application/json' -d '{
    "index.blocks.read_only_allow_delete": null
  }'

Delete or shrink old indices to reclaim space quickly:

curl -sk -XDELETE 'https://es01:9200/filebeat-8.13.0-2026.05.*' -u elastic:$ES_PASS

Prevent recurrence with ILM rollover/delete and, if needed, adjust watermarks temporarily while you add disk:

curl -sk -XPUT 'https://es01:9200/_cluster/settings' -u elastic:$ES_PASS \
  -H 'Content-Type: application/json' -d '{
    "transient": {
      "cluster.routing.allocation.disk.watermark.flood_stage": "97%"
    }
  }'

Filebeat’s queued events flush automatically once the block is gone.

What to watch out for

• The 403 is about disk, not credentials — do not chase auth.
• On older ES the block persists after freeing space; you must null the setting.
• Raising watermarks is a stopgap; the real fix is ILM plus adequate disk.
• The block hits every index with a shard on the full node, so impact is broad.

Related

• Filebeat Error Guide: ‘503 Service Unavailable’
• Filebeat Error Guide: ‘ILM policy not found’
• Filebeat Error Guide: ‘429 Too Many Requests’