🚨 Logstash Error: 'FORBIDDEN/12/index read-only / allow delete' — Cause, Fix, and Troubleshooting Guide

What this error means

When a data node crosses the flood-stage disk watermark, Elasticsearch protects itself by marking every index with a shard on that node read-only. Writes are then rejected with a cluster_block_exception, which the Logstash elasticsearch output surfaces as repeated 403 retries on its bulk requests:

[INFO ][logstash.outputs.elasticsearch][main] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"index [logstash-2026.07.12] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})

The flood-stage watermark defaults to 95% (cluster.routing.allocation.disk.watermark.flood_stage). This is entirely an Elasticsearch-side protection — the block is applied by the cluster, not by Logstash — but it manifests in your pipeline as bulk actions that fail and retry forever. Until disk is freed and the block is cleared, those events cannot be indexed.

Where it surfaces

• retrying failed action with response code: 403 with FORBIDDEN/12/index read-only / allow delete (api) in logstash-plain.log.
• The message is a cluster_block_exception naming a specific index as blocked.
• Ingest stalls or the queue grows because the output retries the blocked bulks indefinitely.
• New documents stop appearing in Kibana even though Logstash is running and reachable.
• _cat/allocation shows one or more data nodes at or above 95% disk.

Identity and permission causes

• Flood-stage watermark crossed — a data node passed 95% disk, so ES applied the read-only-allow-delete block cluster-wide to affected indices.
• Runaway index growth — no ILM rollover/delete, so indices grow until they fill the disk.
• Undersized data nodes — the disk is simply too small for the retained data volume.
• Old indices and snapshots never cleaned — stale logstash-* indices and local snapshots accumulate and consume the disk.
• Block left set after a prior incident — disk was freed earlier but index.blocks.read_only_allow_delete was never cleared.

Tracing the failed authorization

Confirm the block and identify which nodes are actually short on disk. Start with allocation — this shows disk use per node:

# Per-node disk usage; look for nodes at/over 95%
curl -s 'https://es.internal:9200/_cat/allocation?v'

# Which indices carry the read-only-allow-delete block?
curl -s 'https://es.internal:9200/_all/_settings/index.blocks.read_only_allow_delete?flat_settings=true'

Check the effective watermark settings, including defaults, so you know the threshold that tripped:

curl -s 'https://es.internal:9200/_cluster/settings?include_defaults=true&flat_settings=true' \
  | grep -i 'disk.watermark'

On the Logstash side, the output is behaving correctly — it retries 403s. The relevant config is simply which cluster it writes to. Logstash .conf files use a Ruby-like DSL, so the output is shown as ruby:

output {
  elasticsearch {
    hosts => ["https://es.internal:9200"]
    index => "logstash-%{+YYYY.MM.dd}"
  }
}

There is nothing to fix in the pipeline — the block must be resolved on the Elasticsearch cluster.

Resolution

Fix this in order: free disk first, then clear the block. Clearing the block while the disk is still full only lets ES re-apply it on the next write.

First, free disk on the affected data nodes — delete old indices, remove stale snapshots, clear logs, or add storage:

# Delete an old, no-longer-needed index (example)
curl -s -X DELETE 'https://es.internal:9200/logstash-2026.05.01'

# Re-check that nodes are back under the watermark
curl -s 'https://es.internal:9200/_cat/allocation?v'

Once nodes are safely below the flood-stage watermark, clear the read-only-allow-delete block. Setting it to null on all indices lifts the block:

curl -s -X PUT 'https://es.internal:9200/_all/_settings' \
  -H 'Content-Type: application/json' \
  -d '{"index.blocks.read_only_allow_delete": null}'

Writes resume immediately; Logstash’s retried bulks succeed and the queue drains. For the long term, put the indices under ILM with rollover and delete phases so they never fill the disk again:

# Sketch of an ILM policy: roll over by size/age, delete when old
curl -s -X PUT 'https://es.internal:9200/_ilm/policy/logstash-policy' \
  -H 'Content-Type: application/json' \
  -d '{"policy":{"phases":{
        "hot":{"actions":{"rollover":{"max_size":"50gb","max_age":"1d"}}},
        "delete":{"min_age":"14d","actions":{"delete":{}}}}}}'

Hardening access

• Clearing the block without freeing disk is a temporary fix — ES re-applies it the moment the node crosses flood-stage again.
• The block is applied per index, not per node — an index with a single shard on a full node gets blocked even if other nodes have space.
• read_only_allow_delete still permits deletes, which is deliberate: it lets you delete data to recover, but blocks all writes and updates.
• Alert on disk watermarks (_cat/allocation) well before 95% so you act at the high watermark, not after the flood-stage block lands.
• ILM rollover/delete is the real fix; manually deleting indices during every incident does not scale.

Related identity errors

• Handle Elasticsearch output 429 rejections in Logstash
• Recover a full Logstash dead letter queue
• Fix the Logstash mapper_parsing_exception