🚨 Logstash Error: 'FORBIDDEN/12/index read-only / allow delete' — Cause, Fix, and Troubleshooting Guide

Overview

When a data node crosses the flood-stage disk watermark, Elasticsearch protects itself by marking every index with a shard on that node read-only. Writes are then rejected with a cluster_block_exception, which the Logstash elasticsearch output surfaces as repeated 403 retries on its bulk requests:

[INFO ][logstash.outputs.elasticsearch][main] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"index [logstash-2026.07.12] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})

The flood-stage watermark defaults to 95% (cluster.routing.allocation.disk.watermark.flood_stage). This is entirely an Elasticsearch-side protection — the block is applied by the cluster, not by Logstash — but it manifests in your pipeline as bulk actions that fail and retry forever. Until disk is freed and the block is cleared, those events cannot be indexed.

Symptoms

• retrying failed action with response code: 403 with FORBIDDEN/12/index read-only / allow delete (api) in logstash-plain.log.
• The message is a cluster_block_exception naming a specific index as blocked.
• Ingest stalls or the queue grows because the output retries the blocked bulks indefinitely.
• New documents stop appearing in Kibana even though Logstash is running and reachable.
• _cat/allocation shows one or more data nodes at or above 95% disk.

Common Root Causes

• Flood-stage watermark crossed — a data node passed 95% disk, so ES applied the read-only-allow-delete block cluster-wide to affected indices.
• Runaway index growth — no ILM rollover/delete, so indices grow until they fill the disk.
• Undersized data nodes — the disk is simply too small for the retained data volume.
• Old indices and snapshots never cleaned — stale logstash-* indices and local snapshots accumulate and consume the disk.
• Block left set after a prior incident — disk was freed earlier but index.blocks.read_only_allow_delete was never cleared.

How to diagnose

Confirm the block and identify which nodes are actually short on disk. Start with allocation — this shows disk use per node:

# Per-node disk usage; look for nodes at/over 95%
curl -s 'https://es.internal:9200/_cat/allocation?v'

# Which indices carry the read-only-allow-delete block?
curl -s 'https://es.internal:9200/_all/_settings/index.blocks.read_only_allow_delete?flat_settings=true'

Check the effective watermark settings, including defaults, so you know the threshold that tripped:

curl -s 'https://es.internal:9200/_cluster/settings?include_defaults=true&flat_settings=true' \
  | grep -i 'disk.watermark'

On the Logstash side, the output is behaving correctly — it retries 403s. The relevant config is simply which cluster it writes to. Logstash .conf files use a Ruby-like DSL, so the output is shown as ruby:

output {
  elasticsearch {
    hosts => ["https://es.internal:9200"]
    index => "logstash-%{+YYYY.MM.dd}"
  }
}

There is nothing to fix in the pipeline — the block must be resolved on the Elasticsearch cluster.

Fixes

Fix this in order: free disk first, then clear the block. Clearing the block while the disk is still full only lets ES re-apply it on the next write.

First, free disk on the affected data nodes — delete old indices, remove stale snapshots, clear logs, or add storage:

# Delete an old, no-longer-needed index (example)
curl -s -X DELETE 'https://es.internal:9200/logstash-2026.05.01'

# Re-check that nodes are back under the watermark
curl -s 'https://es.internal:9200/_cat/allocation?v'

Once nodes are safely below the flood-stage watermark, clear the read-only-allow-delete block. Setting it to null on all indices lifts the block:

curl -s -X PUT 'https://es.internal:9200/_all/_settings' \
  -H 'Content-Type: application/json' \
  -d '{"index.blocks.read_only_allow_delete": null}'

Writes resume immediately; Logstash’s retried bulks succeed and the queue drains. For the long term, put the indices under ILM with rollover and delete phases so they never fill the disk again:

# Sketch of an ILM policy: roll over by size/age, delete when old
curl -s -X PUT 'https://es.internal:9200/_ilm/policy/logstash-policy' \
  -H 'Content-Type: application/json' \
  -d '{"policy":{"phases":{
        "hot":{"actions":{"rollover":{"max_size":"50gb","max_age":"1d"}}},
        "delete":{"min_age":"14d","actions":{"delete":{}}}}}}'

What to watch out for

• Clearing the block without freeing disk is a temporary fix — ES re-applies it the moment the node crosses flood-stage again.
• The block is applied per index, not per node — an index with a single shard on a full node gets blocked even if other nodes have space.
• read_only_allow_delete still permits deletes, which is deliberate: it lets you delete data to recover, but blocks all writes and updates.
• Alert on disk watermarks (_cat/allocation) well before 95% so you act at the high watermark, not after the flood-stage block lands.
• ILM rollover/delete is the real fix; manually deleting indices during every incident does not scale.

Related

• Handle Elasticsearch output 429 rejections in Logstash
• Recover a full Logstash dead letter queue
• Fix the Logstash mapper_parsing_exception