🚨 Logstash Error: 'retrying failed action with response code: 403' — Cause, Fix, and Troubleshooting Guide

Overview

The Logstash elasticsearch output ships events in bulk requests. When Elasticsearch rejects a bulk write, the output retries — and if the rejection is a 403, it will retry a request that can never succeed:

[INFO ][logstash.outputs.elasticsearch][main] retrying failed action with response code: 403 ({"type"=>"security_exception", "reason"=>"action [indices:data/write/bulk[s]] is unauthorized for user [logstash_writer] with effective roles [logstash_writer_role]"})

This is an authorization failure, not authentication and not rate limiting. The user logged in fine (that would be 401), and Elasticsearch is not asking you to back off (that would be 429). The role attached to the user simply does not have the index privileges needed to write to the target index pattern. Because the output treats it as retryable, these events loop until they exhaust retries and are routed to the dead-letter queue or dropped — while the log fills with the same 403 line.

Symptoms

• Repeated retrying failed action with response code: 403 with "type"=>"security_exception" naming the user and effective role.
• The reason names the blocked action, typically indices:data/write/bulk[s] or indices:admin/auto_create.
• Events never land in Elasticsearch; the target index stays empty or stops growing while the pipeline appears “busy”.
• DLQ growth (when enabled) or silent drops after retries are exhausted.
• Authentication itself works — a manual curl -u to _cluster/health with the same credentials returns 200, which rules out 401.

Common Root Causes

• Role lacks index write privileges — the role has no write, create_doc, or create_index on the target index pattern, so bulk writes are denied.
• Auto-create blocked — the first event to a new daily index needs create_index (or auto_create); without it the write fails even though existing indices would accept it.
• Index pattern mismatch — the role grants privileges on app-* but the output writes to logstash-* (or a renamed pattern), so nothing matches.
• Document- or field-level security — DLS/FLS on the role filters the write path and blocks the bulk action.
• Cluster block on the index — a read-only block (for example from a disk watermark, index.blocks.write) surfaces as an authorization-style rejection.

How to diagnose

First, positively distinguish 403 from 401 and 429, because the fix for each is different:

# 401 => bad/missing credentials; 200 => auth is fine, so a later 403 is authorization
curl -s -o /dev/null -w '%{http_code}\n' -u logstash_writer:$ES_PW \
  https://es.internal:9200/_cluster/health

Look at what the role actually grants versus what the write needs:

# Inspect the user and its roles
curl -s -u admin:$ADMIN_PW https://es.internal:9200/_security/user/logstash_writer?pretty

# Ask ES directly whether the write is authorized on the real target pattern
curl -s -u logstash_writer:$ES_PW -H 'Content-Type: application/json' \
  https://es.internal:9200/_security/user/_has_privileges -d '{
    "index": [
      { "names": ["app-logs-*"], "privileges": ["write","create_index","create_doc"] }
    ]
  }'

Confirm the output is writing to the pattern you think it is:

output {
  elasticsearch {
    hosts    => ["https://es.internal:9200"]
    user     => "logstash_writer"
    password => "${ES_PW}"
    index    => "app-logs-%{+YYYY.MM.dd}"
  }
}

Rule out an index-level write block, which can masquerade as an authorization problem:

curl -s -u admin:$ADMIN_PW 'https://es.internal:9200/app-logs-*/_settings?pretty' | grep -i block

Fixes

Grant the writer role the index privileges the bulk path requires, on the exact patterns you write to, plus the cluster privileges needed for templates and ILM:

PUT _security/role/logstash_writer_role
{
  "cluster": ["manage_index_templates", "manage_ilm", "monitor"],
  "indices": [
    {
      "names": ["logstash-*", "app-logs-*"],
      "privileges": ["write", "create_index", "create_doc"]
    }
  ]
}

Map the role to the user (idempotent — safe to re-apply):

PUT _security/user/logstash_writer
{
  "password": "${ES_PW}",
  "roles": ["logstash_writer_role"]
}

Verify with the privileges API before restarting, so you are not guessing:

curl -s -u logstash_writer:$ES_PW -H 'Content-Type: application/json' \
  https://es.internal:9200/_security/user/_has_privileges -d '{
    "index": [ { "names": ["app-logs-*"], "privileges": ["write","create_doc"] } ]
  }'
# expect "has_all_requested": true

If the 403 is actually a write block from a disk watermark, clear the block after freeing space rather than touching roles:

curl -s -u admin:$ADMIN_PW -X PUT -H 'Content-Type: application/json' \
  'https://es.internal:9200/app-logs-*/_settings' \
  -d '{"index.blocks.write": null, "index.blocks.read_only_allow_delete": null}'

Then restart and confirm the retries stop:

sudo systemctl restart logstash
sudo tail -f /var/log/logstash/logstash-plain.log | grep -Ei 'response code: 40[13]|security_exception'

What to watch out for

• 403 is not 429 — do not “fix” an authorization error by tuning flush_size or backoff; the requests will still fail.
• The first write to each new daily index needs create_index/auto_create; a role with only write succeeds on existing indices and fails at midnight rollover.
• Keep the role’s names patterns in sync with the output’s index template — a rename on one side silently breaks the other.
• DLS/FLS on the writer role can block the whole bulk action; writer roles rarely need document-level security, so scope it out.
• Alert on the response code: 403 log pattern and on DLQ growth so an authorization regression is caught before data is lost.

Related

• Elasticsearch output 429 rejections
• mapper_parsing_exception on write
• Dead-letter queue full