🚨 Logstash Error: 'Failed to install template' — Cause, Fix, and Troubleshooting Guide

Overview

On startup the Logstash elasticsearch output installs an index template so new indices are created with the right field mappings. This is on by default (manage_template => true), and the template name defaults to logstash — or ecs-logstash when ECS compatibility is enabled. If that install call is rejected, the output logs a failure and the template never lands:

[ERROR][logstash.outputs.elasticsearch][main] Failed to install template {:message=>"Got response code '403' contacting Elasticsearch at URL 'https://es.internal:9200/_index_template/ecs-logstash'", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError}

A 403 here is an authorization failure: the user authenticated, but its role is not allowed to manage index templates. The consequence is subtle — Logstash keeps shipping events, but without your template Elasticsearch auto-creates fields with guessed types (a numeric field becomes text, a timestamp becomes keyword), and you only discover it later when a mapping is wrong or a mapper_parsing_exception appears downstream.

Symptoms

• Failed to install template on every pipeline start, usually with response code 403 and security_exception.
• The same failure with a connection error (Elasticsearch Unreachable) instead of 403 when ES is down during startup.
• Indices get created with dynamically-mapped, wrong-typed fields; Kibana shows keyword where you expected date or long.
• With a custom template => path, the log names a missing file or invalid JSON rather than a privilege error.
• The error references _index_template/... (composable) on ES 8.x, or _template/... (legacy) on older configs — a clue when the API style is mismatched.

Common Root Causes

• Missing template privileges — the writer role lacks the cluster privilege manage_index_templates (and manage_ilm when ILM is on), so ES returns 403 security_exception.
• Elasticsearch unreachable at startup — the node is down or mid-restart when the output tries to install, so the call never gets an auth check.
• Invalid or missing custom template file — template => "/path/to/tpl.json" points at a file that does not exist, is not readable by the logstash user, or is malformed JSON.
• Template API mismatch — a legacy _template payload sent against an ES version that expects composable _index_template (or vice versa), producing a rejection.
• Existing conflicting template — a template with the same name and incompatible settings already exists and blocks the overwrite.

How to diagnose

First confirm which template name and API the output is targeting, and whether ES is even reachable with Logstash’s own credentials:

# Reachability + auth as the logstash user sees it
curl -s -o /dev/null -w '%{http_code}\n' -u logstash_writer:$ES_PW https://es.internal:9200/

# Does the composable template already exist / what does it contain?
curl -s -u logstash_writer:$ES_PW https://es.internal:9200/_index_template/ecs-logstash?pretty

Ask Elasticsearch directly whether the role is authorized, rather than guessing:

curl -s -u logstash_writer:$ES_PW -H 'Content-Type: application/json' \
  https://es.internal:9200/_security/user/_has_privileges -d '{
    "cluster": ["manage_index_templates","manage_ilm"]
  }'

A "has_all_requested": false here is the smoking gun. Inspect the output block to see what it is trying to install:

output {
  elasticsearch {
    hosts    => ["https://es.internal:9200"]
    user     => "logstash_writer"
    password => "${ES_PW}"
    index    => "app-logs-%{+YYYY.MM.dd}"
    # manage_template => true      # default
    # template_name   => "ecs-logstash"
    # template        => "/etc/logstash/templates/app.json"
  }
}

If you use a custom template, validate the file exists, is readable by the service user, and is well-formed JSON before blaming ES:

sudo -u logstash test -r /etc/logstash/templates/app.json && echo readable
jq empty /etc/logstash/templates/app.json && echo 'valid JSON'

Fixes

The most common fix is granting the role the cluster privileges it needs. Define or update the role via the security API:

PUT _security/role/logstash_writer_role
{
  "cluster": ["manage_index_templates", "manage_ilm", "monitor"],
  "indices": [
    {
      "names": ["app-logs-*"],
      "privileges": ["write", "create_index", "create_doc"]
    }
  ]
}

Then confirm the user is mapped to that role and re-run _has_privileges. Once it returns true, restart Logstash and the template installs cleanly.

If templates are managed externally (by your platform team, Terraform, or a separate bootstrap job), tell Logstash to stop trying:

output {
  elasticsearch {
    hosts           => ["https://es.internal:9200"]
    user            => "logstash_writer"
    password        => "${ES_PW}"
    index           => "app-logs-%{+YYYY.MM.dd}"
    manage_template => false
  }
}

For a custom template, point at a validated, readable JSON file and let Logstash own it:

output {
  elasticsearch {
    hosts         => ["https://es.internal:9200"]
    user          => "logstash_writer"
    password      => "${ES_PW}"
    template      => "/etc/logstash/templates/app.json"
    template_name => "app-logs"
    index         => "app-logs-%{+YYYY.MM.dd}"
  }
}

If ES was simply unreachable at startup, fix connectivity/credentials first, then restart and watch the install succeed:

sudo systemctl restart logstash
sudo tail -f /var/log/logstash/logstash-plain.log | grep -Ei 'template|403|security_exception'

What to watch out for

• manage_template => false is only safe if a correct template genuinely exists — otherwise you silently fall back to dynamic mapping and wrong field types.
• Match the API style to your ES version: ES 8.x uses composable _index_template; do not force a legacy _template payload against it.
• manage_index_templates is a cluster privilege, not an index privilege; granting index privileges alone will not clear the 403.
• When ILM is enabled, the template references an ILM policy — the role also needs manage_ilm, or the install fails for a different reason.
• Keep custom template JSON owned by and readable by the logstash user, and validate it with jq in CI so a bad edit never reaches production.

Related

• Elasticsearch output 429 rejections
• mapper_parsing_exception on write
• Elasticsearch Unreachable