🚨 Filebeat Error Guide: 'ILM policy not found' — Restore Lifecycle Management

Overview

Filebeat wires its data stream to an Index Lifecycle Management (ILM) policy during setup. When the template references a policy that does not exist in Elasticsearch, index creation or setup fails with:

Exiting: error loading index lifecycle policy: index lifecycle policy [filebeat] not found: illegal_argument_exception: index.lifecycle.name [filebeat] refers to a non-existent ILM policy

ILM controls rollover, retention, and deletion of time-series indices. Filebeat’s index template sets index.lifecycle.name to a policy (default filebeat) that must already be loaded. If setup was skipped, the policy was deleted, or a custom name was configured without loading it, the reference dangles and Elasticsearch rejects the operation. The result is that indices either fail to create or never roll over, growing without bound.

Symptoms

• error loading index lifecycle policy: index lifecycle policy [...] not found on startup or during filebeat setup --index-management.
• Filebeat data-stream indices stop rolling over and grow past their intended size/age.
• _ilm/policy/filebeat returns 404.
• The index template’s index.lifecycle.name points at a policy that _ilm/explain reports as missing.
• New indices fail to create with illegal_argument_exception referencing the non-existent policy.

Common Root Causes

• Setup never ran — filebeat setup --index-management was skipped, so the default policy was never created.
• Policy deleted — someone removed the filebeat policy but the template still references it.
• Custom policy name unloaded — setup.ilm.policy_name was changed without loading the corresponding policy.
• Insufficient privileges — the setup user lacks manage_ilm, so the policy PUT silently failed earlier.
• ILM disabled mismatch — setup.ilm.enabled: false while the template still carries index.lifecycle.name.
• Restored template without policy — a template snapshot was restored but the ILM policy was not.

Diagnostic Workflow

Confirm whether the referenced policy exists:

curl -sk 'https://es01:9200/_ilm/policy/filebeat?pretty' -u elastic:$ES_PASS

Check what policy the write index expects and its current ILM state:

curl -sk 'https://es01:9200/filebeat-*/_ilm/explain?pretty' -u elastic:$ES_PASS \
  | grep -A3 'policy\|step\|error'

Inspect the template’s lifecycle reference:

curl -sk 'https://es01:9200/_index_template/filebeat-8.13.0?pretty' -u elastic:$ES_PASS \
  | grep -A2 lifecycle

Verify the setup user can manage ILM, then reload just the policy:

curl -sk 'https://es01:9200/_security/_authenticate?pretty' -u elastic:$ES_PASS
filebeat setup --index-management -e -v 2>&1 | tail -30

Example Root Cause Analysis

A cluster cleanup script deleted “unused” ILM policies, including filebeat, while the Filebeat index template still set index.lifecycle.name: filebeat. The next daily rollover failed and new-index creation logged index lifecycle policy [filebeat] not found. _ilm/policy/filebeat returned 404, and _ilm/explain on the write index showed a step: ERROR referencing the missing policy.

Reloading the policy via Filebeat’s own setup restored it without hand-writing JSON:

filebeat setup --index-management \
  -E output.elasticsearch.username=elastic \
  -E output.elasticsearch.password="$ES_PASS"

_ilm/policy/filebeat then returned the phased policy, and _ilm/explain moved the write index out of the ERROR step on the next poll. To prevent recurrence, the team added a guard so the cleanup script never touches policies referenced by an active template.

Prevention Best Practices

• Always run filebeat setup --index-management (with a manage_ilm-capable user) before first ingestion in a new cluster.
• Never delete ILM policies without checking _index_template references first — a dangling index.lifecycle.name breaks rollover.
• Keep setup.ilm.policy_name and the loaded policy in lockstep in configuration management.
• Include ILM policies alongside templates in any snapshot/restore or disaster-recovery runbook.
• Monitor _ilm/explain for indices stuck in the ERROR step so a missing policy surfaces before disks fill.

Quick Command Reference

curl -sk 'https://es01:9200/_ilm/policy/filebeat?pretty' -u elastic:$ES_PASS
curl -sk 'https://es01:9200/filebeat-*/_ilm/explain?pretty' -u elastic:$ES_PASS
curl -sk 'https://es01:9200/_index_template/filebeat-8.13.0?pretty' -u elastic:$ES_PASS | grep lifecycle
filebeat setup --index-management -e -v

Conclusion

ILM policy not found means Filebeat’s template points at a lifecycle policy that no longer exists in Elasticsearch. Confirm the gap with _ilm/policy and _ilm/explain, then reload it with filebeat setup --index-management using a privileged user. Protect the policy from cleanup scripts and keep it in your DR runbook so rollover never silently stops. More lifecycle fixes are in the Filebeat guides.