Filebeat ILM & Index Template Design Prompt
Design Filebeat's index lifecycle management, index template, and alias/rollover strategy so log indices roll over cleanly, age through hot/warm/cold, and delete on schedule.
- Target user
- Engineers managing Filebeat index lifecycle in Elasticsearch
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a senior Elasticsearch data-lifecycle engineer who has designed Filebeat ILM policies that keep cluster storage bounded while meeting retention SLAs. You know that Filebeat's ILM behavior is a chain of template -> alias -> policy -> rollover and that one broken link silently disables the rest. I will provide: - My current Filebeat `setup.ilm`, `setup.template`, and any `output.elasticsearch.index`/`indices` settings: [ILM/TEMPLATE CONFIG] - Elasticsearch version and whether data tiers (hot/warm/cold/frozen) exist: [ES TIERS] - Daily index volume, average doc size, and shard sizing target: [VOLUME] - Required retention (how long searchable, how long stored, legal hold): [RETENTION] - The goal/symptom (indices not rolling, shard bloat, disk filling, wrong template applied, multiple log types in one index): [GOAL] Your job: 1. **Explain the ILM chain for Filebeat.** Walk through how `setup.ilm.enabled` creates the write alias, how the bootstrap index and rollover alias work, and why the index template must match the alias pattern — so I can see where a break happens. 2. **Size the rollover.** Recommend `max_size`/`max_primary_shard_size`, `max_age`, and `max_docs` rollover conditions against my daily volume and a ~30-50GB/shard target, showing the arithmetic (daily GB / shard target -> shards or rollover cadence). 3. **Design the phase transitions.** Lay out hot -> warm -> cold -> delete with `min_age` per phase, matching my data tiers, and note per-phase actions (forcemerge, shrink, freeze, set_priority) that are safe versus costly. 4. **Handle multiple log types.** If I ship several datasets, decide whether to split indices via `output.elasticsearch.indices` conditionals (each with its own alias/policy) or keep one — with the shard-count and mapping trade-offs. 5. **Migration safety.** If I am changing an existing policy/template, spell out that changes apply only at next rollover, how to force a controlled rollover to validate, and how to avoid orphaning the current write index. Output as: (a) an annotated ILM chain diagram in text (template -> alias -> policy -> rollover), (b) the rollover-sizing arithmetic, (c) the corrected `setup.ilm`/`setup.template` config plus the ILM policy JSON with per-phase comments, (d) a validation checklist (confirm alias points at write index, force a test rollover, check next index picked up new template). Treat any delete phase change as a data-loss review gate.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Filebeat Conditional Output Routing Design Prompt
Route events from a single Filebeat to different Elasticsearch data streams, indices, or ingest pipelines based on conditions, so each log type lands in the right place with the right retention without running multiple agents.
-
Filebeat Filestream High-Volume Harvester Scaling Prompt
Scale the filestream input on high-volume hosts so thousands of active files are harvested promptly without exhausting file descriptors, stalling the scanner, or letting slow outputs starve the harvesters.
-
Filebeat Timestamp and Date Parsing Design Prompt
Set @timestamp from the event's own log time — across mixed formats and timezones — so documents sort by when the event happened, not when Filebeat read them, without dropping unparseable lines.
-
Filebeat Docker Autodiscover with Hints Prompt
Design a Filebeat Docker autodiscover configuration driven by container labels (hints) so per-service multiline, modules, and JSON parsing are applied automatically as containers come and go.
More Filebeat prompts & error guides
Browse every Filebeat prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.