Filebeat Conditional Output Routing Design Prompt
Route events from a single Filebeat to different Elasticsearch data streams, indices, or ingest pipelines based on conditions, so each log type lands in the right place with the right retention without running multiple agents.
- Target user
- Engineers routing mixed log streams to per-type indices, pipelines, or data streams in Filebeat
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a senior Filebeat engineer who splits one agent's mixed event stream to the correct destinations — data streams, indices, or ingest pipelines — using conditions, cleanly and without index explosion. I will provide: - The distinct log types this Filebeat handles and a few sample events for each (with the fields that distinguish them) - Where each type should land (target data stream / index name, retention/ILM expectation, and any per-type ingest pipeline) - My current `output.elasticsearch` block and whether I use classic indices or data streams - Filebeat version Your job: 1. **Choose the routing model** — decide between `output.elasticsearch.indices` with `when` conditions, `pipelines` with `when`, or per-type `data_stream` naming (dataset/namespace), and justify the choice for classic indices vs data streams. 2. **Write reliable conditions** — express each rule with the right condition (`equals`, `contains`, `regexp`, `has_fields`) against fields that actually exist at output time, and order them so the first match wins where that matters. 3. **Set a safe default** — define the fallback index/pipeline for anything that matches no rule, and make unmatched events observable (a distinct default or a tag) rather than silently blended in. 4. **Bound the destinations** — if a routing key comes from event data, constrain it to a known allow-list so a malformed or hostile value cannot spawn arbitrary indices; flag any unbounded key as a risk. 5. **Line up the backing config** — list the index templates, ILM policies, and pipelines each destination requires so routing does not fail on a missing template or an unknown pipeline. 6. **Verify** — give a test plan: feed one sample of each type, confirm it lands in the intended destination and pipeline, and confirm the default catches an unmatched sample. Output as: the complete `output.elasticsearch` routing block, each condition explained in plain English, the list of templates/ILM/pipelines to create, and a per-type validation checklist. Default to caution: always define and monitor a default destination, verify conditions against real events, and never let a user-controlled field become an unbounded index name.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Filebeat Elasticsearch Output Tuning Prompt
Tune the Filebeat Elasticsearch output — bulk sizing, worker count, compression, and load balancing — so ingest keeps up with log volume without overwhelming the cluster.
-
Filebeat ILM & Index Template Design Prompt
Design Filebeat's index lifecycle management, index template, and alias/rollover strategy so log indices roll over cleanly, age through hot/warm/cold, and delete on schedule.
-
Filebeat Ingest Pipeline Integration Prompt
Design and wire an Elasticsearch ingest pipeline to a Filebeat output so parsing, enrichment, and field mapping happen at ingest time with clean error handling.
-
Filebeat Kafka Output Design Prompt
Design a Filebeat -> Kafka output — topic/partition strategy, keying, acks, compression, and delivery semantics — so logs land in Kafka durably and in balanced partitions.
More Filebeat prompts & error guides
Browse every Filebeat prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.