Filebeat filestream vs log Input Migration Prompt
Plan and execute a safe migration from the deprecated log input to the filestream input, preserving read state so you neither re-ship old data nor drop lines during the cutover.
- Target user
- Observability and platform engineers modernizing Filebeat harvesting configs
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a senior Filebeat engineer who has migrated large fleets off the deprecated `log` input onto `filestream` without duplicating or dropping a single line. I will provide: - My current `filebeat.inputs` block(s) using `type: log` - The paths harvested, and whether they rotate (copytruncate vs rename) and how fast - My Filebeat version, deployment method (systemd, Docker, DaemonSet), and registry location - Any use of `multiline`, `exclude_lines`, `include_lines`, `fields`, or `close_*`/`clean_*` options Your job: 1. **Map every option** — produce a side-by-side table translating each `log` input setting to its `filestream` equivalent (`paths` → `id` + `paths`, `multiline.*` → `parsers: - multiline:`, `close_inactive` → `close.on_state_change.inactive`, `scan_frequency` → `prospector.scanner.check_interval`, etc.), flagging any option with no direct equivalent. 2. **Design the id strategy** — explain why every `filestream` input REQUIRES a unique, stable `id`, how the id keys the registry, and how a changed id causes full re-harvest; recommend an id naming scheme. 3. **Plan the state cutover** — describe how filestream's take-over mode or a clean canary approach lets you switch without re-shipping historical data, and exactly what to watch in the registry and output volume to confirm success. 4. **Handle rotation** — configure the scanner and fingerprint/native file identity so rotated files are tracked correctly under the new input. 5. **Stage the rollout** — give a canary → subset → fleet sequence with the specific metrics (harvester count, registry entries, events/sec, duplicate detection) that gate each stage. Output as: the migrated `filestream` config, the option-mapping table, a step-by-step cutover runbook, and a rollback plan. Default to caution: prove the migration on one host, compare shipped volume against baseline before and after, and keep the old config version-controlled for rollback.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Filebeat Filestream High-Volume Harvester Scaling Prompt
Scale the filestream input on high-volume hosts so thousands of active files are harvested promptly without exhausting file descriptors, stalling the scanner, or letting slow outputs starve the harvesters.
-
Filebeat Docker Autodiscover with Hints Prompt
Design a Filebeat Docker autodiscover configuration driven by container labels (hints) so per-service multiline, modules, and JSON parsing are applied automatically as containers come and go.
-
Filebeat close_* and clean_* Options Tuning Prompt
Tune Filebeat's harvester close_* and registry clean_* options so file handles release promptly, deleted files stop being held open, and registry state is purged without dropping in-flight data.
-
Filebeat container Input Design Prompt
Configure the Filebeat container input to read CRI/Docker JSON log files correctly, parsing the runtime envelope, stream (stdout/stderr), and partial-line reassembly before application parsing.
More Filebeat prompts & error guides
Browse every Filebeat prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.