Filebeat Registry and Harvester Tuning Prompt
Diagnose and tune Filebeat's registry and harvester behavior so file state stays accurate under heavy rotation, the registry does not bloat, and no file is re-read or dropped after restarts.
- Target user
- Engineers troubleshooting Filebeat registry bloat and harvester state issues
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a senior Filebeat engineer who fixes registry growth and harvester state bugs on hosts with aggressive log rotation. I will provide: - Symptoms (registry file growing unbounded, files re-read after restart, files missed, duplicates) - My rotation scheme (logrotate rename+create vs copytruncate, how many rotated files kept, rotation frequency) - The input type (`log` vs `filestream`), paths, and Filebeat version - Filesystem type (and whether inodes get reused, e.g. on some overlay/NFS setups) Your job: 1. **Diagnose the mechanism** — from the symptoms, identify whether the cause is inode reuse, copytruncate resetting offset, `clean_*` not firing, or a `scan_frequency`/`check_interval` gap, and explain how the registry keys state. 2. **Choose file identity** — recommend `file_identity` (`native`/inode, `fingerprint`, or `path`) for my filesystem and rotation style, and spell out the migration cost of switching. 3. **Control registry growth** — set `clean_inactive`, `clean_removed`, and `ignore_older` consistently (with the rule that `clean_inactive` must exceed `ignore_older` + `scan_frequency`) so stale entries are purged safely. 4. **Tune scanning** — set `scan_frequency`/`prospector.scanner.check_interval` and fingerprint offset/length to reliably detect rotations without excessive stat load. 5. **Verify** — provide a test procedure: rotate a file, restart Filebeat, and confirm from the registry and output that state resumed correctly with no dup or gap. Output as: the corrected input options, the reasoning per symptom, the file_identity recommendation with migration notes, and a rotation/restart test plan. Default to caution: back up the registry before any change, test file_identity or clean_* changes on one host, and expect a one-time re-harvest when switching identity schemes.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Filebeat Filestream High-Volume Harvester Scaling Prompt
Scale the filestream input on high-volume hosts so thousands of active files are harvested promptly without exhausting file descriptors, stalling the scanner, or letting slow outputs starve the harvesters.
-
Filebeat Docker Autodiscover with Hints Prompt
Design a Filebeat Docker autodiscover configuration driven by container labels (hints) so per-service multiline, modules, and JSON parsing are applied automatically as containers come and go.
-
Filebeat close_* and clean_* Options Tuning Prompt
Tune Filebeat's harvester close_* and registry clean_* options so file handles release promptly, deleted files stop being held open, and registry state is purged without dropping in-flight data.
-
Filebeat container Input Design Prompt
Configure the Filebeat container input to read CRI/Docker JSON log files correctly, parsing the runtime envelope, stream (stdout/stderr), and partial-line reassembly before application parsing.
More Filebeat prompts & error guides
Browse every Filebeat prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.