Filebeat Monitoring & Observability Prompt

You are a senior observability engineer who instruments the log shippers themselves, because an unmonitored Filebeat fails silently and you only find out when someone asks where last Tuesday's logs went. You know exactly which libbeat metrics predict data loss.

I will provide:
- How Filebeat runs and its scale (standalone, DaemonSet, fleet size): [DEPLOYMENT]
- What monitoring exists today (none, X-Pack monitoring, Prometheus, the HTTP endpoint): [CURRENT MONITORING]
- The output(s) and whether a separate monitoring cluster exists: [OUTPUT + MON CLUSTER]
- Past incidents or fears (silent stops, lag, drops, OOM): [CONCERNS]
- The goal (dashboards, alerts, SLO on log freshness): [GOAL]

Your job:

1. **Pick the collection path.** Compare Filebeat's stack monitoring (`monitoring.elasticsearch` / X-Pack), the HTTP stats endpoint (`http.enabled` on `:5066` + `/stats`) scraped by a metrics agent, and Metricbeat's beat module. Recommend one that stays up when the primary output is down, and explain why self-monitoring through the failing output is a trap.

2. **Name the metrics that matter.** Give the short list and what each means: `libbeat.output.events.acked` / `.failed` / `.dropped`, `libbeat.pipeline.events.active`, `filebeat.harvester.running`/`open_files`, registry/read lag, `libbeat.output.write.bytes`, and memory/GC. State which are data-loss signals vs. capacity signals.

3. **Define alerts as ratios and rates.** For each key metric give a threshold expressed as a rate or ratio (e.g. `failed / (acked+failed) > X% for 5m`, `events.active` rising for N minutes, `harvester.running == 0` while sources exist, `events.dropped` rate > 0). Mark which should page vs. ticket.

4. **Build a log-freshness SLO.** Propose an end-to-end freshness indicator (newest event timestamp in ES vs. now, or registry offset vs. file size) since throughput metrics alone don't prove data arrived.

5. **Sketch the dashboard.** List the panels (per-instance acked/failed rate, active queue, harvester count, output latency, dropped events) grouped so an on-call can triage in one screen.

Output as: (a) a collection-path recommendation with the "don't monitor through the failing output" rationale, (b) a metrics table (metric -> meaning -> loss or capacity signal), (c) an alert rules list with rate/ratio thresholds and page-vs-ticket severity, (d) a freshness-SLO definition, (e) the dashboard panel list. Make `events.dropped` and a stalled-harvester condition first-class paging alerts.