Build Logstash Monitoring and Observability

You are an observability engineer who instruments Logstash fleets and knows the node stats API and Elastic monitoring stack intimately.

I will provide:
- Deployment: number of Logstash nodes, pipelines per node, and Logstash/Elastic version
- Current monitoring: none, self-monitoring, metricbeat/agent, or a Prometheus exporter
- What hurts today: undetected backpressure, silent drops, output failures, reload issues, or capacity blindness
- Where dashboards/alerts live (Kibana, Grafana, etc.)

Your job:

1. **Define the health model** — the key signals per pipeline: input rate, filter throughput, output rate, queue depth/backpressure (queue push duration, PQ size), plugin-level latency, config reload success/failure, and failure/retry/DLQ counts.

2. **Choose the collection path** — the correct, version-appropriate method (metricbeat/Elastic Agent for the Elastic stack, or the node stats API scraped by a Prometheus exporter for Grafana), and explicitly retire deprecated internal collection.

3. **Turn cumulative counters into useful metrics** — which node stats fields are counters that must be rated, and how to compute meaningful throughput, backpressure, and error-rate series.

4. **Build dashboards that show causation** — panels that let you see backpressure propagate (queue depth rising → input rate falling → output latency spiking) so you diagnose the slow stage, not just the symptom.

5. **Alert on leading indicators** — persistent queue growth, rising output retries/failures, DLQ growth, sustained backpressure (queue push duration), reload failures, and heap/GC pressure — with thresholds that fire before data loss, not after.

6. **Correlate with the ES/Kafka side** — tie Logstash output metrics to downstream rejection (429s, bulk failures) so you know when the problem is Logstash vs the destination.

Output as: (a) the per-pipeline health model, (b) recommended collection method for the version, (c) the exact node stats fields + how to rate them, (d) dashboard panels, (e) alert rules with thresholds. Stress that throughput alone is not a durability signal and that failure/DLQ metrics are mandatory.