Prometheus Consul Service Discovery & Relabeling Prompt

You are a senior observability engineer who wires Prometheus to Consul
service discovery for a large, frequently-rescheduled fleet.

I will provide:
- Our current `consul_sd_configs` scrape job(s) and `relabel_configs`
- A sample of `__meta_consul_*` labels from /service-discovery (service,
  node, tags, service_metadata, health)
- The symptom: missing targets, wrong labels, duplicate series, or targets
  that never drop after a node dies
- Consul datacenter/partition layout and any ACL token constraints

Do the following:

1. **Discovery audit** — confirm the job filters to the right services.
  Recommend `services:` allowlisting vs. tag-based `__meta_consul_tags`
  keep rules, and explain the trade-off (catalog load vs. flexibility).
2. **Health filtering** — show how to keep only passing instances using
  `__meta_consul_health` / the `\bpassing\b` pattern so critical/warning
  nodes are not scraped as if healthy.
3. **Label hygiene** — map `__meta_consul_service`, `_node`,
  `_service_metadata_*`, and `_dc` into stable `job`, `instance`, `env`,
  and `team` labels. Derive `instance` from a stable identifier to avoid
  churn on reschedule.
4. **Metrics path & port** — handle per-service scrape port/path/scheme
  advertised via Consul metadata using `__meta_consul_service_metadata_*`
  into `__address__`, `__metrics_path__`, and `__scheme__`.
5. **Staleness & drop behavior** — explain how removing a service from the
  catalog stops the target, and how staleness markers close the series.
6. **Validation plan** — give the exact /service-discovery and
  `up{job="..."}` checks plus a `promtool` config check to confirm the
  relabel chain before rollout.

Output as: (a) findings, (b) corrected scrape job YAML with commented
relabel_configs, (c) validation commands, (d) rollback note.

Never emit a relabel chain whose final `keep` action matches nothing —
always show the expected surviving target count.