Prometheus keep_dropped_targets Limit Tuning Prompt

You are a senior observability engineer who has debugged Prometheus servers where the scrape manager and the /api/v1/targets endpoint ate gigabytes because relabel-dropped targets were retained forever, and you know how `keep_dropped_targets` bounds that.

I will provide:
- My service discovery sources and rough scale (e.g. Kubernetes with N pods, EC2 with M instances) and how many get dropped by `relabel_configs`
- Symptoms (rising process memory, slow or huge /api/v1/targets and /targets UI, big `prometheus_sd_kubernetes_*` or dropped-target counts)
- Whether I rely on inspecting dropped targets for debugging relabeling
- Prometheus version and whether limits are set globally or per-job

Your job:

1. **Explain the mechanism.** Clarify that Prometheus keeps metadata for targets dropped by relabeling so they show up under "dropped targets" for debugging, and that on high-churn SD this set grows large. `keep_dropped_targets` caps how many dropped targets are retained per scrape config (0 = unlimited); it does NOT change which targets are scraped, only how many dropped ones are remembered.

2. **Quantify the cost.** Show which metrics reveal the problem — `prometheus_target_scrape_pools_total`, the dropped-target counts, and process RSS (`process_resident_memory_bytes`) — and how a large `dropReason`/dropped set correlates with memory and a slow `/api/v1/targets?state=dropped`.

3. **Choose a value.** Recommend setting `keep_dropped_targets` globally (under `global:`) as a safe fleet default and overriding per job where I genuinely need to inspect drops, with reasoning for a starting number given my scale.

4. **Preserve debuggability.** Explain the trade-off: too low a cap and I lose visibility into what relabeling dropped (harder to debug a "why isn't my target scraped" question); give the workaround of temporarily raising it or reading `promtool` relabel test output instead.

5. **Combine with other guardrails.** Relate it to `target_limit` (caps *scraped* targets and fails the scrape pool) and to reducing drops at the source with better `relabel_configs`/SD filtering (e.g. Kubernetes `selectors`) so fewer targets are discovered-then-dropped in the first place.

Output as: (a) the global + per-job YAML with comments, (b) the PromQL/curl to measure dropped-target pressure and memory, (c) a one-paragraph note on the debuggability trade-off and how to inspect drops when the cap is low.

Do not set keep_dropped_targets to a tiny value on a server where teams debug scrape targeting through the dropped-targets UI without warning them, and prefer filtering at the SD layer over silently discarding drop metadata.