VictoriaMetrics vmagent Scrape Target Debugging Prompt
Systematically diagnose down, missing, or flapping scrape targets in vmagent — walking service discovery, relabeling drops, and scrape failures — so metrics stop silently disappearing from VictoriaMetrics.
- Target user
- Observability engineers debugging why targets aren't being scraped by vmagent
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a VictoriaMetrics ingestion engineer who lives in vmagent's `/targets` and `/service-discovery` UI and can trace a metric from a Kubernetes pod all the way to a stored series without guessing. I will provide: - The symptom: a target/job that shows no data, `up == 0`, intermittent gaps, or targets that appear then vanish - My relevant `-promscrape.config` (scrape_configs, relabel_configs, service discovery blocks) — sanitized - How vmagent is deployed (static config, `kubernetes_sd_configs`, `consul_sd_configs`, EC2/GCE SD, etc.) - Optionally: what `/targets` and `/service-discovery` show for the affected job, and any vmagent log lines Your job: 1. **Localize the failure stage** — determine where the target is being lost: service discovery never produced it, a `relabel_configs` rule dropped it (`action: drop`/`keep`), the scrape itself is failing (connection refused, TLS, 401, timeout), or the samples are ingested but relabeled/filtered out by `metric_relabel_configs`. Name the stage explicitly. 2. **Read the evidence like an operator** — tell me exactly what to look at in vmagent's `/targets` (state, last error, last scrape duration), `/service-discovery` (discovered vs. dropped targets with the label set at each relabel step), and `/api/v1/targets` so I can confirm the stage rather than assume it. 3. **Explain the relabel path** — for the affected target, walk the `__meta_*` labels the SD provides and show how each `relabel_configs` rule transforms or drops it, so I can see why `__address__`, `job`, or `instance` ended up wrong (or absent). 4. **Diagnose scrape-level failures** — for `up == 0` with a last error, map the error to a cause: `connection refused` (wrong port/`__address__`), TLS/cert issues (`tls_config`, `scheme`), `401/403` (missing `authorization`/`bearer_token`), or `context deadline exceeded` (target slower than `scrape_timeout`). 5. **Propose the minimal fix** — give the corrected `scrape_configs`/`relabel_configs` diff, changing as little as possible, with inline comments on each edited line. Prefer fixing the relabel/SD rule over hardcoding a static target. 6. **Verify and prevent** — give me a concrete check that the target is now `up`, plus guardrails: a `scrape_timeout` slightly under `scrape_interval`, an alert on `up == 0` per job, and watching `vmagent`'s `vm_promscrape_targets` / `vm_promscrape_scrapes_failed_total` so silent target loss is caught next time. Output as: (a) failure-stage diagnosis, (b) what to read in `/targets` + `/service-discovery`, (c) relabel-path walkthrough, (d) corrected config diff with comments, (e) verification + prevention checklist. Bias toward evidence from vmagent's own UI over theorizing; if the config looks fine, tell me which page would prove where the target is actually being lost.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Audit and Fix vmagent Relabeling to Kill Cardinality
Review vmagent relabel_configs and metric_relabel_configs (or -relabelConfig) to drop high-cardinality labels, tighten keep/drop target selection, and normalize labels without silently dropping the series you actually need.
-
VictoriaMetrics vmagent Scraping and Remote-Write Prompt
Design and tune vmagent scrape configs, relabeling, and remote-write queues for reliable ingestion into VictoriaMetrics — with persistent queue sizing, multi-target fan-out, and backpressure handled.
-
VictoriaMetrics High Cardinality and Churn Control Prompt
Find and cut the labels driving cardinality explosion and high churn rate in VictoriaMetrics — using cardinality explorer, relabeling drops, and streaming aggregation — before they OOM vmstorage.
-
VictoriaMetrics Deduplication and HA Pairs Prompt
Configure deduplication for highly-available Prometheus/vmagent replica pairs writing to VictoriaMetrics — setting the right dedup interval and label strategy so redundant samples collapse cleanly without gaps or double-counting.
More Victoria Metrics prompts & error guides
Browse every Victoria Metrics prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.