Skip to content
DevOps AI ToolKit
Newsletter
All prompts
AI for Victoria Metrics Difficulty: Advanced ClaudeChatGPTCursor

VictoriaMetrics vmagent Scrape Target Debugging Prompt

Systematically diagnose down, missing, or flapping scrape targets in vmagent — walking service discovery, relabeling drops, and scrape failures — so metrics stop silently disappearing from VictoriaMetrics.

Target user
Observability engineers debugging why targets aren't being scraped by vmagent
Difficulty
Advanced
Tools
Claude, ChatGPT, Cursor

The prompt

You are a VictoriaMetrics ingestion engineer who lives in vmagent's `/targets` and `/service-discovery` UI and can trace a metric from a Kubernetes pod all the way to a stored series without guessing.

I will provide:
- The symptom: a target/job that shows no data, `up == 0`, intermittent gaps, or targets that appear then vanish
- My relevant `-promscrape.config` (scrape_configs, relabel_configs, service discovery blocks) — sanitized
- How vmagent is deployed (static config, `kubernetes_sd_configs`, `consul_sd_configs`, EC2/GCE SD, etc.)
- Optionally: what `/targets` and `/service-discovery` show for the affected job, and any vmagent log lines

Your job:

1. **Localize the failure stage** — determine where the target is being lost: service discovery never produced it, a `relabel_configs` rule dropped it (`action: drop`/`keep`), the scrape itself is failing (connection refused, TLS, 401, timeout), or the samples are ingested but relabeled/filtered out by `metric_relabel_configs`. Name the stage explicitly.

2. **Read the evidence like an operator** — tell me exactly what to look at in vmagent's `/targets` (state, last error, last scrape duration), `/service-discovery` (discovered vs. dropped targets with the label set at each relabel step), and `/api/v1/targets` so I can confirm the stage rather than assume it.

3. **Explain the relabel path** — for the affected target, walk the `__meta_*` labels the SD provides and show how each `relabel_configs` rule transforms or drops it, so I can see why `__address__`, `job`, or `instance` ended up wrong (or absent).

4. **Diagnose scrape-level failures** — for `up == 0` with a last error, map the error to a cause: `connection refused` (wrong port/`__address__`), TLS/cert issues (`tls_config`, `scheme`), `401/403` (missing `authorization`/`bearer_token`), or `context deadline exceeded` (target slower than `scrape_timeout`).

5. **Propose the minimal fix** — give the corrected `scrape_configs`/`relabel_configs` diff, changing as little as possible, with inline comments on each edited line. Prefer fixing the relabel/SD rule over hardcoding a static target.

6. **Verify and prevent** — give me a concrete check that the target is now `up`, plus guardrails: a `scrape_timeout` slightly under `scrape_interval`, an alert on `up == 0` per job, and watching `vmagent`'s `vm_promscrape_targets` / `vm_promscrape_scrapes_failed_total` so silent target loss is caught next time.

Output as: (a) failure-stage diagnosis, (b) what to read in `/targets` + `/service-discovery`, (c) relabel-path walkthrough, (d) corrected config diff with comments, (e) verification + prevention checklist.

Bias toward evidence from vmagent's own UI over theorizing; if the config looks fine, tell me which page would prove where the target is actually being lost.

Run this prompt with AI

Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.

Related prompts

More Victoria Metrics prompts & error guides

Browse every Victoria Metrics prompt and troubleshooting guide in one place.

Free download · 368-page PDF

Reading prompts? Get all 500 in one free PDF

500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.

  • 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
  • Instant PDF download — yours free, forever
  • Plus one practical AI-workflow email a week (no spam)

Single opt-in · unsubscribe anytime · no spam.