Skip to content
DevOps AI ToolKit
Newsletter
Prometheus Troubleshooting Toolkit

Prometheus Troubleshooting Toolkit

Use this Prometheus troubleshooting toolkit to diagnose down targets, scrape timeouts, silent or noisy alerts, missing metrics, high cardinality, remote-write failures, and TSDB problems.

Paste your error and get a triage plan.

Paste a log line, CLI error, stack trace, service failure, or config snippet and get a structured troubleshooting plan. Your snippet is carried straight into the AI Incident Response Assistant with Prometheus context prefilled.

Open the Prometheus Incident Assistant → Browse Prometheus prompts

Do not paste secrets, tokens, private keys, passwords, or customer data. Your snippet stays in your browser until you open the assistant.

Top 25 Prometheus Errors and Failure Modes

The production failures engineers hit most — each links to a full cause → fix → prevention guide.

Target down (up == 0)

Prometheus reports a target as down and the up metric drops to zero, so its metrics stop being collected.

Scrape timeout (context deadline exceeded)

A scrape exceeds the configured scrape_timeout and fails with a context deadline exceeded error before metrics are returned.

Service discovery failure

Service discovery returns no targets or the wrong ones because relabeling rules silently drop them from the scrape pool.

PromQL query timeout (too many samples)

A query is aborted because it loads too many samples or runs past the query timeout, returning a query timed out error.

Alert not firing

An alert rule stays stuck in pending or never evaluates true, so no notification is ever sent for a real problem.

Alert firing too often

An alert flaps or re-fires repeatedly because its threshold, for duration, or grouping is tuned too aggressively.

Missing labels

Expected labels are absent from a series, breaking joins, routing, and dashboards that depend on them.

Duplicate series (found duplicate series for match group)

A many-to-one join or scrape produces duplicate series for the same label set, causing found duplicate series errors.

High cardinality

Unbounded label values explode the active series count, driving up memory usage and slowing queries.

Remote write failure

Prometheus fails to ship samples to a remote endpoint, returning 4xx or 5xx errors and backing up its remote write queue.

WAL corruption

A corrupted write-ahead log prevents Prometheus from opening its storage and completing startup replay.

TSDB compaction error

Background block compaction fails, leaving overlapping or unmergeable blocks and growing disk usage.

Disk full

The TSDB data directory runs out of space, causing write failures and potential data loss until retention or storage is fixed.

Out of memory (OOMKilled)

Prometheus is killed by the OOM killer when head series and query memory exceed the container or host limit.

Prometheus reload failed

A configuration reload is rejected because of a YAML or validation error, so the old config keeps running.

Rule evaluation failed

A recording or alerting rule errors during evaluation, so its output series and alerts are not produced.

Recording rule missing

A recording rule series is absent from queries and dashboards because the rule is misnamed, not loaded, or failing to evaluate.

Alertmanager notification failed

Alertmanager fails to deliver a notification to a receiver because of a bad config, auth error, or unreachable endpoint.

Blackbox probe failed

A Blackbox Exporter probe reports failure for an HTTP, TCP, or ICMP target, so synthetic checks and their alerts break.

Node exporter down

The node_exporter target is unreachable or crashing, so host CPU, memory, and disk metrics disappear.

Kubernetes scrape config error

A misconfigured Kubernetes service discovery or relabeling block causes pods and services to be scraped incorrectly or not at all.

TLS scrape error (x509)

A scrape over HTTPS fails with an x509 unknown authority or expired certificate error during the TLS handshake.

Basic auth scrape error (401/403)

A scrape is rejected with 401 or 403 because of missing or incorrect basic auth or bearer token credentials.

Metrics endpoint 404

The target returns 404 for the metrics path, usually because the scrape metrics_path or port is wrong.

Metrics endpoint 500

The application errors while rendering its metrics endpoint, returning 500 and giving Prometheus nothing to scrape.

Validate your config before you redeploy

Catch the structural mistakes that cause outages — 100% in your browser, nothing uploaded.

AI Alert Rule GeneratorYAML Validator (scrape / rules config) Not sure? Start with the Incident Assistant →

Best Prometheus Prompts

Turn symptoms, logs, and config into a structured plan with prompts tuned for Prometheus.

Target Down / Scrape Failure Triage

Diagnose why a target is down and restore scraping

Use this prompt

Scrape Timeout Slow Target Diagnosis

Find why scrapes time out and tune scrape_timeout

Use this prompt

Scrape Config & Service Discovery Debugging

Debug scrape jobs, relabeling, and service discovery

Use this prompt

PromQL Query Optimization

Debug and speed up slow or wrong PromQL queries

Use this prompt

Prometheus Alert Rule Generator

Generate and review alerting rules from intent

Use this prompt

High-Cardinality Control & Reduction

Diagnose cardinality spikes and cut active series

Use this prompt

Remote Write Queue & Backpressure Tuning

Fix remote write failures and shipping backpressure

Use this prompt

Config Reload Validation (promtool check)

Validate config before reload to avoid failed reloads

Use this prompt

WAL / TSDB Corruption Recovery

Recover a Prometheus that won't start after WAL corruption

Use this prompt

Download the Prometheus Troubleshooting Runbook Pack

Runbooks for scrape, query, and alerting problems — down targets, cardinality, remote-write, and silent alerts.

  • Target-down & scrape-timeout checks
  • PromQL & rule-evaluation debugging
  • High-cardinality diagnosis
  • Remote-write & TSDB triage
  • Alertmanager notification checks
Download the Prometheus Runbook Pack

All Prometheus Troubleshooting Guides

Grouped by failure-mode type — each guide covers cause, fix, validation, and prevention.

Authentication & TLS (4)

node_exporter permission denied collecting server returned HTTP status 401 Unauthorized x509: certificate signed by unknown authority x509: certificate has expired or is not yet valid

Networking (7)

connection reset by peer lookup ... no such host metrics endpoint 404 metrics endpoint 500 connect: connection refused Bad Gateway Thanos Store Gateway Caching Tiers Explained

Storage (8)

Thanos thanos_shipper_uploads_total Not Increasing: Blocks Not Shipping to Object Storage Debugging Thanos Sidecar Uploads to Object Storage (S3/GCS/Azure) out of sequence m-mapped chunk compaction failed no space left on device replaying WAL opening storage failed Prometheus TSDB Internals: Head Block, WAL, Compaction & Retention Explained

Deployment & Builds (1)

Building Incident Timelines From Prometheus Data With AI

Runtime (4)

OOMKilled rule manager error evaluating rule Generating Blackbox Exporter Probe Configs With AI Safely Detecting Dead Targets in Prometheus with absent() and Staleness Markers

Performance (20)

load too many samples into memory HTTP status 429 Too Many Requests dial tcp: i/o timeout unexpected EOF cannot allocate memory server returned HTTP status 503 Service Unavailable body size limit exceeded found multiple scrape configs with job name invalid metric type label_limit exceeded lock DB directory: resource temporarily unavailable scrape sample limit exceeded metric_relabel_configs as a Cardinality Firewall OpenTelemetry Collector Backpressure: memory_limiter, batch, and Queues context deadline exceeded query timed out too many open files Protecting the Prometheus Read Path: max-samples, timeout, and Concurrency Cutting Cardinality at Ingest With vmagent Stream Aggregation Scaling Prometheus Scraping: Functional Sharding, Hashmod, and Agent Mode

API errors (1)

remote_write server returned HTTP status 500

Configuration (12)

not a valid duration string vector cannot contain metrics with the same labelset missing labels Error loading config (--config.file=/etc/prometheus/prometheus.yml) duplicate sample for timestamp found error when loading rules parse error: unexpected context deadline exceeded Debugging Prometheus Relabeling Drops With AI Without Guessing AI Instrumentation Review: Catching Label Explosions at Code Time Enriching Prometheus Alert Annotations With Live Query Context Enforcing Tenant Labels in Multi-Tenant Prometheus and Mimir

Other (33)

Prometheus vs Datadog: Open-Source vs SaaS Observability Comparing ChatGPT and Claude for Data Analysis Thanos thanos_shipper_upload_failures_total Rising: Fix Failed Block Uploads HTTP response to HTTPS client not ready to serve traffic target_limit exceeded alert firing too often alertmanager failed to join cluster Alert Stuck 'Pending' and Never Firing binary expression must contain only scalar and instant vector types Empty query result exceeded maximum resolution of 11000 points per timeseries found duplicate series for the match group out of bounds rate should only be used with counters up == 0 Alertmanager Grouping Timers: group_wait, group_interval, and repeat_interval The $__rate_interval Trap: Why Grafana rate() Panels Lie When You Zoom Native Histograms vs Classic Buckets: Getting Quantiles You Can Trust many-to-many matching not allowed out of order sample remote_write 429 quantile_over_time vs histogram_quantile: Which Percentile to Trust Reviewing AI-Generated Grafana Alert Rules Before They Go Live Catching PromQL Unit Mistakes With AI Before They Mislead Migrating Nagios Checks to Prometheus Alerts With AI Unit Testing Prometheus Alert Rules With Promtool and AI What Is Infrastructure Observability? A 2026 Guide Alertmanager Inhibition Rules and Silences Done Right Grafana Dashboards as Code with Grafonnet: A GitOps Workflow That Scales Prometheus Federation vs Remote-Write: Which to Use and When PromQL rate() vs irate() vs increase(): When Each One Lies to You PromQL Subqueries and _over_time: Trend Analysis Without the Guesswork
Browse the full Prometheus category

Prometheus troubleshooting FAQ

Why is my Prometheus target down?
A target shows as down (up == 0) when Prometheus cannot complete a scrape. The most common causes are connection refused because the exporter isn't listening, DNS or network errors, a wrong port or metrics_path, TLS or auth rejections, or scrapes that exceed scrape_timeout. Start on the Targets page in the Prometheus UI, read the exact error next to the target, then curl the endpoint from the Prometheus host to reproduce it.
How do I fix a Prometheus scrape timeout?
A scrape timeout shows up as "context deadline exceeded" when a target takes longer than scrape_timeout to respond. Fix it by measuring how long the /metrics endpoint actually takes, reducing what the target exposes if it's rendering too many series, and only then raising scrape_timeout (which must stay at or below scrape_interval). Overloaded exporters and high-cardinality endpoints are the usual culprits.
Why is my Prometheus alert not firing?
An alert that never fires is usually stuck in pending or evaluating to no data. Check that the expression returns samples in the query browser, that the for duration isn't longer than the condition ever holds, that the rule group is actually loaded, and that the series has the labels your routing expects. Use the Alerts page to see whether the rule is inactive, pending, or firing before blaming Alertmanager.
How do I reduce Prometheus high cardinality?
High cardinality comes from labels with unbounded values such as user IDs, request paths, or full URLs. Identify the worst offenders with topk queries against scrape_samples_scraped and the TSDB status page, then drop or aggregate those labels using metric_relabel_configs, fix instrumentation to use bounded label values, and move expensive aggregations into recording rules. This cuts active series, memory, and query latency.
When should I use the Alert Rule Generator?
Use the Prometheus Alert Rule Generator when you know what you want to be paged about but not the exact PromQL and rule structure. It turns plain intent, such as "page me when p99 latency exceeds 500ms for five minutes," into a validated alerting rule with sensible thresholds, for durations, labels, and annotations, and it's a fast way to review or standardize AI-generated rules before committing them.
How do I recover Prometheus after WAL corruption or a full disk?
If Prometheus won't start with a WAL corruption or "opening storage failed" error, check disk space first, since a full TSDB directory is a frequent root cause. Free space or expand the volume, then let Prometheus truncate the corrupted WAL segment on restart; as a last resort remove the offending segment or the wal directory, accepting the loss of the most recent unpersisted samples. Lowering retention and adding disk alerts prevents a repeat.