🚨 Prometheus Error Guide: 'lookup ... no such host' — Fix Scrape DNS

Overview

Prometheus records this on a target’s status page when it cannot resolve the target’s hostname to an address before scraping:

Get "http://api-metrics:9100/metrics": dial tcp: lookup api-metrics on 10.96.0.10:53: no such host

In Kubernetes the fully qualified variant is common when a Service or namespace is wrong:

Get "http://svc.ns.svc.cluster.local:8080/metrics": dial tcp: lookup svc.ns.svc.cluster.local on 10.96.0.10:53: no such host

The target never gets scraped: up{} is 0 and the target status shows the resolver (10.96.0.10:53) that returned NXDOMAIN. This is a name-resolution failure, not a connectivity or auth failure — the address was never obtained.

Symptoms

• Target is DOWN with a lookup <name> on <resolver>:53: no such host error.
• up{job="..."} == 0 for one or more instances that share a name pattern.
• The error names the resolver Prometheus is actually using — often revealing the wrong DNS server.
• New targets from service discovery fail while static IP targets stay healthy.
• nslookup/dig for the same name from the Prometheus pod/host also fails.

Common Root Causes

• Typo or wrong FQDN — misspelled Service name, wrong namespace, or missing .svc.cluster.local suffix.
• Search-domain assumptions — a short name that resolves in one namespace but not from Prometheus’s namespace.
• Service or endpoint deleted — the target’s DNS record no longer exists (scaled to zero, renamed).
• Wrong resolver — Prometheus using a node resolver instead of cluster DNS (or vice versa), so cluster names don’t resolve.
• CoreDNS/kube-dns outage or misconfig — the resolver itself is failing or NXDOMAIN-ing valid names.
• Relabeling built a bad __address__ — service discovery produced a hostname that never existed.

Diagnostic Workflow

Resolve the exact name from the same context Prometheus runs in:

# From the Prometheus host or pod
dig +short api-metrics
dig +short svc.ns.svc.cluster.local

# Which resolver and search domains are actually in effect?
cat /etc/resolv.conf

In Kubernetes, verify the Service and its endpoints truly exist:

kubectl -n ns get svc svc
kubectl -n ns get endpointslices -l kubernetes.io/service-name=svc

Inspect what address Prometheus derived after relabeling — the Service Discovery page shows both discovered labels and the final __address__. Confirm the config’s target string:

scrape_configs:
  - job_name: api
    static_configs:
      - targets: ["api-metrics.monitoring.svc.cluster.local:9100"]

Confirm which targets are affected and whether they share a name pattern:

up == 0
count by (job) (up == 0)

Example Root Cause Analysis

After moving Prometheus into a dedicated monitoring namespace, several targets began failing with lookup api-metrics on 10.96.0.10:53: no such host. The scrape config used the short name api-metrics:9100, which had resolved when Prometheus lived in the default namespace. From monitoring, the search-domain expansion no longer matched.

dig +short api-metrics.default.svc.cluster.local resolved instantly, confirming the record existed under its FQDN. Updating the target to the fully qualified api-metrics.default.svc.cluster.local:9100 fixed every affected target and up{} returned to 1. The lesson: never rely on namespace-local short names for cross-namespace scraping.

Prevention Best Practices

• Always use fully qualified names for cross-namespace/cross-host targets instead of short names that depend on search domains.
• In Kubernetes, prefer service-discovery roles (endpointslice) over hardcoded hostnames so records track reality.
• Alert on up == 0 grouped by job, and separately watch count(up == 0) spiking (mass DNS failure = resolver problem).
• Monitor CoreDNS/kube-dns health so resolver outages are attributed correctly instead of blamed on targets.
• Validate relabeling produces resolvable __address__ values by checking the Service Discovery page after config changes.

Quick Command Reference

# Resolve from Prometheus's context
dig +short NAME
cat /etc/resolv.conf

# Kubernetes existence checks
kubectl -n NS get svc NAME
kubectl -n NS get endpointslices -l kubernetes.io/service-name=NAME

up == 0
count by (job) (up == 0)

Conclusion

no such host is a pure DNS failure: Prometheus asked the named resolver for the target and got NXDOMAIN. Reproduce the lookup from Prometheus’s own network context, confirm the name and namespace, and prefer FQDNs or service discovery over fragile short names. When many targets fail at once, suspect the resolver itself rather than the targets.