🚨 Prometheus Error Guide: 'missing labels' — Restore Labels That Break Joins & Routing

Overview

There is no crash for a missing label — the symptom is a series that renders but lacks a label you rely on, so joins return nothing and routing silently misfires. In the expression browser the series is missing the team (or env, cluster) label you expected:

# expected
http_requests_total{job="api", instance="node-1:9100", team="payments"}

# actual — team is gone
http_requests_total{job="api", instance="node-1:9100"}

A PromQL join built on that label then evaluates to empty:

Empty query result

Symptoms

• A dashboard panel or alert that filters/groups by a label suddenly shows no data.
• A vector match (on(...) / group_left) returns an empty result even though both metrics exist.
• Alertmanager routes an alert to the wrong receiver because a routing label (team, severity, env) is absent.
• The series exists in the expression browser but is missing a label present on other targets.
• A relabeling change deployed recently and labels disappeared right after.

Common Root Causes

• A relabel_configs rule dropped or overwrote the label before the scrape (target labels).
• A metric_relabel_configs rule stripped the label after the scrape but before ingestion.
• The exporter never exposed the label, and you assumed service discovery would add it.
• __meta_* service-discovery labels were not mapped into real labels, so they were discarded.
• honor_labels handling — an exporter (like Pushgateway) set a label the server overwrote, or vice versa.
• A join using on() with the wrong label set, so matching labels do not line up between the two sides.
• Inconsistent labels across targets — some instances carry the label, some do not.

Diagnostic Workflow

Start in the expression browser. Query the bare metric and read the actual label set on the returned series:

http_requests_total{job="api"}

If the label is missing, check whether it is being produced at scrape time. up carries the target labels, so it is a quick way to see what the server attached:

up{job="api"}

Look at the scrape config. relabel_configs acts on target/__meta_* labels before the scrape; metric_relabel_configs acts on every sample after the scrape:

# prometheus.yml
scrape_configs:
  - job_name: api
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # keep only pods we mean to scrape
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: api
        action: keep
      # map a discovery meta-label into a real label
      - source_labels: [__meta_kubernetes_pod_label_team]
        target_label: team
      - source_labels: [__meta_kubernetes_pod_node_name]
        target_label: node
    metric_relabel_configs:
      # this kind of rule silently deletes a label
      - regex: team
        action: labeldrop

The labeldrop above is a classic culprit — it removes team from every sample after scrape. Removing it restores the label. To confirm which meta-labels are available for mapping, open Status > Targets in the UI and expand the target’s “Discovered Labels”.

For a join that returns empty, verify both sides share the matching labels. A one-to-many join needs group_left and a common label set:

sum by (instance) (rate(http_requests_total{job="api"}[5m]))
  * on(instance) group_left(team)
  node_meta{job="api"}

If node_meta lacks instance in the same format (e.g. node-1:9100 vs node-1), the on(instance) match fails and you get an empty result. Normalize the label with relabeling so both sides agree.

When ingesting from Pushgateway or a federation endpoint, honor_labels decides who wins on a conflict:

  - job_name: pushgateway
    honor_labels: true      # keep labels the source sent, don't overwrite with job/instance
    static_configs:
      - targets: ['node-1:9091']

Validate the whole config before reloading:

promtool check config prometheus.yml
curl -s http://node-1:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, discovered: .discoveredLabels}'

Example Root Cause Analysis

A cost-attribution dashboard grouped http_requests_total by team, and one service’s panels went blank. Querying http_requests_total{job="api"} in the expression browser showed the series present but with no team label, while other jobs still had it. up{job="api"} confirmed the target was healthy, so this was a labeling problem, not a scrape failure.

Reading the scrape config revealed a recently added metric_relabel_configs rule with action: labeldrop and regex: team — added to reduce cardinality on a different, high-churn job but applied to the wrong job_name. Because metric_relabel_configs runs after the scrape, the label was collected and then thrown away. Removing that rule from the api job restored team on every sample, and the dashboard’s sum by (team) grouping came back on the next scrape.

Prevention Best Practices

• Know the boundary: relabel_configs shapes target labels before scrape; metric_relabel_configs shapes samples after scrape. Use the right one.
• Map __meta_* discovery labels into real labels explicitly — they are dropped unless you target_label them.
• Keep label names and formats consistent across targets so on() joins line up (especially instance).
• Set honor_labels deliberately for Pushgateway and federation, and document why.
• Run promtool check config in CI and diff discoveredLabels from the targets API after relabeling changes.
• Prefer group_left/group_right with an explicit label list so joins are readable and intentional.

Quick Command Reference

# Validate config and view target labels
promtool check config prometheus.yml
curl -s http://node-1:9090/api/v1/targets \
  | jq '.data.activeTargets[] | {job: .labels.job, labels: .labels}'

# Inspect the real label set on a series
http_requests_total{job="api"}

# See what target labels the server attached
up{job="api"}

# A join that needs a shared, correctly-formatted match label
rate(http_requests_total{job="api"}[5m]) * on(instance) group_left(team) node_meta

Conclusion

Missing labels are almost always self-inflicted by relabeling: a labeldrop, an overwrite, or an unmapped __meta_* label. Confirm the actual label set in the expression browser, decide whether the problem is before scrape (relabel_configs) or after (metric_relabel_configs), and keep join labels consistent across targets. Validate every change with promtool and the targets API. More configuration walkthroughs are in the Prometheus stack guide.