Prometheus Scrape Config and Relabeling Deep Dive

Relabeling is where most engineers’ understanding of Prometheus quietly stops. They copy a relabel_configs block from a chart, it works, and they never figure out why. Then one day they need to drop a noisy metric or rewrite a label, they paste something that looks right, and it silently does nothing — or worse, drops every target. I’ve debugged that confusion enough times to write down the model that actually makes relabeling click.

Two different relabeling stages

The single most important thing: there are two relabeling phases and they run at different times on different data.

relabel_configs runs before scraping, on the target’s labels (the meta-labels from service discovery). It decides which targets to scrape and what labels the target carries.
metric_relabel_configs runs after scraping, on every sample’s labels. It decides which metrics to keep and lets you rewrite or drop labels on the metrics themselves.

Put plainly: relabel_configs shapes targets, metric_relabel_configs shapes metrics. Use the first to pick what to scrape, the second to control cardinality. Mixing them up is the root of 90% of relabeling confusion.

The anatomy of a relabel rule

Every relabel rule is the same five-ish fields, and once you internalize the mental model it’s mechanical:

- source_labels: [__meta_kubernetes_pod_label_app]
  separator: ";"
  regex: (.+)
  target_label: app
  replacement: "$1"
  action: replace

Read it as a pipeline: take the values of source_labels, join them with separator, match against regex, and if it matches, perform action. For replace, write replacement (with regex capture groups like $1) into target_label.

The action is the verb that changes everything:

keep — drop the target/metric unless the regex matches. This is your allow-list.
drop — drop the target/metric if the regex matches. Your deny-list.
replace — write into target_label (the default action).
labelmap — copy labels matching a regex into new names.
labeldrop / labelkeep — remove or retain labels by name.
hashmod — hash a label into N buckets, used for sharding scrapes.

Picking targets with keep and drop

The classic Kubernetes pattern: only scrape pods that opt in with an annotation.

relabel_configs:
  # only keep pods annotated prometheus.io/scrape: "true"
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: "true"

  # use the annotated path, defaulting to /metrics
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)

  # rewrite the scrape address to the annotated port
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__

Those __-prefixed labels are special. __address__ is the host:port Prometheus will scrape, __metrics_path__ is the URL path, and __meta_* labels come from service discovery. They’re all dropped before storage unless you copy them to a real label name — which is why the replace rules above promote pod labels into permanent labels.

Controlling cardinality with metric_relabel_configs

This is the lever that saves your TSDB. A noisy exporter floods you with a high-cardinality metric you’ll never query — drop it after scrape:

metric_relabel_configs:
  # drop a notoriously high-cardinality histogram entirely
  - source_labels: [__name__]
    regex: "apiserver_request_duration_seconds_bucket"
    action: drop

  # drop a useless label that explodes cardinality
  - regex: "id"
    action: labeldrop

  # keep only metrics from a specific subsystem
  - source_labels: [__name__]
    regex: "myapp_.*"
    action: keep

The labeldrop rule is the cardinality scalpel. If a metric carries a pod_id or container_hash label that multiplies series count for no analytical value, drop the label and watch your active series fall. Just be careful: dropping a label that distinguishes otherwise-identical series causes Prometheus to error on duplicate series, so drop labels that are genuinely redundant.

Sharding scrapes with hashmod

When one Prometheus can’t scrape everything, hashmod splits targets deterministically across replicas:

relabel_configs:
  - source_labels: [__address__]
    modulus: 3
    target_label: __tmp_shard
    action: hashmod
  - source_labels: [__tmp_shard]
    regex: "0"          # this replica handles shard 0
    action: keep

Each replica keeps only its shard. Hashing on __address__ ensures a given target consistently lands on the same replica, so you can scale scraping horizontally without overlap.

Debugging relabeling without guessing

Two techniques save hours:

Use the /service-discovery page. Prometheus’ web UI shows, per target, the meta-labels before relabeling and the final labels after, plus whether the target was dropped and why. This is the ground truth — stop reasoning in your head and read what Prometheus actually computed.
Build regexes incrementally. Start with regex: (.+) and action: keep to confirm the source label has the value you think, then tighten. Most “it does nothing” bugs are a meta-label name typo — __meta_kubernetes_pod_label_app vs __meta_kubernetes_pod_labelpresent_app — that the service-discovery page reveals instantly.

The mental model that sticks

Targets first (relabel_configs), metrics second (metric_relabel_configs). keep is allow-list, drop is deny-list. __-prefixed labels are scaffolding that vanishes unless you promote it. And when in doubt, the service-discovery page shows you exactly what Prometheus saw. Hold those four ideas and relabeling stops being voodoo.

For the cardinality discipline this enables, see our metric-cardinality and scrape-tuning guides in the Prometheus and monitoring category. And our monitoring alert assistant can review the rules you build on top of these relabeled metrics.

Relabeling semantics are stable but meta-label names depend on your service discovery. Confirm names on the targets and service-discovery pages of your own Prometheus.