🚨 Prometheus Error Guide: 'found multiple scrape configs with job name' Duplicate Job

Exact Error Message

Two scrape configs sharing a job_name fail the config load with a precise message:

ts=2026-06-27T11:21:09.402Z caller=main.go:1183 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="found multiple scrape configs with job name \"node\""

promtool reports the same failure:

$ promtool check config /etc/prometheus/prometheus.yml
Checking /etc/prometheus/prometheus.yml
  FAILED: found multiple scrape configs with job name "node"

On a live reload it returns HTTP 400:

$ curl -s -XPOST http://localhost:9090/-/reload
failed to reload config: couldn't load configuration (--config.file=/etc/prometheus/prometheus.yml): found multiple scrape configs with job name "node"

What the Error Means

Every entry under scrape_configs (and every file pulled in via scrape_config_files) must have a unique job_name. The job_name becomes the job label on all series from that config and is the primary key Prometheus uses to track a scrape pool. When two configs declare the same name, Prometheus cannot tell which pool a series belongs to, so it rejects the entire config at load time. Like all config errors, this fails the whole file — at startup the process won’t boot; on reload the previous config is kept and HTTP 400 is returned.

This is distinct from two targets producing the same series (an ingestion-time out of order problem). Here the collision is at the config level, caught before any scrape runs.

Common Causes

1. Two scrape_configs entries share job_name — usually copy-paste, leaving both blocks named "node".
2. An included file plus inline config collide — scrape_config_files pulls in a file defining job_name: node while prometheus.yml also defines it inline.
3. file_sd and static_configs collision — a job using file_sd_configs is duplicated by a second static job under the same name.
4. Glob in scrape_config_files matches the same job twice — two files in the glob both define the same job_name.
5. Helm / operator additionalScrapeConfigs duplicating a default job — your chart values add a job named node or kubernetes-pods that the chart already ships.
6. Templated config rendering the same name twice — a loop in a Jinja/envsubst template emitting duplicate job_name values.

How to Reproduce the Error

Define the same job_name in two places:

# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s

scrape_config_files:
  - /etc/prometheus/scrape.d/*.yml

scrape_configs:
  - job_name: "node"
    static_configs:
      - targets: ["10.0.0.11:9100"]

# /etc/prometheus/scrape.d/node.yml
scrape_configs:
  - job_name: "node"                       # collides with the inline job above
    static_configs:
      - targets: ["10.0.0.12:9100"]

promtool check config /etc/prometheus/prometheus.yml
# -> FAILED: found multiple scrape configs with job name "node"

Diagnostic Commands

List every job_name across the main file and all included scrape files, then find duplicates:

grep -rhoE '^\s*-?\s*job_name:\s*.*' /etc/prometheus/prometheus.yml /etc/prometheus/scrape.d/ \
  | sed -E 's/.*job_name:\s*//; s/["'"'"']//g' \
  | sort | uniq -d

node

Validate the merged config (this is what catches the collision authoritatively):

promtool check config /etc/prometheus/prometheus.yml

Inspect the currently loaded config to see which jobs the running server already has:

curl -s http://localhost:9090/api/v1/status/config | jq -r '.data.yaml' | grep -E 'job_name'

Confirm the reload result and read the log:

curl -s -o /dev/stderr -w '\nHTTP %{http_code}\n' -XPOST http://localhost:9090/-/reload
journalctl -u prometheus --no-pager | grep -iE 'multiple scrape configs|Error loading config' | tail -10

For Kubernetes operator setups, check the rendered secret rather than your values file:

kubectl -n monitoring get secret prometheus-k8s -o jsonpath='{.data.prometheus\.yaml\.gz}' \
  | base64 -d | gunzip | grep -E 'job_name'

Step-by-Step Resolution

1. Find the duplicate with the grep | sort | uniq -d one-liner above. It lists every job_name defined more than once across all included files.

2. Decide: merge or rename. If both blocks scrape the same logical fleet, merge their targets into one job:

   scrape_configs:
     - job_name: "node"
       static_configs:
         - targets: ["10.0.0.11:9100", "10.0.0.12:9100"]

3. If they are genuinely different fleets, give each a distinct name:

   scrape_configs:
     - job_name: "node-prod"
       static_configs:
         - targets: ["10.0.0.11:9100"]
     - job_name: "node-staging"
       static_configs:
         - targets: ["10.0.0.12:9100"]

4. Resolve inline vs included collisions. If scrape_config_files already provides the job, remove the inline duplicate (or vice versa). Keep each job defined in exactly one file.

5. Fix operator/Helm duplicates. If your additionalScrapeConfigs re-declares a default job, rename it or remove it. Better, model the target as a ServiceMonitor/PodMonitor so the operator owns the job and avoids hand-written collisions entirely:

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     name: node-exporter
     namespace: monitoring
   spec:
     selector:
       matchLabels: { app: node-exporter }
     endpoints:
       - port: metrics

6. Validate, reload, confirm:

   promtool check config /etc/prometheus/prometheus.yml \
     && curl -s -XPOST http://localhost:9090/-/reload \
     && curl -s http://localhost:9090/api/v1/status/config | jq -r '.data.yaml' | grep job_name

Prevention and Best Practices

• Run promtool check config in CI; it deterministically catches duplicate job_name before deploy.
• Keep one job per file when using scrape_config_files, and name the file after the job to make collisions visually obvious.
• Never copy a scrape block without changing its job_name; treat job_name as a unique key.
• In Kubernetes, prefer ServiceMonitor/PodMonitor over additionalScrapeConfigs so the operator guarantees uniqueness.
• Add a pre-commit hook running the grep ... | uniq -d check across your scrape directory.
• The free incident assistant can scan a multi-file scrape layout and report which files declare the same job; see more under Prometheus and monitoring.

Related Errors

• Config reload failed / HTTP 400 — the general Error loading config family; duplicate job_name is one specific case.
• Invalid rule group / parse error — the rule-file equivalent of a load-time rejection.
• out of order sample — what you get after the config loads if two distinct jobs still produce identical series.

Frequently Asked Questions

Can two scrape configs ever share a job_name? No. job_name must be globally unique across prometheus.yml and every file in scrape_config_files. Duplicates reject the whole config.

My duplicate is across two files — why does grep on prometheus.yml alone miss it? Because the collision is between an inline job and one pulled in via scrape_config_files. Grep across the main file and the included directory together.

Should I merge the jobs or rename them? Merge when both blocks scrape the same logical fleet (combine their targets). Rename when they are different fleets that happen to share a copied name.

How do I find the duplicate in a Helm/operator setup? Decode the rendered Prometheus secret (prometheus.yaml.gz) and grep job_name there. Your values file may look clean while the chart’s defaults plus your additionalScrapeConfigs collide after rendering.

Does this error stop the whole server? At startup, yes — the process exits. On reload, the running config is kept and you get HTTP 400, so the server keeps scraping with the last good config until you fix and reload.