🚨 Prometheus Error Guide: 'Error loading config (--config.file=/etc/prometheus/prometheus.yml)' Reload Failure

Exact Error Message

When Prometheus cannot parse prometheus.yml at startup, it refuses to boot and logs:

ts=2026-06-27T09:14:02.331Z caller=main.go:1183 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="parsing YAML file /etc/prometheus/prometheus.yml: yaml: line 24: mapping values are not allowed in this context"
ts=2026-06-27T09:14:02.331Z caller=main.go:1186 level=info msg="completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=1.812ms

When you reload a running Prometheus over HTTP, the failure surfaces as an HTTP 400 with the reason in the body:

$ curl -s -XPOST http://localhost:9090/-/reload
failed to reload config: couldn't load configuration (--config.file=/etc/prometheus/prometheus.yml): parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:
  line 31: field scrap_interval not found in type config.plain

If the lifecycle endpoint is not enabled, you instead get a 405:

$ curl -s -XPOST http://localhost:9090/-/reload
Lifecycle API is not enabled.

What the Error Means

Prometheus reads prometheus.yml once at startup and again on every reload (SIGHUP or POST /-/reload). The whole file is parsed into a strongly typed config struct. If parsing fails at startup, the process exits non-zero and the service never comes up. If it fails on reload, the running config is kept and the new one is rejected with HTTP 400 — the server stays up but serves the old config.

Two failure layers exist. The YAML layer rejects malformed syntax (mapping values are not allowed, bad indentation). The schema layer rejects valid YAML that does not match the config struct (field X not found in type config.plain, invalid durations, bad regex). The reload body always names the offending line and reason, so read it first.

Common Causes

1. YAML syntax or indentation errors — tabs instead of spaces, a misaligned list item, or a missing colon (mapping values are not allowed in this context).
2. Unknown field — a typo like scrap_interval or a stanza placed under the wrong parent (field ... not found in type config.plain).
3. Bad regex in a relabel rule — an unbalanced group or invalid RE2 syntax (error parsing regexp).
4. Invalid duration — scrape_interval: 30 instead of 30s, or 1minute (not a valid duration string).
5. Missing secret file path — bearer_token_file or tls_config.cert_file points at a path that does not exist.
6. --web.enable-lifecycle not set — POST /-/reload returns HTTP 405 “Lifecycle API is not enabled.”
7. Environment variable not expanded — Prometheus does not expand ${VAR} unless started with --enable-feature=expand-external-labels (and even then only in external_labels); a literal ${TOKEN} lands in the config.

How to Reproduce the Error

Introduce a schema typo and start Prometheus in the foreground:

# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "node"
    scrap_interval: 10s        # typo: should be scrape_interval
    static_configs:
      - targets: ["localhost:9100"]

prometheus --config.file=/etc/prometheus/prometheus.yml
# -> level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)"
#    err="... field scrap_interval not found in type config.ScrapeConfig"

For the reload path, fix startup but break it after the server is running, then reload:

curl -s -XPOST http://localhost:9090/-/reload
# -> failed to reload config: couldn't load configuration ... yaml: ...

Diagnostic Commands

Validate the file before touching the running server — this is the single most important habit:

promtool check config /etc/prometheus/prometheus.yml

Checking /etc/prometheus/prometheus.yml
  FAILED: parsing YAML file /etc/prometheus/prometheus.yml: yaml: line 24: mapping values are not allowed in this context

Trigger a reload and read the HTTP body (a non-empty body means failure):

curl -s -o /dev/stderr -w '\nHTTP %{http_code}\n' -XPOST http://localhost:9090/-/reload

Confirm what config the server currently has loaded (the last good one survives a failed reload):

curl -s http://localhost:9090/api/v1/status/config | jq -r '.data.yaml' | head -40

Read the service log for the startup or reload error line:

journalctl -u prometheus --no-pager | grep -iE 'Error loading config|reload|err=' | tail -20

Lint YAML structure independently of Prometheus’s schema:

yamllint /etc/prometheus/prometheus.yml || python3 -c 'import yaml,sys; yaml.safe_load(open("/etc/prometheus/prometheus.yml"))'

Step-by-Step Resolution

1. Run promtool check config and read the line number and reason. Never reload a file you have not validated.

2. Fix YAML syntax — replace tabs with spaces (Prometheus YAML is space-indented only), align list items, and ensure every key has a colon:

   scrape_configs:
     - job_name: "node"
       scrape_interval: 10s
       static_configs:
         - targets: ["localhost:9100"]

3. Fix unknown fields — correct typos and verify the stanza is under the right parent. scrape_interval lives under global or a scrape_configs entry, never at the top level.

4. Fix durations and regex — use unit suffixes (30s, 1m, 2h) and test relabel regexes as RE2. A bad regex shows as error parsing regexp.

5. Provide secret files — make sure bearer_token_file, cert_file, and key_file paths exist and are readable by the prometheus user.

6. Enable the lifecycle API so POST /-/reload works instead of returning 405. Add the flag to the unit:

   # /etc/systemd/system/prometheus.service
   ExecStart=/usr/local/bin/prometheus \
     --config.file=/etc/prometheus/prometheus.yml \
     --web.enable-lifecycle

   sudo systemctl daemon-reload && sudo systemctl restart prometheus

7. Expand env vars yourself — Prometheus will not interpolate ${TOKEN}. Render the file with envsubst before deploying, or use a *_file field pointing at the secret:

   envsubst < prometheus.yml.tmpl > /etc/prometheus/prometheus.yml
   promtool check config /etc/prometheus/prometheus.yml

8. Validate, then reload, then confirm:

   promtool check config /etc/prometheus/prometheus.yml \
     && curl -s -XPOST http://localhost:9090/-/reload \
     && curl -s http://localhost:9090/api/v1/status/config | jq -r '.data.yaml' | head

Prevention and Best Practices

• Gate every config change behind promtool check config in CI; reject the merge if it fails.
• Always reload with POST /-/reload (lifecycle enabled) over SIGHUP so you get a clear HTTP 400 body on failure instead of a silent rejection.
• Keep prometheus.yml under version control and render env vars with envsubst at deploy time, not at runtime.
• Alert on a successful reload by watching prometheus_config_last_reload_successful == 0.
• Pin secret material to *_file fields so the YAML never contains literal tokens that break expansion assumptions.
• The free incident assistant can diff a failing config against the last good /api/v1/status/config and point at the offending stanza; see more under Prometheus and monitoring.

Related Errors

• Invalid rule group / parse error — the same reload path fails when rule_files contain a bad PromQL expression.
• Duplicate scrape job name — a specific Error loading config variant caused by two scrape configs sharing a job_name.
• out of order sample — an ingestion-path error that occurs after the config loads successfully.

Frequently Asked Questions

Why does POST /-/reload return HTTP 405? The lifecycle API is disabled by default. Start Prometheus with --web.enable-lifecycle and the endpoint becomes available; until then the server only reloads on SIGHUP.

Does a failed reload take Prometheus down? No. A failed POST /-/reload returns HTTP 400 and the previously loaded config stays active. Only a failed config at startup prevents the process from booting.

Why isn’t my ${TOKEN} environment variable being expanded? Prometheus does not interpolate shell-style variables in prometheus.yml. Render the file with envsubst before deployment, or reference a secret via a *_file field.

How do I see the config Prometheus is actually running? Query GET /api/v1/status/config and read .data.yaml. After a failed reload this still shows the last good config, which confirms the new one was rejected.

Can promtool check config catch every reload failure? It catches YAML and schema errors and validates referenced rule files and secret file paths. It cannot catch runtime issues like an unreachable file_sd path, but it stops the vast majority of Error loading config failures before they reach the server.