🚨 Grafana Error Guide: 'Alertmanager is not ready' — Fix Grafana Alerting Notifications

Overview

Grafana Alerting raises this when the Alertmanager component — the part that receives firing alerts and dispatches notifications — has not finished initializing or has rejected its configuration. The Alerting UI and API return:

the Alertmanager is not ready: config not available or invalid

For an external Alertmanager wired as a datasource, the equivalent is:

failed to load Alertmanager config: alertmanager not initialised

Until the Alertmanager is ready, firing alert rules evaluate but their notifications are never delivered — a dangerous silent gap, because rules look healthy while pages simply don’t go out.

Symptoms

• The Alerting → Contact points / Notification policies pages show “Alertmanager is not ready”.
• Alert rules fire (visible as firing in the Alerting UI) but no notifications are sent.
• Saving a contact point or notification policy fails with an invalid-config error.
• An external Alertmanager datasource health check fails or reports not initialised.
• In HA, only some Grafana replicas report ready; notifications are inconsistent.

Common Root Causes

• Invalid notification configuration — a contact point or template with a syntax error that the Alertmanager refuses to load, leaving it without a valid config.
• A referenced secret/template missing — a contact point points at an env var or template that doesn’t exist.
• External Alertmanager unreachable or unhealthy when Grafana is configured to use one instead of the built-in.
• Clustering/gossip problems in HA (multiple Grafana replicas or an external Alertmanager cluster) so no node reaches a ready state.
• Startup ordering — the Alertmanager hasn’t finished loading on a fresh boot; transient during startup.
• Corrupt or partially-migrated alerting state after an upgrade, so the stored config is invalid.

Diagnostic Workflow

1. Read the readiness reason in the log. Grafana logs why the Alertmanager isn’t ready:

journalctl -u grafana-server --since '10 min ago' | grep -i 'alertmanager\|notifier\|contact point'

Look for invalid config, a named template, or a missing receiver — that names the offending piece.

2. Check the Alerting API health. The built-in Alertmanager exposes status:

# Built-in Grafana Alertmanager status
curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" \
  http://localhost:3000/api/alertmanager/grafana/config/api/v1/status | jq '.cluster,.config != null'

If config is null, the Alertmanager has no valid config loaded.

3. Validate the notification config. Pull the current config and check contact points/templates for the flagged error:

curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" \
  http://localhost:3000/api/v1/provisioning/contact-points | jq '.[].name'
curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" \
  http://localhost:3000/api/v1/provisioning/templates | jq 'keys'

A template with a Go-template syntax error (unclosed {{ }}) will block the whole Alertmanager.

4. Fix invalid provisioned config. If contact points/policies are provisioned as YAML, correct the syntax and reload:

apiVersion: 1
contactPoints:
  - orgId: 1
    name: slack-critical
    receivers:
      - uid: slack-critical-1
        type: slack
        settings:
          url: ${SLACK_WEBHOOK_URL}   # must resolve; missing secret => not ready

Ensure every referenced secret/env var actually exists.

5. For an external Alertmanager, verify it’s healthy and reachable:

# From the Grafana host
curl -s http://alertmanager:9093/-/ready
amtool check-config /etc/alertmanager/alertmanager.yml

Confirm the Grafana alertmanager datasource points at the right URL and implementation.

6. Reset the config if it’s corrupt. As a last resort (with a DB backup), restore a known-good notification config via provisioning or the API, then restart Grafana and re-check readiness.

Example Root Cause Analysis

A team added a custom notification template to make Slack alerts richer, saved it, and within minutes on-call stopped receiving pages — yet the Alerting UI clearly showed rules firing. The Contact points page displayed “the Alertmanager is not ready”.

The Grafana log pointed straight at it: failed to load config: template: :1: unexpected "}" in operand. The new template had an unbalanced {{ }}. Because the Alertmanager loads all templates as one config unit, that single syntax error made the entire Alertmanager config invalid, so it never became ready and dispatched nothing — even though every other contact point was fine.

Fix: corrected the template’s braces (validated the Go template locally), re-saved, and the Alertmanager reloaded to ready immediately; queued notifications flushed. The scary part was the silence: firing rules with zero delivery. They added an alert on Alertmanager readiness so the next bad config would page a fallback channel instead of failing silently.

Prevention Best Practices

• Validate notification templates and contact points (locally or in staging) before saving — one syntax error disables all notifications.
• Reference only existing secrets/env vars in contact points; a missing webhook URL keeps the Alertmanager un-ready.
• Monitor Alertmanager readiness and alert via an independent channel, so a bad config pages a fallback instead of going silent.
• Provision alerting config as reviewed code, not ad-hoc UI edits, so changes are validated in CI.
• Keep external Alertmanager healthy and its cluster quorate, and confirm the datasource URL/implementation.
• Back up alerting config before upgrades so a corrupt migration can be restored to a known-good state.

Quick Command Reference

# Read why the Alertmanager isn't ready
journalctl -u grafana-server --since '10 min ago' | grep -i 'alertmanager\|template\|contact point'

# Built-in Alertmanager status (is config loaded?)
curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" \
  http://localhost:3000/api/alertmanager/grafana/config/api/v1/status | jq '.config != null'

# Inspect contact points and templates
curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" \
  http://localhost:3000/api/v1/provisioning/contact-points | jq '.[].name'

# External Alertmanager health
curl -s http://alertmanager:9093/-/ready && amtool check-config /etc/alertmanager/alertmanager.yml

Conclusion

“The Alertmanager is not ready” means Grafana’s notification dispatcher has no valid config loaded — usually a single invalid contact point or notification template that invalidates the whole config, or an unreachable external Alertmanager. The danger is silent: rules fire but nothing is delivered. Read the log for the named error, validate contact points and templates, confirm every referenced secret exists, and monitor Alertmanager readiness on an independent channel so bad config never quietly stops your pages.