🚨 Grafana Error Guide: 'Failed to migrate legacy alerting' — Fix the Unified Alerting Upgrade

Overview

Grafana raises this at startup when it tries to migrate legacy dashboard-based alerting to unified alerting (Grafana Alerting) and the migration transaction fails, aborting boot. The log reads:

Server shutdown: migration failed: failed to upgrade legacy alerting: alert_rule migration error

A common concrete cause underneath it is an alert attached to a deleted dashboard or datasource:

failed to migrate alert ... could not find dashboard with id 42 for alert "High CPU"

Because the migration runs inside a database transaction that must fully succeed, one bad legacy alert can block the whole instance from starting until it’s resolved or the migration is retried cleanly.

Symptoms

• Grafana fails to start after enabling unified alerting or upgrading to a version where it’s the default.
• The log shows failed to upgrade legacy alerting / alert_rule migration error and the server exits.
• The failure references a specific alert, dashboard id, datasource, or notification channel.
• Rolling back the Grafana version lets it start again on legacy alerting.
• After a partial migration, some alerts exist as unified rules and others are missing.

Common Root Causes

• Orphaned legacy alerts — an alert whose dashboard or panel was deleted, so the migration can’t resolve it.
• Alert referencing a missing datasource that was removed but left the alert behind.
• Duplicate or invalid notification channels that can’t be converted into contact points.
• A corrupt or partially applied prior migration (e.g. after a crash mid-upgrade).
• Insufficient DB permissions/locks during the migration transaction (SQLite locked, Postgres role limits).
• Version skew — jumping across several major versions so the migration path is longer than expected.

Diagnostic Workflow

1. Back up the database first — before anything else. The migration mutates alerting tables:

# SQLite
sqlite3 /var/lib/grafana/grafana.db ".backup '/backup/grafana-pre-migration.db'"
# Postgres
pg_dump -Fc grafana > /backup/grafana-pre-migration.dump

2. Read the exact alert/dashboard the migration choked on. The log names it:

journalctl -u grafana-server --since '15 min ago' | grep -i 'migrat\|legacy alert\|alert_rule'

3. Find orphaned legacy alerts in the DB. Alerts pointing at dashboards that no longer exist are the usual blocker:

-- Legacy alerts whose dashboard is gone
SELECT a.id, a.name, a.dashboard_id
FROM alert a
LEFT JOIN dashboard d ON d.id = a.dashboard_id
WHERE d.id IS NULL;

-- Legacy alerts pointing at a missing datasource
SELECT id, name FROM alert
WHERE panel_id NOT IN (SELECT id FROM dashboard);

Delete or fix the orphans (with a backup in hand), then retry startup.

4. Temporarily start on legacy alerting to clean up. If you must get the instance up, pin legacy in grafana.ini while you fix data:

[alerting]
enabled = true

[unified_alerting]
enabled = false

Then clean the orphaned alerts through the UI (delete the stale alert rules/panels) before re-enabling unified alerting.

5. Force a clean re-migration once data is fixed. Enable unified alerting and, if a partial migration left junk, use the documented reset:

[unified_alerting]
enabled = true

[unified_alerting.upgrade]
clean_upgrade = true   # drops previously-migrated unified alerting data and re-runs

clean_upgrade/force_migration discards partially migrated unified-alerting state — only run it with a backup and understanding you’ll re-migrate from legacy.

6. Verify the result. After a successful boot, confirm rules exist:

curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" \
  http://localhost:3000/api/v1/provisioning/alert-rules | jq 'length'

Example Root Cause Analysis

An instance running for years on legacy alerting was upgraded to a version where unified alerting is mandatory. It refused to boot: failed to upgrade legacy alerting ... could not find dashboard with id 87 for alert "Nightly Batch Lag".

Rolling back let it start. A query joining alert to dashboard found three legacy alerts whose dashboards had been deleted months earlier — the alert rows were never cleaned up, so under legacy alerting they simply sat dormant, but the migration insisted on resolving every alert’s dashboard.

Fix: with a fresh DB backup, the team re-pinned legacy alerting, deleted the three orphaned alerts, then re-enabled unified alerting. The migration completed on the next boot and all valid alerts became unified rules. The blocker was stale data the old alerting engine had tolerated but the migration would not.

Prevention Best Practices

• Take a full DB backup and dry-run in staging before enabling the migration in production.
• Clean up orphaned legacy alerts (deleted dashboards/datasources) before upgrading, while still on legacy.
• Upgrade incrementally across major versions rather than jumping many at once, so the migration path is short.
• Ensure the DB isn’t locked/contended during the upgrade (quiet SQLite, adequate Postgres privileges).
• Keep clean_upgrade/force_migration documented and reserved for a backed-up, understood reset — not a first response.
• Verify alert coverage post-migration with a known test condition so silent gaps are caught immediately.

Quick Command Reference

# Back up before migrating
sqlite3 /var/lib/grafana/grafana.db ".backup '/backup/pre.db'"

# Read the migration failure
journalctl -u grafana-server --since '15 min ago' | grep -i 'migrat\|legacy alert'

# Find orphaned legacy alerts (SQLite/Postgres)
sqlite3 /var/lib/grafana/grafana.db \
  "SELECT a.id,a.name FROM alert a LEFT JOIN dashboard d ON d.id=a.dashboard_id WHERE d.id IS NULL;"

# Confirm unified rules exist after upgrade
curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" \
  http://localhost:3000/api/v1/provisioning/alert-rules | jq 'length'

Conclusion

failed to upgrade legacy alerting aborts startup because the migration runs as an all-or-nothing transaction and one invalid legacy alert — usually orphaned by a deleted dashboard or datasource — blocks the whole thing. Back up the DB, read exactly which alert failed, clean the orphaned data while pinned to legacy alerting, then re-run the migration (using clean_upgrade only with a backup). Cleaning stale alerts before the upgrade prevents the failure entirely.