🚨 Grafana Error Guide: 'migration failed' on Startup — Fix Broken Schema Migrations

Overview

Every time Grafana starts, it compares its expected schema version against the backend database and runs any pending schema migrations before it serves traffic. This is normal and usually invisible — you’ll see Starting DB migrations in the logs and then the web server comes up. But when a migration cannot be applied, Grafana logs a fatal error and refuses to start. The database is left in a partially-migrated state, and every subsequent restart fails on the same step.

The literal errors you’ll see in journalctl -u grafana-server or kubectl logs deploy/grafana:

Starting DB migrations
migration failed (id = create index UQE_user_login - v2): duplicate key name 'UQE_user_login'

Failed to migrate

Error: ✗ migration failed (id = add index alert_notification_state ...): Error 1071: Specified key was too long; max key length is 767 bytes

Sync failed

Unlike a datasource error, this stops the whole application. Migrations are transactional per-step on Postgres but not fully transactional on MySQL, so an interrupted upgrade can leave half-applied DDL that the next run trips over.

Symptoms

• Grafana crash-loops right after logging Starting DB migrations.
• Logs show migration failed (id = ...) naming a specific migration id.
• The failure appears immediately after a version upgrade or downgrade.
• MySQL-specific Error 1071 or Error 1709 key-length errors.
• duplicate column name / duplicate key name on a step that was partly applied.

Common Root Causes

1. Interrupted prior upgrade left a half-applied migration

If a previous Grafana start was killed (OOM, node eviction, kubectl delete pod) mid-migration on MySQL, the DDL partially applied but the migration_log row was never written as successful. The next boot retries the step and hits an object that already exists.

migration failed (id = add unique index user.login): duplicate key name 'UQE_user_login'

Inspect the migration bookkeeping table:

SELECT migration_id, success, error, timestamp
FROM migration_log
WHERE success = 0
ORDER BY timestamp DESC;

2. MySQL utf8mb4 index key length limit

Grafana requires utf8mb4. On older MySQL (5.6 / MariaDB) with innodb_large_prefix off and ROW_FORMAT=COMPACT, a 4-byte charset makes indexed VARCHAR columns exceed the 767-byte prefix limit.

[database]
type = mysql
host = 10.0.0.5:3306
name = grafana
user = grafana

Error: ✗ migration failed (id = add index ...): Error 1071: Specified key was too long; max key length is 767 bytes
Error 1709: Index column size too large. The maximum column size is 767 bytes.

Fix at the server level:

SET GLOBAL innodb_large_prefix = ON;
SET GLOBAL innodb_file_format = Barracuda;
SET GLOBAL innodb_default_row_format = DYNAMIC;

Then ensure the Grafana database uses CHARACTER SET utf8mb4 and tables use ROW_FORMAT=DYNAMIC.

3. Downgrade to an older Grafana version

Migrations only go forward. If you roll a deployment back to an older image after the newer version already migrated the schema, the old binary sees columns/tables it doesn’t understand — or the newer schema violates an old assumption.

migration failed (id = ...): Error 1054: Unknown column 'is_service_account' in 'field list'

The correct recovery is to roll forward to the version that matches the schema, or restore a pre-upgrade DB backup.

4. DB user lacks DDL permissions

Migrations issue CREATE TABLE, ALTER TABLE, CREATE INDEX. A user with only SELECT/INSERT/UPDATE/DELETE fails the moment a new migration ships.

migration failed (id = create table ...): Error 1142: CREATE command denied to user 'grafana'@'%' for table 'alert_rule'

GRANT ALL PRIVILEGES ON grafana.* TO 'grafana'@'%';
FLUSH PRIVILEGES;

5. Dirty or manually-edited migration_log

Someone deleted rows from migration_log, or a failed row (success = 0) blocks the sequence. Grafana treats a logged failure as a hard stop.

Sync failed
migration already run but marked as failed (id = ...)

Diagnostic Workflow

Step 1 — Capture the failing migration id. It’s the single most important datum.

journalctl -u grafana-server -n 80 --no-pager | grep -iE "migration|Sync failed|Starting DB"
kubectl logs deploy/grafana --tail=80 | grep -i migration

Step 2 — Inspect the migration_log table to see what already ran and which step failed.

mysql -h 10.0.0.5 -u grafana -p grafana \
  -e "SELECT migration_id, success FROM migration_log WHERE success=0;"
# Postgres:
psql -h 10.0.0.5 -U grafana -d grafana \
  -c "SELECT migration_id, success FROM migration_log WHERE success = false;"

Step 3 — Confirm charset/row format (MySQL).

mysql -h 10.0.0.5 -u grafana -p grafana -e "
  SELECT DEFAULT_CHARACTER_SET_NAME FROM information_schema.SCHEMATA WHERE SCHEMA_NAME='grafana';
  SHOW VARIABLES LIKE 'innodb_large_prefix';
  SHOW VARIABLES LIKE 'innodb_default_row_format';"

Step 4 — Verify the DB user’s grants.

mysql -h 10.0.0.5 -u grafana -p -e "SHOW GRANTS FOR CURRENT_USER();"

Step 5 — Check the running version vs the schema. A mismatch after a rollback points at a downgrade.

grafana-cli --version

Example Root Cause Analysis

An operator upgraded Grafana from 9.5 to 10.2. The pod OOM-killed mid-startup, was rescheduled, and then entered CrashLoopBackOff with:

Starting DB migrations
migration failed (id = add unique index dashboard_org_id_folder_id_title): duplicate key name 'UQE_dashboard_org_id_folder_id_title'

Querying migration_log showed the step was not recorded as successful (success = 0), yet the index already existed — the classic signature of a MySQL migration interrupted between the DDL and the log write. Recovery: take a DB backup, drop the orphaned index manually (ALTER TABLE dashboard DROP INDEX UQE_dashboard_org_id_folder_id_title;), delete the failed migration_log row, and restart. Grafana re-ran the step cleanly, wrote success = 1, and continued through the remaining migrations. Root cause: an OOM kill during a non-transactional MySQL migration.

Prevention Best Practices

• Always back up the database immediately before an upgrade — mysqldump / pg_dump. A backup makes every migration failure recoverable.
• Give Grafana enough memory so it isn’t OOM-killed mid-migration; migrations on large tables spike memory.
• Use utf8mb4 + DYNAMIC row format from day one on MySQL to avoid the 767-byte trap.
• Never downgrade across a schema change — pin image tags and roll forward.
• Grant DDL privileges to the Grafana user permanently, not just at install.
• Run a single instance during upgrades (scale to 1 replica) so two pods don’t race the same migration.
• Read the release notes for breaking schema changes before jumping major versions.

Quick Command Reference

# Find the failing migration id
journalctl -u grafana-server -n 80 --no-pager | grep -i migration
kubectl logs deploy/grafana --tail=80 | grep -i migration

# Inspect migration bookkeeping
mysql -h 10.0.0.5 -u grafana -p grafana -e "SELECT migration_id,success FROM migration_log WHERE success=0;"
psql  -h 10.0.0.5 -U grafana -d grafana -c "SELECT migration_id,success FROM migration_log WHERE success=false;"

# MySQL charset / row-format checks
mysql -e "SHOW VARIABLES LIKE 'innodb_large_prefix';"
mysql -e "SHOW VARIABLES LIKE 'innodb_default_row_format';"

# Grants
mysql -h 10.0.0.5 -u grafana -p -e "SHOW GRANTS FOR CURRENT_USER();"

# Backup BEFORE touching anything
mysqldump -h 10.0.0.5 -u grafana -p grafana > grafana-backup.sql
pg_dump   -h 10.0.0.5 -U grafana grafana > grafana-backup.sql

# Restart after fixing
systemctl restart grafana-server
kubectl rollout restart deploy/grafana

Conclusion

When Grafana logs migration failed / Failed to migrate and won’t start, diagnose in this order:

1. Interrupted prior upgrade — orphaned DDL plus a success = 0 row in migration_log; drop the partial object and clear the failed row.
2. utf8mb4 key-length limit — Error 1071 / Error 1709; enable innodb_large_prefix and ROW_FORMAT=DYNAMIC.
3. Downgrade to an older version — Unknown column; roll forward or restore a backup, never downgrade across schema changes.
4. Missing DDL grants — CREATE command denied; grant ALTER/CREATE to the Grafana user.
5. Dirty migration_log — a lingering failed row blocks the sequence; fix the underlying cause, then repair the row.

See the Grafana category and the related guide on database connection failures for adjacent backend problems.