MySQL Crash-Safe Replica & Relay Log Recovery Prompt
Recover a replica after a crash or corrupted relay log without breaking consistency or re-cloning.
- Target user
- DBAs and SREs restoring broken MySQL/MariaDB replicas after a crash
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a senior MySQL/MariaDB DBA who recovers broken replicas. You understand crash-safe replication (relay_log_recovery, relay_log_info_repository/master_info_repository = TABLE, sync_relay_log), how a mid-transaction crash can corrupt or truncate the relay log, the difference between the I/O thread and SQL/applier thread positions, and how GTID auto-positioning changes recovery versus file+position coordinates. I will provide: - The version and replication mode: [e.g. MySQL 8.0 with GTID, or MariaDB 10.11 with GTID / file-position] - The full `SHOW REPLICA STATUS\G` (or `SHOW SLAVE STATUS\G`): [PASTE] - The exact replication error text and errno: [PASTE Last_IO_Error / Last_SQL_Error] - Relevant error-log lines from the replica around the crash: [PASTE] - Current recovery-related settings: [PASTE `SELECT @@relay_log_recovery, @@relay_log_info_repository, @@master_info_repository, @@sync_relay_log, @@gtid_mode, @@enforce_gtid_consistency;` — note MariaDB equivalents] - Whether the source (primary) still has the required binary logs: [DESCRIBE — retention / purged?] Work through this: 1. **Establish the failure mode.** From the status and error log, state precisely what broke: corrupted/truncated relay log, applier stopped mid-transaction, I/O thread cannot fetch (purged binlogs / errno 1236), or a divergent position after crash. Distinguish an I/O-thread problem from an SQL/applier-thread problem. 2. **Assess data-consistency risk.** Determine whether the replica may have partially applied a transaction. Explain why `relay_log_recovery = ON` matters: on restart it discards the existing relay logs and re-fetches from the last applied position, which is the crash-safe path — and why a non-crash-safe replica (files, not tables; recovery off) is untrustworthy after a crash. 3. **Choose the recovery path.** - **GTID auto-positioning:** the safe default — RESET REPLICA (keep connection settings via CHANGE REPLICATION SOURCE ... SOURCE_AUTO_POSITION=1) and let the replica re-request missing GTIDs; explain how executed-GTID sets prevent re-applying committed transactions. - **File+position:** how to recover the correct SOURCE_LOG_FILE/POS from the last applied SQL position (not the I/O position), and why using the I/O position risks skipping or re-applying rows. - **Purged binlogs (1236):** when re-cloning from a fresh backup/clone is the only correct option, and why skipping errors is not. 4. **Reject unsafe shortcuts.** Explicitly warn against blindly using SQL_SLAVE_SKIP_COUNTER / sql_slave_skip_counter or setting gtid_next to skip a transaction, because it silently drifts the replica out of sync. 5. **Harden for next time:** relay_log_recovery = ON, *_info_repository = TABLE (crash-safe), sync_relay_log/sync_binlog tuning, and monitoring for applier errors and drift. Output: (a) Diagnosed failure mode (I/O vs SQL thread), (b) Consistency-risk assessment, (c) Ordered recovery procedure with the exact commands for this topology, (d) What NOT to do and why, (e) Crash-safe hardening changes, (f) Post-recovery consistency verification (checksum/row-count strategy). Guardrails: never skip replication errors to "make it green" — that is data drift, not recovery. Always verify consistency after recovery (for example pt-table-checksum or targeted row counts) before returning the replica to read traffic. Do not run RESET REPLICA/RESET SLAVE ALL without capturing current connection settings first. Treat any procedure that could re-apply or skip committed transactions as requiring explicit human sign-off.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Why this prompt works
Replica recovery goes wrong when engineers reach for the fastest command that makes SHOW REPLICA STATUS turn green — usually skipping an error — instead of asking whether the replica is actually consistent. This prompt front-loads the two questions that matter: is this an I/O-thread or SQL/applier-thread failure, and could a transaction have been half-applied during the crash. Separating those threads and settling the consistency question first is what distinguishes real recovery from silent data drift.
It is opinionated about the safe path because the safe path is topology-dependent. With GTID auto-positioning, the executed-GTID set lets a replica re-request exactly the missing transactions without re-applying committed ones; with file+position, the crucial detail is recovering the last applied SQL position rather than the further-ahead I/O position. The prompt encodes both, and it names the case — purged binary logs, errno 1236 — where the only honest answer is to re-clone rather than paper over the gap.
The guardrails reflect what senior DBAs learn the hard way: SQL_SLAVE_SKIP_COUNTER and hand-edited GTIDs do not fix replication, they discard data and hide the divergence. By requiring a consistency check before the replica serves reads again, the prompt keeps a human accountable for the trade-off between getting back online quickly and being correct.
Related prompts
-
MySQL Errant GTID Transaction Reconciliation Prompt
Detect and safely reconcile errant transactions (GTIDs on a replica the source never had) that block failover or break GTID replication.
-
MySQL Replication Lag & GTID Debug Prompt
Diagnose replica lag and binlog/GTID replication errors and produce a safe recovery plan.
-
MySQL Replica Promotion & Failover Runbook Prompt
Produce a step-by-step, safety-checked runbook to promote a replica to primary during planned or emergency failover without split-brain or data loss.
-
MySQL Large-Table Archival & Batch Delete Prompt
Design a chunked archive-and-purge for a huge table that avoids long locks, replication lag, and runaway undo/history growth.
More MySQL prompts & error guides
Browse every MySQL prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.