🚨 Prometheus Error Guide: 'out of sequence m-mapped chunk' — Recover the TSDB Head

Overview

On startup, while replaying the write-ahead log and loading memory-mapped head chunks, Prometheus can hit an inconsistency between a series’ in-order expectation and the m-mapped chunk files on disk:

ts=2026-07-06T07:41:12.664Z caller=head.go:723 level=error component=tsdb msg="Loading on-disk chunks failed, discarding chunk files completely" err="out of sequence m-mapped chunk for series ref 481239, last chunk: [1720248000000, 1720251600000], new: [1720247400000, 1720250000000]"

You may also see the follow-on line as Prometheus decides to rebuild from the WAL:

ts=2026-07-06T07:41:12.665Z caller=head.go:731 level=info component=tsdb msg="Deleting mmap chunk files"

Prometheus stores recent, still-active samples in an in-memory head block. To survive restarts it memory-maps completed head chunks into data/chunks_head/ and records everything in the WAL under data/wal/. This error means a head chunk file references samples whose time range overlaps or precedes what the series already has — an “out of sequence” ordering violation in the on-disk head chunk metadata, usually from an unclean shutdown, a crash mid-flush, or storage that reordered/lost writes. Prometheus protects data integrity by discarding the corrupt chunks_head files and rebuilding the head from the WAL.

Symptoms

• Prometheus logs out of sequence m-mapped chunk for series ref … (and often Loading on-disk chunks failed, discarding chunk files completely) during startup.
• The process starts, but head/WAL replay is slower than usual and recent data may be briefly re-derived from the WAL.
• It follows a hard kill, OOMKill, node crash, power loss, or storage that was moved/snapshotted while Prometheus was running.
• In severe cases (corrupt WAL and head chunks) startup fails outright with a related opening storage failed error.
• prometheus_tsdb_head_series and query results for very recent data may look momentarily incomplete until replay finishes.

Common Root Causes

1. Unclean shutdown / crash — SIGKILL, OOMKill, or node failure interrupted a head-chunk flush, leaving the chunks_head files inconsistent with the WAL.
2. Storage layer reordering or losing writes — networked/overlay storage, a snapshot-restore, or a filesystem that doesn’t honor write ordering under fsync.
3. Running two Prometheus processes on the same data dir — a second instance (e.g. a botched restart or a stale container) writing the same data/ directory.
4. Disk full during a flush — no space left on device mid-write can truncate a chunk file, later read back as out of sequence.
5. Manual tampering / partial copy — copying data/ while Prometheus was live, or restoring only part of the TSDB directory.

Diagnostic Workflow

Read the TSDB errors around startup:

journalctl -u prometheus --no-pager | grep -iE 'out of sequence|mmap|chunks_head|Loading on-disk chunks|Deleting mmap' | tail -20

Confirm no second process is touching the data directory (a classic cause):

sudo lsof +D /var/lib/prometheus/data 2>/dev/null | awk '{print $1,$2}' | sort -u
ps -ef | grep -i '[p]rometheus'

Inspect the TSDB directory structure — head chunks live under chunks_head/, the WAL under wal/:

ls -la /var/lib/prometheus/data/chunks_head/
ls -la /var/lib/prometheus/data/wal/
df -h /var/lib/prometheus

Validate overall TSDB block health (persisted blocks, not the head):

promtool tsdb analyze /var/lib/prometheus/data | head -30

If Prometheus is up, watch replay and head-series metrics to confirm it recovered:

prometheus_tsdb_head_series

rate(prometheus_tsdb_wal_corruptions_total[10m])

Example Root Cause Analysis

A Prometheus pod was OOMKilled during a large series churn. On restart, it logged out of sequence m-mapped chunk for series ref 481239 … followed by Loading on-disk chunks failed, discarding chunk files completely and Deleting mmap chunk files.

The on-call engineer first checked for a second writer with lsof +D and ps -ef — only one process, so this wasn’t a split-brain data-dir conflict. df -h showed the volume was 96% full, and journalctl showed an earlier no space left on device warning right before the OOMKill. The head-chunk flush had been interrupted mid-write by both memory pressure and near-full disk, leaving chunks_head/ inconsistent with the WAL.

Recovery was intentionally minimal. Because Prometheus itself had already chosen to discard and rebuild from the WAL, the engineer let startup complete rather than deleting anything by hand. After startup, prometheus_tsdb_head_series climbed back to its normal level and recent queries returned complete data — the WAL had preserved the samples the corrupt head chunks referenced.

The durable fix targeted the trigger, not the symptom: the memory limit was raised, a TSDB-disk-usage alert was added at 80%, and retention/head memory were tuned so a churn spike couldn’t OOM the process again. Only if Prometheus had failed to start would they have taken the last-resort step of stopping the service and moving chunks_head/ aside (after backing up data/) to force a clean WAL rebuild.

Prevention Best Practices

• Always shut down Prometheus gracefully (SIGTERM, not SIGKILL) so head chunks flush cleanly; give containers a generous terminationGracePeriodSeconds.
• Never point two Prometheus processes at the same data/ directory — enforce it with the lock file and single-writer deployment.
• Right-size memory to avoid OOMKills during series churn, and alert on process_resident_memory_bytes approaching the limit.
• Alert on TSDB disk usage well before full (80%) — a disk-full mid-flush is a common corruption trigger.
• Use durable, write-ordering-respecting storage; avoid snapshotting or copying data/ while Prometheus is running — for backups use promtool tsdb snapshots or the admin snapshot API.
• Keep a recent TSDB snapshot/backup so a truly unrecoverable head can be restored without total data loss.

Quick Command Reference

# Startup TSDB / head-chunk errors
journalctl -u prometheus --no-pager | grep -iE 'out of sequence|mmap|chunks_head|Deleting mmap' | tail -20

# Ensure only one process owns the data dir
sudo lsof +D /var/lib/prometheus/data 2>/dev/null | awk '{print $1,$2}' | sort -u
ps -ef | grep -i '[p]rometheus'

# TSDB directory + disk headroom
ls -la /var/lib/prometheus/data/chunks_head/ /var/lib/prometheus/data/wal/
df -h /var/lib/prometheus

# Analyze persisted TSDB blocks
promtool tsdb analyze /var/lib/prometheus/data | head -30

# LAST RESORT (Prometheus won't start): back up, then move head chunks aside for a clean WAL rebuild
sudo systemctl stop prometheus
sudo cp -a /var/lib/prometheus/data /var/lib/prometheus/data.bak
sudo mv /var/lib/prometheus/data/chunks_head /var/lib/prometheus/data/chunks_head.corrupt
sudo systemctl start prometheus

Conclusion

out of sequence m-mapped chunk for series ref is Prometheus catching an ordering inconsistency between its on-disk head chunks and the WAL, and protecting your data by discarding the corrupt chunks_head files and rebuilding from the log. In most cases the right move is to let it recover on its own: the WAL usually holds the samples the bad chunks referenced. Focus your effort on the trigger — an OOMKill, a full disk, a hard crash, or two processes sharing one data directory — because that is what will otherwise bring the error back. Graceful shutdowns, single-writer deployments, memory and disk headroom, and regular TSDB snapshots turn this from a recurring outage into a one-time, self-healing blip.