🚨 Prometheus Error Guide: 'opening storage failed' TSDB / WAL Corruption

Overview

opening storage failed is the fatal error Prometheus logs when it cannot open its local TSDB on startup. Prometheus stores data as a write-ahead log (WAL) plus immutable on-disk blocks under its data directory. On boot it replays the WAL into memory and loads block metadata. If the WAL was truncated mid-write (unclean shutdown, OOM kill, disk full) or a block’s index/meta is corrupt, the open fails and Prometheus exits instead of serving with bad data.

You will see this at startup, followed by the process exiting:

ts=2026-06-23T14:18:01.554Z caller=main.go:1126 level=error msg="Opening storage failed" err="opening storage failed: /prometheus/wal/00001234: invalid checksum"

Other variants name a corrupt block or a WAL segment:

err="opening storage failed: block dir: \"/prometheus/01HXYZ...\": meta.json: unexpected end of JSON input"

err="opening storage failed: read WAL: corruption after offset 17825792 in segment 1234"

It is a startup-blocking condition: the service will crash-loop until the corrupt artifact is repaired or removed. The good news is that the WAL repair path and block-level corruption are usually recoverable with at most a small, bounded data loss.

Symptoms

• Prometheus crash-loops on start, logging Opening storage failed and exiting non-zero.
• systemd shows the unit in activating (auto-restart) or failed.
• The error names a specific WAL segment offset or a block directory / meta.json.
• The event follows an OOM kill, a kill -9, a power loss, or a disk-full condition.

systemctl status prometheus --no-pager | grep -E 'Active|Main PID|status'

   Active: activating (auto-restart) (Result: exit-code) since ...

Common Root Causes

1. Unclean shutdown truncated the WAL

A kill -9, container OOM, or power loss can leave the last WAL segment half-written, failing the checksum on replay.

journalctl -u prometheus --no-pager | grep -iE 'corruption|invalid checksum|unexpected end' | tail -5

err="read WAL: corruption after offset 17825792 in segment 1234"

A named segment offset is the signature of a truncated WAL — usually repairable in place.

2. Disk full during a write

If the data volume filled while Prometheus was writing, the WAL or a block gets partially written.

df -h /prometheus

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1    100G  100G     0  100% /prometheus

100% Use% at the time of the crash means writes were truncated; you must free space before any recovery will hold.

3. OOM kill mid-compaction left a corrupt block

The kernel OOM-killer terminating Prometheus during block compaction can leave a block with an incomplete index or meta.json.

dmesg -T | grep -i 'killed process' | grep -i prometheus | tail -3

[Tue Jun 23 14:17:51 2026] Out of memory: Killed process 4412 (prometheus) total-vm:...

Pair an OOM timestamp with a “block dir … meta.json” error to confirm a compaction-time corruption.

4. A single corrupt block (bad index or meta.json)

One block’s metadata is unreadable while the rest of the TSDB is fine. Identify the block:

ls -1 /prometheus | grep -E '^[0-9A-Z]{26}$'
cat /prometheus/01HXYZABCDEF0123456789ABCD/meta.json 2>&1 | head

cat: .../meta.json: unexpected end of JSON input

The named block is the only casualty; removing it costs just that block’s time window.

5. WAL replay exhausting memory (looks like a hang, then OOM)

A very large WAL after a long outage can OOM on replay, creating a loop: replay -> OOM -> truncated WAL -> corruption.

ls -lh /prometheus/wal/ | tail
du -sh /prometheus/wal

2.7G    /prometheus/wal

A multi-gigabyte WAL on a memory-limited instance replays slowly and risks OOM; raise the memory limit for the recovery boot.

6. Filesystem or permission damage

Underlying filesystem errors or wrong ownership after a volume restore block the open entirely.

ls -ld /prometheus
touch /prometheus/.write-test && echo "writable" || echo "NOT writable"

drwxr-xr-x 4 root root 4096 Jun 23 14:00 /prometheus
NOT writable

If the data dir is owned by root but Prometheus runs as nobody/prometheus, the open fails on permissions, not corruption.

Diagnostic Workflow

Step 1: Read the exact failure artifact

journalctl -u prometheus --no-pager | grep -iE 'Opening storage failed' | tail -3

The message names either a WAL segment+offset (repairable) or a block dir (removable) — this decides the recovery path.

Step 2: Rule out disk-full and permissions first

df -h /prometheus
ls -ld /prometheus && id prometheus

Free space and correct ownership are prerequisites; fix these before touching the WAL or blocks.

Step 3: Back up the data directory

# Stop the service, then snapshot before any repair
systemctl stop prometheus
cp -a /prometheus /prometheus.bak-$(date +%F)

Always copy before repairing so a failed recovery is reversible.

Step 4a: Repair a corrupt WAL (segment error)

Prometheus repairs a truncated WAL automatically on the next start by dropping everything after the corruption point. If it does not self-heal, remove the trailing corrupt segment named in the error:

ls -1 /prometheus/wal/
# Remove only the last, corrupt segment named in the log:
rm /prometheus/wal/00001234
systemctl start prometheus

You lose only the un-flushed samples after the corruption offset.

Step 4b: Remove a single corrupt block (block-dir error)

# Verify it is the named block, then move it aside:
mv /prometheus/01HXYZABCDEF0123456789ABCD /tmp/corrupt-block/
systemctl start prometheus

You lose only that block’s ~2h (or compacted) window; all other blocks load normally.

Step 5: Start and confirm clean replay

systemctl start prometheus
journalctl -u prometheus --no-pager -f | grep -iE 'WAL|replay|head|TSDB started'

level=info msg="WAL segment loaded" ...
level=info msg="TSDB started"

TSDB started confirms a clean open.

Example Root Cause Analysis

Prometheus crash-loops after the node was abruptly power-cycled. The unit is in auto-restart and the log shows:

err="opening storage failed: read WAL: corruption after offset 9437184 in segment 1102"

Disk and permissions are fine:

df -h /prometheus

/dev/nvme1n1    100G   58G   42G  59% /prometheus

So this is a truncated WAL from the power loss, not a disk-full or permission issue — the named segment 1102 was half-written. After stopping the service and backing up the directory, the trailing corrupt segment is removed:

systemctl stop prometheus
cp -a /prometheus /prometheus.bak-2026-06-23
ls -1 /prometheus/wal/ | tail -3

00001100
00001101
00001102

rm /prometheus/wal/00001102
systemctl start prometheus

Prometheus replays the remaining WAL cleanly and logs TSDB started. The only loss is the few seconds of in-flight samples after offset 9437184 that were never durably written — well within the acceptable window.

Prevention Best Practices

• Give Prometheus a graceful shutdown: set a generous systemd/container stop timeout so the WAL flushes; avoid kill -9 and OOM kills, which are the top corruption source.
• Right-size memory and alert on process_resident_memory_bytes vs the limit so compaction/replay never triggers an OOM mid-write.
• Alert on disk usage of the data volume well before it fills; a full disk during a write reliably corrupts the WAL or a block.
• Keep retention and block sizes sane, and back up via the TSDB snapshot admin API (POST /api/v1/admin/tsdb/snapshot) rather than copying a live directory.
• For durability beyond the local TSDB, ship to a remote-write long-term store so a local corruption never means total data loss.
• The free incident assistant can read the startup error and tell you whether it’s a WAL-repair or block-removal recovery; more storage guidance is under Prometheus and monitoring.

Quick Command Reference

# Exact failure artifact (WAL segment vs block dir)
journalctl -u prometheus --no-pager | grep -iE 'Opening storage failed' | tail -3

# Prerequisites: disk and permissions
df -h /prometheus
ls -ld /prometheus && id prometheus

# Back up before any repair
systemctl stop prometheus
cp -a /prometheus /prometheus.bak-$(date +%F)

# Repair WAL: remove the named trailing corrupt segment
ls -1 /prometheus/wal/ && rm /prometheus/wal/<SEGMENT>

# Remove a single corrupt block
mv /prometheus/<BLOCK_ULID> /tmp/corrupt-block/

# Start and watch replay
systemctl start prometheus
journalctl -u prometheus --no-pager -f | grep -iE 'WAL|replay|TSDB started'

Conclusion

opening storage failed means Prometheus refused to start on a corrupt WAL or block. Recover methodically:

1. Read the error — it names a WAL segment+offset (repair) or a block dir (remove).
2. Rule out disk-full and permissions before touching data.
3. Back up the data directory so the repair is reversible.
4. Drop the trailing corrupt WAL segment, or move the single corrupt block aside.
5. Start and confirm TSDB started with a clean replay.

The data loss is bounded to the un-flushed tail or one block. Prevent recurrence by eliminating ungraceful shutdowns, OOM kills, and full disks — and by shipping to a remote store so local corruption is never catastrophic.