🚨 Logstash Error Guide: 'java.io.IOException: Page file ... different size' — Recover a Corrupt Persistent Queue

Overview

With queue.type: persisted, Logstash stores in-flight events in page files under path.queue and tracks position with checkpoint files. If those files are truncated or inconsistent — usually after a crash, disk-full, or power loss mid-write — the queue fails to open on startup:

[FATAL][logstash.runner] An unexpected error occurred!
{:error=>#<Java::JavaIo::IOException: Page file size is 8399 but expected 67108864>,
:message=>"Page file size is 8399 but expected 67108864",
:backtrace=>["org.logstash.ackedqueue.io.MmapPageIOV2.open(MmapPageIOV2.java:...)",
"org.logstash.ackedqueue.Queue.openPage(Queue.java:...)",
"org.logstash.ackedqueue.Queue.recover(Queue.java:...)"]}

Variants include Page file ... exists but is different size than expected, checkpoint ... doesn't exist, or Element index is out of bounds. All mean the persistent queue on disk is inconsistent and Logstash refuses to start rather than replay corrupt data.

Symptoms

• Startup aborts with java.io.IOException referencing a page.N file, checkpoint, or MmapPageIO in the backtrace.
• The failure began right after an OOM kill, kill -9, node reboot, or a full path.data volume.
• The systemd unit crash-loops; ingest is fully stopped because the pipeline never opens.
• path.queue/<pipeline>/ contains page.* and checkpoint.* files, some unexpectedly small or zero-byte.
• df shows (or recently showed) 100% usage on the volume holding path.data.

Common Root Causes

• Ungraceful shutdown mid-write — kill -9, OOM kill, or power loss truncated a page or checkpoint file.
• Disk full during a queue write — path.data volume hit 100%, leaving a partially written page.
• Filesystem corruption — an unclean unmount or storage fault damaged the queue files.
• Shared/relocated path.queue — the queue directory was moved, copied incompletely, or shared between instances.
• Version downgrade — starting an older Logstash against a newer on-disk queue format.
• Manual tampering — someone deleted or edited page/checkpoint files by hand.

Diagnostic Workflow

Read the exact IOException and note which page/checkpoint it names:

grep -A15 'java.io.IOException' /var/log/logstash/logstash-plain.log | tail -25

Locate the queue and inspect the page/checkpoint files for anomalies (zero-byte or wrong-size pages):

grep -E 'path.queue|queue.type|path.data' /etc/logstash/logstash.yml
ls -la /var/lib/logstash/queue/main/
du -sh /var/lib/logstash/queue/main/

Check the volume for the disk-full condition that often causes truncation:

df -h /var/lib/logstash
journalctl -u logstash --since '2 hours ago' | grep -Ei 'space|ioexception|oom|killed'

Confirm the queue settings so you know how much data is at risk. In logstash.yml:

queue.type: persisted
path.queue: "/var/lib/logstash/queue"
queue.max_bytes: 4gb
queue.checkpoint.writes: 1024      # lower = more durable, higher = faster

If the data in the queue is expendable, stop Logstash and reset the corrupt pipeline queue so it starts clean:

sudo systemctl stop logstash
# Back up first in case you want to attempt salvage:
sudo mv /var/lib/logstash/queue/main /var/lib/logstash/queue/main.corrupt.$(date +%s)
sudo systemctl start logstash

Verify the pipeline opens and events flow to Elasticsearch again:

curl -s localhost:9600/_node/stats/pipelines?pretty | grep -A6 '"queue"'
curl -s 'http://es:9200/_cat/indices/logstash-*?v&h=index,docs.count'

Example Root Cause Analysis

A Logstash node ran queue.type: persisted on a 20 GB volume shared with system logs. Overnight, a log flood filled the volume to 100% exactly as Logstash was extending page.42. The write was truncated, and the next morning’s routine restart failed with Page file size is 8399 but expected 67108864, crash-looping the service and halting all ingest.

df -h /var/lib/logstash showed the volume had been full, and journalctl confirmed No space left on device at the crash timestamp. ls -la on the queue directory showed page.42 at 8399 bytes instead of the expected 64 MB — the truncated page named in the exception. The queue held only a few seconds of unacknowledged events, so the team moved the corrupt main queue aside (preserving it as a backup), freed disk by relocating system logs off the volume, and restarted. Logstash opened a fresh queue and resumed. They then moved path.queue to its own dedicated, monitored volume so a future log flood could not corrupt it.

Prevention Best Practices

• Put path.data/path.queue on a dedicated volume with generous headroom, and alert on disk usage well before 100% — full disks are the top cause of PQ corruption.
• Always stop Logstash with SIGTERM and a long TimeoutStopSec so the queue checkpoints cleanly; avoid kill -9 and short deploy timeouts.
• Tune queue.checkpoint.writes toward durability (lower value) on hosts prone to unclean shutdown, accepting a small throughput cost.
• Never share, copy partially, or manually edit queue files; treat path.queue as opaque, single-owner state.
• Keep Logstash versions consistent across restarts — don’t downgrade against a newer on-disk queue format.
• Back up (move aside) a corrupt queue before wiping it, so you retain the option to salvage acknowledged-but-unshipped events.

Quick Command Reference

# Exact IOException and the file it names
grep -A15 'java.io.IOException' /var/log/logstash/logstash-plain.log | tail -25

# Inspect the queue files (look for wrong-size / zero-byte pages)
ls -la /var/lib/logstash/queue/main/

# Was the disk full? (the usual root cause)
df -h /var/lib/logstash
journalctl -u logstash --since '2 hours ago' | grep -Ei 'space|ioexception'

# Reset the corrupt queue (data loss) after backing it up
sudo systemctl stop logstash
sudo mv /var/lib/logstash/queue/main /var/lib/logstash/queue/main.corrupt.$(date +%s)
sudo systemctl start logstash

# Confirm recovery
curl -s localhost:9600/_node/stats/pipelines?pretty | grep -A6 '"queue"'

Conclusion

A java.io.IOException about a page file being a different size than expected means Logstash’s persistent queue on disk is inconsistent — almost always because an ungraceful shutdown or a full disk truncated a page or checkpoint mid-write. Logstash refuses to start rather than replay corrupt data. Diagnose by matching the named page file to a zero/wrong-size file and checking df/journalctl for a disk-full or OOM event. If the queued data is expendable, back up and reset the pipeline’s queue directory to start clean. Prevent recurrence by giving the queue a dedicated, monitored volume and always shutting Logstash down gracefully.