🚨 Logstash Error: 'No space left on device' — Cause, Fix, and Troubleshooting Guide

Overview

Logstash writes to disk in more places than people expect: the persistent queue, the dead letter queue, and its own logs all consume the filesystem under path.data and path.logs. When that filesystem fills up, any write fails and the error surfaces in /var/log/logstash/logstash-plain.log:

[ERROR][org.logstash.execution.AbstractPipelineExt][main] java.io.IOException: No space left on device

The same condition can appear as the lower-level JRuby form:

Errno::ENOSPC: No space left on device

The most common victim is the persistent queue (PQ). When the PQ cannot write a new page, the pipeline stalls waiting for space, and in some cases the process aborts outright. This is not a Logstash bug — it is a full disk — so the fix is to reclaim space and, more importantly, to cap how much disk Logstash is allowed to consume so it can never fill the volume again.

Symptoms

• Repeated java.io.IOException: No space left on device or Errno::ENOSPC in the Logstash log.
• The pipeline stalls: events stop flowing to outputs and upstream inputs (Beats, the PQ) back up.
• df -h shows the filesystem holding path.data and/or path.logs at or near 100% used.
• Logstash’s own log file stops growing (it cannot write) or the service crashes and systemd restarts it into the same wall.
• Large files accumulate under /var/lib/logstash/queue, /var/lib/logstash/dead_letter_queue, or /var/log/logstash.

Common Root Causes

• Persistent queue growth from backpressure — a slow or stuck output (e.g. Elasticsearch rejecting writes) lets the PQ grow unbounded until it fills path.queue.
• Uncapped dead letter queue — failed events pile into path.dead_letter_queue and nothing drains or expires them, so the DLQ eats the disk.
• Unrotated Logstash logs — /var/log/logstash/logstash-plain.log and GC logs grow without rotation, especially with debug logging left on.
• Undersized volume — path.data sits on a volume that is simply too small for the throughput and buffering the pipeline needs.
• A shared volume filled by something else — another service on the same filesystem consumed the space, and Logstash is the first to complain.

How to diagnose

Find which filesystem is full, then find what is consuming it. Start at the filesystem level:

df -h /var/lib/logstash /var/log/logstash

Then measure the actual Logstash directories to identify the offender — PQ, DLQ, or logs:

sudo du -sh /var/lib/logstash/* /var/log/logstash/* 2>/dev/null | sort -h

Inspect the persistent queue and dead letter queue directories specifically:

sudo du -sh /var/lib/logstash/queue /var/lib/logstash/dead_letter_queue
sudo ls -lh /var/lib/logstash/queue/main

Check whether the PQ is growing because an output is backed up — a stuck Elasticsearch output is the classic cause:

curl -s localhost:9600/_node/stats/pipelines?pretty | grep -A6 -i 'queue\|events'
sudo grep -Ei 'unreachable|429|backpressure|_dead_letter' /var/log/logstash/logstash-plain.log | tail

Look at the current queue settings in /etc/logstash/logstash.yml to see whether any caps are in place:

queue.type: persisted
# if these are absent, the queues are effectively uncapped by size:
# queue.max_bytes: ...
# dead_letter_queue.max_bytes: ...

Fixes

First reclaim space so the pipeline can move again. Rotate or clear old logs (never delete the live PQ pages while Logstash is running unless you accept event loss):

sudo find /var/log/logstash -name '*.log.*' -mtime +3 -delete
sudo logrotate -f /etc/logrotate.d/logstash 2>/dev/null || true

Then cap the queues in /etc/logstash/logstash.yml so Logstash can never consume the whole disk. Size the caps to fit comfortably within the volume with headroom to spare:

queue.type: persisted
queue.max_bytes: 1gb

dead_letter_queue.enable: true
dead_letter_queue.max_bytes: 1gb

If the PQ filled because of backpressure, the real fix is upstream — unblock or slow the stuck output. Investigate the output that stopped accepting events (see the Elasticsearch and backpressure guides) and let the queue drain once it recovers:

sudo tail -f /var/log/logstash/logstash-plain.log | grep -Ei 'Restored|drained|queue'

If the volume is genuinely too small for the workload, move path.data to a larger disk and update logstash.yml:

path.data: /mnt/logstash-data

Then restart and confirm space is under control:

sudo systemctl restart logstash && df -h /var/lib/logstash

What to watch out for

• Never delete files out of the live persistent queue directory while Logstash is running — those pages are unacked events, and removing them loses data. Cap the size instead and let it drain.
• Capping queue.max_bytes means that when the PQ is full, inputs apply backpressure (they block) rather than crash — that is intended, but it means a stuck output will now throttle ingestion instead of filling the disk.
• A capped DLQ drops the oldest dead-lettered events when full; if you care about them, drain the DLQ with a dedicated pipeline rather than relying on the cap alone.
• Make sure logrotate is actually configured for Logstash logs — debug-level logging can fill a disk on its own.
• Alert on filesystem usage for path.data and on the No space left on device log pattern, so you catch a backing-up queue before it stalls the whole pipeline.

Related

• Persistent queue corruption
• Dead letter queue full
• Java heap space (OutOfMemoryError)