🚨 Logstash Error Guide: 'the dead_letter_queue is full' — Drain and Size the DLQ

Overview

When the Logstash dead-letter queue (DLQ) reaches its size cap and no reader is draining it, new events that cannot be written to Elasticsearch have nowhere to go and Logstash logs:

[WARN ][logstash.outputs.elasticsearch.dlqwriter] cannot write event to DLQ
(configured max_bytes reached): the dead_letter_queue is full
[WARN ][org.logstash.common.io.DeadLetterQueueWriter] Unable to write event to the
dead letter queue: the queue is full (max_bytes=1073741824)

The DLQ stores events that Elasticsearch rejected as non-retryable — mapping conflicts (400) and similar. It is bounded by dead_letter_queue.max_bytes (default 1 GB). If failing events accumulate faster than a DLQ-reader pipeline consumes them, the queue fills and, depending on dead_letter_queue.storage_policy, either drops the oldest entries or blocks new writes.

Symptoms

• Repeated the dead_letter_queue is full / max_bytes reached warnings in logstash-plain.log.
• path.data/dead_letter_queue/<pipeline_id>/ grows to exactly max_bytes and stops.
• Events that fail indexing are silently dropped (with storage_policy: drop_older) or the pipeline stalls (drop_newer).
• A steady stream of 400 mapping-conflict errors from the elasticsearch output preceding the DLQ fill.
• No DLQ-reader pipeline exists, so nothing ever empties the queue.

Common Root Causes

• No DLQ reader configured — DLQ is enabled but no dead_letter_queue input pipeline drains it, so it only grows.
• Persistent mapping conflicts — a field arriving as both a string and an object generates a flood of 400s that all land in the DLQ.
• max_bytes too small for the failure volume during an incident.
• Root failure never fixed — the underlying bad data keeps arriving, refilling the DLQ as fast as you drain it.
• storage_policy mismatch — drop_newer blocks the pipeline; drop_older silently discards data you needed.
• Disk pressure — path.data is on a small volume and the DLQ competes with the persistent queue for space.

Diagnostic Workflow

Confirm the DLQ is enabled and see its configured cap in logstash.yml:

grep -E 'dead_letter_queue' /etc/logstash/logstash.yml

# /etc/logstash/logstash.yml
dead_letter_queue.enable: true
dead_letter_queue.max_bytes: 1024mb
dead_letter_queue.storage_policy: drop_older
path.dead_letter_queue: "/var/lib/logstash/dead_letter_queue"

Measure current DLQ size on disk per pipeline:

du -sh /var/lib/logstash/dead_letter_queue/*/
ls -la /var/lib/logstash/dead_letter_queue/main/

Read DLQ metrics from the monitoring API:

curl -s localhost:9600/_node/stats/pipelines?pretty | grep -A6 dead_letter_queue

Find out WHY events are failing — inspect the elasticsearch output errors preceding the DLQ writes:

grep -E '"status"=>400|mapper_parsing_exception|illegal_argument' \
  /var/log/logstash/logstash-plain.log | tail
curl -s 'http://es:9200/logstash-2026.07.10/_mapping?pretty' | grep -A3 problem_field

Set up a reader pipeline to drain the DLQ (so it stops being “full”) in pipelines.yml:

- pipeline.id: dlq-drain
  config.string: |
    input {
      dead_letter_queue {
        path => "/var/lib/logstash/dead_letter_queue"
        pipeline_id => "main"
        commit_offsets => true
      }
    }
    output {
      # reshape / correct the event, then re-send or archive it
      file { path => "/var/log/logstash/dlq-review-%{+YYYY.MM.dd}.log" }
    }

Example Root Cause Analysis

An application started emitting a user field that was sometimes a string ("alice") and sometimes an object ({"id":1}). Elasticsearch had mapped user as an object, so every string variant failed with mapper_parsing_exception (400) and was routed to the DLQ. With no reader pipeline configured and max_bytes: 1024mb, the DLQ hit 1 GB in a few hours and Logstash began logging the dead_letter_queue is full. Because storage_policy was drop_older, older failed events were being silently discarded.

The operator confirmed the mapping conflict via _mapping and the 400 errors in the log. They fixed the root cause with a mutate/rename in the pipeline to coerce user into user.name consistently, added a dlq-drain reader pipeline to reprocess the backlog through the corrected logic, and raised max_bytes to 4gb temporarily to absorb the drain. Once the corrected events reindexed, DLQ size dropped to zero and stayed there.

Prevention Best Practices

• If you enable the DLQ, always run a companion dead_letter_queue reader pipeline — an unread DLQ can only fill.
• Fix the root cause of failed writes (usually mapping conflicts): enforce consistent field types with an index template and mutate coercion.
• Choose storage_policy deliberately: drop_older protects the pipeline but loses data; drop_newer/blocking preserves data but risks backpressure. Alert either way.
• Monitor DLQ size (_node/stats/pipelines and du on path.dead_letter_queue) and alert well before max_bytes.
• Put path.data (and thus the DLQ) on a volume with headroom, sized for the worst-case failure burst.
• Use strict index templates so a rogue field type is rejected predictably and can be caught early rather than flooding the DLQ.

Quick Command Reference

# DLQ config and on-disk size
grep dead_letter_queue /etc/logstash/logstash.yml
du -sh /var/lib/logstash/dead_letter_queue/*/

# DLQ metrics
curl -s localhost:9600/_node/stats/pipelines?pretty | grep -A6 dead_letter_queue

# Why are writes failing?
grep -E '400|mapper_parsing_exception' /var/log/logstash/logstash-plain.log | tail
curl -s 'http://es:9200/logstash-*/_mapping?pretty'

# Drain the DLQ (reader pipeline in pipelines.yml), then confirm it empties
du -sh /var/lib/logstash/dead_letter_queue/main/

Conclusion

the dead_letter_queue is full means Logstash has been routing non-retryable failed events (usually mapping-conflict 400s) into the DLQ faster than anything drains it, and the queue hit max_bytes. The durable fix is two-fold: stop the failures at the source by enforcing consistent field types with index templates and mutate coercion, and run a dead_letter_queue reader pipeline to reprocess or archive the backlog. Size max_bytes for your worst burst, choose storage_policy consciously, and alert on DLQ growth so you act before events are dropped.