🚨 Logstash Error Guide: 'java.lang.OutOfMemoryError: Java heap space' — Right-Size the JVM Heap

Overview

Logstash runs inside a JVM, and when the pipeline holds more live objects than the configured heap can store, the JVM throws an unrecoverable error and the process usually dies:

[FATAL][org.logstash.Logstash] Logstash stopped processing because of an error: (SystemExit) exit
java.lang.OutOfMemoryError: Java heap space
	at org.jruby.RubyString.substr19(RubyString.java:...)
	at org.logstash.ackedqueue.Queue.readBatch(Queue.java:...)

You will also see it emitted by the JVM directly to stderr or written to the file named by -XX:HeapDumpPath when -XX:+HeapDumpOnOutOfMemoryError is set. Once this fires, the affected worker threads are in an undefined state and Logstash cannot be trusted to keep processing.

Symptoms

• Logstash exits or hangs shortly after start under load, sometimes minutes after appearing healthy.
• logstash-plain.log ends with java.lang.OutOfMemoryError: Java heap space and a FATAL stop line.
• A java_pid<PID>.hprof heap-dump file appears in the Logstash working directory or HeapDumpPath.
• jstat -gcutil shows the old generation pinned at 100.00 with continuous full GCs (high FGC/FGCT).
• Throughput collapses just before the crash as the JVM spends all its time in garbage collection.
• Under systemd the unit flaps: active (running) then failed, restarting in a crash loop.

Common Root Causes

• Heap too small for the workload — the default -Xmx1g cannot hold large batches of enriched events.
• Oversized pipeline.batch.size × pipeline.workers — total in-flight events = batch size × workers, all resident in heap at once.
• Large events — multi-megabyte log lines, big message fields, or deeply nested JSON multiply memory per event.
• Memory-hungry filters — aggregate, ruby, large translate dictionaries, or a grok with catastrophic backtracking retaining state.
• In-memory queue with backpressure — a slow output (Elasticsearch) backs events up in the memory queue until heap is exhausted.
• Xms ≠ Xmx — a heap that must grow fragments and stalls; production should pin both equal.
• Heap larger than container memory limit — the JVM believes it has more heap than the cgroup allows and the kernel OOM-kills it (different symptom, same root cause of mis-sizing).

Diagnostic Workflow

First, confirm the heap actually configured — not what you think is configured. Logstash reads config/jvm.options (and LS_JAVA_OPTS):

grep -E '^-Xm(s|x)' /etc/logstash/jvm.options
# Effective flags at runtime:
jps -lv | grep logstash
ps -ef | grep -o '\-Xmx[0-9a-zA-Z]*'

Watch garbage collection live — if old-gen O stays near 100 and full GCs never reclaim, you are heap-bound:

PID=$(pgrep -f 'org.logstash.Logstash')
jstat -gcutil "$PID" 2000 10

Inspect the pipeline settings that drive in-flight memory:

# /etc/logstash/pipelines.yml
- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"
  pipeline.workers: 8
  pipeline.batch.size: 2000     # 8 x 2000 = 16,000 events in heap at once
  queue.type: memory

Enable a heap dump so the next crash is analyzable, in jvm.options:

-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/log/logstash/heapdump.hprof
-Xms4g
-Xmx4g

Check whether a slow Elasticsearch output is causing the backpressure that fills the heap:

curl -s localhost:9600/_node/stats/pipelines?pretty | \
  grep -A3 -E '"events"|"duration_in_millis"'
curl -s 'http://es:9200/_cat/thread_pool/write?v&h=node_name,active,queue,rejected'

Pull JVM memory stats from the monitoring API to see heap pressure without attaching a profiler:

curl -s localhost:9600/_node/stats/jvm?pretty | grep -A6 heap_used

Example Root Cause Analysis

A team ran Logstash on a 8 GB VM with the stock -Xmx1g. After adding a translate filter backed by a 400 MB CSV lookup and bumping pipeline.batch.size to 4000 across 6 workers, Logstash began crashing within ten minutes of start with java.lang.OutOfMemoryError: Java heap space.

jstat -gcutil showed O stuck at 100.00 with FGC incrementing every second — the classic GC death spiral. The heap dump, opened in Eclipse MAT, revealed two dominators: the fully-resident translate dictionary (~600 MB on-heap after parsing) and 24,000 in-flight event objects (6 workers × 4000 batch). Together they exceeded the 1 GB heap before any output flushed.

The fix was threefold: raise -Xms/-Xmx to a matched 4g (leaving headroom on the 8 GB box for the OS and page cache), drop pipeline.batch.size to 1000, and move the translate lookup to an external elasticsearch filter so the dictionary no longer lived on Logstash’s heap. GC pressure dropped to a few percent old-gen and the crashes stopped.

Prevention Best Practices

• Set -Xms equal to -Xmx in jvm.options, and keep the heap at no more than ~50–65% of available RAM so the OS page cache and off-heap buffers have room.
• Size in-flight memory deliberately: total resident events ≈ pipeline.workers × pipeline.batch.size. Start small (workers = CPU count, batch 125) and raise only with evidence.
• Keep large lookup data (translate dictionaries, enrichment tables) off-heap by using the elasticsearch, jdbc_static, or memcached filters instead of giant in-memory maps.
• Always enable -XX:+HeapDumpOnOutOfMemoryError and -XX:HeapDumpPath in production so every OOM leaves evidence.
• Prefer the persistent queue (queue.type: persisted) so backpressure spills to disk instead of ballooning the heap.
• Under containers, set the heap explicitly and never let it exceed the cgroup memory limit; alert on jvm.mem.heap_used_percent from the _node/stats API.

Quick Command Reference

# Show configured and effective heap
grep -E '^-Xm(s|x)' /etc/logstash/jvm.options
jps -lv | grep logstash

# Live GC / heap pressure
PID=$(pgrep -f 'org.logstash.Logstash'); jstat -gcutil "$PID" 2000

# Heap and pipeline stats from the monitoring API
curl -s localhost:9600/_node/stats/jvm?pretty | grep -A6 heap_used
curl -s localhost:9600/_node/stats/pipelines?pretty

# Capture a heap dump on demand
jmap -dump:live,format=b,file=/var/log/logstash/manual.hprof "$PID"

# Check ES write pressure that may be filling the heap
curl -s 'http://es:9200/_cat/thread_pool/write?v&h=node_name,active,queue,rejected'

Conclusion

java.lang.OutOfMemoryError: Java heap space in Logstash is almost always a sizing problem: the heap is too small for the batch size, worker count, event size, or on-heap lookup data you have configured. Pin -Xms and -Xmx to a matched value with headroom for the OS, size in-flight events with pipeline.workers × pipeline.batch.size, move big dictionaries off-heap, and switch to a persistent queue so backpressure spills to disk. Enable heap dumps up front so the next incident is diagnosable in minutes instead of guesses.