🚨 Logstash Error: 'OutOfDirectMemoryError' — Cause, Fix, and Troubleshooting Guide

Overview

The beats, http, and tcp inputs are built on Netty, which allocates off-heap direct memory for its network buffers rather than using the JVM heap. When that direct-memory pool is exhausted, Netty cannot allocate a buffer for the next connection and throws an OutOfDirectMemoryError, often alongside a resource-leak warning. In /var/log/logstash/logstash-plain.log it looks like this:

[ERROR][io.netty.util.ResourceLeakDetector][nioEventLoopGroup-2-1] ...
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 1056964615, max: 1073741824)

The critical word is direct — this is not a heap OutOfMemoryError: Java heap space (that is a separate problem). The used/max figures in the message are the off-heap pool, not -Xmx. Here max is 1073741824 (1 GiB), which is the default because MaxDirectMemorySize defaults to the value of -Xmx. When many Filebeat clients connect at once, or push large batches, Netty’s buffers can outgrow that pool and the input starts failing connections.

Symptoms

• io.netty.util.internal.OutOfDirectMemoryError: failed to allocate N byte(s) of direct memory in the log.
• ResourceLeakDetector warnings from nioEventLoopGroup-* threads preceding or surrounding the error.
• Filebeat clients log connection resets or publish timeouts and retry, driving even more connections.
• The error tracks with connection count or batch size, not with pipeline heap pressure.
• used approaches max in the message, and max equals your -Xmx value (the default direct-memory cap).

Common Root Causes

• Too many concurrent Filebeat connections — each open connection holds Netty buffers; a large fleet all pointing at one Logstash node exhausts the pool.
• MaxDirectMemorySize too low — left at its default (equal to -Xmx), the off-heap pool is smaller than the workload needs.
• Large client batches / pipelining — high bulk_max_size and pipelining on Filebeat mean bigger in-flight buffers per connection.
• Connections not timing out — idle or half-open beats connections are never reaped, so their buffers are never freed.
• A single Logstash node taking the whole fleet — no load balancing or queue in front, so all buffer pressure lands on one process.
• An old beats input plugin — earlier plugin/Netty versions had heavier or leak-prone buffer handling.

How to diagnose

Confirm this is a direct memory problem, not heap. The message says so, but verify the configured cap and heap side by side:

grep -E 'Xmx|MaxDirectMemorySize' /etc/logstash/jvm.options

# Confirm the running process flags
sudo -u logstash jcmd $(pgrep -f logstash) VM.flags | tr ' ' '\n' | grep -i direct

Count how many Filebeat clients are actually connected to the beats port — connection count is the usual driver:

# Established connections to the beats input port
ss -tan state established '( dport = :5044 or sport = :5044 )' | wc -l

Check the Logstash JVM stats for the direct-memory pool over time:

curl -s localhost:9600/_node/stats/jvm?pretty | grep -A6 -i 'non_heap\|buffer'

Look at what the Filebeat clients are configured to send, since batch size and pipelining multiply per-connection buffer usage:

# filebeat.yml (as-found — inspect these)
output.logstash:
  hosts: ["logstash.internal:5044"]
  bulk_max_size: 2048
  pipelining: 2

Fixes

Raise the direct-memory cap in /etc/logstash/jvm.options so Netty has room for its buffers. Set it explicitly rather than inheriting -Xmx:

# /etc/logstash/jvm.options
-Xms2g
-Xmx2g
-XX:MaxDirectMemorySize=2g

Restart to apply, and confirm the new cap is in effect:

sudo systemctl restart logstash
sudo -u logstash jcmd $(pgrep -f logstash) VM.flags | tr ' ' '\n' | grep -i MaxDirectMemorySize

Reduce per-connection pressure from the clients. Lower Filebeat’s batch size and pipelining so each connection holds smaller in-flight buffers:

# filebeat.yml
output.logstash:
  hosts:
    - "logstash-a.internal:5044"
    - "logstash-b.internal:5044"
  loadbalance: true
  bulk_max_size: 1024
  pipelining: 1

Reap idle connections by setting client_inactivity_timeout on the beats input, so half-open connections release their buffers:

input {
  beats {
    port => 5044
    client_inactivity_timeout => 60
  }
}

Spread the fleet across multiple Logstash nodes (loadbalance: true above) or put a queue such as Kafka in front so no single node absorbs all connections. And keep the logstash-input-beats plugin current for the latest buffer-handling fixes:

sudo -u logstash /usr/share/logstash/bin/logstash-plugin update logstash-input-beats

What to watch out for

• Direct memory is not heap. Do not “fix” this by raising -Xmx alone — if MaxDirectMemorySize is unset it will track -Xmx, but set it explicitly so the two are decoupled and intentional.
• Total memory budget must fit the host. -Xmx + MaxDirectMemorySize + OS overhead must be well under physical RAM, or you trade a Netty error for an OOM-killer kill.
• Client settings multiply. bulk_max_size × pipelining × connection_count is roughly what drives buffer demand; dialing all three down is more effective than any single change.
• Load-balance, don’t overload one node. A flat fan-in from a big fleet onto one Logstash is the root cause more often than a genuinely small cap.
• This is distinct from java.lang.OutOfMemoryError: Java heap space. If your error says heap rather than direct, the tuning is different — see the heap-space guide.

Related

• Logstash ‘OutOfMemoryError: Java heap space’
• Logstash beats input ‘Connection reset by peer’
• Logstash ‘circuit_breaking_exception: Data too large’