🚨 Logstash Error Guide: 'Ruby exception occurred' — Fix NoMethodError for nil in the ruby Filter

Overview

The ruby filter lets you run arbitrary Ruby against each event, either inline via code => or from an external script. It is the escape hatch you reach for when no built-in filter does the transformation you need. That power comes with a sharp edge: any exception your Ruby raises while processing an event is caught by the filter, logged, and — by default — tags the event with _rubyexception and lets it pass through. The pipeline keeps running, but the event is mangled and your logs fill with stack traces.

The most common failure is calling a method on a field that turned out to be nil because it was absent on that particular event. In /var/log/logstash/logstash-plain.log it looks like this:

[ERROR][logstash.filters.ruby] Ruby exception occurred: undefined method `strip' for nil:NilClass

With a fuller backtrace when log.level is debug:

[ERROR][logstash.filters.ruby] Ruby exception occurred:
  {:script=>"(ruby filter code)",
   :exception=>"undefined method `split' for nil:NilClass",
   :backtrace=>["(ruby filter code):3:in `block in register'",
                "org/logstash/execution/...:in `filter'"]}

The key clue is for nil:NilClass. Logstash’s event.get("[some][field]") returns nil when the field does not exist, and calling any method — .strip, .split, .to_f, .each — on nil raises NoMethodError. Because real-world log streams are heterogeneous, a field present on 99% of events but missing on the other 1% will throw on exactly those events, making the error look intermittent and confusing.

Symptoms

• logstash-plain.log repeatedly logs Ruby exception occurred: undefined method '<x>' for nil:NilClass, often at a steady low rate rather than on every event.
• A share of your documents in Elasticsearch carry a _rubyexception tag (searchable as tags:_rubyexception).
• The field the ruby filter was supposed to compute is missing or stale on the affected events.
• Throughput dips and CPU rises under load because exception handling and backtrace logging are expensive per event.
• The pipeline passes --config.test_and_exit (syntax is valid) yet still throws at runtime — because the bug is data-dependent, not syntactic.

Common Root Causes

• Unguarded field access — calling a method on event.get("[field]") without checking it returned non-nil; missing on some events, present on others.
• Wrong field-reference string — event.get("user.name") (dotted) instead of event.get("[user][name]") (bracketed) returns nil because the literal key user.name doesn’t exist.
• Type assumptions — treating a value as a String when it arrived as an Integer or Array, so .strip or .split is undefined for that class.
• Using the pre-8.0 event[...] API — old snippets copied from the internet use event["field"], which is not how you access fields in modern Logstash; use event.get/event.set.
• Nested-hash navigation — chaining event.get("[a]")["b"] where [a] is nil, so indexing into nil blows up.
• Locale/format edge cases — Integer(str) or Time.parse(str) raising ArgumentError on a malformed value that only appears in a fraction of events.
• Referencing an undefined local/variable — a typo’d variable name evaluates to a method call on main, raising NameError/NoMethodError.

Diagnostic Workflow

First confirm the config is structurally valid, so you know you’re chasing a runtime data bug rather than a syntax error:

sudo -u logstash /usr/share/logstash/bin/logstash \
  -f /etc/logstash/conf.d/derive-fields.conf \
  --path.settings /etc/logstash \
  --config.test_and_exit

A Configuration OK here proves the Ruby parses; it says nothing about whether it survives real data. Here is a pipeline that will throw on any event missing [url][original]. Logstash .conf uses a Ruby-flavored DSL, so it reads naturally:

input {
  beats { port => 5044 }
}

filter {
  ruby {
    # BUG: assumes [url][original] always exists. It doesn't.
    code => '
      path = event.get("[url][original]")
      segments = path.split("/")          # NoMethodError when path is nil
      event.set("[url][depth]", segments.length - 1)
    '
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "web-%{+YYYY.MM.dd}"
  }
}

The robust version guards the nil and validates type before operating:

filter {
  ruby {
    code => '
      path = event.get("[url][original]")
      if path.is_a?(String) && !path.empty?
        segments = path.split("/").reject(&:empty?)
        event.set("[url][depth]", segments.length)
      else
        event.tag("url_original_missing")
      end
    '
    # Route any surviving exception to an explicit tag/field so you can see it downstream.
    tag_on_exception => "_ruby_derive_fields_error"
  }
}

To catch the offending events without spelunking through the whole stream, reproduce with a minimal stdin/stdout pipeline that replays a suspect document:

echo '{"message":"no url here"}' | sudo -u logstash /usr/share/logstash/bin/logstash \
  --path.settings /etc/logstash \
  -e 'input { stdin { codec => json } }
      filter { ruby { code => "event.set(\"depth\", event.get(\"[url][original]\").split(\"/\").length)" } }
      output { stdout { codec => rubydebug } }'

Then find the events already poisoned in production by their exception tag, and tail the log for live backtraces:

# In Kibana / via the ES API
curl -s 'http://localhost:9200/web-*/_count' \
  -H 'Content-Type: application/json' \
  -d '{"query":{"term":{"tags":"_rubyexception"}}}'

sudo tail -f /var/log/logstash/logstash-plain.log | grep -A5 'Ruby exception occurred'

Example Root Cause Analysis

An SRE team enriched web-access logs with a computed [url][depth]. Everything looked healthy for a week, then Kibana showed a slow trickle of documents tagged _rubyexception, and logstash-plain.log carried:

[ERROR][logstash.filters.ruby] Ruby exception occurred: undefined method `split' for nil:NilClass

The ruby filter computed depth from [url][original]. That field is populated by an upstream grok on the request line — but a subset of health-check requests hit the load balancer with a bare CONNECT and no path, so grok failed to set [url][original] on those events, leaving it nil. Calling .split on nil threw on exactly those ~0.5% of events, which is why the error was intermittent and never showed up in tests built from “normal” sample logs.

The fix had two layers. Immediately, the inline code was wrapped in an is_a?(String) guard that tags url_original_missing instead of computing depth, so no event throws and the malformed ones stay queryable. Structurally, the team moved the logic into an external script_path file with an RSpec test (ruby filter scripts support a test block) covering the missing-field case, and added tag_on_exception => "_ruby_depth_error" so any future unhandled exception lands under a specific, alertable tag rather than the generic _rubyexception. Runtime exceptions dropped to zero and the previously poisoned documents were reindexed once the guard shipped.

Prevention Best Practices

• Treat every event.get(...) as possibly nil: guard with if val or val.is_a?(String) before calling methods on it.
• Always use bracketed field references — event.get("[a][b]"), never event.get("a.b") — and the modern event.get/event.set API, never the deprecated event[...] accessor.
• Move non-trivial logic into an external script_path file and write the accompanying test block so edge cases (missing field, wrong type) are covered before deploy.
• Set tag_on_exception to a filter-specific tag so failures are attributable and alertable instead of hiding in the generic _rubyexception.
• Validate and coerce types explicitly (Integer(str) rescue nil, String(val)) rather than assuming the wire format.
• Prefer built-in filters (mutate, grok, dissect, translate) when they can do the job; reach for ruby only when they genuinely can’t, keeping the surface area for exceptions small.

Quick Command Reference

# Validate config syntax (does not exercise runtime data paths)
sudo -u logstash /usr/share/logstash/bin/logstash \
  -f /etc/logstash/conf.d/derive-fields.conf \
  --path.settings /etc/logstash --config.test_and_exit

# Replay a single suspect event through an inline pipeline
echo '{"message":"no url here"}' | sudo -u logstash /usr/share/logstash/bin/logstash \
  --path.settings /etc/logstash \
  -e 'input { stdin { codec => json } } output { stdout { codec => rubydebug } }'

# Count events poisoned by a ruby exception
curl -s 'http://localhost:9200/web-*/_count' -H 'Content-Type: application/json' \
  -d '{"query":{"term":{"tags":"_rubyexception"}}}'

# Live-tail ruby exceptions with a few lines of backtrace
sudo tail -f /var/log/logstash/logstash-plain.log | grep -A5 'Ruby exception occurred'

# Run a ruby filter script's built-in tests
sudo -u logstash /usr/share/logstash/bin/logstash \
  -e 'filter { ruby { path => "/etc/logstash/scripts/url_depth.rb" } }' \
  -t --path.settings /etc/logstash

Conclusion

Ruby exception occurred: undefined method '<x>' for nil:NilClass means your inline Ruby operated on a field that was absent on some events. It passes --config.test_and_exit because the problem is data, not syntax, which is why it shows up as an intermittent trickle of _rubyexception-tagged documents rather than an outright pipeline failure. Guard every event.get against nil, use bracketed field references and the event.get/event.set API, and set a specific tag_on_exception so failures are visible. For anything beyond a one-liner, move the logic to an external script with tests — the ruby filter is a scalpel, and it cuts cleanest when its edge cases are covered before deploy.