🚨 Logstash Error Guide: 'pattern %{FOO} not defined' — Fix Grok Compile Failures

Overview

Unlike _grokparsefailure, which is a soft runtime tag, an undefined grok pattern is a hard failure. When Logstash compiles a grok filter and encounters a pattern name it has never heard of — %{FOO} where FOO is not a built-in pattern and was never registered — the filter fails to compile and the entire pipeline refuses to start. Nothing flows until you fix it.

The error surfaces during pipeline startup or a --config.test_and_exit run, and it is explicit about the missing name:

[ERROR][logstash.filters.grok] Error while attempting to compile grok pattern
[ERROR][logstash.javapipeline] Pipeline error {:pipeline_id=>"main",
  :exception=>#<Grok::PatternError: pattern %{APPLOG} not defined>}
[ERROR][logstash.agent] Failed to execute action
  {:action=>LogStash::PipelineAction::Create/pipeline_id:main,
   :exception=>"Java::JavaLang::IllegalStateException",
   :message=>"Unable to configure plugins: (ConfigurationError) pattern %{APPLOG} not defined"}

You may also see the shorter “Grok compile failed” wording, or no pattern found for %{...}, depending on the Logstash version. All of them mean the same thing: the grok engine cannot resolve one of the %{PATTERN} references in your match expression to an actual regular expression, so it aborts before processing a single event.

Symptoms

• The Logstash service fails to start, or restarts in a crash loop, immediately after a config change that touched a grok filter.
• logstash-plain.log contains pattern %{...} not defined, Grok compile failed, or no pattern found for on startup.
• bin/logstash --config.test_and_exit returns a non-zero exit code and prints the compile error instead of Configuration OK.
• Under systemd, journalctl -u logstash shows the pipeline creation failing and the process exiting.
• The failure is deterministic — it happens on every start, not intermittently, because it is a compile-time error, not a data-dependent one.

Common Root Causes

• A typo in a built-in pattern name — %{IPADRESS} instead of %{IPADDRESS}, %{TIMSTAMP_ISO8601} instead of %{TIMESTAMP_ISO8601}. Grok has no fuzzy matching; one wrong character means “not defined.”
• A custom pattern that was never registered — the config references %{APPLOG} but no patterns_dir was configured and no inline pattern_definitions block defines it.
• A wrong or unreadable patterns_dir — the directory is set but the path is wrong, the file has bad permissions, or Logstash runs as a user that cannot read it, so the custom definitions never load.
• Case sensitivity — grok pattern names are case-sensitive and conventionally uppercase; %{httpdate} will not resolve to %{HTTPDATE}.
• A pattern that depends on a missing sub-pattern — a custom pattern is defined in terms of another custom pattern that itself was never registered, so the dependency fails to compile.
• Version-specific or plugin-provided patterns — a pattern name copied from a blog post belongs to a newer logstash-patterns-core release or a plugin you have not installed.

Diagnostic Workflow

Because this is a compile-time error, config-test mode reproduces it instantly and safely — you never have to restart the live service to see it:

# Reproduce the compile failure without touching the running pipeline
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/app.conf --config.test_and_exit

# Shorthand
/usr/share/logstash/bin/logstash -t -f /etc/logstash/conf.d/app.conf

If a pattern is undefined you will get the compile error and a non-zero exit code. A successful compile prints Configuration OK.

Here is a config that triggers the failure — it references %{APPLOG}, which does not exist as a built-in:

# /etc/logstash/conf.d/app.conf  (BROKEN — %{APPLOG} is undefined)
input {
  beats { port => 5044 }
}

filter {
  grok {
    match => { "message" => "%{APPLOG}" }
  }
}

output {
  elasticsearch { hosts => ["localhost:9200"] }
}

There are two correct ways to define the missing pattern. The first is an inline pattern_definitions block, ideal for one or two custom patterns:

filter {
  grok {
    # Define APPLOG right here, in terms of built-in patterns
    pattern_definitions => {
      "APPLOG" => "%{TIMESTAMP_ISO8601:ts} %{LOGLEVEL:level} \[%{DATA:thread}\] %{JAVACLASS:class} - %{GREEDYDATA:msg}"
    }
    match => { "message" => "%{APPLOG}" }
  }
}

The second is an external patterns file, better when you reuse patterns across many pipelines. Create the file:

sudo mkdir -p /etc/logstash/patterns

# /etc/logstash/patterns/custom
APPLOG %{TIMESTAMP_ISO8601:ts} %{LOGLEVEL:level} \[%{DATA:thread}\] %{JAVACLASS:class} - %{GREEDYDATA:msg}

Then point the grok filter at the directory:

filter {
  grok {
    patterns_dir => ["/etc/logstash/patterns"]
    match => { "message" => "%{APPLOG}" }
  }
}

Make sure the Logstash user can read it, then re-test:

sudo chown -R logstash:logstash /etc/logstash/patterns
sudo chmod 644 /etc/logstash/patterns/custom
/usr/share/logstash/bin/logstash -t -f /etc/logstash/conf.d/app.conf

To confirm whether a name is built-in before you rely on it, grep the shipped pattern definitions:

# List every built-in pattern name available to your install
find /usr/share/logstash/vendor/bundle -path '*patterns*' -type f \
  -exec grep -hE '^[A-Z0-9_]+ ' {} + | awk '{print $1}' | sort -u | grep -i time

Example Root Cause Analysis

An engineer copies a Java-application grok pattern from an internal wiki into a new pipeline and the Logstash service enters a restart loop. journalctl -u logstash -f shows:

[ERROR][logstash.filters.grok] Error while attempting to compile grok pattern
Grok::PatternError: pattern %{JAVALOGLEVEL} not defined

The pattern text is %{TIMESTAMP_ISO8601:ts} %{JAVALOGLEVEL:level} %{GREEDYDATA:msg}. The engineer assumes JAVALOGLEVEL is a standard pattern, but grepping the built-ins shows only LOGLEVEL exists — JAVALOGLEVEL was a custom pattern defined in the wiki author’s own patterns_dir, which was never copied over.

The fix is to register the missing pattern. Because it is only used here, an inline definition is cleanest:

filter {
  grok {
    pattern_definitions => {
      # LOGLEVEL is built-in; JAVALOGLEVEL just constrains it to the levels this app emits
      "JAVALOGLEVEL" => "(?:TRACE|DEBUG|INFO|WARN|ERROR|FATAL)"
    }
    match => { "message" => "%{TIMESTAMP_ISO8601:ts} %{JAVALOGLEVEL:level} %{GREEDYDATA:msg}" }
  }
}

Re-running bin/logstash -t now prints Configuration OK, and the service starts cleanly. The broader lesson: any %{NAME} that is not in logstash-patterns-core must be defined by you, either inline or in a patterns_dir file that ships alongside the pipeline config. A pattern that “works on someone else’s machine” almost always depends on a custom definition that must travel with it.

Prevention Best Practices

• Validate every config with --config.test_and_exit in CI before it reaches a running node, so a compile failure is caught in a pull request rather than in a crash loop.
• Keep custom patterns in a version-controlled patterns_dir that is deployed together with the pipeline configs — never rely on a pattern that lives only on one host.
• Verify pattern names against your install’s logstash-patterns-core rather than copying from blog posts; names and availability change between versions.
• Prefer inline pattern_definitions for one-off patterns to keep the definition next to its use and avoid path/permission problems.
• Set file ownership and permissions so the logstash service user can read every patterns file (chown logstash:logstash, chmod 644).
• Watch for case and spelling — pattern names are uppercase and case-sensitive; a linter or code review that flags unknown %{...} names prevents most of these failures.

Quick Command Reference

# Reproduce / verify the compile error
bin/logstash -t -f /etc/logstash/conf.d/app.conf
bin/logstash -f /etc/logstash/conf.d/app.conf --config.test_and_exit

# Watch startup errors live
sudo journalctl -u logstash -f
sudo tail -f /var/log/logstash/logstash-plain.log

# List all built-in grok pattern names
find /usr/share/logstash/vendor/bundle -path '*patterns*' -type f \
  -exec grep -hE '^[A-Z0-9_]+ ' {} + | awk '{print $1}' | sort -u

# Check that the Logstash user can read your patterns dir
sudo -u logstash test -r /etc/logstash/patterns/custom && echo readable || echo UNREADABLE

# Fix ownership/permissions on a patterns directory
sudo chown -R logstash:logstash /etc/logstash/patterns && sudo chmod 644 /etc/logstash/patterns/*

Conclusion

“Pattern not defined” and “Grok compile failed” are startup-blocking errors: the pipeline will not run until every %{PATTERN} reference resolves to a real regular expression. The fix is almost always one of three things — correct a typo in a built-in name, register a custom pattern via pattern_definitions or a patterns_dir file, or fix the path and permissions so that file actually loads. Because the failure is deterministic and compile-time, you can catch it every single time with bin/logstash -t before deploying. Wire that config test into CI, keep your custom patterns version-controlled and deployed alongside the pipeline, and this class of outage disappears entirely — a broken pattern never reaches a running node.