🚨 Logstash Error Guide: 'JSON parse error, original data now in message field' — Fix the json Filter

Overview

When you run the json filter (or the json codec) over a field that is not valid JSON, Logstash does not drop the event — it logs a warning, leaves the raw text where it was, and adds a _jsonparsefailure tag so you can route the failure downstream. The canonical warning in /var/log/logstash/logstash-plain.log reads:

[WARN ][logstash.filters.json][main][f3a1...] Error parsing json {:source=>"message", :raw=>"{\"level\":\"info\",\"msg\":\"started\" ", :exception=>#<LogStash::Json::ParserError: Unexpected end-of-input: expected close marker for Object>}

Older and codec-path variants phrase it as the message this guide is named for, and always attach the failure tag:

[WARN ][logstash.codecs.json] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('}' (code 125))>, :data=>"...}"}

The key idea: the json filter tried to turn a text field into structured fields, the underlying Jackson parser rejected it, and rather than lose data Logstash keeps the original string in the message field (or your configured source) and marks the event with _jsonparsefailure. Your job is to find why the text is not valid JSON and either fix the producer or handle the failure gracefully.

Symptoms

• Events flow through, but they carry a _jsonparsefailure value in their tags field.
• Fields you expected to be parsed out (level, msg, trace_id) are missing; the whole payload sits unparsed in message.
• logstash-plain.log shows repeated Error parsing json / JSON parse error WARN lines, often at a steady rate matching one noisy source.
• Kibana shows documents where structured fields are absent and only message and tags:[_jsonparsefailure] are populated.
• The volume of failures spikes right after a log-format change on an upstream service, or when a new source that emits plain text is added to the same pipeline.

Common Root Causes

• The source field is not JSON at all — a plain-text log line (nginx access log, a stack trace) is being fed to a json filter that expects message to be an object.
• Truncated or concatenated events — a multi-line JSON document split across lines, or two JSON objects glued together on one line, so the parser hits Unexpected end-of-input or Unexpected character.
• Trailing commas or single quotes — JavaScript-style objects ({'level':'info',}) are not strict JSON; Jackson rejects single quotes and trailing commas.
• Wrong source setting — the filter parses message when the JSON actually lives in a nested field, so it parses envelope text that surrounds the payload.
• Double-encoding / already-parsed input — the json codec on the input already parsed the payload, then a json filter runs again over a field that is now a Ruby hash string, not JSON.
• Embedded newlines or unescaped control characters — raw tabs/newlines inside string values that were never escaped make the document invalid.
• A leading log prefix — a timestamp or syslog priority prepended to the JSON (2026-07-10T10:00:00 {"level":"info"}) means the field does not start with {.

Diagnostic Workflow

First confirm the pipeline itself is valid, so you are debugging data rather than syntax:

sudo -u logstash /usr/share/logstash/bin/logstash \
  -f /etc/logstash/conf.d/app-json.conf \
  --config.test_and_exit --path.settings /etc/logstash

Now build a pipeline that parses JSON and isolates failures instead of silently tagging them. Logstash .conf uses a Ruby-like DSL, so this is shown as ruby:

filter {
  json {
    source => "message"
    target => "app"                 # parsed fields land under [app], keeping message intact
    tag_on_failure => ["_jsonparsefailure"]
    skip_on_invalid_json => true    # do not even attempt parse on obviously non-JSON lines
  }

  # Route the bad ones somewhere you can inspect them
  if "_jsonparsefailure" in [tags] {
    mutate { add_field => { "[@metadata][parse_failed]" => "true" } }
  }
}

output {
  if [@metadata][parse_failed] == "true" {
    file { path => "/var/log/logstash/json-parse-failures-%{+YYYY.MM.dd}.log" }
  } else {
    elasticsearch {
      hosts => ["https://es.internal:9200"]
      index => "app-logs-%{+YYYY.MM.dd}"
    }
  }
}

Capture a few real failing lines and test them by hand — jq will point at the exact offending character:

# Pull raw failing payloads that Logstash quarantined
tail -n 20 /var/log/logstash/json-parse-failures-2026-07-10.log

# Validate a single suspect line; jq reports the precise parse position
echo '{"level":"info","msg":"started" ' | jq .

jq: error (at <stdin>:1): Unexpected end of input (Expected object) ...

The --log.level debug flag on Logstash will also print each event as it enters and leaves the filter, which is invaluable when you cannot tell which source is producing the bad lines.

Example Root Cause Analysis

A platform team routed both their Go API (structured JSON logs) and their legacy Nginx access logs into one pipeline that unconditionally applied a json { source => "message" } filter. Elasticsearch filled with documents tagged _jsonparsefailure, and logstash-plain.log logged Error parsing json several times a second.

Tailing the quarantine file revealed the culprit immediately:

192.0.2.10 - - [10/Jul/2026:10:00:01 +0000] "GET /health HTTP/1.1" 200 12

That is a plain-text Nginx line, not JSON — it does not start with {, so Jackson rejected it with Unexpected character ('1' (code 49)). The json filter was correct; the input mix was wrong. Two structured sources were being forced through one JSON-only parser.

The fix was to gate the json filter on a field the two sources did not share, and only parse when the line actually looks like JSON:

filter {
  if [message] =~ /^\s*\{/ {          # only attempt JSON on lines that start with {
    json {
      source => "message"
      target => "app"
      skip_on_invalid_json => true
    }
  } else {
    grok {                            # parse the Nginx lines with a pattern instead
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
  }
}

After deploying, the _jsonparsefailure rate dropped to zero, the API logs were fully structured under [app], and the Nginx lines were parsed by grok. The broader lesson: _jsonparsefailure almost always means the wrong data reached the filter, not that the filter is broken — segregate sources or guard the filter with a conditional so only genuine JSON is parsed.

Prevention Best Practices

• Never point a json filter at a mixed stream. Split sources by input, by a tag, or with an if [message] =~ /^\s*\{/ guard so only JSON lines are parsed.
• Set skip_on_invalid_json => true so obviously non-JSON lines are skipped cheaply instead of throwing per-event parser exceptions.
• Use a target (e.g. target => "app") so parsed fields nest under one key and the original message is preserved for replay when a failure does occur.
• Route _jsonparsefailure events to a dedicated file or index rather than mixing them into your main data, so failures are visible and replayable, not buried.
• Fix producers to emit one compact JSON object per line (JSON Lines); avoid pretty-printed multi-line JSON and JavaScript-isms like single quotes or trailing commas.
• Strip any leading syslog/timestamp prefix with grok or dissect before the json filter so the field handed to json starts at the {.

Quick Command Reference

# Validate the pipeline syntax
sudo -u logstash /usr/share/logstash/bin/logstash \
  -f /etc/logstash/conf.d/app-json.conf \
  --config.test_and_exit --path.settings /etc/logstash

# Count parse failures in the Logstash log
grep -c 'Error parsing json\|JSON parse error' /var/log/logstash/logstash-plain.log

# Inspect quarantined raw payloads
tail -n 50 /var/log/logstash/json-parse-failures-*.log

# Validate a single suspect line and get the exact failure position
echo '{"level":"info","msg":"started" ' | jq .

# Run Logstash in the foreground with debug logging to trace events
sudo -u logstash /usr/share/logstash/bin/logstash \
  -f /etc/logstash/conf.d/app-json.conf --log.level debug --path.settings /etc/logstash

Conclusion

The _jsonparsefailure tag and the JSON parse error, original data now in message field warning are Logstash telling you it protected your data: the text it was handed was not valid JSON, so it kept the raw string in message and flagged the event instead of dropping it. The fix is rarely in the parser and almost always in the data path — a plain-text line in a JSON-only stream, a truncated or concatenated document, or a leading prefix before the {. Guard the json filter with a conditional, set skip_on_invalid_json, and route failures to their own index so you can see and replay them. Validate suspect lines with jq, and confirm the fix by watching the failure count in logstash-plain.log fall to zero.