🚨 Logstash Error: '_jsonparsefailure' — Cause, Fix, and Troubleshooting Guide

Overview

The Logstash json filter takes a field, parses it as a JSON document, and merges the result into the event. When the field it was told to parse is not valid JSON, the filter cannot fail the event outright — it adds the tag _jsonparsefailure and passes the event through untouched. In /var/log/logstash/logstash-plain.log the underlying parse error is logged like this:

[WARN ][logstash.filters.json ][main] Error parsing json {:source=>"message", :raw=>"{\"level\":\"info\", \"msg\":\"started\"", :exception=>#<LogStash::Json::ParserError: Unexpected end-of-input: expected close marker for Object>}

This is the filter-side failure, and it is distinct from a Jackson JSON parse error raised at the elasticsearch output (that is a different problem, covered separately). Here, the json filter is telling you that the specific field named in :source — usually message — did not contain a complete, well-formed JSON object. The :raw value in the log is the exact text it choked on, which is the single most useful clue: in the example above the object is truncated, missing its closing }.

Symptoms

• Events arrive in Elasticsearch carrying a _jsonparsefailure tag but with the JSON left unparsed inside message.
• logstash-plain.log shows Error parsing json with a LogStash::Json::ParserError and a :raw snippet.
• The exception text hints at the shape of the problem: Unexpected end-of-input (truncated), Unexpected character (not JSON), or Unrecognized token.
• Some events parse fine and others fail — typically the failing ones are large or multiline records that were split.
• Downstream dashboards show fields missing on a subset of events because the merge never happened.

Common Root Causes

• The field is not JSON at all — plain text, a syslog line, or a key=value log was routed into a json filter.
• Truncated JSON from split multiline logs — a pretty-printed or stack-trace-bearing JSON record spanning several lines was ingested as several separate events, so each event holds a fragment.
• Invalid JSON dialect — single quotes instead of double quotes, trailing commas, unquoted keys, or NaN/Infinity values that strict JSON rejects.
• Concatenated objects — two JSON documents on one line ({...}{...}) with no array wrapper; strict parsers stop after the first.
• Wrong source field — the filter points at a field that holds only part of the payload, or a field that is already a parsed object.
• A codec already parsed it — an input codec => json plus a redundant json filter double-parses, and the second pass sees a non-string.

How to diagnose

Start by isolating the exact bytes that failed. The :raw value in the log is the failing input; copy it and validate it with a strict parser:

# Pull recent failures and their raw payloads
grep 'logstash.filters.json' /var/log/logstash/logstash-plain.log | tail -20

# Validate a captured payload with a strict JSON parser
echo '{"level":"info", "msg":"started"' | jq .
# parse error: Unexpected end of input — confirms truncation

Route tagged events to a file so you can inspect real failures without touching Elasticsearch. Add a temporary debug output keyed on the tag:

output {
  if "_jsonparsefailure" in [tags] {
    file { path => "/var/lib/logstash/json-failures.log" }
  }
}

Then look at whether the failures are fragments of a larger record. If the raw values are consistently missing their opening { or closing }, the payload is being split before Logstash — the framing is the problem, not the parser:

# Are failures short fragments rather than whole documents?
awk '{ print length, $0 }' /var/lib/logstash/json-failures.log | sort -n | head

For reference, a strict json filter that only runs on plausibly-JSON input looks like this:

filter {
  if [message] =~ /^\s*\{/ {
    json {
      source => "message"
      skip_on_invalid_json => true
    }
  }
}

Fixes

Guard the filter with a conditional so only messages that actually look like JSON are parsed. This alone eliminates _jsonparsefailure on mixed log streams:

filter {
  if [message] =~ /^\s*\{/ {
    json { source => "message" }
  }
}

Set skip_on_invalid_json => true so a non-JSON field is skipped quietly instead of tagging the event and logging a warning for every line:

filter {
  json {
    source => "message"
    skip_on_invalid_json => true
  }
}

Fix multiline framing upstream when the failures are truncated documents. Each event must be one complete JSON record. If Filebeat ships the logs, stitch the lines there so Logstash receives whole documents:

# filebeat.yml — reassemble a JSON object that starts with "{"
filebeat.inputs:
  - type: filestream
    id: app-json
    paths:
      - /var/log/app/*.json.log
    parsers:
      - multiline:
          type: pattern
          pattern: '^\{'
          negate: true
          match: after

Route the survivors aside rather than letting malformed data pollute your index. Tag-based branching keeps clean data flowing while you triage the rest:

output {
  if "_jsonparsefailure" in [tags] {
    file { path => "/var/lib/logstash/json-failures-%{+YYYY.MM.dd}.log" }
  } else {
    elasticsearch { hosts => ["https://es.internal:9200"] index => "app-%{+YYYY.MM.dd}" }
  }
}

What to watch out for

• :raw is your evidence. Always read the raw snippet before changing config — it tells you whether you have non-JSON, truncation, or a dialect issue, which need different fixes.
• Do not double-parse. If an input codec => json already decoded the payload, a second json filter on message will fail or misbehave; remove one of them.
• Multiline belongs in one place. Do the stitching in Filebeat or with a multiline codec on a file input — never both, or you will re-split what you just joined.
• skip_on_invalid_json hides the signal. It stops the tag and the warning, which is convenient, but you lose visibility into genuinely broken producers — keep a sampled failure output while you stabilize.
• Trailing commas and single quotes are not JSON. If an upstream service emits them, fix the producer; Logstash’s parser is deliberately strict.

Related

• Logstash Jackson JSON parse error at the output
• Logstash ‘_grokparsefailure’ tag
• Logstash ‘_dateparsefailure’ tag