🚨 Filebeat Error Guide: 'Error decoding JSON' — Fix JSON Log Parsing Failures

Overview

When you tell Filebeat to parse log lines as JSON and a line isn’t valid JSON, the input logs a decode error and passes the line through as raw text with an error field:

Error decoding JSON: invalid character 'x' looking for beginning of value

You enable JSON parsing either with the legacy json.* options on a log input or the parsers: - ndjson: block on a filestream input. Filebeat then expects exactly one complete JSON object per line. If a line is plain text, truncated, pretty-printed across multiple lines, or double-encoded, the decoder fails. By default Filebeat still ships the line — but the structured fields you wanted are missing and an error.message is attached, so downstream dashboards break.

Symptoms

• Repeated Error decoding JSON lines in the Filebeat log, often one per bad source line.
• Documents in Elasticsearch have an error.type: json / error.message field instead of parsed keys.
• Some events parse correctly and others don’t, from the same file.
• Multiline / pretty-printed JSON produces unexpected end of JSON input.
• Fields like @timestamp or message behave oddly after a key collision.

Common Root Causes

• Non-JSON lines mixed into a JSON file — startup banners, stack traces, or plain-text lines among NDJSON.
• Pretty-printed (multiline) JSON — one object spread across several lines instead of one object per line.
• Truncated lines from log rotation cutting a line in half.
• Double-encoded JSON — a JSON string that itself contains escaped JSON.
• Key collisions — a JSON key like message or type overwriting a Filebeat-managed field without overwrite_keys / proper target.

Diagnostic Workflow

Confirm the source really is one-object-per-line NDJSON, then configure the parser to match.

head -n 5 /var/log/app/app.json | jq .        # each line must be a standalone JSON object
grep -vE '^\{.*\}$' /var/log/app/app.json | head   # find non-object lines

Use the filestream NDJSON parser and route decoded fields under a target key to avoid collisions:

# /etc/filebeat/filebeat.yml
filebeat.inputs:
  - type: filestream
    id: app-json
    paths: ["/var/log/app/app.json"]
    parsers:
      - ndjson:
          target: "json"          # nest parsed fields, avoid clobbering top-level
          add_error_key: true     # attach error.message on failure instead of silent drop
          overwrite_keys: false
          message_key: message    # apply multiline/timestamp to this key

filebeat test config -c /etc/filebeat/filebeat.yml
filebeat -e -c /etc/filebeat/filebeat.yml -d "processors,input"

If the source is pretty-printed, reassemble objects with a multiline parser before NDJSON:

    parsers:
      - multiline:
          type: pattern
          pattern: '^\{'
          negate: true
          match: after
      - ndjson:
          target: "json"

Example Root Cause Analysis

A service logged clean NDJSON, but its startup emitted two human-readable banner lines before switching to JSON. Filebeat parsed every JSON line fine and logged Error decoding JSON: invalid character 'S' looking for beginning of value for the banners (Starting service...). Those two events landed with error.message and no fields.

The banners were harmless, so the fix was to keep add_error_key: true (so failures are visible, not silent) and drop the un-parseable startup lines with an exclude_lines filter:

- type: filestream
  id: app-json
  paths: ["/var/log/app/app.json"]
  exclude_lines: ['^Starting service', '^Loaded config']
  parsers:
    - ndjson: { target: "json", add_error_key: true }

Non-JSON noise was excluded at the input; genuine JSON parsed cleanly with no more decode errors.

Prevention Best Practices

• Emit strict one-object-per-line NDJSON from applications — never pretty-print logs destined for Filebeat.
• Always parse JSON under a target: key so application keys can’t collide with Filebeat’s top-level fields.
• Keep add_error_key: true so malformed lines are flagged, not silently mishandled.
• Use exclude_lines to drop known non-JSON noise (banners, blank lines) before the NDJSON parser.
• If objects span lines, chain a multiline parser before ndjson to reassemble them.

Quick Command Reference

head -n5 app.json | jq .                    # verify each line is valid JSON
grep -vE '^\{.*\}$' app.json                 # find non-object lines
filebeat test config                         # validate parser config
filebeat -e -d "processors,input"            # watch decode behavior live
# config: parsers -> ndjson { target, add_error_key, overwrite_keys }
# config: exclude_lines to drop non-JSON noise

Conclusion

Error decoding JSON means a line Filebeat was told to treat as JSON isn’t valid JSON — usually plain-text noise, pretty-printed objects, or truncation. Verify the source is strict NDJSON with jq, parse under a target: key with add_error_key: true, exclude known non-JSON lines, and chain a multiline parser when objects span lines. Fixing the log format at the source is the cleanest long-term answer; the input-level config handles the stragglers.