🚨 Loki Error Guide: 'pipeline error' (__error__="JSONParserErr") — Handle Lines That Don't Match Your Parser

Overview

When a LogQL query runs a parser stage over a stream and some lines do not match that format, Loki does not fail the whole query. Instead it attaches a special error label to each offending line and Grafana surfaces the result as a “pipeline error.” The lines carry a __error__ label naming the parser that failed:

__error__="JSONParserErr"
__error__="LogfmtParserErr"

This is distinct from a LogQL syntax error, where the query text itself is malformed and nothing runs. Here the query is valid — it just met a line it could not parse. The most common cause is applying | json or | logfmt to a stream that mixes formats: some lines are structured, some are plain text, a startup banner, or a stack trace. The fix is not to remove the parser but to handle the non-matching lines deliberately: filter them out, inspect them, or scope the parser so it only runs where it applies.

Symptoms

• Grafana shows a “pipeline error” annotation on a panel even though results still return.
• Some result rows carry __error__="JSONParserErr" or __error__="LogfmtParserErr" while others parse cleanly.
• Fields you expect from | json/| logfmt are empty for a subset of lines.
• A | label_format or | line_format referencing a missing label produces errors on lines lacking that field.
• The error rate correlates with specific line types (banners, multi-line traces, non-JSON access logs) mixed into an otherwise structured stream.

Common Root Causes

• Mixed formats in one stream — the app emits JSON for most events but plain-text startup/shutdown banners or panics that | json cannot parse.
• | logfmt on non-logfmt lines — a stream where only some lines are key=value formatted, so the rest fail with LogfmtParserErr.
• Parser applied too broadly — | json runs across a multi-tenant or multi-app selector where only one app actually emits JSON.
• label_format on a missing label — a template references a field that some lines do not have, producing per-line errors.
• Nested or malformed JSON — lines that are truncated, double-encoded, or contain embedded newlines that break strict JSON parsing.
• Line filters placed after the parser — filtering on a parsed field that is empty on failed lines, so failures leak into the output instead of being excluded.

How to diagnose

1. Confirm it is a pipeline error, not a syntax error — a syntax error rejects the query outright; a pipeline error returns rows with __error__ set. Inspect the failing lines directly:

   {namespace="prod", app="api"} | json | __error__ != ""

2. Identify which parser is failing by reading the error value on those rows:

   {namespace="prod", app="api"} | json | line_format "{{__error__}} {{__line__}}"

3. Count the failure rate to judge how much of the stream is affected:

   sum(count_over_time({namespace="prod", app="api"} | json | __error__="JSONParserErr" [5m]))

4. Check the stream really is single-format by sampling raw lines with logcli:

   logcli query '{namespace="prod", app="api"}' --limit=20 --since=15m \
     --addr=https://loki-gateway/ --org-id=tenant-a

5. Look for a too-broad selector feeding the parser — if the selector spans multiple apps, narrow it so the parser only sees the format it expects:

   {namespace="prod"} | json    # too broad; some apps are not JSON

Fixes

Filter out failed lines after the parser — keep only successfully parsed lines by matching an empty error label, which clears the pipeline error:

{namespace="prod", app="api"} | json | __error__="" | level="error"

Inspect the failures on purpose when you want to see what did not parse, by selecting the non-empty error label instead:

{namespace="prod", app="api"} | json | __error__ != ""

Scope the parser to the stream that actually matches — tighten the selector so | json/| logfmt only runs where lines are structured:

{namespace="prod", app="api", format="json"} | json | status="500"

Parse selectively into a single field rather than the whole line, which avoids failing on lines that are not fully JSON:

{namespace="prod", app="api"} | json message="log.msg" | message=~"timeout.*"

Drop the error label when it is expected noise — once you have decided the failures are benign, remove __error__ (and __error_details__) so panels render cleanly:

{namespace="prod", app="api"} | logfmt | __error__="" | drop __error__

What to watch out for

• A pipeline error is not a syntax error — do not “fix” valid query text; fix how non-matching lines are handled.
• | __error__="" silently discards the failed lines; if those lines matter (panics, non-JSON errors), inspect them with | __error__ != "" before excluding them.
• Applying | json to a mixed stream is a smell — the cleaner fix is often to split formats at ingest so each stream is uniform.
• label_format/line_format referencing a missing label will error per line; guard templates or ensure the field exists first.
• drop __error__ only hides the label; it does not make the line parse — the underlying data is still unparsed, so do not rely on parsed fields from those lines.

Related

• Loki Error Guide: ‘parse error’ (LogQL syntax) — the syntax-error sibling where the query text itself is invalid.
• Loki Error Guide: ‘queries require at least one matcher’ — another query-shape rule that trips exploratory LogQL.
• Loki Error Guide: ‘max entries limit per query exceeded’ — the result-size limit you hit once the parser and filters are correct.