🚨 Loki Error Guide: 'max line size exceeded' — Truncate, Split, or Raise max_line_size

Overview

Loki’s distributor drops individual log entries that are larger than max_line_size. The push response and distributor logs show:

rpc error: code = Code(400) desc = max line size (256KB) exceeded while adding (312KB) line to stream {app="checkout"}

Only the oversized entries are rejected — the rest of the batch is accepted — so this often shows up as silently missing lines rather than a hard failure. It exists to stop pathological entries (giant stack traces, base64 payloads, minified JSON dumps) from bloating chunks and slowing queries.

Symptoms

• Specific big log lines never appear in Loki while smaller ones from the same stream do.
• loki_discarded_samples_total{reason="line_too_long"} increases.
• Distributor logs contain max line size ... exceeded.
• The gaps correlate with error paths that emit large stack traces or serialized objects.
• Clients (Alloy/Promtail) may not surface the drop clearly since the batch otherwise succeeds.

Common Root Causes

• Multiline stack traces collapsed into one entry — a pipeline that joins a whole Java/Python traceback into a single line.
• Serialized payloads in logs — full request/response bodies, base64 blobs, or large JSON logged verbatim.
• Verbose debug logging — dumping entire objects or config at DEBUG level.
• max_line_size too small — the default (256KB) below what a legitimate structured log needs.
• Multiline stage misconfiguration — a multiline regex that never terminates, concatenating many lines.
• Third-party libraries logging large diagnostic dumps.

Diagnostic Workflow

Confirm entries are being dropped for line length:

sum by (tenant) (rate(loki_discarded_samples_total{reason="line_too_long"}[5m]))

Read the distributor logs to find the offending stream and size:

kubectl logs -l app.kubernetes.io/component=distributor --since=15m \
  | grep -i "max line size"

Check the configured limit:

limits_config:
  max_line_size: 256KB
  max_line_size_truncate: false   # true = truncate instead of drop

Find oversized lines at the source before they reach Loki (example with Alloy loki.process to drop or truncate):

loki.process "trim" {
  stage.drop {
    longer_than = "256KB"
    drop_counter_reason = "line_too_long"
  }
  forward_to = [loki.write.default.receiver]
}

Or, keep the line but truncate on the Loki side instead of dropping:

limits_config:
  max_line_size: 256KB
  max_line_size_truncate: true

Fix a runaway multiline stage so tracebacks aren’t concatenated indefinitely:

stage.multiline {
  firstline     = "^\\d{4}-\\d{2}-\\d{2}"
  max_lines     = 128
  max_wait_time = "3s"
}

Example Root Cause Analysis

A checkout service’s error logs were mysteriously missing exactly the entries engineers most wanted — the ones with full stack traces. loki_discarded_samples_total{reason="line_too_long"} showed a steady trickle, and distributor logs reported max line size (256KB) exceeded while adding (312KB) on {app="checkout"}.

The cause was a multiline stage with no max_lines cap: deep exceptions with hundreds of frames plus a serialized request body were collapsed into one 300KB+ line. Two changes fixed it: they set max_lines = 128 and max_wait_time = 3s on the multiline stage so a single logical event stayed bounded, and enabled max_line_size_truncate: true as a safety net so any still-oversized line was truncated (and kept) rather than silently dropped. Missing error logs returned, and the truncation counter showed the safety net rarely triggered — the multiline cap was the real fix.

Prevention Best Practices

• Cap multiline stages with max_lines and max_wait_time so tracebacks can’t concatenate without bound.
• Avoid logging full request/response bodies or base64 blobs; log references/IDs and store payloads elsewhere.
• Set max_line_size_truncate: true so oversized lines are truncated (and preserved) instead of dropped.
• Alert on loki_discarded_samples_total{reason="line_too_long"} so silent drops are visible.
• Raise max_line_size per tenant only when a legitimate structured-log format needs it, and mind chunk/query impact.
• Keep DEBUG object dumps out of production.

Quick Command Reference

# Drops due to oversized lines
sum by (tenant) (rate(loki_discarded_samples_total{reason="line_too_long"}[5m]))

# Distributor logs naming the stream and size
kubectl logs -l app.kubernetes.io/component=distributor --since=15m | grep -i "max line size"

# Effective limit and truncate setting
logcli --addr=http://loki-gateway/ config | grep -iE 'max_line_size'

Conclusion

max line size exceeded means individual entries are larger than max_line_size and are being dropped at the distributor — usually giant stack traces, serialized payloads, or a multiline stage with no line cap. Find the source with the discard metric and distributor logs, cap multiline stages, and stop logging huge payloads. Enable max_line_size_truncate so oversized lines are preserved-but-truncated rather than lost, alert on the discard counter, and raise the limit per tenant only when a real log format demands it.