🚨 Filebeat Error Guide: 'mapper_parsing_exception' — Fix Field Parse Failures

Overview

Elasticsearch rejects individual bulk items when a field value cannot be coerced into the type its mapping declares. Filebeat surfaces the rejection from the _bulk response and, because it is non-retryable, drops the offending event after logging:

Failed to publish events caused by: mapper_parsing_exception: failed to parse field [http.response.status_code] of type [long] in document with id 'AX...'. Preview of field's value: 'unknown'

The mapping for http.response.status_code is long, but the event carried the string "unknown". Elasticsearch cannot cast it, so it rejects just that document ("status": 400 in the bulk item) rather than the whole batch. Filebeat treats mapping errors as permanent — retrying would fail identically — so it discards the event and moves on. The data is genuinely lost unless you fix the source or the mapping and replay.

Symptoms

• mapper_parsing_exception: failed to parse field [...] in the Filebeat log, naming a specific field and its declared type.
• Some events are missing from Kibana while most from the same source index fine — only the malformed ones drop.
• The bulk response shows per-item "status": 400 while overall "errors": true.
• libbeat.output.events.dropped increments; filebeat.events.done still climbs for the good events.
• Errors cluster around a field that recently changed shape (e.g. a number that sometimes arrives as text).

Common Root Causes

• Type conflict — a field mapped as long/date/ip receives a string like "unknown", "-", or "N/A".
• Dynamic mapping locked early — the first document set the type, and later documents disagree.
• Date format mismatch — a timestamp string does not match the field’s format, so date parsing fails.
• Object vs scalar clash — a field is sometimes a string and sometimes a nested object.
• Ingest pipeline gap — a convert/grok processor that should normalize the field is missing or failing silently.
• Malformed source logs — the application emits placeholder text in a numeric field during errors.

Diagnostic Workflow

Read the Filebeat log to capture the exact field, type, and rejected value:

journalctl -u filebeat --since '30 min ago' | grep -i mapper_parsing_exception

Inspect the current mapping for that field to confirm its declared type:

curl -sk 'https://es01:9200/filebeat-*/_mapping/field/http.response.status_code?pretty' \
  -u elastic:$ES_PASS

Reproduce the rejection with a one-off bulk request carrying the bad value:

curl -sk -H 'Content-Type: application/x-ndjson' -u elastic:$ES_PASS \
  'https://es01:9200/filebeat-000001/_bulk' --data-binary $'
{"index":{}}
{"http":{"response":{"status_code":"unknown"}}}
' | jq '.items[0].index.error'

If you use ignore_malformed as a stopgap, verify which docs were flagged:

curl -sk 'https://es01:9200/filebeat-*/_search?pretty' -u elastic:$ES_PASS \
  -H 'Content-Type: application/json' -d '{"query":{"exists":{"field":"_ignored"}},"size":1}'

Example Root Cause Analysis

An access-log pipeline indexed cleanly for weeks, then started dropping a trickle of events with failed to parse field [http.response.status_code] of type [long]. The _mapping/field call confirmed long; the reproduction bulk with "status_code":"unknown" returned the identical mapper_parsing_exception. The upstream service, when it could not reach a backend, logged status_code=unknown instead of a numeric code.

The durable fix was an ingest pipeline that coerces or drops the non-numeric value before indexing:

PUT _ingest/pipeline/filebeat-http-fix
{ "processors": [
  { "convert": { "field": "http.response.status_code", "type": "long",
                 "ignore_failure": true } },
  { "remove": { "field": "http.response.status_code", "if":
      "ctx.http?.response?.status_code instanceof String" } }
] }

Pointing output.elasticsearch.pipeline: filebeat-http-fix at it stopped the drops — malformed values were normalized instead of rejected, and valid codes indexed as numbers.

Prevention Best Practices

• Define explicit index templates for your fields rather than relying on the first document’s dynamic type.
• Normalize types in an ingest pipeline (convert, grok, script) close to the source so placeholders never reach the mapping.
• Fix the application to emit consistent types — a numeric field should never carry "unknown" or "-".
• Use ignore_malformed: true on volatile numeric fields as a safety net so one bad value does not drop the whole document.
• Alert on libbeat.output.events.dropped so silent per-event data loss is visible.

Quick Command Reference

journalctl -u filebeat --since '30 min ago' | grep mapper_parsing_exception
curl -sk 'https://es01:9200/filebeat-*/_mapping/field/<field>?pretty' -u elastic:$ES_PASS
curl -sk 'https://es01:9200/_bulk' -H 'Content-Type: application/x-ndjson' -u elastic:$ES_PASS --data-binary @probe.ndjson | jq '.items[0]'
curl -sk 'https://es01:9200/_ingest/pipeline/filebeat-http-fix?pretty' -u elastic:$ES_PASS

Conclusion

mapper_parsing_exception is a data-shape mismatch, not a connectivity problem: a value’s type disagrees with its mapping, so Elasticsearch rejects just that document and Filebeat drops it. Read the field name and rejected value from the log, confirm the mapping, then fix the type at the source or with an ingest convert — and add ignore_malformed so a stray value never costs you a whole event. More mapping fixes are in the Filebeat guides.