🚨 Filebeat Error Guide: 'mapping conflict' — Resolve Field Type Clashes

Overview

Elasticsearch rejects a bulk item when a field’s inferred type conflicts with the type already established in the index mapping. Filebeat surfaces the per-item rejection and, because it is non-retryable, drops the event:

Failed to publish events caused by: illegal_argument_exception: mapper [host.ip] cannot be changed from type [ip] to type [text]; mapping conflict for document with id 'AX...'

Once a field is mapped (say host.ip as ip), Elasticsearch will not silently change that type. A later event where the same field arrives as free text — or as an object where a scalar was expected — cannot be indexed into the existing mapping, so that document is rejected with illegal_argument_exception. This differs from mapper_parsing_exception (a value that fails to parse into a valid type): here two events disagree about the field’s shape, and the first one won. The conflicting events are lost until the shapes are reconciled.

Symptoms

• illegal_argument_exception: mapper [...] cannot be changed from type [X] to type [Y] in the Filebeat log.
• Only events from one source drop, while others sharing the index succeed.
• The bulk response shows per-item "status": 400 with mapper in the reason.
• libbeat.output.events.dropped climbs for the conflicting source.
• The clash appears after a new log source or a field-naming change reuses an existing field name with a different shape.

Common Root Causes

• Same field, two types across sources — one service sends host.ip as an IP, another as a hostname string.
• Scalar vs. object clash — a field is a string in some events and a nested object ({ "name": ... }) in others.
• Dynamic mapping locked by first doc — the earliest document fixed a type the later ones violate.
• Missing explicit template — relying on dynamic mapping lets whichever event arrives first win.
• Field reuse — a generic name like tags, error, or code used with different structures.
• Numeric-vs-keyword drift — a value indexed as long early, then arriving as a keyword-like string.

Diagnostic Workflow

Capture the exact field and the from/to types from the Filebeat log:

journalctl -u filebeat --since '30 min ago' | grep -i 'cannot be changed'

Inspect the field’s current mapping to see the established type:

curl -sk 'https://es01:9200/filebeat-*/_mapping/field/host.ip?pretty' -u elastic:$ES_PASS

Reproduce the conflict by sending the alternate shape:

curl -sk -H 'Content-Type: application/x-ndjson' -u elastic:$ES_PASS \
  'https://es01:9200/filebeat-000001/_bulk' --data-binary $'
{"index":{}}
{"host":{"ip":"web-frontend-01"}}
' | jq '.items[0].index.error'

Find which sources emit each shape so you know what to normalize:

curl -sk 'https://es01:9200/filebeat-*/_search?pretty' -u elastic:$ES_PASS \
  -H 'Content-Type: application/json' \
  -d '{"size":0,"aggs":{"srcs":{"terms":{"field":"agent.hostname"}}}}'

Example Root Cause Analysis

Two application teams shipped to the same filebeat-* data stream. Team A emitted host.ip as a real address; Team B, misusing the field, emitted the machine’s DNS name as a string. Because Team A’s events indexed first, _mapping/field/host.ip showed type ip. Team B’s events then failed with mapper [host.ip] cannot be changed from type [ip] to type [text] and were silently dropped.

The clean fix was to stop reusing host.ip for a hostname. An ingest pipeline renamed Team B’s misplaced value into the correct ECS field before indexing:

PUT _ingest/pipeline/filebeat-hostname-fix
{ "processors": [
  { "rename": { "field": "host.ip", "target_field": "host.name",
                "if": "!(ctx.host?.ip =~ /^\\d+\\.\\d+\\.\\d+\\.\\d+$/)",
                "ignore_missing": true } }
] }

With output.elasticsearch.pipeline: filebeat-hostname-fix, non-IP values moved to host.name and stopped colliding with the ip mapping. Existing conflicting data was reindexed into a corrected index. Long term, both teams adopted the ECS field definitions so the shapes could never diverge again.

Prevention Best Practices

• Define an explicit index template with locked field types so the first document cannot dictate the mapping.
• Standardize on ECS (Elastic Common Schema) field names and types across every source writing to the same index.
• Normalize divergent shapes in an ingest pipeline (rename, convert, script) before they hit the mapping.
• Give distinct log sources distinct indices/data streams when their schemas genuinely differ.
• Alert on libbeat.output.events.dropped so silent per-event loss from a mapping clash is visible immediately.

Quick Command Reference

journalctl -u filebeat --since '30 min ago' | grep 'cannot be changed'
curl -sk 'https://es01:9200/filebeat-*/_mapping/field/<field>?pretty' -u elastic:$ES_PASS
curl -sk 'https://es01:9200/_bulk' -H 'Content-Type: application/x-ndjson' -u elastic:$ES_PASS --data-binary @probe.ndjson | jq '.items[0]'
curl -sk 'https://es01:9200/_ingest/pipeline/filebeat-hostname-fix?pretty' -u elastic:$ES_PASS

Conclusion

A mapping conflict means two events disagree about a field’s type or shape and the first one already fixed the mapping — so the later ones drop. Read the from/to types, confirm the established mapping, then reconcile the shapes with an explicit template, ECS field names, or an ingest rename/convert. Separate genuinely different schemas into different indices, and alert on dropped events so nothing vanishes silently. More mapping fixes are in the Filebeat guides.