🚨 Filebeat Error: 'version_conflict_engine_exception' — Cause, Fix, and Troubleshooting Guide

Overview

Elasticsearch rejects a bulk item with a version conflict when a document with the same _id already exists and the write is a create (or an out-of-order versioned update). With Filebeat this shows up when events carry a deterministic _id:

Cannot index event ...: 409 Conflict: {"type":"version_conflict_engine_exception","reason":"[abc123]: version conflict, document already exists (current version [1])","index":"filebeat-8.13.0-2026.07.12"}

By default Filebeat lets Elasticsearch auto-generate IDs, so conflicts are rare. They appear when you set a custom document _id (via add_id, a fingerprint processor, or pipeline-set _id) and the same event is shipped twice — after a restart, a registry reset, or two Filebeats reading the same file. The conflicting item is dropped as a duplicate; unique events still index fine.

Symptoms

• version_conflict_engine_exception / 409 Conflict on specific bulk items.
• Duplicate-suppression is intended (custom _id) but conflicts spike after a restart or replay.
• libbeat.output.events.failed (or dropped) increments while others succeed.
• Two Filebeat instances harvesting the same paths.

Common Root Causes

• Custom deterministic _id plus re-shipping the same events (restart, registry loss).
• Two Filebeats reading the same file and writing the same _id.
• op_type: create on an ID that already exists.
• Registry reset/rollback causing a replay of already-indexed lines.
• Fingerprint collisions — a fingerprint/add_id field that is not actually unique.

How to diagnose

Check whether events set a custom _id and inspect the processors:

grep -A4 'processors' /etc/filebeat/filebeat.yml
grep -Ei 'add_id|fingerprint|_id' /etc/filebeat/filebeat.yml

Confirm only one Filebeat harvests the paths and inspect a conflicting doc:

pgrep -a filebeat
curl -sk 'https://es01:9200/filebeat-*/_doc/abc123?pretty' -u elastic:$ES_PASS

Fixes

If duplicate suppression is intentional and conflicts are just idempotent replays, they are usually safe to ignore — the duplicate is simply not re-indexed. To silence the noise, keep auto-generated IDs unless you truly need dedup:

# Remove custom _id to let ES assign unique IDs (no conflicts possible)
processors:
  # - add_id: ~        # remove or comment out if dedup is not required

If you need idempotency, base the fingerprint on genuinely unique fields:

processors:
  - fingerprint:
      fields: ["message", "@timestamp", "log.file.path", "log.offset"]
      target_field: "@metadata._id"
      method: "sha256"

Ensure only one Filebeat harvests each path (dedupe inputs / fix a duplicate deployment), and avoid registry resets that force replays.

What to watch out for

• With auto-generated IDs this error cannot occur — it always implies a custom _id.
• Two Filebeats on the same files is a classic cause; check for stray instances.
• A weak fingerprint (too few fields) causes false collisions across distinct events.
• Idempotent replays are safe; investigate only if unique events are being dropped.

Related

• Filebeat Error Guide: ‘data path already locked by another beat’
• Filebeat Error Guide: ‘id already exists in the filestream input’
• Filebeat Error Guide: ‘mapper_parsing_exception’