🚨 Filebeat Error Guide: 'registry file is corrupt' — Recover a Broken Filebeat Registry

Overview

Filebeat reports a corrupt registry when the on-disk state it uses to remember read offsets can no longer be parsed or validated:

registry file is corrupt, could not read active markers: /var/lib/filebeat/registry/filebeat/active.dat

The registry is Filebeat’s memory of “how far did I read in each file.” It is stored under path.data as a set of small data files plus a meta.json. When one of those files is truncated, half-written, or has an inconsistent marker, Filebeat can no longer trust the offsets. Depending on version it will either exit or drop back to reading files from the start. Either way, ingestion is disrupted and you risk duplicated or missed lines until the registry is repaired or reset.

Symptoms

• Startup logs mention registry file is corrupt, could not read active markers, or a bad meta.json.
• Filebeat re-reads entire files from the beginning, flooding downstream with duplicates.
• The service crash-loops, or starts but never advances offsets.
• The host suffered a crash, disk-full event, or was cloned from an image with a stale registry.
• active.dat, log.json, or meta.json under the registry directory are size zero or unexpectedly small.

Common Root Causes

• Truncated write from a crash or power loss while Filebeat was flushing the registry.
• Disk full so the registry data file was only partially written.
• VM/image cloning that captured a registry mid-write, or copied a registry that no longer matches the files present.
• Manual edits to registry files (never edit them by hand).
• Filesystem corruption on the path.data volume itself.

Diagnostic Workflow

Confirm which registry file is broken and whether the offsets are recoverable before you reset anything.

ls -la /var/lib/filebeat/registry/filebeat/
sudo cat /var/lib/filebeat/registry/filebeat/meta.json          # should be small valid JSON
file /var/lib/filebeat/registry/filebeat/*.dat

Validate config and run a dry foreground start to capture the exact failure:

# /etc/filebeat/filebeat.yml
path.data: /var/lib/filebeat
filebeat.inputs:
  - type: filestream
    id: app
    paths: ["/var/log/app/*.log"]
    ignore_older: 72h        # bounds a rebuild so old files aren't re-read

filebeat test config -c /etc/filebeat/filebeat.yml
sudo systemctl stop filebeat
filebeat -e -c /etc/filebeat/filebeat.yml         # watch the corruption error live

Back up the broken registry before removing it, in case you want to inspect offsets later:

sudo cp -a /var/lib/filebeat/registry /root/filebeat-registry.bak

Example Root Cause Analysis

A fleet was scaled by cloning a golden VM image that had Filebeat installed and running at snapshot time. Every clone booted with a meta.json captured mid-flush. On first boot each node logged registry file is corrupt and re-read all existing logs, doubling ingestion volume and spiking Elasticsearch load.

The root cause was baking a live registry into the image. The fix on each node was to reset the registry once, and the permanent fix was to clear the registry during image build:

sudo systemctl stop filebeat
sudo rm -rf /var/lib/filebeat/registry
sudo systemctl start filebeat

The golden image was rebuilt with systemctl stop filebeat && rm -rf /var/lib/filebeat/registry as the last provisioning step, so future clones start with a clean state.

Prevention Best Practices

• Always stop Filebeat and delete path.data/registry as the final step when building golden images or containers.
• Alert on the path.data volume filling up so registry flushes never truncate.
• Never hand-edit registry files; treat them as opaque Filebeat state.
• Keep a short registry flush interval on busy hosts to minimize the corruptible window.
• Set ignore_older on inputs so that any registry reset re-reads only recent files, not months of history.

Quick Command Reference

ls -la /var/lib/filebeat/registry/filebeat/        # inspect registry files
cat .../meta.json                                    # should be valid JSON
sudo cp -a /var/lib/filebeat/registry /root/reg.bak  # back up before reset
sudo systemctl stop filebeat                         # stop before touching registry
sudo rm -rf /var/lib/filebeat/registry               # reset corrupt registry
sudo systemctl start filebeat                        # rebuild clean registry
filebeat -e -c filebeat.yml                          # foreground to confirm recovery

Conclusion

A corrupt registry is Filebeat losing its record of read offsets, usually from a crash, disk-full event, or a cloned image captured mid-write. Back up the broken registry, then reset it and let Filebeat rebuild — pairing the reset with ignore_older avoids a flood of duplicate logs. The durable fix is process discipline: stop Filebeat and clear the registry when baking images, and alert on the data volume so writes never truncate.