🚨 Filebeat Error Guide: 'could not start registrar: Error loading state' — Fix Registry Load Failures

Overview

Filebeat writes this at startup and then exits when the registrar — the component that persists read offsets — cannot load its saved state from disk:

Exiting: could not start registrar: Error loading state: Error decoding states: invalid character '\x00' looking for beginning of value

The registrar keeps track of how far Filebeat has read in every file so it can resume without duplicating or losing lines. On startup it reads the registry under the data path (default /var/lib/filebeat/registry). If that state cannot be decoded — because the file is truncated, zero-filled from a crash, permission-blocked, or JSON-corrupt — Filebeat treats it as fatal and refuses to run rather than risk shipping wrong offsets.

Symptoms

• Filebeat exits immediately on start; systemctl status filebeat shows it in a restart loop.
• Logs contain Exiting: could not start registrar: Error loading state.
• The decode error mentions invalid character '\x00', unexpected end of JSON input, or Error decoding states.
• The host recently had a hard power loss, OOM kill, or full disk during a Filebeat write.
• The data path was moved, restored from a backup, or has wrong ownership after a redeploy.

Common Root Causes

• Unclean shutdown mid-write — a crash or power loss while the registry was being flushed leaves a truncated or null-byte-padded file.
• Disk full during registry write — the registry gets partially written and corrupted.
• Wrong permissions on the data path — Filebeat can list the directory but cannot read the registry file it needs.
• Registry restored from an inconsistent backup — a snapshot copied while Filebeat was writing.
• Registry format mismatch — pointing a newer Filebeat at an old registry file layout without migration, or vice versa.

Diagnostic Workflow

Locate the data path, then inspect the registry file for corruption before deciding whether to repair or reset.

grep -i 'path.data' /etc/filebeat/filebeat.yml    # default: /var/lib/filebeat
ls -la /var/lib/filebeat/registry/filebeat/
file /var/lib/filebeat/registry/filebeat/*.json

Check the file for null bytes or truncation (the classic \x00 cause):

sudo hexdump -C /var/lib/filebeat/registry/filebeat/log.json | tail    # trailing 00 00 00 = corruption
sudo python3 -m json.tool /var/lib/filebeat/registry/filebeat/log.json  # fails on invalid JSON

Verify the config and data-path access as the service user:

# /etc/filebeat/filebeat.yml (relevant keys)
path.data: /var/lib/filebeat
filebeat.registry.flush: 1s     # flush more often to shrink the crash window

filebeat test config -c /etc/filebeat/filebeat.yml
sudo -u filebeat test -r /var/lib/filebeat/registry/filebeat/log.json && echo readable

Example Root Cause Analysis

A node’s disk filled to 100% overnight. Filebeat tried to flush the registry, got ENOSPC, and left log.json padded with null bytes. On the next restart the registrar failed with invalid character '\x00' looking for beginning of value and Filebeat entered a crash loop.

hexdump -C confirmed a trailing run of 00 bytes. Because the offsets were unrecoverable, the fix was to free disk, stop Filebeat, remove the corrupt registry, and let Filebeat rebuild it:

sudo systemctl stop filebeat
sudo rm -rf /var/lib/filebeat/registry
sudo systemctl start filebeat

Filebeat re-created a fresh registry and re-read files from the configured start position. To avoid re-ingesting everything, they set ignore_older on the inputs so only recent files were re-read.

Prevention Best Practices

• Alert on disk usage for the path.data volume well before 85% so registry writes never hit ENOSPC.
• Keep filebeat.registry.flush short (e.g. 1s) on high-throughput hosts to shrink the window a crash can corrupt.
• Never snapshot/restore the registry while Filebeat is running — stop the service first.
• Ensure the data path is owned by and readable/writable to the Filebeat service user across redeploys.
• When you must reset the registry, set ignore_older on inputs so a rebuild doesn’t re-ship gigabytes of old logs.

Quick Command Reference

grep -i 'path.data' /etc/filebeat/filebeat.yml         # find data path
ls -la /var/lib/filebeat/registry/filebeat/            # inspect registry files
hexdump -C .../log.json | tail                          # look for trailing null bytes
python3 -m json.tool .../log.json                       # validate registry JSON
sudo systemctl stop filebeat                            # stop before touching registry
sudo rm -rf /var/lib/filebeat/registry                  # reset a corrupt registry
sudo systemctl start filebeat                           # rebuild fresh registry

Conclusion

could not start registrar: Error loading state is Filebeat refusing to run on a corrupt or unreadable registry rather than risking wrong offsets. Confirm the cause with hexdump/json.tool — usually null bytes from a disk-full crash — then either fix permissions or delete the registry so Filebeat rebuilds it. Pair a reset with ignore_older to avoid re-ingesting old data, and prevent recurrence by alerting on the data-path disk and keeping the registry flush interval short.