🚨 Filebeat Error Guide: 'harvester ... file was truncated' — Fix Truncated Log Handling

Overview

Filebeat logs this when a file it is harvesting suddenly becomes smaller than the offset it has already read to:

Harvester for file /var/log/app/app.log was truncated, offset reset from 1048576 to 0

Filebeat tracks a byte offset per file. On each read it expects the file to be at least as large as its stored offset. If the current size is smaller, Filebeat concludes the file was truncated in place — typically by a copytruncate log rotation or an application that reopens its log with O_TRUNC — and resets the offset to 0 so it can re-read from the new beginning. This is usually correct behavior, but a misconfigured rotation or an unexpected truncation can cause missed or duplicated lines around the event.

Symptoms

• Recurring was truncated, offset reset warnings, often on a schedule matching log rotation.
• A gap or burst of duplicated log lines right at rotation time.
• Logs briefly stop then resume after each rotation.
• The application uses logrotate with copytruncate, or reopens its own log file periodically.
• Truncation warnings on files that should only ever grow (a sign of unintended O_TRUNC).

Common Root Causes

• copytruncate rotation — logrotate copies then truncates the live file, so Filebeat sees the size drop.
• App reopening its log with truncate — some frameworks truncate on restart or log-level change.
• A shared file zeroed by another process (e.g. a > file redirect used to “clear” a log).
• Rapid rotation where truncation happens faster than Filebeat’s scan interval, so some lines are never read.
• Filesystem/container quirks reporting a smaller size transiently.

Diagnostic Workflow

Confirm how the file is rotated, then align Filebeat’s close/rotation handling with it.

grep -Rn 'copytruncate\|create\|/var/log/app' /etc/logrotate.d/
stat /var/log/app/app.log        # watch size/inode across a rotation
sudo logrotate -d /etc/logrotate.d/app   # dry-run to see the rotation strategy

For copytruncate, keep the default filestream behavior but make scans frequent enough to catch pre-truncate data:

# /etc/filebeat/filebeat.yml
filebeat.inputs:
  - type: filestream
    id: app
    paths: ["/var/log/app/app.log"]
    prospector.scanner.check_interval: 1s   # scan often so lines are read before truncation
    close.on_state_change.inactive: 5m
    ignore_older: 24h

filebeat test config -c /etc/filebeat/filebeat.yml
filebeat -e -c /etc/filebeat/filebeat.yml -d "harvester"   # observe truncation + offset reset

Example Root Cause Analysis

An app’s logs were rotated hourly with copytruncate. Under load, up to a second of log lines was written between logrotate’s cp and its truncate; Filebeat sometimes hadn’t read those final bytes before the file was zeroed, so a handful of lines were lost each hour. The warnings confirmed truncation was happening as designed, but the scan interval was too slow to keep up.

Two changes fixed it. First, they moved from copytruncate to standard create-based rotation so the old file kept its inode and was read to completion:

# /etc/logrotate.d/app
/var/log/app/app.log {
    daily
    rotate 7
    create 0640 app app
    # no copytruncate — logrotate renames, app reopens a fresh file
}

Second, they lowered check_interval to 1s. After the change, the old rotated file was harvested fully before being removed, and truncation resets disappeared.

Prevention Best Practices

• Prefer rename/create rotation (create in logrotate) over copytruncate so Filebeat reads rotated files to completion.
• If copytruncate is unavoidable, keep prospector.scanner.check_interval low so lines are read before truncation.
• Never clear live logs with > file; use proper rotation instead.
• Set ignore_older and close options so rotated files are handled predictably.
• Alert on frequent truncation warnings — they can indicate an app truncating its own log unexpectedly.

Quick Command Reference

grep -Rn 'copytruncate' /etc/logrotate.d/       # detect copytruncate rotation
logrotate -d /etc/logrotate.d/app                # dry-run rotation strategy
stat /var/log/app/app.log                        # watch size/inode over rotation
filebeat -e -d "harvester"                        # observe truncation + offset reset
# config: prospector.scanner.check_interval, close.on_state_change.inactive

Conclusion

file was truncated means Filebeat saw a file shrink below its read offset and reset to re-read from the start — normal for copytruncate rotation but a risk for line loss under load. The robust fix is to switch to rename/create rotation so rotated files keep their inode and are read to completion; where copytruncate must stay, lower the scan interval so lines are captured before truncation. Persistent truncation warnings on grow-only files are worth investigating as unintended O_TRUNC behavior.