🚨 Filebeat Error: 'State for ... not removed, file still open' — Cause, Fix, and Troubleshooting Guide

Overview

Filebeat keeps a state entry per file in its registry and periodically tries to clean up entries for files that have been deleted or moved. If it wants to remove a state but the harvester still holds the file open, it declines and logs:

State for /var/log/app/app.log.1 should be removed, but harvester not finished resp. state not removed: file still open

This is a housekeeping message, not a fatal error, but a steady stream of it points at a real problem: harvesters are not closing after files rotate or are deleted, so state accumulates and descriptors leak. Under clean_removed/clean_inactive, Filebeat wants to drop the state, but it will not while the file is actively open. Left unchecked this grows the registry and can lead to too many open files.

Symptoms

• Repeated state not removed: file still open for rotated files (e.g. *.log.1).
• Registry file grows; /proc/<pid>/fd count climbs over time.
• Deleted or rotated files keep descriptors held (visible as (deleted) in /proc/<pid>/fd).
• Eventually correlates with too many open files under heavy rotation.

Common Root Causes

• close.* timers too long — harvesters keep files open long after writes stop.
• close_removed disabled — Filebeat keeps reading deleted files.
• Aggressive rotation — files rotate faster than harvesters close.
• clean_inactive shorter than ignore_older — a misconfiguration that fights itself.
• Slow output — harvesters stay open waiting to finish sending buffered lines.

How to diagnose

Watch the housekeeping messages and correlate with descriptor growth:

journalctl -u filebeat -f | grep -i 'state.*not removed\|file still open'
ls /proc/$(pgrep -x filebeat)/fd | wc -l

Look for descriptors held on deleted files:

sudo ls -l /proc/$(pgrep -x filebeat)/fd | grep -i deleted | head

Fixes

Close harvesters promptly after rotation and when files are removed:

filebeat.inputs:
  - type: filestream
    id: app
    paths:
      - /var/log/app/*.log
    close.on_state_change.inactive: 5m
    close.on_state_change.renamed: true
    close.on_state_change.removed: true
    close.reader.after_interval: 5m

Keep the clean/ignore relationship correct — clean_inactive must be larger than ignore_older + scan_frequency:

    clean_inactive: 72h
    ignore_older: 48h

Restart Filebeat and confirm the state-not-removed messages subside and the descriptor count stabilizes.

What to watch out for

• The message alone is harmless; a sustained stream under rotation is the warning sign.
• clean_inactive must exceed ignore_older or Filebeat can drop state for files it still reads.
• Closing on removed/renamed is what actually releases descriptors after rotation.
• If the output is slow, harvesters stay open to flush — fix output health too.

Related

• Filebeat Error: ‘too many open files’
• Filebeat Error Guide: ‘harvester … file was truncated’
• Filebeat Error Guide: ‘registry file is corrupt’