🚨 Filebeat Error: 'Harvester crawler could not be started' — Cause, Fix, and Troubleshooting Guide

Overview

A harvester is the goroutine that opens and reads a single file. When Filebeat matches a file to an input but cannot open or begin reading it, it logs that the harvester could not be started, with the OS-level reason attached:

Harvester could not be started on new file: /var/log/app/app.log, Err: error setting up harvester: Harvester setup failed. Unexpected file opening error: Failed opening /var/log/app/app.log: open /var/log/app/app.log: too many open files

The file is discovered correctly — the failure is at the open() syscall or the initial read. The nested Err: names the real cause: an exhausted file-descriptor limit, a symlink loop, an unreadable/encoding-mismatched file, or a device error. Only that one file is skipped; other harvesters keep running, so ingestion is partial rather than fully down.

Symptoms

• Harvester could not be started on new file: <path>, Err: ... for specific files.
• Some files ship while others silently do not.
• The nested Err: mentions too many open files, no such file, or an encoding/read error.
• The count of failures grows as more files match a broad glob.

Common Root Causes

• File-descriptor exhaustion — more matched files than the process nofile limit allows.
• Symlink loop or dangling symlink in the harvested path.
• Encoding mismatch — a declared encoding that cannot decode the file’s bytes.
• Transient file — the file rotated or was deleted between discovery and open.
• Permissions / SELinux blocking the open (a distinct permission denied variant).

How to diagnose

Run with harvester debug logging to see each open attempt and its error:

filebeat -e -d "harvester,input"

Check the open-file limit and current usage for the Filebeat process:

cat /proc/$(pgrep -x filebeat)/limits | grep 'open files'
ls /proc/$(pgrep -x filebeat)/fd | wc -l

Count how many files your globs actually match — a runaway glob is a common cause:

ls -1 /var/log/app/**/*.log 2>/dev/null | wc -l

Fixes

Raise the file-descriptor limit for the service and cap concurrent harvesters:

# /etc/systemd/system/filebeat.service.d/override.conf
[Service]
LimitNOFILE=65536

systemctl daemon-reload && systemctl restart filebeat

Bound how many files stay open at once and close idle ones sooner:

filebeat.inputs:
  - type: filestream
    id: app
    paths:
      - /var/log/app/*.log
    harvester_limit: 256
    close.on_state_change.inactive: 5m
    prospector.scanner.symlinks: false   # avoid symlink loops

Narrow overly-broad globs so Filebeat is not trying to open thousands of files, then restart and confirm harvesters start cleanly.

What to watch out for

• harvester_limit bounds concurrency but files still need descriptors — size LimitNOFILE accordingly.
• Broad ** globs are the usual reason limits blow up; scope paths tightly.
• A dangling symlink reports as a harvester start failure, not a missing-file error.
• A permission denied variant needs an ACL/SELinux fix, not a limit bump.

Related

• Filebeat Error Guide: ‘Harvester could not be started … permission denied’
• Filebeat Error: ‘too many open files’
• Filebeat Error Guide: ‘harvester … file was truncated’