🚨 Filebeat Error Guide: 'Harvester could not be started ... permission denied' — Fix Log File Access

Overview

Filebeat logs this when a harvester tries to open a file it has matched with a glob but the OS refuses the open() call. The event is logged per file and the file is skipped, so no lines from it are ever shipped:

Harvester could not be started on new file: /var/log/app/app.log, Err: error setting up harvester: Harvester setup failed. Unexpected file opening error: open /var/log/app/app.log: permission denied

The harvester is the component that reads a single file. When open() returns EACCES, Filebeat cannot read the file’s contents even though the path exists and matched an input path pattern. Filebeat itself keeps running and continues harvesting every other file it can read, which is why this often goes unnoticed until logs from one application silently stop arriving.

Symptoms

• Logs from a specific file or directory never appear in Elasticsearch/Logstash, while other inputs work fine.
• journalctl -u filebeat shows repeated permission denied harvester errors, one line per unreadable file.
• The path clearly exists and matches the input paths: glob when you check by hand.
• Running Filebeat manually as root works, but the systemd service (running as user filebeat or root with a restricted User=) fails.
• Newly rotated files are readable but the active log is not, or vice versa.

Common Root Causes

• The Filebeat process user lacks read on the file. The service runs as user filebeat, but the log is mode 0600 owned by appuser.
• No execute (traverse) bit on a parent directory. Reading /var/log/app/app.log requires x on /var, /var/log, and /var/log/app.
• SELinux or AppArmor denies access. On RHEL-family hosts, Filebeat runs confined and cannot read files outside var_log_t contexts.
• Restrictive umask on the log writer. The application creates logs 0600, excluding group/other reads.
• Container volume ownership mismatch. The UID inside the Filebeat container differs from the UID that owns the mounted host logs.

Diagnostic Workflow

First confirm the exact identity Filebeat runs as, then test readability as that identity.

systemctl show filebeat -p User -p Group
sudo -u filebeat cat /var/log/app/app.log | head    # reproduce the denial as the real user
namei -l /var/log/app/app.log                        # show perms on every path component

Validate the config and inputs without touching the pipeline:

# /etc/filebeat/filebeat.yml (excerpt under test)
filebeat.inputs:
  - type: filestream
    id: app-logs
    paths:
      - /var/log/app/*.log

filebeat test config -c /etc/filebeat/filebeat.yml    # YAML + schema validity
filebeat -e -c /etc/filebeat/filebeat.yml             # run in foreground, watch harvester errors live

Check SELinux denials if the file perms look correct:

sudo ausearch -m avc -ts recent | grep filebeat
getenforce
ls -Z /var/log/app/app.log

Example Root Cause Analysis

A team shipped an app that wrote /var/log/payments/txn.log with mode 0600, owner payments:payments. Filebeat ran as User=filebeat. namei -l showed the file line as -rw------- payments payments txn.log — no read bit for group or other, and filebeat was not in the payments group. Everything upstream was fine; the harvester simply never opened the file.

The fix was to add a supplementary group read grant rather than loosen the file to world-readable (payment logs are sensitive):

sudo usermod -aG payments filebeat
sudo chmod 0640 /var/log/payments/txn.log      # group can read
sudo systemctl restart filebeat

After restart, sudo -u filebeat cat succeeded, the harvester started, and the backlog flushed within seconds.

Prevention Best Practices

• Add the filebeat user to the log owner’s group and set logs to 0640 instead of making them world-readable.
• Ensure every parent directory in the log path has the x (traverse) bit for the Filebeat user or group.
• On SELinux hosts, keep logs in var_log_t-labeled directories, or add a targeted policy — never blanket-disable SELinux.
• In containers, match the container UID/GID to the host log owner, or run Filebeat with a user: that has read access to the mounted volume.
• Set the log writer’s umask so files land as 0640, avoiding the 0600 trap on new files.

Quick Command Reference

systemctl show filebeat -p User -p Group          # confirm run-as identity
namei -l /var/log/app/app.log                      # perms on each path component
sudo -u filebeat cat /var/log/app/app.log          # reproduce denial as Filebeat's user
sudo usermod -aG appgroup filebeat                  # grant group access
sudo chmod 0640 /var/log/app/app.log               # allow group read
sudo ausearch -m avc -ts recent | grep filebeat    # check SELinux denials
filebeat test config                                # validate config
filebeat -e                                          # foreground run to watch harvester errors

Conclusion

Harvester could not be started ... permission denied is almost always a plain filesystem-access problem, not a Filebeat bug. Reproduce the denial as the exact user Filebeat runs as, walk the full path with namei -l, and grant the minimum access needed — a group read bit is safer than world-readable logs. On RHEL hosts, always rule out SELinux before touching Unix permissions. Once the process user can open() the file, the harvester starts on the next scan and ships the backlog automatically.