Logstash Error: 'Permission denied'

Overview

The Logstash file input tails files on disk and records how far it has read in a sincedb file so it can resume after a restart. That means it needs two kinds of filesystem access: it must be able to read each source log, and it must be able to write its own sincedb tracking file. When either is missing, the input logs a Permission denied and stops making progress. In /var/log/logstash/logstash-plain.log the two failures look like this:

[WARN ][filewatch.tailmode.handlers.createinitial][main] failed to open /var/log/app/app.log: Permission denied - /var/log/app/app.log
[ERROR][logstash.inputs.file ][main] Permission denied - /var/lib/logstash/plugins/inputs/file/.sincedb_a1b2c3

These are two distinct problems that happen to share a message. The first line is a read failure on the source path; the second is a write failure on the sincedb path. Logstash runs as the unprivileged logstash service user, and system logs under /var/log are frequently owned root:adm with mode 640, so a fresh install often cannot read them at all.

Symptoms

Repeated failed to open … Permission denied for a source path the input is configured to watch.
Permission denied naming a file under .../plugins/inputs/file/.sincedb_* or a custom sincedb_path.
The pipeline starts cleanly but no events from that file input ever reach the output.
ls -l on the log shows root:adm ownership and mode 640; the logstash user is in neither.
After moving a working config to a host with SELinux enforcing, the input silently reads nothing.

Common Root Causes

Source log not readable by the logstash user — files under /var/log are typically root:adm mode 640, and logstash is not in the adm group.
sincedb_path not writable — the default sincedb directory (or a custom sincedb_path) is owned by root, so the input cannot persist its read position.
Wrong ownership after a manual copy — logs or the sincedb directory were created by root during setup and never chowned back.
Restrictive directory mode — the log lives in a directory the logstash user cannot traverse (x bit missing on a parent directory).
SELinux denials — on RHEL-family hosts the logstash_t domain is denied access to files with a label it is not allowed to read/write, even when Unix permissions look correct.
Running as root to “fix” it — a tempting workaround that creates root-owned sincedb files and a security problem, without addressing the real permission model.

How to diagnose

First reproduce the failure as the service user rather than as root — this is the single most useful check, because root can read everything and hides the problem:

# Can the logstash user actually read the source log?
sudo -u logstash test -r /var/log/app/app.log && echo READABLE || echo DENIED

# What owns it, and what mode is it?
ls -l /var/log/app/app.log
# -rw-r----- 1 root adm 20480 Jul 12 09:14 /var/log/app/app.log

Confirm the logstash user’s group membership and whether it is in adm:

id logstash
groups logstash

Check that the sincedb directory is writable by the same user:

sudo -u logstash test -w /var/lib/logstash && echo WRITABLE || echo DENIED
ls -ld /var/lib/logstash

If Unix permissions look correct but reads still fail on a RHEL-family host, suspect SELinux and look for AVC denials naming logstash_t:

getenforce
sudo ausearch -m avc -ts recent | grep -i logstash
# type=AVC ... denied { read } ... scontext=system_u:system_r:logstash_t ...

For reference, a minimal file input with an explicit, service-user-owned sincedb_path looks like this. Logstash .conf files use a Ruby-like DSL, so the block is shown as ruby:

input {
  file {
    path => "/var/log/app/app.log"
    start_position => "beginning"
    sincedb_path => "/var/lib/logstash/sincedb_app"
    mode => "tail"
  }
}

Fixes

Grant read access to the source logs. For system logs under /var/log, add the service user to the adm group, then restart so the new group membership takes effect:

sudo usermod -aG adm logstash
sudo systemctl restart logstash

For a single application directory where you do not want to grant the whole adm group, use a POSIX ACL to give just the logstash user read-and-traverse access:

# Read the file, traverse (x) the directory
sudo setfacl -m u:logstash:rx /var/log/app/app.log
sudo setfacl -m u:logstash:rx /var/log/app

Make the sincedb path writable. Point sincedb_path at a directory the logstash user owns, and fix ownership on the state directory:

sudo chown -R logstash:logstash /var/lib/logstash
sudo -u logstash touch /var/lib/logstash/sincedb_app && echo "sincedb writable"

with the matching config:

input {
  file {
    path => "/var/log/app/*.log"
    sincedb_path => "/var/lib/logstash/sincedb_app"
    start_position => "beginning"
  }
}

Handle SELinux properly rather than disabling it. If ausearch shows logstash_t denials, generate a targeted policy from the denials, or relabel the target so logstash_t is allowed to read it — do not blanket-set the directory to public_content_t for secrets:

sudo ausearch -m avc -ts recent | audit2allow -M logstash-file
sudo semodule -i logstash-file.pp

What to watch out for

Do not run Logstash as root to sidestep this. It creates root-owned sincedb files that the service user cannot rewrite later, and it is a security regression.
Group changes need a restart. Adding logstash to adm only affects processes started afterward; a running service keeps its old group set until restarted.
Directory traverse matters too. A readable file inside a 700 root-owned directory is still unreachable — the logstash user needs x on every parent directory.
Log rotation can reset ACLs. If logrotate recreates files with default ownership, your setfacl grant disappears; prefer group membership or a create directive in the logrotate config that preserves access.
A wrong sincedb_path silently resets progress. If the input cannot write sincedb, it may re-read from the start on every restart — watch for duplicate events, not just the error line.

Logstash Error: 'Permission denied' — Cause, Fix, and Troubleshooting Guide

Fixing errors like this? Get 500 free DevOps AI prompts