🚨 OpenTelemetry Error Guide: 'permission denied' in the filelog receiver — Fix Log Access

Overview

This error appears when the OpenTelemetry Collector’s filelog receiver (or its checkpoint file_storage extension) cannot open a path the OS won’t let it read. It shows up in the Collector logs:

warn	fileconsumer/file.go:XXX	Failed to open file	{"kind": "receiver", "name": "filelog", "component": "fileconsumer", "error": "open /var/log/pods/app_web_9f/web/0.log: permission denied"}

The storage-extension variant, seen at startup, reads:

error	service/service.go:XXX	failed to build extensions: failed to create extension "file_storage": mkdir /var/lib/otelcol/storage: permission denied

permission denied (EACCES) means the path exists but the Collector process — running as a specific UID inside its container — lacks read (or write, for storage) access. No log lines are ingested from the affected files.

Symptoms

• The filelog receiver starts but ingests nothing from some or all paths.
• Repeated Failed to open file ... permission denied warnings for /var/log/pods/... or /var/log/containers/....
• The Collector CrashLoopBackOffs at startup when it’s the file_storage extension directory that is unreadable.
• Only some files fail — often those owned by root or another UID with restrictive modes.
• Works on one node/OS but not another due to differing log file ownership.

Common Root Causes

• Non-root container UID — the Collector runs as an unprivileged UID (e.g. 10001) but the host log files are 0600 root:root.
• hostPath not mounted or read-only where write is needed — /var/log (and the symlink targets it points to) isn’t fully mounted into the container.
• Symlink target not mounted — /var/log/containers/*.log symlinks into /var/log/pods/...; mounting only one path leaves the real files unreadable.
• Storage directory not writable — the file_storage checkpoint directory is on a volume the Collector UID can’t write.
• SELinux/AppArmor — the host security module denies the container access to /var/log despite POSIX permissions.
• fsGroup / securityContext mismatch — the pod’s security context doesn’t grant access to the mounted paths.

How to diagnose

Find out what UID the Collector actually runs as, then test whether that UID can read the file:

kubectl get pod -l app=otel-collector -o jsonpath='{.items[0].spec.securityContext}'
kubectl exec ds/otel-collector -- id
kubectl exec ds/otel-collector -- ls -l /var/log/pods/app_web_9f/web/0.log
kubectl exec ds/otel-collector -- head -c1 /var/log/pods/app_web_9f/web/0.log && echo READABLE

Confirm the host paths — and the symlink targets — are actually mounted into the container:

kubectl get ds otel-collector -o yaml | grep -A3 -E 'hostPath|mountPath'
kubectl exec ds/otel-collector -- ls -ld /var/log /var/log/pods /var/log/containers

If it’s the storage extension, check the directory ownership and mode:

kubectl exec ds/otel-collector -- ls -ld /var/lib/otelcol/storage

On SELinux hosts, an avc: denied in the node’s audit log confirms a policy denial rather than a POSIX one:

# on the node
journalctl -k | grep -i 'avc.*denied' | tail

Fixes

Mount both the container log directory and its symlink target, and run the Collector with enough privilege to read root-owned log files. A typical DaemonSet snippet:

spec:
  template:
    spec:
      securityContext:
        runAsUser: 0          # read root-owned /var/log/pods files
        runAsGroup: 0
      containers:
        - name: otel-collector
          volumeMounts:
            - name: varlogpods
              mountPath: /var/log/pods
              readOnly: true
            - name: varlogcontainers
              mountPath: /var/log/containers
              readOnly: true
            - name: storage
              mountPath: /var/lib/otelcol/storage
      volumes:
        - name: varlogpods
          hostPath: { path: /var/log/pods }
        - name: varlogcontainers
          hostPath: { path: /var/log/containers }
        - name: storage
          hostPath: { path: /var/lib/otelcol/storage, type: DirectoryOrCreate }

Point the filelog receiver at the real files and the file_storage extension at a writable directory:

extensions:
  file_storage:
    directory: /var/lib/otelcol/storage

receivers:
  filelog:
    include: [ /var/log/pods/*/*/*.log ]
    include_file_path: true
    storage: file_storage        # checkpoint offsets survive restarts
    operators:
      - type: container           # parse CRI/containerd log format

service:
  extensions: [file_storage]
  pipelines:
    logs:
      receivers: [filelog]
      processors: [batch]
      exporters: [otlp]

If you cannot run as root, instead grant the Collector’s group access to the log files at the host level (or use fsGroup), but read-only root is usually simplest for node log collection. For SELinux hosts, add the appropriate context (e.g. label the volume or use a policy that permits container access to /var/log) rather than disabling enforcement.

What to watch out for

• Mounting /var/log/containers without /var/log/pods fails silently — the visible files are symlinks whose real targets you didn’t mount. Mount both.
• runAsUser: 0 reads root logs but widens the Collector’s blast radius; keep the mounts readOnly: true and scope the config tightly.
• The file_storage directory must be writable; a read-only mount there causes the mkdir/startup failure, not the receiver warning.
• Offsets are stored per path — if you change the storage directory or lose the volume, the receiver may re-read files from the start and duplicate logs.
• On SELinux/AppArmor nodes, POSIX permissions can look correct while the LSM still denies access; check the node’s kernel audit log.

Related

• OpenTelemetry Error Guide: ‘failed to parse config’
• OpenTelemetry Error Guide: ‘address already in use’
• OpenTelemetry Error Guide: ‘resource attribute service.name not set’