🚨 Prometheus Error Guide: 'node_exporter permission denied collecting' Collector Failure

Exact Error Message

This is an exporter-side error, not a Prometheus-core error. It appears in the node_exporter log when an individual collector cannot read a sysfs/procfs path:

level=error ts=2026-06-27T09:12:44.501Z caller=collector.go:173 msg="collector failed" name=rapl duration_seconds=0.00 err="open /sys/class/powercap/intel-rapl:0/energy_uj: permission denied"

The textfile collector produces a very similar line when its directory is unreadable:

level=error ts=2026-06-27T09:12:44.612Z caller=collector.go:173 msg="collector failed" name=textfile duration_seconds=0.00 err="failed to read textfile collector directory \"/var/lib/node_exporter/textfile\": open /var/lib/node_exporter/textfile: permission denied"

The critical detail: the scrape still succeeds. Prometheus gets an HTTP 200 and most metrics arrive. The failed collector just contributes nothing, and you see a 0 for its success metric:

node_scrape_collector_success{collector="rapl"} 0
node_scrape_collector_success{collector="textfile"} 0
node_scrape_collector_success{collector="cpu"} 1

Because the endpoint is healthy, up stays 1 and no target alert fires — the data simply goes missing.

What the Error Means

node_exporter is, by good practice, run as an unprivileged user rather than root. Most collectors read world-readable files under /proc and /sys and work fine that way. But a handful of collectors need elevated access:

• rapl and powercap read /sys/class/powercap/intel-rapl:*/energy_uj, which is root-readable only on recent kernels (a fix for a power side-channel).
• hwmon reads some sensors that are mode 0600.
• textfile reads whatever you drop in its directory — if those files are owned by root and 0600, the exporter user cannot open them.
• systemd talks to the systemd D-Bus, which needs the socket to be reachable.
• perf uses perf_event_open, gated by CAP_PERFMON/perf_event_paranoid.

When such a collector hits permission denied, node_exporter does not crash. It logs collector failed, sets node_scrape_collector_success{collector="..."} 0, and continues serving every other collector. The failure is therefore silent unless you are watching that metric.

Common Causes

• A hardened systemd unit hides the paths. ProtectSystem, PrivateDevices=true, ProtectKernelTunables=true, or ProtectControlGroups=true make parts of /sys, /proc, and /dev invisible or read-only to the service. The collector then sees permission denied (or the path is simply gone).
• Textfile directory owned by root. A cron job writes *.prom files as root with mode 0600, and the node_exporter user cannot read the directory or its files.
• Container missing bind mounts. Running node_exporter in a container without --path.procfs, --path.sysfs, and host /proc, /sys, /run mounted, or running as a user that cannot read what is mounted.
• SELinux or AppArmor denial. The policy blocks the exporter from reading a path even though Unix permissions would allow it. You will see an AVC denial rather than a Unix EACCES source.
• Privileged collectors without privilege. systemd needs D-Bus access; rapl/perf need root or a capability such as CAP_PERFMON/CAP_SYS_RAWIO.

How to Reproduce the Error

Run node_exporter as an unprivileged user and point the textfile collector at a root-only directory:

sudo mkdir -p /var/lib/node_exporter/textfile
echo 'my_metric 1' | sudo tee /var/lib/node_exporter/textfile/test.prom >/dev/null
sudo chown -R root:root /var/lib/node_exporter/textfile
sudo chmod 700 /var/lib/node_exporter/textfile

sudo -u node_exporter /usr/local/bin/node_exporter \
  --collector.textfile.directory=/var/lib/node_exporter/textfile

The log immediately shows the textfile permission denied, and a scrape confirms the collector reports 0:

curl -s http://localhost:9100/metrics | grep 'collector="textfile"'

node_scrape_collector_success{collector="textfile"} 0

The rapl case reproduces on a kernel where energy_uj is root-only: enable --collector.rapl as the unprivileged user and watch for the open /sys/class/powercap/...: permission denied line.

Diagnostic Commands

All of these are read-only. Start by listing every collector that is currently failing:

curl -s http://localhost:9100/metrics | grep node_scrape_collector_success | grep ' 0$'

See how long each collector is taking (a failing one usually shows ~0s):

curl -s http://localhost:9100/metrics | grep node_scrape_collector_duration

Inspect the running unit — this reveals the hardening keys that hide paths:

systemctl cat node_exporter

Check the textfile directory ownership and the user the exporter runs as:

ls -la /var/lib/node_exporter/textfile
id node_exporter

Reproduce the exact denied read as the exporter user (read-only test of the path):

sudo -u node_exporter ls -la /sys/class/powercap/intel-rapl:0/energy_uj
sudo -u node_exporter cat /sys/class/powercap/intel-rapl:0/energy_uj

Pull the permission-denied lines straight from the journal:

journalctl -u node_exporter -n 60 --no-pager | grep -i 'permission denied'

If Unix permissions look correct but access is still denied, suspect SELinux:

sudo ausearch -m avc -ts recent

Step-by-Step Resolution

1. Identify the failing collector. Use the node_scrape_collector_success == 0 query below — it names exactly which collector is broken so you are not guessing:

node_scrape_collector_success == 0

2. Decide: keep it or disable it. If you genuinely do not need the data (rapl power telemetry on a VM, for example), disabling the collector is the correct, lowest-risk fix:

# /etc/systemd/system/node_exporter.service drop-in
[Service]
ExecStart=
ExecStart=/usr/local/bin/node_exporter --no-collector.rapl

3. If you need the data, relax the systemd sandbox. Hardened units commonly hide the very paths these collectors read. Loosen only the keys that block you, and prefer scoped read-only access over turning protection fully off:

[Service]
User=node_exporter
Group=node_exporter
# These three commonly cause sysfs/procfs/rapl denials:
ProtectKernelTunables=false
PrivateDevices=false
# Keep the rest of the FS read-only, but allow the paths you need:
ProtectSystem=strict
ReadOnlyPaths=/sys /proc
ReadWritePaths=/var/lib/node_exporter/textfile
# For rapl/perf, grant a capability instead of running as root:
AmbientCapabilities=CAP_PERFMON

Then reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart node_exporter

4. Fix textfile ownership instead of widening the sandbox. The textfile fix is almost always ownership, not sandboxing — chown the directory and files to the exporter user:

sudo chown -R node_exporter:node_exporter /var/lib/node_exporter/textfile
sudo chmod 755 /var/lib/node_exporter/textfile
sudo chmod 644 /var/lib/node_exporter/textfile/*.prom

Make the cron job that writes the files do so as node_exporter, or chown them after writing.

5. For containers, add the bind mounts. Mount the host paths and point the exporter at them:

services:
  node-exporter:
    image: prom/node-exporter:latest
    pid: host
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--path.rootfs=/host/root'
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/host/root:ro,rslave

6. For SELinux/AppArmor, fix the policy — do not disable enforcement. Resist setenforce 0. Read the AVC with ausearch, then generate and install a targeted policy module (audit2allow -M) or correct the file context with restorecon/chcon.

Prevention and Best Practices

• Alert on node_scrape_collector_success == 0. Because the scrape stays healthy, this is the only signal that a collector has silently gone dark. Add a rule per node.
• Disable collectors you do not need. Fewer enabled collectors means fewer privileged paths to manage and a smaller attack surface.
• Own the textfile directory at provisioning time. Bake node_exporter:node_exporter ownership and 0755/0644 modes into your configuration management so a root-owned cron file never reintroduces the error.
• Keep the systemd sandbox, but scope it. Prefer ReadOnlyPaths/ReadWritePaths and AmbientCapabilities over blanket false so you retain most of the hardening.
• Test as the service user. After any change, run sudo -u node_exporter against the contested path to confirm access before relying on a scrape.

Related Errors

• target down / up == 0 — the exporter is unreachable entirely, not just one collector failing. See target down (up == 0). With a collector permission error, up stays 1.
• connection refused — Prometheus cannot open a TCP connection to :9100 at all. See scrape connection refused. Different layer: that is the listener, this is a per-collector filesystem read.

Frequently Asked Questions

Why does the scrape succeed but I’m missing metrics? node_exporter serves every collector that worked and skips the one that failed. The HTTP response is 200, up is 1, and only the failed collector’s series are absent — with node_scrape_collector_success{collector="..."} 0 flagging which one.

Should I just run node_exporter as root? No. Running as root removes the whole point of the unprivileged design. Disable the collector you do not need, grant a specific capability (CAP_PERFMON for perf/rapl), or fix file ownership. Root is a last resort.

My textfile metrics disappeared after a cron run. Why? The cron job almost certainly wrote the .prom files as root with restrictive modes. chown them back to node_exporter and have the job write as that user so the directory stays readable.

The Unix permissions look fine but it’s still denied — what now? Suspect SELinux or AppArmor, or a systemd sandbox key. Run ausearch -m avc -ts recent for an AVC denial and check systemctl cat node_exporter for ProtectKernelTunables, PrivateDevices, or ProtectSystem hiding the path.