🚨 Telegraf Error Guide: '[inputs.exec] exec: exit status 1' — Fix Failing Exec Commands

Overview

The exec input runs an external command each interval and parses its stdout into metrics. When the command exits non-zero, Telegraf logs the failure with the command and its exit code:

2026-07-10T12:00:00Z E! [inputs.exec] Error in plugin: exec: exit status 1 for command '/usr/local/bin/collect-metrics.sh': fork/exec /usr/local/bin/collect-metrics.sh: permission denied

A common companion is a non-zero exit with the command’s own stderr captured in the message:

E! [inputs.exec] Error in plugin: exec: exit status 1 for command '/opt/scripts/queue_depth.py': Traceback (most recent call last): ModuleNotFoundError: No module named 'redis'

Telegraf discards the output of a failed command, so no metrics from that exec block reach your outputs.

Symptoms

• Metrics produced by a specific script are missing while built-in inputs work fine.
• journalctl -u telegraf repeats exec: exit status 1 for command '...' every interval.
• The script runs fine in your interactive shell but fails under the telegraf service user.
• The error sometimes includes the script’s stderr (a stack trace, command not found, or permission denied).
• Gather time spikes if the command hangs before eventually failing on the exec timeout.

Common Root Causes

• Not executable / wrong permissions — the script lacks +x or is not readable by the telegraf user.
• Wrong interpreter or missing shebang — no #!/usr/bin/env python3, or a Python/binary not on the service PATH.
• Missing runtime dependency — a module, gem, or binary available in your shell but not in the service environment (ModuleNotFoundError, command not found).
• Different environment under systemd — no HOME, a minimal PATH, or missing env vars the script relies on.
• Working directory assumptions — the script uses a relative path that only resolves from your interactive cwd.
• The script genuinely returns non-zero — it hits an error condition (API 500, empty result) and exits 1 by design.
• Output not matching data_format — the command succeeds but prints non-parseable text, causing downstream parse failures often confused with exit errors.

Diagnostic Workflow

Run the exact command as the telegraf service user, not as yourself. This reproduces the systemd environment far more faithfully than your login shell:

sudo -u telegraf /usr/local/bin/collect-metrics.sh; echo "exit=$?"

Inspect the environment systemd actually gives the service, since a stripped PATH is the classic cause:

systemctl show telegraf -p ExecMainPID
sudo cat /proc/$(pgrep -x telegraf)/environ | tr '\0' '\n'

Then run only the exec input under Telegraf with debug to see stdout, stderr, and parsing together:

telegraf --config /etc/telegraf/telegraf.conf --test --input-filter exec --debug

Make the exec block explicit about interpreter, format, and timeout so failures are unambiguous:

[[inputs.exec]]
  commands = ["/usr/local/bin/collect-metrics.sh"]
  timeout = "10s"
  data_format = "influx"
  # Optionally namespace the resulting metrics
  name_override = "queue_depth"

  # Provide any env the script needs; systemd's PATH is minimal
  [inputs.exec.environment]
    PATH = "/usr/local/bin:/usr/bin:/bin"
    HOME = "/var/lib/telegraf"

The script itself must print line-protocol (for data_format = "influx") on stdout and exit 0:

#!/usr/bin/env bash
set -euo pipefail
depth=$(redis-cli -h 127.0.0.1 LLEN jobs)
echo "queue_depth,queue=jobs value=${depth}i"

Example Root Cause Analysis

A Python collector worked perfectly when the on-call engineer ran it, but Telegraf logged exec: exit status 1 ... ModuleNotFoundError: No module named 'redis'. Running sudo -u telegraf /opt/scripts/queue_depth.py reproduced the traceback exactly.

The script’s shebang was #!/usr/bin/env python3, which under the telegraf service resolved to the system Python at /usr/bin/python3 — where the redis package was not installed. The engineer had installed redis into a user virtualenv that only their shell activated. The fix was to point the shebang (and the exec command) at the interpreter that actually had the dependency, and to install the package system-wide for the service:

[[inputs.exec]]
  commands = ["/opt/venvs/telegraf/bin/python3 /opt/scripts/queue_depth.py"]
  timeout = "10s"
  data_format = "influx"

After installing redis into /opt/venvs/telegraf and referencing that interpreter, sudo -u telegraf ran the command to a clean exit 0 and Telegraf collected queue_depth. The lesson: always reproduce exec failures as the service user, because “works in my shell” hides PATH, interpreter, and dependency differences.

Prevention Best Practices

• Always test exec commands with sudo -u telegraf before deploying; your login shell is not representative.
• Give scripts an absolute interpreter path or set PATH explicitly in [inputs.exec.environment].
• Add set -euo pipefail to shell collectors so failures are loud and exit codes are honest.
• Install script dependencies into an environment the service user can reach, not a personal virtualenv.
• Set a sane timeout so a hung command fails fast instead of stalling the collection interval.
• Emit valid line protocol (or JSON matching your data_format) and validate it with telegraf --test before shipping.
• Keep exec scripts idempotent and side-effect free; they run on every interval.

Quick Command Reference

# Reproduce as the service user (most important step)
sudo -u telegraf /usr/local/bin/collect-metrics.sh; echo "exit=$?"

# Inspect the running service's environment (PATH/HOME)
sudo cat /proc/$(pgrep -x telegraf)/environ | tr '\0' '\n'

# Run only the exec input with debug
telegraf --config /etc/telegraf/telegraf.conf --test --input-filter exec --debug

# Check the script is executable and readable by telegraf
ls -l /usr/local/bin/collect-metrics.sh
sudo -u telegraf test -x /usr/local/bin/collect-metrics.sh && echo OK

# Watch exec errors live
journalctl -u telegraf -f | grep -i exec

Conclusion

[inputs.exec] exec: exit status 1 means the external command returned non-zero, most often due to permissions, a missing interpreter or dependency, or a systemd environment that differs from your shell. Reproduce with sudo -u telegraf, inspect the service PATH, and make the interpreter, data_format, and timeout explicit in the config. Once the command exits 0 and prints valid line protocol under the service user, telegraf --test will show the metrics flowing.