🚨 Telegraf Error Guide: 'outputs.influxdb 401 Unauthorized' — Fix InfluxDB Write Auth

Overview

Telegraf logs this whenever the InfluxDB output plugin authenticates against the write endpoint and InfluxDB rejects the credentials. Every flush interval fails and metrics pile up in the buffer:

E! [outputs.influxdb] When writing to [http://influxdb:8086]: 401 Unauthorized: authorization failed

The v1 line-protocol variant is worded slightly differently but means the same thing:

E! [outputs.influxdb] E! Post "http://influxdb:8086/write?db=telemetry": 401 Unauthorized

A 401 is an authentication failure: InfluxDB does not accept who you claim to be. It is distinct from a 403 (authenticated but not permitted) and a 404 (endpoint or database missing), which have their own guides.

Symptoms

• Telegraf starts cleanly but no data ever lands in InfluxDB.
• The agent log repeats 401 Unauthorized once per flush_interval.
• metric buffer overflow warnings appear later as the unflushed buffer fills.
• Manual curl writes with the same credentials also return 401.
• Other Telegraf instances pointed at the same InfluxDB with different credentials write fine.

Common Root Causes

• Wrong username/password in [[outputs.influxdb]] — a typo, an unescaped special character, or a rotated password.
• Auth enabled on InfluxDB but no credentials configured in Telegraf at all (auth-enabled = true in InfluxDB 1.x).
• Using v1 username/password against InfluxDB 2.x/OSS which expects a token via the influxdb_v2 plugin.
• Environment variable not expanded — password = "$INFLUX_PASSWORD" where the variable is empty in Telegraf’s unit environment.
• A reverse proxy (nginx/Traefik) stripping or requiring its own Authorization header before the request reaches InfluxDB.
• Password contains characters that break TOML parsing when unquoted.

Diagnostic Workflow

First, confirm which plugin and endpoint Telegraf is actually using:

telegraf --config /etc/telegraf/telegraf.conf config | grep -A15 'outputs.influxdb'

Inspect the output block. For InfluxDB 1.x with auth enabled you need both fields:

[[outputs.influxdb]]
  urls = ["http://influxdb:8086"]
  database = "telemetry"
  username = "telegraf"
  password = "${INFLUX_PASSWORD}"

Verify the credentials actually reach the process — a common cause is an empty env var in the systemd unit:

sudo systemctl show telegraf -p Environment
sudo cat /etc/default/telegraf | grep INFLUX

Reproduce the write outside Telegraf with the same credentials so you can isolate the plugin from the network:

curl -i -XPOST "http://influxdb:8086/write?db=telemetry" \
  -u telegraf:"$INFLUX_PASSWORD" \
  --data-binary 'probe,host=diag value=1'

A 204 No Content means the credentials are good and the problem is in Telegraf’s config; a 401 confirms the credentials themselves are wrong. Check the InfluxDB user exists and can write:

influx -username admin -password "$ADMIN_PW" -execute 'SHOW USERS'
influx -username admin -password "$ADMIN_PW" -execute 'SHOW GRANTS FOR telegraf'

Example Root Cause Analysis

A team migrated InfluxDB behind an nginx TLS terminator and rotated the Telegraf password in the InfluxDB SET PASSWORD step, but the new value only landed in a Vault secret. The systemd unit still sourced /etc/default/telegraf, which held the old password. curl with the Vault value returned 204; curl with the file value returned 401 — proving Telegraf was reading a stale secret. The fix was to render /etc/default/telegraf from Vault on deploy and add systemctl restart telegraf. Data resumed within one flush interval, and the backed-up buffer drained on the next flush because metric_buffer_limit had held the intervening points.

Prevention Best Practices

• Store the password in /etc/default/telegraf or a systemd EnvironmentFile and reference it as ${INFLUX_PASSWORD}; never inline secrets in telegraf.conf.
• Wire credential rotation to also update Telegraf’s environment file and restart the service in the same automation step.
• Give the Telegraf InfluxDB user a write-only grant on exactly one database, so a leaked credential has minimal blast radius.
• Alert on a sustained nonzero internal_write error count so a 401 is caught in minutes, not when a dashboard goes flat.
• Use the correct plugin for the InfluxDB major version — influxdb for 1.x, influxdb_v2 for 2.x/Cloud.

Quick Command Reference

# Show the effective output config Telegraf parsed
telegraf --config /etc/telegraf/telegraf.conf config | grep -A15 outputs.influxdb

# Confirm the password env var reaches the service
sudo systemctl show telegraf -p Environment

# Reproduce the authenticated write
curl -i -XPOST "http://influxdb:8086/write?db=telemetry" \
  -u telegraf:"$INFLUX_PASSWORD" --data-binary 'probe value=1'

# Verify the InfluxDB user and grants
influx -username admin -password "$ADMIN_PW" -execute 'SHOW GRANTS FOR telegraf'

# Watch for the error live
journalctl -u telegraf -f | grep -i 401

Conclusion

A Telegraf outputs.influxdb 401 Unauthorized is almost always a credential mismatch between what Telegraf sends and what InfluxDB expects — a stale password, an empty environment variable, or the wrong plugin for the InfluxDB version. Reproduce the write with curl first: if curl also fails the fix is in InfluxDB’s users and grants, and if curl succeeds the fix is in Telegraf’s config or environment. For the closely related “authenticated but forbidden” case see a 403 guide, and for missing databases see the database not found guide. More fixes in the Telegraf guides.