🚨 Telegraf Error Guide: '[outputs.datadog] received bad status code, 403' — Fix Datadog API Auth

Overview

The datadog output ships metrics to Datadog’s HTTP intake API using an API key. When Datadog rejects the key, the plugin logs the HTTP status it received:

2026-07-12T12:00:00Z E! [outputs.datadog] Error in plugin: received bad status code, 403

A closely related form shows the write being retried and the metric buffer filling because nothing is being accepted:

2026-07-12T12:00:00Z W! [agent] Metric buffer overflow; 1000 metrics have been dropped

Telegraf keeps collecting, but no metrics reach Datadog until the API key and site are corrected.

Symptoms

• No metrics appear in the Datadog Metrics Explorer while the local Telegraf agent looks healthy.
• journalctl -u telegraf repeats [outputs.datadog] received bad status code, 403 on each flush.
• The metric buffer grows and eventually logs overflow warnings because writes never succeed.
• A key that works for one region fails after the account was created in the EU (or US5/AP1) site.
• The error started after a key rotation, or after moving the config to a host where ${DD_API_KEY} is not set.

Common Root Causes

• Invalid or revoked API key — the apikey is mistyped, truncated, or was deleted in the Datadog console.
• Missing environment variable — the config uses ${DD_API_KEY} but that variable is not exported to the Telegraf service, so the key resolves to an empty string.
• Wrong Datadog site/URL — the account lives in the EU (datadoghq.eu) or another region, but the output points at the default US intake (datadoghq.com), so the key is not recognized.
• Using an APP key instead of an API key — application keys and API keys are different; the intake API only accepts API keys.
• Proxy or egress firewall — an outbound proxy strips or blocks the request, returning a 403 from an intermediary rather than Datadog.
• Org/permissions issue — the key belongs to a different organization than the dashboards you are checking.

Diagnostic Workflow

First verify the key directly against the Datadog validate endpoint. A {"valid":true} response confirms the key and site are correct; a 403 confirms the key is the problem:

# US site (default)
curl -s "https://api.datadoghq.com/api/v1/validate" \
  -H "DD-API-KEY: ${DD_API_KEY}"

# EU site
curl -s "https://api.datadoghq.eu/api/v1/validate" \
  -H "DD-API-KEY: ${DD_API_KEY}"

Confirm the variable is actually visible to the Telegraf service, not just your shell:

sudo systemctl show telegraf -p Environment
# or, if injected via an env file:
sudo cat /etc/default/telegraf | grep DD_API_KEY

Then run only the datadog output under Telegraf to watch the flush after a fix:

telegraf --config /etc/telegraf/telegraf.conf --test --debug

A correct output block references the key from the environment and sets the URL to match your account’s site:

[[outputs.datadog]]
  apikey = "${DD_API_KEY}"
  # US1 (default) — omit url or set the US intake:
  url = "https://app.datadoghq.com/api/v1/series"

For an EU-region account, point url at the EU intake instead:

[[outputs.datadog]]
  apikey = "${DD_API_KEY}"
  url = "https://app.datadoghq.eu/api/v1/series"
  http_proxy_url = "http://10.0.0.9:3128"   # only if egress requires a proxy

Inject the key into the service through a systemd drop-in or env file so ${DD_API_KEY} resolves:

sudo systemctl edit telegraf
# [Service]
# Environment="DD_API_KEY=xxxxxxxxxxxxxxxx"
sudo systemctl restart telegraf

Example Root Cause Analysis

A team migrated Telegraf to a new fleet and metrics stopped landing in Datadog, with [outputs.datadog] received bad status code, 403 on every flush. The API key was valid — it worked from a laptop with curl against api.datadoghq.com. The difference was the account: this org had been provisioned in the EU region.

Running the validate call against api.datadoghq.eu returned {"valid":true}, while the US endpoint returned 403 for the same key. The Telegraf output still used the default US intake, so Datadog’s US site rejected an EU key. Setting url = "https://app.datadoghq.eu/api/v1/series" fixed collection instantly. The lesson: a Datadog 403 is an authentication failure, and when the key itself validates, the mismatch is almost always the site/region — validate the key against the exact site URL before touching the key.

Prevention Best Practices

• Store the API key only in ${DD_API_KEY} via a systemd drop-in or env file; never commit it inline to version control.
• Pin the url explicitly to your account’s Datadog site so a default-US assumption cannot silently break EU/US5/AP1 orgs.
• Use API keys (not application keys) for the intake, and label keys per host/fleet so rotation is traceable.
• Validate new keys with the /api/v1/validate endpoint against the correct site as part of onboarding.
• Alert on outputs.datadog write errors and metric buffer overflow so a 403 is caught before data gaps widen.
• When behind a proxy, set http_proxy_url in the plugin and confirm the proxy allows the Datadog intake host.

Quick Command Reference

# Validate the key against the correct site
curl -s "https://api.datadoghq.com/api/v1/validate" -H "DD-API-KEY: ${DD_API_KEY}"
curl -s "https://api.datadoghq.eu/api/v1/validate"  -H "DD-API-KEY: ${DD_API_KEY}"

# Confirm the service can see the env var
sudo systemctl show telegraf -p Environment

# Test flush with debug
telegraf --config /etc/telegraf/telegraf.conf --test --debug

# Watch Datadog output errors live
journalctl -u telegraf -f | grep -i datadog

Related Guides

• Telegraf [outputs.influxdb] 401 unauthorized
• Telegraf write 413 request entity too large
• Telegraf metric buffer overflow

More fixes in the Telegraf guides.

Conclusion

[outputs.datadog] received bad status code, 403 is an authentication rejection — the API key is invalid, empty (an unset ${DD_API_KEY}), or being sent to the wrong Datadog site. Validate the key with the /api/v1/validate endpoint against the exact region URL, confirm the env var is visible to the service, and pin the plugin url to your account’s site. Get the key and site aligned, and metrics flow to Datadog again without buffer overflow.