🚨 Telegraf Error Guide: '[outputs.cloudwatch] NoCredentialProviders' — Fix AWS Credentials

Overview

The cloudwatch output resolves AWS credentials through the standard provider chain (environment variables, shared credentials file, and the EC2/ECS instance role). When none of those sources yields usable credentials, the AWS SDK returns a chain error and Telegraf logs:

2026-07-12T12:00:00Z E! [outputs.cloudwatch] Error in plugin: NoCredentialProviders: no valid providers in chain. Deprecated.

A closely related failure appears when credentials are found but the region is not set, so the SDK cannot build an endpoint:

E! [outputs.cloudwatch] Error in plugin: MissingRegion: could not find region configuration

Until credentials and a region resolve, no metrics reach CloudWatch and the output backs up Telegraf’s metric buffer.

Symptoms

• No custom metrics appear in the target CloudWatch namespace while Telegraf keeps collecting.
• journalctl -u telegraf repeats NoCredentialProviders: no valid providers in chain on every flush.
• The output works when run interactively (your shell has AWS_* env vars) but fails under systemd.
• The instance has no IAM role attached, or IMDS is blocked so the role’s credentials cannot be fetched.
• After credentials resolve, writes fail instead with AccessDenied for cloudwatch:PutMetricData.

Common Root Causes

• No IAM role / instance profile — the EC2 instance (or ECS task) has no role, so the instance-metadata provider returns nothing.
• IMDS blocked or IMDSv2 required — a hop-limit of 1, disabled IMDS, or a firewall to 169.254.169.254 prevents the SDK from reading role credentials.
• No env or shared-file fallback — AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY are unset and ~/.aws/credentials does not exist for the telegraf user.
• Region not configured — the region option is empty and AWS_REGION is unset, so the SDK cannot pick an endpoint.
• Missing IAM permission — credentials resolve but the role lacks cloudwatch:PutMetricData, surfacing as AccessDenied.
• Env vars set only in your shell — the systemd service does not inherit interactive-shell exports, so the chain is empty under the service.

Diagnostic Workflow

First confirm which credential source should apply. On EC2, check that a role is attached and IMDS is reachable from the host:

TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 60")
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/iam/security-credentials/

Verify the resolved identity as the service user (empty output means the chain is empty for that user):

sudo -u telegraf env AWS_REGION=us-east-1 aws sts get-caller-identity

Confirm what the systemd service actually sees:

sudo systemctl show telegraf -p Environment

Run only the outputs under test once credentials resolve:

telegraf --config /etc/telegraf/telegraf.conf --test --debug

The cloudwatch output needs a namespace and a region; on EC2 with an instance role, no keys are needed:

[[outputs.cloudwatch]]
  region = "us-east-1"
  namespace = "Telegraf"
  high_resolution_metrics = false

Off EC2 (or with a dedicated user), point the output at a named profile or supply keys via the SDK’s standard sources:

[[outputs.cloudwatch]]
  region = "us-east-1"
  namespace = "Telegraf"
  # Uses ~/.aws/credentials profile of the telegraf user:
  profile = "telegraf"
  # shared_credential_file = "/etc/telegraf/.aws/credentials"

Attach an IAM policy granting exactly the write action the plugin needs:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": "cloudwatch:PutMetricData", "Resource": "*" }
  ]
}

If IMDSv2 is enforced, raise the metadata hop limit so a containerized/agent process can reach it:

aws ec2 modify-instance-metadata-options \
  --instance-id i-0abc123 --http-tokens required --http-put-response-hop-limit 2

Example Root Cause Analysis

An SRE moved a Telegraf agent from a bare EC2 host into a container on the same instance and CloudWatch metrics stopped, logging NoCredentialProviders: no valid providers in chain. On the host, aws sts get-caller-identity worked fine — the instance role was attached and valid — so credentials clearly existed somewhere.

The container, however, could not reach IMDS: the instance had IMDSv2 enforced with --http-put-response-hop-limit 1, and the extra network hop into the container’s bridge exceeded that limit, so the SDK’s instance-metadata provider timed out and the chain came up empty. Raising the hop limit to 2 with modify-instance-metadata-options let the container fetch the role’s credentials, and metrics resumed. The lesson: NoCredentialProviders is about the credential chain being empty for that specific process — a working aws CLI on the host does not prove the agent’s process can reach the same source.

Prevention Best Practices

• Prefer an EC2 instance role or ECS task role over long-lived keys so credentials rotate automatically.
• Always set region explicitly in the output; relying on an unset AWS_REGION invites MissingRegion failures.
• When enforcing IMDSv2, set the hop limit to 2 for containerized agents so they can still reach the metadata endpoint.
• Grant least privilege — cloudwatch:PutMetricData is the only action the output needs.
• If you must use a shared credentials file, own it by the telegraf user and reference it explicitly, not the operator’s ~/.aws.
• Confirm the service identity with sudo -u telegraf aws sts get-caller-identity, not just your interactive shell.

Quick Command Reference

# Check the instance role and IMDS reachability (IMDSv2)
TOKEN=$(curl -s -X PUT http://169.254.169.254/latest/api/token \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 60")
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/iam/security-credentials/

# Resolve identity as the service user
sudo -u telegraf env AWS_REGION=us-east-1 aws sts get-caller-identity

# See what the service environment holds
sudo systemctl show telegraf -p Environment

# Run the outputs under test
telegraf --config /etc/telegraf/telegraf.conf --test --debug

# Raise IMDS hop limit for containerized agents
aws ec2 modify-instance-metadata-options \
  --instance-id i-0abc123 --http-tokens required --http-put-response-hop-limit 2

Related Guides

• Telegraf HTTP connection refused
• Telegraf output context deadline exceeded
• Telegraf metric buffer overflow

More fixes in the Telegraf guides.

Conclusion

[outputs.cloudwatch] NoCredentialProviders: no valid providers in chain means the AWS SDK found no usable credentials for the Telegraf process — no instance role, no environment keys, no shared file, or an IMDS endpoint it cannot reach. Attach an IAM role with cloudwatch:PutMetricData, set region explicitly, and confirm the identity with sudo -u telegraf aws sts get-caller-identity. When IMDSv2 is enforced for containerized agents, raise the metadata hop limit, and metrics will flow into CloudWatch.