🚨 Filebeat Error Guide: '401 Unauthorized' — Restore Elasticsearch Output Auth

Overview

Filebeat’s Elasticsearch output authenticates every bulk request. When the credentials are missing, wrong, or expired, Elasticsearch returns 401 and Filebeat backs off and retries:

Failed to connect to backoff(elasticsearch(https://es01:9200)): 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [filebeat_writer] for REST request [/_bulk]"}]}}

A 401 is authentication failure — Elasticsearch does not recognize the identity presented. This is distinct from 403 (authenticated but not authorized). The username/password or API key in output.elasticsearch is wrong, empty, or the user was disabled or rotated. The TCP connection and TLS handshake succeed; the request is rejected at the security layer. Filebeat keeps events queued and retries, so nothing is lost, but ingestion halts until valid credentials are supplied.

Symptoms

• 401 Unauthorized with security_exception ... unable to authenticate user in the Filebeat log.
• filebeat test output connects and negotiates TLS but fails at the authentication step.
• No documents index; the queue grows.
• The error begins right after a password rotation, API-key expiry, or a secrets deploy.
• A direct curl with the same credentials also returns 401, confirming it is the credentials, not Filebeat.

Common Root Causes

• Wrong password — a rotated Elasticsearch password not propagated to the Filebeat config or keystore.
• Missing credentials — output.elasticsearch.username/password unset on a secured cluster.
• Expired or revoked API key — output.elasticsearch.api_key past its expiry or invalidated.
• Disabled user — the service account was disabled or deleted in Elasticsearch.
• Keystore mismatch — the config references ${ES_PASSWORD} but the Filebeat keystore entry is stale or absent.
• Wrong realm — the user exists in a different realm than the cluster expects.

Diagnostic Workflow

Run the output test to confirm the failure is at authentication, not connection or TLS:

filebeat test output

Verify the credentials Filebeat uses actually authenticate:

curl -sk 'https://es01:9200/_security/_authenticate?pretty' -u filebeat_writer:$FB_PASS

Check whether the config reads from the keystore and that the key exists:

grep -A6 'output.elasticsearch' /etc/filebeat/filebeat.yml
filebeat keystore list

If using an API key, confirm it is valid and unexpired:

curl -sk 'https://es01:9200/_security/_authenticate?pretty' \
  -H "Authorization: ApiKey $FB_API_KEY"
curl -sk 'https://es01:9200/_security/api_key?name=filebeat&pretty' -u elastic:$ES_PASS \
  | jq '.api_keys[] | {id, invalidated, expiration}'

Example Root Cause Analysis

Every Filebeat host started logging 401 Unauthorized ... unable to authenticate user [filebeat_writer] at the same moment. A scheduled credential-rotation job had changed the filebeat_writer password in Elasticsearch but the new value never reached the Filebeat keystore, so agents kept presenting the old password.

The reproduction curl -u filebeat_writer:$FB_PASS returned 401 with the stored value but 200 with the freshly rotated one, pinpointing a stale secret. Updating the keystore and restarting fixed it:

filebeat keystore add ES_PASSWORD --force     # paste the rotated password
systemctl restart filebeat

With output.elasticsearch.password: "${ES_PASSWORD}" now resolving to the current value, filebeat test output authenticated and the queued events flushed. The team then wired the rotation job to push the new secret to every Filebeat keystore atomically so config and credential rotate together.

Prevention Best Practices

• Store the password/API key in the Filebeat keystore and update it as part of the same rotation that changes it in Elasticsearch.
• Use a dedicated least-privilege user (or API key) for the output, and monitor its expiry.
• Alert on Filebeat 401 rates so a botched rotation pages you before the queue backs up.
• Test credentials with _security/_authenticate in CI/provisioning before deploying config changes.
• Prefer API keys with explicit expirations and automated renewal over long-lived shared passwords.

Quick Command Reference

filebeat test output
curl -sk 'https://es01:9200/_security/_authenticate?pretty' -u filebeat_writer:$FB_PASS
filebeat keystore list
filebeat keystore add ES_PASSWORD --force
curl -sk 'https://es01:9200/_security/_authenticate?pretty' -H "Authorization: ApiKey $FB_API_KEY"

Conclusion

401 Unauthorized means Elasticsearch does not recognize the identity Filebeat presents — a wrong, missing, or expired password or API key, most often after a rotation that never reached the agent. Confirm with _security/_authenticate, update the Filebeat keystore, and tie credential rotation to secret delivery so the two never drift apart. Fix the credential and the queued events drain automatically. More auth fixes are in the Filebeat guides.