🚨 Logstash Error: 'Got response code '401'' — Cause, Fix, and Troubleshooting Guide

Overview

When Elasticsearch security is enabled, the elasticsearch output must present valid credentials on every request, starting with the compatibility healthcheck it runs at startup. If those credentials are missing, wrong, or expired, Elasticsearch answers 401 Unauthorized and the output refuses to consider the cluster usable:

[WARN ][logstash.outputs.elasticsearch][main] Could not connect to a compatible version of Elasticsearch. Got response code '401' contacting Elasticsearch at URL 'https://es.internal:9200/'
[ERROR][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"Got response code '401' ... unable to authenticate user [logstash_writer]"}

401 is an authentication failure — Elasticsearch could not verify who you are. That is distinct from 403 (you authenticated but lack the privilege) and from a transport-layer Elasticsearch Unreachable error (the request never got a response at all). A 401 means the connection succeeded, ES replied, and it rejected your identity.

Symptoms

• Got response code '401' and unable to authenticate user [...] in logstash-plain.log.
• The output logs Could not connect to a compatible version of Elasticsearch even though the node is up and reachable.
• Events back up in the queue and the pipeline stalls because the output never becomes healthy.
• The same URL responds normally to a curl with correct credentials but returns 401 from Logstash.
• It appears right after enabling security, rotating a password, or deploying to an environment where the credential env var was never set.

Common Root Causes

• No credentials configured — security is on, but the output block sets neither user/password nor api_key.
• Wrong or expired password — the password was rotated in Elasticsearch but not updated in Logstash, or it simply has a typo.
• Empty credential variable — password => "${ES_PW}" resolves to an empty string because ES_PW is not set for the service user.
• Wrong cloud_auth — for Elastic Cloud, cloud_auth names the wrong user or carries a stale password.
• API key mismatch — api_key is malformed, was invalidated, or is not in the required id:api_key form.

How to diagnose

Reproduce the check Logstash makes, from the Logstash host, with the exact credentials it uses. If this curl returns 401, the problem is the credentials, not Logstash:

# Should return cluster info, not 401
curl -u logstash_writer:$ES_PW https://es.internal:9200/

# Explicit status code
curl -s -o /dev/null -w '%{http_code}\n' -u logstash_writer:$ES_PW https://es.internal:9200/

Confirm the credential env var is actually present for the service — a common cause is that it is set in your shell but not in the unit environment:

systemctl show logstash -p Environment | tr ' ' '\n' | grep ES_PW
sudo -u logstash bash -c '[ -n "$ES_PW" ] && echo set || echo MISSING'

Then check the output block. It must supply credentials one of the supported ways — user/password, api_key, or cloud_id/cloud_auth. Logstash .conf files use a Ruby-like DSL, so the output is shown as ruby:

output {
  elasticsearch {
    hosts    => ["https://es.internal:9200"]
    user     => "logstash_writer"
    password => "${ES_PW}"   # empty here → 401
    index    => "app-logs-%{+YYYY.MM.dd}"
  }
}

If ${ES_PW} is blank, the output sends an empty password and ES answers 401.

Fixes

Supply valid credentials. For a native user, set user and password, sourcing the secret from the Logstash keystore rather than hard-coding it:

# Store the secret in the keystore, then reference it as ${ES_PW}
/usr/share/logstash/bin/logstash-keystore add ES_PW

output {
  elasticsearch {
    hosts    => ["https://es.internal:9200"]
    user     => "logstash_writer"
    password => "${ES_PW}"
    index    => "app-logs-%{+YYYY.MM.dd}"
  }
}

To authenticate with an API key instead of a username/password, use api_key in id:api_key form (and drop user/password):

output {
  elasticsearch {
    hosts   => ["https://es.internal:9200"]
    api_key => "${ES_API_KEY}"   # value is "id:key"
    index   => "app-logs-%{+YYYY.MM.dd}"
  }
}

For Elastic Cloud, prefer cloud_id with cloud_auth:

output {
  elasticsearch {
    cloud_id   => "${ES_CLOUD_ID}"
    cloud_auth => "logstash_writer:${ES_PW}"
    index      => "app-logs-%{+YYYY.MM.dd}"
  }
}

Verify the fix by tailing the log after a restart — the 401 and the Could not connect to a compatible version warnings should stop and the queue should drain.

What to watch out for

• A 401 is authentication; a 403 is authorization. If curl -u succeeds but Logstash still fails on writes, you likely have a privilege problem, not a credential one.
• Setting a secret in your interactive shell does not set it for the systemd service — use the keystore or the unit Environment, and confirm with sudo -u logstash.
• api_key must be the id:api_key pair, not just the key portion; a bare key resolves to 401.
• After rotating the ES password, update the keystore value and restart Logstash — the old value is cached until the pipeline restarts.
• Do not confuse this with Elasticsearch Unreachable: if there is no response code at all, the failure is transport-layer, not auth.

Related

• Restore a dead Logstash Elasticsearch output connection
• Fix the Logstash PKIX SSL handshake exception
• Handle Elasticsearch output 429 rejections in Logstash