☸️ Kubernetes Error Guide: 'the server has asked for the client to provide credentials' kubectl 401 Failure

Exact Error Message

$ kubectl get pods
error: You must be logged in to the server (Unauthorized)

Run with more detail and you will see the fuller form:

error: You must be logged in to the server (the server has asked for the
client to provide credentials)

Overview

This is an authentication failure: HTTP 401 Unauthorized. The API server received your request, could not verify who you are, and asked the client to present valid credentials. Note the difference from 403 Forbidden (is forbidden: User ... cannot ...), which means you were authenticated but lack permission. A 401 means the identity itself was rejected.

kubectl authenticates using whatever is configured for the current context: a client certificate, a bearer token, or an exec plugin that fetches a token from a cloud provider or OIDC issuer. When that credential is expired, malformed, revoked, or simply absent, the API server cannot authenticate the request and returns this error. The fix is to restore a valid credential in your kubeconfig.

Symptoms

• All kubectl commands fail with Unauthorized or the must be logged in message.
• The failure is immediate and identical for every resource, indicating identity rejection rather than a per-object permission issue.
• It started working before and stopped, often exactly when a token or certificate expired.
• Other users or a freshly generated kubeconfig work against the same cluster.

Common Root Causes

1. Expired client certificate

Client certs embedded in a kubeconfig (common with kubeadm admin configs) are typically valid for one year. Once expired, the API server rejects them with 401.

2. Expired or revoked bearer token

A static or OIDC token in users[].user.token has a limited lifetime. After expiry it no longer authenticates.

3. Exec credential plugin failing

Cloud kubeconfigs use an exec block (aws eks get-token, gke-gcloud-auth-plugin, az). If the underlying CLI is misconfigured, logged out, or missing, no token is produced.

4. Wrong or stale kubeconfig context

Pointing at an old cluster, a rotated CA, or a user whose credentials were rotated out yields 401.

5. Clock skew

A client clock far out of sync makes short-lived tokens appear invalid, causing authentication to fail.

Diagnostic Workflow

Step 1: Confirm which context and user are active

kubectl config current-context
kubectl config view --minify -o jsonpath='{.users[0].name}{"\n"}'

Step 2: Determine which credential type is in use

kubectl config view --minify -o jsonpath='{.users[0].user}'

Look for client-certificate-data, token, or an exec block; each has a different fix.

Step 3: Check client certificate expiry, if used

kubectl config view --raw -o jsonpath='{.users[0].user.client-certificate-data}' \
  | base64 -d | openssl x509 -noout -enddate

An notAfter date in the past confirms an expired cert.

Step 4: Test the exec plugin directly, if used

aws eks get-token --cluster-name <cluster>   # or the plugin your kubeconfig names

An error here explains why kubectl gets no token.

Step 5: Inspect the raw 401 with verbose logging

kubectl get pods -v=8 2>&1 | grep -i "response status"

Confirm it is 401 Unauthorized, distinguishing it from 403.

Step-by-Step Resolution

1. Regenerate the kubeconfig from your provider when using managed clusters; this refreshes the exec plugin and endpoints:

aws eks update-kubeconfig --name <cluster> --region <region>
# or
gcloud container clusters get-credentials <cluster> --region <region>

2. Renew an expired kubeadm client certificate on the control-plane node:

sudo kubeadm certs renew admin.conf
sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

3. Re-authenticate the underlying CLI for exec-based auth so the plugin can mint tokens:

aws sso login          # or: gcloud auth login / az login

Ensure the required plugin binary (for example gke-gcloud-auth-plugin) is installed and on PATH.

4. Replace a stale static token in the user entry with a current one, or switch to an exec plugin so tokens refresh automatically.

5. Fix clock skew if tokens are short-lived:

sudo timedatectl set-ntp true

6. Verify authentication is restored:

kubectl auth whoami
kubectl get nodes

kubectl auth whoami printing your identity confirms the credential is now accepted.

Prevention

• Prefer exec credential plugins over long-lived static tokens so credentials refresh automatically and short-lived tokens are used.
• Track certificate expiry with kubeadm certs check-expiration and renew well before the deadline.
• Automate kubeconfig generation in onboarding and CI rather than sharing hand-edited files that go stale.
• Keep provider CLIs and auth plugins updated and part of your standard tooling install.
• Enable NTP on all workstations and CI runners to avoid clock-skew authentication failures.

Related Errors

• error: You must be logged in to the server (Unauthorized) — the shorter form of the same 401.
• is forbidden: User "..." cannot get resource — a 403, meaning authenticated but not authorized (an RBAC issue).
• Unable to connect to the server: EOF — a connection-layer failure before authentication even happens.
• Error: exec plugin: invalid apiVersion — a version mismatch in the kubeconfig exec block, breaking token retrieval.

Frequently Asked Questions

Is this a 401 or a 403? This message is a 401 Unauthorized, meaning the server could not verify your identity. A 403 Forbidden (is forbidden: ... cannot ...) means you are authenticated but lack permission. Treat them differently.

Why did it work yesterday and fail today? The most common reason is that a client certificate or token expired overnight. Check notAfter on the cert or the token lifetime; renewing the credential restores access.

How do I fix exec-plugin auth? Run the provider’s login command (aws sso login, gcloud auth login, az login) and confirm the auth plugin binary is installed and on PATH, then retry kubectl. Regenerating the kubeconfig also refreshes the exec block.

What does kubectl auth whoami tell me? It reports the identity the API server sees for your current credentials. If it errors, your credential is not authenticating; if it prints a user, the 401 is resolved. Handy troubleshooting prompts live in the DevOps AI prompt library.

Can clock skew really cause this? Yes. Short-lived tokens carry issued-at and expiry timestamps, so a workstation clock that is far off makes valid tokens look expired. Enabling NTP fixes it. For more, see the Kubernetes & Helm guides.