🚨 GCP Error Guide: 'could not find default credentials' on GKE — Fix Workload Identity

Overview

On a GKE cluster with Workload Identity, pods are supposed to get Google credentials automatically from the metadata server — no key files. When that chain is misconfigured, the Google client libraries can’t find any credentials and fail with the Application Default Credentials error:

google.auth.exceptions.DefaultCredentialsError:
 Could not automatically determine credentials. Please set
 GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials
 and re-run the application. For more information, please see
 https://cloud.google.com/docs/authentication/external/set-up-adc

A closely related variant appears when credentials are found but the Workload Identity binding is missing, so the metadata server returns a 403:

compute.googleapis.com metadata: unable to fetch token:
 google: could not fetch token from metadata: ... status 403

Both point at the same root problem: the Kubernetes service account (KSA) the pod runs as is not correctly wired to a Google service account (GSA).

Symptoms

• Pods log DefaultCredentialsError / “Could not automatically determine credentials” on the first Google API call.
• curl to the metadata server from inside the pod returns 403 or the wrong (node) service account.
• The workload runs fine locally with a key file but fails once deployed to GKE with Workload Identity enabled.
• Some pods work and others don’t — the difference is which KSA (and namespace) they run under.
• After removing a mounted JSON key to “go keyless,” the app immediately breaks.

Common Root Causes

• KSA not annotated — the Kubernetes service account is missing the iam.gke.io/gcp-service-account annotation pointing at the GSA.
• Missing IAM binding — the GSA lacks the roles/iam.workloadIdentityUser binding for serviceAccount:PROJECT.svc.id.goog[NAMESPACE/KSA].
• Pod uses the wrong (or default) KSA — the Deployment has no serviceAccountName, so it runs as default, which was never wired up.
• Workload Identity not enabled — the cluster or the specific node pool doesn’t have Workload Identity / the GKE metadata server enabled.
• Namespace mismatch — the annotation and the IAM member string reference different namespaces than where the pod actually runs.
• GSA lacks the target permission — Workload Identity resolves, but the GSA doesn’t have the role for the API being called (a 403 after auth, not a credentials error).
• Recently removed key file — the app relied on GOOGLE_APPLICATION_CREDENTIALS and that env/mount was deleted before the WI chain was proven.

Diagnostic Workflow

First confirm Workload Identity is enabled on the cluster and node pool:

gcloud container clusters describe CLUSTER --region REGION \
  --format="value(workloadIdentityConfig.workloadPool)"
gcloud container node-pools describe POOL --cluster CLUSTER --region REGION \
  --format="value(config.workloadMetadataConfig.mode)"

The pool mode should be GKE_METADATA; the workload pool should be PROJECT_ID.svc.id.goog.

Find which KSA the pod actually uses, then check its annotation:

kubectl get pod POD -n NAMESPACE -o jsonpath='{.spec.serviceAccountName}{"\n"}'
kubectl get serviceaccount KSA_NAME -n NAMESPACE \
  -o jsonpath='{.metadata.annotations.iam\.gke\.io/gcp-service-account}{"\n"}'

An empty annotation is a smoking gun. Next confirm the IAM binding on the GSA:

gcloud iam service-accounts get-iam-policy \
  GSA_NAME@PROJECT_ID.iam.gserviceaccount.com \
  --flatten="bindings[].members" \
  --filter="bindings.role:roles/iam.workloadIdentityUser"

You should see serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]. Then test the metadata server from inside the running pod:

kubectl exec -it POD -n NAMESPACE -- \
  curl -s -H "Metadata-Flavor: Google" \
  "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email"

This should print the GSA email. If it prints the node’s default compute SA or returns 403, the WI mapping is wrong.

Example Root Cause Analysis

After migrating a service to Workload Identity, a team deleted the mounted JSON key and the pod immediately began crashing with Could not automatically determine credentials. They had created the GSA, annotated a KSA, and added the IAM binding — so on paper everything was done.

The metadata-server test told the real story: kubectl get pod ... -o jsonpath='{.spec.serviceAccountName}' returned default. The Deployment had never set serviceAccountName, so the pod ran as the namespace’s default KSA — which had no annotation — while all the wiring had been applied to a different, purpose-built KSA.

The fix was one line in the pod spec:

spec:
  serviceAccountName: app-ksa   # was implicitly "default"

After redeploying, the metadata endpoint returned the correct GSA email and credentials resolved automatically. The team added an admission-policy check to reject workloads running as default in that namespace.

Prevention Best Practices

• Always set serviceAccountName — never let production pods fall back to the default KSA; make it explicit in every Deployment.
• Wire the full chain as one unit — annotate the KSA and add the roles/iam.workloadIdentityUser binding together, and keep the namespace/KSA names identical in both.
• Prove keyless before removing keys — verify the metadata-server test returns the right GSA email while the old key is still in place, then remove the key.
• Enable Workload Identity on every node pool — a mixed cluster where one pool lacks GKE_METADATA causes intermittent, pool-dependent failures.
• Grant the GSA only the roles it needs — least privilege on the Google side, so a resolved identity still can’t do more than the workload requires.
• Codify it — manage the annotation and IAM binding in Terraform/Helm so the mapping can’t drift or be half-applied.

Quick Command Reference

# Is Workload Identity enabled on the cluster/pool?
gcloud container clusters describe CLUSTER --region REGION \
  --format="value(workloadIdentityConfig.workloadPool)"

# Which KSA does the pod run as?
kubectl get pod POD -n NAMESPACE -o jsonpath='{.spec.serviceAccountName}{"\n"}'

# Is the KSA annotated with the GSA?
kubectl get sa KSA_NAME -n NAMESPACE \
  -o jsonpath='{.metadata.annotations.iam\.gke\.io/gcp-service-account}{"\n"}'

# Does the GSA have the workloadIdentityUser binding?
gcloud iam service-accounts get-iam-policy \
  GSA_NAME@PROJECT_ID.iam.gserviceaccount.com \
  --flatten="bindings[].members" \
  --filter="bindings.role:roles/iam.workloadIdentityUser"

# What identity does the metadata server hand the pod?
kubectl exec -it POD -n NAMESPACE -- curl -s -H "Metadata-Flavor: Google" \
  "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email"

# Bind the KSA to the GSA
gcloud iam service-accounts add-iam-policy-binding \
  GSA_NAME@PROJECT_ID.iam.gserviceaccount.com \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]"

Conclusion

could not find default credentials on GKE almost always means the Workload Identity chain is broken between the pod’s KSA and a Google service account — a missing annotation, a missing workloadIdentityUser binding, a pod silently running as default, or a node pool without the metadata server. Trace it in order: which KSA the pod uses, whether it’s annotated, whether the IAM binding exists, and what the metadata server actually returns. Fix the broken link, prove it with the metadata-email test before deleting any key, and codify the mapping so it never drifts back.