🚨 GCP Error Guide: 'ImagePullBackOff — denied: Permission artifactregistry.repositories.downloadArtifacts denied' on GKE

Exact Error Message

When a GKE pod cannot pull its image from Artifact Registry, kubectl describe pod surfaces an event like this:

Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  61s                 default-scheduler  Successfully assigned prod/checkout-7d9c8b6f4-q2xnv to gke-prod-default-pool-a1b2c3d4-9xkz
  Normal   Pulling    18s (x3 over 58s)   kubelet            Pulling image "us-central1-docker.pkg.dev/acme-prod-417_/services/checkout:1.8.2"
  Warning  Failed     17s (x3 over 57s)   kubelet            Failed to pull image "us-central1-docker.pkg.dev/acme-prod-417/services/checkout:1.8.2": rpc error: code = Unknown desc = failed to pull and unpack image "us-central1-docker.pkg.dev/acme-prod-417/services/checkout:1.8.2": failed to resolve reference: pulling from host us-central1-docker.pkg.dev failed with status code [manifests 1.8.2]: 403 Forbidden
  Warning  Failed     17s (x3 over 57s)   kubelet            Error: ErrImagePull
  Warning  Failed     4s  (x4 over 56s)   kubelet            Error: ImagePullBackOff

A docker pull or gcloud probe against the same repository returns the underlying IAM denial:

denied: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource
"projects/acme-prod-417/locations/us-central1/repositories/services" (or it may not exist)

What the Error Means

The kubelet on a GKE node tried to pull a container image from Artifact Registry and the registry returned HTTP 403 Forbidden. Underneath the 403 is an IAM authorization failure: the identity making the request lacks the artifactregistry.repositories.downloadArtifacts permission on the target repository. That permission is what grants read/pull access to artifacts; without it, every pull attempt fails and Kubernetes backs off into the ImagePullBackOff state.

The identity doing the pulling is almost never your user account. By default, GKE nodes authenticate to Google APIs using the node pool’s service account (frequently the Compute Engine default service account, PROJECT_NUMBER-compute@developer.gserviceaccount.com). On Workload Identity clusters the node-bootstrap pull still uses the node service account, while application-level pulls can involve other identities. The trailing (or it may not exist) is Google’s deliberately vague phrasing — it almost always means a permissions problem, not a missing repository.

Common Causes

• The node pool’s service account is missing the roles/artifactregistry.reader role (which contains downloadArtifacts) on the repository or project.
• The cluster pulls from a repository in a different project than the one the nodes run in, and cross-project IAM was never granted.
• A custom, least-privilege node service account was provisioned but never bound to the reader role.
• The Artifact Registry API or repository is in a different region/host than the image reference, so authentication targets the wrong resource.
• Someone migrated from the deprecated Container Registry (gcr.io) to Artifact Registry (pkg.dev) and copied over GCS-style permissions instead of Artifact Registry roles.
• A Deny IAM policy or org-level constraint is explicitly blocking artifactregistry.repositories.downloadArtifacts.

How to Reproduce the Error

You can reproduce this cleanly in a test project:

1. Create an Artifact Registry repository and push an image to us-central1-docker.pkg.dev/PROJECT/REPO/IMAGE:TAG.
2. Create a GKE node pool that uses a fresh, dedicated service account with no Artifact Registry roles attached.
3. Deploy a workload referencing that image.

The pod schedules, the kubelet attempts the pull, and within seconds you get 403 Forbidden followed by ImagePullBackOff, because the node service account has never been granted roles/artifactregistry.reader.

Diagnostic Commands

All commands here are read-only — they inspect state without changing it.

# Confirm which pod is failing and read the events.
kubectl get pods -n prod
kubectl describe pod checkout-7d9c8b6f4-q2xnv -n prod

# Identify the node pool's service account.
gcloud container node-pools describe default-pool \
  --cluster prod --region us-central1 \
  --format="value(config.serviceAccount)"

# View the IAM policy on the repository (cross-project safe).
gcloud artifacts repositories get-iam-policy services \
  --location us-central1 --project acme-prod-417

# Confirm the repository exists and note its exact format/host.
gcloud artifacts repositories describe services \
  --location us-central1 --project acme-prod-417

# Ask IAM directly whether the node SA can pull.
gcloud projects get-iam-policy acme-prod-417 \
  --flatten="bindings[].members" \
  --filter="bindings.members:123456789012-compute@developer.gserviceaccount.com" \
  --format="table(bindings.role)"

# Test the exact permission for the node service account.
gcloud artifacts repositories test-iam-permissions services \
  --location us-central1 --project acme-prod-417 \
  --permissions=artifactregistry.repositories.downloadArtifacts

Step-by-Step Resolution

1. Grant the reader role to the node service account. This is the fix in the overwhelming majority of cases. Scope it to the repository for least privilege:

gcloud artifacts repositories add-iam-policy-binding services \
  --location us-central1 --project acme-prod-417 \
  --member="serviceAccount:123456789012-compute@developer.gserviceaccount.com" \
  --role="roles/artifactregistry.reader"

2. For cross-project pulls, grant the binding in the project that hosts the repository, naming the node service account from the cluster’s project. The member string and the --project flag belong to different projects — that is expected.

3. Roll the affected pods so the kubelet retries the pull:

kubectl rollout restart deployment/checkout -n prod

4. If you use a custom node service account, confirm it is the one actually attached to the node pool; a binding on the wrong SA looks correct but changes nothing.

5. If a Deny policy is involved, inspect organization-level deny policies with the IAM team — a repository-level grant cannot override an org Deny.

Within a minute the pods should transition to Running as the pull succeeds.

Prevention and Best Practices

• Bake roles/artifactregistry.reader for the node service account into your cluster Terraform so new node pools are never created without pull access.
• Prefer a dedicated, least-privilege node service account over the Compute Engine default account, and grant it reader at the repository (not project) level.
• For multi-project setups, document which projects host registries and codify the cross-project bindings.
• Use Workload Identity for application access to GCP APIs, but remember node-level image pulls still rely on the node service account.
• Add a smoke test in CI that runs gcloud artifacts repositories test-iam-permissions for each consuming cluster’s service account after any IAM change.
• Pipe ImagePullBackOff events into your alerting so a broken pull is caught before it becomes a stalled rollout. If you want a structured workflow for that, see our incident response tooling.

Related Errors

• Error: ErrImagePull — the immediate predecessor to ImagePullBackOff; same root cause, caught on the first attempt.
• manifest unknown / name unknown — the image tag or repository genuinely does not exist (versus a permissions denial).
• denied: Permission "artifactregistry.repositories.uploadArtifacts" denied — the push-side equivalent, hit by CI pipelines rather than nodes.
• failed to pull image ... 401 Unauthorized — authentication (no credentials) rather than authorization (wrong permissions).

Frequently Asked Questions

Why does the error say “or it may not exist” if the repository clearly exists? Google returns intentionally ambiguous wording so a caller without permission cannot enumerate which resources are present. In practice, if gcloud artifacts repositories describe succeeds for you, the repository exists and the message is about permissions.

Which identity actually needs the permission — my user or the node? The GKE node pool’s service account. Granting yourself reader access lets you pull locally but does nothing for the kubelet on the node.

Do I need roles/artifactregistry.admin to fix this? No. roles/artifactregistry.reader is sufficient to pull and contains the downloadArtifacts permission. Admin is over-privileged for nodes.

I migrated from gcr.io and it broke — why? Artifact Registry uses its own IAM roles. GCS-bucket permissions that worked for the legacy Container Registry do not apply to pkg.dev repositories; you must grant Artifact Registry roles explicitly.

Will Workload Identity fix the image pull? Not by itself. Workload Identity governs how pods reach Google APIs at runtime; the initial image pull is still performed by the node service account, which is what needs downloadArtifacts.

How long until the pod recovers after I grant the role? IAM changes propagate within seconds to a minute. Triggering a kubectl rollout restart forces an immediate retry rather than waiting for the backoff timer. See more in our GCP guides.