🚨 GCP Error: 'Permission secretmanager.versions.access denied' — Cause, Fix, and Troubleshooting Guide

Overview

Reading a secret value fails when the calling identity lacks secretmanager.versions.access on that secret:

google.api_core.exceptions.PermissionDenied: 403 Permission
'secretmanager.versions.access' denied for resource
'projects/acme-prod-platform/secrets/db-password/versions/latest'
(or it may not exist).

Secret Manager access is granted per secret (or per project) via roles/secretmanager.secretAccessor. A workload can list secret names yet be denied the value, because accessing a version is a distinct, tightly-scoped permission.

Symptoms

• Apps on Cloud Run / GKE / Cloud Functions fail at startup fetching a secret with secretmanager.versions.access denied.
• The identity can see the secret exists but cannot read its payload.
• Works in dev (broad roles) but fails in prod where IAM is least-privilege.
• The (or it may not exist) tail appears even though the secret is clearly there.

Common Root Causes

1. Missing secretAccessor role

The runtime service account was never granted roles/secretmanager.secretAccessor on the secret or project.

2. Wrong runtime identity

The service runs as the default compute/App Engine service account, not the one you granted access to.

3. Grant at the wrong scope

The role was bound on a different project, or on a different secret than the one being read.

4. Secret Manager API disabled, or secret in another project

The API is off, or the secret path points to a project the identity has no access to (hence “or it may not exist”).

How to Diagnose

All read-only.

# Which identity is the workload actually running as?
gcloud run services describe api \
  --region=us-central1 --project=acme-prod-platform \
  --format="value(spec.template.spec.serviceAccountName)"

# Does that identity have accessor on the secret?
gcloud secrets get-iam-policy db-password \
  --project=acme-prod-platform \
  --flatten="bindings[].members" \
  --filter="bindings.role:roles/secretmanager.secretAccessor" \
  --format="table(bindings.members)"

# Test the exact permission for the active identity
gcloud secrets get-iam-policy db-password --project=acme-prod-platform >/dev/null && \
gcloud projects test-iam-permissions acme-prod-platform \
  --permissions=secretmanager.versions.access

# Is the API enabled and the secret present?
gcloud services list --enabled --project=acme-prod-platform \
  --filter="config.name:secretmanager.googleapis.com"
gcloud secrets describe db-password --project=acme-prod-platform --format="value(name)"

If the runtime service account isn’t in the accessor list, that’s the gap.

Fixes

Grant secretAccessor on the specific secret (least privilege — prefer per-secret over project-wide):

gcloud secrets add-iam-policy-binding db-password \
  --project=acme-prod-platform \
  --member="serviceAccount:app-runtime@acme-prod-platform.iam.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor"

Point the workload at the intended service account instead of the default one:

gcloud run services update api \
  --region=us-central1 --project=acme-prod-platform \
  --service-account=app-runtime@acme-prod-platform.iam.gserviceaccount.com

Enable the API if it’s off:

gcloud services enable secretmanager.googleapis.com --project=acme-prod-platform

Allow ~1 minute for IAM propagation, then retry.

What to Watch Out For

• roles/viewer lets you list secret metadata but not read values — you need secretmanager.secretAccessor for the payload.
• Grant on the individual secret, not the whole project, so one workload can’t read every secret.
• The (or it may not exist) wording is deliberately ambiguous — it can mean a real permission gap or a wrong project/secret path; verify both.
• On GKE, the accessor role must be on the identity Workload Identity maps the pod to, not the node service account.

Related

• GCP Error: ‘PERMISSION_DENIED (403)’ Caller Does Not Have Permission
• GCP Error: ‘Request had insufficient authentication scopes’
• GCP Error: ‘Storage objects 403 bucket access denied’
• More in the GCP error guides.