🚨 Kubernetes RBAC Error: 'Forbidden: User cannot list resource' Access Denied

Overview

Kubernetes rejects an API request when the authenticated identity has no RBAC rule allowing the verb on the target resource. The API server returns HTTP 403 and kubectl prints a Forbidden message that names the user, the verb, the resource, and often the namespace:

Error from server (Forbidden): pods is forbidden: User "dev-ci@example.com"
cannot list resource "pods" in API group "" in the namespace "payments"

The same denial appears for service accounts running inside the cluster, worded from the ServiceAccount identity:

pods is forbidden: User "system:serviceaccount:payments:deploy-bot"
cannot create resource "pods" in API group "apps" in the namespace "payments"

This is an authorization failure, not an authentication failure. The credential is valid and the request reached the API server; RBAC simply has no rule that permits it.

Symptoms

• kubectl get, apply, or logs fails with Error from server (Forbidden) while the same command works for a cluster-admin.
• A CI/CD pipeline or controller pod logs cannot create/list/watch resource ... in the namespace ... and its reconcile loop stalls.
• An operator or sidecar crash-loops with 403s against a resource it needs to watch (e.g. configmaps, secrets, leases).
• Access works in one namespace but is denied in another, because a namespaced RoleBinding does not cover the second namespace.
• A newly created ServiceAccount token authenticates fine but every API call is forbidden.

Common Root Causes

• No binding at all — a Role or ClusterRole with the right rules exists, but no RoleBinding/ClusterRoleBinding connects it to the user, group, or ServiceAccount.
• Wrong scope — a namespaced Role/RoleBinding was used where cluster-wide access is needed, or a ClusterRole was bound with a namespaced RoleBinding that only grants it in one namespace.
• Verb gap — the rule grants get/list but the client needs watch, create, update, patch, delete, or deletecollection.
• Wrong API group — pods live in the core group "", but deployments are in apps and ingresses in networking.k8s.io; a rule listing the wrong apiGroups silently fails to match.
• Subresource omitted — pods/log, pods/exec, and pods/portforward are separate resources; granting pods does not grant its subresources.
• Wrong subject identity — the binding names the wrong ServiceAccount namespace, a user string that does not match the OIDC/cert CN, or a group the identity is not a member of.
• Aggregated ClusterRole labels — an aggregated ClusterRole is empty because no source role carries the aggregation label.

Diagnostic Workflow

Start by asking the API server exactly what the identity can do. kubectl auth can-i is the fastest, safest probe:

kubectl auth can-i list pods --namespace payments
kubectl auth can-i create deployments.apps -n payments

Impersonate the failing identity to reproduce the denial without its credentials (requires impersonation rights):

kubectl auth can-i --list --as=dev-ci@example.com -n payments
kubectl auth can-i list pods \
  --as=system:serviceaccount:payments:deploy-bot -n payments

Confirm which identity the request actually uses. A common surprise is that the token maps to a different user/group than expected:

kubectl auth whoami

Find the bindings that reference the subject, then read the roles they point at:

kubectl get rolebindings,clusterrolebindings -A -o wide \
  | grep -i 'deploy-bot\|dev-ci'
kubectl describe rolebinding <name> -n payments
kubectl describe clusterrole <role-name>

Inspect a role’s exact rules to spot verb, apiGroup, or subresource gaps:

kubectl get clusterrole edit -o yaml \
  | grep -A3 -E 'apiGroups|resources|verbs'

Check the API server audit log (if enabled) for the authorization decision and the reason:

grep -i 'forbidden' /var/log/kubernetes/audit.log \
  | grep 'deploy-bot' | tail

Example Root Cause Analysis

A controller ServiceAccount payments/deploy-bot crash-looped with cannot list resource "leases" in API group "coordination.k8s.io". The team had created a Role granting full access to deployments and pods, and a matching RoleBinding, so deployments worked. Leader election, however, needs leases in the coordination.k8s.io API group, which the Role never mentioned.

kubectl auth can-i --list --as=system:serviceaccount:payments:deploy-bot -n payments confirmed the identity had no verb on leases. The fix was a minimal rule addition, not a broader grant:

- apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]

After kubectl apply and a pod restart, leader election succeeded and the crash loop cleared. The tempting shortcut, binding the ServiceAccount to cluster-admin, would have fixed the symptom while handing a webhook-reachable workload full control of the cluster.

Prevention Best Practices

• Grant least privilege deliberately. Start from no access and add only the apiGroups, resources, subresources, and verbs a workload proves it needs. Never paper over a 403 with cluster-admin.
• Prefer namespaced Roles. Use Role/RoleBinding scoped to a namespace unless the workload genuinely needs cluster-wide reach; reserve ClusterRoleBinding for infrastructure controllers.
• Pin ServiceAccounts per workload. Give each Deployment its own ServiceAccount instead of sharing default, so a compromised pod’s blast radius is one role.
• Validate in CI. Run kubectl auth can-i --list --as=... in a pipeline check to assert an identity has exactly the expected permissions and nothing more.
• Audit standing access. Periodically review ClusterRoleBindings and any subject bound to cluster-admin, edit, or wildcard (*) rules.
• Disable auto-mounted tokens where a pod does not call the API server (automountServiceAccountToken: false).

Quick Command Reference

# Can the current user do it?
kubectl auth can-i <verb> <resource>.<group> -n <ns>

# What can an identity do (impersonation)?
kubectl auth can-i --list --as=<user-or-sa> -n <ns>
kubectl auth can-i --list \
  --as=system:serviceaccount:<ns>:<sa> -n <ns>

# Who am I actually authenticating as?
kubectl auth whoami

# Find and read the bindings/roles
kubectl get rolebindings,clusterrolebindings -A -o wide | grep <subject>
kubectl describe clusterrole <role>

# Inspect rules for verb/apiGroup gaps
kubectl get role <name> -n <ns> -o yaml

Conclusion

A Kubernetes Forbidden: User cannot list resource error is RBAC working as designed: an authenticated identity hit a resource it was never granted. Read the message literally — it names the identity, verb, resource, API group, and namespace — then use kubectl auth can-i --list --as=... to confirm the gap and describe the roles behind the subject. Close it with the smallest possible rule addition scoped to the right namespace, and validate the result. Resist the urge to bind cluster-admin; the whole value of RBAC is that a leaked ServiceAccount token can only do what its narrow role allows.