🚨 OpenTelemetry Error Guide: 'pods is forbidden' in k8sattributes — Fix Collector RBAC

Overview

This error appears when the k8sattributes processor tries to enrich telemetry with Kubernetes metadata but the Collector’s ServiceAccount lacks permission to read pods. The Kubernetes API returns a Forbidden response and the Collector logs it on startup and on every reconnect:

pods is forbidden: User "system:serviceaccount:observability:otel-collector" cannot list resource "pods" in API group "" at the cluster scope

Depending on which informer fails first, you may see the same shape for other resources the processor watches:

replicasets.apps is forbidden: User "system:serviceaccount:observability:otel-collector" cannot list resource "replicasets" in API group "apps" at the cluster scope

The message is a Kubernetes RBAC denial: the processor’s list/watch calls against the API server are rejected, so it cannot build its pod cache and telemetry goes un-enriched (no k8s.pod.name, k8s.namespace.name, etc.).

Symptoms

• Spans, metrics, and logs arrive but are missing k8s.* resource attributes like k8s.pod.name or k8s.deployment.name.
• Collector logs repeat pods is forbidden (and often replicasets.apps is forbidden, namespaces is forbidden).
• The processor logs failed to watch / informer errors and retries in a loop.
• The error names a ServiceAccount in the form system:serviceaccount:<namespace>:<name>.
• It started right after deploying the Collector via a bare manifest or a chart with RBAC disabled.

Common Root Causes

• No ClusterRole — the Collector’s ServiceAccount was never granted the list/watch/get verbs on pods.
• Namespaced Role instead of ClusterRole — a Role/RoleBinding only grants access in one namespace, but the processor watches cluster-wide.
• Binding to the wrong ServiceAccount — the ClusterRoleBinding subject name or namespace does not match the pod’s actual ServiceAccount.
• Missing resources in the rule — pods is granted but replicasets or namespaces (needed to resolve owner metadata) are not.
• Default ServiceAccount — the Deployment omits serviceAccountName, so it runs as default with no permissions.
• RBAC disabled in the chart — the Helm values left clusterRole.create: false, so no role was rendered.

Diagnostic Workflow

Confirm exactly which ServiceAccount the Collector runs as and whether it can list pods cluster-wide, using kubectl auth can-i to reproduce the denial:

# Which ServiceAccount is the Collector pod using?
kubectl get pod -n observability -l app=otel-collector \
  -o jsonpath='{.items[0].spec.serviceAccountName}'

# Reproduce the exact permission check the processor performs
kubectl auth can-i list pods \
  --as=system:serviceaccount:observability:otel-collector -A

# Watch the RBAC denials in the Collector log
kubectl logs -n observability deploy/otel-collector | grep -i 'forbidden'

Grant a ClusterRole with the verbs and resources the k8sattributes processor needs, and bind it to the Collector’s ServiceAccount. The processor reads pods and namespaces (core group "") plus replicasets (the apps group) to resolve owner metadata:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-collector
rules:
  - apiGroups: [""]
    resources: ["pods", "namespaces"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["replicasets"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: otel-collector
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: otel-collector
subjects:
  - kind: ServiceAccount
    name: otel-collector       # must match the pod's serviceAccountName
    namespace: observability   # must match the pod's namespace

Make sure the Deployment actually uses that ServiceAccount and the processor is wired into the pipeline:

# Deployment spec.template.spec
serviceAccountName: otel-collector

processors:
  k8sattributes:
    auth_type: serviceAccount
    passthrough: false
    extract:
      metadata:
        - k8s.namespace.name
        - k8s.pod.name
        - k8s.deployment.name
    pod_association:
      - sources:
          - from: resource_attribute
            name: k8s.pod.ip

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [k8sattributes, batch]
      exporters: [otlphttp]

Example Root Cause Analysis

A team deployed the Collector from a hand-written manifest that created a ServiceAccount named otel-collector in the observability namespace but no RBAC objects. On startup the k8sattributes processor’s pod informer immediately logged pods is forbidden: User "system:serviceaccount:observability:otel-collector" cannot list resource "pods"..., and every span reached the backend without any k8s.* attributes, breaking the Kubernetes views in their dashboards.

The fix had two parts. First, a ClusterRole granting get/list/watch on pods and namespaces (core group) plus replicasets (apps group) was created and bound to the observability:otel-collector ServiceAccount with a ClusterRoleBinding. Second, kubectl auth can-i list pods --as=system:serviceaccount:observability:otel-collector -A was used to confirm the grant returned yes before rolling the Deployment. After the restart the informers synced, the forbidden errors stopped, and enriched attributes like k8s.pod.name and k8s.deployment.name appeared on new spans within seconds.

Prevention Best Practices

• Deploy the Collector with a dedicated ServiceAccount and always set serviceAccountName explicitly — never rely on default.
• Grant a cluster-scoped ClusterRole/ClusterRoleBinding because k8sattributes watches resources across all namespaces.
• Include every resource the processor reads: pods, namespaces, and replicasets (and nodes if you extract node metadata).
• Keep RBAC rules least-privilege: only get, list, watch — the processor never needs write verbs.
• Validate access with kubectl auth can-i ... --as=system:serviceaccount:<ns>:<sa> in CI before rolling out.
• If using the Helm chart, keep clusterRole.create: true and confirm the rendered subject matches the pod’s ServiceAccount.

Quick Command Reference

# Confirm the pod's ServiceAccount
kubectl get pod -n observability -l app=otel-collector \
  -o jsonpath='{.items[0].spec.serviceAccountName}'

# Reproduce the RBAC check the processor makes
kubectl auth can-i list pods --as=system:serviceaccount:observability:otel-collector -A

# Inspect the current binding and its subjects
kubectl describe clusterrolebinding otel-collector

# Watch for forbidden errors after applying RBAC
kubectl logs -n observability deploy/otel-collector -f | grep -i forbidden

Conclusion

pods is forbidden is a Kubernetes RBAC denial, not a Collector bug: the k8sattributes processor cannot list the pods it needs to enrich telemetry because its ServiceAccount lacks a cluster-scoped grant. Create a ClusterRole with get/list/watch on pods, namespaces, and replicasets, bind it to the exact system:serviceaccount:<namespace>:<name> the pod runs as, and verify with kubectl auth can-i before rolling out. Once the informers can read the API, the missing k8s.* attributes return on their own.

Related

• OpenTelemetry Error Guide: ‘permission denied’ in the filelog receiver
• OpenTelemetry Error Guide: ‘failed to parse config’
• OpenTelemetry Error Guide: ‘connection refused’ to the Collector