🚨 VictoriaMetrics Error: 'cannot discover kubernetes targets ... 403 Forbidden' — Cause, Fix, and Troubleshooting Guide

Overview

vmagent’s kubernetes_sd_configs discovers scrape targets by calling the Kubernetes API for pods, endpoints, services, and nodes. Those calls use the pod’s mounted ServiceAccount token, and the API server enforces RBAC on them. When the ServiceAccount lacks list/watch permission on the requested object, discovery fails:

cannot discover kubernetes targets: unexpected status code returned from "https://kubernetes.default.svc/api/v1/pods": 403 Forbidden; the ServiceAccount is missing list/watch RBAC

A 403 is an authorization failure — the request reached the API server and was rejected. That is distinct from a 401 (bad or missing token) or a connection error (running outside the cluster with no kubeconfig). The path in the message (/api/v1/pods) tells you which resource the missing RBAC applies to.

Symptoms

• vmagent logs cannot discover kubernetes targets and the target list stays empty for the affected role.
• The /targets page in vmagent shows zero endpoints for a kubernetes_sd_configs job.
• The error names a specific API path (/api/v1/pods, /api/v1/endpoints, /api/v1/services).
• Metrics stop flowing for workloads that were previously auto-discovered after an RBAC or ServiceAccount change.

Common Root Causes

• The ServiceAccount lacks list/watch RBAC on pods, endpoints, services, or nodes — no ClusterRole grants those verbs, or the binding is missing.
• Wrong or missing SA token mount — automountServiceAccountToken: false, or the pod runs under default instead of the intended ServiceAccount.
• Running outside the cluster without a kubeconfig — vmagent runs off-cluster and has no -promscrape.kubernetesSDCheckInterval reachable in-cluster credentials.
• API server throttling — heavy discovery load triggers 429/priority-and-fairness pushback, which can surface alongside discovery failures on large clusters.

How to diagnose

Confirm exactly which verb and resource the ServiceAccount is denied, using an impersonation check from any kubectl host:

kubectl auth can-i list pods \
  --as=system:serviceaccount:monitoring:vmagent

Verify the token is actually mounted inside the vmagent pod:

kubectl exec -n monitoring deploy/vmagent -- \
  ls /var/run/secrets/kubernetes.io/serviceaccount/

Inspect which ServiceAccount the pod really runs as:

kubectl get pod -n monitoring -l app=vmagent \
  -o jsonpath='{.items[0].spec.serviceAccountName}'

Fixes

1. Create a ClusterRole granting get/list/watch and bind it to vmagent’s ServiceAccount. vmagent needs read access to the objects it discovers:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: vmagent
rules:
  - apiGroups: [""]
    resources: ["pods", "endpoints", "services", "nodes", "nodes/metrics"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vmagent
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: vmagent
subjects:
  - kind: ServiceAccount
    name: vmagent
    namespace: monitoring

2. Mount the SA token. Ensure the Deployment sets serviceAccountName: vmagent and does not disable automountServiceAccountToken, so the token lands at /var/run/secrets/kubernetes.io/serviceaccount/.

3. Provide a kubeconfig when running out-of-cluster. If vmagent runs outside Kubernetes, point each kubernetes_sd_configs entry at a kubeconfig with the needed permissions rather than relying on the in-cluster token.

4. Verify access with kubectl auth can-i. After applying RBAC, confirm the grant resolves before restarting vmagent:

kubectl auth can-i list pods \
  --as=system:serviceaccount:monitoring:vmagent
# expect: yes

What to watch out for

• 403 is authorization (missing RBAC); 401 is authentication (bad token). Fix the right one — adding RBAC will not help a 401.
• RBAC must cover every role you use: discovering endpoints needs endpoints permissions even if you think of the job as pod discovery.
• ClusterRole vs Role matters — cluster-wide discovery requires a ClusterRole/ClusterRoleBinding, not a namespaced Role.
• On large clusters, prefer the endpoints/endpointslices roles and reasonable filters to reduce API server load and avoid throttling.

Related

• VictoriaMetrics Error Guide: cannot load relabel configs
• VictoriaMetrics Error Guide: remote write connection refused
• VictoriaMetrics Error Guide: promscrape maxScrapeSize