☸️ Kubernetes Error Guide: 'v1beta1.metrics.k8s.io: the server is currently unable to handle the request'

Exact Error Message

Error from server (ServiceUnavailable): the server is currently unable to handle the request
(get pods.metrics.k8s.io)

error: Metrics API not available

You will also see it in kubectl get apiservices:

v1beta1.metrics.k8s.io   kube-system/metrics-server   False (MissingEndpoints)   42m

Overview

kubectl top and Horizontal Pod Autoscalers do not read raw metrics; they query the aggregated Metrics API served under v1beta1.metrics.k8s.io. That API group is registered as an APIService object that points the kube-apiserver at the metrics-server Deployment. When metrics-server is unhealthy, unreachable, or its APIService registration is broken, the aggregation layer returns ServiceUnavailable for anything under metrics.k8s.io.

The consequence is broad: kubectl top nodes/kubectl top pods fail outright, and any HPA that targets CPU or memory can no longer read utilization, so it stops scaling and reports unknown for the metric. This is an availability problem in the aggregation path, not a bug in your workloads.

Symptoms

• kubectl top nodes and kubectl top pods return Metrics API not available or ServiceUnavailable.
• kubectl get apiservice v1beta1.metrics.k8s.io shows AVAILABLE: False.
• HPAs show <unknown>/50% under TARGETS and stop scaling.
• kubectl describe hpa events include failed to get cpu utilization: unable to get metrics.

Common Root Causes

1. metrics-server pod not running

The Deployment is 0/1, crash-looping, or has no ready endpoints, so the APIService has nothing to route to (MissingEndpoints).

2. TLS handshake failing to kubelets

metrics-server cannot scrape kubelets because the kubelet serving certificate is not trusted. Logs show x509: cannot validate certificate and pods stay unready.

3. Aggregation layer connectivity blocked

The kube-apiserver cannot reach the metrics-server Service (NetworkPolicy, firewall, or a misconfigured --enable-aggregator-routing).

4. Stale or duplicate APIService registration

A leftover v1beta1.metrics.k8s.io APIService points at a Service that no longer exists.

Diagnostic Workflow

Step 1: Check the APIService availability

kubectl get apiservice v1beta1.metrics.k8s.io
kubectl describe apiservice v1beta1.metrics.k8s.io

The describe output’s Status conditions name the reason — MissingEndpoints, FailedDiscoveryCheck, and so on.

Step 2: Check the metrics-server Deployment and pods

kubectl get deploy metrics-server -n kube-system
kubectl get pods -n kube-system -l k8s-app=metrics-server

Step 3: Read the pod logs for the real cause

kubectl logs -n kube-system -l k8s-app=metrics-server --tail=50

Look for x509 certificate errors or dial tcp ... i/o timeout to the kubelet port 10250.

Step 4: Confirm the Service has endpoints

kubectl get endpoints metrics-server -n kube-system

An empty endpoints list confirms MissingEndpoints.

Step-by-Step Resolution

1. Establish whether the pods are running and ready:

kubectl get pods -n kube-system -l k8s-app=metrics-server -o wide

2. If the pod is crash-looping on TLS to kubelets, add the insecure kubelet-TLS flag (acceptable in many clusters that use self-signed kubelet certs):

kubectl patch deploy metrics-server -n kube-system --type=json \
  -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'

3. Ensure it prefers the correct address type so it can reach nodes:

kubectl patch deploy metrics-server -n kube-system --type=json \
  -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-preferred-address-types=InternalIP"}]'

4. Restart and wait for readiness:

kubectl rollout restart deploy metrics-server -n kube-system
kubectl rollout status deploy metrics-server -n kube-system

5. Confirm the APIService flips to available:

kubectl get apiservice v1beta1.metrics.k8s.io

6. Validate the end-to-end path with the commands that were failing:

kubectl top nodes
kubectl top pods -A
kubectl get hpa -A

For generating hardened metrics-server manifests and HPA definitions, the DevOps AI prompt library has reusable prompts.

Prevention

• Run metrics-server with two replicas and a PodDisruptionBudget so a single node drain does not take the Metrics API offline.
• Pin the metrics-server version to match your Kubernetes minor version; version skew causes discovery failures.
• Monitor the APIService directly: alert on v1beta1.metrics.k8s.io reporting AVAILABLE: False.
• Fix kubelet serving certificates properly (enable RotateKubeletServerCertificate) rather than relying on --kubelet-insecure-tls in production.
• Ensure NetworkPolicies allow the kube-apiserver to reach the metrics-server Service on its serving port.
• Set resource requests on metrics-server so it is not evicted under node pressure, which would silently break HPAs.

Related Errors

• unable to fetch metrics from resource metrics API — the HPA-side view of this same outage.
• no metrics returned from resource metrics API — pods exist but no data yet; often a transient scrape delay.
• couldn't get resource list for metrics.k8s.io/v1beta1 — discovery failing during kubectl api-resources.
• FailedGetResourceMetric in HPA events — the downstream scaling symptom.

Frequently Asked Questions

Why did all my HPAs stop scaling at once? Every CPU/memory-based HPA reads from the same metrics.k8s.io API. When metrics-server is down, every one of them loses its metric source simultaneously and reports <unknown>.

How do I confirm it is metrics-server and not the apiserver? Run kubectl describe apiservice v1beta1.metrics.k8s.io. A MissingEndpoints or FailedDiscoveryCheck condition points at metrics-server; a healthy APIService with failing top points elsewhere.

Is --kubelet-insecure-tls safe? It disables verification of the kubelet’s serving certificate, which is fine for lab clusters but weakens security in production. The durable fix is proper kubelet serving-certificate rotation signed by the cluster CA.

Does removing the APIService fix it? Only if the registration itself is stale. If metrics-server is simply unhealthy, deleting the APIService just removes the Metrics API entirely — repair the Deployment instead.

For more autoscaling and control-plane fixes, see the Kubernetes & Helm guides.