🚨 Kubernetes Error Guide: 'unable to fetch metrics from resource metrics API' — Fix HPA Scaling

Overview

unable to fetch metrics from resource metrics API is what a HorizontalPodAutoscaler reports when it cannot read the CPU/memory numbers it needs to decide whether to scale. Without metrics, the HPA has no signal, so it freezes the replica count and marks itself as unable to scale.

You will see it in the HPA’s status and events:

Warning  FailedGetResourceMetric       horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
Warning  FailedComputeMetricsReplicas  horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)

And in kubectl get hpa, the target column shows <unknown>:

NAME       REFERENCE          TARGETS         MINPODS   MAXPODS   REPLICAS
web-hpa    Deployment/web     <unknown>/70%   2         10        2

The HPA controller reaches the metrics through the aggregated APIs metrics.k8s.io (for CPU/memory, served by metrics-server) and custom.metrics.k8s.io/external.metrics.k8s.io (served by an adapter like Prometheus Adapter or KEDA). This error means the relevant metrics API is missing, unhealthy, or returning nothing for the pods in question.

Symptoms

• kubectl get hpa shows <unknown> in the TARGETS column and the replica count never changes.
• HPA events contain FailedGetResourceMetric and unable to fetch metrics from resource metrics API.
• kubectl top pods also fails (error: Metrics API not available) when the issue is metrics-server itself.
• For custom/external metrics, the message names custom.metrics.k8s.io or external.metrics.k8s.io instead of pods.metrics.k8s.io.

kubectl describe hpa web-hpa

Conditions:
  Type            Status  Reason                   Message
  AbleToScale     True    SucceededGetScale        the HPA controller was able to get the target's scale
  ScalingActive   False   FailedGetResourceMetric  the HPA was unable to compute the replica count:
                          failed to get cpu utilization: unable to get metrics for resource cpu:
                          no metrics returned from resource metrics API

Common Root Causes

1. metrics-server is not installed or not running

The most common cause for CPU/memory HPAs: there is no metrics-server serving metrics.k8s.io, so the aggregated API returns nothing.

kubectl get apiservices | grep metrics
kubectl -n kube-system get deploy metrics-server

v1beta1.metrics.k8s.io    kube-system/metrics-server   False (MissingEndpoints)   2m

False (MissingEndpoints) means the API is registered but has no healthy backend. Install or fix metrics-server.

2. metrics-server running but failing to reach kubelets

metrics-server exists but its pods crash-loop or can’t scrape kubelets — commonly a TLS problem (x509: cannot validate certificate because kubelet serving certs aren’t signed by the cluster CA) or a wrong --kubelet-preferred-address-types.

kubectl -n kube-system logs deploy/metrics-server | tail

E0708 scraper.go: unable to fully scrape metrics: unable to fetch metrics from node worker-1:
Get "https://10.0.1.7:10250/metrics/resource": x509: cannot validate certificate for 10.0.1.7

The fix is either kubelet serving certs signed by the cluster (--rotate-server-certificates + approved CSRs) or, in dev clusters, --kubelet-insecure-tls on metrics-server.

3. The target pods have no resource requests

A CPU-utilization HPA is a percentage of the request. If the containers have no resources.requests.cpu, utilization is undefined and the metric comes back invalid even when metrics-server works.

kubectl get deploy web -o jsonpath='{.spec.template.spec.containers[0].resources}'

{}

No request means no denominator — set resources.requests.cpu on the container.

4. Metrics not yet available for brand-new pods

Right after a pod starts, metrics-server has not scraped it yet (scrape interval is ~15s, plus startup). During that window the HPA legitimately reports <unknown>; it should clear within a minute. Persisting beyond that points at causes 1–3.

5. Custom/external metrics adapter missing or misconfigured

For an HPA on a custom or external metric, the error names custom.metrics.k8s.io or external.metrics.k8s.io. That API is served by an adapter (Prometheus Adapter, KEDA’s metrics adapter, a cloud adapter) — if it’s absent, unhealthy, or the metric query returns no series, the HPA can’t fetch it.

kubectl get apiservices | grep -E 'custom|external'
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | head

6. API aggregation layer broken

The metrics APIs are aggregated APIs; if the apiserver’s aggregation layer can’t reach the adapter Service (network policy, wrong Service, expired requestheader certs), every metrics query fails regardless of adapter health.

kubectl get apiservice v1beta1.metrics.k8s.io -o yaml | grep -A5 status

Diagnostic Workflow

Step 1: Confirm what the HPA sees

kubectl get hpa
kubectl describe hpa <name>

<unknown> targets plus FailedGetResourceMetric confirms a metrics-fetch problem (not a scaling-policy problem). Note whether the message says pods.metrics.k8s.io (resource metrics) or custom/external.metrics.k8s.io.

Step 2: Check the metrics APIService is Available

kubectl get apiservices | grep metrics

Look for True in the AVAILABLE column. False (MissingEndpoints) or False (FailedDiscoveryCheck) tells you the backing adapter is the problem.

Step 3: Test the resource metrics path directly

kubectl top nodes
kubectl top pods -n <ns>
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/<ns>/pods" | head

If kubectl top fails the same way, the issue is metrics-server, not the HPA config.

Step 4: Inspect metrics-server (or the adapter) health

kubectl -n kube-system get pods -l k8s-app=metrics-server
kubectl -n kube-system logs deploy/metrics-server --tail=30

Scrape/TLS errors here explain an empty resource metrics API.

Step 5: Verify the workload has requests

kubectl get deploy <name> -o jsonpath='{.spec.template.spec.containers[*].resources.requests}'

An empty result on a CPU-utilization HPA is the cause — no request, no percentage.

Step 6: For custom metrics, query the custom API

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq '.resources[].name' | head

An error or missing metric name means the adapter isn’t exposing what the HPA references.

Example Root Cause Analysis

A new web deployment has an HPA that never scales; TARGETS shows <unknown>/70%.

kubectl describe hpa web-hpa | grep -A2 FailedGetResourceMetric

Warning  FailedGetResourceMetric  failed to get cpu utilization:
unable to get metrics for resource cpu: no metrics returned from resource metrics API

First, is metrics-server even serving?

kubectl top pods -l app=web

NAME                   CPU(cores)   MEMORY(bytes)
web-6d9f7c4b8b-4kd2n   6m           40Mi

kubectl top works — so metrics-server is healthy and the resource metrics API returns data. That points at the workload, not the metrics stack. Check requests:

kubectl get deploy web -o jsonpath='{.spec.template.spec.containers[0].resources}'

{"limits":{"memory":"256Mi"}}

There is a memory limit but no CPU request. A CPU-utilization HPA computes usage as a percentage of the request; with no request there is no denominator, so the metric is invalid and the HPA reports <unknown>.

Fix — add a CPU request:

kubectl set resources deployment web --requests=cpu=100m

Within a scrape cycle the HPA populates:

NAME       REFERENCE        TARGETS   MINPODS   MAXPODS   REPLICAS
web-hpa    Deployment/web   6%/70%    2         10        2

The HPA now has a usable ratio and will scale when CPU crosses the target.

Prevention Best Practices

• Always set resources.requests.cpu (and memory for memory-based HPAs) on any workload with a utilization HPA — the target percentage is meaningless without a request.
• Install metrics-server as a core cluster add-on and monitor the v1beta1.metrics.k8s.io APIService for Available=True.
• Alert on HPA ScalingActive=False / FailedGetResourceMetric so a broken metrics pipeline is caught before a traffic spike, not during one.
• For custom/external metrics, health-check the adapter’s APIService and validate the exact metric name the HPA references exists in the API.
• After cluster upgrades, re-verify metrics-server kubelet TLS (serving cert rotation) — a common regression that silently empties the metrics API.
• Give new deployments a brief grace period; a transient <unknown> right after rollout is normal until the first scrape completes. See more in Kubernetes & Helm guides.

Quick Command Reference

# What the HPA sees
kubectl get hpa
kubectl describe hpa <name>

# Is the metrics API registered and Available?
kubectl get apiservices | grep -E 'metrics'

# Does the resource metrics path return data?
kubectl top nodes
kubectl top pods -n <ns>
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/<ns>/pods" | head

# metrics-server health and scrape errors
kubectl -n kube-system get pods -l k8s-app=metrics-server
kubectl -n kube-system logs deploy/metrics-server --tail=30

# Does the workload have requests?
kubectl get deploy <name> -o jsonpath='{.spec.template.spec.containers[*].resources.requests}'
kubectl set resources deployment <name> --requests=cpu=100m

# Custom metrics API (for non-resource HPAs)
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq '.resources[].name'

Conclusion

unable to fetch metrics from resource metrics API means the HPA cannot read the metric it scales on, so it freezes at the current replica count and shows <unknown>. The usual root causes:

1. metrics-server is not installed, so metrics.k8s.io has no backend.
2. metrics-server runs but can’t scrape kubelets (usually a TLS/cert problem).
3. The target pods have no resources.requests.cpu, so utilization is undefined.
4. The pods are brand new and haven’t been scraped yet (transient, self-clears).
5. A custom/external metrics adapter is missing or returns no series.
6. The API aggregation layer can’t reach the adapter Service.

Start by checking whether the metrics APIService is Available and whether kubectl top works — that one test splits “metrics stack broken” from “workload misconfigured.” For turning HPA and metrics failures into a ranked fix list, the free incident assistant can help.