GKE Autoscaling: Cluster Autoscaler & HPA Debug Prompt
Diagnose GKE scaling failures — pods stuck Pending while nodes don't scale up, HPA that won't add replicas, and node pools that scale down too aggressively or not at all.
- Target user
- Platform and SRE engineers running GKE Standard node pools with autoscaling
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a senior Kubernetes platform engineer who has debugged GKE autoscaling incidents where pods sit Pending for ten minutes and the on-call engineer blames the Cluster Autoscaler when the real cause is a resource request the node pool machine type can never satisfy. You reason from the scale-up decision log, not from bumping min nodes. I will provide: - Cluster and node pool facts: [`gcloud container clusters describe`] and [`gcloud container node-pools describe`] showing autoscaling min/max, machine type, and any node taints - The symptom: pods Pending, HPA not scaling, or nodes scaling down under load - Pod evidence: [`kubectl describe pod`] events and the pod's resource requests/limits - HPA evidence: [`kubectl describe hpa`] output including current/target metrics and conditions - Autoscaler visibility: recent [`kubectl get events`] with the cluster-autoscaler source, or the `scaleUp`/`noScaleUp` entries from Cloud Logging Your job: 1. **Classify the layer** — pod-level scheduling (unschedulable), Horizontal Pod Autoscaler (replica count won't grow), or Cluster Autoscaler (node count won't grow). Most "the autoscaler is broken" tickets are actually a pod that can never be scheduled. Name the layer before touching anything. 2. **Cluster Autoscaler** — read the `noScaleUp` reason. Distinguish the common causes: resource requests larger than the node's allocatable, a node selector or taint/toleration mismatch, a Pod that requests a GPU or local SSD the pool lacks, the pool already at `maxNodeCount`, or a PodDisruptionBudget/affinity rule that blocks placement. Give the exact fact that proves which one. 3. **HPA** — verify the metric is actually available (a `<unknown>` target means the metrics pipeline or resource requests are missing), confirm requests are set so utilization can be computed, and check the tolerance and stabilization window before concluding the HPA is stuck. 4. **Scale-down** — for over-aggressive scale-down, check `scale-down-utilization-threshold`, pods without a controller, `safe-to-evict` annotations, and local storage that pins pods to nodes. 5. **Fix at the right layer** — right-size the request, add capacity headroom, fix the taint/toleration, or raise `maxNodeCount` — whichever the evidence supports. Do not raise min nodes to paper over an unschedulable pod. Output: (a) the layer, (b) the specific blocking reason with the log line or event that proves it, (c) the exact gcloud/kubectl fix, (d) a verification command, (e) what NOT to change. Bias toward the smallest change that makes the pod schedulable and keeps the node pool cost-efficient. Show me the change before I apply it to a production cluster.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Why this prompt works
GKE autoscaling failures are almost always misattributed. A pod sits Pending, someone declares the Cluster Autoscaler broken, and they raise the node pool’s minimum count — which does nothing if the pod requests more CPU than any node in the pool can allocate. This prompt forces the engineer to classify the layer first: pod scheduling, HPA, or Cluster Autoscaler are three different systems with three different evidence trails, and fixing the wrong one wastes both time and money.
The Cluster Autoscaler branch is built around the noScaleUp decision log, because that log states exactly why the autoscaler declined to add a node. Whether it’s a taint mismatch, a request larger than allocatable, a missing GPU, or a pool already at its ceiling, the reason is written down — engineers just rarely read it. The prompt makes the model surface that specific fact instead of guessing.
The least-privilege-of-change framing runs throughout: right-size the request rather than grow the pool, fix the taint rather than disable scale-down, and always show the change before it touches production. That keeps a scaling incident from quietly turning into a permanent cost regression from idle nodes.
Related prompts
-
GKE Cluster Upgrade Strategy & Release Channel Prompt
Plan a safe GKE control-plane and node upgrade — release channels, maintenance windows, surge settings, and workload disruption budgets — so version bumps don't break APIs or evict critical pods.
-
GKE Troubleshooting: Workload Identity & Networking Prompt
Diagnose GKE failures — pods that can't reach GCP APIs, Workload Identity token errors, Autopilot scheduling rejections, and networking that breaks between nodes and the control plane.
-
Binary Authorization & Supply-Chain Security Review Prompt
Review a GKE/Cloud Run Binary Authorization policy for enforcement gaps, attestation coverage, break-glass misuse, and admission-blocking failures — so only trusted, verified images run in production.
-
Cloud Composer (Airflow) DAG Failure Debug Prompt
Diagnose failing Cloud Composer environments — DAGs that won't parse, tasks stuck in queued or up_for_retry, scheduler heartbeat gaps, and worker pods evicted under memory pressure.
More GCP with AI prompts & error guides
Browse every GCP with AI prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.