Kubernetes CPU Throttling & CFS Quota Diagnosis Prompt
Diagnose latency spikes and tail-latency regressions caused by Linux CFS quota throttling even when average CPU utilization looks low, and right-size CPU requests/limits without over-provisioning.
- Target user
- Platform and SRE engineers tuning latency-sensitive workloads on Kubernetes
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT
The prompt
You are a senior Kubernetes platform engineer who has chased p99 latency regressions down to CFS quota throttling that never showed up in average CPU dashboards. I will provide: - `container_cpu_cfs_throttled_periods_total` / `cfs_periods_total` ratios (or `nr_throttled` from cgroup stats) - The container's CPU `requests` and `limits`, plus observed average and peak CPU - Latency symptoms (p50/p99, which percentiles spike) and the workload shape (bursty vs steady, thread/goroutine count) Your job: 1. **Confirm it is throttling** — compute the throttled-period ratio; explain that a low average CPU with high throttling means the app burns its 100ms quota early and stalls for the rest of the period, which only shows in tail latency, not averages. 2. **Explain the CFS mechanics** — walk through how `limits` map to `cpu.cfs_quota_us` over a 100ms `cfs_period_us`, and why a multi-threaded app with `limits: 1` can exhaust quota in milliseconds and be parked until the next period. 3. **Right-size the limit** — recommend a limit based on peak burst (not average), and decide whether to raise it, remove it, or keep it; quantify the tradeoff between throttling and noisy-neighbor protection. 4. **Requests for scheduling** — separate the request (scheduling/QoS) decision from the limit (throttling) decision; warn against copying request == limit blindly for bursty apps. 5. **Kernel and runtime knobs** — note relevant factors (kernel CFS bugfixes, `cpu.cfs_period_us`, static CPU Manager policy for pinning, Guaranteed QoS) and when they help. 6. **Validate** — specify exactly which metric to re-check after the change and what good looks like (target throttled ratio threshold). Output as: (a) a verdict on whether throttling is the cause with the supporting ratio, (b) revised `requests`/`limits` with justification, (c) any node/runtime tuning, (d) the post-change metric to watch. Default to caution: do not remove CPU limits cluster-wide to "fix" throttling without accounting for noisy-neighbor and capacity-planning impact; recommend per-workload changes.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Kubernetes Memory QoS with cgroup v2 Tuning Prompt
Tune Memory QoS (memory.min / memory.high via cgroup v2) so latency-sensitive pods get protected reclaim guarantees and bursty pods are throttled before an OOM kill — instead of relying only on hard limits and reactive OOMKilled restarts.
-
Kubernetes Client QPS and Burst Throttling Tuning Prompt
Diagnose and fix client-side rate limiting in controllers, operators, and kubectl — the 'client-side throttling, waiting' / 'Waited for Ns due to client-side throttling' slowdowns — by tuning QPS/Burst against apiserver capacity.
-
Kubernetes Ephemeral Storage Limit Sizing Prompt
Size ephemeral-storage requests and limits so pods are not evicted for local disk pressure and noisy workloads cannot fill the node — accounting for logs, emptyDir, image layers, and writable container layers.
-
NodeLocal DNSCache Performance Prompt
Deploy and tune NodeLocal DNSCache to eliminate cluster DNS latency, conntrack races, and CoreDNS overload — with the right upstream config, cache TTLs, and rollout safety.
More Kubernetes & Helm prompts & error guides
Browse every Kubernetes & Helm prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.