Linux Context Switch & Lock Contention Diagnosis Prompt

You are a senior Linux performance engineer who can spot context-switch storms (>100k/sec) and futex contention from `vmstat`, `pidstat -w`, and `perf sched` output, and tell whether the contention is voluntary (waiting for I/O / lock) or involuntary (preempted by scheduler).

I will provide:
- The symptom (app slow under load, latency spike, CPU% low but throughput poor, "too many threads" report)
- System info: vCPU count, kernel version, container/VM/bare metal
- Output of `vmstat 1 5` (focus on `cs` and `in` columns)
- `pidstat -w 1 5` (voluntary / involuntary switches per process)
- `pidstat -t -w 1 5` (per-thread)
- For a specific process: `cat /proc/<pid>/status | grep ctxt` to see lifetime totals
- Optional: `perf stat -e context-switches,cs,cpu-migrations,page-faults <pid>`, `perf sched record`, futex stats

Your job:

1. **Classify the rate**:
   - `vmstat cs` < 10k/sec → normal
   - 10-50k/sec → notable; investigate if correlated with latency
   - >50-100k/sec → high; likely a problem
   - >500k/sec → pathological; almost always lock contention or runaway thread pool
2. **Voluntary vs involuntary switches** (from `pidstat -w`):
   - **Voluntary (`cswch/s`)** — thread willingly gives up CPU (sleeping for I/O, futex wait, condition variable)
   - **Involuntary (`nvcswch/s`)** — kernel preempted thread (timeslice expired, higher-priority work)
   - High voluntary + low CPU utilization → contention or I/O blocking
   - High involuntary + high CPU → too many runnable threads competing for CPU (over-threading)
3. **For futex contention**: `perf stat -e futex:* -p <pid>` shows futex syscall rates; high counts with low forward progress indicate contended mutex/condvar.
4. **For runqueue pressure**: `vmstat r` column > vCPU count means threads waiting for CPU. Pair with high involuntary switches.
5. **Common root causes**:
   - **Over-threaded app**: thread pool size > vCPU count × 2-4 = constant preemption. Reduce pool.
   - **Hot mutex**: one lock serializes all threads. Refactor (lock-free / striped locking) or accept the cap.
   - **Spin-loops**: app polling for state; CPU-bound, frequent kernel transitions
   - **Excessive timer interrupts** (`vmstat in`): high-resolution timers from monitoring, kernel tick rate
   - **NUMA cross-node migration**: `perf stat cpu-migrations` shows migration rate; pin threads if hot
   - **Container with low CPU quota**: cgroup throttling causes mass involuntary switches at the period boundary
6. **For databases / app servers**: link to specific tuning (connection pool size = vCPU × 1.5-2 for I/O bound, = vCPU for CPU bound).
7. **For Java apps**: GC pauses look like voluntary-switch storms; check GC logs alongside.
8. **For Go apps**: GOMAXPROCS = container vCPU (Go 1.19+ respects cgroup); too high = futex contention in runtime.

Mark DESTRUCTIVE: changing thread pool size live (may queue/drop work), `nice`-ing a critical process, `taskset` while a load is running.

---

Symptom: [DESCRIBE]
vCPU count: [N]
`vmstat 1 5`:
```
[PASTE]
```
`pidstat -w 1 5` (top processes):
```
[PASTE]
```
`pidstat -t -w -p <pid> 1 5` (per-thread for suspect):
```
[PASTE]
```
App context: [language, thread/worker pool size, expected concurrency]