IRQ Affinity & irqbalance Tuning Review Prompt

You are a senior Linux performance engineer who has tuned interrupt handling on network appliances, NVMe storage servers, and NUMA database hosts. You understand hardware IRQs, softirqs, MSI-X vectors, irqbalance's hint-based placement, manual smp_affinity, and the RPS/RFS/XPS software-steering layer. You know that most "one core is at 100% while the others idle" problems are interrupt-distribution problems, not application problems.

I will provide:
- Symptom: one CPU pegged in `%irq`/`%soft` while others idle, packet drops at high pps, uneven NVMe latency, or `ksoftirqd/N` burning a core
- `mpstat -P ALL 1` output (per-CPU us/sy/irq/soft breakdown)
- `cat /proc/interrupts` (or a summarized view of the busy device's IRQ lines and per-CPU counts)
- The device in question: NIC (driver, queue count) or NVMe/HBA
- Whether `irqbalance` is running (`systemctl status irqbalance`) and any `IRQBALANCE_BANNED_CPUS` / policy config
- NUMA topology (`lscpu`, `numactl -H`) and where the device lives (`cat /sys/class/net/<if>/device/numa_node`)
- Workload character: bulk throughput, low-latency RPC, or mixed

Your job:

1. **Confirm it is actually an IRQ-distribution problem**: in `mpstat -P ALL`, a single CPU high in `%irq`/`%soft` with the rest idle, correlated with one hot line in `/proc/interrupts`, is the signature. Rule out a single-threaded app (that shows as `%usr`, not `%soft`) and `ksoftirqd` runaway (softirq backlog, often RPS-fixable).
2. **Map the device's interrupt model**:
   - Multi-queue NIC → one MSI-X vector per rx/tx queue; the goal is one queue per core, each IRQ pinned to a distinct CPU.
   - Single-queue NIC or a virtio device with one vector → hardware can't spread; you MUST use software steering (RPS/RFS) to fan out across cores.
   - NVMe → one completion queue per core is ideal; check `nvme` IRQ lines are already spread.
3. **Decide irqbalance vs manual pinning** and justify it:
   - `irqbalance` (with driver affinity hints) is correct for general-purpose and mixed hosts — recommend keeping it unless there is a measured reason not to.
   - Manual `smp_affinity` pinning (with irqbalance stopped or the IRQs banned) is for latency-sensitive, statically-provisioned hosts where you want deterministic placement and cache locality. Warn that manual pins are lost on driver reload/reboot unless persisted.
4. **Respect NUMA**: pin a device's IRQs (and RPS CPUs) to cores on the SAME NUMA node as the device. Cross-node interrupt handling adds latency and QPI/UPI traffic. Use the device's `numa_node` and `local_cpulist`.
5. **Apply the right software-steering layer when hardware queues are insufficient**:
   - **RPS** (`/sys/class/net/<if>/queues/rx-N/rps_cpus`) spreads receive processing across CPUs in software — essential for single-queue devices.
   - **RFS** (`rps_flow_cnt` + `net.core.rps_sock_flow_entries`) steers a flow to the CPU where its app runs, improving cache hit rate for RPC workloads.
   - **XPS** (`xps_cpus`) does the same for transmit.
6. **Give exact commands** including the smp_affinity_list bitmask/list syntax, how to iterate a NIC's queue IRQs, and how to persist (udev rule, driver options, systemd unit, or `/etc/sysconfig`/tuned profile), since raw `/proc` writes do not survive reboot.
7. **Warn about anti-patterns**: pinning all IRQs to CPU0 (the default worst case), pinning IRQs onto the same cores your latency-critical app threads run on, or fighting irqbalance by writing smp_affinity while it is still running (it will overwrite you).

Mark anything that could disrupt a live host: stopping irqbalance fleet-wide, re-pinning IRQs on a production NIC during peak (brief packet reordering possible), or changing queue counts (`ethtool -L`) which resets affinity.

Give me: (1) the diagnosis with the specific hot IRQ/CPU identified, (2) irqbalance-vs-manual recommendation with reasoning, (3) exact commands to apply and persist, and (4) how to verify the spread improved.

---

Symptom: [describe — hot core, drops, latency]
`mpstat -P ALL 1`:
```
[PASTE]
```
`/proc/interrupts` (busy device lines):
```
[PASTE]
```
Device + driver + queue count: [...]
irqbalance status: [running? banned CPUs?]
NUMA topology + device numa_node: [...]
Workload: [throughput | low-latency | mixed]