☁️ OpenStack Error: 'Cannot fit instance NUMA topology' NUMA/CPU Pinning Scheduling Failure

Exact Error Message

$ openstack server show pinned-vm-01 -c fault -f value
{'code': 500, 'created': '2026-07-17T09:14:22Z',
 'message': 'No valid host was found. There are not enough hosts available.'}

The real reason is in the nova-scheduler log, where the NUMATopologyFilter rejects every host:

nova.scheduler.filters.numa_topology_filter [req-...]
  (compute-03) REQ: numa_topology, host has no available NUMA topology that
  can fit the requested instance NUMA topology, cannot fit instance topology
nova.virt.hardware [req-...] Requested instance NUMA topology cannot fit the
  given host NUMA topology

What It Means

When a flavor requests dedicated CPUs (hw:cpu_policy=dedicated), a fixed NUMA layout (hw:numa_nodes), or hugepages (hw:mem_page_size), Nova must place the instance on a compute host that has enough free pinned CPUs and memory within a single NUMA node (or the requested number of nodes). The NUMATopologyFilter walks each host and rejects any that cannot satisfy the layout, so if no host fits you get No valid host was found even when total free CPU/RAM looks plentiful.

The trap is that capacity is per-NUMA-node, not per-host. A host with 32 free vCPUs spread as 16+16 across two nodes cannot place a 20-pinned-CPU single-node instance. This is a placement/topology mismatch, not a hard resource shortage.

Common Causes

• The flavor requests more pinned CPUs than any single host NUMA node has free.
• Hugepages (hw:mem_page_size=1GB) are exhausted or not pre-allocated on the target nodes.
• cpu_dedicated_set / vcpu_pin_set in nova.conf is too small or misaligned with the socket layout.
• Existing pinned instances have fragmented the NUMA nodes, leaving no contiguous free set.
• The flavor mixes shared and dedicated CPUs incorrectly, or requests hw:numa_nodes larger than the host has.

Diagnostic Commands

Inspect the flavor’s NUMA and pinning extra specs:

openstack flavor show m1.pinned.large -c properties -f value

Look at the physical NUMA layout and free memory per node on the compute host:

numactl --hardware

Check how many CPUs Nova is allowed to pin and how many are already claimed:

sudo grep -E 'cpu_dedicated_set|vcpu_pin_set|cpu_shared_set' /etc/nova/nova.conf
sudo virsh list --all

Confirm hugepage availability per node if the flavor uses them:

grep -H '' /sys/devices/system/node/node*/hugepages/hugepages-1048576kB/*_hugepages

See exactly why the scheduler filtered every host:

sudo grep -i 'numa' /var/log/nova/nova-scheduler.log | tail -n 30

Step-by-Step Resolution

1. Compare the flavor’s request against real host capacity. If the flavor asks for more pinned CPUs than one node has free, that is your mismatch:

openstack flavor show m1.pinned.large -f value -c vcpus -c properties
numactl --hardware | grep -E 'node [0-9]+ cpus|free'

2. If the request is larger than one node, spread it across nodes explicitly instead of forcing a single node:

openstack flavor set m1.pinned.large --property hw:numa_nodes=2

3. If pinned CPUs are simply fragmented, free capacity by migrating a pinned instance off a busy host:

openstack server migrate --live-migration --host compute-05 other-pinned-vm

4. If cpu_dedicated_set is undersized or crosses sockets awkwardly, widen it to a clean per-socket range and restart nova-compute:

sudo crudini --set /etc/nova/nova.conf compute cpu_dedicated_set 4-15,20-31
sudo systemctl restart nova-compute

5. For hugepage flavors, pre-allocate 1G pages on the node and confirm they are free:

sudo sysctl -w vm.nr_hugepages=0   # verify current, then set per-node in grub/kernel cmdline
grep -H '' /sys/devices/system/node/node*/hugepages/hugepages-1048576kB/free_hugepages

6. Rebuild or re-launch the instance and confirm it schedules:

openstack server create --flavor m1.pinned.large --image ubuntu-22.04 \
  --network tenant-net pinned-vm-01
openstack server show pinned-vm-01 -c status -f value

Prevention

• Size pinned flavors to fit inside a single host NUMA node whenever possible; only request multiple hw:numa_nodes deliberately.
• Reserve a clean, socket-aligned cpu_dedicated_set and keep host CPUs for the OS out of it via isolcpus/cpu_shared_set.
• Pre-allocate hugepages at boot on hosts that serve hugepage flavors, and track free pages per node.
• Use host aggregates and the pci/aggregate filters to steer pinned workloads onto dedicated NUMA-tuned hosts.
• Monitor NUMA fragmentation; drain and repack pinned instances before nodes become unplaceable.

Related Errors

• No valid host was found. There are not enough hosts available. — the generic scheduler message that NUMA failures surface as.
• PCI device not found — a co-located SR-IOV/PCI passthrough failure on the same pinned flavors.
• Insufficient compute resources — a genuine vCPU/RAM shortage rather than a topology mismatch.
• Instance requires hugepages but none are available — a hugepage-specific variant of this NUMA error.

Frequently Asked Questions

Why does it fail when the host has plenty of free CPUs? Pinned capacity is counted per NUMA node, not per host. Free CPUs split across two nodes cannot satisfy a single-node request; check numactl --hardware to see the real per-node free count.

How do I let a large instance span two NUMA nodes? Set hw:numa_nodes=2 on the flavor so Nova splits the topology across nodes, but expect a memory-latency cost for cross-node access.

What is cpu_dedicated_set and how do I size it? It is the pool of host CPUs Nova may pin guests to. Align it to physical socket ranges and exclude CPUs the host OS needs. For generating tuned flavor and nova.conf snippets, the OpenStack prompts library can help.

Do hugepages cause this error too? Yes. If a flavor sets hw:mem_page_size and the node has no free hugepages of that size, the NUMA filter rejects the host. Pre-allocate pages per node.

Can live migration fix fragmentation? Yes — migrating pinned instances to repack nodes frees contiguous CPU sets so large pinned flavors can schedule again. For more Nova scheduling fixes, see the OpenStack guides.