🚨 OpenStack Error Guide: 'Filtering removed all hosts' Nova Filter Chain

Overview

Filtering removed all hosts for the request is logged by nova-scheduler when the filter chain — the in-process Python filters that run after Placement returns candidates — eliminates the last host. Unlike “No valid host was found” (which can be raised by an empty Placement result), this message is unambiguous: Placement gave Nova some hosts, but a specific filter rejected them all. The companion line Filter <Name> returned 0 hosts names the exact culprit.

You will see this pair in the nova-scheduler log:

DEBUG nova.filters [req-...] Filter NUMATopologyFilter returned 0 hosts
INFO nova.filters [req-...] Filtering removed all hosts for the request with instance ID 'a1b2...'. Filter results: ['ComputeFilter: (start: 4, end: 4)', 'ImagePropertiesFilter: (start: 4, end: 4)', 'NUMATopologyFilter: (start: 4, end: 0)']

The instance then goes to ERROR with a NoValidHost fault. It is a per-request decision driven by the flavor’s extra_specs, the image’s properties, the host’s reported capabilities, and your aggregate layout — so the same flavor can pass on one image and fail on another.

Symptoms

• nova-scheduler logs Filter <Name> returned 0 hosts followed by Filtering removed all hosts.
• Instance fault is No valid host was found, but openstack allocation candidate list does return rows (Placement is fine; a Nova filter is not).
• The Filter results line shows a single filter dropping the count from N to 0.

openstack server show gpu-01 -c fault -f value

{'code': 500, 'message': 'No valid host was found. There are not enough hosts available.', ...}

# Placement says there IS capacity, so the elimination is a Nova filter:
openstack allocation candidate list --resource VCPU=4 --resource MEMORY_MB=8192 -c '#allocations' -f value | head

4

Common Root Causes

1. Understanding the filter chain

The scheduler runs each enabled filter in order. Every filter takes the current host list and returns a (possibly smaller) list. The (start: N, end: M) annotation in the Filter results line is how many hosts entered and left each filter. The filter where end becomes 0 is the one that killed the request.

grep -E '^(enabled_filters|scheduler_default_filters)' /etc/nova/nova.conf

enabled_filters = ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,NUMATopologyFilter,PciPassthroughFilter,AggregateMultiTenancyIsolation,ServerGroupAntiAffinityFilter

Read the chain top to bottom; the end: 0 filter is your fix target.

2. NUMATopologyFilter

Flavors with hw:numa_nodes, hw:cpu_policy=dedicated, or hugepages (hw:mem_page_size) require a host that can fit the requested NUMA layout in free, pinned CPUs and same-node memory. Fragmentation or no pinning support zeroes the list.

openstack flavor show m1.numa -c properties -f value

{'hw:numa_nodes': '2', 'hw:cpu_policy': 'dedicated', 'hw:mem_page_size': '1GB'}

DEBUG nova.virt.hardware [req-...] No host NUMA topology while instance has NUMA topology... (host has no free pinned CPUs on a node)
DEBUG nova.filters Filter NUMATopologyFilter returned 0 hosts

If hosts lack cpu_dedicated_set / hugepages or are already packed, this filter wins.

3. PciPassthroughFilter

Flavors requesting PCI aliases (pci_passthrough:alias=gpu:1) only pass on hosts that advertise free matching devices via pci_passthrough_whitelist. No free device, no host.

openstack flavor show g1.gpu -c properties -f value
grep -E '^(alias|passthrough_whitelist|device_spec)' /etc/nova/nova.conf

{'pci_passthrough:alias': 'a100:1'}

DEBUG nova.filters Filter PciPassthroughFilter returned 0 hosts

Check that the host actually has the device free with openstack hypervisor show <HOST> -c pci_devices (or query Placement device RPs).

4. AggregateMultiTenancyIsolation

This filter restricts hosts in an aggregate tagged filter_tenant_id to specific projects. If your aggregate’s filter_tenant_id does not include the booting project, those hosts are removed — and if the only spare capacity is in that aggregate, you get zero.

openstack aggregate show isolated-pool -c properties -f value

{'filter_tenant_id': '5f2a9c...,8b71d4...'}

If the request’s project is not in that list, the filter drops the aggregate’s hosts.

5. ImagePropertiesFilter

The image can carry properties (hw_architecture, hw_vif_model, hypervisor_type, os_distro) that a host must match. An image built for aarch64 cannot land on x86 hosts; the filter removes them.

openstack image show ubuntu-arm64 -c properties -f value

{'hw_architecture': 'aarch64', 'hypervisor_type': 'qemu'}

DEBUG nova.filters Filter ImagePropertiesFilter returned 0 hosts

6. Enabling scheduler debug to see per-filter counts

By default the Filter results line is at INFO, but the individual returned 0 hosts lines and the per-host reasons are at DEBUG. Turn on debug on the scheduler to see why each host failed a filter.

# Kolla-Ansible: set debug in the scheduler config, then restart
grep -E '^debug' /etc/kolla/nova-scheduler/nova.conf
docker restart nova_scheduler
# Traditional packages
sudo crudini --set /etc/nova/nova.conf DEFAULT debug True
sudo systemctl restart nova-scheduler

Remember to revert debug = False afterwards — the scheduler is chatty.

Diagnostic Workflow

Step 1: Confirm Placement is not the problem

openstack allocation candidate list \
  --resource VCPU=4 --resource MEMORY_MB=8192 --resource DISK_GB=40

If this returns rows, Placement has capacity and the elimination is purely in the Nova filter chain — proceed.

Step 2: Grab the Filter results line

# Kolla-Ansible
docker logs nova_scheduler 2>&1 | grep -i "Filtering removed all hosts" | tail -3
# Traditional packages
sudo grep -i "Filtering removed all hosts" /var/log/nova/nova-scheduler.log | tail -3

The (start: N, end: 0) annotation tells you which filter to investigate. Note its name.

Step 3: Enable debug and re-trigger to capture per-host reasons

sudo crudini --set /etc/nova/nova.conf DEFAULT debug True
sudo systemctl restart nova-scheduler   # or: docker restart nova_scheduler
openstack server create --flavor <FLAVOR> --image <IMAGE> --network <NET> retry-01
docker logs nova_scheduler 2>&1 | grep -iE "returned 0 hosts|No host NUMA|did not match|insufficient" | tail -20

Step 4: Inspect the offending filter’s inputs

# NUMA / dedicated CPU
openstack flavor show <FLAVOR> -c properties -f value
# PCI
grep -E '^(alias|device_spec|passthrough_whitelist)' /etc/nova/nova.conf
# Multi-tenancy isolation
openstack aggregate show <AGGREGATE> -c properties -f value
# Image properties
openstack image show <IMAGE> -c properties -f value

Match the requirement to what the hosts actually advertise.

Step 5: Fix, then revert debug and retry

After correcting the flavor/image/aggregate or freeing the required resource:

sudo crudini --set /etc/nova/nova.conf DEFAULT debug False
sudo systemctl restart nova-scheduler   # or: docker restart nova_scheduler
openstack server create --flavor <FLAVOR> --image <IMAGE> --network <NET> final-01

Example Root Cause Analysis

A GPU instance infer-02 fails with No valid host was found, yet openstack allocation candidate list --resource VCPU=8 --resource MEMORY_MB=16384 returns four candidates — so Placement is healthy.

The scheduler log narrows it down:

INFO nova.filters [req-...] Filtering removed all hosts for the request with instance ID 'c4d5...'. Filter results: ['ComputeFilter: (start: 4, end: 4)', 'PciPassthroughFilter: (start: 4, end: 4)', 'AggregateMultiTenancyIsolation: (start: 4, end: 0)']

PciPassthroughFilter passed (the GPU hosts have free A100s), but AggregateMultiTenancyIsolation dropped all four. Inspecting the GPU aggregate:

openstack aggregate show gpu-pool -c properties -f value

{'filter_tenant_id': '5f2a9c2b...'}

The booting project is 8b71d4e0..., which is not in filter_tenant_id. The aggregate was locked to a single tenant and the new project was never added.

Fix: append the project to the aggregate’s allowed tenants:

openstack aggregate set --property filter_tenant_id="5f2a9c2b...,8b71d4e0..." gpu-pool
openstack server create --flavor g1.gpu --image ubuntu --network internal infer-02

The hosts now survive the isolation filter and the instance reaches ACTIVE.

Prevention Best Practices

• Treat Filter <Name> returned 0 hosts as a routed alert: parse nova-scheduler logs and surface the offending filter name so on-call jumps straight to the right config.
• Keep specialized flavors (NUMA, PCI, dedicated CPU) tested against a known-good host in CI; a flavor that no host can satisfy is a latent outage.
• Document every filter_tenant_id aggregate and update it as part of project onboarding — multi-tenancy isolation failures are invisible until someone in a new project boots there.
• Build images with explicit, correct hw_architecture / hypervisor_type properties so ImagePropertiesFilter never silently excludes the whole fleet.
• Monitor free pinned CPUs and hugepages per NUMA node so you defragment before NUMATopologyFilter starts returning zero.
• For ad-hoc triage, the free incident assistant can read a Filter results line and name the filter plus likely fix. See more in OpenStack guides.

Quick Command Reference

# Prove Placement has capacity (so it's a Nova filter)
openstack allocation candidate list --resource VCPU=4 --resource MEMORY_MB=8192 --resource DISK_GB=40

# Which filter zeroed the list?
docker logs nova_scheduler 2>&1 | grep -i "Filtering removed all hosts" | tail -3
sudo grep -i "Filtering removed all hosts" /var/log/nova/nova-scheduler.log | tail -3

# Enable scheduler debug for per-host reasons (revert after!)
sudo crudini --set /etc/nova/nova.conf DEFAULT debug True
sudo systemctl restart nova-scheduler   # or docker restart nova_scheduler
docker logs nova_scheduler 2>&1 | grep -iE "returned 0 hosts|did not match|No host NUMA" | tail -20

# Inspect the offending filter's inputs
openstack flavor show <FLAVOR> -c properties -f value
openstack image show <IMAGE> -c properties -f value
openstack aggregate show <AGGREGATE> -c properties -f value
grep -E '^(enabled_filters|alias|device_spec)' /etc/nova/nova.conf

# Fix a multi-tenancy isolation aggregate
openstack aggregate set --property filter_tenant_id="<id1>,<id2>" <AGGREGATE>

# Revert debug
sudo crudini --set /etc/nova/nova.conf DEFAULT debug False
sudo systemctl restart nova-scheduler

Conclusion

Filtering removed all hosts means Placement supplied candidates but a Nova filter rejected them all — and the (start: N, end: 0) annotation names which one. The usual root causes:

1. NUMATopologyFilter — no host fits the requested NUMA/pinning/hugepage layout.
2. PciPassthroughFilter — no host has a free matching PCI/GPU device.
3. AggregateMultiTenancyIsolation — the booting project is not in the aggregate’s filter_tenant_id.
4. ImagePropertiesFilter — the image’s architecture/hypervisor properties match no host.
5. A misordered or overly strict enabled_filters chain.

Read the Filter results line first, enable scheduler debug to see the per-host reason, then fix the flavor, image, aggregate, or capacity that the named filter is rejecting.