🚨 OpenStack Error Guide: 'No space left on device' — Fix Nova Compute Disk Exhaustion

Overview

No space left on device (errno 28, ENOSPC) is the error nova-compute and libvirt raise when a compute host’s disk fills up and there is no room to download an image, create an ephemeral/root disk, or write a running guest’s writes. Because Nova stages images in a cache and creates instance disks under /var/lib/nova/instances, a full partition blocks new spawns on that host and can pause or crash running guests.

The literal errors you will see:

libvirt.libvirtError: internal error: process exited while connecting to monitor: Could not write ...: No space left on device

ERROR nova.compute.manager [instance: ...] OSError: [Errno 28] No space left on device

It occurs during instance spawn (image fetch/convert, disk create), snapshot, or while a guest is running and its qcow2 grows. The tell is that the failure is filesystem-level on one or a few compute hosts, while hosts with free disk keep working.

Symptoms

• Instances fail to spawn on specific hosts with ENOSPC/No space left on device.
• Running guests pause (paused state) when their disk can’t grow.
• df shows a Nova or image-cache partition at 100%.

openstack server show <instance> -c "OS-EXT-SRV-ATTR:host" -c status -c fault

| status | ERROR                                                        |
| fault  | {'message': "[Errno 28] No space left on device", ...}       |

docker logs nova_compute 2>&1 | grep -iE "No space left|Errno 28|ENOSPC" | tail -3

ERROR nova.compute.manager OSError: [Errno 28] No space left on device

Common Root Causes

1. /var/lib/nova/instances partition full

This is where instance root/ephemeral disks live; when it fills, no new instance can be created there.

df -h /var/lib/nova/instances
du -sh /var/lib/nova/instances/* 2>/dev/null | sort -rh | head

/dev/mapper/nova  200G  200G   0   100% /var/lib/nova/instances

A 100%-used instances partition is the direct cause of spawn failures on that host.

2. Image cache growth

Nova caches base images under _base; without periodic cleanup it grows unbounded.

du -sh /var/lib/nova/instances/_base 2>/dev/null
docker exec nova_compute grep -E 'remove_unused_base_images|image_cache' /etc/nova/nova.conf

120G  /var/lib/nova/instances/_base

If remove_unused_base_images is disabled or the interval is long, stale base images consume the partition.

3. disk_allocation_ratio overcommit vs real capacity

Placement schedules based on an overcommit ratio; if it’s too aggressive, the scheduler places more disk than the host physically has.

docker exec nova_compute grep -E 'disk_allocation_ratio' /etc/nova/nova.conf
openstack resource provider list --resource DISK_GB=1 2>/dev/null | head

disk_allocation_ratio = 3.0

A high ratio lets Placement believe there is disk that physically isn’t there, so hosts fill before Placement thinks they’re full.

4. Orphaned disks from failed deletes / evacuations

Instances that failed to delete cleanly leave disk directories behind, silently consuming space.

ls -1 /var/lib/nova/instances/ | grep -vE '_base|locks|compute_nodes' | head
openstack server list --host <host> --all-projects -c ID -f value | sort > /tmp/known.txt
# compare directory UUIDs to /tmp/known.txt to find orphans

# a directory whose UUID is NOT in the server list is an orphan

5. Runaway logs or core dumps on the same partition

If logs, journald, or core dumps share the partition, they can fill it independently of Nova.

df -h /
du -sh /var/log/* 2>/dev/null | sort -rh | head

40G  /var/log/kolla

6. qcow2 growth exceeding flavor assumptions

Thin qcow2 disks grow toward their virtual size; many active guests writing at once can fill the host even under a modest overcommit.

qemu-img info /var/lib/nova/instances/<uuid>/disk 2>/dev/null | grep -E 'disk size|virtual size'

virtual size: 80 GiB
disk size: 78 GiB

Diagnostic Workflow

Step 1: Identify the full partition on the host

df -h | sort -k5 -rh | head
df -h /var/lib/nova/instances /var/log /

Step 2: Find what’s consuming it

du -sh /var/lib/nova/instances/* 2>/dev/null | sort -rh | head
du -sh /var/lib/nova/instances/_base 2>/dev/null

Step 3: Check overcommit vs physical capacity

docker exec nova_compute grep -E 'disk_allocation_ratio|reserved_host_disk_mb' /etc/nova/nova.conf

Step 4: Look for orphaned instance directories

openstack server list --host <host> --all-projects -c ID -f value | sort > /tmp/known.txt
ls -1 /var/lib/nova/instances/ | grep -E '^[0-9a-f-]{36}$' | sort > /tmp/dirs.txt
comm -23 /tmp/dirs.txt /tmp/known.txt   # directories with no matching instance

Step 5: Check base-image cleanup config

docker exec nova_compute grep -E 'remove_unused_base_images|remove_unused_original_minimum_age_seconds' \
  /etc/nova/nova.conf

Example Root Cause Analysis

Instances scheduled to compute-05 fail with [Errno 28] No space left on device, while other hosts are fine. On the host:

df -h /var/lib/nova/instances

/dev/mapper/nova  200G  200G   0   100% /var/lib/nova/instances

The partition is full. Breaking down usage:

du -sh /var/lib/nova/instances/_base
du -sh /var/lib/nova/instances/* 2>/dev/null | sort -rh | head -3

150G  /var/lib/nova/instances/_base

The _base image cache is 150 GB — three-quarters of the partition. The config shows cleanup was disabled:

docker exec nova_compute grep 'remove_unused_base_images' /etc/nova/nova.conf

remove_unused_base_images = False

Stale base images accumulated for months. The fix is to enable cache cleanup (and, for immediate relief, remove base images not referenced by any running instance), then confirm space is reclaimed:

# set remove_unused_base_images = True via config override, then:
kolla-ansible reconfigure -t nova
# immediate relief: nova's imagecache manager prunes unreferenced _base files on its next run
df -h /var/lib/nova/instances   # free space restored
openstack server create ...     # spawns succeed on compute-05 again

Longer term, right-size disk_allocation_ratio to real capacity and add per-host disk alerting so a host is drained before it hits 100%.

Prevention Best Practices

• Enable remove_unused_base_images and tune the minimum-age so the _base cache can’t grow unbounded — an oversized image cache is the most common cause.
• Alert on per-compute-host disk usage (instances partition, /, /var/log) well before 100%; a full host silently blocks all new spawns there.
• Set disk_allocation_ratio to match real capacity and use reserved_host_disk_mb so Placement won’t schedule a host into ENOSPC.
• Periodically reconcile instance directories against the Nova server list to catch orphaned disks from failed deletes/evacuations.
• Keep logs, journald, and core dumps off the Nova instances partition (or capped) so they can’t fill it independently.
• Account for qcow2 growth: thin disks trend toward virtual size, so modest overcommit plus many active guests can still fill a host.
• Paste the df output and Nova log into the free incident assistant to pinpoint the consuming directory, and see more OpenStack guides.

Quick Command Reference

# Which partition is full?
df -h | sort -k5 -rh | head
df -h /var/lib/nova/instances /var/log /

# What is consuming it?
du -sh /var/lib/nova/instances/* 2>/dev/null | sort -rh | head
du -sh /var/lib/nova/instances/_base 2>/dev/null

# Overcommit vs capacity
docker exec nova_compute grep -E 'disk_allocation_ratio|reserved_host_disk_mb' /etc/nova/nova.conf

# Orphaned instance directories
openstack server list --host <host> --all-projects -c ID -f value | sort > /tmp/known.txt
ls -1 /var/lib/nova/instances/ | grep -E '^[0-9a-f-]{36}$' | sort > /tmp/dirs.txt
comm -23 /tmp/dirs.txt /tmp/known.txt

# Base-image cleanup config
docker exec nova_compute grep -E 'remove_unused_base_images|remove_unused_original_minimum_age_seconds' /etc/nova/nova.conf

# Reclaim after enabling cleanup
kolla-ansible reconfigure -t nova

Conclusion

No space left on device on a compute host is a filesystem-level failure: the disk Nova writes instance and image data to is full, so spawns fail and running guests can pause. Typical root causes:

1. A full /var/lib/nova/instances partition.
2. An unbounded _base image cache with cleanup disabled.
3. Too-aggressive disk_allocation_ratio letting Placement overschedule disk.
4. Orphaned disks from failed deletes or evacuations.
5. Runaway logs/core dumps sharing the partition.
6. qcow2 growth exceeding the overcommit assumptions.

Find the full partition with df first, then du the instances directory and the _base cache — that quickly separates an image-cache problem from an overcommit or orphaned-disk problem and tells you what to reclaim.