🚨 OpenStack Error Guide: 'HTTP 507 Insufficient Storage' — fix Swift full/unmounted drives

Overview

507 Insufficient Storage is what the Swift proxy returns to a client (or logs from a backend node) when it cannot place an object because the target storage devices are full, unmounted, or otherwise unavailable. Swift writes each replica to a device chosen by the ring; if enough of those devices reject the write for lack of space or a failed mount, the proxy has nowhere to put the required number of replicas and answers 507.

The literal errors you will see:

HTTP/1.1 507 Insufficient Storage

object-server: ERROR __call__ error with PUT ... Insufficient Storage
proxy-server: ERROR with Object server <ip>:6200/d3 re: Trying to PUT: 507 Insufficient Storage

It occurs on a client PUT (uploading an object, a Glance image backed by Swift, a backup) — reads (GET) of existing objects usually still work. A 507 is Swift telling you the write path is out of usable capacity, not that the request was malformed.

Symptoms

• Uploads fail with 507 Insufficient Storage; downloads of existing objects still succeed.
• Glance image uploads (with Swift backend) or backup jobs fail intermittently as some nodes fill.
• Object-server logs on one or more storage nodes show Insufficient Storage or mount_check errors.

swift stat
curl -s -o /dev/null -w "%{http_code}\n" -X PUT \
  -H "X-Auth-Token: $TOKEN" "$STORAGE_URL/testcont/obj" --data-binary @/etc/hostname

507

docker logs swift_object_server 2>&1 | grep -iE "Insufficient Storage|mount_check|not mounted" | tail -3

object-server ERROR __call__ error with PUT /d3/... Insufficient Storage

Common Root Causes

1. Object disks are actually full

The physical devices that hold object replicas have hit (or crossed) their usable limit, so writes are refused.

docker exec swift_object_server df -h /srv/node/* 2>/dev/null

/dev/sdc   1.8T  1.8T   12G  100% /srv/node/d3

A device at ~100% cannot accept new object replicas.

2. A drive is unmounted and mount_check is on

Swift’s mount_check = true refuses to write to a device that isn’t a real mountpoint (to avoid filling the root filesystem). A drive that dropped its mount then returns 507 rather than silently writing to /.

docker exec swift_object_server cat /proc/mounts | grep /srv/node
docker exec swift_object_server grep mount_check /etc/swift/object-server.conf

mount_check = true
# ...and /srv/node/d3 is missing from /proc/mounts

3. Ring imbalance — weight concentrates data

If some devices have much higher ring weight (or fewer devices exist in a zone), they fill first while others stay empty, producing 507s even though the cluster as a whole has space.

docker exec swift_object_server swift-ring-builder /etc/swift/object.builder 2>/dev/null | head -20

Devices: id region zone ip:port ... weight ... balance
     3   1     1   <ip>:6200 ... 3000.00 ...  42.13

A high positive balance on hot devices signals uneven placement.

4. Failed / read-only disk

A disk that went read-only (filesystem errors) or failed hardware rejects writes; Swift treats it as unavailable and may return 507 when replicas can’t be placed.

docker exec swift_object_server mount | grep 'srv/node'
sudo dmesg | grep -iE 'I/O error|read-only|EXT4-fs error' | tail

/dev/sdc on /srv/node/d3 type xfs (ro,...)

5. Inodes exhausted (many small objects)

A device can be well under its byte capacity yet out of inodes if it holds huge numbers of small objects — new files can’t be created.

docker exec swift_object_server df -i /srv/node/* 2>/dev/null

/dev/sdc   ...   IUse% 100%  /srv/node/d3

6. Fallocate reserve / fallocate_reserve threshold

Swift can reserve headroom (fallocate_reserve); once free space drops below it, the object server rejects PUTs with 507 to protect the device before it truly fills.

docker exec swift_object_server grep -i fallocate_reserve /etc/swift/object-server.conf

fallocate_reserve = 1%

Diagnostic Workflow

Step 1: Reproduce and confirm it’s a write-side 507

swift stat
curl -s -o /dev/null -w "%{http_code}\n" -X PUT \
  -H "X-Auth-Token: $TOKEN" "$STORAGE_URL/testcont/obj" --data-binary @/etc/hostname

A 507 on PUT (with GETs of existing objects working) confirms a capacity/availability problem, not auth or a bad request.

Step 2: Find which node/device is complaining

docker logs swift_proxy_server 2>&1 | grep -iE "507|Insufficient Storage" | tail -10
docker logs swift_object_server 2>&1 | grep -iE "Insufficient Storage|mount_check|not mounted" | tail -10

The proxy log names the backend ip:port/device that returned 507.

Step 3: Check space, inodes, and mounts on that device

docker exec swift_object_server df -h /srv/node/*
docker exec swift_object_server df -i /srv/node/*
docker exec swift_object_server cat /proc/mounts | grep /srv/node

Full bytes, 100% inodes, or a missing mount each explain the 507.

Step 4: Check the drive health and mount flags

docker exec swift_object_server mount | grep srv/node
sudo dmesg | grep -iE 'I/O error|read-only|xfs|ext4' | tail

A ro mount or I/O errors means the disk failed or went read-only.

Step 5: Inspect ring balance

docker exec swift_object_server swift-ring-builder /etc/swift/object.builder | head -30
docker exec swift_object_server swift-recon --diskusage 2>/dev/null | tail -20

swift-recon --diskusage shows the fullest/emptiest devices across the cluster.

Example Root Cause Analysis

Backups start failing at 04:00 with 507 Insufficient Storage; existing objects still download fine. The proxy log names one backend:

docker logs swift_proxy_server 2>&1 | grep 507 | tail -3

proxy-server ERROR with Object server <ip>:6200/d3 re: Trying to PUT: 507 Insufficient Storage

Checking that device, bytes look fine but the mount is gone:

docker exec swift_object_server df -h /srv/node/d3
docker exec swift_object_server cat /proc/mounts | grep d3

df: /srv/node/d3: No such file or directory
# (no d3 line in /proc/mounts)

With mount_check = true, Swift correctly refused to write to the unmounted device — the underlying disk dropped its XFS mount after an I/O error:

sudo dmesg | grep -iE 'sdc|xfs' | tail -3

XFS (sdc): metadata I/O error ... shutting down filesystem

Fix: unmount cleanly, run a filesystem check (or replace the disk), remount, and confirm Swift can write again:

sudo umount /srv/node/d3 2>/dev/null
sudo xfs_repair /dev/sdc
sudo mount /srv/node/d3
docker exec swift_object_server cat /proc/mounts | grep d3   # mounted again
curl -s -o /dev/null -w "%{http_code}\n" -X PUT \
  -H "X-Auth-Token: $TOKEN" "$STORAGE_URL/testcont/obj" --data @/etc/hostname   # 201

If the disk is dead, remove it from the ring (swift-ring-builder ... remove), rebalance, and push the new ring so Swift stops targeting it.

Prevention Best Practices

• Monitor per-device disk usage and inodes with swift-recon --diskusage/--driveaudit; alert well before any device reaches its fallocate_reserve threshold.
• Keep mount_check = true so a dropped mount fails loudly with 507 instead of silently filling the root filesystem — then alert on unmounted /srv/node/* devices.
• Balance the ring: keep device weights proportional to capacity and rebalance after adding/removing drives so no device fills far ahead of the others.
• Watch for read-only/I/O error in dmesg and the object-server logs; pull failing drives from the ring promptly and let replication rebuild the data elsewhere.
• Plan capacity ahead of the fill curve — add drives/nodes and rebalance before the cluster’s fullest device crosses ~80%.
• For clusters with many tiny objects, size for inodes, not just bytes, when choosing filesystems and drive counts.
• For triage, drop the proxy/object 507 log lines into the free incident assistant to pinpoint the offending device, and see more OpenStack guides.

Quick Command Reference

# Reproduce a write-side 507
swift stat
curl -s -o /dev/null -w "%{http_code}\n" -X PUT \
  -H "X-Auth-Token: $TOKEN" "$STORAGE_URL/testcont/obj" --data @/etc/hostname

# Which node/device returned 507
docker logs swift_proxy_server 2>&1 | grep -iE "507|Insufficient Storage" | tail -10
docker logs swift_object_server 2>&1 | grep -iE "Insufficient Storage|mount_check|not mounted" | tail -10

# Space, inodes, mounts on that device
docker exec swift_object_server df -h /srv/node/*
docker exec swift_object_server df -i /srv/node/*
docker exec swift_object_server cat /proc/mounts | grep /srv/node

# Drive health & ring balance
sudo dmesg | grep -iE 'I/O error|read-only|xfs|ext4' | tail
docker exec swift_object_server swift-ring-builder /etc/swift/object.builder | head -30
docker exec swift_object_server swift-recon --diskusage | tail -20

Conclusion

507 Insufficient Storage is Swift’s write-path signal that it can’t place the required replicas: the target devices are full, unmounted, failed, or below their reserve. Reads of existing objects keep working, which is the tell that this is capacity/availability, not auth or a malformed request. Typical root causes:

1. Object disks that are genuinely full.
2. An unmounted device correctly refused by mount_check = true.
3. Ring imbalance concentrating data on a few hot devices.
4. A failed or read-only disk rejecting writes.
5. Exhausted inodes from huge numbers of small objects.
6. Free space dropping below the fallocate_reserve threshold.

Let the proxy log name the offending ip:port/device, then check space, inodes, and the mount on exactly that device — the fix is almost always full, unmounted, or failed, and the log points you straight at which.