☁️ OpenStack Error: 'container sync' not replicating — fix Swift container-sync failures

Exact Error Message

container-sync: ERROR Syncing "/srv/node/d3/containers/1234/.../a1b2.db"
to remote https://swift-b.example.com/v1/AUTH_9f/backup:
Response status 401 Unauthorized
container-sync: ERROR Unhandled exception while syncing
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/swift/container/sync.py", line 512, in container_sync_row
    self.swift.put_object(...)
swiftclient.exceptions.ClientException: Object PUT failed: 401 Unauthorized
container-sync: Since Thu Jul 17 09:12:04 2026: 0 synced [412 failures], ...

You may also see container-sync: ERROR ... Connection refused or X-Container-Sync-Key not set in /var/log/swift/container-sync.log (or via journalctl -u swift-container-sync), and objects that simply never appear in the destination container.

What It Means

Swift container sync is a background daemon (swift-container-sync) that copies objects from a source container to a destination container, which may live in an entirely separate Swift cluster. It is driven by two things: a shared realm/key defined in container-sync-realms.conf, and per-container metadata (X-Container-Sync-To and X-Container-Sync-Key) that you set with the Swift client.

When sync “fails”, the daemon is usually running fine but every row it tries to push is rejected — most often with 401 Unauthorized (the keys on the two containers do not match) or a connection error (the destination is unreachable or not in allowed_sync_hosts). Because container sync is eventually-consistent and silent by design, the only symptom users see is that objects never replicate.

Common Causes

• The X-Container-Sync-Key on the source and destination containers do not match.
• container-sync-realms.conf is missing, has mismatched realm keys, or differs between clusters.
• The destination host is not listed in allowed_sync_hosts (intra-cluster sync) or the realm cluster endpoint is wrong.
• The swift-container-sync daemon is not running on the object nodes.
• Network/TLS failure reaching the destination endpoint (firewall, expired cert, wrong port).
• The destination container does not exist or the account is read-only/over quota.

Diagnostic Commands

Check that the sync daemon is actually running on the object nodes:

systemctl status swift-container-sync
sudo tail -f /var/log/swift/container-sync.log

Inspect the source container’s sync metadata:

swift stat backup
# Look for: X-Container-Sync-To and X-Container-Sync-Key

Verify the realm configuration is present and readable:

sudo swift-recon --md5   # confirms ring/conf consistency across nodes
cat /etc/swift/container-sync-realms.conf

Confirm the destination endpoint is reachable from a source object node:

curl -kI https://swift-b.example.com/v1/AUTH_9f/backup

Step-by-Step Resolution

1. Confirm both containers share an identical sync key. Set it on each side with the same secret:

swift post -k 'a-strong-shared-secret' backup
# on the destination cluster/account:
swift post -k 'a-strong-shared-secret' backup

2. Point the source container at the destination using a realm/cluster path or a full URL:

swift post -t '//realm1/clusterB/AUTH_9f/backup' -k 'a-strong-shared-secret' backup

3. Make sure the realm exists and matches on both clusters. /etc/swift/container-sync-realms.conf must define the same realm name and key on every node:

[realm1]
key = realm-shared-key
cluster_clusterB = https://swift-b.example.com/v1/

4. If you use full URLs instead of realms, add the destination to allowed_sync_hosts in container-server.conf on the destination nodes, then restart:

sudo systemctl restart swift-container-server swift-container-sync

5. Re-run and watch the counters move from failures to synced:

sudo grep 'synced' /var/log/swift/container-sync.log | tail -5

container-sync: Since ...: 118 synced [0 failures], 0 skipped, 0 deletes

6. Verify objects now appear on the destination:

swift --os-storage-url https://swift-b.example.com/v1/AUTH_9f list backup

Prevention

• Store the sync key in your configuration management and set it on both containers in the same run, so they can never drift.
• Prefer realm-based sync (container-sync-realms.conf) over hard-coded URLs; it centralizes keys and cluster endpoints.
• Monitor /var/log/swift/container-sync.log for a nonzero failures count and alert on it.
• Keep clocks in sync (NTP/chrony) across clusters; container sync relies on consistent timestamps.
• Test reachability and TLS trust between clusters before enabling sync in production.

Related Errors

• HTTP 507 Insufficient Storage — a full or unmounted Swift drive, unrelated to sync auth.
• 401 Unauthorized on normal PUTs — a Keystone/auth token problem rather than a sync key mismatch.
• Connection refused from swift-proxy — proxy or memcached outage on the destination.
• X-Container-Sync-Key not set — sync metadata was never applied to the container.

Frequently Asked Questions

Why do objects sync one way but not the other? Container sync is directional. Each container’s X-Container-Sync-To points at exactly one destination, so you must configure sync on both containers if you want bidirectional replication.

Does container sync copy objects that already existed before I enabled it? Yes, the daemon walks the container database and will eventually push pre-existing rows, but large backlogs can take a long time to drain.

Why is the log full of 401 errors even though my key looks right? The most common cause is trailing whitespace or a key set on only one side. Re-apply the exact same -k value to both containers and restart swift-container-sync.

Should I use container sync or the newer replication tools? Container sync remains the built-in cross-cluster option, but for large or complex topologies many operators script it or use the AI-assisted patterns in the prompt library. For more object-storage fixes, see the OpenStack guides.