🚨 OpenStack Error Guide: 'Swift 503 unable to connect to memcached' proxy failure

Exact Error Message

$ swift stat
Account HEAD failed: https://swift.example.com/v1/AUTH_9c4f... 503 Service Unavailable

In the swift proxy log:

Jun 27 16:08:51 ctrl1 proxy-server: ERROR Unhandled exception in request:
Jun 27 16:08:51 ctrl1 proxy-server: Error connecting to memcached: 10.0.0.11:11211
Jun 27 16:08:51 ctrl1 proxy-server: STDERR: ConnectionError: [Errno 111] Connection refused
Jun 27 16:08:51 ctrl1 proxy-server: 10.0.0.50 10.0.0.50 27/Jun/2026/16/08/51 HEAD /v1/AUTH_9c4f...
HTTP/1.0 503 - python-swiftclient ... - - - tx... - 0.0031 - - 1719504531.1 1719504531.1 -

What the Error Means

Swift’s proxy server uses memcached for two critical things: caching authentication tokens (via the authtoken / keystoneauth middleware) and caching account/container existence and ratelimit state. When the proxy cannot reach any configured memcached node, middleware that depends on it cannot validate tokens or look up cached state, and the proxy returns 503 Service Unavailable to the client.

The 503 ... unable to connect to memcached is therefore usually not a storage-node problem at all — the object servers may be perfectly healthy. It is a control-path failure in the proxy tier. Either memcached is down, the proxy is pointed at the wrong address/port, a firewall blocks 11211, or every node in the memcache_servers list is unreachable so the proxy has no cache to fall back on.

Common Causes

• memcached service is down or crashed on one or more control nodes.
• Wrong memcache_servers list in proxy-server.conf (or the [filter:cache] / [filter:authtoken] sections) pointing at a dead or renamed host.
• Firewall / security group blocking port 11211 between proxy and memcached.
• memcached bound to localhost only while the proxy expects it on the management IP.
• All memcached nodes unreachable at once (network partition or a rolling restart that took them all down).
• memcached out of memory / evicting aggressively so token lookups miss and re-auth storms overwhelm Keystone, surfacing as intermittent 503s.

How to Reproduce the Error

Stop memcached under a running proxy:

1. Confirm swift stat works normally.
2. Stop the memcached service the proxy depends on.
3. Run swift stat again.

The proxy’s cache/authtoken middleware fails to connect, and the client receives 503 Service Unavailable. Pointing memcache_servers at a non-listening port reproduces the misconfiguration variant.

Diagnostic Commands

Read-only. Confirm memcached is alive and reachable from the proxy node.

# Is memcached running where the proxy expects it?
# Kolla-Ansible
docker ps --filter name=memcached
docker logs swift_proxy_server 2>&1 | grep -i "memcache\|503" | tail
# Traditional packages
systemctl status memcached
journalctl -u swift-proxy | grep -i "memcache\|503" | tail

Verify the port is open and answering from the proxy host (read-only stats):

# Is the port listening?
ss -ltnp | grep 11211

# Ask memcached for its stats without modifying anything
printf 'stats\r\nquit\r\n' | nc 10.0.0.11 11211 | head

STAT pid 1342
STAT uptime 5821
STAT curr_connections 18
STAT get_hits 90412
STAT get_misses 1203

Check what the proxy is configured to use:

# Kolla-Ansible
docker exec swift_proxy_server grep -R memcache_servers /etc/swift/proxy-server.conf
# Traditional
grep -R memcache_servers /etc/swift/proxy-server.conf

Step-by-Step Resolution

1. Confirm memcached is actually down or unreachable. Use ss -ltnp | grep 11211 on each configured node and the stats probe above from the proxy host. A refused connection points at a dead service or a firewall.

2. Restart memcached if it has crashed:

   docker restart memcached            # Kolla-Ansible
   systemctl restart memcached         # Traditional

   Re-run swift stat; if it recovers, the proxy simply lost its cache backend.

3. Fix the memcache_servers list if the proxy points at the wrong host. Compare the addresses in proxy-server.conf against the nodes actually running memcached. Correct them so every entry is a live, reachable host:11211.

4. Open port 11211 between proxy and memcached if a firewall or security group is blocking it. Confirm with the nc probe from the proxy host after the change.

5. Bind memcached to the management IP, not localhost, if the proxy reaches it over the network. A memcached listening only on 127.0.0.1 is invisible to a proxy on another node.

6. Restart the swift proxy so it re-reads config and re-establishes connections, then verify end to end:

   docker restart swift_proxy_server   # Kolla-Ansible
   systemctl restart swift-proxy       # Traditional
   swift stat

Prevention and Best Practices

• Configure multiple memcached nodes in memcache_servers so the loss of one does not blackhole the proxy’s token cache.
• Monitor memcached on every control node and alert on the service being down or on curr_connections dropping to zero from the proxy tier.
• Watch eviction and miss rates — a memcached running hot and evicting tokens causes re-auth storms that surface as intermittent 503s; size its memory for the token volume.
• Never restart all memcached instances at once during maintenance; roll them so the proxy always has at least one live cache.
• Pin memcached to the management IP and keep port 11211 explicitly allowed between the proxy and cache tiers in firewall rules.

Related Errors

• 503 Service Unavailable from Swift without a memcached message — usually a storage-node or ring problem, not the cache tier.
• 401 Unauthorized after a memcached restart — tokens were cached only in memcached and are gone; clients simply re-authenticate.
• Error connecting to memcached in keystonemiddleware logs for other services (Nova, Neutron) — the same root cause affecting a different consumer. See our /categories/openstack/ guides.
• proxy-server: ERROR with Object server — a backend storage error, distinct from the memcached control path.

Frequently Asked Questions

Does losing memcached lose my stored objects? No. memcached only caches tokens and account/container metadata. Object data lives on the storage nodes and is untouched. Restoring memcached restores normal proxy operation.

Why does the proxy 503 instead of just skipping the cache? The authtoken middleware needs memcached to validate and cache tokens. Without it, it cannot safely authorize requests, so the proxy fails closed with 503 rather than serving unauthenticated traffic.

How many memcached nodes should I run? At least as many as your control-plane nodes, all listed in memcache_servers, so a single node failure or restart never leaves the proxy with no cache.

Could a firewall cause this without memcached being down? Yes. If port 11211 is blocked between proxy and cache, memcached is healthy but unreachable, and the proxy behaves exactly as if it were down. The nc probe from the proxy host distinguishes the two.

Is the fix different under Kolla-Ansible? The diagnosis and fix are the same; only access differs — docker ps/docker restart memcached and docker logs swift_proxy_server versus systemctl and journalctl.