🚨 OpenStack Error Guide: 'ToozConnectionError: Error connecting to etcd' — fix Tooz coordination

Overview

ToozConnectionError is the error the Tooz library raises when an OpenStack service tries to reach its distributed-coordination backend — etcd in most Kolla-Ansible clouds — and can’t. Tooz gives services like Cinder (volume locks), Neutron, and anything using distributed locks a way to serialize work across controllers. When the etcd backend is unreachable or has lost quorum, lock acquisition fails and those operations hang or error out.

The literal errors you will see:

tooz.coordination.ToozConnectionError: Error connecting to etcd: <urllib3.exceptions.NewConnectionError: Failed to establish a new connection: [Errno 111] Connection refused>

etcdserver: request timed out
tooz.coordination.ToozError: <etcd error> etcdserver: no leader

It occurs whenever a service takes a coordination lock: a Cinder volume state transition, an agent registering into a Tooz group, or a service starting up and joining the coordinator. Because the lock path is shared, several services can start logging coordination errors at once — the tell that the problem is etcd/the coordination backend, not the individual service.

Symptoms

• Cinder volume create/delete/attach operations hang, then time out; volumes sit in transitional states.
• Service logs show ToozConnectionError, etcdserver: no leader, or request timed out.
• A service refuses to finish starting because it can’t join the coordinator.

openstack volume service list -c Binary -c Host -c State

+------------------+-------------------+-------+
| Binary           | Host              | State |
+------------------+-------------------+-------+
| cinder-scheduler | controller        | up    |
| cinder-volume    | controller@lvm    | down  |
+------------------+-------------------+-------+

docker logs cinder_volume 2>&1 | grep -iE "Tooz|etcd|coordination" | tail -3

ERROR cinder.coordination tooz.coordination.ToozConnectionError: Error connecting to etcd: [Errno 111] Connection refused

Common Root Causes

1. etcd is down or in a restart loop

If the etcd container crashed or is crash-looping, no client can connect and every lock attempt is refused.

docker ps --filter name=etcd --format '{{.Names}} {{.Status}}'
docker logs etcd 2>&1 | tail -20

etcd Restarting (1) 6 seconds ago

2. Cluster quorum loss (no leader)

etcd needs a majority of members to elect a leader and accept writes. With a majority down, the cluster goes read-only/unavailable and lock acquisition (a write) fails with “no leader”.

docker exec etcd etcdctl endpoint status --cluster -w table 2>/dev/null
docker exec etcd etcdctl endpoint health --cluster 2>/dev/null

<vip>:2379 is unhealthy: failed to commit proposal: context deadline exceeded
Error: unhealthy cluster

A 3-member cluster survives losing 1 member; losing 2 breaks quorum and writes stop.

3. DB space exceeded (mvcc alarm)

Without compaction, etcd’s backend grows until it hits quota-backend-bytes and raises a NOSPACE alarm that blocks writes — so locks can’t be taken even though members are “up”.

docker exec etcd etcdctl alarm list 2>/dev/null
docker exec etcd etcdctl endpoint status -w table 2>/dev/null

memberID:... alarm:NOSPACE

mvcc: database space exceeded

4. Network / firewall blocks 2379 (client) or 2380 (peer)

A firewall rule or security-group change cuts the client port (2379) between a service host and etcd, or the peer port (2380) between members, partitioning the cluster.

ss -ltnp | grep -E ':23(79|80)'      # on an etcd host
nc -vz <ETCD_HOST> 2379              # from a service host
sudo iptables -L -n | grep -E '2379|2380'

Connection to <ETCD_HOST> 2379 port [tcp/*] failed: Connection timed out

A timed out (not refused) usually means a firewall is dropping packets.

5. Wrong backend_url or TLS mismatch

After a redeploy or VIP change, [coordination] backend_url can point at the wrong host/scheme (http vs https), so the client never establishes a session.

docker exec cinder_volume grep -A2 '\[coordination\]' /etc/cinder/cinder.conf

[coordination]
backend_url = etcd3+http://<vip>:2379

A https-configured etcd behind an etcd3+http:// URL (or the reverse) fails to connect.

6. Stale lock keys after an ungraceful crash

If a lock holder died hard, a lease-less key can linger and block a new holder until the lease expires or the key is understood.

docker exec etcd etcdctl get --prefix / --keys-only 2>/dev/null | grep -i lock | head

/tooz/locks/cinder-...

Diagnostic Workflow

Step 1: Confirm it’s coordination, not just one service

openstack volume service list -c Binary -c Host -c State
docker logs cinder_volume 2>&1 | grep -iE "Tooz|etcd|coordination" | tail -10

If multiple lock-taking services log Tooz errors together, suspect etcd.

Step 2: Check the etcd containers and cluster health

docker ps --filter name=etcd
docker exec etcd etcdctl endpoint health --cluster
docker exec etcd etcdctl endpoint status --cluster -w table

Look for unhealthy members, a missing leader, or a member that won’t stay up.

Step 3: Check for alarms and DB size

docker exec etcd etcdctl alarm list
docker exec etcd etcdctl endpoint status -w table   # DB SIZE column

A NOSPACE alarm means you must defrag and disarm before writes recover.

Step 4: Verify the client path and config

nc -vz <ETCD_HOST> 2379
docker exec cinder_volume grep -A2 '\[coordination\]' /etc/cinder/cinder.conf

refused → etcd down; timed out → firewall; connects but service still errors → check the URL scheme/TLS.

Step 5: Recover quorum, then reclaim space

# If a member is just down, restart it and re-check health
docker restart etcd
docker exec etcd etcdctl endpoint health --cluster

# Clear a space alarm (only after confirming members are healthy)
docker exec etcd etcdctl defrag --cluster
docker exec etcd etcdctl alarm disarm

Example Root Cause Analysis

At 02:40 the on-call sees Cinder volume creates hang and cinder-volume flips to down. The log:

ERROR cinder.coordination tooz.coordination.ToozError: etcdserver: mvcc: database space exceeded

mvcc: database space exceeded points at etcd’s backend, not Cinder. Checking alarms:

docker exec etcd etcdctl alarm list

memberID:12637028 alarm:NOSPACE

The DB filled because auto-compaction was never configured, so revisions accumulated until the 2 GiB default quota tripped. Fix: compact to the current revision, defrag to reclaim disk, then disarm the alarm:

REV=$(docker exec etcd etcdctl endpoint status -w json | grep -o '"revision":[0-9]*' | head -1 | cut -d: -f2)
docker exec etcd etcdctl compact "$REV"
docker exec etcd etcdctl defrag --cluster
docker exec etcd etcdctl alarm disarm
docker exec etcd etcdctl endpoint health --cluster   # healthy
openstack volume create --size 1 test-recover        # lock succeeds

Longer term, enable auto-compaction and raise --quota-backend-bytes deliberately so revisions can’t silently exhaust the keyspace again.

Prevention Best Practices

• Monitor etcd directly: member health, has_leader, leader changes, DB size vs. quota, and any raised alarm. Page on quorum loss.
• Configure auto-compaction (--auto-compaction-mode=revision or periodic) and a deliberate --quota-backend-bytes so the backend can’t fill unnoticed.
• Run etcd with an odd member count (3 or 5) so a single failure never breaks quorum.
• Keep [coordination] backend_url (host, port, and http/https scheme) managed by config management so a VIP change updates every service consistently.
• Pre-open 2379/2380 in firewalls and security groups for all service and etcd hosts, and test with nc -vz after any network change.
• Snapshot before maintenance: etcdctl snapshot save protects you before any member replacement or forced rebuild.
• For triage, drop the simultaneous ToozConnectionError traces into the free incident assistant to confirm a coordination-wide outage, and see more OpenStack guides.

Quick Command Reference

# Is coordination broken? (lock-taking services error together)
openstack volume service list -c Binary -c Host -c State
docker logs cinder_volume 2>&1 | grep -iE "Tooz|etcd|coordination" | tail -10

# etcd containers & cluster health
docker ps --filter name=etcd
docker exec etcd etcdctl endpoint health --cluster
docker exec etcd etcdctl endpoint status --cluster -w table

# Alarms & DB size
docker exec etcd etcdctl alarm list
docker exec etcd etcdctl endpoint status -w table

# Client path & config
nc -vz <ETCD_HOST> 2379
docker exec cinder_volume grep -A2 '\[coordination\]' /etc/cinder/cinder.conf

# Recover: restart, compact, defrag, disarm
docker restart etcd
docker exec etcd etcdctl defrag --cluster
docker exec etcd etcdctl alarm disarm

Conclusion

ToozConnectionError and etcdserver: no leader are coordination-backend failures: the etcd cluster behind Tooz, not the calling service, is the problem. The shared, multi-service nature of the errors is the diagnostic signature. Typical root causes:

1. etcd is down or crash-looping (nothing to connect to).
2. Quorum loss — a majority of members down leaves the cluster with no leader.
3. A NOSPACE (mvcc: database space exceeded) alarm blocking writes.
4. A firewall or security-group change blocking 2379/2380.
5. A wrong backend_url host/scheme or TLS mismatch after a redeploy.
6. Stale lock keys lingering after an ungraceful crash.

Check member health and whether a leader exists first; the refused vs. timed out vs. no leader vs. NOSPACE distinction tells you which of these you’re chasing — and always confirm quorum before touching cluster membership.