🚨 OpenStack Error Guide: 'error connecting to the cluster' — Fix Cinder/Nova Ceph RADOS Timeout

Overview

error connecting to the cluster (with an underlying rados_connect timed out or [errno 110] Connection timed out) is what Cinder and Nova raise when the RBD driver cannot open a session to the Ceph cluster. Because Cinder uses Ceph for volumes and Nova often uses it for ephemeral disks and image caching, a Ceph reachability or auth problem blocks volume creation, attach, and instance spawn simultaneously.

The literal errors you will see:

rados.Error: error connecting to the cluster: [errno 110] RADOS timed out (Connection timed out)

ERROR cinder.volume.drivers.rbd [-] Error connecting to ceph cluster.: rados.TimedOut: [errno 110] RADOS timed out

It occurs whenever a service opens a RADOS connection: cinder-volume creating a volume, nova-compute spawning an RBD-backed instance, or Glance storing an image. The tell is that the failure is at the Ceph transport layer — mon quorum, network, or cephx auth — not in the OpenStack service logic.

Symptoms

• Volumes stick in creating then flip to error; instances fail to spawn on RBD-backed hosts.
• cinder-volume or nova-compute logs RADOS timed out / error connecting to the cluster.
• The Cinder backend shows down in the service list.

openstack volume service list -c Binary -c Host -c Status -c State

+------------------+-------------------+---------+-------+
| Binary           | Host              | Status  | State |
+------------------+-------------------+---------+-------+
| cinder-volume    | controller@rbd-1  | enabled | down  |
+------------------+-------------------+---------+-------+

docker logs cinder_volume 2>&1 | grep -iE "RADOS|connecting to ceph|errno 110" | tail -3

ERROR cinder.volume.drivers.rbd Error connecting to ceph cluster.: rados.TimedOut: [errno 110] RADOS timed out

Common Root Causes

1. Ceph monitors unreachable (network/firewall)

If the service host can’t reach the mons on 3300/6789, RADOS times out.

grep -E 'mon_host|mon host' /etc/ceph/ceph.conf
nc -vz <mon-ip> 6789
nc -vz <mon-ip> 3300

Connection to 10.0.0.21 6789 port [tcp/*] failed: Connection timed out

A timed out (not refused) usually means a firewall or routing change cut the path to the mon.

2. Ceph cluster unhealthy / no mon quorum

If the mons lost quorum, no client can connect regardless of network.

docker exec ceph_mon ceph -s 2>/dev/null | head -15
# or from a client with admin keyring
ceph health detail

  cluster:
    health: HEALTH_WARN
    services:
      mon: 1/3 daemons up (out of quorum)

Fewer than a majority of mons up means no quorum and every RADOS connect will time out.

3. cephx keyring wrong or missing

The service authenticates with a cephx key; a missing or stale keyring for the cinder/nova client fails auth (sometimes surfacing as a connect timeout).

ls -l /etc/ceph/ceph.client.cinder.keyring
docker exec ceph_mon ceph auth get client.cinder 2>/dev/null
grep -E 'rbd_user|rbd_secret_uuid' /etc/cinder/cinder.conf

[client.cinder]
    key = AQ...==
    caps mon = "profile rbd"
    caps osd = "profile rbd pool=volumes"

If the on-host keyring key doesn’t match ceph auth get, auth fails.

4. Wrong pool or missing caps

The client key may lack caps on the target pool, so the connect/open fails.

docker exec ceph_mon ceph osd pool ls 2>/dev/null
grep -E 'rbd_pool|^\[rbd' /etc/cinder/cinder.conf

volumes
images
vms

A rbd_pool that isn’t in the list, or a key without caps on it, breaks the driver.

5. Clock skew between client and mons

Large time drift breaks cephx and can manifest as connection failures.

timedatectl status | grep -E 'synchronized|NTP'
docker exec ceph_mon ceph -s 2>/dev/null | grep -i clock

System clock synchronized: no
health: HEALTH_WARN clock skew detected on mon.controller-02

6. Wrong mon_host / fsid after redeploy

After a Ceph redeploy or IP change, a stale ceph.conf on the OpenStack host points at dead mons.

grep -E 'fsid|mon_host' /etc/ceph/ceph.conf
docker exec ceph_mon ceph fsid 2>/dev/null

fsid = 11111111-2222-3333-4444-555555555555   # on the OpenStack host
22222222-... # actual cluster fsid — mismatch!

Diagnostic Workflow

Step 1: Reach the mons from the failing host

grep -E 'mon_host' /etc/ceph/ceph.conf
for m in <mon-ip-1> <mon-ip-2> <mon-ip-3>; do nc -vz $m 6789; done

timed out on all mons → network/firewall; refused → mon daemon down.

Step 2: Check cluster health and quorum

docker exec ceph_mon ceph -s
docker exec ceph_mon ceph quorum_status 2>/dev/null | grep -i quorum_names

Step 3: Verify the client keyring matches Ceph

grep -E 'rbd_user|rbd_pool|rbd_secret_uuid' /etc/cinder/cinder.conf
docker exec ceph_mon ceph auth get client.cinder
# compare the key with /etc/ceph/ceph.client.cinder.keyring on the host

Step 4: Test a raw RADOS/RBD op with that key

rbd -p volumes --id cinder --keyring /etc/ceph/ceph.client.cinder.keyring ls 2>&1 | head

A timeout here isolates the problem to Ceph/transport, independent of Cinder.

Step 5: Check clock and config freshness

timedatectl status | grep synchronized
grep -E 'fsid|mon_host' /etc/ceph/ceph.conf
docker exec ceph_mon ceph fsid

Example Root Cause Analysis

Overnight, all new volumes fail and cinder-volume shows down. The log:

ERROR cinder.volume.drivers.rbd Error connecting to ceph cluster.: rados.TimedOut: [errno 110] RADOS timed out

The operator tests mon reachability from the controller:

for m in 10.0.0.21 10.0.0.22 10.0.0.23; do nc -vz $m 6789; done

Connection to 10.0.0.21 6789 port [tcp/*] succeeded!
Connection to 10.0.0.22 6789 port [tcp/*] failed: Connection timed out
Connection to 10.0.0.23 6789 port [tcp/*] failed: Connection timed out

Two of three mons are unreachable. ceph -s from the surviving mon confirms lost quorum:

docker exec ceph_mon ceph -s | head

health: HEALTH_WARN
mon: 1/3 daemons up (out of quorum)

The two down mon hosts were caught by a new firewall rule pushed the night before that dropped inter-mon traffic. Restoring the rule brings the mons back and re-forms quorum:

# after fixing the firewall on the mon hosts
docker exec ceph_mon ceph -s | grep -E 'mon:|health'   # 3/3, HEALTH_OK
openstack volume service list -c Binary -c State        # cinder-volume 'up'
openstack volume create --size 1 test-rebalance          # succeeds

Longer term, add mon-reachability and quorum alerting, and gate firewall changes behind a check that preserves Ceph mon and OSD ports.

Prevention Best Practices

• Monitor Ceph health and mon quorum directly; page on HEALTH_WARN/HEALTH_ERR and any mon out of quorum, since it blocks Cinder and Nova at once.
• Pre-open Ceph ports (3300, 6789 for mons; 6800-7300 for OSDs) to all OpenStack hosts and test with nc -vz after any firewall or network change.
• Manage ceph.conf and client keyrings with config management so a redeploy or IP change updates every OpenStack host’s mon_host and fsid atomically.
• Keep clocks synced with NTP/chrony on mons and clients; cephx and quorum are sensitive to skew.
• Validate client caps (ceph auth get client.cinder) match the pools Cinder/Nova/Glance actually use.
• Test the credential path independently with rbd --id cinder ls so you can isolate Ceph problems from OpenStack ones.
• Feed the RADOS timed out traces into the free incident assistant to separate a network/quorum outage from an auth problem, and see more OpenStack guides.

Quick Command Reference

# Reach the mons from the failing host
grep -E 'mon_host' /etc/ceph/ceph.conf
for m in <mon-ip-1> <mon-ip-2> <mon-ip-3>; do nc -vz $m 6789; done

# Cluster health & quorum
docker exec ceph_mon ceph -s
docker exec ceph_mon ceph quorum_status | grep -i quorum_names

# Keyring & caps
grep -E 'rbd_user|rbd_pool|rbd_secret_uuid' /etc/cinder/cinder.conf
docker exec ceph_mon ceph auth get client.cinder

# Isolate to Ceph with a raw op
rbd -p volumes --id cinder --keyring /etc/ceph/ceph.client.cinder.keyring ls

# Clock & config freshness
timedatectl status | grep synchronized
grep -E 'fsid|mon_host' /etc/ceph/ceph.conf
docker exec ceph_mon ceph fsid

# Cinder backend state
openstack volume service list -c Binary -c Host -c State

Conclusion

error connecting to the cluster / RADOS timed out is a Ceph transport failure: the RBD driver can’t reach or authenticate to the cluster, so Cinder volumes and Nova RBD instances fail together. Typical root causes:

1. Ceph monitors unreachable due to a firewall or routing change.
2. The Ceph cluster lost mon quorum and rejects all clients.
3. A wrong or missing cephx keyring for the cinder/nova client.
4. The client key lacks caps on the target pool, or the pool name is wrong.
5. Clock skew between client and mons breaking cephx.
6. A stale ceph.conf (wrong mon_host/fsid) after a Ceph redeploy.

Test mon reachability and ceph -s first: timed out vs refused and quorum state tell you whether you’re chasing the network, the mons, or the client credentials.