☁️ OpenStack Error: 'Failed to initialize driver' — fix cinder-volume backend startup

Exact Error Message

# /var/log/cinder/cinder-volume.log
ERROR cinder.volume.manager [-] Failed to initialize driver.
ERROR cinder.volume.manager Traceback (most recent call last):
ERROR cinder.volume.manager   File ".../cinder/volume/manager.py", line 512, in init_host
ERROR cinder.volume.manager   File ".../cinder/volume/drivers/rbd.py", line 411, in check_for_setup_error
ERROR cinder.volume.manager cinder.exception.VolumeBackendAPIException:
    Bad or unexpected response from the storage volume backend API:
    Error connecting to ceph cluster: [errno 13] RADOS permission denied
WARNING cinder.volume.manager [-] Update driver status failed: (config name rbd-1)
    is uninitialized.

The service-level symptom is a backend that is down/disabled in cinder service-list, and every volume create on that backend failing with No valid backend was found.

What It Means

cinder-volume loads one driver per backend listed in enabled_backends. During startup it calls the driver’s check_for_setup_error() to confirm it can actually talk to the storage system (Ceph/RBD, LVM/iSCSI, NetApp, Pure, etc.). Failed to initialize driver means that check raised — the backend credentials, connectivity, or a required library/binary is wrong — so Cinder marks that backend uninitialized and refuses to schedule volumes to it.

This is a configuration or dependency problem on the storage node, not a problem with any individual volume. Until the driver initializes, the backend does not report capacity to the scheduler and appears effectively offline.

Common Causes

• Wrong or missing backend credentials (Ceph keyring/rbd_user, iSCSI/vendor API login).
• A required client library or CLI is missing (e.g. python3-rados/ceph-common, vendor SDK).
• Network/firewall blocks the storage backend (Ceph mons, iSCSI portal, management API).
• enabled_backends references a config section that does not exist or has a typo.
• The LVM volume group named in volume_group is missing on an LVM backend.
• A backend TLS certificate is untrusted or the management endpoint URL is wrong.

Diagnostic Commands

Check which backends Cinder thinks are up:

openstack volume service list
sudo tail -f /var/log/cinder/cinder-volume.log

Confirm the enabled backends and their driver settings:

sudo crudini --get /etc/cinder/cinder.conf DEFAULT enabled_backends
sudo crudini --get /etc/cinder/cinder.conf rbd-1 volume_driver
sudo crudini --get /etc/cinder/cinder.conf rbd-1 rbd_user

Reproduce the backend connectivity check by hand. For a Ceph/RBD backend:

sudo -u cinder rbd -p volumes --id cinder --keyring /etc/ceph/ceph.client.cinder.keyring ls

For an LVM backend, confirm the volume group exists:

sudo vgs cinder-volumes

Step-by-Step Resolution

1. Read the traceback’s last line — it names the real cause (permission denied, connection refused, missing library). Fix that specific dependency first. For Ceph, install and verify the client:

sudo apt-get install -y ceph-common python3-rados   # or dnf install
sudo -u cinder ceph -s --id cinder --keyring /etc/ceph/ceph.client.cinder.keyring

2. Correct the backend credentials/section in cinder.conf. Ensure enabled_backends matches real section names exactly:

[DEFAULT]
enabled_backends = rbd-1

[rbd-1]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
rbd_pool = volumes
rbd_user = cinder
rbd_ceph_conf = /etc/ceph/ceph.conf
volume_backend_name = ceph-ssd

3. Fix file ownership so the cinder user can read the keyring/config:

sudo chown cinder:cinder /etc/ceph/ceph.client.cinder.keyring
sudo chmod 600 /etc/ceph/ceph.client.cinder.keyring

4. Restart the volume service and watch the driver initialize cleanly:

sudo systemctl restart cinder-volume
sudo grep -E 'Driver initialization completed|Failed to initialize driver' \
  /var/log/cinder/cinder-volume.log | tail

INFO cinder.volume.manager [-] Driver initialization completed successfully. (config name rbd-1)

5. Confirm the backend is now up and reporting capacity:

openstack volume service list --service cinder-volume

| cinder-volume | storage-01@rbd-1 | nova | enabled | up |

6. Validate end to end by creating a small test volume:

openstack volume create --size 1 driver-smoke-test
openstack volume show driver-smoke-test -c status   # available

Prevention

• Manage cinder.conf and backend credentials with configuration management so enabled_backends and section names never drift.
• Bake required driver dependencies (ceph-common, vendor SDKs) into the storage node image.
• Monitor openstack volume service list and alert when a cinder-volume backend goes down.
• Test backend connectivity (rbd/vgs/vendor API) as a pre-deploy check before restarting the service.
• Keep keyring/config file ownership as cinder:cinder with restrictive permissions.

Related Errors

• No valid backend was found — the scheduler side of a backend that failed to initialize.
• VolumeBackendAPIException: Bad or unexpected response — the generic wrapper this error uses.
• RADOS permission denied — a Ceph keyring/caps problem specifically.
• Volume group "cinder-volumes" not found — an LVM backend missing its VG.

Frequently Asked Questions

Why is only one of my backends down? Each backend in enabled_backends initializes independently. A credential, dependency, or connectivity problem on one backend takes only that backend offline while the others stay up.

The config looks right — why does the driver still fail? The traceback’s final line is authoritative. It usually points at something outside cinder.conf: a missing client library, unreadable keyring, blocked port, or missing LVM volume group. Reproduce that check by hand.

Do I need to restart cinder-volume after fixing it? Yes. check_for_setup_error() runs at startup, so the backend stays uninitialized until you restart the service and see “Driver initialization completed successfully.”

How do I stop this recurring after upgrades? Pin driver dependencies in your node image and validate connectivity before rollout — the automation patterns in the prompt library help script that. For more storage fixes, see the OpenStack guides.