Five OpenStack Ceph Tuning Levers to Baseline Before Rollout

If you only have an afternoon, focus on five levers: set RBD client caching to write-back and raise rbd_cache_max_dirty and rbd_cache_size past their defaults, keep rbd_exclusive_cinder_pool true, let BlueStore’s memory autotune manage OSD RAM while you size RocksDB on NVMe, and fix host-level CPU and network settings before touching the cluster. Baseline every change with fio first.

---

TL;DR:

• Use write back caching only with reliable guest flushes, then double rbd_cache_max_dirty from its 24 MiB default and monitor dirty data age.
• Keep rbd_exclusive_cinder_pool true to avoid separate size queries for every image, especially in large volume fleets, and reduce unnecessary cluster load.
• Leave BlueStore memory autotuning enabled, and size RocksDB block.db at roughly 1% to 2% of primary capacity on fast NVMe.
• Place active Cinder volumes on NVMe backed pools; erasure coded pools suit cold snapshots and backups, while unmeasured placement group increases risk lock contention.
• Baseline every change with fio using 4k random I/O at iodepth=1 and 4MB sequential tests; repeat runs and roll changes out host by host.

---

Table of Contents

• RBD and librbd cache: policies and practical settings
• Cinder and Nova settings that reduce Ceph load and improve latency
• Pool design and CRUSH considerations for block workloads
• OSD and BlueStore tuning: memory, cache autotune, and RocksDB sizing
• Network and host-level tuning that unlock Ceph performance
• Benchmarking, validation, and rollback: fio, Ceph counters, and staged rollouts
• Hyperconverged deployments: resource isolation and OpenStack integration notes
• A pragmatic, ops-first take on Ceph tuning
• How our team can help you tune Ceph without the guesswork
• FAQ
• Sources

RBD and librbd cache: policies and practical settings

rbd_cache controls whether librbd buffers I/O client-side, and rbd_cache_policy decides how. Write-back buffers writes and acknowledges them before they hit the OSDs, which coalesces small, contiguous writes into fewer, larger requests. Write-around skips caching writes but still caches reads. Write-through caches reads but forces every write to disk before acknowledging it, trading speed for safety.

For typical OpenStack VM workloads, write-back is usually the right default, but only when the guest OS issues proper flushes. Linux kernels at 2.6.32 or later handle virtio flush semantics correctly, so write-back caching behaves safely on modern guest images.

The defaults are rbd_cache_max_dirty = 24 MiB and rbd_cache_size = 32 MiB. These work, but they’re conservative for busy volumes.

• Double rbd_cache_max_dirty first and watch dirty-data age before going further.
• Apply changes at the client config level (/etc/ceph/ceph.conf on compute nodes) rather than per image, so Nova’s libvirt driver picks them up consistently.
• Verify with ceph daemon client.<id> perf dump and simple rbd map plus fio read/write tests before and after.

Pro Tip: Change one cache parameter at a time and let a workload run for at least an hour before judging the result; cache effects often show up in tail latency, not average throughput.

Cinder and Nova settings that reduce Ceph load and improve latency

The single biggest Cinder lever is rbd_exclusive_cinder_pool. When it’s false, the driver queries every image’s size independently, which gets expensive fast in large-volume fleets. Leaving it true avoids that per-image overhead entirely and is the recommended setting for anything beyond a small pilot.

A few other Cinder options matter too:

• rbd_store_chunk_size controls object size for new volumes, affecting how writes stripe across OSDs.
• rbd_max_clone_depth limits how many layers of copy-on-write clones stack before Cinder flattens a volume, which matters for boot-from-volume performance.
• rados_connect_timeout protects against a hung RADOS session blocking the whole Cinder service.

On the Nova side, use cache="none" or cache="writeback" for the libvirt disk driver depending on whether your guest filesystem issues flushes reliably; none is the safer production default. After any change, watch Cinder API latency, OSD op latency, and the ratio of client reads to writes in ceph -s to confirm the change helped rather than just shifted load elsewhere.

Pool design and CRUSH considerations for block workloads

Erasure-coded pools cut raw storage cost but add CPU overhead and latency on small, random I/O, so they fit cold or archival data better than active boot disks.

• Use device-class-aware CRUSH rules to put hot Cinder volumes on NVMe-backed pools and cold snapshots or backups on HDD pools.
• Size placement groups for even distribution, but resist the urge to raise PG counts reflexively. Lab testing on very fast clusters shows PG counts have dramatic effects on throughput, but the same source cautions that blind PG increases without measurement are not a fix, since they can also increase PG lock contention.
• Set rbd_read_from_replica_policy to balance read-sourcing across OSDs when your cluster has asymmetric network paths or you want to spread read load off the primary.

OSD and BlueStore tuning: memory, cache autotune, and RocksDB sizing

BlueStore’s cache autotune is the setting most engineers should leave alone rather than fight. bluestore_cache_autotune, paired with osd_memory_target, dynamically balances cache size against actual memory pressure, which avoids the manual thrashing that comes from hardcoding cache sizes on mixed workloads. osd_memory_cache_min sets a floor so autotune doesn’t starve the cache under load.

Where manual sizing still matters is RocksDB’s block.db device. A common practitioner heuristic is sizing it at roughly 1 to 2% of primary storage capacity on fast NVMe, since metadata-heavy small-random I/O depends heavily on DB device speed, not spinning-disk throughput.

• If performance counters show RocksDB index thrashing alongside rising bluefs.read_bytes, consider lowering rocksdb_cache_shard_bits before increasing cache size outright.
• Watch OSD thread shard counts and async messenger thread settings together. Lab benchmarks found OSD threading changes materially affect throughput and IOPS, though the right shard count depends on core count and workload mix.
• For OSDs holding millions of small objects, multithreaded onode scanning can cut recovery time from minutes to seconds on NVMe-backed DB devices.

Pro Tip: Reduce rocksdb_cache_shard_bits as a targeted fix for compaction-related latency spikes rather than a general-purpose knob; it trades a little extra lock contention for a lot less index thrashing.

Network and host-level tuning that unlock Ceph performance

Before touching the cluster config, fix the hosts underneath it using tools like Kubezilla Desktop for streamlined Kubernetes management and monitoring. Host-level and network changes frequently deliver outsized gains compared to Ceph-only tuning, according to operator benchmarks from a recent Ceph performance review.

1. Set the CPU governor to performance and pin C-states with processor.max_cstate=1 to eliminate wake-latency jitter on OSD and MON hosts.
2. Size NICs to the actual replication and client traffic you expect, and tune async messenger thread counts to match available cores.
3. Put RocksDB’s block.db on NVMe, since DB device speed drives both compaction behavior and OSD boot and recovery times.
4. Roll changes out host by host, watching CPU utilization, network throughput, and Ceph OSD perf counters before moving to the next node.

Benchmarking, validation, and rollback: fio, Ceph counters, and staged rollouts

Never tune blind. The same benchmark review recommends fio as the standard tool for simulating VM-like I/O patterns.

1. Run a 4k random read/write test at iodepth=1 to measure IOPS and tail latency, the pattern closest to a typical guest filesystem.
2. Run a 4MB sequential test to measure raw throughput ceilings.
3. Capture ceph -s, OSD perf dumps, and RocksDB cache hit/miss ratios before and after each change.
4. Roll out one host or pool at a time, watching for backfill pressure, and keep the previous config value on hand for immediate rollback.
5. Save config diffs and metric snapshots together; staged changes with recorded snapshots are what let you tell a real improvement from noise.

Pro Tip: Never trust a single fio run. Repeat each test at least twice after a rollout; the second run filters out cache-warming artifacts from the first.

Hyperconverged deployments: resource isolation and OpenStack integration notes

When Nova and Ceph OSDs share hosts, resource contention is the real risk, not raw tuning. Start mgr/cephadm/autotune_memory_target_ratio at 0.2 on hyperconverged nodes and leave osd_memory_target_autotune enabled so OSD memory adjusts automatically as VM density changes.

• Reserve host memory for OSDs using Nova’s NovaReservedHostMemory, and adjust cpu_allocation_ratio downward to account for OSD CPU usage.
• Limit OSD container CPU allocation on HCI nodes so a backfill event doesn’t starve Nova’s scheduler or running VMs.
• Bake tuned defaults into an initial-ceph.conf used with openstack overcloud ceph deploy, so new nodes inherit safe settings instead of starting from scratch.

A pragmatic, ops-first take on Ceph tuning

The biggest mistake we see is engineers overriding autotune settings because a blog post told them a specific number worked somewhere else. Autotune exists because workload mix varies, and a fixed cache size that helped one cluster can thrash another. Change one thing, measure it, and keep the rollback path ready. For a deeper playbook, our Ceph and OpenStack integration tuning checklist walks through the same sequence in runbook form.

— James

How our team can help you tune Ceph without the guesswork

Working through RBD cache policies, RocksDB sizing, and host tuning on a live production cluster takes time most teams don’t have free. Our OpenStack / Kolla-Ansible Review walks through your current Ceph and Cinder configuration and flags the highest-impact changes for your workload, priced at $450 one-off. For teams managing storage with no spare hours at all, our Managed OpenStack and Kolla-Ansible Services handle ongoing tuning and operations directly.

• Get a focused OpenStack / Kolla-Ansible Review scoped to your Ceph and Cinder configuration.
• Hand off ongoing tuning and day-to-day operations through Managed OpenStack and Kolla-Ansible Services.
• Pull ready-made runbooks from our OpenStack Troubleshooting Toolkit to apply these changes yourself.

If you’re ready to move, book a review and we’ll start with your current config and your latest fio numbers.

FAQ

What is the safest first Ceph tuning change to make?

Baseline your current performance with fio before changing anything, then start with RBD client cache settings since they’re reversible per-client without touching the cluster. Raising rbd_cache_max_dirty past its 24 MiB default is a common low-risk first step.

Should I leave BlueStore memory autotune enabled?

Yes, in almost all cases. BlueStore’s autotune dynamically balances OSD memory and cache size, and manual overrides should only follow clear evidence from measurement that a fixed value performs better for your specific workload.

Why does rbd_exclusive_cinder_pool matter so much?

Leaving rbd_exclusive_cinder_pool set to true stops the Cinder driver from issuing a separate size query for every image, which adds up quickly in large-volume fleets and otherwise becomes one of the heavier sources of unnecessary load on the cluster.

How big should my RocksDB block.db device be?

A common operational guideline sizes RocksDB’s block.db at roughly 1 to 2% of primary storage capacity on fast NVMe, which keeps metadata lookups fast for small, random I/O patterns typical of VM volumes.

Can I increase placement groups to fix slow performance?

Not blindly. Lab testing shows PG counts can meaningfully affect throughput on very fast clusters, but the same benchmarks warn against raising PG counts without measurement, since it can increase lock contention instead of helping.

Sources

• Ceph block device cache settings — Red Hat Ceph Storage documentation
• Cinder RBD driver configuration — OpenStack documentation
• BlueStore configuration reference — Ceph documentation
• Ceph

Recommended

• Ceph + OpenStack Integration Tuning Prompt
• Tuning Galera Flow Control for OpenStack Databases
• Managing Glance Images at Scale in OpenStack
• Integrating Cinder With NetApp and NFS Backends in OpenStack