Rehearsed runbook OpenStack upgrade strategy for US production clouds

Use a staged rolling upgrade with N+1 hops: upgrade one release at a time, put placement ahead of nova, pin RPC versions so old and new services can talk, and never skip a verified database backup. Before you touch a maintenance window, read the release notes for breaking changes, run a full restore drill, and rehearse the entire sequence in staging. If you do nothing else this week, schedule that rehearsal window now.

---

TL;DR:

• Conduct a phased, N+1 rolling upgrade with careful staging, ensuring placement is upgraded first and RPC versions are pinned during the transition.
• Verify backups with a full restore test and review release notes for all intermediate versions before starting the upgrade process.
• Sequence OpenStack services by upgrading placement first, followed by nova-conductor, nova-api, and then compute nodes in manageable batches.
• Keep RPC version pins in place until all services, including compute nodes, report the new version, and perform controlled load balancer drains during shutdowns.
• Rehearse the entire upgrade and rollback procedures in staging to identify telemetry blind spots and avoid common failure points during production deployment.

---

Table of Contents

• Choosing the Right OpenStack Upgrade Strategy
• What Should You Check Before Starting an OpenStack Upgrade?
• Which Order Should You Upgrade OpenStack Services In?
• RPC Pinning and Graceful Shutdown: The Mechanics That Keep Mixed Versions Talking
• How Do Online Data Migrations Work During an OpenStack Upgrade?
• Automating Upgrades Without Losing Control of the Process
• Testing, Rollback, and the Validation Gates That Catch Problems Early
• Your OpenStack Upgrade Checklist for the Maintenance Window
• Lessons From Watching OpenStack Upgrades Go Right and Wrong
• How Devopsaitoolkit Supports Your Next OpenStack Upgrade
• Where to Verify the Official OpenStack Upgrade Guidance
• Sources
• FAQ

Choosing the Right OpenStack Upgrade Strategy

Three patterns dominate real production clouds, and picking the wrong one is how a four-hour maintenance window turns into a weekend.

In-place rolling upgrades move services one release at a time while the cluster keeps serving traffic. You upgrade a subset of nodes, let them run alongside the old version, then finish the batch. This is the pattern most OpenStack projects are built to support, and it’s the default for a reason: it needs no duplicate infrastructure and keeps rollback relatively contained to whatever batch you just touched.

Blue-green control-plane cutovers stand up a second, upgraded control plane next to the old one, validate it, then flip traffic over. It’s the most reversible option. You can point traffic back with a load balancer change instead of a package downgrade. The cost is real: double the control-plane hardware for the overlap window, plus the complexity of keeping two stacks in sync during the dual-run period.

Fast-forward, hop-by-hop upgrades skip intermediate releases when a cloud has fallen multiple versions behind. They save calendar time but multiply risk, since you’re absorbing several releases’ worth of breaking changes and data migrations in one pass.

For most production clouds, the safest default is a rolling N+1 upgrade, rehearsed in staging until the runbook stops surprising you:

• Rolling N+1: lowest infrastructure overhead, moderate rollback complexity, best rehearsal-to-confidence ratio
• Blue-green: highest reversibility, highest cost, best for control planes that can’t tolerate any risk
• Fast-forward: fastest calendar time, highest risk, only for clouds already dangerously behind on versions

What Should You Check Before Starting an OpenStack Upgrade?

Every OpenStack upgrade strategy lives or dies on what happens before the maintenance window opens, not during it.

1. Read the release notes release by release. Never skip from your current version to the target without reading every intermediate release’s notes. Deprecated config options, renamed flags, and removed drivers hide in these documents, and they are the single most common cause of a failed upgrade that looked fine on paper.
2. Run a full database backup, then prove you can restore it. A backup nobody has restored is a hope, not a plan. Take a full dump, restore it to a scratch instance, and confirm the schema and row counts match before you schedule anything.
3. Inventory your custom configuration. Any override to a default policy file, scheduler filter, or driver setting needs a name and an owner before the upgrade starts, because upgrade scripts assume defaults.
4. Run preflight checks against cluster health. Confirm database quorum, load balancer node status, disk headroom on controllers, message queue depth, and the health of every service you’re about to touch.

Nova’s own upgrade documentation treats backups and preflight checks as non-negotiable steps, not optional hygiene, and for good reason: nova-conductor and nova-api are the two services most likely to throw errors mid-migration if the database isn’t in a known-good state.

Pro Tip: Keep a running diff of every config file against the project’s sample config for each release. When something breaks three versions later, that diff is the fastest way to find out whether you’re the cause.

Which Order Should You Upgrade OpenStack Services In?

Sequence matters more than almost anything else in an OpenStack migration plan. Get the order wrong and you’ll spend the maintenance window chasing API errors that have nothing to do with the actual bug.

• Placement goes first, always. Nova, and increasingly other services, depend on placement for resource tracking. Upgrading nova before placement leaves nova talking to an API that doesn’t understand its requests.
• Nova follows in a fixed sequence: nova-conductor, then nova-api, then nova-compute nodes in batches. Conductor brokers database access for compute nodes, so it has to understand the new schema before computes start reporting against it.
• Batch your compute node upgrades. Never upgrade every hypervisor at once. Take a slice, verify instance operations still work, then move to the next slice.
• **Glance uses an expand, migrate, contract pattern for rolling upgrades.

Bring a new node online that expands the database schema, let it join the load balancer rotation, upgrade the remaining nodes one at a time, then contract the schema once every node runs the new release, exactly as the Glance rolling-upgrade guide lays out.

• Check Ceph, RabbitMQ, and Galera version prerequisites before you start the OpenStack upgrade itself. RabbitMQ quorum queues in particular have specific version floors, and Galera cluster nodes need to agree on wsrep versions before you touch the services sitting on top of them.

RPC Pinning and Graceful Shutdown: The Mechanics That Keep Mixed Versions Talking

This is where most OpenStack upgrade challenges actually surface, because it’s the part where old code and new code have to cooperate for real, not just coexist on paper.

RPC version pinning, set through [upgrade_levels], tells newer services to speak an older message format so they stay compatible with services you haven’t upgraded yet. Set compute=auto (or the specific prior release name) before you touch nova-compute nodes, and leave it pinned until every compute node is running the new code. Only then do you remove the pin and let services negotiate their native version. Nova’s upgrade documentation treats this pinning as the mechanism that makes rolling upgrades survivable rather than theoretical.

Shutdown signals matter just as much as pinning:

• Send SIGTERM to a service you’re taking down for upgrade. It finishes in-flight work and exits cleanly instead of dropping requests mid-transaction.
• Send SIG_HUP when you only need a config reload, not a full restart, which avoids an unnecessary worker cycle.
• Add workers back gradually after a restart rather than all at once, so a newly restarted service doesn’t get slammed with a full queue backlog the instant it comes online.

Your load balancer needs its own drain sequence. Pull a control-plane node out of rotation, wait for its in-flight connections to close, upgrade it, run a health check, then add it back before pulling the next node. Skipping the drain step is how you generate a burst of client-side 500 errors that has nothing to do with the upgrade itself and everything to do with yanking a live node out of a pool.

Pro Tip: Keep the RPC pin in place for one full upgrade cycle longer than you think you need. Removing it too early, right after the last compute node reports healthy, is a common source of intermittent RPC errors that only show up under load a day or two later.

How Do Online Data Migrations Work During an OpenStack Upgrade?

Schema changes during a live upgrade follow an expand, migrate, contract workflow, and skipping a phase is how you end up with a database that neither the old nor the new code can read correctly.

1. Expand: add new columns or tables without removing anything the old code still depends on.
2. Migrate: move data into the new structures while both old and new code paths remain functional against the database.
3. Contract: drop the old columns or tables only after every service instance is confirmed running the new release.

Run the actual data migration with nova-manage db online_data_migrations --max-count, which processes a bounded batch of rows and returns an exit code telling you whether more work remains. Rerun the command until it reports zero remaining rows, as Nova’s admin documentation specifies.

Chunk size is a database capacity decision, not a script default. A smaller --max-count value spreads migration load over more passes but keeps each pass gentle on a busy Galera cluster; a larger value finishes faster but risks lock contention on tables still serving live reads and writes. Watch write latency and replication lag on your database during the run, and pause between passes if either climbs. There’s no universal safe chunk size across every deployment, so treat your first migration run in staging as the calibration exercise for production.

Automating Upgrades Without Losing Control of the Process

Automation speeds up an OpenStack upgrade process, but it doesn’t replace judgment about when to stop and look.

OpenStack-Ansible ships run-upgrade.sh along with playbooks that upgrade hosts, infrastructure services, and OpenStack components in sequence. Run its preflight checks first, since the project’s own major-upgrade documentation flags RabbitMQ and Ceph as services that sometimes need manual attention the script won’t catch on its own.

TripleO uses a different model: prepare, run, converge. You prepare the new container images and environment files, run the upgrade against the overcloud stack, then converge to confirm the stack’s actual state matches what Heat expects, following the sequence TripleO’s upgrade workflow documents.

Juju charms generally support only single-step N+1 upgrades. There’s no built-in fast-forward path, so multi-release jumps have to be broken into individual hops handled manually between charm runs.

• Keep every playbook and upgrade script under version control, not just the target infrastructure code.
• Build a rollback toggle into your automation rather than writing rollback steps only for the manual path.
• Never let an automation run continue unattended through a service you haven’t rehearsed upgrading before.

Testing, Rollback, and the Validation Gates That Catch Problems Early

A rehearsal environment only earns its keep if it forces the same failure modes production would produce. That means testing actual user journeys (booting an instance, attaching a volume, live-migrating a VM) not just confirming that services start.

1. Build the rehearsal environment from the same database size class and config inventory as production, not a scaled-down toy version.
2. Rehearse the full sequence: backup, preflight, service order, RPC pinning, data migrations, and unpinning, in that order, at least once before the real window.
3. Define numeric health gates before you start: API error rate under a fixed threshold, message queue depth below a set ceiling, and scheduler throughput holding steady. If a gate fails, you stop and diagnose before moving to the next batch.
4. Time-box your rollback decision. If a batch isn’t healthy within your predefined window, restore configs, restore the database snapshot, and downgrade packages rather than pushing forward hoping it self-corrects.
5. Collect logs, health-gate metrics, and the exact command history from the window as evidence for post-upgrade review or audit.

Pro Tip: Practitioner field guides recommend reserving some extra compute and database capacity headroom before an upgrade window, as described in VEXXHOST’s operator playbook, specifically so a batch that runs hot during migration doesn’t force an emergency rollback you could have avoided with slack capacity.

Your OpenStack Upgrade Checklist for the Maintenance Window

Print this, tape it to the monitor, and don’t skip a line because the last upgrade went smoothly.

Before the window opens:

• Confirm backups are restored and verified, not just taken.
• Reserve capacity buffer on compute and database tiers.
• Annotate release notes and your config inventory with what’s changing for this specific hop.

During cutover:

• Follow service order exactly: placement, nova-conductor, nova-api, nova-compute batches.
• Monitor database write load continuously through every migration pass.
• Run online_data_migrations in chunks and confirm a zero-remaining exit status before moving on.
• Remove RPC version pins only after every service reports the new version.

After the window closes:

• Contract the database schema once all nodes confirm the new release.
• Verify a sample of real tenant workloads, not just service health checks.
• Run smoke tests and archive logs and metrics as your upgrade evidence.

Lessons From Watching OpenStack Upgrades Go Right and Wrong

The gap between a smooth OpenStack upgrade and a three-day incident almost never comes down to a missing feature or an unsupported driver. It comes down to telemetry gaps operators didn’t know they had until the upgrade exposed them. A queue depth alert that only fires at a threshold nobody has hit before. A database dashboard that shows connections but not lock wait time. Fix those blind spots before your rehearsal, not during your production window.

The biggest pattern I’ve seen in failed upgrades: teams rehearse the happy path and skip the failure path. They test that the upgrade works, but never test that the rollback works. A runbook that only documents forward motion isn’t a runbook, it’s a hope with extra steps. Document every config delta between releases as you find it, because that log becomes the fastest diagnostic tool the next time something behaves strangely three versions later. If you’re still building out your production baseline, our guide to production-ready OpenStack clouds covers the observability foundations that make rehearsals actually predictive.

— James

How Devopsaitoolkit Supports Your Next OpenStack Upgrade

Every step in this OpenStack upgrade strategy, from the release-note diff to the rollback runbook, is faster to execute when you’re not building the templates from scratch under deadline pressure. There are tools and resources that provide version-controlled runbook templates, AI prompt libraries built for Linux and OpenStack workflows, and audits to help with upgrade planning before the window opens.

None of this replaces the work covered above. Reading release notes, rehearsing in staging, and verifying your backups still has to happen no matter what tools sit on your desk. What Devopsaitoolkit adds is speed and consistency: instead of writing your preflight checklist and RPC pinning notes from memory every release cycle, you start from a tested template and adjust for what changed. If you manage Linux hosts alongside your OpenStack estate, the Linux admin prompt library covers the surrounding automation work, and the pricing page breaks down what’s included in an audit versus a self-serve toolkit. Start at Devopsaitoolkit and pick the option that matches how hands-on you want this next upgrade to be.

Where to Verify the Official OpenStack Upgrade Guidance

Bookmark the primary sources and subscribe to their release notes rather than relying on secondhand summaries: Nova’s admin upgrade docs, Glance’s rolling upgrade guide, OpenStack-Ansible’s major upgrade docs, and TripleO’s upgrade workflow. Each project updates these pages faster than any third-party guide can track.

FAQ

What Is the Safest OpenStack Upgrade Strategy for Production?

A staged rolling upgrade with N+1 hops is the safest default: upgrade one release at a time, keep placement ahead of nova, pin RPC versions during the transition, and verify database backups before starting.

How Long Should an OpenStack Rolling Upgrade Take?

There’s no fixed duration since it depends on cluster size and how many compute batches you run, but rehearsing the full sequence in staging first is what lets you estimate your own window accurately instead of guessing.

Can You Skip Versions During an OpenStack Migration Plan?

Fast-forward or hop-by-hop upgrades exist for clouds that have fallen multiple releases behind, but they carry higher risk because you absorb several releases’ worth of breaking changes and data migrations at once.

When Should You Remove RPC Version Pins After Upgrading OpenStack Components?

Remove [upgrade_levels] pins only after every service instance, including every compute node, reports the new version, and leave them in place slightly longer than you think necessary to avoid intermittent RPC errors under load.

What’s the Biggest Cause of Failed OpenStack Upgrades?

Skipping the rollback rehearsal is the most common failure pattern. Teams test that the forward upgrade path works but never validate that config and database restores actually succeed under time pressure.

Recommended

• Planning OpenStack Upgrades Safely Without Downtime
• How to Build a Production-Ready OpenStack Cloud (2026 Guide)
• Automating OpenStack with the Python SDK and CLI
• Deploying OpenStack with Kolla-Ansible: A Practical Guide