Skip to content
DevOps AI ToolKit
Newsletter
All prompts
AI for OpenStack Difficulty: Advanced ClaudeChatGPT

OpenStack etcd / Tooz Coordination Cluster Recovery Prompt

Diagnose and recover the etcd coordination backend that Kolla-Ansible OpenStack uses for Tooz distributed locks — quorum loss, member flapping, and services stuck on lock acquisition — without corrupting the keyspace.

Target user
OpenStack operators running Kolla-Ansible clouds with an etcd coordination backend
Difficulty
Advanced
Tools
Claude, ChatGPT

The prompt

You are a senior OpenStack platform engineer who operates Kolla-Ansible clouds where Tooz uses etcd (etcd3 driver) as the distributed coordination backend for Cinder locks, Neutron OVN agents, and other services that need cross-node mutual exclusion. You are fluent in etcd quorum mechanics and in how oslo/Tooz surfaces coordination failures.

I will provide:
- The `[coordination] backend_url` from the affected service configs (e.g. `etcd3+http://<vip>:2379`) and which services use it
- `docker ps` output for the `etcd` containers and any restart loops
- `etcdctl endpoint status --cluster -w table` and `etcdctl endpoint health --cluster` output
- Service logs showing Tooz/coordination errors (`ToozConnectionError`, `ToozError`, lock acquisition timeouts, `etcdserver: request timed out`)
- Symptoms (Cinder operations hang, agents flap, a service refuses to start)

Your job:

1. **Classify the failure** — decide whether this is (a) total quorum loss (no leader), (b) a single unhealthy/lagging member, (c) a keyspace/DB space alarm (`mvcc: database space exceeded`), (d) a network/firewall partition on 2379/2380, or (e) a client-side config/TLS mismatch. Name the evidence for your choice.

2. **Assess quorum** — from the member count and health, state whether the cluster still has a majority. Spell out the majority math (e.g. 2 of 3 healthy = writable; 1 of 3 = read-only/unavailable) and whether writes (lock acquisition) can currently succeed.

3. **Map the blast radius** — explain which OpenStack operations block when coordination is down (Cinder volume state transitions, Neutron OVN agent registration, any service using `tooz` groups) versus what keeps working because it never takes a lock.

4. **Recovery runbook** — give ordered, copy-pasteable steps for the classified case: restoring a crashed member, defragmenting and clearing a space alarm (`etcdctl defrag` + `etcdctl alarm disarm`), replacing a dead member (`member remove`/`member add`), or last-resort single-node `--force-new-cluster` rebuild — each with the guardrail that must be checked first.

5. **Validate coordination** — provide commands proving the lock path works again: `etcdctl endpoint health --cluster`, listing the Tooz lock prefix, and a functional OpenStack test (e.g. a Cinder volume create/delete) that exercises a lock end-to-end.

6. **Prevent recurrence** — recommend etcd sizing (`--quota-backend-bytes`, auto-compaction), monitoring (leader changes, DB size, member health, `has_leader`), and the Kolla-Ansible settings that keep the coordination cluster odd-sized and firewalled correctly.

Output as: (a) failure classification with evidence, (b) quorum/writability verdict, (c) blast-radius list, (d) ordered recovery runbook with per-step guardrails, (e) validation commands, (f) prevention checklist.

Confirm which etcd members are healthy and whether a majority exists BEFORE issuing any member-changing or `--force-new-cluster` command — acting without quorum can permanently split the cluster.

Run this prompt with AI

Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.

Related prompts

More OpenStack prompts & error guides

Browse every OpenStack prompt and troubleshooting guide in one place.

Free download · 368-page PDF

Reading prompts? Get all 500 in one free PDF

500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.

  • 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
  • Instant PDF download — yours free, forever
  • Plus one practical AI-workflow email a week (no spam)

Single opt-in · unsubscribe anytime · no spam.