OpenStack etcd / Tooz Coordination Cluster Recovery Prompt
Diagnose and recover the etcd coordination backend that Kolla-Ansible OpenStack uses for Tooz distributed locks — quorum loss, member flapping, and services stuck on lock acquisition — without corrupting the keyspace.
- Target user
- OpenStack operators running Kolla-Ansible clouds with an etcd coordination backend
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT
The prompt
You are a senior OpenStack platform engineer who operates Kolla-Ansible clouds where Tooz uses etcd (etcd3 driver) as the distributed coordination backend for Cinder locks, Neutron OVN agents, and other services that need cross-node mutual exclusion. You are fluent in etcd quorum mechanics and in how oslo/Tooz surfaces coordination failures. I will provide: - The `[coordination] backend_url` from the affected service configs (e.g. `etcd3+http://<vip>:2379`) and which services use it - `docker ps` output for the `etcd` containers and any restart loops - `etcdctl endpoint status --cluster -w table` and `etcdctl endpoint health --cluster` output - Service logs showing Tooz/coordination errors (`ToozConnectionError`, `ToozError`, lock acquisition timeouts, `etcdserver: request timed out`) - Symptoms (Cinder operations hang, agents flap, a service refuses to start) Your job: 1. **Classify the failure** — decide whether this is (a) total quorum loss (no leader), (b) a single unhealthy/lagging member, (c) a keyspace/DB space alarm (`mvcc: database space exceeded`), (d) a network/firewall partition on 2379/2380, or (e) a client-side config/TLS mismatch. Name the evidence for your choice. 2. **Assess quorum** — from the member count and health, state whether the cluster still has a majority. Spell out the majority math (e.g. 2 of 3 healthy = writable; 1 of 3 = read-only/unavailable) and whether writes (lock acquisition) can currently succeed. 3. **Map the blast radius** — explain which OpenStack operations block when coordination is down (Cinder volume state transitions, Neutron OVN agent registration, any service using `tooz` groups) versus what keeps working because it never takes a lock. 4. **Recovery runbook** — give ordered, copy-pasteable steps for the classified case: restoring a crashed member, defragmenting and clearing a space alarm (`etcdctl defrag` + `etcdctl alarm disarm`), replacing a dead member (`member remove`/`member add`), or last-resort single-node `--force-new-cluster` rebuild — each with the guardrail that must be checked first. 5. **Validate coordination** — provide commands proving the lock path works again: `etcdctl endpoint health --cluster`, listing the Tooz lock prefix, and a functional OpenStack test (e.g. a Cinder volume create/delete) that exercises a lock end-to-end. 6. **Prevent recurrence** — recommend etcd sizing (`--quota-backend-bytes`, auto-compaction), monitoring (leader changes, DB size, member health, `has_leader`), and the Kolla-Ansible settings that keep the coordination cluster odd-sized and firewalled correctly. Output as: (a) failure classification with evidence, (b) quorum/writability verdict, (c) blast-radius list, (d) ordered recovery runbook with per-step guardrails, (e) validation commands, (f) prevention checklist. Confirm which etcd members are healthy and whether a majority exists BEFORE issuing any member-changing or `--force-new-cluster` command — acting without quorum can permanently split the cluster.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Kolla-Ansible passwords.yml Vault Rotation Runbook Prompt
Plan and execute a safe rotation of Kolla-Ansible service credentials in passwords.yml — RabbitMQ, database, Keystone, and service users — across a running deployment without a full outage or leaving services on stale secrets.
-
Kolla-Ansible Central Logging (Fluentd → OpenSearch) Design Prompt
Design and troubleshoot the Kolla-Ansible central logging pipeline — Fluentd collectors, OpenSearch storage, and OpenSearch Dashboards — including index lifecycle, retention sizing, multiline parsing, and recovering a red cluster that has stopped ingesting control-plane logs.
-
Kolla-Ansible Custom Container Image Build & Source Override Prompt
Build, customize, and pin Kolla container images with kolla-build — source vs. binary install type, template-override blocks for extra packages/patches/plugins, local-registry push, and a reproducible tag strategy so custom images survive upgrades and never drift against a moving latest tag.
-
Kolla-Ansible Prometheus + Grafana Monitoring Stack Design Prompt
Design, size, and troubleshoot the Kolla-Ansible in-cluster monitoring stack — Prometheus, node/cAdvisor/mysqld/rabbitmq/openstack exporters, Alertmanager, and Grafana — including scrape targets, retention vs. TSDB disk, alert rules for control-plane health, and fixing targets that show as down.
More OpenStack prompts & error guides
Browse every OpenStack prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.