OpenStack Production Troubleshooting

Diagnostic commands run in order — each one narrows the fault

1. Reproduce the timeout and time it

   time openstack --debug server list 2>&1 | tail -30

   Copy

   A response at almost exactly the proxy timeout (commonly 60s) confirms the proxy gave up, not the client.

2. Check HAProxy backend health

   echo "show stat" | sudo socat stdio /var/run/haproxy/admin.sock | cut -d, -f1,2,18,19

   Copy

   Any backend not in status UP is the layer that failed. This is the single fastest way to localize a 504.

3. Check which service container is unhealthy

   sudo docker ps --filter "health=unhealthy" --format "{{.Names}}\\t{{.Status}}"

   Copy

   On a Kolla-Ansible deployment the container health check usually fails before the API does.

4. Read the service API log for slow requests

   sudo tail -200 /var/log/kolla/keystone/keystone-apache-admin-access.log | awk '$NF > 5000000'

   Copy

   Apache logs request duration in microseconds. Requests over ~5s are the ones dragging the proxy toward its timeout.

Diagnostic commands run in order — each one narrows the fault

1. Check cluster and partition status

   sudo docker exec rabbitmq rabbitmqctl cluster_status

   Copy

   A non-empty `partitions` section means a network partition — RPC will time out until it is resolved.

2. Find queues with backed-up messages

   sudo docker exec rabbitmq rabbitmqctl list_queues name messages consumers --no-table-headers | awk '$2 > 100'

   Copy

   A queue with messages and ZERO consumers is a service that has stopped consuming — that is your failing component.

3. Check for blocked connections

   sudo docker exec rabbitmq rabbitmqctl list_connections name state | grep -v running

   Copy

   "blocked" or "blocking" means RabbitMQ hit a memory or disk alarm and is applying backpressure to publishers.

4. Check the resource alarms

   sudo docker exec rabbitmq rabbitmqctl status | grep -A5 alarms

   Copy

   Any raised alarm stalls the whole message bus and surfaces as timeouts in every OpenStack service at once.

Diagnostic commands run in order — each one narrows the fault

1. Check compute service state

   openstack compute service list --long

   Copy

   A service that is `enabled` but `down` has stopped reporting to the conductor — check its host clock and the message bus first.

2. Read the scheduler failure reason

   sudo tail -200 /var/log/kolla/nova/nova-scheduler.log | grep -iE "no valid host|filter" | tail -20

   Copy

   "No valid host was found" is always followed by which filter rejected every candidate — that filter names the real constraint.

3. Check hypervisor capacity

   openstack hypervisor stats show

   Copy

   Compare free vcpus/ram/disk against the flavor being requested; also check the allocation ratios before assuming exhaustion.

Diagnostic commands run in order — each one narrows the fault

1. Check agent state

   openstack network agent list --long

   Copy

   An agent with `alive = XXX` has missed its heartbeats. Check its host before touching any router.

2. Inspect the router namespace

   sudo ip netns list | grep qrouter && sudo ip netns exec qrouter-<ID> ip -brief address

   Copy

   A router namespace with no addresses (or missing entirely) explains total loss of external connectivity for that tenant.

3. Read the L3 agent log

   sudo tail -200 /var/log/kolla/neutron/neutron-l3-agent.log | grep -iE "error|traceback" | tail -20

   Copy

   Look for the first traceback after the last successful sync — the ones after it are usually cascade failures.

Diagnostic commands run in order — each one narrows the fault

1. Check volume service state

   openstack volume service list

   Copy

   A `cinder-volume` backend that is down means every create on that backend will sit in "creating" until it times out.

2. Read the scheduler decision

   sudo tail -200 /var/log/kolla/cinder/cinder-scheduler.log | grep -iE "no valid|filter|capab" | tail -20

   Copy

   Names the backend filter that rejected the request — usually capacity, or a backend that never reported capabilities.

3. Check a stuck volume

   openstack volume show <VOLUME_ID> -f value -c status -c "os-vol-host-attr:host"

   Copy

   The host attribute tells you which backend owns it; a volume stuck in "creating" with no host never got scheduled at all.