🚨 OpenStack Error Guide: 'Layer7 wrong status' — HAProxy Backend DOWN on the Control-Plane VIP

Overview

In a Kolla-Ansible deployment, every OpenStack API is fronted by HAProxy on a control-plane VIP (managed by keepalived). HAProxy load-balances each API — Keystone, Nova, Neutron, Glance, Cinder — across the controller nodes. When the backend servers for a service fail their health checks, HAProxy marks them DOWN, and once every server in a backend is down HAProxy returns 503 Service Unavailable or the request simply times out.

The literal errors you will see in the HAProxy log:

Server nova_api/controller-02 is DOWN, reason: Layer7 wrong status, code: 503, check duration: 4ms. 2 active and 0 backup servers left. 0 sessions active...
backend nova_api has no server available!

And from the client side hitting the VIP:

$ openstack server list
Service Unavailable (HTTP 503)

The tell is that the API on the VIP fails, but the service container on an individual controller may still be reachable directly on its own host/port. The problem is between HAProxy and the backend — the backend service is dead, unhealthy, or too slow — not HAProxy itself.

Symptoms

• API calls through the VIP return 503 Service Unavailable or hang until the client times out.
• docker logs haproxy shows Server <backend>/<host> is DOWN and backend <name> has no server available!.
• One service is affected (e.g. only Glance 503s) while others on the same VIP work fine.
• The affected service’s container is Restarting, Exited, or unresponsive on a controller.

docker exec haproxy sh -c 'echo "show stat" | socat stdio /var/lib/kolla/haproxy/haproxy.sock' \
  | awk -F, '$18=="DOWN" || $18=="MAINT" {print $1"/"$2" -> "$18}'

glance_api/controller-01 -> DOWN
glance_api/controller-02 -> DOWN
glance_api/controller-03 -> DOWN

All three servers DOWN in one backend means every request to that service 503s.

Common Root Causes

1. The backend service container is down or OOM-killed

If the API container crashed, was OOM-killed, or is stuck restarting, HAProxy’s health check fails and marks it DOWN.

docker ps -a --filter name=glance_api --format '{{.Names}} {{.Status}}'
docker inspect glance_api --format '{{.State.OOMKilled}} {{.State.ExitCode}}'

glance_api Restarting (137) 6 seconds ago
true 137

Exit 137 with OOMKilled=true means the kernel killed it for memory.

2. The service is up but failing its health check

Kolla health checks hit an API path; if the service returns 500/503 (e.g. it can’t reach its database or Keystone), HAProxy sees Layer7 wrong status and marks it DOWN even though the process is running.

docker exec haproxy sh -c 'echo "show stat" | socat stdio /var/lib/kolla/haproxy/haproxy.sock' \
  | awk -F, '$1=="glance_api"{print $2, $18, $37, $38}'    # svname status check_status last_chk

controller-01 DOWN L7STS L7STS 503

L7STS / code 503 means the backend answered but with a bad HTTP status — look at that service’s own log next.

3. A dependency the backend needs is broken

The container is healthy but its own downstream — MariaDB (Galera), RabbitMQ, or Keystone — is unreachable, so it returns errors on every request.

docker logs glance_api 2>&1 | grep -iE "error|refused|timeout|unavailable" | tail -5

ERROR ... OperationalError: (2003, "Can't connect to MySQL server on '<vip>' (111)")

4. HAProxy timeouts too tight for a slow backend

A slow image upload or a heavy Nova list can exceed timeout server/timeout client in haproxy.cfg, so HAProxy tears the connection down and the client sees a timeout or 504.

docker exec haproxy grep -E 'timeout (client|server|connect|check)' /etc/haproxy/haproxy.cfg

    timeout client   1m
    timeout server   1m
    timeout connect  10s

A 1-minute timeout server will cut off a large Glance upload mid-flight.

5. Firewall / network between HAProxy and the backend host

A firewall change or routing blip blocks HAProxy from reaching the backend port, producing Layer4 connection problem in the check status.

docker exec haproxy sh -c 'echo "show stat" | socat stdio /var/lib/kolla/haproxy/haproxy.sock' \
  | awk -F, '$1=="nova_api"{print $2,$18,$38}'

controller-03 DOWN L4CON

L4CON (Layer4 connection problem) points at network/port reachability, not the app.

Diagnostic Workflow

Step 1: Ask HAProxy which backends are down (read-only)

docker exec haproxy sh -c 'echo "show stat" | socat stdio /var/lib/kolla/haproxy/haproxy.sock' \
  | awk -F, 'NR==1{next} $2!="FRONTEND" && $2!="BACKEND" {print $1"/"$2" -> "$18" ("$38")"}'

$18 is status (UP/DOWN), $38 is the check result (L7STS, L4CON, L7OK). This tells you whether the failure is app-layer or network-layer.

Step 2: Read the HAProxy log for the transition and reason

docker logs haproxy 2>&1 | grep -iE "is DOWN|no server available" | tail -10

Layer7 wrong status, code: 503 → the backend app is unhealthy. Layer4 connection problem → network/port. Layer7 timeout → the backend is too slow.

Step 3: Inspect the failed backend container

docker ps -a --filter name=glance_api --format '{{.Names}} {{.Status}}'
docker inspect glance_api --format 'OOM={{.State.OOMKilled}} Exit={{.State.ExitCode}} Restarts={{.RestartCount}}'
docker logs glance_api 2>&1 | tail -30

Step 4: Test the backend directly, bypassing HAProxy

# From the controller, hit the service on its real bind port (not the VIP)
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:9292/    # glance-api example

A 300/200 direct but 503 via the VIP confirms the health check / HAProxy path; a failure direct confirms the service itself.

Step 5: Check the service’s own dependencies

docker logs glance_api 2>&1 | grep -iE "OperationalError|MessagingTimeout|Unauthorized|refused" | tail -5
docker exec haproxy grep -E 'timeout (server|client|connect)' /etc/haproxy/haproxy.cfg

Example Root Cause Analysis

At 14:20 an operator reports openstack image list returns 503 Service Unavailable, while openstack server list works. Only Glance is affected. Querying the HAProxy stats socket:

docker exec haproxy sh -c 'echo "show stat" | socat stdio /var/lib/kolla/haproxy/haproxy.sock' \
  | awk -F, '$1=="glance_api"{print $2,$18,$38}'

controller-01 DOWN L7STS
controller-02 DOWN L7STS
controller-03 DOWN L7STS

All three glance backends are DOWN with L7STS — the app answers but with a bad status, so this is not a network issue. Checking one container:

docker inspect glance_api --format 'OOM={{.State.OOMKilled}} Exit={{.State.ExitCode}}'
docker logs glance_api 2>&1 | tail -5

OOM=false Exit=0
ERROR glance.common.wsgi ... OperationalError: (2003, "Can't connect to MySQL server on '<vip>' (111)")

Glance is up but cannot reach its database on the VIP, so it 503s every health check. The real fault is one step down: a Galera node was drained and the MariaDB backend in HAProxy has no server available either. The narrowest correct fix is to restore the database backend (bring the Galera node back / repair the cluster), not to touch Glance or HAProxy. Once MariaDB answers on the VIP, Glance’s health checks pass and HAProxy re-marks the backends UP:

docker exec haproxy sh -c 'echo "show stat" | socat stdio /var/lib/kolla/haproxy/haproxy.sock' \
  | awk -F, '$1=="glance_api"{print $2,$18}'    # controllers flip back to UP
openstack image list    # succeeds

If a single service container were genuinely wedged (not a dependency), the narrowest action is to restart just that one backend container — for example docker restart glance_api on the affected controller — never docker restart haproxy, which would drop every service on the VIP at once.

Prevention Best Practices

• Monitor the HAProxy stats socket directly: alert on any backend DOWN and on backend ... has no server available. Distinguish L7STS (app), L4CON (network), and L7TOUT (slow) in the alert.
• Set memory limits and monitoring on API containers so an OOM (exit 137) is caught before it takes a whole backend down.
• Alert on the shared dependencies (MariaDB/Galera, RabbitMQ, Keystone) separately — a single 503 storm across many services usually points at one of them, not the APIs.
• Right-size HAProxy timeouts: raise timeout server for services with large payloads (Glance uploads) rather than letting big requests get cut off.
• Never blind-restart haproxy to “fix” a 503 — it drops every service on the VIP. Restart only the specific failed backend container after confirming it, and prefer fixing the dependency.
• Keep at least two healthy backends per service so a single controller reboot never empties a backend. See more OpenStack guides.

Quick Command Reference

# Which backends are DOWN and why (read-only)
docker exec haproxy sh -c 'echo "show stat" | socat stdio /var/lib/kolla/haproxy/haproxy.sock' \
  | awk -F, '$18=="DOWN"{print $1"/"$2" "$38}'

# HAProxy log: transitions and empty backends
docker logs haproxy 2>&1 | grep -iE "is DOWN|no server available" | tail -10

# Inspect the failing backend container
docker ps -a --filter name=<service>_api --format '{{.Names}} {{.Status}}'
docker inspect <service>_api --format 'OOM={{.State.OOMKilled}} Exit={{.State.ExitCode}}'
docker logs <service>_api 2>&1 | tail -30

# Bypass the VIP: test the backend directly on the controller
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:<port>/

# Check timeouts in haproxy.cfg
docker exec haproxy grep -E 'timeout (client|server|connect|check)' /etc/haproxy/haproxy.cfg

# Recover: restart ONLY the failed backend service (not haproxy)
docker restart <service>_api

Conclusion

An HAProxy Server ... is DOWN / backend has no server available on the control-plane VIP means the backend behind a specific OpenStack API failed its health check — HAProxy is doing its job. The diagnostic signature is a single service 503-ing on the VIP while others work. Typical root causes:

1. The backend API container is down, restarting, or OOM-killed (exit 137).
2. The container is up but returns a bad HTTP status (L7STS) on its health check.
3. A dependency the backend needs — MariaDB/Galera, RabbitMQ, Keystone — is unreachable.
4. HAProxy timeout server/timeout client is too tight for a slow request.
5. A firewall/routing change blocks HAProxy from the backend port (L4CON).

Read the stats socket and log first — the check status (L7STS vs. L4CON vs. L7TOUT) tells you whether you’re chasing the app, the network, or a timeout. Then restart the narrowest thing: the failed backend service, never HAProxy itself.