☁️ OpenStack Error: 'provisioning_status ERROR' Octavia Load Balancer Amphora Failure

Exact Error Message

$ openstack loadbalancer show lb-prod-web
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| provisioning_status | ERROR                                |
| operating_status    | OFFLINE                              |
| vip_address         | 10.0.10.42                           |
+---------------------+--------------------------------------+

The Octavia worker log tells the real story:

ERROR octavia.controller.worker.v2.controller_worker [-] Task
'octavia-create-amp-for-lb-subflow-octavia-compute-connectivity-wait'
failed: octavia.amphorae.driver_exceptions.exceptions.TimeOutException:
contacting the amphora timed out
WARNING octavia.controller.worker.v2.tasks.compute_tasks [-]
Amphora compute instance failed to become active, deleting.
Nova instance status: ERROR

What It Means

Octavia realizes load balancers by booting one or more amphora Nova instances (small VMs running HAProxy) and configuring them over a dedicated management network. When any step in that create flow fails — the amphora VM never boots, never gets an IP on the management network, or never answers on its REST API port — the worker rolls the flow back and marks the load balancer provisioning_status ERROR.

ERROR is a terminal provisioning state. The load balancer will not self-heal, and most further API calls against it are rejected until you either recover it or delete it. The root cause is almost always downstream: Nova can’t schedule the amphora, the amphora image/flavor is wrong, or the lb-mgmt-net path between the controller and the amphora is broken.

Common Causes

• The amphora image is missing, untagged, or the configured amp_image_tag no longer matches any image in Glance.
• Nova cannot schedule the amphora flavor (no valid host, quota exhausted, or the amp_flavor_id was deleted).
• The lb-mgmt-net (management network) is unreachable from the controller, so the health/REST connection to the amphora times out.
• Security groups or the health-manager firewall block UDP 5555 (health) or TCP 9443 (amphora API).
• Certificate expiry: the client/server certs Octavia uses to talk to the amphora API have expired.
• The octavia-health-manager or octavia-worker service is down on a controller.

Diagnostic Commands

Confirm the load balancer state and pull the amphora record:

openstack loadbalancer show lb-prod-web
openstack loadbalancer amphora list --loadbalancer lb-prod-web

Check the amphora Nova instance directly (Octavia often deletes failed ones, so look fast or check logs):

openstack server list --all-projects --name amphora
openstack server show <amphora-instance-id> -f value -c fault

Verify the amphora image and flavor Octavia is configured to use:

grep -E 'amp_image_tag|amp_flavor_id' /etc/octavia/octavia.conf
openstack image list --tag amphora
openstack flavor show <amp_flavor_id>

Tail the worker and health-manager logs on the controller:

journalctl -u octavia-worker -u octavia-health-manager --since "20 min ago"

Step-by-Step Resolution

1. Read the worker log to classify the failure. Nova instance status: ERROR means Nova scheduling failed; contacting the amphora timed out means the VM booted but the management path is broken.

2. If scheduling failed, fix the image/flavor. Re-tag or re-upload the amphora image so it matches amp_image_tag, and confirm the flavor still exists:

openstack image set --tag amphora <amphora-image-id>
openstack flavor show $(grep amp_flavor_id /etc/octavia/octavia.conf | awk '{print $3}')

3. If the amphora booted but timed out, test the management path. From a controller, confirm you can reach the amphora API port and that health-manager is listening:

openstack loadbalancer amphora show <amphora-id> -f value -c lb_network_ip
nc -vz <lb_network_ip> 9443
ss -lunp | grep 5555

4. Check certificate validity — expired certs are a silent, common cause after a year of uptime:

openssl x509 -enddate -noout -in /etc/octavia/certs/client.pem

5. Once the underlying issue is fixed, recover the load balancer instead of leaving it in ERROR. failover rebuilds the amphora:

openstack loadbalancer failover lb-prod-web
openstack loadbalancer show lb-prod-web -f value -c provisioning_status

6. If recovery is impossible (image permanently gone, tenant cleanup), delete and recreate:

openstack loadbalancer delete lb-prod-web --cascade

Prevention

• Pin and protect the amphora image: keep it tagged, and monitor that openstack image list --tag amphora always returns a result.
• Alert on octavia-worker and octavia-health-manager service health so a dead controller service surfaces before it strands a build.
• Monitor certificate expiry for /etc/octavia/certs/ and rotate well ahead of the not-after date.
• Reserve headroom in the Octavia project quota and on hosts in the amphora availability zone so scheduling never starves.
• Keep the lb-mgmt-net isolated but reachable, and verify UDP 5555 / TCP 9443 are open end-to-end after any firewall change.

Related Errors

• Nova instance status: ERROR on the amphora — a pure Nova scheduling/boot failure surfaced through Octavia.
• operating_status: OFFLINE with provisioning_status: ACTIVE — the LB provisioned but health checks fail; a member/monitor problem, not a build failure.
• certificate verify failed in the worker log — expired or mismatched amphora certs.
• octavia.amphorae.driver_exceptions.exceptions.TimeOutException — the management-network reachability variant of this same failure.

Frequently Asked Questions

Why is my load balancer stuck in ERROR and not recovering on its own? ERROR is a terminal provisioning state; Octavia does not auto-heal it. You must run openstack loadbalancer failover after fixing the root cause, or delete and recreate the LB.

How do I rebuild a failed amphora without deleting the load balancer? Use openstack loadbalancer failover <lb>, which reprovisions the amphora VM and re-applies the listener/pool configuration while keeping the VIP and object IDs.

What most often causes the amphora timeout? A broken lb-mgmt-net path or expired certificates. The controller boots the VM successfully but cannot reach its REST API on port 9443, so the create flow times out and rolls back.

Can I see why the amphora VM itself failed? Yes — grab the amphora instance ID from openstack loadbalancer amphora list and run openstack server show <id> -c fault, but do it quickly because Octavia deletes failed amphorae during rollback.

Where do I find related fixes? Browse the full OpenStack guides for more Octavia, Nova, and Neutron troubleshooting, and grab a ready-made investigation prompt from the prompt library.