🚨 OpenStack Error Guide: 'Could not connect to instance. Retrying' — Fix Octavia Amphora Unreachable

Overview

Could not connect to instance. Retrying. is the error the Octavia controller (worker/health-manager) logs when it cannot reach a freshly booted amphora over the load-balancer management network (lb-mgmt-net) to configure it. Because every load balancer is realized by one or more amphora VMs that the control plane must talk to on port 9443, a management-network or amphora-boot problem leaves load balancers stuck in PENDING_CREATE and eventually ERROR.

The literal errors you will see:

WARNING octavia.amphorae.drivers.haproxy.rest_api_driver Could not connect to instance. Retrying.

ERROR octavia.controller.worker.v2.controller_worker ... TimeOutException: contacting the amphora timed out

It occurs during load-balancer create, listener/pool changes, or a failover — any operation where the controller must push config to the amphora agent. The tell is that the amphora VM often boots fine in Nova, but the control plane can’t reach its agent on the management network.

Symptoms

• Load balancers stay in PENDING_CREATE then flip to ERROR.
• octavia-worker/health-manager logs Could not connect to instance. Retrying.
• The amphora Nova instance is ACTIVE but the LB never provisions.

openstack loadbalancer show <lb-id> -c provisioning_status -c operating_status

| provisioning_status | ERROR   |
| operating_status    | OFFLINE |

docker logs octavia_worker 2>&1 | grep -iE "Could not connect|timed out|amphora" | tail -5

WARNING octavia.amphorae.drivers.haproxy.rest_api_driver Could not connect to instance. Retrying.

Common Root Causes

1. lb-mgmt-net not reachable from the controllers

The controllers reach amphorae over lb-mgmt-net; a routing/bridge gap makes 9443 unreachable.

openstack loadbalancer amphora list --loadbalancer <lb-id> -c id -c lb_network_ip -c status
nc -vz <amphora-lb-network-ip> 9443

Connection to 172.16.0.14 9443 port [tcp/*] failed: Connection timed out

A timeout to the amphora’s lb_network_ip on 9443 means the management path is broken.

2. Security group blocks 9443 to the amphora

The amphora’s security group must allow the controller to reach the agent port.

openstack port list --network lb-mgmt-net -c ID -c "Fixed IP Addresses" | head
openstack security group rule list <amphora-sg> -c "IP Protocol" -c "Port Range" -c "Remote IP Prefix"

| tcp | 9443:9443 | 10.0.0.0/24 |

If no rule permits the controller subnet to 9443, connections are dropped.

3. Amphora failed to boot / cloud-init didn’t start the agent

If the amphora image is bad or metadata failed, the agent never listens on 9443.

openstack loadbalancer amphora list --loadbalancer <lb-id> -c id -c status -c compute_id
openstack server show <compute_id> -c status -c fault

| status | BOOTING |

An amphora stuck in BOOTING, or a Nova instance in ERROR, means the agent was never reachable to begin with.

4. Certificate mismatch (controller<->amphora mTLS)

The controller and amphora authenticate with certificates; a mismatch after a cert rotation causes connection failures.

docker exec octavia_worker grep -E 'server_ca|client_cert|ca_certificates' \
  /etc/octavia/octavia.conf
docker logs octavia_worker 2>&1 | grep -iE "certificate|SSL|handshake" | tail -5

ERROR ... SSLError: certificate verify failed

An SSLError/handshake failure (rather than a timeout) points at cert material, not the network.

5. Health-manager can’t receive heartbeats

Amphorae send UDP heartbeats to the health-manager on 5555; if blocked, the LB may be marked failed and repeatedly failed-over.

docker exec octavia_health_manager grep -E 'bind_port|controller_ip_port_list|heartbeat' \
  /etc/octavia/octavia.conf
ss -lunp | grep 5555

controller_ip_port_list = 10.0.0.11:5555,10.0.0.12:5555

If nothing listens on 5555 or the port is blocked, heartbeats are lost and failovers churn.

6. Amphora flavor/quota exhaustion

If the Octavia service project is out of Nova quota or the flavor can’t schedule, amphorae never come up.

openstack quota show <octavia-project-id> -c instances -c cores -c ram
docker logs octavia_worker 2>&1 | grep -iE "quota|No valid host" | tail -5

ERROR ... Quota exceeded for instances

Diagnostic Workflow

Step 1: Is the amphora up in Nova and Octavia?

openstack loadbalancer amphora list --loadbalancer <lb-id> -c id -c status -c lb_network_ip -c compute_id
openstack server show <compute_id> -c status -c fault

An amphora ACTIVE in Nova but unreachable is a network/cert issue; one stuck BOOTING/ERROR is a boot/quota issue.

Step 2: Test the management path to 9443

nc -vz <amphora-lb-network-ip> 9443

timed out → routing/security group; refused → agent not listening (boot problem).

Step 3: Distinguish network from certificate failures in the log

docker logs octavia_worker 2>&1 | grep -iE "Could not connect|SSL|certificate|timed out" | tail -10

SSLError/handshake → certs; plain timed out → network.

Step 4: Check security groups and lb-mgmt-net

openstack security group rule list <amphora-sg> -c "IP Protocol" -c "Port Range" -c "Remote IP Prefix"
openstack network show lb-mgmt-net -c status -c admin_state_up

Step 5: Verify health-manager heartbeat path

ss -lunp | grep 5555
docker exec octavia_health_manager grep -E 'controller_ip_port_list' /etc/octavia/octavia.conf

Example Root Cause Analysis

A new load balancer sits in PENDING_CREATE for minutes, then ERROR. The worker log repeats:

WARNING octavia.amphorae.drivers.haproxy.rest_api_driver Could not connect to instance. Retrying.

The amphora is ACTIVE in Nova, so it booted. The operator tests the management path:

openstack loadbalancer amphora list --loadbalancer <lb-id> -c lb_network_ip -c status
nc -vz 172.16.0.14 9443

| 172.16.0.14 | BOOTING |
Connection to 172.16.0.14 9443 port [tcp/*] failed: Connection timed out

A timeout (not refused), and no SSLError in the log, points at the network path. Checking the amphora security group:

openstack security group rule list lb-mgmt-sg -c "IP Protocol" -c "Port Range" -c "Remote IP Prefix"

| tcp | 22:22 | 0.0.0.0/0 |

There is no rule allowing the controller subnet to reach 9443 — a recent security-group cleanup removed it. Re-adding the rule restores connectivity:

openstack security group rule create --protocol tcp --dst-port 9443 \
  --remote-ip <controller-subnet>/24 lb-mgmt-sg
# Octavia retries and the amphora becomes reachable
openstack loadbalancer show <lb-id> -c provisioning_status   # ACTIVE

For the already-ERROR load balancer, trigger a failover so it rebuilds cleanly. Longer term, protect the lb-mgmt security group with config management so a cleanup can’t silently remove the 9443 rule.

Prevention Best Practices

• Treat the amphora security-group rules (9443 from controllers, 5555 heartbeats) as critical config under version control — a removed rule silently breaks all LB provisioning.
• Monitor lb-mgmt-net reachability from the controllers to a canary amphora IP on 9443, and alert on failure.
• Watch load balancers stuck in PENDING_CREATE and amphorae stuck in BOOTING as early signals of a management-plane break.
• Keep Octavia’s controller/amphora certificates rotated in lockstep; a one-sided rotation causes SSLError, distinct from a network timeout.
• Ensure the health-manager’s 5555 UDP path is open both ways, or heartbeat loss triggers needless failover storms.
• Reserve Nova quota (instances, cores, RAM) and management-network IPs for the Octavia project including failover/spare headroom.
• Feed the worker log into the free incident assistant to separate a cert failure from a network timeout, and see more OpenStack guides.

Quick Command Reference

# Amphora state in Octavia and Nova
openstack loadbalancer amphora list --loadbalancer <lb-id> -c id -c status -c lb_network_ip -c compute_id
openstack server show <compute_id> -c status -c fault

# Test the management path
nc -vz <amphora-lb-network-ip> 9443

# Network vs certificate in the log
docker logs octavia_worker 2>&1 | grep -iE "Could not connect|SSL|certificate|timed out" | tail -10

# Security groups & lb-mgmt-net
openstack security group rule list <amphora-sg> -c "IP Protocol" -c "Port Range" -c "Remote IP Prefix"
openstack network show lb-mgmt-net -c status -c admin_state_up

# Heartbeat path
ss -lunp | grep 5555
docker exec octavia_health_manager grep -E 'controller_ip_port_list' /etc/octavia/octavia.conf

# Rebuild a failed LB
openstack loadbalancer failover <lb-id>

Conclusion

Could not connect to instance. Retrying. means Octavia’s control plane can’t reach the amphora agent on the management network, so load balancers never finish provisioning. Typical root causes:

1. lb-mgmt-net routing/bridge gap making 9443 unreachable.
2. A security group missing the controller-to-9443 rule.
3. The amphora failed to boot or its agent never started.
4. A controller/amphora certificate mismatch after a rotation.
5. Blocked 5555 heartbeats causing failover churn.
6. Nova quota or scheduling exhaustion preventing amphora boot.

Check whether the amphora is ACTIVE in Nova and test 9443 first: timed out vs refused, and the presence or absence of SSLError, tells you whether you’re chasing the network, the certificates, or the amphora boot.