🚨 OpenStack Error Guide: '169.254.169.254 unreachable' — Metadata Service Unavailable

Overview

Every OpenStack instance fetches its configuration — SSH keys, hostname, user-data — from the metadata service at the link-local address 169.254.169.254. The request path is: the instance hits 169.254.169.254, a metadata proxy running inside the Neutron router (qrouter) or DHCP (qdhcp) namespace catches it, forwards it to neutron-metadata-agent, which adds the instance/tenant headers and proxies to nova-api-metadata. If any hop in that chain is broken, the instance can’t reach metadata and cloud-init fails — no SSH key injected, no hostname set.

The literal symptom, from inside the instance (or its console log):

$ curl http://169.254.169.254/latest/meta-data/
curl: (7) Failed to connect to 169.254.169.254 port 80: Connection timed out

cloud-init[812]: url_helper.py ... Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id'
  failed [50/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded]

The tell is that the instance boots and networks fine (it may even have a DHCP lease) but cloud-init logs metadata timeouts. The failure is in the metadata proxy/agent chain or the L3/DHCP routing to 169.254.169.254, not in the instance itself.

Symptoms

• cloud-init in the instance console log shows repeated timeouts to 169.254.169.254.
• SSH key not injected, hostname not set, user-data scripts didn’t run.
• curl http://169.254.169.254/latest/meta-data/ from the instance times out or refuses.
• The neutron_metadata_agent container or a DHCP/L3 agent is down or erroring.

docker ps --filter name=neutron_metadata_agent --format '{{.Names}} {{.Status}}'
openstack network agent list -c "Agent Type" -c Host -c Alive --agent-type metadata

+----------------+------------+-------+
| Agent Type     | Host       | Alive |
+----------------+------------+-------+
| Metadata agent | network-01 | :-)   |
+----------------+------------+-------+

Common Root Causes

1. neutron-metadata-agent is down or erroring

The agent is the core of the chain. If its container is down, or it can’t reach nova-api-metadata, every metadata request stalls.

docker ps --filter name=neutron_metadata_agent --format '{{.Names}} {{.Status}}'
docker logs neutron_metadata_agent 2>&1 | grep -iE "error|refused|unauthorized|timeout" | tail -5

ERROR neutron.agent.metadata.agent ... Connection refused to nova metadata at <vip>:8775

2. No metadata proxy in the qrouter/qdhcp namespace

The proxy (haproxy or neutron-ns-metadata-proxy) must run inside the namespace that serves the instance’s subnet. If it isn’t there, nothing listens for 169.254.169.254.

ip netns list | grep -E 'qrouter|qdhcp'
docker exec neutron_l3_agent ip netns exec qrouter-<router-id> \
  ip route get 169.254.169.254
docker exec neutron_l3_agent ip netns exec qrouter-<router-id> \
  ss -ltnp | grep -E ':80|9697'

# empty output = no metadata proxy listening in the namespace

3. Isolated subnet with no router (DHCP namespace must serve metadata)

If the subnet has no router (isolated/provider network), metadata must be served from the qdhcp namespace, which requires enable_isolated_metadata = True in the DHCP agent config.

docker exec neutron_dhcp_agent grep -E 'enable_isolated_metadata|force_metadata' \
  /etc/neutron/dhcp_agent.ini

enable_isolated_metadata = false

false on an isolated network means the instance has no metadata route at all.

4. nova-api-metadata unreachable

The metadata agent forwards to nova-api-metadata (port 8775). If that service is down or the shared metadata secret mismatches, the agent gets refused or 500s.

docker ps --filter name=nova_api_metadata --format '{{.Names}} {{.Status}}'
docker logs nova_api_metadata 2>&1 | grep -iE "error|401|403|500|secret" | tail -5

ERROR nova.api.metadata ... Expected X-Instance-ID-Signature ... does not match

A signature mismatch means metadata_proxy_shared_secret differs between Neutron and Nova.

5. Security group or route blocks 169.254.169.254

An overly strict security group or a missing host route can stop the request from ever leaving the instance to the namespace.

openstack security group rule list <sg-id> -c "IP Protocol" -c "Port Range" -c Direction
# From the instance:
ip route get 169.254.169.254

# The 169.254.169.254 route should point at the DHCP/router gateway on the subnet

Diagnostic Workflow

Step 1: Reproduce from inside the instance (read-only)

# From the instance console/SSH:
curl -s -m 5 http://169.254.169.254/latest/meta-data/ || echo "metadata UNREACHABLE"
ip route get 169.254.169.254

Connection timed out → nothing is answering (proxy/route). Connection refused → something’s there but rejecting. No route → routing/DHCP option problem.

Step 2: Confirm the metadata agent is up and healthy

docker ps --filter name=neutron_metadata_agent --format '{{.Names}} {{.Status}}'
openstack network agent list --agent-type metadata -c Host -c Alive
docker logs neutron_metadata_agent 2>&1 | grep -iE "error|refused|timeout|unauthorized" | tail -10

Step 3: Find and inspect the serving namespace

# Identify the router/network for the instance's subnet, then:
ip netns list | grep -E 'qrouter|qdhcp'
docker exec neutron_l3_agent ip netns exec qrouter-<router-id> ss -ltnp | grep -E ':80|9697'
docker exec neutron_dhcp_agent ip netns exec qdhcp-<network-id> ss -ltnp | grep -E ':80|9697'

A metadata proxy should be listening in whichever namespace serves the subnet. If neither has it, metadata has nowhere to land.

Step 4: Test the path from the namespace to nova-api-metadata

docker exec neutron_metadata_agent grep -E 'nova_metadata_host|nova_metadata_port|metadata_proxy_shared_secret' \
  /etc/neutron/metadata_agent.ini
docker ps --filter name=nova_api_metadata --format '{{.Names}} {{.Status}}'
docker logs nova_api_metadata 2>&1 | grep -iE "error|signature|401|500" | tail -5

Step 5: Check isolated-metadata config, security groups, and routes

docker exec neutron_dhcp_agent grep -E 'enable_isolated_metadata|force_metadata' /etc/neutron/dhcp_agent.ini
openstack security group rule list <sg-id> -c Direction -c "IP Protocol" -c "Port Range"

Example Root Cause Analysis

A tenant reports new instances on an isolated provider network boot but have no SSH key and cloud-init shows metadata timeouts. From the instance:

curl -s -m 5 http://169.254.169.254/latest/meta-data/ || echo "metadata UNREACHABLE"
ip route get 169.254.169.254

metadata UNREACHABLE
# no specific route to 169.254.169.254 (only the default via the subnet gateway)

The instance has no route to the metadata address. Because this subnet has no Neutron router, metadata must come from the qdhcp namespace. Checking the DHCP namespace and its config:

docker exec neutron_dhcp_agent ip netns exec qdhcp-<network-id> ss -ltnp | grep -E ':80|9697'
docker exec neutron_dhcp_agent grep -E 'enable_isolated_metadata' /etc/neutron/dhcp_agent.ini

# (no listener in the namespace)
enable_isolated_metadata = false

The metadata agent and nova-api-metadata are both healthy — the real cause is that isolated metadata is disabled, so the DHCP agent never pushes the 169.254.169.254 host route via DHCP option 121 nor runs a proxy in qdhcp. The narrowest fix is to enable isolated metadata on the DHCP agent and restart only that agent (not the whole network stack):

# Set enable_isolated_metadata = True in dhcp_agent.ini (via Kolla config), then:
docker restart neutron_dhcp_agent

After the DHCP agent restarts and the instance renews its lease (or is rebooted), it receives the host route to 169.254.169.254, a proxy appears in the qdhcp namespace, and cloud-init succeeds:

docker exec neutron_dhcp_agent ip netns exec qdhcp-<network-id> ss -ltnp | grep -E ':80|9697'
# From the instance after DHCP renew:
curl -s http://169.254.169.254/latest/meta-data/instance-id

Restarting only neutron_dhcp_agent keeps the L3 agent and running routers untouched, so existing traffic is unaffected.

Prevention Best Practices

• Monitor the metadata chain end to end: neutron_metadata_agent liveness, nova_api_metadata up, and a synthetic curl to 169.254.169.254 from a canary instance.
• Set enable_isolated_metadata = True (or force_metadata) on the DHCP agent for any deployment that uses router-less provider/isolated networks — the top cause of “no route to metadata.”
• Keep metadata_proxy_shared_secret identical between Neutron’s metadata_agent.ini and Nova — manage it with config management so a partial redeploy can’t split it.
• Ensure security groups and images don’t block or reroute the link-local 169.254.169.254; verify with ip route get from the instance.
• Restart the narrowest agent: fix the metadata or DHCP agent specifically, never blind-restart every Neutron agent (which would drop DHCP/L3 for all tenants). See more OpenStack guides.
• Watch cloud-init in console logs after image or network changes so a broken metadata path is caught before users file tickets.

Quick Command Reference

# Reproduce from the instance (read-only)
curl -s -m 5 http://169.254.169.254/latest/meta-data/ || echo "UNREACHABLE"
ip route get 169.254.169.254

# Metadata agent health
docker ps --filter name=neutron_metadata_agent --format '{{.Names}} {{.Status}}'
openstack network agent list --agent-type metadata -c Host -c Alive
docker logs neutron_metadata_agent 2>&1 | grep -iE "error|refused|timeout" | tail -10

# Is a proxy listening in the serving namespace?
ip netns list | grep -E 'qrouter|qdhcp'
docker exec neutron_l3_agent ip netns exec qrouter-<router-id> ss -ltnp | grep -E ':80|9697'
docker exec neutron_dhcp_agent ip netns exec qdhcp-<network-id> ss -ltnp | grep -E ':80|9697'

# Path to nova-api-metadata + shared secret
docker exec neutron_metadata_agent grep -E 'nova_metadata_host|nova_metadata_port|metadata_proxy_shared_secret' \
  /etc/neutron/metadata_agent.ini
docker ps --filter name=nova_api_metadata --format '{{.Names}} {{.Status}}'
docker logs nova_api_metadata 2>&1 | grep -iE "error|signature|401|500" | tail -5

# Isolated networks need this ON
docker exec neutron_dhcp_agent grep -E 'enable_isolated_metadata|force_metadata' /etc/neutron/dhcp_agent.ini

# Recover: restart ONLY the affected agent
docker restart neutron_dhcp_agent      # or neutron_metadata_agent

Conclusion

Instances that can’t reach 169.254.169.254 and cloud-init timeouts mean a break somewhere in the metadata chain: instance → namespace proxy → neutron-metadata-agent → nova-api-metadata. The instance itself is usually fine — it booted and got an IP. Typical root causes:

1. neutron-metadata-agent is down or can’t reach nova-api-metadata.
2. No metadata proxy is running in the serving qrouter/qdhcp namespace.
3. An isolated/router-less network with enable_isolated_metadata = false.
4. nova-api-metadata is down or the metadata_proxy_shared_secret mismatches (signature errors).
5. A security group or missing host route blocks 169.254.169.254 at the instance.

Reproduce with curl from the instance, then walk the chain — namespace proxy, metadata agent, nova-api-metadata — and restart only the narrowest agent (usually the DHCP or metadata agent), never the whole Neutron stack.