🚨 OpenStack Error Guide: '1040 Too many connections' — Fix MariaDB/Galera Connection Exhaustion

Overview

(1040, 'Too many connections') is the MariaDB/Galera error every OpenStack service hits when the database has reached max_connections and refuses new sessions. Because Nova, Neutron, Keystone, Cinder, Glance and Placement all pool connections to the same Galera cluster (usually through an HAProxy VIP), a connection ceiling surfaces as DBConnectionError and 500s across the whole control plane at once.

The literal errors you will see:

oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (1040, 'Too many connections')

ERROR oslo_db.sqlalchemy.engines ... (1040, 'Too many connections')
(Background on this error at: https://sqlalche.me/e/...)

It occurs whenever a service tries to open a new DB session — an API request, a periodic task, or a token validation. Like a RabbitMQ outage, the symptom appears in many services simultaneously, which points at the shared database, not any one project.

Symptoms

• Multiple services log DBConnectionError / (1040, 'Too many connections') together.
• openstack commands intermittently 500; Horizon shows “An error occurred”.
• Keystone token issuance fails, cascading auth errors everywhere.

docker exec mariadb mysql -uroot -p"$DBPASS" -e "SHOW STATUS LIKE 'Threads_connected';"

+-------------------+-------+
| Variable_name     | Value |
+-------------------+-------+
| Threads_connected | 500   |
+-------------------+-------+

docker logs keystone 2>&1 | grep -iE "1040|Too many connections" | tail -3

ERROR oslo_db ... (1040, 'Too many connections')

Common Root Causes

1. max_connections set too low for the fleet

The default or a conservative value is quickly exhausted once dozens of services each open a pool.

docker exec mariadb mysql -uroot -p"$DBPASS" -e "SHOW VARIABLES LIKE 'max_connections';"

+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 512   |
+-----------------+-------+

Compare the value against the sum of every service’s pool (below); if they can collectively exceed it, exhaustion is a matter of time.

2. Oversized SQLAlchemy pools per service

Each service opens max_pool_size + max_overflow connections PER worker. Many workers x many services multiplies fast.

grep -E 'max_pool_size|max_overflow' /etc/nova/nova.conf /etc/neutron/neutron.conf \
  /etc/keystone/keystone.conf 2>/dev/null

max_pool_size = 30
max_overflow = 50

30 + 50 = 80 connections per worker; with 8 API workers that’s 640 from one service alone.

3. HAProxy fans all services onto one Galera node

The common single-writer HAProxy config sends every connection to one active Galera node, so its max_connections is the real ceiling — not the cluster’s aggregate.

docker exec haproxy grep -A6 'listen mariadb' /etc/haproxy/haproxy.cfg
echo "---"
docker exec mariadb mysql -uroot -p"$DBPASS" \
  -e "SHOW STATUS LIKE 'wsrep_cluster_size';"

    server controller-01 10.0.0.11:3306 check
    server controller-02 10.0.0.12:3306 check backup
    server controller-03 10.0.0.13:3306 check backup

Only controller-01 takes traffic; its per-node max_connections gates everything.

4. Leaked / long-lived connections

A service (or a stuck migration/report task) holds connections without releasing them, slowly consuming the pool.

docker exec mariadb mysql -uroot -p"$DBPASS" -e \
  "SELECT user, host, COUNT(*) c FROM information_schema.processlist \
   GROUP BY user, host ORDER BY c DESC LIMIT 10;"

+----------+-------------------+-----+
| user     | host              | c   |
+----------+-------------------+-----+
| nova     | controller-01:... | 210 |
| neutron  | controller-01:... | 140 |
+----------+-------------------+-----+

One user holding a disproportionate share suggests a leak or runaway worker count.

5. A boot storm or retry storm

A surge of API calls (bulk launch, orchestration) spikes concurrent connections past the ceiling; failing requests retry and make it worse.

docker exec mariadb mysql -uroot -p"$DBPASS" -e "SHOW STATUS LIKE 'Max_used_connections';"

| Max_used_connections | 512 |

Max_used_connections pinned at max_connections confirms you hit the wall.

6. open_files_limit / systemd caps below what MariaDB needs

MariaDB can silently cap max_connections if the OS file-descriptor limit is lower than required.

docker exec mariadb mysql -uroot -p"$DBPASS" -e "SHOW VARIABLES LIKE 'open_files_limit';"

| open_files_limit | 1024 |

A low open_files_limit prevents max_connections from taking effect even if you raise it.

Diagnostic Workflow

Step 1: Confirm it’s the shared DB, not one service

for s in nova_api neutron_server keystone cinder_api; do
  echo "== $s =="; docker logs $s 2>&1 | grep -c "Too many connections"
done

Errors across many services at once = database ceiling.

Step 2: Read the live connection state

docker exec mariadb mysql -uroot -p"$DBPASS" -e \
  "SHOW STATUS LIKE 'Threads_connected'; SHOW STATUS LIKE 'Max_used_connections'; \
   SHOW VARIABLES LIKE 'max_connections';"

Step 3: Find who is holding connections

docker exec mariadb mysql -uroot -p"$DBPASS" -e \
  "SELECT user, COUNT(*) c FROM information_schema.processlist GROUP BY user ORDER BY c DESC;"

Step 4: Check HAProxy single-node fan-out

docker exec haproxy grep -A6 'listen mariadb' /etc/haproxy/haproxy.cfg
docker exec mariadb mysql -uroot -p"$DBPASS" -e "SHOW STATUS LIKE 'wsrep_cluster_size';"

Step 5: Sum the configured pools against the ceiling

grep -rE 'max_pool_size|max_overflow' /etc/*/*.conf 2>/dev/null

Multiply per-worker pool by worker counts across services and compare to max_connections.

Example Root Cause Analysis

At 09:00, right after a scheduled bulk launch of 200 instances, Horizon starts 500ing and Keystone logs flood with (1040, 'Too many connections'). The DB confirms it’s maxed:

docker exec mariadb mysql -uroot -p"$DBPASS" -e \
  "SHOW STATUS LIKE 'Threads_connected'; SHOW VARIABLES LIKE 'max_connections';"

| Threads_connected | 512 |
| max_connections   | 512 |

Connections are pinned. HAProxy shows all traffic hitting one node:

    server controller-01 10.0.0.11:3306 check
    server controller-02 ... check backup

So the ceiling is one node’s 512, not the cluster’s. The process list shows Nova holding the lion’s share after the launch storm — its pools plus a high worker count. The immediate fix is to raise the per-node limit and file-descriptor cap, then reconfigure Nova’s pools sensibly:

# Immediate relief: raise the runtime ceiling
docker exec mariadb mysql -uroot -p"$DBPASS" -e "SET GLOBAL max_connections = 4096;"

Then persist a higher max_connections and open_files_limit via Kolla config override, and right-size max_pool_size/max_overflow and worker counts so the aggregate can’t exceed the ceiling. Reconfigure and verify:

kolla-ansible reconfigure -t mariadb,nova   # after editing config overrides
docker exec mariadb mysql -uroot -p"$DBPASS" -e "SHOW STATUS LIKE 'Threads_connected';"

Longer term, spread reads across Galera nodes or right-size pools so a boot storm can’t exhaust a single writer.

Prevention Best Practices

• Size max_connections (and open_files_limit) to exceed the summed worst-case of every service’s (max_pool_size + max_overflow) x workers, with headroom for boot storms.
• Right-size SQLAlchemy pools; huge max_pool_size/max_overflow values multiplied by worker counts are the usual culprit, not the DB itself.
• Remember the HAProxy single-writer pattern: the per-node limit is the real ceiling — raise it on the active node, not just conceptually across the cluster.
• Alert on Threads_connected approaching max_connections and on Max_used_connections so you see the wall coming.
• Watch information_schema.processlist for a single user hoarding connections — the signature of a leak or an over-worked service.
• Keep MariaDB’s file-descriptor limit high enough that a raised max_connections actually takes effect (verify with SHOW VARIABLES LIKE 'open_files_limit').
• Paste the simultaneous DBConnectionError traces into the free incident assistant to confirm a shared-DB ceiling, and see more OpenStack guides.

Quick Command Reference

# Is the whole control plane hitting it?
for s in nova_api neutron_server keystone cinder_api; do \
  echo "== $s =="; docker logs $s 2>&1 | grep -c "Too many connections"; done

# Live connection state
docker exec mariadb mysql -uroot -p"$DBPASS" -e \
  "SHOW STATUS LIKE 'Threads_connected'; SHOW STATUS LIKE 'Max_used_connections'; \
   SHOW VARIABLES LIKE 'max_connections'; SHOW VARIABLES LIKE 'open_files_limit';"

# Who holds the connections?
docker exec mariadb mysql -uroot -p"$DBPASS" -e \
  "SELECT user, COUNT(*) c FROM information_schema.processlist GROUP BY user ORDER BY c DESC;"

# HAProxy fan-out & cluster size
docker exec haproxy grep -A6 'listen mariadb' /etc/haproxy/haproxy.cfg
docker exec mariadb mysql -uroot -p"$DBPASS" -e "SHOW STATUS LIKE 'wsrep_cluster_size';"

# Configured pools across services
grep -rE 'max_pool_size|max_overflow' /etc/*/*.conf 2>/dev/null

# Immediate relief (persist via config override afterward)
docker exec mariadb mysql -uroot -p"$DBPASS" -e "SET GLOBAL max_connections = 4096;"

Conclusion

(1040, 'Too many connections') is a shared-database ceiling: MariaDB/Galera hit max_connections and every service that needs a new session fails at once. The simultaneous, cross-service pattern is the diagnostic signature. Typical root causes:

1. max_connections set too low for the fleet.
2. Oversized SQLAlchemy pools multiplied across workers and services.
3. HAProxy funneling all services to one Galera node, so its per-node limit is the ceiling.
4. Leaked or long-lived connections held by a runaway service.
5. A boot/retry storm spiking concurrent connections past the wall.
6. A low OS file-descriptor limit silently capping max_connections.

Check Threads_connected vs max_connections first, then the HAProxy fan-out and the per-user process list — the connection accounting tells you whether to raise the ceiling, shrink the pools, or hunt a leak.