🚨 NGINX Error Guide: '(99: Cannot assign requested address)' — Fix Port Exhaustion

Overview

NGINX logs this when it tries to open a new connection to an upstream but the kernel has no local ephemeral port available for the source (client) side of the socket:

2026/07/09 16:03:22 [crit] 1442#1442: *91237 connect() to 10.0.3.20:8080 failed (99: Cannot assign requested address) while connecting to upstream, client: 198.51.100.4, server: api.example.com, request: "POST /v1/events HTTP/1.1", upstream: "http://10.0.3.20:8080/v1/events", host: "api.example.com"

Clients see intermittent 502 Bad Gateway under load. The error is EADDRNOTAVAIL from the kernel: every outbound connection to the same upstream_ip:port needs a unique local source port, and NGINX has run out because thousands of short-lived connections are stacked up in TIME_WAIT. It is almost always a symptom of NGINX opening a brand-new TCP connection per proxied request instead of reusing a keepalive pool.

Symptoms

• Intermittent 502s that correlate with traffic peaks, not with backend health.
• The error log shows connect() to <upstream> failed (99: Cannot assign requested address).
• ss -tan state time-wait | wc -l on the NGINX host is very high (tens of thousands).
• The exhaustion is per destination ip:port — one busy upstream fails while others are fine.
• Adding backend capacity doesn’t help; the bottleneck is local port allocation on the proxy.

Common Root Causes

• No upstream keepalive — NGINX opens and closes a fresh TCP connection for every request, each consuming an ephemeral port that lingers in TIME_WAIT for ~60s.
• A narrow ephemeral port range — net.ipv4.ip_local_port_range left at a small window, so few ports are available per destination.
• Very high request rate to a single upstream — even with a decent range, per-request connections exhaust ports faster than they recycle.
• HTTP/1.0 to the upstream — keepalive requires HTTP/1.1 and clearing the Connection header; missing either forces new connections.
• Proxying everything to one ip:port — the ~28k default ephemeral ports all target the same tuple, which is the exhaustible resource.

Diagnostic Workflow

Confirm it’s port exhaustion by counting TIME_WAIT sockets and checking the range:

ss -tan | awk '{print $1}' | sort | uniq -c        # connection states
ss -tan state time-wait | wc -l                     # how many TIME_WAIT
cat /proc/sys/net/ipv4/ip_local_port_range          # available ephemeral ports

See how many connections target the busy upstream specifically:

ss -tan | grep ':8080' | awk '{print $NF}' | sort | uniq -c | sort -rn | head

The primary fix is an upstream keepalive pool so NGINX reuses connections instead of minting one per request:

upstream backend {
    server 10.0.3.20:8080;
    keepalive 64;                 # pooled idle connections per worker
    keepalive_requests 1000;      # requests per pooled connection before recycle
    keepalive_timeout 60s;
}

server {
    location / {
        proxy_pass http://backend;
        proxy_http_version 1.1;           # required for keepalive
        proxy_set_header Connection "";   # clear "close" so the connection is reused
    }
}

If the request rate genuinely needs more ports, widen the ephemeral range (secondary measure):

sysctl -w net.ipv4.ip_local_port_range="1024 65535"
# persist in /etc/sysctl.d/ and apply with: sysctl --system

Validate NGINX and reload:

nginx -t && nginx -s reload

Example Root Cause Analysis

An event-ingest API behind NGINX started throwing 502s only during traffic spikes. The error log showed connect() to 10.0.3.20:8080 failed (99: Cannot assign requested address), and ss -tan state time-wait | wc -l on the proxy read over 40,000 — nearly all targeting the single ingest upstream on :8080.

The root cause was that the upstream block had no keepalive and the location used the defaults, so NGINX opened a new TCP connection for every one of the thousands of POSTs per second. Each closed connection sat in TIME_WAIT for 60 seconds, and with the default ephemeral range the proxy simply ran out of source ports to that one destination. Adding keepalive 64; to the upstream plus proxy_http_version 1.1; and proxy_set_header Connection ""; in the location collapsed the connection churn: reused pooled connections meant far fewer ephemeral ports in flight. TIME_WAIT dropped by an order of magnitude and the spike-time 502s vanished. Widening ip_local_port_range was kept as a secondary safety margin, not the fix.

Prevention Best Practices

• Always configure an upstream keepalive pool for high-throughput proxying, with proxy_http_version 1.1 and proxy_set_header Connection "" — this is the real fix.
• Size keepalive per worker for your concurrency, and set keepalive_requests/keepalive_timeout so connections recycle without churning.
• Widen net.ipv4.ip_local_port_range as a secondary measure; don’t rely on it instead of keepalive.
• Prefer tuning connection reuse over enabling net.ipv4.tcp_tw_reuse blindly — reuse the connections and you avoid the TIME_WAIT pile-up entirely.
• Spread load across multiple upstream ip:port targets where possible; exhaustion is per destination tuple.
• Alert on TIME_WAIT socket counts and on Cannot assign requested address in the error log so you catch it before it causes 502s.

Quick Command Reference

# Confirm TIME_WAIT pile-up and ephemeral range
ss -tan state time-wait | wc -l
cat /proc/sys/net/ipv4/ip_local_port_range

# Which upstream tuple dominates connections
ss -tan | grep ':UPSTREAM_PORT' | awk '{print $NF}' | sort | uniq -c | sort -rn | head

# Widen the range (secondary measure)
sysctl -w net.ipv4.ip_local_port_range="1024 65535"

# Validate and reload after adding keepalive
nginx -t && nginx -s reload

Conclusion

connect() failed (99: Cannot assign requested address) means NGINX ran out of local ephemeral ports to an upstream because it was opening a fresh TCP connection per request and piling them into TIME_WAIT. The durable fix is an upstream keepalive pool with proxy_http_version 1.1 and a cleared Connection header so connections are reused. Widening ip_local_port_range helps at the margin, but connection reuse is what makes the 502s disappear under load.