🚨 NGINX Error Guide: 'upstream timed out (110: Connection timed out)' — Fix Slow Backends

Overview

NGINX logs this in the error log the moment a proxied backend takes longer to respond than a proxy_*_timeout allows. The client usually sees a 504 Gateway Time-out, and the error log records exactly which phase timed out:

2026/07/09 14:22:07 [error] 1180#1180: *84213 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 203.0.113.9, server: api.example.com, request: "GET /reports/export HTTP/1.1", upstream: "http://10.0.2.15:8080/reports/export", host: "api.example.com"

The phrase after “while” is the important part. while reading response header from upstream means the backend accepted the connection but was too slow to send the first byte of the response (governed by proxy_read_timeout). while connecting to upstream points at proxy_connect_timeout, and while sending request to upstream points at proxy_send_timeout.

Symptoms

• Clients receive 504 Gateway Time-out after a consistent delay (often exactly 60 seconds — the default proxy_read_timeout).
• The error log fills with upstream timed out (110: Connection timed out) lines, each naming a specific upstream: address.
• Only slow endpoints (reports, exports, search, third-party calls) fail; fast endpoints on the same upstream are fine.
• $upstream_response_time in the access log sits at or just above your timeout value.
• The backend’s own logs show the request completing successfully a few seconds after NGINX gave up.

Common Root Causes

• A genuinely slow endpoint — a report, export, or aggregation query that legitimately takes longer than proxy_read_timeout.
• An overloaded backend — CPU saturation, a full thread/worker pool, or GC pauses making every response slow.
• A slow downstream dependency — the backend is itself waiting on a database, cache, or third-party API.
• Timeouts left at defaults — the 60s defaults are too short for long-running work, or too long to fail fast for a hung backend.
• Network path problems — packet loss or a firewall silently dropping the connection between NGINX and the upstream.
• Connection pool exhaustion — keepalive to the upstream misconfigured, so new connections queue behind saturated ones.

Diagnostic Workflow

First confirm which phase is timing out — read the word after “while” in the error log:

grep 'upstream timed out' /var/log/nginx/error.log | tail -20
grep 'upstream timed out' /var/log/nginx/error.log | grep -oE 'while [a-z ]+from upstream|while connecting|while sending' | sort | uniq -c

Measure how long upstream responses actually take by adding timing to your log format, then reload:

http {
    log_format upstream_timing '$remote_addr "$request" '
                               'status=$status '
                               'request_time=$request_time '
                               'upstream_connect=$upstream_connect_time '
                               'upstream_header=$upstream_header_time '
                               'upstream_response=$upstream_response_time '
                               'upstream=$upstream_addr';

    access_log /var/log/nginx/timing.log upstream_timing;
}

Then watch which endpoints run long:

awk '{print $NF, $0}' /var/log/nginx/timing.log | sort -rn | head

Test the backend directly, bypassing NGINX, to prove where the latency lives:

time curl -s -o /dev/null -w '%{time_total}\n' http://10.0.2.15:8080/reports/export

If the direct call is also slow, the problem is the backend, not NGINX. If it is fast, look at the network path or connection pooling. Set timeouts deliberately in the proxy location:

location /reports/ {
    proxy_pass http://backend;
    proxy_connect_timeout 5s;    # fail fast if the backend can't be reached
    proxy_send_timeout   30s;    # time to send the request body
    proxy_read_timeout  120s;    # allow long-running reports to finish
    proxy_next_upstream error timeout http_502 http_504;
}

Validate and reload after any change:

nginx -t && nginx -s reload

Example Root Cause Analysis

An analytics API returned 504 on /reports/export at exactly 60 seconds, every time. The error log showed while reading response header from upstream, pointing at proxy_read_timeout. The team’s first instinct was to raise the timeout — but a direct curl to the backend showed the export finishing in about 75 seconds, and the backend logs confirmed a successful 200 at that mark.

The real issue was a synchronous CSV export that scanned an unindexed table. Rather than paper over it by pushing proxy_read_timeout ever higher, they added the missing database index (dropping the export to 4 seconds) and set proxy_read_timeout 120s on that one location as a safety margin for large accounts. The 504s stopped, and — because they scoped the longer timeout to /reports/ only — a genuinely hung backend elsewhere still failed fast at the default instead of pinning a worker for two minutes.

Prevention Best Practices

• Set proxy_connect_timeout low (2-5s) so an unreachable backend fails fast, and tune proxy_read_timeout per route to match each endpoint’s real work, rather than one global value.
• Log $upstream_connect_time, $upstream_header_time, and $upstream_response_time so you can see latency creeping up before it becomes a 504.
• Fix the backend, not just the timeout — a rising proxy_read_timeout is a smell that a slow query or dependency needs attention.
• Use proxy_next_upstream error timeout with multiple upstream servers so a single slow node doesn’t take the request down.
• Alert on $upstream_response_time percentiles and on the count of upstream timed out log lines.
• Enable upstream keepalive (keepalive in the upstream block plus proxy_http_version 1.1 and proxy_set_header Connection "") to avoid per-request connection overhead.

Quick Command Reference

# See which phase is timing out
grep 'upstream timed out' /var/log/nginx/error.log | tail -20

# Rank slowest upstream responses (with timing log_format enabled)
sort -t= -k6 -rn /var/log/nginx/timing.log | head

# Time the backend directly, bypassing NGINX
time curl -s -o /dev/null -w '%{time_total}\n' http://BACKEND_IP:PORT/PATH

# Validate config and reload after changing timeouts
nginx -t && nginx -s reload

Conclusion

upstream timed out (110: Connection timed out) is NGINX telling you the backend was slower than a proxy_*_timeout you set (or left at its default). The word after “while” pinpoints the phase — connect, send, or read — and $upstream_response_time tells you whether the timeout is too aggressive or the backend is genuinely slow. Raise the timeout only where long work is legitimate, scope it per location, and treat a steadily climbing proxy_read_timeout as a sign to fix the backend rather than hide the latency.