NGINX Configuration

NGINX fails in a small number of very recognisable ways, and almost all of them come from four places: which server block matched, which location matched, what got sent upstream, and what the certificate chain actually contains. Get those four right and the 502/503/504 triage becomes mechanical. This reference is organized around them, and every section ends at the errors it produces.

The config model: contexts, testing, reloading

Directives are only legal in certain contexts, and they inherit downward — main → events / http → server → location. A directive placed one level too high or too low is the single most common config error.

# The only safe reload sequence
sudo nginx -t && sudo systemctl reload nginx

Config-shape failures: unknown directive usually means a missing module, directive not allowed here is the wrong context, and unexpected end of file, expecting brace is exactly what it says. Invalid number of arguments is usually a missing semicolon on the line above.

Server blocks: which one answered?

NGINX picks a server block by listen, then by server_name, in a defined order:

1. exact match                    server_name api.example.com;
2. longest wildcard starting *    server_name *.example.com;
3. longest wildcard ending *      server_name www.example.*;
4. first matching regex           server_name ~^api\d+\.example\.com$;
5. default_server for that listen (or the FIRST block if none is marked)

server {
    listen 443 ssl default_server;
    server_name _;
    ssl_reject_handshake on;   # refuse unknown hostnames rather than serving a random vhost
    return 444;
}

Related: conflicting server name, duplicate default server, and could not build server_names_hash — the last needs server_names_hash_bucket_size raised when you have long domain names.

Location matching: the rule people get wrong

Matching is not top-to-bottom. NGINX evaluates in this order:

1. = /exact          exact match wins immediately, stops here
2. prefix matches    the LONGEST one is remembered (not used yet)
3. ^~ /prefix        if the longest prefix used ^~, stop — skip regex entirely
4. ~ / ~* regex      checked IN FILE ORDER; the FIRST match wins
5. otherwise         fall back to the longest prefix remembered in step 2

location = /healthz        { return 200 "ok\n"; }      # 1: exact, cheapest
location ^~ /static/       { root /var/www; }          # 3: stop regex for assets
location ~* \.(jpg|png)$   { expires 30d; }            # 4: regex, file order matters
location /                 { proxy_pass http://app; }  # 5: catch-all fallback

The practical consequences: a regex block written below another regex block never runs if the one above also matches, and ^~ is how you stop an expensive regex from being evaluated for every static asset. See also duplicate location and rewrite or internal redirection cycle, which is nearly always a try_files or rewrite pointing back at itself.

Reverse proxy essentials

upstream app {
    server 10.0.1.10:8080 max_fails=3 fail_timeout=10s;
    server 10.0.1.11:8080 max_fails=3 fail_timeout=10s;
    keepalive 32;                       # reuse upstream connections
}

server {
    location / {
        proxy_pass http://app;

        # Keepalive to the upstream requires BOTH of these
        proxy_http_version 1.1;
        proxy_set_header Connection "";

        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Two details worth internalising. Upstream keepalive silently does nothing without proxy_http_version 1.1 and clearing the Connection header — and without it, a busy proxy opens a fresh TCP connection per request and eventually exhausts ephemeral ports, surfacing as cannot assign requested address. And proxy_set_header does not merge across levels: defining any proxy_set_header inside a location discards every one inherited from the server block.

Timeouts, buffers, and the errors they cause

Most 5xx responses from NGINX are one of these limits, not a bug.

Which status you get is diagnostic:

| Status | What NGINX is telling you | | --- | --- | | 502 | It reached the upstream and got something unusable — connection refused/reset, or an invalid HTTP response. | | 503 | It had no upstream to try: every server is marked down by max_fails, or a limit_req/limit_conn rejected the request. | | 504 | The upstream accepted the connection and then did not answer within proxy_read_timeout. | | 499 | The client gave up and disconnected first. Not an NGINX fault — usually the backend is slow. |

That distinction saves a great deal of time: 502 means look at whether the backend is running; 504 means it is running and slow; 503 means NGINX decided not to send the request at all; 499 means the user left.

Guides for each: 502 bad gateway, 503 service temporarily unavailable, 504 gateway timeout, upstream timed out, no live upstreams, connect failed connection refused, upstream prematurely closed connection, upstream sent too big header, 400 request header too large and 413 request entity too large.

TLS and certificate chains

Nearly every “works in my browser, fails from curl or Java” report is one thing: an incomplete certificate chain.

server {
    listen 443 ssl;
    http2 on;
    server_name example.com;

    # MUST be the full chain: leaf + intermediates, in that order.
    ssl_certificate     /etc/ssl/example.com/fullchain.pem;
    ssl_certificate_key /etc/ssl/example.com/privkey.pem;

    ssl_protocols       TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers off;
    ssl_session_cache   shared:SSL:10m;
    ssl_session_timeout 1d;

    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
}

# What chain is the server REALLY sending? (count the certificates)
openssl s_client -connect example.com:443 -servername example.com -showcerts </dev/null

# Expiry, without the noise
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates

# Do the certificate and key actually match? (the two hashes must be identical)
openssl x509 -noout -modulus -in fullchain.pem | openssl md5
openssl rsa  -noout -modulus -in privkey.pem   | openssl md5

-servername matters: without it you are not sending SNI, so on a multi-site host you will be shown the default server’s certificate and conclude the wrong thing.

TLS failures each have a guide: cannot load certificate, PEM no start line (usually a truncated or wrongly-formatted file), SSL private key mismatch, ssl_do_handshake failed, peer closed connection during SSL handshake and, for TLS to the backend, upstream SSL certificate verify failed. Certificate content problems: x509 certificate expired and name mismatch.

Rate limiting and connection limits

http {
    limit_req_zone  $binary_remote_addr zone=perip:10m rate=10r/s;
    limit_conn_zone $binary_remote_addr zone=conns:10m;

    server {
        location /api/ {
            limit_req  zone=perip burst=20 nodelay;
            limit_conn conns 10;
            limit_req_status 429;      # 503 by default — 429 is far more honest
            proxy_pass http://app;
        }
    }
}

burst allows a short spike to queue; nodelay serves those queued requests immediately rather than spacing them out, which is usually what you want for an API. Without nodelay a burst is delayed rather than rejected, which looks like latency rather than throttling.

$binary_remote_addr is the right zone key because it stores 4 bytes instead of a string — a 10m zone holds roughly 160,000 addresses. Behind a load balancer, $remote_addr is the balancer, so you need real_ip configured or you will rate-limit everyone as one client.

Logging that names the culprit

The default access log cannot tell you whether NGINX or the backend was slow. Fix that once:

log_format upstreamlog
    '$remote_addr - $remote_user [$time_local] "$request" '
    '$status $body_bytes_sent '
    'rt=$request_time uct=$upstream_connect_time '
    'uht=$upstream_header_time urt=$upstream_response_time '
    'us=$upstream_status ua=$upstream_addr';

access_log /var/log/nginx/access.log upstreamlog;
error_log  /var/log/nginx/error.log warn;

# Slowest requests in the last 10k lines
tail -10000 /var/log/nginx/access.log | awk '{print $NF, $0}' | sort -rn | head -20

# Error log, most recent first — the error log is where NGINX explains itself
tail -100 /var/log/nginx/error.log

The last line of an error-log entry names the upstream and the client, which is usually enough to identify the failing backend without any other tooling.

Performance and limits

Raising worker_connections without raising the file-descriptor limit does nothing — the two must move together, which is why too many open files and worker_connections are not enough so often appear at the same moment.

Troubleshooting specific errors

The NGINX failures engineers hit most often, each with a dedicated guide:

• 503 service temporarily unavailable — every upstream marked down, or a rate limit rejecting.
• 502 bad gateway — reached the backend, got something unusable.
• 504 gateway timeout — backend accepted and did not answer in time.
• 400 request header too large — raise large_client_header_buffers.
• 413 request entity too large — raise client_max_body_size.
• connect failed, connection refused upstream — nothing listening on the backend port.
• upstream sent too big header — raise proxy_buffer_size.
• upstream prematurely closed connection — the backend died mid-response.
• cannot load certificate — path, permissions, or the wrong PEM.
• SSL private key mismatch — cert and key are not a pair.
• directive not allowed here — right directive, wrong context.
• cannot assign requested address — ephemeral port exhaustion; enable upstream keepalive.
• too many open files — raise worker_rlimit_nofile and the OS limit together.
• 403 forbidden and permission denied connecting upstream — filesystem permissions, or SELinux blocking the proxy connection.

For anything else, validate the file in the NGINX config validator or paste the message into the Incident Assistant.

Production checklist

• nginx -t before every reload. Reload, do not restart, unless listen changed.
• nginx -T when a directive “isn’t working”. Read the resolved config, not the files.
• Define an explicit default_server that rejects unknown hostnames.
• Decide the proxy_pass trailing slash deliberately. It rewrites the URI.
• Enable upstream keepalive properly — proxy_http_version 1.1 plus an empty Connection header.
• Re-set every proxy_set_header you need inside a location that defines any of them.
• Serve the full chain, and verify it with openssl s_client -showcerts from outside the host.
• Use a resolver for backends whose IP changes. Startup resolution is once, forever.
• limit_req_status 429 so throttling is distinguishable from an outage.
• Log $upstream_response_time and $request_time. Without both you cannot tell who was slow.
• Move worker_connections and the fd limit together.
• Alert on certificate expiry, not on the outage it causes.

Frequently asked questions

Why does my proxy_pass return 404 from the application?

Almost always the trailing slash. proxy_pass http://backend; passes the original URI unchanged, so /api/users arrives as /api/users. proxy_pass http://backend/; replaces the matched location prefix, so the same request arrives as /users. Both are legitimate — the bug is a mismatch between which one you wrote and what the application expects. Check the access log on the backend to see the path it actually received; that settles it immediately.

What is the difference between 502, 503 and 504?

They describe three different failures. 502 means NGINX connected to an upstream and got something it could not use — connection refused or reset, or a malformed HTTP response. 503 means NGINX had nothing to send the request to: every upstream is marked down by max_fails, or a limit_req/limit_conn rejected it. 504 means the upstream accepted the connection and then failed to respond within proxy_read_timeout. So 502 says check whether the backend is running, 504 says it is running and slow, and 503 says NGINX never tried.

Why does my site work in a browser but fail from curl?

An incomplete certificate chain. ssl_certificate must contain the leaf certificate followed by every intermediate — that is what fullchain.pem is. Browsers frequently repair a missing intermediate by fetching it themselves, so the site appears fine, while curl, Java, Go and mobile clients reject it. Check what you actually serve with openssl s_client -connect host:443 -servername host -showcerts and count the certificates returned.

Do I need to reload NGINX after renewing a certificate?

Yes. NGINX loads certificates into the worker processes at startup and keeps using them until the workers are replaced, so a renewed file on disk changes nothing until you reload. A graceful nginx -s reload picks up the new certificate with no dropped connections, which is why certbot’s deploy hook does exactly that. A certificate that renewed successfully and still serves as expired is nearly always a missing reload hook.

Why is my location block never matching?

Because location matching is not top-to-bottom. NGINX takes an exact = match first, then remembers the longest prefix match, then — unless that prefix used ^~ — evaluates regex locations in file order and takes the first hit, only falling back to the remembered prefix if no regex matched. So a regex block placed below another matching regex never runs, and a ^~ prefix will deliberately prevent any regex from being considered. Dump the config with nginx -T and walk the order.

How do I stop NGINX proxying to a dead backend IP?

Use a resolver. A plain server name:port; inside an upstream block is resolved once at startup and never re-checked, so a container or Kubernetes service that changes address keeps receiving traffic at the old IP until a reload. Define resolver with a valid DNS server and use a variable in proxy_pass so the name is re-resolved with its TTL — remembering that a variable in proxy_pass also changes URI handling, so set the path explicitly.

Related resources

• Tool: NGINX config validator — paste a config and check directives, proxying and TLS before a reload.
• Guide: Linux Commands — the host-level networking and filesystem diagnosis underneath these errors.
• Guide: Cloud Security — TLS, certificates and the trust model in more depth.
• Stack hub: NGINX command centre — the top NGINX errors, tools and runbook in one place.
• Tool: Incident Assistant — paste an error and get an ordered triage plan.