🚨 NGINX Error Guide: '(13: Permission denied) while connecting to upstream' (SELinux)

Exact Error Message

When NGINX cannot complete an outbound connection to a backend it logs a connect() failure in the error log. The hallmark of the SELinux variant is errno (13: Permission denied):

2026/06/27 10:14:02 [crit] 2841#2841: *512 connect() to 127.0.0.1:9000 failed (13: Permission denied) while connecting to upstream, client: 203.0.113.44, server: app.example.com, request: "GET /api/health HTTP/1.1", upstream: "http://127.0.0.1:9000/", host: "app.example.com"

At the same moment, the kernel audit subsystem records an Access Vector Cache (AVC) denial for the NGINX process. This is the smoking gun:

type=AVC msg=audit(1782900842.115:9013): avc:  denied  { name_connect } for  pid=2841 comm="nginx" dest=9000 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:hi_reserved_port_t:s0 tclass=tcp_socket permissive=0

The browser-facing symptom is usually an HTTP 502 Bad Gateway (or 504 on timeout) returned to clients while the upstream service itself is healthy.

What the Error Means

NGINX runs as a reverse proxy and opens a TCP socket to an upstream (PHP-FPM, a Node app, Gunicorn, Tomcat, another NGINX, etc.). On RHEL, Rocky Linux, AlmaLinux, CentOS Stream and Fedora, SELinux is enabled and enforcing by default. The NGINX worker process runs in the httpd_t security domain, and SELinux policy tightly restricts which network operations that domain may perform.

By default the httpd_t domain is not allowed to make arbitrary outbound network connections. When NGINX calls connect(), the kernel asks SELinux for permission. If policy denies the name_connect permission on that destination port, the connect() call returns errno 13 (EACCES, “Permission denied”), and NGINX surfaces it verbatim.

Crucially, this happens before any packet leaves the box. The backend never sees the connection attempt. That is what distinguishes it from a backend being down: with errno 13 the socket is blocked by local policy, not refused by a remote.

Common Causes

• httpd_can_network_connect is off. This SELinux boolean (off by default) governs whether httpd_t may open general outbound TCP connections. Proxying to a TCP backend on a non-standard port almost always requires it.
• Backend port is not labeled http_port_t. SELinux allows httpd_t to connect to a fixed set of ports labeled http_port_t (80, 81, 443, 488, 8008, 8009, 8443, 9000 in many policies). A backend on 3000, 5000, 8081, or any unlisted port is denied unless the boolean is on or the port is relabeled.
• httpd_can_network_relay needed for upstream proxying. When NGINX acts purely as a relay/proxy, this boolean (or httpd_can_network_connect) must be enabled.
• A custom or moved upstream port (for example moving PHP-FPM from 9000 to 9001) that no longer carries an allowed label.
• Recent OS hardening or migration to a RHEL-family distro where SELinux is enforcing, when the same config worked on Debian/Ubuntu (which ship AppArmor and a more permissive default).

How to Reproduce the Error

On an enforcing RHEL-family host, configure NGINX to proxy to a backend listening on an unlabeled port and confirm SELinux is active:

getenforce
# Enforcing

A minimal proxy block pointed at, say, a Node app on port 3000:

upstream app { server 127.0.0.1:3000; }
server {
    listen 80;
    server_name app.example.com;
    location / { proxy_pass http://app; }
}

With the backend running and reachable locally, a request through NGINX returns 502, and error.log shows connect() ... failed (13: Permission denied). Meanwhile a direct curl to the backend succeeds, proving the service is up and the block is purely SELinux policy.

Diagnostic Commands

Start by confirming the config is valid and identifying whether the problem is policy versus a dead backend. All commands here are read-only:

# 1. Validate NGINX configuration syntax
sudo nginx -t

# 2. Is SELinux actually enforcing?
getenforce

# 3. Check the key boolean (current and persistent state)
getsebool httpd_can_network_connect

# 4. Is the backend actually listening locally?
ss -ltnp | grep ':3000'

# 5. Prove the backend answers when NGINX is bypassed
curl -I http://127.0.0.1:3000/

# 6. Pull recent AVC denials from the audit log
sudo ausearch -m avc -ts recent

# 7. Get a human-readable analysis and suggested fix
sudo sealert -a /var/log/audit/audit.log

Interpretation:

• If ss shows the backend listening and curl returns a response, the service is healthy. A 502 on top of that points squarely at SELinux.
• If ausearch returns an entry with comm="nginx", scontext=...:httpd_t:s0, and denied { name_connect }, you have confirmed the SELinux cause. Note the dest= port and tcontext label.
• sealert reads the same audit data and prints the exact setsebool/semanage command it recommends, which is invaluable for confirming the right fix.

Step-by-Step Resolution

1. Confirm the denial. Run sudo ausearch -m avc -ts recent and verify the entry references nginx, httpd_t, and the name_connect permission on your backend port. If there is no AVC and getenforce says Enforcing, errno 13 may instead be a filesystem/socket permission issue rather than network policy.

2. Decide between the two correct fixes. There are two policy-clean approaches; pick based on your situation.

Option A — allow outbound connections (most common). Enabling the boolean lets httpd_t connect to any port. This is the standard fix when NGINX proxies to one or more dynamic/local backends:

sudo setsebool -P httpd_can_network_connect on

The -P flag makes the change persistent across reboots. Omit it only for a throwaway test.

Option B — label the specific port (least privilege). If you prefer to keep the boolean off and only permit the one backend port, relabel that port as http_port_t:

sudo semanage port -a -t http_port_t -p tcp 3000

If semanage is missing, install it with sudo dnf install policycoreutils-python-utils. To inspect existing labels first, run sudo semanage port -l | grep http_port_t. Use -m instead of -a to modify a port that already has a different label.

3. Reload NGINX and re-test. No NGINX config change is required for the SELinux fix, but reload to clear cached upstream state and re-validate syntax:

sudo nginx -t && sudo systemctl reload nginx

4. Verify. Re-run curl -I https://app.example.com/ through NGINX. The 502 should be gone. Re-run sudo ausearch -m avc -ts recent to confirm no new denials appear. If your incident workflow needs a structured writeup, the incident-response dashboard can generate one from the AVC and log evidence.

Never just run setenforce 0. Disabling SELinux makes the symptom vanish but leaves the host unprotected and masks the real policy gap. Use the boolean or port label instead.

Prevention and Best Practices

• Bake the boolean into provisioning. Set httpd_can_network_connect (or the port label) in your Ansible, Puppet, or kickstart so freshly built hosts proxy correctly from day one. Ansible’s ansible.posix.seboolean module makes the state declarative.
• Prefer least privilege when feasible. If you have a small, fixed set of backend ports, label them with semanage port rather than opening the broad boolean. Reserve the boolean for hosts with many dynamic upstreams.
• Standardize backend ports. Keeping PHP-FPM on 9000 (already http_port_t in most policies) avoids relabeling entirely.
• Monitor for AVC denials. Ship audit.log to your log pipeline and alert on denied { name_connect } ... comm="nginx" so policy gaps surface before users hit 502s.
• Test in permissive mode during migrations. Temporarily setenforce 0 only on a staging box to enumerate every denial via ausearch, generate policy, then return to enforcing.
• Audit getsebool -a | grep httpd periodically to document which booleans your fleet depends on.

Related Errors

• (111: Connection refused) while connecting to upstream — errno 111 means the connection left NGINX and was actively refused by the destination: the backend is down, crashed, or not listening on that port. This is the opposite of errno 13. If ss -ltnp shows nothing on the port, you are looking at a dead backend, not SELinux.
• (13: Permission denied) while connecting to upstream (unix:/...sock) — the same errno but over a Unix socket usually points at filesystem/SELinux file context on the socket (httpd_sys_rw_content_t vs the socket’s label), not name_connect.
• upstream timed out (110: Connection timed out) — the backend accepted but did not respond in time, or a firewall is dropping packets silently.
• 502 Bad Gateway — the generic client-facing result of any of the above; always read error.log to find the underlying errno.

For more NGINX fixes, browse the NGINX category.

Frequently Asked Questions

How do I know it is SELinux and not the backend being down? Check the errno. Errno 13 (Permission denied) plus an AVC denied { name_connect } for comm="nginx" means SELinux blocked the connection locally. Errno 111 (Connection refused) means the packet reached the destination and was refused, so the backend is down. Confirm with curl -I directly to the backend: if that works but NGINX still fails with errno 13, it is SELinux.

Should I use the boolean or the port label? Use setsebool -P httpd_can_network_connect on when NGINX proxies to many or dynamic ports, it is the simplest robust fix. Use semanage port -a -t http_port_t -p tcp <port> for least privilege when you have one or two fixed backend ports and want to keep the boolean off.

Will setenforce 0 fix it? It will make the error disappear, but it disables SELinux enforcement system-wide and is not a fix. Use it only briefly on staging to enumerate denials, then return to Enforcing and apply the boolean or port label.

Why did this work fine on Ubuntu? Debian and Ubuntu ship AppArmor with a permissive default for web servers, while RHEL-family distros ship SELinux enforcing. The same NGINX config that proxies freely on Ubuntu hits the httpd_t network restriction on RHEL, Rocky, and Alma.