🚨 NGINX Error Guide: 'worker process exited on signal 11' — Fix Worker Crashes

Overview

NGINX logs this when the master process notices one of its workers died from a fatal signal and restarts it. Signal 11 (SIGSEGV) is a segmentation fault — the worker touched invalid memory and the kernel killed it:

2026/07/09 08:52:19 [alert] 771#771: worker process 8123 exited on signal 11 (core dumped)

You may also see signal 6 (SIGABRT, an assertion/abort) or signal 9 (SIGKILL, usually the OOM killer). The master immediately forks a replacement worker, so NGINX often keeps serving — but every crash drops the in-flight requests that worker was handling, and a tight crash loop degrades the whole service. Unlike config errors, this is a runtime memory fault, so nginx -t passes cleanly; the problem only shows under real traffic.

Symptoms

• The error log repeats worker process NNNN exited on signal 11 (core dumped) (or signal 6/9) at [alert] level.
• Intermittent dropped connections, resets, or 502s that don’t map to any backend failure.
• nginx -t passes — the config is syntactically valid; the crash is at runtime.
• Crashes cluster around a specific request pattern (a route, a large upload, a particular header, a TLS handshake) or around memory pressure.
• Worker PIDs in ps keep changing as workers die and respawn.

Common Root Causes

• A buggy third-party module — a dynamically loaded module (Lua, a custom filter, an old cache/security module) with a memory bug that faults under certain input.
• A mismatched or corrupt build — a module compiled against a different NGINX version, or a partially upgraded binary.
• OOM killer (signal 9) — the host ran out of memory and the kernel killed workers; often paired with large buffers or many connections.
• Malformed input hitting an edge case — a specific request, header, or TLS handshake that trips a crash path in NGINX or a module.
• Hardware/library faults — a broken OpenSSL/library ABI mismatch after a system upgrade.

Diagnostic Workflow

First identify the signal and how often it happens — signal 11 vs 9 points at very different causes:

grep -E 'exited on signal' /var/log/nginx/error.log | tail -30
grep -E 'exited on signal' /var/log/nginx/error.log | grep -oE 'signal [0-9]+' | sort | uniq -c

Rule out OOM immediately, since signal 9 is usually the kernel, not a bug:

dmesg -T | grep -iE 'killed process|out of memory|oom'
journalctl -k | grep -i 'oom'

Check which modules are loaded — third-party modules are the most common source of segfaults:

nginx -V 2>&1 | tr ' ' '\n' | grep -iE 'module|add-dynamic'
grep -rn 'load_module' /etc/nginx/

Enable core dumps so you can capture a backtrace of the fault. In nginx.conf:

worker_rlimit_core 500m;
working_directory /var/dump/nginx;   # must exist and be writable by the worker user

After a crash reproduces, inspect the core with a debugger to see which module/function faulted:

ls -lh /var/dump/nginx/
gdb $(command -v nginx) /var/dump/nginx/core.8123 -ex 'bt full' -batch 2>/dev/null | head -40

To confirm a suspect module, disable it (comment its load_module), then validate and reload:

nginx -t && nginx -s reload

Example Root Cause Analysis

A site began logging worker process exited on signal 11 (core dumped) a few times a minute after a routine package update. nginx -t passed, and the crashes didn’t correlate with any backend. dmesg showed no OOM kills, ruling out memory pressure and pointing at an actual segfault rather than a signal 9.

The team enabled worker_rlimit_core and a writable working_directory, reproduced a crash, and ran gdb with bt full on the core. The backtrace pointed straight into a third-party dynamic module that had NOT been rebuilt during the update — its ABI no longer matched the newly upgraded NGINX binary, so it dereferenced garbage on certain requests. Commenting out that module’s load_module line and reloading stopped the crashes instantly. The permanent fix was rebuilding the module against the current NGINX version (and pinning that step into the upgrade pipeline so binary and modules always move together).

Prevention Best Practices

• Keep third-party/dynamic modules rebuilt in lockstep with the NGINX binary; an ABI mismatch after an upgrade is a classic segfault source.
• Enable worker_rlimit_core and a writable working_directory on hosts where crashes might occur, so you can get a backtrace instead of guessing.
• Distinguish the signal first: 11/6 means a code/memory bug (often a module), 9 means the OOM killer — check dmesg before anything else.
• Minimize third-party modules to what you actually need; each one is attack surface and a potential crash path.
• Load-test new modules and NGINX upgrades in staging with production-like traffic before rolling out.
• Alert on exited on signal in the error log so a slow crash loop doesn’t silently drop requests for hours.

Quick Command Reference

# Count and identify the signals workers are dying on
grep -E 'exited on signal' /var/log/nginx/error.log | grep -oE 'signal [0-9]+' | sort | uniq -c

# Rule out the OOM killer
dmesg -T | grep -iE 'killed process|out of memory'

# List loaded dynamic modules
nginx -V 2>&1 | tr ' ' '\n' | grep -i module
grep -rn 'load_module' /etc/nginx/

# Backtrace a core dump (after enabling worker_rlimit_core)
gdb $(command -v nginx) /path/to/core -ex 'bt full' -batch | head -40

# Validate and reload after disabling a suspect module
nginx -t && nginx -s reload

Conclusion

worker process exited on signal 11 is a runtime segfault, not a config error — which is why nginx -t passes while workers keep dying. Start by reading the signal: 11/6 points at a code or module bug (frequently a third-party module out of sync with the NGINX binary), while signal 9 is usually the OOM killer, confirmable in dmesg. Enable core dumps, get a backtrace to name the faulting module, and fix the root cause — rebuild or remove the offending module — rather than living with a crash-restart loop that quietly drops requests.