🚨 Linux Error: Segmentation fault (core dumped) — Cause, Fix, and Troubleshooting Guide

Summary

Segmentation fault (core dumped) means a process accessed memory it was not allowed to touch. The CPU raised a fault, the kernel delivered SIGSEGV (signal 11), the process died, and — because a core pattern is configured — a core dump was written for post-mortem analysis. The message is printed by the shell reporting the child’s exit signal. Your job is to capture the core and read the backtrace to find where and why.

Common Symptoms

• A program prints Segmentation fault (core dumped) and exits with code 139 (128 + 11).
• dmesg shows a segfault at <addr> ip <addr> sp <addr> line naming the binary and library.
• The crash is reproducible with certain inputs, or intermittent under load (a memory-corruption smell).
• A systemd service repeatedly restarts, with coredumpctl listing fresh cores each time.

Most Likely Causes of the ‘Segmentation fault (core dumped)’ Error

Common production causes of the Segmentation fault (core dumped) error:

1. A bug in the program or a linked library — null/dangling pointer, buffer overflow, use-after-free, stack overflow from unbounded recursion.
2. ABI / shared-library mismatch — the binary was built against one .so version and is running against an incompatible one.
3. Corrupted or incompatible input data — malformed files/records that the parser mishandles.
4. Hardware/memory fault — bad RAM, showing as intermittent, non-deterministic crashes across different programs.
5. Stack size exhaustion — deep recursion or large stack allocations hitting ulimit -s.

Quick Triage

# Confirm the exit signal (139 = SIGSEGV)
your-program; echo "exit=$?"

# Kernel's view of the fault — names binary + faulting library
dmesg | grep -i segfault | tail

# Is core dumping even enabled?
ulimit -c
cat /proc/sys/kernel/core_pattern

An exit of 139 plus a dmesg segfault line pointing at a specific .so narrows the fault to that library immediately.

Diagnostic Commands

# Kernel segfault records: 'segfault at ADDR ip ADDR sp ADDR error N in LIB'
dmesg -T | grep -i segfault

# Where do cores go? (systemd-coredump vs a path pattern)
cat /proc/sys/kernel/core_pattern

# On systemd hosts, list and inspect captured cores
coredumpctl list
coredumpctl info <PID|COMM>
coredumpctl gdb <PID|COMM>        # opens gdb on the core

# Ensure cores are allowed for a manual repro
ulimit -c            # 0 means disabled
ulimit -c unlimited

# Load the core in gdb manually (non-systemd core_pattern)
gdb /path/to/program /path/to/core
# then, inside gdb:
#   bt full           full backtrace with locals
#   info registers
#   info sharedlibrary

# Check the library the binary actually links (ABI mismatch hunt)
ldd /path/to/program

# Install debug symbols so the backtrace is readable (see Fix)

coredumpctl gdb <name> is the fastest path on modern Ubuntu/RHEL: it pulls the captured core and opens gdb with the matching executable in one step.

Fix / Remediation

1. Enable core capture if it was off, then reproduce:

   ulimit -c unlimited
   # systemd hosts already capture via:
   cat /proc/sys/kernel/core_pattern   # |/lib/systemd/systemd-coredump ...

2. Install matching debug symbols so the backtrace names real functions. Ubuntu/Debian:

   sudo apt-get install -y <pkg>-dbgsym gdb
   # (enable the ddebs.ubuntu.com dbgsym repo first)

   RHEL/Rocky:

   sudo dnf install -y gdb
   sudo debuginfo-install -y <pkg>

3. Get the backtrace and identify the faulting frame:

   coredumpctl gdb <COMM>
   (gdb) bt full
   (gdb) frame <N>
   (gdb) list

4. Resolve a library mismatch if ldd/info sharedlibrary shows an unexpected version — reinstall/pin the correct package or rebuild against the deployed ABI.

5. Fix the code for the classic bugs (null-deref, overflow, use-after-free). Build with sanitizers to catch it early:

   gcc -g -fsanitize=address,undefined -o prog prog.c
   ./prog        # ASan prints the exact faulting line

6. Raise the stack limit only if the crash is genuine deep recursion:

   ulimit -s 16384      # 16 MB

Warning: Do not blindly increase ulimit -s or paper over the crash by auto-restarting the service — a segfault under load is often memory corruption that can silently corrupt data or be exploitable. Get the backtrace and fix the root cause; on suspected bad RAM, run memtester/memtest86+ and drain the host.

Validation

your-program; echo "exit=$?"     # expect 0, not 139
dmesg -T | grep -i segfault | tail   # no new entries after the fix
coredumpctl list                 # no fresh cores accumulating

A clean exit code, no new dmesg segfault lines, and no fresh cores after replaying the triggering input confirm the fix.

Prevention

• Keep debug symbols and gdb available on production-adjacent hosts so the first crash is analyzable, not a re-run gamble.
• Run CI builds with AddressSanitizer/UBSan and fuzzing on parsers that handle untrusted input.
• Pin library versions between build and runtime to avoid ABI drift; verify with ldd in the image.
• Leave systemd-coredump enabled and monitor coredumpctl list so repeated crashes page someone.
• Include RAM tests in hardware burn-in to catch faults that masquerade as software segfaults.

Related Errors

• Out of memory: killed process
• Cannot allocate memory
• Exec format error
• Text file busy

Final Notes

Segmentation fault (core dumped) is SIGSEGV plus a written core: the process touched forbidden memory and the kernel saved the evidence. Read the dmesg segfault line for the faulting library, then use coredumpctl gdb (or plain gdb on the core) with debug symbols to get bt full. Most cases are a code bug or an ABI mismatch; intermittent, cross-program crashes point at hardware. Fix the root cause rather than raising limits or auto-restarting into the same fault.

Want faster Linux incident response? Use DevOps AI Toolkit to turn production errors into clear diagnostics, remediation steps, and reusable runbooks.