🐧 eu-unstrip: Fixing Missing debuginfo When Reading a Core Dump

The Symptom

You extracted a core dump captured by systemd-coredump, opened it, and every frame in the backtrace is useless:

(gdb) bt
#0  0x00007f2a1c0b4e2f in ?? ()
#1  0x00007f2a1c0a1120 in ?? ()
#2  0x000055a3f1c90b27 in ?? ()

Those ?? markers mean gdb found the binary but not the debuginfo that maps addresses back to function names and source lines. The build-id embedded in the ELF file did not resolve to any separate debug file on this machine. This guide is about turning those ?? frames back into real symbols by matching and installing the correct debuginfo.

Reading eu-unstrip -n Output

The fastest way to see what is missing is eu-unstrip from the elfutils package. Point it at the core:

eu-unstrip -n --core=/tmp/myapp.core

0x400000+0x21000  a3f1c9c8...b27 0x0 /usr/bin/myapp /usr/bin/myapp
0x7f2a...+0x1f0000 5d8e2f11...a10 -   /lib64/libc.so.6 libc.so.6
0x7f2b...+0x9000   9c14ab07...ee0 .   /lib64/libssl.so.3 libssl.so.3

Read the columns carefully, because the third field is the whole story:

• Column 2 is the 40-character build-id — the unique fingerprint of that exact binary build.
• Column 3 is the debuginfo status. A path or . means debug data is available or embedded; a bare - means no matching debuginfo was found and any symbols for that module will be ??.

So the libc.so.6 line above (-) is exactly why your libc frames are unresolved. Build-ids are how everything lines up: debuginfo is stored and looked up by build-id, so a debug package built from a different build of the library will not match even if the version string looks similar.

Why Symbols Show as ??

There are only a few root causes, and eu-unstrip tells you which one you have:

• The debuginfo package for that binary is simply not installed.
• Installed debuginfo has a build-id that differs from the binary in the core (mismatched package versions between the crashing host and the triage host).
• The binary was built locally and stripped, and its debuginfo was never saved or was overwritten by a rebuild.
• The library came from a third-party or container image whose debug symbols are not in your distro’s debuginfo repos.

The fix is always the same shape: obtain debuginfo whose build-id equals the one eu-unstrip printed.

Step-by-Step Resolution

1. Enumerate the missing modules and capture their build-ids:

eu-unstrip -n --core=/tmp/myapp.core

2. Install debuginfo for the packages that show -. On Fedora/RHEL, dnf debuginfo-install resolves the right debug packages for a binary:

sudo dnf debuginfo-install glibc openssl-libs

On Debian/Ubuntu, enable the -dbgsym repositories and install the matching *-dbgsym packages:

sudo apt install libc6-dbgsym libssl3-dbgsym

3. Prefer debuginfod, which fetches exactly the right debuginfo by build-id on demand — no guessing at package names. Enable the service and point clients at an index:

sudo systemctl enable --now debuginfod.service    # optional local server
export DEBUGINFOD_URLS="https://debuginfod.fedoraproject.org/"

Distributions ship a default in /etc/debuginfod/*.urls; sourcing /etc/profile.d/debuginfod.sh sets DEBUGINFOD_URLS for you.

4. Verify debuginfod can retrieve the debug file for a specific build-id straight from eu-unstrip. Use the build-id (strip the offset) with debuginfod-find:

debuginfod-find debuginfo 5d8e2f11...a10

/home/you/.cache/debuginfod_client/5d8e2f11...a10/debuginfo

5. Re-run eu-unstrip and confirm the - entries now resolve to a debug path instead:

eu-unstrip -n --core=/tmp/myapp.core

6. Open the core again. With DEBUGINFOD_URLS set, gdb will download missing debuginfo automatically on first use (answer y when it prompts, or set set debuginfod enabled on):

gdb /usr/bin/myapp /tmp/myapp.core

(gdb) bt
#0  0x00007f2a... in __memcpy_avx_unaligned () at memcpy-avx.S:250
#1  0x000055a3... in copy_record (dst=..., src=...) at record.c:88

Real functions and source lines mean the build-ids finally match. Standardizing this debuginfo workflow across a fleet pairs well with a saved recipe from our DevOps AI prompt library.

Prevention and Best Practices

• Set DEBUGINFOD_URLS globally on triage hosts so symbol fetching is automatic and always build-id correct.
• Keep the triage host on the same distro release as production; installing a mismatched debuginfo version is the number-one cause of persistent ?? frames.
• For in-house binaries, archive the unstripped build (or its .debug file) keyed by build-id in an artifact store or a private debuginfod.
• Confirm a fetch worked with debuginfod-find debuginfo <build-id> before spending time in gdb.
• Never assume the same version string means the same build — always compare the 40-hex build-id from eu-unstrip.

Frequently Asked Questions

What does a bare dash in eu-unstrip -n output mean? In the debuginfo column, - means no matching debug file was found for that module’s build-id, so its stack frames will render as ?? until you install matching debuginfo.

How do I install debuginfo that actually matches the core? Use debuginfod (export DEBUGINFOD_URLS=...) so files are fetched by build-id, or dnf debuginfo-install <pkg> / apt install <pkg>-dbgsym, ensuring the package versions match the crashing host.

What is DEBUGINFOD_URLS and why does it matter? It is the environment variable listing debuginfod servers that gdb, eu-stack, and debuginfod-find query to download debug symbols by build-id automatically, so you never hand-pick the wrong debug package.

Why do symbols still show ?? after I installed a debuginfo package? Almost always a build-id mismatch: the installed debuginfo came from a different build than the binary in the core. Compare build-ids with eu-unstrip -n and use debuginfod to fetch the exact one. For more debugging walkthroughs, see the Linux admin guides.