Linux Kernel Module Troubleshooting Prompt

You are a senior Linux sysadmin who has debugged kernel module failures across NVIDIA drivers, custom hardware drivers, ZFS DKMS, and Secure Boot. You can read `modprobe -v` and `dmesg` to spot signing mismatches, ABI breaks, and dependency loops.

I will provide:
- The symptom (`modprobe` fails, hardware not detected, "module not found," DKMS build fails, Secure Boot rejected)
- The module name and what it should do
- `modprobe -v <module>` output
- `dmesg | tail -50` lines around the load attempt
- Kernel version (`uname -r`)
- For DKMS: `dkms status`
- Whether Secure Boot is enabled (`mokutil --sb-state`)

Your job:

1. **Identify the failure mode**:
   - **"Module not found"** — kernel modules in `/lib/modules/$(uname -r)/`; missing for current kernel
   - **"Unknown symbol in module"** — module compiled against different kernel; ABI mismatch
   - **"Required key not available"** — Secure Boot rejecting unsigned module
   - **"Operation not permitted"** — Secure Boot lockdown mode
   - **DKMS build fail** — kernel-devel headers missing, compiler error
   - **Hardware not detected** — module loads but doesn't bind; udev rule, kernel parameter, or PCI ID mismatch
   - **Loop / dependency error** — `modprobe -v` shows dep chain; circular or missing dep
2. **For "not found" on current kernel**:
   - `find /lib/modules/$(uname -r)/ -name "<module>.ko*"` — is it there?
   - May be in a kernel package separate from base (e.g., `linux-modules-extra` on Ubuntu)
   - `modprobe -v --first-time <module>` — verbose without re-loading
3. **For Secure Boot rejection**:
   - `mokutil --sb-state` shows Secure Boot status
   - `dmesg | grep "key was rejected"` confirms
   - Options: disable Secure Boot (BIOS), enroll module signing key (MOK), use distro-signed modules only
   - **MOK enrollment**: `mokutil --import <pub.der>`, reboot, enroll via BIOS prompt
4. **For DKMS build failures**:
   - `dkms status` shows installed module versions and target kernels
   - `cat /var/lib/dkms/<module>/<ver>/build/make.log` — actual build error
   - Common: `linux-headers-$(uname -r)` not installed; install matching version
   - After kernel upgrade: `dkms autoinstall` rebuilds for new kernel
5. **For hardware not detected**:
   - `lspci -k` shows kernel driver bound to each device. `Kernel driver in use: <none>` = no binding.
   - `lspci -nn` shows vendor:device IDs; compare with module's `modinfo` aliases
   - `modinfo <module> | grep alias` — list of PCI IDs the module claims
   - If hardware not in alias list: kernel doesn't know; check upstream patches, vendor drivers
6. **For ABI mismatch ("unknown symbol")**:
   - Module compiled against kernel A, loaded into kernel B
   - Rebuild with DKMS or distro package matching current kernel
   - `modinfo <module> | grep vermagic` — vs `uname -r`
7. **For blacklisting / load order**:
   - `/etc/modprobe.d/<name>.conf` — `blacklist <module>`, `options <module> param=value`, `install` / `alias`
   - `/etc/modules-load.d/<name>.conf` — load at boot
   - initramfs needs rebuild after blacklist changes affecting boot-time modules (`update-initramfs -u` / `dracut -f`)
8. **For `rmmod` / unload failures**:
   - `lsmod` shows refcount; if non-zero, something uses the module
   - `lsof` / `fuser` for hardware modules
   - Can't `rmmod` modules with non-zero refcount; identify user first

Mark DESTRUCTIVE: blacklisting a module needed for booting (renders boot broken if initramfs not updated), `rmmod` of a module the running kernel depends on, force-loading unsigned modules under Secure Boot lockdown.

---

Symptom: [DESCRIBE]
Module: [name]
`uname -r`:
```
[PASTE]
```
`modprobe -v <module>`:
```
[PASTE]
```
`dmesg | tail -50`:
```
[PASTE]
```
`lspci -k` (for hardware): or `lsusb -v` for USB:
```
[PASTE relevant device]
```
Secure Boot state: [`mokutil --sb-state`]
DKMS status (if applicable):
```
[PASTE `dkms status`]
```

Why this prompt works

Kernel module errors look opaque because the failure can be at any of several layers: not present, not built for current kernel, signature rejected, ABI mismatch, or hardware not matching. The error messages are similar across these. This prompt forces a layer-by-layer walk.

How to use it

1. Always check uname -r against where the module lives: /lib/modules/$(uname -r)/. Missing for the running kernel is the #1 issue.
2. Check Secure Boot state. Half of “won’t load” issues are signature rejection.
3. For hardware, lspci -nn + modinfo alias identifies whether the kernel even has a driver for this PCI ID.
4. For DKMS, read the build log. It’s verbose but has the actual error.

Useful commands

# What modules are loaded
lsmod
lsmod | grep <module>

# Find the module file
find /lib/modules/$(uname -r)/ -name "<module>*.ko*"
modinfo <module>
modinfo <module> | grep -E "filename|vermagic|alias|signature"

# Load attempt
sudo modprobe -v <module>                  # verbose
sudo modprobe -v --first-time <module>     # error if already loaded
sudo insmod /path/to/module.ko             # raw load (no deps)

# Recent kernel messages
sudo dmesg | tail -50
sudo dmesg | grep -i <module>
sudo journalctl -k --since "5 minutes ago"

# Secure Boot
sudo mokutil --sb-state
sudo mokutil --list-enrolled              # list trusted keys

# DKMS
sudo dkms status
sudo dkms autoinstall                      # rebuild all for current kernel
sudo dkms install <module>/<version>
cat /var/lib/dkms/<module>/<version>/build/make.log

# Required headers / build env
ls /usr/src/linux-headers-$(uname -r)/      # Debian/Ubuntu
ls /usr/src/kernels/$(uname -r)/            # RHEL/Fedora
sudo apt install linux-headers-$(uname -r)  # Ubuntu
sudo dnf install kernel-devel-$(uname -r)   # RHEL

# Hardware view
lspci -k                                    # show driver per device
lspci -nn                                   # show vendor:device IDs
lsusb -v                                    # USB
sudo lshw -class network

# Module parameters
sudo modprobe <module> param=value
cat /sys/module/<module>/parameters/<param>

# Blacklist / load order
cat /etc/modprobe.d/blacklist.conf
ls /etc/modprobe.d/
ls /etc/modules-load.d/

# Rebuild initramfs after module config changes
sudo update-initramfs -u                    # Debian/Ubuntu
sudo dracut -f                              # RHEL/Fedora

# Unload
sudo modprobe -r <module>
sudo rmmod <module>

Sign your own modules (MOK enrollment)

# Generate key
sudo openssl req -new -x509 -newkey rsa:2048 -keyout MOK.priv -outform DER -out MOK.der -nodes -days 36500 -subj "/CN=my-key/"

# Sign module
sudo /usr/src/linux-headers-$(uname -r)/scripts/sign-file sha256 MOK.priv MOK.der /path/to/module.ko

# Enroll key (requires reboot)
sudo mokutil --import MOK.der
# Set a one-time password; reboot; complete enrollment in BIOS MokManager

# Verify post-enrollment
sudo mokutil --list-enrolled

DKMS rebuild after kernel upgrade

# After kernel update, DKMS should auto-build for new kernel.
# If it doesn't, manually:
sudo apt install linux-headers-$(uname -r)  # ensure headers
sudo dkms autoinstall

# Reboot, verify
lsmod | grep <module>

Common findings this catches

• uname -r doesn’t match the only /lib/modules/ directory — running an old kernel; reboot to apply update.
• modinfo shows vermagic: 5.15.0 running on 6.5.0 kernel — ABI mismatch; rebuild via DKMS.
• DKMS build log: fatal error: linux/module.h: No such file or directory — headers package missing or wrong version.
• dmesg shows Loading of unsigned module is rejected — Secure Boot lockdown; sign module or disable SB.
• lspci -k shows Kernel driver in use: (empty) — no driver bound; check vendor:device ID in modinfo alias.
• modprobe -r fails with “Module is in use” — lsmod shows refcount > 0; identify user.
• NVIDIA driver fails after kernel update — DKMS rebuild failed silently; check build log.

When to escalate

• Vendor driver vs kernel version compatibility — coordinate with hardware vendor; sometimes kernel patches needed.
• Secure Boot lockdown preventing all custom modules — security policy issue, not technical.
• Kernel oops/panic on module load — file a kernel bug; isolate the failing module by booting with blacklist=<mod> (via grub cmdline).