🚨 Linux Error: 'Failed to initialize NVML: Driver/library version mismatch' — Cause, Fix, and Troubleshooting Guide

Overview

On a GPU or ML node, nvidia-smi suddenly refuses to run and every CUDA workload fails to see the GPU:

Failed to initialize NVML: Driver/library version mismatch

NVML (the NVIDIA Management Library) is the userspace library that nvidia-smi, CUDA, and container runtimes use to talk to the GPU. This error means the loaded NVIDIA kernel module and the userspace driver libraries on disk are different versions. It happens almost every time the NVIDIA driver package is upgraded while the machine is still running: the new .so libraries land on disk immediately, but the kernel already has the old nvidia.ko module loaded and in use. NVML sees the version numbers disagree and bails out.

Crucially, nothing is corrupt — the fix is to get both halves back in sync by reloading the module or rebooting. This most often bites ML training/inference hosts and GPU Kubernetes nodes where unattended-upgrades quietly pulled a new driver overnight.

Symptoms

• nvidia-smi prints Failed to initialize NVML: Driver/library version mismatch and exits non-zero.
• CUDA applications fail with CUDA error: system has unsupported display driver / cuda driver combination or cannot enumerate devices.
• GPU pods on a Kubernetes node go Pending/CrashLoopBackOff; the NVIDIA device plugin logs NVML init failures and advertises zero GPUs.
• The node was fine yesterday; an apt/dnf history shows a recent NVIDIA driver upgrade with no reboot since.

nvidia-smi

Failed to initialize NVML: Driver/library version mismatch

Common Root Causes

1. Unattended/partial driver upgrade without a reboot

unattended-upgrades (Ubuntu/Debian) or a dnf/yum update installed a newer nvidia-driver package. The userspace libs are now the new version; the running kernel still holds the old nvidia.ko. Until the module is reloaded or the box rebooted, they mismatch.

2. New kernel installed, module rebuilt for it, but old module still loaded

A kernel update triggered DKMS to build the nvidia module for the new kernel. The still-running old kernel has the old module loaded, and the on-disk libraries were updated too — three moving parts, two of them out of step.

3. Version pin drift between the driver and the kernel module (kmod)

The driver libraries and the packaged kmod come from different repositories or were pinned to different versions, so a partial upgrade leaves the userspace .so at one version and the built module at another.

4. A manual .run installer over a package-managed driver (or vice versa)

Mixing the .run installer and distro packages leaves libraries from one source and a module from the other, producing a permanent mismatch until one is cleaned out.

How to diagnose

Step 1: Read the version each half reports

# Userspace / running-driver view (this is what NVML compares against)
cat /proc/driver/nvidia/version

NVRM version: NVIDIA UNIX x86_64 Kernel Module  560.40.07  ...

# The kernel module file on disk (what would load on next modprobe/reboot)
modinfo -F version nvidia

560.85.02

When /proc/driver/nvidia/version (the loaded module) and modinfo (the on-disk module) disagree, that is the mismatch, confirmed.

Step 2: Confirm which package versions are installed

# Debian/Ubuntu
dpkg -l | grep -i nvidia
# RHEL/Rocky/Fedora
rpm -qa | grep -i nvidia

Look for a userspace library package (e.g. libnvidia-*) at a newer version than the loaded module reports.

Step 3: Check the upgrade history to confirm the trigger

# Debian/Ubuntu — did unattended-upgrades touch the driver?
grep -i nvidia /var/log/apt/history.log
grep -i nvidia /var/log/unattended-upgrades/unattended-upgrades.log
# RHEL family
dnf history | head; rpm -q --last kernel | head
uptime   # confirm no reboot since the upgrade

Step 4: See whether any process is holding the GPU

sudo lsof /dev/nvidia* 2>/dev/null
lsmod | grep nvidia

You cannot unload the module while a process (training job, nvidia-persistenced, a container) holds /dev/nvidia*. This decides whether you can reload live or must reboot.

Fixes

Option A: Reboot (simplest, always works)

If the node can drain, reboot — it loads the new module cleanly to match the new libraries:

# Optional: drain a Kubernetes node first
kubectl drain gpu-node-03 --ignore-daemonsets --delete-emptydir-data

sudo reboot

After it comes back:

nvidia-smi   # should now print the GPU table

Option B: Reload the modules live (no reboot)

Only possible when no process is using the GPU. Stop GPU workloads and the persistence daemon first, then unload and reload the module stack:

# 1. Stop anything holding /dev/nvidia*
sudo systemctl stop nvidia-persistenced
sudo lsof /dev/nvidia* 2>/dev/null      # must return nothing

# 2. Unload the NVIDIA module stack (order matters — dependents first)
sudo rmmod nvidia_uvm nvidia_drm nvidia_modeset nvidia

# 3. Reload — modprobe pulls the current on-disk module and its deps
sudo modprobe nvidia

# 4. Restart persistence mode and verify
sudo systemctl start nvidia-persistenced
nvidia-smi

If rmmod complains the module is in use, a process still holds the GPU (step 1’s lsof will show it) — stop it or fall back to a reboot.

Prevent it from recurring

Pin the driver and the kernel module to the same version and keep unattended-upgrades from moving them out from under a running box:

# Debian/Ubuntu — hold the driver packages
sudo apt-mark hold 'nvidia-driver-*' 'libnvidia-*' 'nvidia-dkms-*'

# Or exclude NVIDIA from unattended-upgrades:
# in /etc/apt/apt.conf.d/50unattended-upgrades add to Package-Blacklist:
#   "nvidia-";
#   "libnvidia-";

# RHEL/Rocky/Fedora — pin via versionlock or exclude in dnf.conf
sudo dnf install -y python3-dnf-plugin-versionlock
sudo dnf versionlock add 'nvidia-driver-*' 'kmod-nvidia-*'

Then upgrade drivers deliberately, in a maintenance window, followed by a reboot — never as a silent background update.

What to watch out for

• Nothing is broken — do not reinstall the driver in a panic. The libraries and module are both fine; they are just different versions. Reload or reboot first.
• Reload order matters. Unload the dependents (nvidia_uvm, nvidia_drm, nvidia_modeset) before nvidia, or rmmod nvidia fails with “Module nvidia is in use”.
• nvidia-persistenced holds the module. Stop it before rmmod, or the unload fails even with no user jobs running.
• A live reload does not survive if the loaded libs were also swapped for a third version. If reload still mismatches, reboot — it is the only way to guarantee all three (kernel, module, libs) agree.
• On Kubernetes, drain the node first. Reloading or rebooting under active GPU pods loses their work; the NVIDIA device plugin re-advertises GPUs once nvidia-smi succeeds again.
• The real prevention is version pinning + coordinated upgrades, not repeated live reloads. An un-pinned GPU fleet will hit this again on the next unattended driver bump.

Related

• Linux Error: modprobe module not found
• Linux Error: cannot open shared object file
• Linux Error Guide: ‘Failed to start’ systemd Service Won’t Start
• All Linux admin guides

Want faster Linux incident response? Use the free incident assistant to turn GPU driver and kernel-module errors into clear diagnostics and reusable runbooks.