🐳 Docker Error Guide: 'nvidia-container-cli: initialization error' on --gpus all

Exact Error Message

$ docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: nvml error: driver/library version mismatch: unknown.

Other common variants of the same hook failure:

nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory
nvidia-container-cli: initialization error: driver error: failed to process request: unknown

What It Means

--gpus all tells Docker to invoke the NVIDIA Container Toolkit, which runs a prestart OCI hook (nvidia-container-cli) to inject the host GPU driver libraries and device nodes into the container. initialization error means that hook failed before the container even started, usually because the toolkit could not talk to the host NVIDIA driver through NVML (the NVIDIA Management Library).

The most frequent trigger is a driver/library version mismatch: the kernel module currently loaded and the userspace libnvidia-ml library are different versions, typically after a driver package upgrade without a reboot. Because this is a hook failure, the container is never created and no application output appears.

Common Causes

• The NVIDIA driver was upgraded but the machine was not rebooted, so the loaded kernel module and userspace libraries disagree.
• The NVIDIA Container Toolkit is not installed, or the Docker runtime hook was never registered.
• /etc/docker/daemon.json does not declare the nvidia runtime, or the daemon was not restarted after installing the toolkit.
• The host driver is missing or broken; nvidia-smi fails on the host itself.
• A kernel update loaded a new kernel without a matching rebuilt NVIDIA module (DKMS not rebuilt).
• Toolkit and driver major versions are incompatible (toolkit too old for a new driver branch).

Diagnostic Commands

First confirm the host driver works at all; if this fails, the container never will:

nvidia-smi

Check the loaded kernel module version against the userspace library version:

cat /proc/driver/nvidia/version

Confirm the toolkit is installed and the CLI runs standalone:

nvidia-container-cli --version
nvidia-container-cli info

Verify Docker knows about the nvidia runtime and hook:

docker info | grep -i runtime
cat /etc/docker/daemon.json

Step-by-Step Resolution

1. Fix the host driver first. Run nvidia-smi on the host. If it reports Failed to initialize NVML: Driver/library version mismatch, a driver upgrade is pending a reboot:

sudo reboot

If you cannot reboot immediately, reload the kernel modules (only when no GPU process is running):

sudo rmmod nvidia_uvm nvidia_drm nvidia_modeset nvidia
sudo modprobe nvidia

2. Install or reinstall the NVIDIA Container Toolkit if nvidia-container-cli is missing:

sudo apt-get install -y nvidia-container-toolkit

3. Register the runtime with Docker and restart the daemon:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

4. Confirm the runtime is now present:

docker info | grep -i nvidia

5. Re-run the GPU container. It should now print the driver and GPU table:

docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

6. If you hit libnvidia-ml.so.1: cannot open shared object file, the host driver is not fully installed. Reinstall the matching driver package and reboot:

sudo apt-get install --reinstall -y nvidia-driver-550
sudo reboot

7. After a kernel update, ensure the DKMS module rebuilt for the new kernel:

sudo dkms status
sudo dkms autoinstall

Prevention

• Always reboot (or reload modules) after a driver upgrade so the kernel module and userspace library stay in lockstep; the mismatch is the number-one cause.
• Pin the driver and toolkit versions in your image-build and provisioning pipelines so a surprise upgrade cannot drift them apart. The DevOps AI prompt library has prompts that emit a validated GPU-host bootstrap sequence.
• Add a health check that runs docker run --rm --gpus all ... nvidia-smi after provisioning and fails the build if the hook errors.
• Use DKMS so NVIDIA modules rebuild automatically on kernel updates, avoiding a missing module after reboot.
• Keep the toolkit at or above the version required by your driver branch; check the compatibility matrix before bumping the driver.

Related Errors

• Failed to initialize NVML: Driver/library version mismatch — the same root cause seen directly from nvidia-smi on the host.
• could not select device driver "" with capabilities: [[gpu]] — the nvidia runtime/hook is not registered with Docker at all.
• unknown or invalid runtime name: nvidia — daemon.json is missing the nvidia runtime entry.
• no CUDA-capable device is detected — the container started but sees no GPU, often a --gpus scoping issue.

Frequently Asked Questions

Why do I need to reboot after upgrading the driver? The upgrade installs new userspace libraries immediately but the old kernel module stays loaded until reboot. NVML refuses to work when they disagree, producing the version mismatch.

Do I need the toolkit if I already have the driver? Yes. The host driver lets the machine use the GPU; the NVIDIA Container Toolkit is what injects that driver into containers via the runtime hook. Both are required for --gpus all.

How do I avoid rebooting a production host? Cordon and drain GPU workloads, then unload and reload the nvidia modules with rmmod/modprobe. This only works if no process is holding the GPU open.

The host nvidia-smi works but the container still fails, why? The toolkit or Docker runtime registration is likely missing. Run nvidia-ctk runtime configure --runtime=docker and restart Docker.

Does this apply to Kubernetes GPU pods too? The same driver/toolkit relationship underlies the NVIDIA device plugin, so a host-level mismatch breaks pods identically. For more, see the Docker guides.