🚨 Docker Error Guide: 'could not select device driver with capabilities [[gpu]]' — Enable NVIDIA GPUs

Overview

This error appears when you ask Docker to expose GPUs to a container — with --gpus all or a Compose deviceRequests — but the daemon has no driver registered that can satisfy a GPU capability request:

docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

The empty "" driver name is the tell: Docker looked for a device driver that advertises the gpu capability and found none registered. This is almost always because the NVIDIA Container Toolkit is not installed, or is installed but not wired into the Docker daemon. The host GPU and NVIDIA driver may be perfectly healthy — what is missing is the shim that lets Docker inject the GPU into containers.

Symptoms

• docker run --gpus all ... fails immediately with the message above; the container never starts.
• The same command works on a colleague’s machine or a cloud GPU image but not on a freshly provisioned host.
• nvidia-smi works on the host but fails inside any container.
• Compose services with a gpu capability under deploy.resources.reservations.devices fail to create.
• Occurs right after a fresh Docker install, a Docker upgrade that reset daemon.json, or a driver reinstall.

Common Root Causes

• The NVIDIA Container Toolkit is not installed — Docker itself has no built-in GPU support; the toolkit provides the runtime that does.
• The toolkit is installed but not registered with Docker — nvidia-ctk runtime configure was never run, so daemon.json has no nvidia runtime and no GPU device driver.
• The daemon was not restarted after registering the runtime, so the new configuration is not loaded.
• The host NVIDIA driver is missing or broken — nvidia-smi fails on the host too, so there is no GPU for the toolkit to expose.
• A version mismatch between the kernel driver, the toolkit, and Docker after a partial upgrade.

Diagnostic Workflow

First confirm the host itself can see the GPU — if this fails, fix the driver before touching Docker:

nvidia-smi
lspci | grep -i nvidia

Check whether the NVIDIA Container Toolkit and its runtime are present and known to Docker:

nvidia-ctk --version
docker info | grep -i runtime
which nvidia-container-runtime

If docker info does not list an nvidia runtime, that is the root cause. Inspect the daemon config that should register it:

cat /etc/docker/daemon.json
journalctl -u docker --since '15 min ago' | grep -i 'runtime\|gpu\|nvidia'

Register the runtime with the toolkit’s helper, which writes the correct daemon.json stanza:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

After registration, daemon.json should contain the nvidia runtime — verify it looks like this:

{
  "runtimes": {
    "nvidia": {
      "path": "nvidia-container-runtime",
      "args": []
    }
  }
}

Finally, confirm the GPU is now visible inside a container:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Example Root Cause Analysis

An ML engineer provisioned a new g5 GPU instance, installed Docker from the convenience script, and immediately hit could not select device driver "" with capabilities: [[gpu]] when running their training image with --gpus all.

nvidia-smi on the host printed the full GPU table — so the kernel driver was fine and the hardware was visible. But docker info | grep -i runtime listed only runc; there was no nvidia runtime. cat /etc/docker/daemon.json showed the file did not even exist. The image and command were correct; Docker simply had no driver capable of satisfying [[gpu]] because the NVIDIA Container Toolkit had never been installed on this fresh host.

The fix was to install the toolkit and register it:

# install the toolkit package (repo setup omitted), then:
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

docker info then listed both runc and nvidia runtimes, and nvidia-smi succeeded inside the container. The root cause was a missing container toolkit on a fresh host — confirmed by a working host nvidia-smi combined with an empty runtime list in docker info.

Prevention Best Practices

• Bake the NVIDIA Container Toolkit and nvidia-ctk runtime configure into your GPU host image / provisioning automation so every node has it from first boot.
• Re-run nvidia-ctk runtime configure --runtime=docker after Docker upgrades that may reset daemon.json, and always restart the daemon.
• Pin and track compatible versions of the kernel driver, the container toolkit, and Docker together; test after any of them changes.
• Add a smoke test to provisioning that runs docker run --rm --gpus all nvidia/cuda:...-base nvidia-smi and fails the build if it does not print the GPU table.
• Keep daemon.json under configuration management so the runtimes block cannot silently drift or be overwritten.

Quick Command Reference

# Does the host see the GPU?
nvidia-smi
lspci | grep -i nvidia

# Is the toolkit installed and registered with Docker?
nvidia-ctk --version
docker info | grep -i runtime

# Register the NVIDIA runtime and restart
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Verify GPU visibility inside a container
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

# Inspect daemon config / logs
cat /etc/docker/daemon.json
journalctl -u docker --since '15 min ago' | grep -i nvidia

Conclusion

could not select device driver "" with capabilities: [[gpu]] means Docker has no GPU-capable device driver registered — not that your GPU is broken. The near-universal cause is a missing or unregistered NVIDIA Container Toolkit. Confirm the host GPU with nvidia-smi, check docker info for an nvidia runtime, then run nvidia-ctk runtime configure --runtime=docker and restart the daemon. Bake that into your GPU host provisioning and add a --gpus all nvidia-smi smoke test so a fresh node never surprises you with this error again.