Docker for AI Workloads on Ubuntu 26.04

In Parts 3 and 4 you gave ai-node01 a working GPU — an NVIDIA driver with CUDA, or an AMD GPU with ROCm — and proved it with nvidia-smi or rocm-smi. That host can now do GPU compute. This lesson wraps that raw capability in containers so an AI workload becomes something you can ship, version, and rebuild identically anywhere — while still reaching the GPU underneath.

What You’ll Learn

• Why containers are the standard packaging for AI infrastructure, and the problems they solve (and the ones they don’t)
• How a container reaches the host GPU driver — the NVIDIA path and the AMD path — and why the container never replaces that driver
• How to install Docker Engine on Ubuntu 26.04 from Docker’s official repository
• How to validate the Docker daemon chain and understand the socket, the docker group, and why that group is root-equivalent
• How to install and verify the NVIDIA Container Toolkit, then run a GPU container and compare it to the host
• How AMD ROCm containers expose the GPU through kernel devices instead of a toolkit
• How to define an AI stack with Docker Compose — services, networks, volumes, GPU reservations, health checks, restart policies, and limits
• How to keep model data persistent so multi-gigabyte downloads survive a container recreate
• How to network AI services safely — private bridge networks, service-name DNS, and not exposing internals to the internet
• How to handle environment variables vs secrets, image tags, logging, and container security
• A layer-by-layer method for troubleshooting GPU containers when they fail

By the end you will run your first GPU-enabled AI container on ai-node01, with model storage that survives a rebuild — the foundation the local-LLM lesson builds on next.

Why Docker for AI Infrastructure

You already have a GPU that works on the host. Why add a container layer at all? Because “works on this host, once, the way I set it up” is not a deployment strategy. AI workloads drag in heavy, fast-moving, version-sensitive dependencies — a specific Python, a specific PyTorch build, a specific CUDA runtime — and reproducing that stack by hand on a second machine is where afternoons disappear.

Contrast the two ways of running the same workload:

  Traditional (on the host)
  --------------------------
  App + Python + PyTorch + libs
        installed directly on
        ai-node01's OS
   → tangled with system packages
   → "works on my node" only
   → hard to reproduce or roll back

  Containerized
  --------------------------
  ┌───────────────────────────┐
  │ Image: app+Python+PyTorch │
  │        +CUDA runtime      │
  └───────────────────────────┘
        runs on ANY host with
        Docker + a working GPU
        driver underneath

A container is a packaged, isolated process: your application plus its user-space dependencies, bundled into an image that runs the same way on any host with a container runtime. It is not a virtual machine — there is no second kernel; the container shares the host’s Linux kernel and just gets its own isolated view of the filesystem, processes, and network.

For AI infrastructure specifically, containers buy you:

• Reproducibility — the same image produces the same environment every time; no “it worked yesterday” drift.
• Isolation — one workload’s dependencies can’t collide with another’s on the same host.
• Consistency — dev, test, and prod run the identical image.
• Rollback — a bad release is one image tag away from the previous known-good one.
• Portability — the image moves between machines, on-prem or cloud, unchanged.
• Versioning — pinned image tags give you an auditable history of exactly what ran.
• CI/CD — images are the natural build artifact for automated pipelines.
• Scaling — many identical replicas of a stateless service are trivial to launch.
• A Kubernetes on-ramp — everything you learn here is the unit Kubernetes schedules later in this series.

Be equally clear about what Docker does not fix. Containers are a packaging and isolation tool, not a magic layer:

• It does not fix a bad or missing host GPU driver — the container depends on it (next section).
• It does not create VRAM you don’t have — a model too big for your GPU is still too big inside a container.
• It does not make an unsupported GPU work, or make an incompatible model run.
• It does not solve storage, networking, security, or monitoring for you — it just gives you clean seams to configure each. That configuration is the rest of this lesson.

🤖 AI Infrastructure Tip — Think of the image as a contract: “given a host with Docker and a working GPU driver, this exact software will run.” The container guarantees the software side of that contract. The GPU driver, the hardware, and the model’s resource appetite are still your job — which is exactly why Parts 3 and 4 came first.

Host vs Container GPU Responsibilities

The single most important idea in this lesson: the container does not contain a GPU driver. The driver lives on the host, inside the kernel, and the container reaches through to it. Get this wrong and every GPU-container problem looks like a mystery.

Read the full stack bottom-up:

   AI code / framework   (in container)
          │
   CUDA / ROCm user-space libs
          │  (in container)
   ─ ─ ─ ─ ─ container boundary ─ ─ ─ ─
   Container runtime bridge
          │
   Host GPU driver          (on host)
          │
   Linux kernel             (on host)
          │
        GPU                 (hardware)

Everything above the dashed line ships inside the image. Everything below it belongs to the host you built in Part 3 or Part 4. The image carries the framework and the CUDA (or ROCm) user-space libraries; the actual driver that commands the hardware is the host’s, and a bridge lets the container’s libraries call it.

The two GPU vendors bridge that gap differently:

  NVIDIA path
  -----------
  Container (CUDA libs)
      │
  NVIDIA Container Toolkit  ← the bridge
      │
  Host NVIDIA driver → kernel → GPU

  AMD path
  --------
  Container (ROCm libs)
      │
  /dev/kfd + /dev/dri devices ← the bridge
      │
  Host amdgpu driver → kernel → GPU

NVIDIA uses a purpose-built shim called the NVIDIA Container Toolkit, which injects the host driver’s libraries and device files into the container at start-up. AMD takes a more Linux-native route: you hand the container the kernel’s GPU device files directly and let ROCm inside the container talk to the host amdgpu driver through them. Same principle, two mechanisms — and in both, the host driver does the real work. If nvidia-smi (or rocm-smi) is broken on the host, no container will see the GPU. Fix the host first.

Installing Docker Engine on Ubuntu 26.04

Ubuntu ships its own docker.io package, and it works — but it trails upstream, and current Compose and GPU features arrive first in Docker’s own Engine. Install from Docker’s official apt repository. Start with the prerequisites and Docker’s signing key:

sudo apt-get update
sudo apt-get install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

This creates /etc/apt/keyrings, downloads Docker’s GPG key there, and makes it world-readable so apt can verify packages signed with it. Nothing is installed yet — you’ve just told the system to trust Docker’s repository.

Now add the repository itself:

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

The $(. /etc/os-release && echo "$VERSION_CODENAME") part reads your release codename automatically — you don’t type it — so the repo line matches the OS you’re on. The final apt-get update refreshes the package index to include Docker’s repository.

❗ Important — Docker’s apt repo may not list a brand-new Ubuntu codename the moment 26.04 ships. If that apt-get update reports it can’t find the 26.04 codename, do not invent one. Check the current, authoritative steps at docs.docker.com/engine/install/ubuntu/ — Docker sometimes has you temporarily use the previous LTS codename until the new one is published.

Install the Engine, CLI, containerd, Buildx, and the Compose plugin in one shot:

sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Then confirm both the Engine and Compose are present:

docker version
docker compose version

docker version prints client and server (daemon) versions — seeing both means the CLI can reach a running daemon. Note that modern Compose is the docker compose subcommand (v2), not the old standalone docker-compose binary. If docker compose version prints a version, you have the plugin and the whole Compose workflow later in this lesson will work.

The Docker Daemon Chain

When you type a docker command, five layers cooperate. Knowing them turns “Docker is broken” into a specific question about which layer:

  docker CLI
      │  (talks over the socket)
  dockerd (daemon)
      │
  containerd
      │
  runc (OCI runtime)
      │
  Linux kernel  → runs your container

The CLI is just a client; it sends your request over a socket to dockerd, the long-running daemon that actually manages images and containers. dockerd delegates container lifecycle to containerd, which uses runc to ask the kernel to create the isolated process. If the daemon isn’t running, the CLI has nothing to talk to — the classic “Cannot connect to the Docker daemon” error.

Verify the daemon is up and enabled:

systemctl status docker

Look for active (running) and, ideally, enabled (so it starts on boot). Example output:

● docker.service - Docker Application Container Engine
   Loaded: loaded (…; enabled; …)
   Active: active (running) since …

That active (running) line is the prerequisite for everything else in this lesson.

Docker Permissions and the Socket

By default, docker commands need sudo, because the daemon listens on the Unix socket /var/run/docker.sock, which is owned by root:docker. You can drop the sudo by adding your user to the docker group:

sudo usermod -aG docker $USER

Then log out and back in (group membership is applied at login) and confirm with docker run --rm hello-world — no sudo. Convenient, but understand exactly what you just granted.

⚠️ Warning — Membership in the docker group is root-equivalent access to the entire host. Anyone who can talk to the Docker socket can start a container that bind-mounts the host’s / and read or modify any file as root — no password required. Only add trusted users to the docker group, and treat it with the same care as sudo access. And never “fix” a permission error with chmod 777 /var/run/docker.sock — that exposes root-equivalent control to every user and process on the machine. If you hit a socket permission error, fix group membership, not the socket mode.

Your First Container (No GPU)

Before you blame the GPU for anything, prove Docker itself works with the smallest possible test:

docker run --rm hello-world

Docker pulls a tiny image, runs it, prints a “Hello from Docker!” confirmation, and — thanks to --rm — deletes the container when it exits. If you see that message, the whole daemon chain from the previous section is healthy. Run one more, slightly more realistic:

docker run --rm ubuntu:24.04 echo ok

This pulls a real Ubuntu image, runs echo ok inside it, prints ok, and cleans up. If both of these succeed, Docker is sound and any later GPU failure is specifically a GPU-access problem — not a Docker problem. That separation is the first move in every troubleshooting section below.

The NVIDIA Container Toolkit

For NVIDIA GPUs, the bridge between container and host driver is the NVIDIA Container Toolkit. It doesn’t install a driver in the image; at container start it injects the host driver’s libraries and device nodes so the CUDA libraries inside the container can reach the real GPU:

  NVIDIA GPU
      │
  Host NVIDIA driver   (from Part 3)
      │
  NVIDIA Container Toolkit
      │
  Docker daemon
      │
  GPU container (CUDA user-space libs)

Add the toolkit’s repository and key, then install it:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

The first curl imports NVIDIA’s signing key; the second downloads the repository list and rewrites it to reference that key; apt-get update then picks up the new repo, and the install pulls the toolkit. Now wire it into Docker and restart the daemon so the change takes effect:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

nvidia-ctk runtime configure --runtime=docker edits Docker’s configuration to register the NVIDIA runtime; the restart makes dockerd reload it. Until you restart the daemon, --gpus won’t work — a common “I installed it but it still fails” trap.

❗ Important — The host NVIDIA driver comes from Part 3 (Ubuntu’s ubuntu-drivers install). The toolkit does not install or replace it. If the host driver isn’t working, install it the Part 3 way first — see NVIDIA GPUs and CUDA on Ubuntu 26.04.

Running a GPU Container (NVIDIA)

The test that proves everything: run nvidia-smi inside a container and confirm it prints the same GPU table as the host.

docker run --rm --gpus all ubuntu:24.04 nvidia-smi

--gpus all tells Docker to expose every GPU to the container via the toolkit; --rm cleans up afterward. Success looks identical to running nvidia-smi on the host — same driver version, same GPU name, same VRAM total:

+-----------------------------------------------------+
| NVIDIA-SMI 5xx.xx   Driver: 5xx.xx  CUDA: 12.x     |
|   0  NVIDIA ...    |  40C  25W | 512/16384MiB   0%  |
+-----------------------------------------------------+

(example output). Run nvidia-smi on the host and compare: the two tables should match. If the container’s table matches the host’s, the full NVIDIA container path works.

You can also run a CUDA base image if you want the CUDA libraries present in the container:

docker run --rm --gpus all nvidia/cuda:12.6.2-base-ubuntu24.04 nvidia-smi

Treat that 12.6.2-base-ubuntu24.04 tag as an example — pick a current, appropriate tag from hub.docker.com/r/nvidia/cuda for your driver. To expose only one GPU instead of all of them, use --gpus '"device=0"' to select GPU index 0.

🛠️ DevOps Tip — The CUDA user-space libraries inside the container still depend on a compatible host driver, exactly as in Part 3. A CUDA image with a newer CUDA version than your host driver supports can still fail even though --gpus all is correct — because the constraint is the host driver’s maximum CUDA version, which the container can’t change. When a GPU container misbehaves, check the host driver’s supported CUDA first. The container ships the runtime libs; the host owns the driver.

AMD ROCm Containers

AMD does not use the NVIDIA Container Toolkit. Instead, you expose the kernel’s GPU device files directly to the container, and ROCm inside the container talks to the host amdgpu driver through them:

docker run --rm \
  --device /dev/kfd \
  --device /dev/dri \
  <rocm-enabled-image> rocm-smi

/dev/kfd is the ROCm compute device — the Kernel Fusion Driver node that GPU compute goes through. /dev/dri holds the direct-rendering GPU device nodes. Your user must be in the render and video groups (set up in Part 4) so those device files are accessible. If rocm-smi inside the container lists your GPU, the AMD container path works.

⚠️ Warning — Do not reach for --privileged as the normal way to run AMD GPU containers. --privileged disables most of Docker’s isolation and hands the container broad host access — use it only as a temporary diagnostic if you suspect a device-permission problem, never as your standing configuration. Passing the two specific --device flags is the correct, least-privilege path. And never chmod 777 device files to “make it work.”

❗ Important — Carry forward Part 4’s honesty: current ROCm officially lists Ubuntu 24.04 and 22.04 on its supported-OS matrix, and 26.04 may not be there yet. Before committing to an AMD container workload on 26.04, verify both your GPU and your OS against the compatibility matrix at rocm.docs.amd.com/en/latest/compatibility/compatibility-matrix.html. See AMD ROCm AI Infrastructure on Ubuntu 26.04 for the full support picture.

NVIDIA vs AMD Container Stack

Both vendors run on the same Docker Engine and the same PyTorch on top. They differ only in the driver, the compute platform, and how the GPU is bridged into the container:

Layer	NVIDIA	AMD
Host driver	NVIDIA driver	amdgpu
Compute platform	CUDA	ROCm
Container bridge	NVIDIA Container Toolkit	Kernel devices /dev/kfd, /dev/dri
Run flag	--gpus all	--device /dev/kfd --device /dev/dri
Container engine	Docker	Docker
Framework on top	PyTorch	PyTorch

The takeaway: once the GPU is bridged in, everything above — Docker, Compose, PyTorch, your model server — is the same skill regardless of vendor. Only the bottom two rows change.

Docker Compose for AI

Running containers with long docker run lines is fine for a one-off test, but an AI deployment is usually several cooperating services — a model server, maybe a web UI, maybe a database — that need consistent networks, volumes, and start-up order. Docker Compose describes that whole stack declaratively in one YAML file, so docker compose up brings it up identically every time.

Learn the vocabulary before you meet a big file:

• services — each container in your stack (e.g. llm, webui), defined once.
• image — which image a service runs; pin a specific tag for reproducibility.
• networks — private networks that let services talk to each other by name.
• volumes — named, persistent storage that outlives a container.
• ports — which container ports are published to the host (host:container).
• environment — configuration passed in as environment variables.
• healthcheck — a command Compose runs to decide if a service is actually ready, not just running.
• depends_on — declares that one service starts after another (optionally, after it’s healthy).
• restart — the restart policy when a container exits.
• deploy.resources — CPU/memory limits and, for AI, the GPU reservation.

One modern-Compose note that saves confusion: do not add a top-level version: key. It’s obsolete — current Compose ignores or deprecates it. Start your file straight at services:.

Giving Compose Access to the GPU

In a plain docker run you write --gpus all. In Compose, the equivalent is a device reservation under deploy.resources:

services:
  llm:
    image: ollama/ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1            # or: count: all
              capabilities: [gpu] # MANDATORY — omit and deployment errors

Read the reservation carefully. driver: nvidia selects the NVIDIA runtime; count: 1 reserves one GPU (use count: all for every GPU); and capabilities: [gpu] is required — leave it out and Compose refuses to start the service. Note that count and device_ids: ['0'] are mutually exclusive: use count for “how many,” or device_ids to name specific GPUs, but not both.

AMD is different here too — there is no driver: nvidia reservation. You map the kernel devices into the service instead (Compose supports a devices: list on a service mapping /dev/kfd and /dev/dri), mirroring the --device flags from the AMD section above. The principle is the same; the syntax follows each vendor’s bridge.

Persistent Model Storage

Containers are disposable — recreate one and its writable layer is gone. AI models are multi-gigabyte downloads you never want to fetch twice. So model data must live outside the container, in a named volume or a bind mount. Adopt a predictable layout on the host:

  /srv/ai/
  ├── models/
  ├── data/
  ├── configs/
  └── logs/

Create it with ownership you control, so containers (and you) can write without fighting permissions:

sudo mkdir -p /srv/ai/models
sudo chown "$USER" /srv/ai/models

mkdir -p creates the tree; chown "$USER" hands you ownership so a bind mount into this path is writable. Prefer NVMe storage here, and watch your root filesystem — a few large models can fill / fast if models land on the boot disk by accident.

You have two ways to persist data, and the difference matters:

• Named volume — Docker manages the storage (e.g. a volume called ollama). Portable, easy to back up as a unit, and the default choice for model caches.
• Bind mount — you map a specific host path (like /srv/ai/models) into the container. You control exactly where the bytes live and can inspect them directly on the host — handy when you want models on a particular NVMe drive.

The lifecycle test that proves it works: download a model into the volume, then recreate the container. Because the model lives in the volume, not the container’s writable layer, the new container finds the model already present — no re-download.

  1. run container  → volume attached
  2. download model → lands in volume
  3. docker rm container
  4. run again       → same volume
  5. model is STILL there  ✓

That “still there” at step 5 is the entire point of persistent storage. If a model re-downloads on recreate, your data isn’t actually in the volume — recheck the mount path.

AI Container Networking

Compose puts your services on a private bridge network and gives each one a DNS name equal to its service name. So a web UI can reach the model server at http://llm:11434 without any IP addresses or host ports — the traffic never leaves Docker’s internal network.

  Browser
     │  http://host:3000
     ▼
  Host published port 3000
     │
  Web UI container
     │  http://llm:11434  (private net)
     ▼
  LLM container  (port 11434)

Two concepts to hold apart:

• Service-name DNS (internal): llm, webui — names that resolve only inside the Compose network. This is how services find each other. No host port needed.
• Published ports (host_port:container_port): the bridge from the outside world into a container. 3000:8080 means “host port 3000 forwards to container port 8080.” Only publish what genuinely needs external access.

And a security decision baked into every published port — what address it binds to:

• 127.0.0.1:3000:8080 binds to localhost only. Reachable from the host itself (or an SSH tunnel), not from the network. Use this for anything not meant for the public.
• 0.0.0.0:3000:8080 (the default for 3000:8080) binds to all interfaces — anyone who can route to the host can reach it.

⚠️ Warning — Do not publish internal AI services (a model API, a database) to 0.0.0.0 on an internet-facing host. A model server with no authentication, exposed on a public IP, is an open door — anyone can drive your GPU or exfiltrate data. Keep the LLM backend on the private Compose network with no host port at all; publish only the UI, and prefer 127.0.0.1 with an SSH tunnel for admin access. Networking fundamentals carry over from the Kali Linux networking lessons if you want to go deeper.

Environment Variables and Secrets

Containers are configured largely through environment variables — a service reads OLLAMA_BASE_URL or a log level from its environment, so the same image behaves differently per deployment without a rebuild. In Compose you set them under environment: or load them from a .env file.

That flexibility tempts people into a serious mistake: environment variables are not secret storage. Anyone who can run docker inspect on the container, or read your Compose file, sees them in plain text.

⚠️ Warning — Never commit credentials — API keys, tokens, database passwords — into a Git-tracked Compose file or a checked-in .env. Keep secret-bearing .env files out of version control (add them to .gitignore) and lock their permissions down (chmod 600 .env). For anything genuinely sensitive, use a secret file referenced by the container rather than an inline environment value, so the secret lives in a protected file the container reads — not in the image, not in the command line, not in your commit history. A leaked API key in a public repo is a same-day incident.

Health Checks: Running Is Not Ready

A container can be running while the AI service inside it is nowhere near able to serve a request — it may still be loading a multi-gigabyte model or initializing the GPU. “Running” and “ready” are different states, and conflating them is why dependent services fail at start-up.

The readiness ladder for an AI service climbs through several rungs:

  process started       (docker ps: Up)
      │
  port is listening
      │
  API responds
      │
  model is loaded
      │
  inference works       ← actually ready

Only the last rung means the service can do its job. A Compose healthcheck: lets you test an actual HTTP request against the API, so Docker reports the container as healthy only once it truly responds — not merely Up. Then depends_on with condition: service_healthy gates a dependent service until the one it needs is genuinely ready.

🤖 AI Infrastructure Tip — depends_on alone only waits for the container to start, not to become ready — and AI containers are unusually slow to become ready because of model load, download, and GPU initialization. Without a health check, a UI can come up and start hammering an LLM backend that’s still loading its model, producing a wave of confusing connection errors. Pair depends_on with condition: service_healthy and a real health check on the backend.

Restart Policies

When a container exits, Docker’s restart policy decides what happens next. For a long-running AI service, restart: unless-stopped is the sensible default — it restarts the container on crash or reboot, but respects a deliberate docker stop. Compare the options:

• no — never restart automatically (the default).
• on-failure — restart only on a non-zero exit code, optionally capped at N tries.
• always — restart no matter what, even after you manually stop it (it comes back on daemon restart).
• unless-stopped — restart automatically, unless you stopped it on purpose. The pragmatic choice for AI services.

⚠️ Warning — A restart policy can hide a real failure. A container stuck in a crash-restart loop looks “sort of alive” in docker ps while it fails identically every few seconds. Don’t let auto-restart substitute for diagnosis — when a container keeps restarting, read docker logs and fix the root cause instead of trusting the policy to paper over it.

Resource Limits

Docker can cap a container’s CPU and RAM through deploy.resources.limits (or mem_limit / cpus). That protects the host: one runaway container can’t starve the others of system memory. Useful and worth setting.

But there’s a hard boundary you must not blur:

❗ Important — Limiting a container’s RAM does not limit its GPU VRAM. System memory is governed by Linux cgroups, which Docker controls; GPU VRAM is governed by the model and the GPU runtime, which Docker’s memory limits don’t touch. If a model needs more VRAM than your GPU has, no mem_limit will save you — the fix is a smaller or more-quantized model, or a bigger GPU (the local-LLM lesson covers this). And do not go looking for a Docker flag that “partitions” or “shares” VRAM the way mem_limit shares RAM — don’t invent one. GPU memory management lives in the runtime, not in Docker cgroups.

Logging

When a container misbehaves, its logs are the first place to look — the daemon captures whatever the container writes to stdout/stderr:

docker logs <container-name>

For a whole Compose stack, follow every service’s logs live:

docker compose logs -f

-f streams new lines as they arrive — invaluable while watching a model load or reproducing a failure. Reading logs before restarting is the discipline that turns “it keeps crashing” into a specific, fixable cause. Centralized, searchable logging across many nodes — shipping these logs somewhere you can query them — is a monitoring concern we build out in Part 8 (coming soon); the observability stack previews where that goes.

Image Tags

An image reference like ollama/ollama:latest has two parts: the name and the tag. The tag decides exactly which build you run — and latest is not a fixed version. It’s a moving pointer that can change under you between pulls, so two hosts pulling latest a week apart may run different software.

🛠️ DevOps Tip — Pin explicit version tags for anything you depend on. latest is convenient for a throwaway test, but risky in production: an upstream release can silently change behavior, break compatibility, or alter GPU requirements the next time you pull. Pinning a specific tag makes deployments reproducible and rollbacks trivial — you always know precisely what ran, and you can go back to the previous tag in seconds. Pin the exact tag in your Compose file; treat latest as “unknown version.”

AI Container Security

Everything so far has security implications; here is the consolidated checklist for AI workloads specifically:

• No --privileged. It disables isolation and is almost never necessary. Grant specific --device or capabilities instead.
• Run as non-root where the image supports it. A process that doesn’t need root inside the container shouldn’t have it.
• Publish the minimum ports, bound to 127.0.0.1 unless a service genuinely must be reachable from the network. Keep model backends off host ports entirely.
• Mount read-only where you can. If a container only needs to read models, mount that volume read-only so a compromised process can’t tamper with them.
• Use updated, trusted, pinned images from sources you trust, on specific tags — not random latest images of unknown provenance.
• Protect the Docker socket. Recall it’s root-equivalent: guard docker group membership, and never mount the socket into a container you don’t fully trust.
• Isolate services on private networks. Only the front door (a UI) should be reachable; the LLM and any datastore stay internal.

🤖 AI Infrastructure Tip — The highest-value AI-container security move is also the simplest: keep the model server on a private network with no published port, and expose only an authenticated front end. Most “someone is using my GPU” incidents trace back to an unauthenticated model API bound to 0.0.0.0 on a public host. Isolation first, then the rest of the checklist.

Troubleshooting GPU Containers

Every GPU-container failure is a broken layer. Diagnose top-down through the same stack every time, and you’ll isolate the fault fast instead of guessing:

  Hardware present?        (lspci)
      │
  Host driver works?       (nvidia-smi / rocm-smi on host)
      │
  Container runtime OK?    (toolkit configured, daemon restarted)
      │
  Container starts?        (docker ps / docker logs)
      │
  AI runtime ready?        (health check / API responds)
      │
  Model loaded?            (inference works)

Work the common failures with Problem → Likely Cause → Check → Fix → Validate.

🔍 Troubleshooting — Cannot connect to the Docker daemon. Problem: any docker command errors with “Cannot connect to the Docker daemon.” Likely cause: dockerd isn’t running. Check: systemctl status docker. Fix: sudo systemctl start docker (and sudo systemctl enable docker so it starts on boot); read the status output if it won’t start. Validate: docker run --rm hello-world succeeds.

🔍 Troubleshooting — Permission denied on the socket. Problem: docker (without sudo) reports “permission denied” on /var/run/docker.sock. Likely cause: your user isn’t in the docker group. Check: groups — is docker listed? Fix: sudo usermod -aG docker $USER, then log out and back in. Do not chmod 777 the socket. Validate: docker run --rm hello-world runs without sudo.

🔍 Troubleshooting — Container has no GPU. Problem: a container that should use the GPU sees none. Likely cause: you launched it without --gpus all (NVIDIA) or without the --device flags (AMD), or the Compose reservation is missing. Check: re-read the run command / Compose deploy.resources.reservations.devices block. Fix: add --gpus all (NVIDIA) or --device /dev/kfd --device /dev/dri (AMD); in Compose, ensure capabilities: [gpu] is present. Validate: docker run --rm --gpus all ubuntu:24.04 nvidia-smi shows the GPU.

🔍 Troubleshooting — Host GPU works but container doesn’t. Problem: nvidia-smi works on the host, but fails inside any container. Likely cause: the NVIDIA Container Toolkit isn’t configured into Docker, or the daemon wasn’t restarted after configuring it. Check: did you run sudo nvidia-ctk runtime configure --runtime=docker and then sudo systemctl restart docker? Fix: run both, in that order. Validate: the container nvidia-smi table matches the host’s.

🔍 Troubleshooting — nvidia-smi fails inside the container. Problem: the container starts but nvidia-smi inside it errors. Likely cause: the host driver is broken, or a CUDA-image version exceeds the host driver’s supported CUDA. Check: does nvidia-smi work on the host? What CUDA version does it support? Fix: repair the host driver first (Part 3); pick a CUDA image tag within the host driver’s supported CUDA. Validate: host and container nvidia-smi agree.

🔍 Troubleshooting — AMD ROCm container can’t see the GPU. Problem: rocm-smi inside the container shows no GPU. Likely cause: the kernel devices weren’t mapped in, or your user isn’t in render/video, or 26.04 isn’t on ROCm’s supported-OS matrix yet. Check: are --device /dev/kfd --device /dev/dri present? groups shows render and video? Is your GPU + OS on the compatibility matrix? Fix: add both --device flags, add the groups (Part 4), and verify OS/GPU support before assuming a bug. Validate: rocm-smi inside the container lists the GPU.

🔍 Troubleshooting — Container exits immediately. Problem: the container starts and stops within seconds. Likely cause: the main process errored on start-up (bad config, missing file, wrong command). Check: docker logs <name> — read the last lines before it died. Fix: correct whatever the logs report (config, mount path, command). Validate: docker ps shows the container staying Up.

🔍 Troubleshooting — Out of memory (system RAM). Problem: the container is killed with an OOM error. Likely cause: the workload exceeded the host’s RAM (or the container’s mem_limit). Check: free -h on the host; docker logs for an OOM-kill message. Fix: raise the limit if the host has headroom, or reduce the workload’s memory footprint. Validate: the container runs to completion without an OOM kill.

🔍 Troubleshooting — Out of VRAM. Problem: the workload fails with a GPU/CUDA out-of-memory error. Likely cause: the model + its runtime needs more VRAM than the GPU has — a container RAM limit does not affect this. Check: nvidia-smi used vs total VRAM during the run. Fix: use a smaller or more-quantized model, reduce context/batch, or move to a bigger-VRAM GPU. Don’t look for a Docker VRAM flag — there isn’t one. Validate: used VRAM stays below total through the whole run.

🔍 Troubleshooting — Model data lost on recreate. Problem: recreating the container re-downloads the model. Likely cause: the model was written to the container’s writable layer, not a volume/bind mount. Check: confirm the volume or bind mount is attached at the model path (docker inspect, or list /srv/ai/models). Fix: mount a named volume or /srv/ai/models at the model directory the runtime uses. Validate: recreate the container; the model is still present (no re-download).

🔍 Troubleshooting — Port already in use. Problem: the container won’t start — “port is already allocated.” Likely cause: another process (often a previous container) already holds that host port. Check: ss -tulpn | grep <port> to see who owns it. Fix: stop the other process/container, or publish a different host port. Validate: the container starts and the port responds.

🔍 Troubleshooting — API works in the container but not from the host. Problem: the API responds inside the container but is unreachable from the host. Likely cause: the port isn’t published, or the service binds to 127.0.0.1 inside the container so only the container’s own loopback sees it. Check: the ports: mapping; whether the service binds to 0.0.0.0 inside the container. Fix: publish the port (host:container) and ensure the service listens on 0.0.0.0 within the container. Validate: curl from the host reaches the API on the published port.

🔍 Troubleshooting — Dependent service starts before the AI API is ready. Problem: a UI or client errors at start-up because the model backend isn’t ready yet. Likely cause: depends_on waited only for start, not readiness, and the model is still loading. Check: does the backend have a healthcheck? Is depends_on using condition: service_healthy? Fix: add a real health check to the backend and gate the dependent with condition: service_healthy. Validate: the dependent starts cleanly only after the backend reports healthy.

🔍 Troubleshooting — Container keeps restarting. Problem: a container cycles endlessly in a restart loop. Likely cause: it crashes on start-up and the restart policy relaunches it into the same failure. Check: docker logs <name> — the same error repeats each cycle. Fix: fix the root cause from the logs; a restart policy is not a fix. Validate: the container reaches a stable Up (and healthy) state and stops cycling.

Hands-On Lab: Run Your First GPU AI Container on ai-node01

🧪 Hands-On Lab — Take the GPU host you built in Part 3 (or Part 4) and turn it into a container host that runs a real GPU workload with persistent model storage. Do these in order on ai-node01. AMD readers substitute the AMD device flags where noted.

1. Confirm the host GPU works. Run nvidia-smi (or rocm-smi) on the host. If it fails, stop and fix the host driver (Part 3 / Part 4) before touching Docker.
2. Install Docker Engine. Add Docker’s apt repo and key, then install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin (commands above).
3. Verify Docker. docker version and docker compose version — confirm client, server, and Compose plugin.
4. Check the daemon. systemctl status docker — expect active (running).
5. Fix permissions. sudo usermod -aG docker $USER, then log out and back in.
6. Prove Docker works (no GPU). docker run --rm hello-world, then docker run --rm ubuntu:24.04 echo ok.
7. Install the NVIDIA Container Toolkit (NVIDIA hosts) — add its repo/key and sudo apt-get install -y nvidia-container-toolkit. AMD hosts skip this.
8. Wire it into Docker. sudo nvidia-ctk runtime configure --runtime=docker, then sudo systemctl restart docker. (AMD: nothing to configure.)
9. Run a GPU container test. NVIDIA: docker run --rm --gpus all ubuntu:24.04 nvidia-smi. AMD: docker run --rm --device /dev/kfd --device /dev/dri <rocm-image> rocm-smi.
10. Compare host vs container. The container’s GPU table must match the host’s.
11. Create model storage. sudo mkdir -p /srv/ai/models && sudo chown "$USER" /srv/ai/models.
12. Launch an AI service with a persistent volume. Start a model-server container (the next lesson uses Ollama) with a named volume or /srv/ai/models mounted for models, and its port published to 127.0.0.1.
13. Download a model into the volume. Pull a small, hardware-appropriate model (the local-LLM lesson covers picking one).
14. Test the port binding. From the host, curl the service’s local port; confirm it responds.
15. Prove persistence. docker rm -f the container, recreate it against the same volume, and confirm the model is still there — no re-download.
16. Read the logs. docker logs <name> — get familiar with what a healthy start-up looks like.
17. Document the node. Record Docker version, toolkit version, image tags, and the volume path so you can reproduce this host later.

  ┌──────────────────────────────────────┐
  │  SUCCESS: ai-node01 runs GPU containers │
  │                                        │
  │  Docker Engine ...... installed ✓      │
  │  Daemon ............. running   ✓      │
  │  GPU in container ... matches host ✓   │
  │  Model volume ....... /srv/ai   ✓      │
  │  Model persists ..... on recreate ✓    │
  │  Port bound local ... 127.0.0.1 ✓      │
  └──────────────────────────────────────┘

What You Learned

• Why containers are the standard for AI infrastructure — reproducibility, isolation, rollback, portability, and a Kubernetes on-ramp — and the limits Docker does not remove (bad drivers, insufficient VRAM, model compatibility, storage/network/security/monitoring).
• That the container carries the framework and CUDA/ROCm user-space libraries, but the host owns the driver — NVIDIA bridges via the Container Toolkit, AMD via /dev/kfd and /dev/dri.
• How to install Docker Engine from Docker’s official apt repo on Ubuntu 26.04, validate the CLI→dockerd→containerd→runc→kernel chain, and treat the docker group as root-equivalent.
• How to install and configure the NVIDIA Container Toolkit, run --gpus all containers, and compare host vs container nvidia-smi.
• How to describe an AI stack in Compose — services, private networks, volumes, the GPU reservations block, health checks, restart policies, and limits — without an obsolete version: key.
• How to keep model data in a named volume or bind mount under /srv/ai so it survives a recreate, and how to publish ports safely with 127.0.0.1 and private networks.
• Operational discipline: pin image tags, keep secrets out of Git and out of environment values, read logs before restarting, and never let a restart policy hide a failure.
• A top-down troubleshooting method — Hardware → Host Driver → Container Runtime → Container → AI Runtime → Model — applied to every common GPU-container failure.

  ai-node01 build-along
  ------------------------------------
  Ubuntu ..................... [done]
  Networking ................. [done]
  Storage .................... [done]
  GPU + Driver (P3/P4) ....... [done]
  Docker ..................... [done]
  GPU Containers ............. [done]
  Persistent Storage ......... [done]
  ------------------------------------
  Local LLM .................. [next]
  Kubernetes ................. [upcoming]
  Monitoring ................. [upcoming]

Recommended Hardware for a Docker AI Lab

Containers make the software reproducible, but they run on real hardware — and Docker removes none of the physical constraints from Part 3. The GPUs below (rendered as cards beneath this lesson, grouped from a learning card to a dedicated AI development system) are the same tiers you’d choose for a bare-metal node, because containers don’t add VRAM, power, or PCIe lanes.

Before buying, cross-check each candidate against your actual machine and workload:

• VRAM — the container can’t conjure GPU memory; the model still has to fit.
• Power and connectors — enough PSU headroom and the exact connectors the card needs.
• Cooling and airflow — sustained container inference runs the GPU hard; the case must move the heat.
• PCIe slot — an available slot of the right generation, with physical clearance.
• Current Linux driver support — confirm the model is supported on Ubuntu 26.04 today.
• Your model’s requirements — match VRAM and compute to what you’ll actually run, not to the biggest card on the shelf.

See the recommended GPUs below, and treat them as starting points, not a shopping list.

Next Lesson

Running Local LLMs on Ubuntu 26.04 — put this container host to work. You’ll run a real local large language model in a GPU container, persist its model data in the volume you just created, call its API, watch VRAM during inference, and learn the storage, networking, and health patterns that make a model server production-worthy. Continue to Part 6 →

To reinforce the container fundamentals underneath all of this, the Docker Academy covers images, volumes, and networking in depth, and the Docker error guides are a fast reference when a container misbehaves. If you skipped ahead, revisit NVIDIA GPUs and CUDA or AMD ROCm to make sure the host GPU works before you containerize it, and the series index shows where Kubernetes and monitoring fit next.