Kubernetes for AI Workloads on Ubuntu 26.04

In Part 6 you had a single node, ai-node01, doing real work: Ubuntu 26.04 with a GPU driver, CUDA or ROCm, Docker, the NVIDIA Container Toolkit, and a local LLM served through an API. That is a capable machine — and a single point of failure with no scheduler. This lesson turns that one node into a GPU-aware Kubernetes cluster so you can schedule and operate AI workloads the way real platforms do: with restarts, placement rules, persistent storage, and service discovery handled for you.

What You’ll Learn

Why you reach for Kubernetes once a single Docker host stops being enough — and why Kubernetes is an orchestrator, not an AI framework
The architecture of a GPU-aware AI cluster: user → Service → Pod → runtime → GPU → node, and how a control plane and worker nodes fit together
The Kubernetes components a DevOps engineer actually needs to reason about, and how kubelet talks to containerd instead of the Docker daemon
How to install Kubernetes on Ubuntu 26.04 from the community pkgs.k8s.io repository, and why versions are held
How to initialize a control plane with kubeadm, install a CNI, and join a GPU worker
How Kubernetes learns about a GPU through a device plugin / GPU Operator, and why the scheduler ignores accelerators until then
The NVIDIA path (nvidia.com/gpu via the GPU Operator) and the AMD path (amd.com/gpu via the ROCm device plugin)
How to schedule onto the right hardware with labels, nodeSelector, affinity, and taints in a heterogeneous cluster
How to deploy the Part-6 LLM as a Deployment with a ConfigMap, a Secret, a PersistentVolumeClaim, health probes, and a ClusterIP Service
A layer-by-layer method for troubleshooting GPU scheduling, from NotReady nodes to Pending GPU Pods

By the end you will run the Ollama model server on Kubernetes, scheduled onto a GPU node, with its model data surviving Pod recreation — the platform foundation the monitoring lesson builds on next.

Why Kubernetes for AI Infrastructure

You finished Part 6 with Docker Compose bringing up a model server on one host. Compose is excellent for exactly that: a small stack on a machine you manage by hand. The trouble starts the moment you care about what happens when that machine — or that container — misbehaves.

Look honestly at the single-node picture you built:

  ai-node01 (Part 6)
  ------------------------------
  Docker + Compose
     └── ollama container
           └── GPU (whole card)
  ------------------------------
  If the container dies    → manual restart
  If the host reboots      → manual bring-up
  If you need a 2nd copy   → copy files by hand
  If the GPU is busy       → no queue, no placement
  Config change            → edit + recreate
  New version              → hope the swap is clean

Every line below the divider is a question Compose answers with “you, at 2 a.m.” An orchestrator answers them declaratively. The specific problems that push AI infrastructure onto Kubernetes are:

Failure recovery — a crashed Pod is restarted automatically; a dead node’s work is rescheduled elsewhere.
Scaling — you declare “I want N replicas” and the system keeps that many running, spread across nodes.
Scheduling — you say “this Pod needs a GPU” and the scheduler finds a node that has one free, instead of you picking hosts by hand.
Restart and self-healing — health probes decide when a container is really broken and needs replacing.
Service discovery — a stable name (ollama:11434) always points at the current Pods, even as they come and go.
Configuration management — config and secrets live as first-class objects, decoupled from the image.
Rollouts — a new image version rolls out gradually and can roll back if it fails.
GPU allocation — the cluster tracks which GPUs are in use and won’t schedule two whole-GPU Pods onto one card.

🤖 AI Infrastructure Tip — Be clear about what Kubernetes is not. It does not train models, serve inference, or make a GPU faster. It is a scheduler and lifecycle manager for containers — the same containers you built in Part 5. Your model server, your CUDA image, your Ollama runtime are unchanged; Kubernetes just decides where they run, how many, and what happens when they fail. If your workload doesn’t run correctly in Docker on one host, it won’t run correctly in Kubernetes either. Fix it in Compose first.

AI Cluster Architecture

A request to your model server travels a specific path once Kubernetes is in front of it. Read it top to bottom:

  User / client
      │  http://ollama:11434
      ▼
  Service (stable name + virtual IP)
      │  load-balances to healthy Pods
      ▼
  Pod  (one running instance)
      │
  Container runtime (containerd)
      │
  GPU  (nvidia.com/gpu reserved)
      │
  Node  (ai-node01, a GPU worker)

The Service is a stable front door; the Pod is a disposable running instance; the runtime and GPU are the same stack from Parts 5 and 6, now scheduled onto a specific node. A Pod is Kubernetes’ smallest unit — one or more containers that always run together on one node, sharing a network identity.

Above that sits the cluster topology. Kubernetes splits responsibilities between a control plane (the brain that makes decisions) and worker nodes (the muscle that runs your Pods):

  ┌─────────────────────────────┐
  │  ai-control01  (control plane) │
  │  decides WHAT runs WHERE      │
  └───────────────┬─────────────┘
                  │  schedules Pods onto
        ┌─────────┴─────────┐
        ▼                   ▼
  ┌───────────┐       ┌───────────┐
  │ ai-node01 │       │ ai-node02 │
  │ GPU worker│       │  (optional)│
  └───────────┘       └───────────┘

For this lab you need two machines: ai-control01 (the control plane) and ai-node01 (your Part-6 GPU worker). A second worker, ai-node02, is optional and lets you practice placement across nodes.

❗ Important — Real AI clusters are heterogeneous: an NVIDIA node here, an AMD node there, a CPU-only node for the web front end. Kubernetes will happily schedule a Pod onto a node whose GPU can’t run its image — a CUDA image lands on ROCm hardware and crashes. Workload compatibility is your responsibility, enforced with labels and affinity (covered below). The scheduler places by resources and rules, not by whether the software will actually work.

Kubernetes Components for AI Engineers

You do not need to become a Kubernetes internals expert, but you do need a mental model of which component to blame when something breaks. Here are the pieces and how they connect to scheduling an AI Pod:

  CONTROL PLANE (ai-control01)
  ----------------------------
  kube-apiserver   ← you talk to this (kubectl)
  etcd             ← cluster state database
  kube-scheduler   ← picks a node for each Pod
  controller-mgr   ← drives actual → desired state

  WORKER (ai-node01)
  ----------------------------
  kubelet          ← runs Pods the node is given
  containerd       ← the container runtime (CRI)
  CNI plugin       ← Pod networking

kube-apiserver — the single front door. Every kubectl command and every component talks to it. If the apiserver is down, the cluster is unmanageable.
etcd — the key-value store holding all cluster state. Back it up; losing it loses the cluster.
kube-scheduler — watches for unscheduled Pods and assigns each to a node that satisfies its resource requests (including nvidia.com/gpu) and placement rules.
kube-controller-manager — the reconcile loop: if you asked for 3 replicas and 2 exist, it creates one more.
kubelet — the per-node agent. It receives Pod specs from the apiserver and makes the runtime run them.
containerd — the actual container runtime that pulls images and starts containers (next section).
CNI plugin — wires up Pod-to-Pod networking so a Service can reach a Pod on any node.

The AI scheduling flow reads across these: you kubectl apply a Deployment → the apiserver stores it → the controller-manager creates Pods → the scheduler finds a GPU node → the kubelet on that node tells containerd to start the container → the CNI gives it network. When a GPU Pod won’t start, that chain is your fault tree.

containerd, Not the Docker Daemon

In Part 5 you installed Docker and learned the CLI → dockerd → containerd → runc → kernel chain. Kubernetes talks to a container runtime through a standard interface called the CRI (Container Runtime Interface), and the runtime it talks to is containerd directly — not the Docker daemon:

  kubelet
     │  CRI (standard interface)
     ▼
  containerd
     │
   runc
     │
  Linux kernel  → runs the container

This sometimes worries people who spent Part 5 learning Docker. Don’t let it — almost everything transfers. Containerd runs the same OCI images you built and pulled with Docker; the same registries, the same image tags, the same volumes and networks concepts, the same logs on stdout/stderr. Docker was a friendly toolbox built on top of containerd; Kubernetes just uses the engine underneath directly. The one habit that changes: on a Kubernetes node you inspect containers with kubectl (or crictl) rather than docker ps — because the Docker CLI isn’t in the picture on the worker.

🛠️ DevOps Tip — Part 5 was not wasted. Every skill — how images are built and tagged, how registries work, why latest is dangerous, how volumes persist data, how to read container logs — applies unchanged here. Kubernetes adds orchestration on top of container knowledge; it does not replace it. The reader who understands Docker deeply learns Kubernetes faster, because the bottom half of the stack is identical.

Cluster Prerequisites

Kubernetes has a short list of per-node requirements that trip up first-time cluster builders. Run this checklist on every node — control plane and workers — before installing anything.

First, turn off swap. The kubelet’s default configuration still refuses to start if swap is on:

sudo swapoff -a
sudo sed -i '/ swap / s/^/#/' /etc/fstab

swapoff -a disables swap now; the sed line comments out the swap entry in /etc/fstab so it stays off across reboots. Verify with free -h — the Swap row should read 0.

❗ Important — Recent Kubernetes can run with swap enabled via the beta NodeSwap feature (failSwapOn: false plus a swapBehavior setting). That is a deliberate, advanced configuration — not the default. For this lab, keep swap off: it is the documented, predictable path, and one less variable when a node misbehaves. Don’t enable NodeSwap unless you specifically need it and have read its current status in the Kubernetes docs.

Next, load the kernel modules Kubernetes networking needs and set the required sysctls:

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system

The overlay module backs container filesystems; br_netfilter lets iptables see bridged traffic so Pod networking rules apply. bridge-nf-call-iptables=1 and ip_forward=1 allow the node to route packets between Pods and off the node. sysctl --system applies them without a reboot. Confirm with lsmod | grep br_netfilter (the module should appear) and sysctl net.ipv4.ip_forward (should print 1).

Lab Node Design

Two machines, one role each, so the concepts stay clear:

  ┌──────────────────────────────────────┐
  │ ai-control01   control plane          │
  │   10.10.10.10  (example IP)           │
  │   no GPU needed                       │
  ├──────────────────────────────────────┤
  │ ai-node01      GPU worker (from P6)   │
  │   10.10.10.21  (example IP)           │
  │   NVIDIA or AMD GPU + driver          │
  ├──────────────────────────────────────┤
  │ ai-node02      GPU worker (optional)  │
  │   10.10.10.22  (example IP)           │
  └──────────────────────────────────────┘

⚠️ Warning — The 10.10.10.x addresses above are documentation examples, not addresses to copy. Use whatever your own network hands out, and make sure every node can reach every other node on the cluster ports. Don’t hard-code these IPs into a real cluster.

The control plane can be a modest machine — it schedules and stores state, it doesn’t run your models. ai-node01 is the GPU worker you already built. Give each node a unique hostname (hostnamectl set-hostname ai-control01) and a stable IP before you begin, because Kubernetes certificates and the kubeconfig bind to them.

Installing Kubernetes on Ubuntu 26.04

Kubernetes packages come from the community repository at pkgs.k8s.io. (The old apt.kubernetes.io / packages.cloud.google.com repo is retired — do not use it.) The repository URL is scoped to a Kubernetes minor version; confirm the current stable minor at kubernetes.io and substitute it below. At the time of writing that is v1.36 — treat it as an example you verify, not a number to trust blindly.

Run this on every node:

sudo apt-get update && sudo apt-get install -y apt-transport-https ca-certificates curl gpg
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.36/deb/Release.key | \
  sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.36/deb/ /' | \
  sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

The first block imports the repo’s signing key; the echo line adds the version-scoped repository; then you install the three tools:

kubelet — the node agent that runs Pods.
kubeadm — the cluster bootstrap tool you’ll use to init and join.
kubectl — the command-line client you’ll drive the cluster with.

The final apt-mark hold pins all three so a routine apt upgrade can’t silently jump them to a new minor.

❗ Important — Kubernetes enforces a version skew policy: the kubelet must not be a newer minor than the apiserver, and you upgrade one minor at a time. That’s why the version is baked into the repo URL and why you hold the packages — an accidental upgrade that skips a minor or gets ahead of the control plane breaks the node. Upgrading Kubernetes is a deliberate act: change the minor in both the keyring path and the repo URL, apt-mark unhold, upgrade, then re-hold. Never let it happen by accident.

Configuring containerd

Kubernetes needs containerd configured with the systemd cgroup driver, and it must match the kubelet’s driver or Pods misbehave under load. Generate a default config and set the flag:

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null
# In config.toml, under the runc options, set:  SystemdCgroup = true
sudo systemctl restart containerd

containerd config default writes a complete default configuration; you edit it to set SystemdCgroup = true in the runc options block, then restart so containerd reloads it. This one flag is the classic gotcha — a cgroup-driver mismatch produces Pods that run fine idle but fall over under memory pressure. Confirm containerd came back with systemctl status containerd (expect active (running)).

Initializing the Control Plane

kubeadm init bootstraps the control plane: it generates the cluster’s certificate authority, starts the apiserver, scheduler, controller-manager, and etcd as Pods, writes an admin kubeconfig, and prints a join command for workers. Run it only on ai-control01:

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

The --pod-network-cidr tells Kubernetes which private IP range to hand out to Pods. That range must match the CNI you install next — the 10.244.0.0/16 value here is an example that pairs with a common CNI default; use the range your chosen CNI documents.

When it finishes, kubeadm prints something like:

Your Kubernetes control-plane has initialized successfully!
...
kubeadm join 10.10.10.10:6443 --token <TOKEN> \
    --discovery-token-ca-cert-hash sha256:<HASH>

(example output).

⚠️ Warning — Treat that kubeadm join line as a placeholder, not a real token. Join tokens are short-lived credentials that let a machine join your cluster — they expire (default 24 hours), and pasting a stale or shared token won’t work. If yours has expired, regenerate a fresh join command on the control plane with kubeadm token create --print-join-command. Never publish a real token or CA hash — anyone with them can add a node to your cluster.

Configuring kubectl

kubeadm init created an admin kubeconfig at /etc/kubernetes/admin.conf, owned by root. Copy it into your user’s home so kubectl can use it without sudo:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl get nodes

The three lines create ~/.kube, copy the admin config into the place kubectl looks by default, and fix ownership so your user can read it. The kubectl get nodes is your first validation:

NAME           STATUS     ROLES           AGE   VERSION
ai-control01   NotReady   control-plane   1m    v1.36.x

(example output). NotReady is expected here, not a failure — the node has no Pod network yet. It flips to Ready once you install a CNI, which is the next step.

CNI Networking

Kubernetes doesn’t ship Pod networking itself; it delegates to a CNI (Container Network Interface) plugin that gives every Pod an IP and lets Pods on different nodes talk to each other. Pick one — layering two CNIs breaks the cluster in confusing ways.

For this lab use Calico, a widely deployed CNI with good documentation. (Its main alternative is Cilium, an eBPF-based CNI worth knowing about as you grow; either works, but choose one.) Current Calico installs through the Tigera operator. Look up the current release tag from the Calico/Tigera docs and substitute it for <VERSION>:

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/<VERSION>/manifests/tigera-operator.yaml
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/<VERSION>/manifests/custom-resources.yaml

The first manifest installs the Tigera operator (a controller that manages Calico); the second tells it what network to build — its Pod CIDR should match the --pod-network-cidr you passed to kubeadm init. The Pod’s traffic then flows Pod → CNI → cluster network:

  Pod A (node ai-control01)
     │
  CNI plugin (Calico)
     │  routes across nodes
  CNI plugin (Calico)
     │
  Pod B (node ai-node01)

Give it a minute, then re-run kubectl get nodes. Once the CNI Pods are running, ai-control01 transitions to Ready. That transition is your validation that networking is up.

Joining the GPU Worker

With the control plane Ready, bring in ai-node01. You already ran the prerequisites, installed Kubernetes, and configured containerd on it. Now run the join command on ai-node01 (use the fresh one from kubeadm token create --print-join-command if the original expired):

sudo kubeadm join 10.10.10.10:6443 --token <TOKEN> \
    --discovery-token-ca-cert-hash sha256:<HASH>

This registers the worker with the apiserver and starts its kubelet under the control plane’s authority. Back on ai-control01, verify:

kubectl get nodes -o wide

NAME           STATUS   ROLES           VERSION
ai-control01   Ready    control-plane   v1.36.x
ai-node01      Ready    <none>          v1.36.x

(example output). Both Ready means you have a working two-node cluster. -o wide adds internal IPs and OS info, handy for confirming the node came up with the address you expect. You now have a cluster — but it still can’t schedule a GPU, because Kubernetes doesn’t know the GPU exists yet.

Verify the GPU Before Kubernetes Integration

Before you ask Kubernetes to schedule onto a GPU, prove the GPU works on the host exactly as you did in Parts 3–6. Kubernetes cannot fix a broken GPU stack — it can only advertise a working one.

On ai-node01, confirm the hardware is present and the driver is healthy:

lspci | grep -i -E 'vga|3d|nvidia|amd'
nvidia-smi        # NVIDIA hosts

lspci proves the card is physically detected on the PCI bus; nvidia-smi proves the host driver from Part 3 is loaded and can talk to it. AMD hosts run the ROCm tools from Part 4 (rocm-smi) instead. If either fails on the host, stop and fix it using NVIDIA GPUs and CUDA or AMD ROCm before going further.

❗ Important — This ordering is not optional. If nvidia-smi (or rocm-smi) fails on the host, no device plugin will advertise a GPU and every GPU Pod will sit Pending forever. “The container can’t see the GPU” almost always means “the host couldn’t either.” Verify the host first, every time.

How GPUs Become Schedulable Resources

Kubernetes knows about CPU and memory natively. It knows nothing about GPUs out of the box — the scheduler does not auto-discover accelerators. A GPU becomes a schedulable resource only after a device plugin (or an operator that installs one) advertises it to the node. The visibility chain looks like this:

  Physical GPU
      │
  Host driver          (Part 3 / Part 4)
      │
  Device plugin /       advertises the GPU
  GPU Operator          to the kubelet
      │
  Node resource         nvidia.com/gpu: 1
      │
  Pod request           limits: nvidia.com/gpu: 1

Until the device plugin runs, kubectl describe node ai-node01 shows no GPU resource, and a Pod asking for one stays Pending. The plugin’s whole job is to turn “there is a working GPU on this host” into “this node has one nvidia.com/gpu the scheduler can hand out.”

NVIDIA GPU Integration

For NVIDIA, the recommended path is the NVIDIA GPU Operator, installed with Helm. The critical detail for this series: your host already has the driver (Part 3) and the NVIDIA Container Toolkit (Part 5), so you install the Operator with driver management turned off and let it manage only the Kubernetes-side pieces:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update
helm install --wait --generate-name -n gpu-operator --create-namespace \
  nvidia/gpu-operator --set driver.enabled=false

driver.enabled=false tells the Operator: don’t install or manage a GPU driver, because the host owns it. With that set, the Operator manages the Kubernetes integration layer:

the device plugin that advertises nvidia.com/gpu to the scheduler,
integration with the NVIDIA Container Toolkit so Pods reach the GPU,
NFD (Node Feature Discovery), which labels nodes by their hardware,
and DCGM, NVIDIA’s GPU telemetry, which the monitoring lesson (Part 8) consumes.

Present the Operator version as current: Helm supports a --version flag if you need to pin one, but don’t hard-code a stale version — check the current release in NVIDIA’s docs. Verify the actual behavior after install rather than assuming every sub-component is present; configurations vary, and you can disable pieces you don’t need.

🤖 AI Infrastructure Tip — There’s a lighter alternative: the NVIDIA k8s-device-plugin DaemonSet on its own. It advertises nvidia.com/gpu and nothing else — no NFD, no DCGM, no toolkit management. For a bare single-node lab that’s enough. Choose the GPU Operator when you want “serious infrastructure” — it also brings the DCGM telemetry you’ll wire into Prometheus in Part 8, so installing it now saves work later. The device plugin is the minimal option; the Operator is the batteries-included one.

A Note on Helm

You just used Helm, the package manager for Kubernetes. Where apt installs .deb packages onto a host, Helm installs charts — bundles of Kubernetes manifests with configurable values — into a cluster. --set driver.enabled=false overrode one chart value; helm repo add registered a chart source. You can list what’s installed with:

helm list -A

-A lists releases across all namespaces. That’s as much Helm as this lesson needs — it’s a tool you’ll meet again in Part 8 for the monitoring stack. The Kubernetes & Helm guides go deeper when you want it.

Verify the NVIDIA GPU Resource

Once the Operator’s Pods are running, ask the node whether it now advertises a GPU:

kubectl describe node ai-node01

In the output, look under Capacity and Allocatable for the GPU resource:

Capacity:
  cpu:             16
  memory:          64Gi
  nvidia.com/gpu:  1
Allocatable:
  nvidia.com/gpu:  1

(example output). nvidia.com/gpu: 1 under both Capacity (total) and Allocatable (available to schedule) means the scheduler can now place a GPU Pod. If that line is absent, the device plugin isn’t advertising — jump to troubleshooting.

Your First GPU Pod

Prove end-to-end GPU scheduling with the smallest possible workload: a Pod that runs nvidia-smi and exits. Save this as gpu-check.yaml:

apiVersion: v1
kind: Pod
metadata: { name: gpu-check }
spec:
  restartPolicy: Never
  containers:
    - name: cuda
      image: nvidia/cuda:12.6.2-base-ubuntu24.04
      command: ["nvidia-smi"]
      resources:
        limits:
          nvidia.com/gpu: 1

The image tag is an example — pick a current one from the nvidia/cuda repository on Docker Hub that suits your driver. restartPolicy: Never means “run once, don’t restart” — right for a one-shot check. Apply it and read the logs:

kubectl apply -f gpu-check.yaml
kubectl logs gpu-check

kubectl logs prints the container’s stdout — here, the nvidia-smi table. If it shows your GPU, the entire chain works: scheduler → GPU node → device plugin → toolkit → container.

✅ Validation — Success looks like this:

kubectl get pod gpu-check shows Completed (it ran and exited cleanly).

kubectl logs gpu-check prints the same GPU name and VRAM total as nvidia-smi on the host.

The Pod was scheduled onto ai-node01 (kubectl get pod gpu-check -o wide shows the node).

If all three hold, Kubernetes can now schedule GPU workloads. Clean up with kubectl delete pod gpu-check.

Understanding the GPU Resource Request

Look again at that resources block — it is where a GPU request differs fundamentally from CPU or memory:

resources:
  limits:
    nvidia.com/gpu: 1

CPU and memory are compressible/divisible: you can request 500m (half a core) or 256Mi, and many Pods share a node’s CPU. A GPU request under limits is different — by default it reserves a whole physical GPU, exclusively, for that Pod. There is no nvidia.com/gpu: 0.5 by default. A GPU is requested only as a limit (Kubernetes treats it as both request and limit), and one card serves one Pod.

That has a direct scheduling consequence: if ai-node01 has one GPU and you ask for two replicas that each request nvidia.com/gpu: 1, exactly one Pod runs and the other sits Pending, waiting for a GPU that doesn’t exist. This isn’t a bug — it’s the scheduler correctly refusing to double-book a card.

❗ Important — Advanced topic: GPU sharing. Whole-GPU scheduling is the default, but several mechanisms can let multiple Pods share one card: MIG (Multi-Instance GPU, which partitions certain data-center GPUs into hardware slices), time-slicing (the device plugin advertises a card more than once and interleaves work), and DRA (Dynamic Resource Allocation, a newer Kubernetes framework for flexible device requests). Every one of these depends on your specific hardware, driver version, Kubernetes version, and operator configuration — MIG only works on GPUs that support it; time-slicing trades isolation for density; DRA’s maturity moves between releases. None is required for this lesson, and none is automatic. If you need GPU sharing, verify what your exact stack supports in the current NVIDIA and Kubernetes docs rather than assuming it works. Don’t design around a sharing feature you haven’t confirmed.

The AMD GPU Path

AMD GPUs are a separate integration from NVIDIA — different plugin, different resource name, different host stack. The resource Kubernetes advertises is amd.com/gpu, published by either the ROCm k8s-device-plugin (a DaemonSet) or the AMD GPU Operator (a Helm chart, into a namespace such as kube-amd-gpu). The host needs the amdgpu driver and ROCm from Part 4. Pods then request amd.com/gpu: 1 exactly the way NVIDIA Pods request nvidia.com/gpu: 1.

❗ Important — Carry forward the Part 4 honesty about ROCm on Ubuntu 26.04. Current ROCm officially supports Ubuntu 24.04 and 22.04; 26.04 may not be on the supported list yet. Before you commit an AMD node to a cluster, verify both your GPU and your OS version against the AMD compatibility matrix at rocm.docs.amd.com/en/latest/compatibility/compatibility-matrix.html. If 26.04 isn’t listed, you may be on unsupported ground — plan accordingly. See AMD ROCm AI Infrastructure for the full picture.

One more trap to name explicitly: an NVIDIA CUDA image will not run on AMD hardware, and an AMD/ROCm image won’t run on NVIDIA. They are different builds against different compute platforms. In a mixed cluster, the Pod’s image and the node’s GPU must match — which is exactly what the next sections enforce.

Heterogeneous Clusters and Node Labels

Real clusters mix hardware, and Kubernetes needs your help to route each Pod to compatible nodes. The tool is labels — arbitrary key/value tags on nodes:

kubectl label node ai-node01 accelerator=nvidia workload=ai

This tags ai-node01 as an NVIDIA, AI-workload node. Labels are pure metadata — they describe a node so you can select it later. Verify with kubectl get nodes --show-labels.

⚠️ Warning — A label does not reserve or allocate a GPU. Labeling a node accelerator=nvidia tells the scheduler “this node is an NVIDIA box”; it does nothing about availability. A Pod that lands there via a label but forgets resources.limits.nvidia.com/gpu: 1 gets no GPU — it runs on the GPU node with no GPU attached. Placement (labels/affinity) and allocation (resource requests) are two separate jobs; you almost always need both.

nodeSelector

The simplest way to steer a Pod to labeled nodes is nodeSelector in the Pod spec:

spec:
  nodeSelector:
    accelerator: nvidia
  containers:
    - name: llm
      image: ollama/ollama
      resources:
        limits:
          nvidia.com/gpu: 1

nodeSelector: { accelerator: nvidia } restricts scheduling to nodes carrying that label; the nvidia.com/gpu: 1 still does the actual reservation. Together they mean “an NVIDIA node and a free GPU on it.” This is how you keep a CUDA image off an AMD node.

Node Affinity

nodeAffinity is the more expressive form of the same idea — it supports “prefer” as well as “require,” and richer matching (in/notin/exists). You reach for it when a simple equality label isn’t enough, for example “require an NVIDIA GPU, and prefer a node also labeled workload=ai.” It’s more verbose than nodeSelector; know it exists, and use nodeSelector until you actually need the extra expressiveness.

Taints, Tolerations, and GPU Economics

Labels and selectors attract the right Pods to GPU nodes. Taints do the opposite — they repel everything that isn’t explicitly allowed, which is how you keep cheap, ordinary workloads off your expensive GPU hardware.

kubectl taint nodes ai-node01 nvidia.com/gpu=present:NoSchedule

This taint says “don’t schedule anything here unless it explicitly tolerates nvidia.com/gpu=present.” A GPU Pod opts in with a matching toleration:

spec:
  tolerations:
    - key: nvidia.com/gpu
      operator: Equal
      value: present
      effect: NoSchedule
  containers:
    - name: llm
      image: ollama/ollama
      resources:
        limits:
          nvidia.com/gpu: 1

Now only Pods that both tolerate the taint and request a GPU can land on ai-node01; a random logging sidecar or web front end stays off it.

🛠️ DevOps Tip — This is GPU-node economics, not fussiness. A GPU node costs far more to buy and run than a CPU node — the card, the power draw, the cooling, and the opportunity cost of a GPU sitting idle under some batch job that never needed it. Taints make the node exclusive to work that requires it, so you’re not paying for an accelerator to run a web server. On a real platform, GPU nodes are almost always tainted. Combine the pair: taint the node so junk stays off, label + request so GPU work gets on.

Deploying the Part-6 LLM to Kubernetes

Time to put it together. You’ll take the Ollama model server from Part 6 and run it as a proper Kubernetes workload: a Deployment that requests a GPU, with configuration, secrets, persistent storage, health probes, and a Service in front.

A Deployment is the standard way to run a long-lived, replicated workload. It manages a ReplicaSet, which manages the Pods:

  Deployment  (desired: N replicas, image, template)
      │
  ReplicaSet  (keeps N Pods alive)
      │
  Pod ... Pod   (the running containers)

Here’s the core of the Deployment — one replica, requesting one GPU:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
spec:
  replicas: 1
  selector:
    matchLabels: { app: ollama }
  template:
    metadata:
      labels: { app: ollama }
    spec:
      nodeSelector:
        accelerator: nvidia
      containers:
        - name: ollama
          image: ollama/ollama
          ports:
            - containerPort: 11434
          resources:
            limits:
              nvidia.com/gpu: 1

replicas: 1 because you have one GPU. The nodeSelector steers it to the NVIDIA node; the GPU limit reserves the card. If you set replicas: 2 with one GPU, the second Pod sits Pending — a concrete demonstration of whole-GPU scheduling.

Configuration: ConfigMap and Secret

Non-secret settings — a model name, a log level — belong in a ConfigMap, decoupled from the image so you can change them without rebuilding:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ollama-config
data:
  OLLAMA_MODEL: "llama3"
  OLLAMA_LOG_LEVEL: "info"

You then inject those into the container as environment variables (via envFrom or individual env entries referencing the ConfigMap). Real credentials — an API token, a registry password — go in a Secret instead.

⚠️ Warning — A Kubernetes Secret is base64-encoded, not encrypted. Base64 is encoding — reversible by anyone with kubectl get secret ... -o yaml and a base64 -d. A Secret is the right place for credentials (it’s handled differently from a ConfigMap, mountable as a file, and can be RBAC-restricted and encrypted at rest by the cluster operator), but do not mistake the default for confidentiality. Never put a real credential in a ConfigMap, never commit a Secret manifest with a live value to Git, and understand that “it’s in a Secret” is not, by itself, “it’s encrypted.”

Persistent Storage: PV and PVC

Pods are disposable — recreate the Ollama Pod and its writable layer is gone, along with any multi-gigabyte model it downloaded. The fix is the same idea as the Docker volume from Part 5, expressed in Kubernetes terms: a PersistentVolume (PV, a piece of storage) claimed by a PersistentVolumeClaim (PVC, a request for storage) and mounted into the Pod.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ollama-models
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 50Gi

Mount that claim at Ollama’s model directory in the Deployment’s Pod template:

      volumes:
        - name: models
          persistentVolumeClaim:
            claimName: ollama-models
      containers:
        - name: ollama
          image: ollama/ollama
          volumeMounts:
            - name: models
              mountPath: /root/.ollama

Now the model lives in the PVC at /root/.ollama, outside the Pod’s lifecycle. Delete and recreate the Pod and the new one finds the model already present — no re-download. That solves the real operational pain: without persistence, every new Pod re-downloads the model, wasting bandwidth and adding minutes to every restart. A shared, persistent cache is what makes restarts and rollouts cheap.

🤖 AI Infrastructure Tip — Storage performance matters for model load time. For this lab, local NVMe backing the PV is fastest and simplest. Network storage — NFS, Ceph, or a cloud volume — lets multiple nodes share one model cache and survives a node loss, which you’ll want on a real multi-node platform, but it adds latency and is not required here. Match the storage class to the job: fast-local for a single GPU node, shared-network when several nodes must pull from the same cache.

The Service

Pods come and go with changing IPs; a Service gives them one stable name and virtual IP. A ClusterIP Service — reachable only inside the cluster — is the correct baseline for a backend model server:

apiVersion: v1
kind: Service
metadata:
  name: ollama
spec:
  selector:
    app: ollama
  ports:
    - port: 11434
      targetPort: 11434

The selector: { app: ollama } wires the Service to the Deployment’s Pods; anything in the cluster can now reach the model at ollama:11434, and the Service load-balances across healthy Pods. This mirrors the service-name DNS you used in Compose (http://llm:11434), now cluster-wide and Pod-count-aware.

Ingress and External Access

❗ Important — A ClusterIP Service is internal only — nothing outside the cluster can reach it, which is exactly right for a backend. Controlled external access (a public URL, TLS, routing rules) is the job of an Ingress or the Gateway API, and that’s Part 9. Do not expose the model server to the internet here. Keep it ClusterIP; the inference-server lesson covers safe external exposure properly.

Health Probes: Started Is Not Ready

An AI container is Running long before it can serve a request — it may still be loading a multi-gigabyte model into VRAM. Kubernetes has three probes for exactly this gap, and getting them right is critical for AI workloads because model load is slow:

startupProbe — guards a slow start. Until it succeeds, the other probes are held off. This is what gives a 90-second model load room to finish.
readinessProbe — gates Service traffic. A Pod that fails readiness is removed from the Service’s load-balancing pool — no requests until the model is actually loaded.
livenessProbe — detects a hung container and restarts it.

          startupProbe:
            httpGet: { path: /, port: 11434 }
            failureThreshold: 30
            periodSeconds: 10
          readinessProbe:
            httpGet: { path: /, port: 11434 }
            periodSeconds: 10
          livenessProbe:
            httpGet: { path: /, port: 11434 }
            periodSeconds: 20

Each probes an HTTP endpoint on the API port, not just “is the process alive.” The startupProbe’s generous failureThreshold × periodSeconds budget (here up to 300 seconds) covers a slow model load before liveness starts counting.

⚠️ Warning — An aggressive livenessProbe is the classic AI-workload footgun. If liveness starts checking immediately with a short timeout, it fails during the normal 90-second model load, Kubernetes concludes the container is hung, restarts it — and the restart begins loading the model again, failing liveness again, forever. That’s a restart loop caused entirely by the probe, not the app. Give slow starts room: use a startupProbe (or a large initialDelaySeconds/failureThreshold on liveness) so “still loading” is never mistaken for “hung.” Container started ≠ model ready.

Logs, Events, and Unschedulable Pods

Three commands answer three different questions, and knowing which to reach for is most of Kubernetes troubleshooting:

kubectl logs <pod>          # what the APPLICATION printed
kubectl describe pod <pod>  # K8s state + recent events for this Pod
kubectl get events          # cluster-wide events (scheduling failures)

kubectl logs shows the container’s own stdout/stderr — use it for application errors (model failed to load, bad config).
kubectl describe pod shows Kubernetes’ view — the Pod’s phase, its conditions, why the scheduler placed it (or couldn’t), and recent events attached to it. Use it when the Pod isn’t behaving and the app logs are empty because the container never started.
kubectl get events surfaces cluster-level happenings, including why a Pod is Pending.

A Pending GPU Pod is the signature AI-cluster failure. The scheduler couldn’t place it. kubectl describe pod (bottom, in Events) or kubectl get events tells you which of these it is:

  Pod Pending — likely causes
  ---------------------------------
  - No GPU resource advertised
      (device plugin not running)
  - All GPUs already allocated
  - nodeSelector matches no node
  - Taint with no matching toleration
  - Not enough CPU / memory
  - PVC still Pending (no storage)
  - Target node NotReady

Work down that list against the describe output and the cause is almost always obvious.

Troubleshooting

GPU scheduling fails one layer at a time. Diagnose top-down through this stack every time and you isolate the fault instead of guessing:

  Cluster healthy?     (kubectl get nodes)
      │
  Node Ready?          (describe node)
      │
  GPU driver OK?       (nvidia-smi on host)
      │
  GPU integration?     (nvidia.com/gpu advertised)
      │
  Scheduler placed?    (Pod not Pending)
      │
  Pod running?         (kubectl logs / describe)
      │
  Application ready?   (readiness / API responds)

Work the common failures with Problem → Likely Cause → Check → Fix → Validate.

🔍 Troubleshooting — Node stuck NotReady. Problem: kubectl get nodes shows a node NotReady. Likely cause: no CNI installed yet, or the kubelet/containerd is down. Check: kubectl describe node <name>; on the node, systemctl status kubelet containerd. Fix: install a CNI if none exists; otherwise start/repair kubelet and containerd, confirm swap is off and the sysctls are set. Validate: the node flips to Ready.

🔍 Troubleshooting — CNI not working / Pods have no network. Problem: Pods stay ContainerCreating or can’t reach each other. Likely cause: no CNI, a broken CNI install, or two CNIs layered. Check: are the CNI Pods running? Does the Pod CIDR match --pod-network-cidr? Fix: install exactly one CNI with the correct CIDR; never layer two. Validate: nodes go Ready and a test Pod gets an IP.

🔍 Troubleshooting — GPU works on host but not in Kubernetes. Problem: nvidia-smi works on the node, but kubectl describe node shows no nvidia.com/gpu. Likely cause: no device plugin / GPU Operator advertising the GPU. Check: are the GPU Operator / device-plugin Pods running in their namespace? Fix: install the GPU Operator (driver.enabled=false) or the device-plugin DaemonSet. Validate: nvidia.com/gpu appears under Capacity/Allocatable.

🔍 Troubleshooting — nvidia.com/gpu missing after install. Problem: the Operator is installed but the resource still isn’t advertised. Likely cause: an Operator component (device plugin, toolkit, NFD) failed to start. Check: kubectl get pods -n gpu-operator; kubectl logs the failing Pod. Fix: resolve what the failing Pod reports (often a driver mismatch or toolkit config). Validate: all Operator Pods Running; resource advertised.

🔍 Troubleshooting — GPU Operator Pods failing. Problem: Pods in the gpu-operator namespace crash or stay Init. Likely cause: host driver not actually working, or driver.enabled set wrong for your setup. Check: nvidia-smi on the host; the failing Pod’s logs and events. Fix: repair the host driver (Part 3); confirm driver.enabled=false matches “host owns the driver.” Validate: Operator Pods Running.

🔍 Troubleshooting — AMD amd.com/gpu not advertised. Problem: an AMD node shows no amd.com/gpu. Likely cause: ROCm device plugin / AMD GPU Operator not running, amdgpu/ROCm host issue, or 26.04 not on ROCm’s supported list. Check: host rocm-smi; the AMD plugin Pods; the AMD compatibility matrix. Fix: deploy the ROCm device plugin / AMD Operator; verify the host ROCm stack and OS support. Validate: amd.com/gpu appears on the node.

🔍 Troubleshooting — GPU Pod stuck Pending. Problem: a Pod requesting a GPU never schedules. Likely cause: no GPU advertised, all GPUs allocated, nodeSelector mismatch, untolerated taint, PVC Pending, or node NotReady. Check: kubectl describe pod <name> — read the Events at the bottom. Fix: address the specific reason (advertise the GPU, free a GPU, fix the selector/toleration, provision the PVC). Validate: the Pod schedules onto a GPU node and runs.

🔍 Troubleshooting — Pod scheduled to the wrong node. Problem: a Pod lands on a node that can’t run it (e.g. CUDA image on a non-NVIDIA node). Likely cause: missing or wrong nodeSelector/affinity. Check: the Pod’s nodeSelector; the node’s labels (kubectl get nodes --show-labels). Fix: label nodes correctly and set the matching nodeSelector (plus the GPU request). Validate: the Pod runs on the intended node.

🔍 Troubleshooting — No GPU inside a running Pod. Problem: the Pod runs but sees no GPU. Likely cause: the Pod didn’t request nvidia.com/gpu (a label alone doesn’t reserve one). Check: the Pod spec’s resources.limits. Fix: add nvidia.com/gpu: 1 (or amd.com/gpu: 1) under limits. Validate: nvidia-smi inside the Pod shows the GPU.

🔍 Troubleshooting — CUDA unavailable in the container. Problem: the Pod has a GPU but CUDA calls fail. Likely cause: image CUDA version exceeds the host driver’s supported CUDA, or toolkit integration missing. Check: host driver’s supported CUDA (nvidia-smi); the image’s CUDA version. Fix: choose a CUDA image within the host driver’s support; confirm the GPU Operator’s toolkit integration. Validate: a CUDA workload runs in the Pod.

🔍 Troubleshooting — ROCm can’t see the GPU in the Pod. Problem: an AMD Pod runs but rocm-smi shows nothing. Likely cause: amd.com/gpu not requested, host amdgpu/ROCm broken, or OS unsupported. Check: the Pod’s resource request; host rocm-smi; compatibility matrix. Fix: request amd.com/gpu, repair the host ROCm stack, verify OS support. Validate: rocm-smi in the Pod lists the GPU.

🔍 Troubleshooting — ImagePullBackOff. Problem: the Pod can’t pull its image. Likely cause: wrong image name/tag, private registry without credentials, or no network. Check: kubectl describe pod — the pull error names the reason. Fix: correct the image reference; add an imagePullSecret for private registries. Validate: the image pulls and the container starts.

🔍 Troubleshooting — CrashLoopBackOff. Problem: the container starts, crashes, and restarts repeatedly. Likely cause: the app errors on startup (bad config, missing mount, wrong command). Check: kubectl logs <pod> (and --previous for the last crash). Fix: correct the root cause from the logs — a restart policy is not a fix. Validate: the Pod reaches a stable Running/Ready.

🔍 Troubleshooting — Model won’t load. Problem: the container runs but the model never finishes loading. Likely cause: insufficient VRAM, missing model in the PVC, or another process holding GPU memory. Check: kubectl logs for GPU-memory errors; nvidia-smi on the node for used VRAM. Fix: use a smaller/quantized model, ensure the model is cached in the PVC, free stray GPU memory. Validate: logs show the model loaded and the API answers.

🔍 Troubleshooting — PVC stuck Pending. Problem: the PVC never binds, so the Pod stays Pending. Likely cause: no matching PV or no default StorageClass to provision one. Check: kubectl describe pvc <name>; kubectl get storageclass. Fix: create a PV or set a default StorageClass that can provision the request. Validate: the PVC shows Bound and the Pod schedules.

🔍 Troubleshooting — Service can’t reach the Pod. Problem: ollama:11434 doesn’t respond from inside the cluster. Likely cause: Service selector doesn’t match Pod labels, or no Pod is Ready. Check: kubectl get endpoints ollama — empty means no matching ready Pods. Fix: align the Service selector with the Pod labels; fix readiness so Pods enter the pool. Validate: the endpoints list the Pod IPs and requests succeed.

🔍 Troubleshooting — Readiness never succeeds. Problem: the Pod runs but stays 0/1 Ready; the Service has no endpoints. Likely cause: the readiness probe targets the wrong path/port, or the model is genuinely not ready. Check: the probe config; kubectl logs for load progress. Fix: point the probe at the real API endpoint; give slow loads a startupProbe. Validate: the Pod goes Ready and joins the Service.

🔍 Troubleshooting — Liveness restart loop. Problem: the Pod restarts every ~minute during model load. Likely cause: an aggressive livenessProbe fails before the model finishes loading. Check: the liveness timing; correlate restarts with the load duration in logs. Fix: add a startupProbe (or raise liveness initialDelaySeconds/failureThreshold). Validate: the Pod loads the model once and stays up.

🔍 Troubleshooting — All GPUs allocated. Problem: a new GPU Pod is Pending though the node has GPUs. Likely cause: every GPU is already reserved by running Pods (whole-GPU scheduling). Check: kubectl describe node — Allocated resources vs Capacity for nvidia.com/gpu. Fix: scale down a workload, add a GPU node, or (advanced, if supported) enable GPU sharing. Validate: an Allocatable GPU frees up and the Pod schedules.

Hands-On Lab: Deploy a GPU-Powered AI Workload to Kubernetes

🧪 Hands-On Lab — Turn your Part-6 GPU host into a two-node cluster and run the Ollama model server on it, scheduled onto the GPU, with its model data surviving a Pod delete. Do these in order.

Prepare both nodes. On ai-control01 and ai-node01: set hostnames, confirm connectivity, swapoff -a and comment swap in /etc/fstab.
Load modules and sysctls. Add overlay + br_netfilter; set bridge-nf-call-iptables=1 and ip_forward=1; sysctl --system.
Install Kubernetes. Add the pkgs.k8s.io repo (current minor), install kubelet kubeadm kubectl, then apt-mark hold all three — on both nodes.
Configure containerd. Generate the default config, set SystemdCgroup = true, restart containerd — on both nodes.
Init the control plane. On ai-control01: sudo kubeadm init --pod-network-cidr=<cni-cidr>; save the printed join command.
Set up kubectl. Copy admin.conf to ~/.kube/config; run kubectl get nodes (expect NotReady).
Install one CNI. Apply Calico via the Tigera operator (current version tag); wait for ai-control01 to go Ready.
Join the worker. On ai-node01: run the join command (regenerate with kubeadm token create --print-join-command if expired).
Verify the cluster. kubectl get nodes -o wide → both Ready.
Verify the GPU on the host. On ai-node01: lspci | grep -i nvidia and nvidia-smi (AMD: rocm-smi). Fix the host first if either fails.
Install GPU integration. NVIDIA: GPU Operator via Helm with driver.enabled=false. AMD: the ROCm device plugin / AMD Operator.
Confirm the resource. kubectl describe node ai-node01 → nvidia.com/gpu (or amd.com/gpu) under Capacity/Allocatable.
Run a one-shot GPU Pod. Apply gpu-check.yaml; kubectl logs gpu-check should match host nvidia-smi. Delete it.
Label the GPU node. kubectl label node ai-node01 accelerator=nvidia workload=ai.
Taint the GPU node. kubectl taint nodes ai-node01 nvidia.com/gpu=present:NoSchedule.
Create the PVC. Apply the ollama-models PersistentVolumeClaim; confirm it Bound.
Create the ConfigMap. Apply ollama-config (model name, log level).
Deploy Ollama. Apply the Deployment (1 replica, GPU limit, nodeSelector, toleration, PVC mount at /root/.ollama, health probes).
Watch it come up. kubectl get pods -w; kubectl describe pod if it’s Pending; kubectl logs to watch the model load.
Create the Service. Apply the ClusterIP ollama Service on port 11434; confirm kubectl get endpoints ollama is populated.
Test from inside the cluster. From a temporary Pod (kubectl run tmp --rm -it --image=curlimages/curl -- sh), curl http://ollama:11434.
Prove persistence. kubectl delete pod the Ollama Pod; the Deployment recreates it; confirm the model is still cached (no re-download).
Prove GPU scheduling limits. Scale to replicas: 2; watch the second Pod sit Pending on a one-GPU cluster; scale back to 1.
Document the cluster. Record Kubernetes minor, CNI + version, GPU integration method, image tags, and the PVC/storage layout.

  ┌────────────────────────────────────────┐
  │  SUCCESS: AI workload on Kubernetes     │
  │                                         │
  │  Cluster ............ 2 nodes Ready ✓   │
  │  GPU advertised ..... nvidia.com/gpu ✓  │
  │  GPU Pod scheduled .. on ai-node01  ✓   │
  │  Ollama Deployment .. Running/Ready ✓   │
  │  Model persists ..... on Pod delete ✓   │
  │  Service reachable .. ollama:11434  ✓   │
  └────────────────────────────────────────┘

Cluster Security Essentials

Kubernetes is powerful enough to be dangerous when misconfigured. This is not a full security course — Part 9 and dedicated material go deeper — but bake in these defaults from day one.

⚠️ Warning — Operate the cluster with least privilege. Do not hand out cluster-admin by default — scope permissions to what a user or workload actually needs. Store credentials in Secrets, and remember a Secret is base64-encoded, not encrypted, so still keep manifests out of Git and restrict access. Do not run privileged Pods or grant broad host access unless a workload provably requires it. Be cautious with hostPath volumes — they punch through isolation to the node’s filesystem. And protect the kubeconfig / API access like root credentials: whoever holds a cluster-admin kubeconfig owns every node, every Secret, and every GPU in the cluster.

Access control in Kubernetes is RBAC — Role-Based Access Control. Briefly: a Role (or ClusterRole) lists allowed verbs on resources, and a RoleBinding grants that Role to a user or service account. Start narrow — a namespace-scoped Role for a workload — and widen only when needed. The Kubernetes & Helm guides cover RBAC and cluster hardening in depth.

Where the Build-Along Stands

  ai-cluster build-along
  ------------------------------------
  Ubuntu nodes ............... [done]
  Kubernetes cluster ......... [done]
  CNI networking ............. [done]
  GPU scheduling ............. [done]
  Persistent models .......... [done]
  Service discovery .......... [done]
  ------------------------------------
  Monitoring ................. [next]
  Inference server ........... [upcoming]
  Production platform ........ [upcoming]

You now have a real GPU-aware platform: a control plane, a GPU worker, scheduled workloads, persistent storage, and a stable Service. But there’s a question you can’t yet answer.

  ┌───────────────┐     ┌───────────────┐
  │ ai-control01  │────▶│  ai-node01    │
  │ control plane │     │  GPU worker   │
  └───────────────┘     │  ollama Pod   │
                        │  GPU: ??? %    │
                        └───────────────┘

Are the GPUs actually busy — or allocated and idle? Are they overheating, running out of VRAM, or throttling under load? Is that expensive card doing real work, or wasting money? Kubernetes scheduled the Pod, but it can’t tell you whether the hardware underneath is healthy or the model is fast. For that you need monitoring — which is exactly Part 8.

GPU Hardware to Consider for a Kubernetes AI Lab

To learn everything in this lesson you need one GPU on ai-node01 — the scheduling, labeling, persistence, and Service concepts are identical whether your cluster has one card or fifty. A second GPU (in ai-node01 or an added ai-node02) only lets you watch the scheduler place work across accelerators; it teaches nothing new conceptually. Don’t buy hardware to learn Kubernetes.

When you do size a GPU node for real workloads, weigh:

VRAM — the model must fit; Kubernetes reserves the whole card but can’t add memory to it.
Driver + Kubernetes compatibility — confirm the GPU is supported by a current driver on Ubuntu 26.04 and by your device plugin / operator.
Power and cooling — a GPU node under sustained inference runs hot and draws real watts; provision PSU headroom and airflow.
PCIe — an available slot of the right generation with physical clearance.
Workload fit — match VRAM and compute to the models you’ll actually run, not the biggest card on the shelf.
Budget — GPU nodes are expensive to buy and to run; taints keep them exclusive to work that needs them.

The recommended GPUs below are grouped from a homelab learning card up to a dedicated AI development system. Treat them as starting points sized to a node’s role in a cluster, not a shopping list.

What You Learned

Why a single Docker host stops being enough, and the specific problems — failure recovery, scaling, scheduling, self-healing, discovery, config, rollouts, GPU allocation — that Kubernetes solves as an orchestrator, not an AI framework.
The architecture of a GPU-aware cluster: user → Service → Pod → runtime → GPU → node, split across a control plane (ai-control01) and GPU workers (ai-node01).
Which Kubernetes components to reason about, and that kubelet drives containerd through the CRI — so your Docker knowledge from Part 5 transfers intact.
How to install Kubernetes from pkgs.k8s.io on Ubuntu 26.04, configure containerd with SystemdCgroup = true, kubeadm init a control plane, install one CNI, and join a GPU worker — with versions held for the skew policy.
That GPUs are not native resources: a device plugin / GPU Operator must advertise nvidia.com/gpu (or amd.com/gpu) before the scheduler can place GPU Pods, and a GPU request reserves a whole card by default.
How to route workloads in a heterogeneous cluster with labels, nodeSelector, affinity, and taints/tolerations — placement and allocation are separate jobs.
How to deploy the Part-6 LLM as a Deployment with a ConfigMap, a Secret (base64 is encoding, not encryption), a PVC for persistent models, health probes tuned for slow model loads, and a ClusterIP Service.
A top-down troubleshooting method — Cluster → Node → GPU Driver → GPU Integration → Scheduler → Pod → Application — applied to every common GPU-scheduling failure.

Next Lesson

Monitoring Ubuntu AI Infrastructure — you can schedule GPU workloads now, but you can’t yet see whether they’re healthy. Part 8 builds the observability layer: Prometheus scraping GPU and cluster metrics, Grafana dashboards, DCGM GPU telemetry, and alerts that fire on real problems — so you finally know whether those GPUs are busy, hot, out of VRAM, or quietly wasting money. Continue to Part 8 →

To reinforce the platform underneath this lesson, the Kubernetes & Helm guides go deep on scheduling, RBAC, and Helm, and the Docker guides and Docker Academy shore up the container fundamentals that Kubernetes builds on. If a node or GPU isn’t cooperating, revisit Docker for AI Workloads and Running Local LLMs to confirm the workload runs on one host first, or the NVIDIA and AMD ROCm lessons to fix the host GPU. The series index shows how the inference server and production platform build on the cluster you just stood up.