Build a Production AI Platform on Ubuntu 26.04

Across nine lessons you went from a bare Ubuntu install to a GPU-aware Kubernetes cluster serving a secured inference API. Every piece works — but “works on my cluster today” is not the same as “I could operate this for real users tomorrow.” This capstone is about that gap: the automation, security, observability, and recovery controls that turn a working demo into a platform you can run, change, and repair without guessing.

What You’ll Learn

• What the word production actually demands — reproducibility, reliability, security, maintainability, observability, automation, controlled change, and recovery — and why each one is a control you build, not a state you reach
• How the whole platform fits together in one architecture: external clients → gateway → inference → GPU nodes → storage → monitoring, plus the Git → CI → registry → GitOps → Kubernetes delivery path
• How to define infrastructure with Terraform or OpenTofu and configure Ubuntu with Ansible, layered under Kubernetes and GitOps
• How to lay out one Git repository for the platform, and run a GitLab CI/CD pipeline that lints, validates, tests, builds, scans, and deploys
• How Argo CD makes Git the source of truth and continuously reconciles the cluster to it
• How to handle secrets, RBAC, NetworkPolicy, and the host firewall so the GPU API is never carelessly exposed
• How to treat container images and models as supply-chain dependencies — pinned, scanned, versioned, and changed on purpose
• How to run rolling updates, node maintenance, and GPU driver upgrades without dropping the service — and why a surge needs a spare GPU
• What high availability, backups, and disaster recovery honestly mean for this platform, and how to answer “can I rebuild it from scratch?”
• How to run production monitoring, alerting, on-call, runbooks, and SLOs, and do evidence-based capacity planning before buying another GPU
• A full production-readiness checklist, a capstone lab, and a troubleshooting matrix for the whole platform

By the end you will have combined every earlier lesson into a single, version-controlled, self-healing, observable platform — and understand which control to reach for when it breaks.

The Capstone: From Working to Operable

Everything up to now answered how do I make this run? Part 3 gave you a GPU driver, Part 5 containers, Part 7 a scheduler, Part 9 a secure API. This lesson answers a different question: if I had to operate this AI service for real users tomorrow, what would I need?

The honest answer is that the workload is the easy part. What separates a demo from a platform is a set of operational controls that have nothing to do with adding features:

  Demo                Production platform
  ----------------    ----------------------------
  It runs         →   It runs the same way twice
  I configured it →   Config lives in Git, reviewed
  I deployed it   →   A pipeline deploys it
  It's up         →   I'm alerted when it isn't
  It's secure(?)  →   Least-privilege, by default
  It works        →   I can rebuild it from zero

None of the right column is a new install command. Each is a control — a deliberate mechanism that makes the platform reproducible, reliable, secure, observable, or recoverable. This capstone builds those controls on top of the cluster you already have. You will not install a bigger model or a faster runtime. You will make the platform you already built operable.

❗ Important — Advanced topic: this lesson is a survey with working examples, not a certification. It touches Terraform/OpenTofu, Ansible, GitLab CI/CD, Argo CD, RBAC, backups, and DR — each of which has its own deep guides on this site. The goal is that you understand how the pieces connect and can build a credible lab version, then go deeper where your role demands it. Where a tool’s exact syntax moves between releases, this lesson tells you to check current docs rather than hard-coding a version that will rot.

The Production Platform Architecture

Here is the whole platform on one page — the strongest picture in the series. Read it in two halves. The top half is the runtime path a user request takes; the bottom half is the delivery path your changes take to reach the cluster.

  RUNTIME PATH (a user request)
  ----------------------------------------
   External clients
        │  HTTPS (TLS)
        ▼
   Gateway  (Gateway API)
     - authentication
     - rate limiting
     - request logging
        │
        ▼
   Inference Service  (vLLM)
        │  ClusterIP: vllm:8000
        ▼
   GPU nodes  ai-node01 / ai-node02
     - whole-GPU scheduling
        │
        ▼
   Storage  (model cache PVC)

  Observed by:
   Prometheus → Grafana → Alertmanager
   (host · GPU · Kubernetes · inference)

  DELIVERY PATH (a change you make)
  ----------------------------------------
   Git repo  (desired state)
        │  push / merge request
        ▼
   CI/CD  (lint→validate→test→
           build→scan→push)
        │
        ▼
   Container registry  (immutable tags)
        │
        ▼
   GitOps controller  (Argo CD)
        │  reconciles cluster → Git
        ▼
   Kubernetes  (applies manifests)

The two paths meet at Kubernetes: the delivery path puts the desired state into the cluster, and the runtime path is what actually serves users once it’s there. Everything in this lesson slots into one of these two diagrams.

The final platform components — the full parts list you are operating:

Layer	Component	Its job
Infrastructure	Terraform / OpenTofu	Provision machines, networks, storage
OS config	Ansible	Configure Ubuntu, prereqs, agents
Orchestration	Kubernetes	Schedule and heal workloads
Delivery	GitLab CI/CD + Argo CD	Build, scan, and reconcile
Serving	vLLM inference + Gateway API	Serve and expose the model
GPU	Driver + device plugin	Advertise nvidia.com/gpu
Storage	PVC (model cache)	Persist model weights
Observability	Prometheus + Grafana + Alertmanager	Metrics, dashboards, alerts
Security	RBAC, NetworkPolicy, secrets, UFW	Least-privilege everywhere
Recovery	Backups + DR runbook	Rebuild from scratch

If that table looks like a lot, remember you already built most of it across Parts 1–9. The capstone is about the connective tissue — the last three rows especially — that lets you operate the first seven.

Infrastructure as Code

Everything you built by hand so far has a hidden liability: it exists only because you typed the commands, in an order only you remember, on machines only you can describe. Rebuild the cluster next month and you will get something subtly different. Infrastructure as Code (IaC) removes that liability by describing the infrastructure in version-controlled files instead of in your shell history.

The contrast is the whole point:

  Manual                IaC
  ------------------    ----------------------
  SSH in and type   →   Declare in a file
  Undocumented      →   Reviewed in a merge req
  One-off, drifts   →   Repeatable, identical
  "How did we...?"  →   git log tells you
  Rebuild = memory  →   Rebuild = apply again

IaC is version-controlled (every change has an author and a diff), repeatable (the same code produces the same infrastructure), and reviewable (a teammate approves it before it lands). On this site the two mainstream tools are Terraform and OpenTofu — they share the same HCL configuration language, so the examples here apply to either. Pick one for your platform and stay consistent.

What IaC actually manages depends on what sits under your cluster. On a cloud or an OpenStack backend, IaC provisions the substrate:

• virtual machines (including cloud GPU instances),
• networks, subnets, and security groups,
• block and object storage,
• DNS records and load balancers.

A minimal, provider-agnostic sketch — treat the resource types as placeholders for whatever your provider names them:

# Illustrative HCL — the exact resource
# TYPE names come from your provider.
variable "gpu_node_count" {
  type    = number
  default = 2
}

resource "example_instance" "gpu_node" {
  count     = var.gpu_node_count
  name      = "ai-node0${count.index + 1}"
  image     = "ubuntu-26-04-lts"
  flavor    = "gpu-large"   # a GPU-backed size
  network   = example_network.ai.id
}

You run terraform plan (or tofu plan) to preview the change, then terraform apply to make it real — and critically, the resulting state file records what was created. That state is precious; we will back it up later.

❗ Important — On bare-metal hardware you own, IaC stops lower. There is no API to conjure a physical GPU server, so provisioning the machine is a manual/PXE/vendor step, and IaC’s role narrows to what does have an API — DNS, load balancers, maybe network gear. That is fine. Don’t force bare metal into a cloud-shaped tool; use IaC for the layers that have APIs and Ansible (next) for the OS. This lesson does not teach Terraform end-to-end — the Infrastructure as Code guides, Terraform guides, and OpenTofu guides do that. Here you only need to see where it fits.

Configuration Management with Ansible

IaC gives you machines. Something still has to turn a blank Ubuntu 26.04 install into a GPU worker or a control-plane node — install packages, create users, set kernel parameters, lay down the GPU and Kubernetes prerequisites from Parts 3 and 7, create directories, and drop in monitoring agents. Doing that by hand is exactly the manual liability IaC removed, and the fix is the same idea applied to the OS: configuration management with Ansible.

Ansible describes the desired state of a machine in YAML playbooks and makes the machine match, over SSH, with no agent to install first. A small, readable task list:

# Illustrative Ansible tasks for a GPU node.
- name: Configure AI worker
  hosts: gpu_nodes
  become: true
  tasks:
    - name: Install base packages
      apt:
        name: [curl, gpg, ca-certificates]
        state: present
        update_cache: true

    - name: Disable swap now (kubelet needs it off)
      command: swapoff -a

    - name: Ensure model cache dir exists
      file:
        path: /srv/models
        state: directory
        mode: "0750"

Each task is idempotent: run the playbook twice and the second run changes nothing, because Ansible only acts when reality differs from the declared state. That is what makes it safe to re-run after every change. Ansible is the right tool for OS-level configuration — packages, users, kernel/sysctl tuning, the GPU driver and Kubernetes prerequisites, directories, and node agents. The Ansible guides go deep; here it is one layer of a stack.

That stack is the mental model to hold for the rest of the lesson — four declarative layers, each handing off to the next:

  1. Terraform / OpenTofu
        provisions machines & networks
              │
  2. Ansible
        configures Ubuntu on them
              │
  3. Kubernetes
        runs apps, services, secrets
              │
  4. GitOps (Argo CD)
        continuously reconciles 1–3's
        Kubernetes state back to Git

Read top to bottom, each layer is more dynamic than the last: infrastructure changes rarely, OS config occasionally, Kubernetes objects often, and GitOps runs continuously. Match the tool to the pace of change and the platform stays comprehensible.

Source Control and Repository Layout

If Git is where the desired state lives, then the shape of the repository is the shape of your platform. One repository, clearly organized, is enough for a lab-to-small-production platform. Here is a layout that mirrors the four-layer stack:

  ai-platform/
  ├── infrastructure/
  │   ├── terraform/     # or opentofu/
  │   └── ansible/
  ├── kubernetes/
  │   ├── base/          # shared objects
  │   ├── inference/     # vLLM Deployment,
  │   │                  #   Service, PVC, config
  │   ├── monitoring/    # Prometheus, Grafana
  │   └── gateway/       # Gateway, HTTPRoute,
  │                      #   cert-manager issuer
  ├── docker/            # Dockerfiles
  ├── scripts/           # helper scripts
  ├── docs/              # runbooks, arch docs
  ├── tests/             # manifest & policy tests
  └── README.md

Every artifact the platform needs to exist has a home here, and anyone who clones the repo can see the entire system at a glance. The Git strategy that keeps it trustworthy:

• Protect main. No direct pushes; changes arrive through merge requests.
• Require review. At least one other engineer approves before merge.
• Version and tag releases so you can point GitOps at a known-good commit.
• Roll back by revert — a bad change is undone by reverting the commit, which flows back out through the same pipeline.
• Never commit secrets. No API keys, no kubeconfigs, no TLS private keys, no Terraform state with credentials. This rule has no exceptions.

🛠️ DevOps Tip — The single most valuable property of this setup is that git log becomes the history of your infrastructure. “Why does the gateway rate-limit at that value? Who changed the model version? When did we add the second GPU node?” — all answerable from commits and merge requests instead of memory or Slack archaeology. That auditability is not a nice-to-have; it is the difference between operating a platform and haunting it. This lesson is not a Git tutorial — if branching and merge requests are new, shore that up separately — but the discipline above is non-negotiable for a platform.

CI/CD Pipelines

A merge to main should not require a human to remember eleven steps in the right order. A CI/CD pipeline runs them automatically on every change: continuous integration validates and builds, continuous delivery gets the result toward the cluster. This site’s ecosystem leans on GitLab CI/CD (GitHub Actions is a fine alternative with the same shape).

A pipeline for this platform, stage by stage:

# Illustrative .gitlab-ci.yml stages.
stages: [lint, validate, test, build, scan,
         push, deploy]

lint:
  stage: lint
  script:
    - yamllint kubernetes/
    - ansible-lint infrastructure/ansible/

validate:
  stage: validate
  script:
    # Catch invalid manifests before the cluster does
    - kubeconform -strict kubernetes/

build:
  stage: build
  script:
    # Immutable tag from the commit SHA — never :latest
    - docker build -t $REGISTRY/vllm:$CI_COMMIT_SHA .

scan:
  stage: scan
  script:
    # Fail the build on known-critical CVEs
    - trivy image $REGISTRY/vllm:$CI_COMMIT_SHA

Walk the stages: lint catches formatting and obvious mistakes; validate runs kubeconform (or helm template) so a malformed manifest fails in CI, not in production; test runs whatever policy/unit checks you have; build produces the image; scan runs Trivy to fail on known-critical vulnerabilities; push sends the image to the registry; deploy updates the desired state (or, with GitOps, that last step is just a commit). Each stage is a gate — a red stage stops the change from advancing.

The image build itself carries production discipline:

• Pin the base image to a specific digest or tag, never a moving one.
• Tag the output immutably — the commit SHA above, not :latest, so every running Pod maps to an exact build.
• Push to a trusted registry — GitLab or GitHub Container Registry, not a random public image.
• Scan every image and treat critical findings as build failures.
• Understand SBOM (a Software Bill of Materials — an inventory of what’s inside the image) and image signing as the next maturity step: they let you prove what you shipped and that it wasn’t tampered with.

⚠️ Warning — :latest is the enemy of reproducibility. A Pod that pulls myimage:latest can run different code on every restart depending on what latest pointed to at pull time — which means an incident you cannot reproduce and a rollback you cannot trust. Tag images immutably (commit SHA or a semantic version), reference that exact tag in your manifests, and let the registry keep the history. “Which build is in production?” must always have a precise answer.

GitOps: Git as the Source of Truth

CI builds and validates. But how does the validated result land in the cluster, and how do you stop the cluster from drifting away from what Git says? The answer is GitOps: a controller running inside the cluster continuously compares the live state to a Git repository and reconciles any difference. Git is the source of truth; the cluster is a reflection of it.

Pick one GitOps controller — this lesson uses Argo CD (Flux is the common alternative; don’t run both). The model:

  Git repo (desired K8s state)
        │  Argo CD watches it
        ▼
  Argo CD controller (in-cluster)
        │  diffs desired vs live
        ▼
  Kubernetes (live state)
        │
   drift?  → Argo CD re-applies Git
   in sync → nothing to do

An Argo CD Application points at a path in your repo and a target cluster/namespace:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: inference
  namespace: argocd
spec:
  project: default
  source:
    repoURL: <your-git-repo-url>
    path: kubernetes/inference
    targetRevision: main
  destination:
    server: https://kubernetes.default.svc
    namespace: inference

source says what Git to watch and where; destination says which cluster and namespace to reconcile. Install Argo CD per its current documentation rather than a hard-coded manifest URL — the install path and version move. The benefits are exactly the production properties this lesson is chasing:

• Auditability — every change is a reviewed Git commit.
• Drift correction — someone kubectl edits the cluster by hand? Argo CD flags it OutOfSync and can revert it to Git.
• Rollback — revert the commit and the controller rolls the cluster back.
• Review — production changes go through merge request, not live typing.

Keep the environment split simple: a lab namespace/branch and a production namespace/branch is plenty. Don’t overengineer a five-environment promotion pipeline for a platform this size. And pick one templating convention — Helm or Kustomize, per your repo’s convention — not both layered together.

Managing Secrets

Your platform has secrets: the vLLM API key, TLS private keys, registry credentials, maybe a database password. The repository layout above forbids committing them — so where do they go? Pick one manageable pattern and apply it consistently. The choices, roughly in order of “how much infrastructure”:

• Kubernetes Secrets + encryption at rest — the cluster operator enables etcd encryption so Secrets aren’t stored in plaintext. Simplest; the Secret still can’t live in Git.
• Sealed Secrets — you encrypt a Secret into a SealedSecret custom resource that is safe to commit; a controller in the cluster decrypts it. This fits GitOps beautifully.
• External Secrets Operator — the cluster pulls secrets from an external store at runtime.
• Vault — a dedicated secrets manager for larger setups.

For a GitOps platform, Sealed Secrets is the clean lab choice: the encrypted form goes in Git, the plaintext never does. Whatever you pick, remember the rule from Part 7:

⚠️ Warning — A Kubernetes Secret is base64-encoded, not encrypted. kubectl get secret -o yaml | base64 -d reveals it instantly. base64 is encoding — reversible by anyone with read access. A Secret is the right place for a credential (RBAC-restrictable, mountable, encryptable at rest by the operator), but “it’s in a Secret” is not confidentiality on its own. Never commit a raw Secret manifest to Git; use one of the patterns above, and cover the full lifecycle — generation, rotation, revocation, storage, and audit — not just the first kubectl create. The Vault guides go deeper when you outgrow the simple options.

Access Control: RBAC, NetworkPolicy, and Firewall

Security on this platform is layered — identity inside Kubernetes, traffic between Pods, and packets at the host. Each layer defaults to least privilege.

RBAC (Role-Based Access Control) governs who and what can do things in the cluster. A Role (namespaced) or ClusterRole (cluster-wide) lists allowed verbs on resources; a RoleBinding grants that Role to a user or a ServiceAccount. Give each workload its own ServiceAccount with only the permissions it needs:

# The inference workload's identity — read-only,
# namespaced, nothing more.
apiVersion: v1
kind: ServiceAccount
metadata:
  name: inference
  namespace: inference
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: inference
  name: inference-read
rules:
  - apiGroups: [""]
    resources: [configmaps, secrets]
    verbs: [get, list]

Separate ServiceAccounts for the inference, monitoring, and deploy identities keep a compromise contained. The cardinal rule:

⚠️ Warning — Do not hand out cluster-admin. A cluster-admin binding owns every node, every Secret, and every GPU in the cluster — one leaked kubeconfig and the whole platform is gone. Scope permissions to the smallest set a user or workload provably needs, prefer namespaced Roles over ClusterRoles, and audit who holds broad access. Least privilege is not paranoia; it is the blast-radius control that decides how bad a bad day gets.

NetworkPolicy controls Pod-to-Pod traffic (it requires a CNI that enforces it, like Calico or Cilium from Part 7). By default Kubernetes lets any Pod talk to any Pod — you want the gateway to reach inference, and nothing random to reach the backend:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-gateway-to-inference
  namespace: inference
spec:
  podSelector:
    matchLabels: { app: vllm }
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels: { role: gateway }
      ports:
        - port: 8000

This says only Pods in the gateway namespace may reach vllm on port 8000; everything else is denied. That is the “allow gateway → inference, deny random Pod → backend” rule expressed as policy.

The host firewall (UFW) is the outermost layer. The Kubernetes control-plane API, the kubelet, SSH, the monitoring stack, and the gateway’s control ports must not be globally reachable from the internet:

sudo ufw default deny incoming
sudo ufw allow 22/tcp          # SSH from trusted nets only
sudo ufw enable
sudo ufw status verbose

ufw status verbose shows exactly what is open — review it and confirm the Kubernetes API port, monitoring ports, and gateway admin ports are not exposed to the world. Only the public HTTPS listener should be reachable externally; everything else stays on the internal network. The security-hardening guides and Linux administration guides go deeper on host hardening.

Supply Chain and Model Provenance

You already scan and pin your images in CI. Production goes one step further: treat everything you pull in as a supply-chain dependency you must vouch for — and that includes the model.

For container images: use trusted base images, scan them, pin them to digests, and adopt signing and SBOMs so you can prove what you shipped and that no one swapped it. That is the software supply chain, and it is now table stakes.

The part teams routinely miss is the model supply chain. A model file is a dependency exactly like a library — but far larger and, unlike code, opaque. Treat it accordingly:

  For every model in production, record:
  ------------------------------------------
  Source     where it came from (registry)
  License    are you allowed to use it?
  Checksum   verify the download integrity
  Version    exact revision, pinned
  Change     reviewed & approved rollout

🤖 AI Infrastructure Tip — Do not pull an arbitrary model straight into production. “Someone found a new model on a hub and pointed the Deployment at it” is how you ship an unlicensed, unverified, or subtly broken artifact to users. Fetch models from a source you trust, verify the checksum, pin the exact revision, and put model changes through the same review as code changes. A model is not configuration you tweak — it is a dependency you govern.

Model versioning follows from that. Run v1 in production and test v2 in staging before any switch, because a model change is not neutral — a new version can change answer quality, VRAM footprint, latency, and throughput all at once. A model that needs more VRAM than the last one can suddenly fail to fit the GPU; a slower model can blow your latency budget. Roll new models out in a controlled way (canary or blue/green, next section) and watch the Part 8 metrics as you do.

Deployment Strategies

When you ship a new image or model, how the swap happens decides whether users notice. Kubernetes Deployments default to a rolling update: new Pods come up, old Pods go down, gradually. For a stateless web app that’s seamless. For a GPU workload there is a catch that trips up everyone the first time.

  Rolling update wants to "surge":
  bring up a NEW Pod before killing OLD.

  2 GPU nodes · 2 replicas · want +1 surge
  ┌──────────┐  ┌──────────┐
  │ ai-node01│  │ ai-node02│
  │ GPU: used│  │ GPU: used│
  └──────────┘  └──────────┘
  surge Pod needs a 3rd GPU → none free
        → surge Pod Pending → rollout stalls

A surge Pod needs a GPU to schedule onto. With every GPU already reserved (whole-GPU scheduling from Part 7), the extra Pod sits Pending and the rollout stalls. You have two ways out: keep a spare GPU for the surge, or set the Deployment strategy to maxSurge: 0, maxUnavailable: 1 so it takes an old Pod down first to free a GPU — accepting a brief capacity dip instead of stalling.

Blue/green and canary are the more controlled strategies, and both are GPU-cost-aware: blue/green runs two full copies (double the GPUs, briefly), canary sends a slice of traffic to the new version (needs enough GPU for both). They cost accelerator capacity you must actually have. A PodDisruptionBudget protects availability during voluntary disruptions like drains:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: vllm-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels: { app: vllm }

❗ Important — A PodDisruptionBudget guarantees at most how many Pods go down during maintenance — it does not create capacity. minAvailable: 1 with a single GPU means a drain that would take that Pod down is simply blocked until you provide somewhere else for it to run. On a one-GPU cluster, a PDB can make node maintenance impossible rather than safe. Budgets protect availability; they don’t manufacture GPUs. Size them against the capacity you actually have.

Node Maintenance and GPU Driver Updates

Nodes need patching, kernels need updating, and GPU drivers need upgrading — on a live platform, without yanking a running model out from under users. Kubernetes gives you three verified commands to do it gracefully.

kubectl cordon ai-node01     # stop NEW pods landing here
kubectl drain ai-node01 \
  --ignore-daemonsets \
  --delete-emptydir-data     # evict existing pods
# ... do maintenance ...
kubectl uncordon ai-node01   # allow scheduling again

cordon marks the node unschedulable so nothing new lands on it. drain evicts the Pods that are already there (respecting your PodDisruptionBudget) so they reschedule elsewhere — which only works if another compatible GPU exists. uncordon returns the node to service. Between drain and uncordon, the node is yours to work on safely.

The GPU driver / kernel update workflow strings these together with the host-side procedure from Part 3:

  1. kubectl drain ai-node01
  2. update GPU driver / kernel
       (the Part 3 method, on the host)
  3. reboot the node
  4. validate GPU:  nvidia-smi
  5. validate K8s:  nvidia.com/gpu advertised
       (kubectl describe node ai-node01)
  6. test a GPU workload on the node
  7. kubectl uncordon ai-node01

Every step is a gate. After the reboot, nvidia-smi must show the GPU with the new driver on the host before you trust Kubernetes; then kubectl describe node must show nvidia.com/gpu advertised again; then a test Pod must actually use it. Only then do you uncordon. Do one node at a time so the service stays up on the others.

⚠️ Warning — GPU drivers and kernels are coupled. A kernel update can leave the previously-working NVIDIA driver unable to load against the new kernel, and the node comes back with no GPU — nvidia-smi fails, the device plugin advertises nothing, and every GPU Pod that reschedules there goes Pending. This is exactly why the workflow validates nvidia-smi before uncordoning: never return a node to the scheduler until you’ve proven its GPU works with the running kernel. Plan reboots into a maintenance window, and keep a way to roll the driver/kernel back.

Ordinary Ubuntu security patching rides the same rails: sudo apt update && sudo apt upgrade for security and kernel updates, planned as a drain → patch → reboot → validate → uncordon cycle rather than an unscheduled reboot that dumps a running model on the floor.

High Availability and Failure Domains

Now the honest conversation about high availability (HA) — because the lab you built is not enterprise-HA, and pretending otherwise is how people get hurt.

HA means removing single points of failure at each layer. Your lab has an obvious one:

  Lab today                  Production HA
  --------------------       --------------------
  1 control-plane node   →   3+ control-plane
   = SINGLE POINT             (stacked etcd,
     OF FAILURE                load-balanced API)
  1 GPU worker           →   multiple GPU workers
  local storage          →   replicated storage

A single control-plane node is a single point of failure: lose it and the cluster is unmanageable (existing Pods may keep running, but nothing can be scheduled, healed, or changed). Production Kubernetes runs an HA control plane — three or more control-plane nodes with a load-balanced API server and a resilient etcd quorum. That’s a real build, out of scope for this lab, but you must know your lab doesn’t have it.

GPU failure has a hard limit no amount of Kubernetes cleverness overcomes:

❗ Important — When a GPU node fails, Kubernetes reschedules its Pods only if another node with a compatible free GPU exists. If none does, the Pods go Pending and stay there — Kubernetes cannot create a GPU it doesn’t have. Redundancy for GPU workloads means spare GPU capacity, not just spare Pods. Two replicas on one GPU is not HA; it is one working Pod and one Pending Pod. Real GPU HA costs real accelerators.

Storage has its own tradeoff. Local NVMe is fastest and simplest but pins a Pod to one node (node affinity) and dies with that node. Replicated or network storage (Ceph, NFS, cloud volumes) survives a node loss and lets any node pull the model, at the cost of latency and complexity. There is no free lunch: fast-local for a single node, replicated-network when you need survivability.

Backups and Disaster Recovery

The final question a platform must answer: if it all burned down, could you rebuild it? That’s disaster recovery, and it starts with knowing what to back up.

  Back up (small, precious):
  ------------------------------------
  ✓ Git repos (the whole platform)
  ✓ Terraform / OpenTofu STATE
  ✓ Kubernetes manifests
  ✓ Secrets (via a proper secret backup)
  ✓ Model config (name, version, args)
  ✓ Persistent data you can't refetch
  ✓ Grafana dashboards
  ✓ Prometheus alert rules

  Maybe skip (large, refetchable):
  ------------------------------------
  ~ Model WEIGHTS — only if reliably
    re-fetchable from a trusted registry

Most of that is tiny and belongs in Git already. Two items deserve care: Terraform/OpenTofu state (lose it and IaC no longer knows what it manages) and secrets (back them up through your secret tooling, never as plaintext). Model weights are the interesting call — they’re huge, and if you can reliably re-fetch the exact pinned version from a trusted registry, you may skip backing them up and re-download on rebuild. The tradeoff is rebuild time versus storage cost: skipping weights makes disaster recovery slower but cheaper, and only works if that registry and version are genuinely dependable.

For the cluster and its persistent volumes, Velero is the standard tool for backing up and restoring Kubernetes objects and PVs. Treat it as optional and verify its current support for your setup rather than assuming — but know it exists when a lab grows up.

The DR question stated plainly:

🔍 Troubleshooting — Can you rebuild from scratch? Problem: the platform is gone — hardware, cluster, all of it. Check: do you have, in backups, all of: Git repos + IaC code + config-management (Ansible) + Kubernetes manifests + a reachable image registry + a secrets backup + the model source? Fix: provision with IaC → configure with Ansible → stand up Kubernetes → GitOps reconciles the manifests → pull images from the registry → restore secrets → re-fetch the pinned model. Validate: the rebuilt platform serves the same API and passes the readiness checks. If any input above is missing, you can’t fully rebuild — fix that gap now, not during the disaster.

RTO and RPO are the two numbers that frame recovery, kept beginner-friendly: RTO (Recovery Time Objective) is how fast must you be back; RPO (Recovery Point Objective) is how much data can you afford to lose. You set the targets based on what the service is worth — this lesson won’t invent numbers for you, but you should be able to state both for your platform.

Production Monitoring, Alerting, and Runbooks

Part 8 already gave you the observability stack — Prometheus scraping host, GPU, Kubernetes, and inference metrics; Grafana dashboards; DCGM GPU telemetry. Production operationalizes it: alerts that page a human, runbooks that tell that human what to do, and SLOs that define “good.”

The alert categories a production AI platform actually needs — each actionable, meaning a human can and must do something when it fires:

  Alert on (actionable):
  ------------------------------------
  Service down / no healthy replicas
  Error rate elevated
  Latency over budget
  GPU worker missing / node NotReady
  GPU hardware error (DCGM)
  Disk filling (host or model cache)
  Pod restart loops (CrashLoop)
  TLS certificate nearing expiry
  Backup job failed

🤖 AI Infrastructure Tip — The failure mode of alerting is noise, not silence. An alert that fires on every transient blip, or on “GPU utilization is high” (which is the GPU doing its job), trains everyone to ignore alerts — and then the real one gets ignored too. Alert on symptoms users feel (service down, errors, latency) and on conditions that will cause an outage if unattended (disk filling, cert expiring, backups failing), not on raw resource numbers. Every alert should be something a human needs to act on; if no action follows, it’s a dashboard panel, not an alert.

Each alert needs an on-call owner and an escalation path, and each needs a runbook — a short document that turns a 3 a.m. page into a checklist. A workable runbook structure:

  RUNBOOK: <alert name>
  ------------------------------------
  Alert:      what fired, and its severity
  Impact:     what users experience
  Checks:     commands to run to diagnose
  Recovery:   steps to fix it
  Validation: how to confirm it's resolved

Store runbooks in the repo’s docs/ so they’re versioned alongside the platform. SLOs (Service Level Objectives) — targets for availability, latency, and error rate — are how you decide when an alert matters. They are workload-specific: a batch pipeline and an interactive chat API have very different acceptable latencies, so you set the targets from your Part 8 measurements. This lesson will not fabricate an SLO number; a made-up “99.9%” is worse than an honest one you derived.

Capacity Planning: When Do You Need Another GPU?

At some point the platform feels slow and the reflex is “buy a GPU.” Resist it until the evidence says so. Capacity planning is diagnosis before purchase.

Buy another GPU when you observe, sustained and correlated in your metrics:

  Evidence you need more GPU:
  ------------------------------------
  ✓ Queue depth growing over time
  ✓ GPU saturated for sustained periods
  ✓ SLO violations under normal load
  ✓ VRAM too small for the model you need
  ✓ Replicas can't schedule (no free GPU)
  ✓ Real, expected traffic growth

  NOT evidence:
  ------------------------------------
  ✗ "utilization hit 80% once"

A brief utilization spike is a GPU working, not a GPU overwhelmed. Before spending money, investigate the cheaper causes: is request volume actually up, or is the model architecture heavier than needed? Would batching help? Is a CPU bottleneck or memory-bandwidth limit starving the GPU (the Part 8 correlation skill)? Is it a scheduling problem — a Pod Pending because of a taint or a request, not because the card is full? Often the fix is configuration, not capital.

When you do expand, the cost is a curve, not a sticker price: hardware, cloud instance hours, power, cooling, storage, network, and the operational time to run it. This lesson won’t quote prices — they move and vary — but plan for the whole curve, not just the card.

Plan Capacity Before Buying More GPUs

The measure-first discipline above is also the honest buying advice: don’t add accelerators to a lab to learn — you learned everything in this series on one GPU. Add them when your own metrics prove sustained demand the current hardware can’t meet. When that day comes, the Example Hardware Expansion Options below range from a capable single workstation card up to a dedicated AI system, sized to a node’s role rather than a shopping list. And because most “we need more GPU” problems are really efficiency problems, the performance-engineering book in the recommended reading below is worth more than another card — it teaches you to find the batching, memory-bandwidth, and scheduling wins first.

Cloud vs On-Prem and OpenStack

Where the platform runs is its own decision, and neither answer is universally right. A balanced view:

Factor	On-prem GPUs	Cloud GPUs
Upfront cost	High (buy hardware)	Low (pay as you go)
Ongoing cost	Power, cooling, ops	Hourly, can add up fast
Control	Full (hardware, data)	Provider-bounded
Scaling speed	Slow (procure, rack)	Fast (API call)
GPU availability	You own it	Subject to supply
Best for	Steady, heavy load	Bursty, variable load
Data locality	On your premises	In the provider

Many real platforms are hybrid: steady baseline on-prem, burst to cloud. Choose by your load shape and constraints, not hype.

OpenStack deserves a mention because it’s how many organizations run private-cloud GPU infrastructure — an API-driven substrate you own. Conceptually it slots right under everything in this lesson:

  OpenStack (private cloud)
        │  provisions
        ▼
  GPU Nova instances
    - PCI passthrough or vGPU
    - GPU flavors
    - host aggregates / AZs
        │  running
        ▼
  Ubuntu 26.04
        │
  Kubernetes → vLLM → your AI platform

OpenStack Nova can hand out GPU-backed instances via PCI passthrough (a whole physical GPU into a VM) or vGPU (a sliced virtual GPU), sized by flavors, placed with host aggregates and availability zones. From there it’s Ubuntu, then Kubernetes, then the same platform you’ve built — IaC (Terraform/OpenTofu) drives the OpenStack API just like any other cloud. This lesson won’t write a full passthrough guide; the OpenStack guides, the managed OpenStack offering, and the OpenStack troubleshooting hub cover the details.

The Production Readiness Checklist

Before you call anything “production,” walk this checklist. It is the whole lesson compressed into gates.

Infrastructure

• Machines and networks defined in Terraform/OpenTofu; state backed up
• Ubuntu configured by Ansible, re-runnable and idempotent
• GPU nodes validated: driver, nvidia.com/gpu advertised

Security

• No cluster-admin handed out; per-workload ServiceAccounts and Roles
• Secrets managed by a real pattern; nothing secret in Git
• NetworkPolicies restrict traffic to what’s needed
• UFW closes the API/SSH/monitoring/gateway control ports to the world
• Images pinned, scanned, from a trusted registry; TLS on the public API

Reliability

• Health probes tuned for slow model loads (Part 7/9)
• Deployment strategy accounts for GPU capacity (surge needs a spare)
• PodDisruptionBudget sized against real capacity
• Node-maintenance and driver-update runbooks written

Monitoring

• Prometheus + Grafana + Alertmanager live (Part 8)
• Actionable alerts with owners, severities, escalation
• SLOs defined from measurement, not invention

Automation

• CI/CD pipeline: lint → validate → test → build → scan → push → deploy
• GitOps reconciling the cluster to Git; drift corrected

Documentation

• Runbooks in docs/; architecture document current
• DR procedure written and, ideally, rehearsed

Hands-On Lab: Build a Production-Style Ubuntu AI Platform

🧪 Hands-On Lab — Combine everything from Parts 1–9 into an operable platform. Nine phases, worked in order. Each phase produces a control you didn’t have as a bare cluster. Where a step reuses an earlier lesson, this is about wiring it into the platform, not rebuilding it.

Phase 1 — Infrastructure

1. Create the ai-platform Git repo with the layout above; protect main.
2. Write Terraform/OpenTofu for your machines/networks (or document the bare-metal provisioning step).
3. plan then apply; commit the code and back up the state.
4. Write an Ansible playbook for the Ubuntu prerequisites (Parts 3 and 7); run it idempotently.

Phase 2 — Kubernetes 5. Bring up the cluster (Part 7) with the GPU device plugin advertising nvidia.com/gpu. 6. Put the base Kubernetes objects under kubernetes/base in Git. 7. Apply RBAC: ServiceAccounts and namespaced Roles for inference/monitoring/deploy. 8. Apply NetworkPolicies (deny-by-default, allow gateway → inference).

Phase 3 — Model Serving 9. Deploy vLLM (Part 9) as a Deployment with a GPU limit, model-cache PVC, ConfigMap, and Secret. 10. Add startup/readiness/liveness probes tuned for the model load. 11. Confirm the ClusterIP Service reaches the model inside the cluster.

Phase 4 — Secure Access 12. Stand up the Gateway API gateway with a TLS listener (cert-manager). 13. Route an HTTPRoute to the vLLM Service; require API-key auth. 14. Add rate limiting; verify authorized (200) vs unauthorized (401) requests.

Phase 5 — Observability 15. Deploy Prometheus + Grafana + Alertmanager (Part 8). 16. Import host/GPU/Kubernetes/inference dashboards. 17. Write actionable alerts (service down, errors, latency, GPU missing, disk, cert expiry, backup failed).

Phase 6 — Automation 18. Write the GitLab CI/CD pipeline: lint → validate → test → build → scan → push. 19. Build images with immutable tags; scan with Trivy; push to a trusted registry. 20. Install Argo CD; create Applications pointing at kubernetes/; confirm sync.

Phase 7 — Reliability 21. Set the Deployment strategy for GPU capacity; add a PodDisruptionBudget. 22. Practice a node drain → (simulated) maintenance → validate GPU → uncordon. 23. Write the node-maintenance and driver-update runbooks into docs/.

Phase 8 — Performance 24. Load-test your own API with a gentle ramp; record latency, TTFT, tokens/sec, queue depth. 25. Correlate a bottleneck (GPU vs CPU vs memory-bandwidth vs scheduling) before concluding “need more GPU.” 26. Define SLOs from what you measured.

Phase 9 — Documentation & Recovery 27. Back up Git, IaC state, manifests, secrets, model config, dashboards, alert rules. 28. Write the DR runbook; if you can, rehearse a rebuild in a scratch environment. 29. Fill in the architecture document (next section) and the readiness checklist above.

✅ Validation — You have a production-style platform when all of these hold:

• The platform is defined in Git and deploys through CI/CD + GitOps — no hand-applied manifests.
• The GPU API is reachable only over authenticated HTTPS; internal ports are firewalled.
• Prometheus is scraping and at least one actionable alert has been tested end-to-end.
• A node can be drained, “maintained,” and uncordoned without taking the service down.
• You can state, from backups, exactly how you would rebuild from scratch — and what (if anything) is missing.

The Final Architecture Document

Every platform needs a one-page summary a new engineer can read on day one. Copy this into docs/ and fill it with your real (sanitized) values:

  AI PLATFORM — ARCHITECTURE SUMMARY
  ====================================
  OS .............. Ubuntu 26.04 LTS
  GPU ............. <model> + <driver>
  Orchestration ... Kubernetes <minor>
  CNI ............. <Calico/Cilium>
  Runtime ......... vLLM (OpenAI-compat)
  Gateway ......... Gateway API + <impl>
  TLS ............. cert-manager
  IaC ............. Terraform / OpenTofu
  Config mgmt ..... Ansible
  CI/CD ........... GitLab CI/CD
  GitOps .......... Argo CD
  Registry ........ <trusted registry>
  Secrets ......... <pattern chosen>
  Monitoring ...... Prometheus + Grafana
  Alerting ........ Alertmanager
  Backups ......... <what + where>
  DR .............. rebuild via IaC+GitOps
  ------------------------------------
  SLOs ............ <your targets>
  On-call ......... <owner + escalation>

Turning This Into a Portfolio Project

This platform is genuinely portfolio-worthy — very few early-career engineers have built GPU-aware Kubernetes with CI/CD, GitOps, and a real recovery story. Publish it, but publish it safely.

⚠️ Warning — Publish sanitized artifacts only. No API keys, no TLS private keys, no kubeconfigs, no real public IPs or hostnames, no secret values, no internal network layout that aids an attacker. Share the structure — the repo layout, example manifests with placeholders, the architecture document, the runbooks, the diagrams from this lesson recreated for your build. A leaked credential in a public portfolio repo is a real incident, not a hypothetical. Scrub before you push, and use a secrets scanner on the repo.

What a sanitized version of this project demonstrates to a reviewer — concepts, not a job guarantee:

• Linux — Ubuntu administration, systemd, networking, firewalling
• GPU — driver/CUDA lifecycle, GPU-aware scheduling
• Containers — image building, pinning, scanning, registries
• Kubernetes — Deployments, Services, storage, RBAC, NetworkPolicy
• AI Ops — inference serving, model supply chain, GPU capacity
• Observability — Prometheus, Grafana, actionable alerting, SLOs
• Production DevOps — IaC, config management, CI/CD, GitOps, backups, DR

That is a coherent, senior-shaped story told through one project. It shows you can operate, not just install — which is the thing production teams actually hire for.

Troubleshooting the Production Platform

Production failures span both paths — the runtime path a request takes and the delivery path a change takes. Keep both stacks in your head:

  RUNTIME stack (a request fails)
  ------------------------------------
  User → DNS → TLS → Gateway → Auth →
  Service → Pod → Runtime → GPU →
  Ubuntu → Hardware

  DEPLOY stack (a change fails)
  ------------------------------------
  Git → CI/CD → Registry → GitOps →
  Kubernetes

Isolate the failing layer, change one thing, validate, document. The matrix below covers the failures this platform actually throws.

Problem	Likely cause	Check	Fix	Validate
CI passes but deploy fails	Cluster/GitOps rejects the change	Argo CD sync status; kubectl describe the object	Correct the manifest; re-sync	Object healthy, app serves
ImagePullBackOff	Wrong tag or missing registry creds	kubectl describe pod pull error	Fix tag; add imagePullSecret	Image pulls, Pod runs
Manifest rejected	Invalid/failing admission	CI kubeconform; apiserver error	Correct schema/policy	kubeconform and apply pass
GPU node missing after reboot	Driver failed against new kernel	nvidia-smi on host	Reinstall/rollback driver (Part 3)	nvidia.com/gpu re-advertised
Driver upgrade breaks GPU	Kernel/driver mismatch	Host nvidia-smi; dmesg	Match driver to kernel; reboot	Test Pod uses GPU
No healthy replicas	Pods crashing or unready	kubectl get pods; logs	Fix root cause; readiness	Endpoints populated
Gateway 503	No ready backend behind route	Service endpoints; Pod readiness	Restore healthy Pods	200 through the gateway
TLS expires	cert-manager didn’t renew	Certificate/Order status	Fix issuer/challenge	Valid cert served
Secrets missing	Secret not applied/sealed wrong	kubectl get secret; controller logs	Re-apply via secret pattern	Pod mounts the secret
GitOps OutOfSync	Manual drift or bad commit	Argo CD diff	Revert drift or fix Git	Status Synced/Healthy
GPU Pod Pending	No free/compatible GPU	kubectl describe pod events	Free/add a GPU; fix selector	Pod schedules
Model storage unavailable	PVC unbound / node affinity	kubectl describe pvc; node	Provision storage; fix affinity	PVC Bound, model loads
Model version raises VRAM	Heavier model won’t fit	Logs for OOM; nvidia-smi VRAM	Roll back or bigger GPU	Model loads, API answers
Monitoring missing after upgrade	Scrape/config broke on upgrade	Prometheus targets page	Fix scrape/ServiceMonitor	Targets UP, dashboards fill
Backup fails	Job error / credentials / space	Backup job logs	Fix cause; re-run	Backup completes, restorable
DR rebuild differs	An input wasn’t captured	Compare rebuilt vs documented	Capture the missing artifact	Rebuild matches architecture doc
New deploy raises latency	Regression in image/model/config	Compare Part 8 latency before/after	Roll back the change	Latency back within SLO

Series Complete

You started this series at a bare Ubuntu prompt. You end it operating a self-healing, observable, version-controlled AI platform. The final build-along box:

  ┌──────────────────────────────────────────┐
  │  PRODUCTION-STYLE AI PLATFORM             │
  │                                           │
  │  Ubuntu nodes ............ [done] ✓       │
  │  GPU driver + CUDA/ROCm .. [done] ✓       │
  │  Containers .............. [done] ✓       │
  │  Local LLM ............... [done] ✓       │
  │  Kubernetes cluster ...... [done] ✓       │
  │  Monitoring stack ........ [done] ✓       │
  │  Secure inference API .... [done] ✓       │
  │  IaC + config mgmt ....... [done] ✓       │
  │  CI/CD + GitOps .......... [done] ✓       │
  │  RBAC + network + secrets  [done] ✓       │
  │  Backups + DR plan ....... [done] ✓       │
  │  Runbooks + SLOs ......... [done] ✓       │
  │                                           │
  │  Status .................. OPERABLE       │
  └──────────────────────────────────────────┘

You built an Ubuntu AI infrastructure platform. Not a demo, not a tutorial you copied — an operable system with the controls that let you run, change, and repair it. The Part 1 → Part 10 progression, seen whole:

  1  Getting started on Ubuntu 26.04
  2  Your first AI server
  3  NVIDIA GPUs + CUDA
  4  AMD GPUs + ROCm
  5  Docker for AI workloads
  6  Running local LLMs
  7  Kubernetes for AI workloads
  8  Monitoring the platform
  9  Secure inference server
  10 Production platform (this)
  --------------------------------------
  bare OS ─────────────────▶ operable
                              platform

Each lesson added one layer. This one added the operational controls that bind the layers into something you can defend the word “production” for — honestly, as a lab-grade build with a clear path to the real thing.

What You Learned

Grouped by the skills you now hold:

• Linux — Ubuntu 26.04 administration, systemd services, networking, and host firewalling for a GPU platform.
• GPU — the driver/CUDA/ROCm lifecycle, GPU-aware scheduling, and the driver–kernel coupling that maintenance must respect.
• Containers — building, pinning, scanning, and signing images, and treating a registry as a trusted supply chain.
• AI Ops — serving inference with vLLM, governing the model supply chain, versioning models, and reasoning about VRAM/latency/throughput.
• Kubernetes — Deployments, Services, storage, probes, RBAC, NetworkPolicy, PodDisruptionBudgets, and graceful node maintenance.
• Observability — Prometheus, Grafana, DCGM GPU telemetry, actionable alerting, on-call, runbooks, and SLOs built from measurement.
• Production DevOps — IaC with Terraform/OpenTofu, configuration management with Ansible, CI/CD, GitOps with Argo CD, secrets management, backups, and disaster recovery.

Above every individual tool sits the real lesson: production is a set of controls — reproducibility, reliability, security, observability, automation, and recovery — that you build deliberately on top of a working system. You can now tell whether a platform has them, and add the ones it lacks.

Next Lesson

There is no Part 11 — this series is complete. You went from a bare Ubuntu 26.04 install to an operable, GPU-accelerated AI platform, and you built every layer yourself. Return to the Ubuntu 26.04 AI Infrastructure series index any time to revisit a lesson: the GPU and ROCm foundations, Docker and local LLMs, the Kubernetes and monitoring platform, or the secure inference server.

Where you go next is a new track, not a continuation of this one. Standalone directions worth exploring as your own future projects:

• Deepen the automation with the Infrastructure as Code, Terraform, OpenTofu, and Ansible guides, and the GitLab CI/CD guides for richer pipelines.
• Harden and scale Kubernetes with the Kubernetes & Helm guides and the security-hardening guides, and manage secrets seriously with the Vault guides.
• Go further on observability through the Prometheus and Grafana guides and the observability stack walkthrough.
• Run private-cloud GPU infrastructure with the OpenStack guides, the managed OpenStack offering, and the OpenStack troubleshooting hub.

You have the platform, the controls, and the diagnostic instincts to operate it. That is the whole point of the series — and it’s yours now. Well done.