Ubuntu 26.04 for AI Infrastructure: Getting Started

AI applications get the attention, but they only run because someone built and operates the machine underneath them. This lesson is the conceptual foundation for that work: what “AI infrastructure” actually is, where Ubuntu 26.04 fits, and the vocabulary — GPUs, VRAM, CUDA, ROCm, inference — you will use for the rest of the series.

What You’ll Learn

What “AI infrastructure” means, and how it differs from the AI application on top of it
Ubuntu’s role as the operating system beneath almost every AI workload
The difference between a CPU and a GPU, and why that difference matters for AI
What GPU acceleration is and which kinds of work it speeds up
How NVIDIA and AMD compare at a high level as AI hardware vendors
What CUDA is (NVIDIA’s GPU compute platform) at a conceptual level
What ROCm is (AMD’s equivalent) and why it is not a drop-in replacement for CUDA
What VRAM is and why GPU memory is often the single biggest constraint you manage
The difference between training a model and running inference with one
What a model server is and why AI services need one
Why containers matter for reproducible, portable AI workloads
Why AI environments need DevOps engineers, not just data scientists
What this 10-part course builds: a working Ubuntu 26.04 AI server, end to end

What Is AI Infrastructure?

When people say “AI,” they usually mean the visible part: a chatbot, an image generator, a coding assistant. That is the AI application — the top layer. Everything holding it up is AI infrastructure, and that is what you, as a DevOps engineer, own.

It helps to see the whole thing as a stack. Each layer depends on the one below it:

   AI Application
        │
   API / Model Server
        │
   Inference Engine
        │
   PyTorch / Runtime
        │
   CUDA / ROCm
        │
 Linux Kernel + Drivers
        │
    GPU Hardware

Why each layer matters:

AI Application — the product the user sees. It sends prompts and receives responses. It does no math itself.
API / Model Server — accepts requests over HTTP, queues them, batches them, and returns results. This is the layer you deploy, scale, and monitor most directly.
Inference Engine — the optimized software that actually runs the model efficiently on the hardware (managing memory, batching, and throughput).
PyTorch / Runtime — the framework the model is expressed in. It translates model operations into GPU calls.
CUDA / ROCm — the vendor compute layer that lets the framework talk to the GPU. Nothing above this line reaches the hardware without it.
Linux Kernel + Drivers — the operating system and the GPU driver. The driver is what makes the card usable; the kernel schedules everything.
GPU Hardware — the physical accelerator doing the parallel math.

A failure at any layer looks like “the AI is broken” to a user. Your job is knowing which layer actually failed. A slow response might be a saturated model server, an out-of-memory GPU, a missing driver, or a cold cache — and the fix for each is different. Most of this series is about building, verifying, and diagnosing these layers one at a time.

Why Ubuntu 26.04 for AI Infrastructure?

This is not a brand preference. There are concrete technical reasons Ubuntu dominates AI infrastructure:

Linux is where servers and GPUs live. GPU compute drivers, frameworks, and inference engines target Linux first. Ubuntu is the most common Linux target for that tooling, so you hit the fewest “unsupported platform” walls.
Cloud availability. Every major cloud offers Ubuntu GPU images. Skills you build here transfer directly to a rented cloud GPU instance.
Container + GPU ecosystem. The tooling that exposes GPUs to containers (the NVIDIA Container Toolkit, covered later in the series) is built and tested on Ubuntu first.
Automation. Ubuntu works cleanly with the automation you already know — cloud-init, apt, systemd, Ansible — so a GPU node can be provisioned repeatably instead of by hand.
Security and updates. A predictable apt update path and security maintenance matter when a box has expensive hardware and network exposure.
The LTS lifecycle. Ubuntu 26.04 is an LTS (“Long Term Support”) release — codenamed “Resolute Raccoon” and released in April 2026 — with five years of standard support. AI hardware is expensive and long-lived; you do not want to rebuild the base OS every nine months.
Kubernetes compatibility. When you scale past one machine, the GPU scheduling and device-plugin ecosystem in Kubernetes is validated against Ubuntu.
AI tooling ecosystem. Most install guides, community answers, and vendor documentation assume Ubuntu. That reduces the time you spend translating instructions.

🛠️ DevOps Tip — Pick the LTS release on purpose. A non-LTS Ubuntu (“interim”) release is supported for only nine months. On a GPU server that you expect to run for years, choosing the April LTS (like 26.04) means your driver and CUDA stack stays on a supported base far longer, which is exactly what you want under expensive hardware.

AI Application vs AI Infrastructure

Two roles build an AI product, and they overlap but are not the same:

  AI Developer                AI Infrastructure Engineer
  ────────────                ──────────────────────────
  Models                      Linux
  Python                      GPUs
  Prompts                     Networking
  Applications                Storage
                              Docker
                              Kubernetes
                              Monitoring
                              Automation

The AI developer works above the model server: choosing models, writing Python, designing prompts, shipping the application. The AI infrastructure engineer works below it: standing up Linux, installing GPU drivers, wiring networking and storage, containerizing workloads, orchestrating them, and monitoring the whole thing.

In a small team one person does both. In a larger one the roles split. The two need each other — a brilliant model is useless on a box that runs out of GPU memory, and a perfectly tuned server is useless with no model to run. This series is the infrastructure side. You do not need to be a data scientist to follow it; you need to be comfortable with Linux and willing to learn the GPU-specific pieces.

CPU vs GPU

The single most important hardware idea in AI infrastructure is the difference between the two kinds of processor in the machine.

A CPU (Central Processing Unit) has a small number of very powerful, general-purpose cores. It is built to do a wide variety of tasks quickly and in the right order — running the operating system, handling control logic, making decisions, and coordinating everything else.

A GPU (Graphics Processing Unit) has a very large number of smaller, simpler units built to do the same math operation across huge amounts of data at once. That pattern — the same multiply-and-add repeated across enormous grids of numbers — is exactly what neural networks are made of.

        CPU                        GPU
   ─────────────            ─────────────────────
   few, powerful            thousands of smaller
   cores                    parallel units

   [ C ][ C ]               [.][.][.][.][.][.][.]
   [ C ][ C ]               [.][.][.][.][.][.][.]
                            [.][.][.][.][.][.][.]
   great at:                [.][.][.][.][.][.][.]
   - OS & control
   - branching logic        great at:
   - one hard task fast     - matrix math
                            - inference/training
                            - many things at once

Neither is “better.” A GPU is bad at running your operating system or complex branching logic — that is the CPU’s job. But for the dense matrix math inside a model, a GPU can be orders of magnitude faster because it does thousands of those operations simultaneously. This is what GPU acceleration means: moving the heavy parallel math off the CPU and onto hardware built for it. A well-built AI node uses the CPU to run Linux, the drivers, and the model server, and uses the GPU purely as the math accelerator.

VRAM: The Memory That Matters

A GPU has its own dedicated memory called VRAM (Video RAM). This is separate from your system RAM. When a model runs on a GPU, it lives in VRAM — not in the RAM your CPU uses. This distinction trips up almost everyone new to AI infrastructure, so it is worth stating plainly: a box with 128 GB of system RAM but a small GPU may still be unable to load a large model, because the model needs VRAM, not RAM.

VRAM gets consumed by several things at once:

Model weights — the numbers that are the model. Larger models have more weights and need more VRAM just to load.
Inference working memory — during inference the GPU holds intermediate results. For text models this includes the KV-cache (key/value cache), memory that grows as the conversation gets longer.
Context size — the more tokens of context a request carries, the more working memory it needs. Longer prompts cost more VRAM.
Batching — serving several requests at once (a batch) raises throughput but multiplies the working memory required.

As an illustrative example only: if a model’s weights occupy some amount of VRAM, then a long conversation with many concurrent users will need meaningfully more on top of that for the KV-cache and batch. The exact number depends on the model, the context length, and how many requests you batch — do not treat any single figure as a spec.

The practical takeaway: GPU memory is usually the first ceiling you hit. Not raw speed, not CPU, not disk — VRAM. Much of running an AI service well is deciding what fits in the VRAM you have and what to do when it does not. Later parts of the series cover how to observe VRAM usage and how quantization (shrinking a model’s memory footprint) buys you headroom.

🤖 AI Infrastructure Tip — When someone reports “the model crashed,” your first instinct should be GPU memory, not the application. An out-of-VRAM condition is one of the most common failure modes in AI infrastructure, and it often appears only under load — when context grows or several requests batch together — not during a quiet first test.

NVIDIA CUDA (a first look)

CUDA (Compute Unified Device Architecture) is NVIDIA’s platform for running general-purpose compute on their GPUs. It is the software layer that lets an AI framework send math to an NVIDIA card. For most of the AI world today, CUDA is the default path, which is why NVIDIA hardware is so common in this space.

Conceptually, the chain looks like this:

  Ubuntu
    │
  NVIDIA Driver
    │
  CUDA
    │
  AI Framework (e.g. PyTorch)
    │
  Model

Read it bottom-up when you diagnose: the model can only run if the framework can reach CUDA, CUDA can only work if the driver is installed and matched, and the driver only loads on a healthy Ubuntu base. Every layer depends on the one below.

You do not install any of this yet. Setting up the NVIDIA driver and the CUDA stack correctly — including using Ubuntu’s own ubuntu-drivers tool to pick a driver — is Part 3 of this series. For now, only understand what CUDA is and where it sits in the stack.

AMD ROCm (a first look)

NVIDIA is not the only option. AMD GPUs use a platform called ROCm (Radeon Open Compute) to do the same job CUDA does — expose the GPU to AI frameworks. So Ubuntu supports two hardware paths that converge at the framework:

              Ubuntu
             /      \
   NVIDIA Driver   AMD Driver
        │             │
      CUDA           ROCm
         \           /
        AI Frameworks
             │
           Model

The important caveat: ROCm is not identical to CUDA, and it is not a drop-in replacement. The two are separate ecosystems. Some tools, versions, and models support one path better than the other, and hardware support in ROCm is more selective. It is genuinely usable and improving, but do not assume that “it works on NVIDIA” automatically means “it works the same on AMD.” Full ROCm setup and its trade-offs are covered in Part 4. Here, just know both paths exist and that they are not interchangeable.

Training vs Inference

There are two very different things you can do with a model, and they have completely different infrastructure profiles.

Training is the process of creating or updating a model’s weights by showing it data. It is enormously expensive: heavy sustained compute, large amounts of VRAM and storage, big datasets, and — at serious scale — many GPUs coordinated across a network in a distributed job that can run for days or weeks.

Inference is using an already-trained model to produce an answer. A request comes in, the model runs once, an answer comes out. It is far lighter per request than training, but it must be fast, reliable, and available around the clock.

  Inference request path:

  Application
      │
     API
      │
  Inference Server
      │
    Model
      │
     GPU

This series emphasizes inference infrastructure. That is a deliberate choice. Inference is where the DevOps skill set matters most and transfers best: you are deploying a service, exposing it over an API, containerizing it, scheduling it, watching latency and GPU metrics, and keeping it up. Training infrastructure is a specialized world of its own; the everyday work of running AI in production is overwhelmingly about serving inference well.

What Does an AI Infrastructure Engineer Do?

Concretely, across this course and the job it represents, you will:

Install and harden Ubuntu 26.04 on a server
Detect and configure GPUs at the hardware level
Install GPU drivers and the CUDA or ROCm stack
Set up Docker and GPU-enabled containers
Run and schedule workloads on Kubernetes with GPU awareness
Deploy models behind a model server
Configure networking so services and nodes can talk
Provision storage for models, caches, and datasets
Monitor GPU utilization, VRAM, and inference latency
Manage secrets and access to model endpoints
Automate provisioning so nodes are reproducible
Troubleshoot failures across every layer of the stack
Secure the system and its exposed endpoints
Control resource usage so one workload cannot starve another

A lot of this is DevOps you may already recognize — with a GPU-shaped twist on every item.

Traditional DevOps vs AI Infrastructure DevOps

Traditional DevOps	AI Infrastructure DevOps
CPU	CPU + GPU
RAM	RAM + VRAM
Docker	GPU-enabled Docker
Kubernetes	GPU scheduling
App latency	Inference latency
CPU metrics	GPU metrics
Web server	Model server
Application artifact	Model artifact

If you are already comfortable with the left column, you are closer than you think. This series teaches the right column. If any of the left-column items feel shaky, the Linux Admins category and the Docker Academy are good places to firm up the fundamentals before you go deeper.

Hands-On Lab: Inspect an Ubuntu System for AI Readiness

🧪 Hands-On Lab — Before you build anything, learn to read the machine you were handed. These commands are safe, read-only, and stable on Ubuntu 26.04. They tell you what hardware Ubuntu can see. This is about hardware visibility, not drivers — whether a GPU works is a Part 3 question; whether Ubuntu can even see the card is what we check now.

Run each command and read what it tells you.

1. Confirm the OS and release.

cat /etc/os-release

This prints the distribution and version. Look for VERSION showing 26.04 and the codename “Resolute Raccoon.” This confirms you are on the LTS base the rest of the series targets.

2. Check the kernel version.

uname -r

The kernel version matters because GPU drivers are built against a specific kernel. If a later driver install misbehaves, the kernel version is one of the first things you will compare notes on.

3. Inspect the CPU.

lscpu

This shows core count, architecture, and virtualization support. On an AI node the CPU runs the OS, drivers, and model server — so you want to know how many cores you have to work with even though the GPU does the heavy math.

4. Check system memory.

free -h

The -h flag prints human-readable sizes. This is your system RAM, used by the CPU and the OS — not VRAM. Keeping the two separate in your head starts here: this number does not tell you whether a model will fit on the GPU.

5. List PCI devices to find a GPU.

lspci

lspci lists everything on the PCI bus, including graphics cards. It is long, so filter it:

lspci | grep -i -E 'vga|3d|nvidia|amd'

This narrows the output to display and 3D controllers and to NVIDIA/AMD devices. If your GPU appears here, Ubuntu can see the card at the hardware level. That is exactly what you want to confirm now. Important: seeing the card here does not mean the driver is installed or that the GPU can do compute — it only means the hardware is present and visible. Making it usable is Part 3.

6. Look at block devices and disks.

lsblk

This shows disks and partitions as a tree. Models and their caches can be large, so you need to know what storage exists before you plan where they live.

7. Check free disk space.

df -h

This shows how full each mounted filesystem is. Model downloads and container images fill disks quickly; a full disk is a mundane but very common cause of failed AI deployments.

8. Inspect network interfaces.

ip addr

This lists interfaces and their IP addresses. An AI service is only useful if clients can reach it, so knowing the node’s addresses is step one for the networking work later in the series. If networking fundamentals are new to you, the Kali Linux networking lessons are a solid primer.

⚠️ Warning — Everything in this lab is read-only and safe to run. Do not install NVIDIA drivers, CUDA, or ROCm at this stage, even if a forum post tempts you. Driver installation is deliberately sequenced into Part 3 so it happens on a clean, verified base — installing it out of order is a common way to end up with a broken graphics stack.

Your AI Lab So Far

  ┌─────────────────────────────┐
  │   Knowledge Foundation  ✓    │
  │                              │
  │  - AI infra stack            │
  │  - CPU vs GPU                │
  │  - VRAM as the constraint    │
  │  - CUDA vs ROCm (concept)    │
  │  - training vs inference     │
  │  - inspect a system          │
  └─────────────────────────────┘
              │
              ▼
     Part 2: build ai-node01
   (your first Ubuntu 26.04
        AI server)

You have the vocabulary and the mental model. Part 2 turns it into a real machine.

What You Learned

AI infrastructure is everything beneath the AI application — a layered stack from the model server down to the GPU — and DevOps engineers own it.
Ubuntu 26.04 “Resolute Raccoon” is an April 2026 LTS with a five-year lifecycle, the de-facto base for GPU drivers, containers, and cloud AI.
A CPU handles the OS and control logic with a few powerful cores; a GPU accelerates parallel matrix math with many small units.
VRAM is the GPU’s own memory, separate from system RAM, and it is usually the first constraint you hit.
CUDA (NVIDIA) and ROCm (AMD) are the vendor compute layers that connect frameworks to GPUs — and they are not interchangeable.
Training creates model weights; inference uses them — and this series focuses on inference infrastructure.
You can inspect any Ubuntu box for AI readiness with safe, read-only commands, distinguishing hardware visibility from driver installation.

Next Lesson

Next you will build the machine itself: install and prepare Ubuntu 26.04, then stand up your first AI server, ai-node01, as the foundation for everything that follows.

Continue to Part 2: Building Your First Ubuntu 26.04 AI Server →