NVIDIA GPUs and CUDA on Ubuntu 26.04

In Parts 1 and 2 you built ai-node01: a clean Ubuntu 26.04 server with networking, storage, and a GPU that lspci can see. Seeing the card is not the same as using it. This lesson turns that detected GPU into a working compute device — installing the NVIDIA driver the Ubuntu way, understanding what CUDA actually is, and proving the GPU does real work with PyTorch.

What You’ll Learn

• How to identify the NVIDIA GPU in your machine from the command line
• The relationship between hardware → driver → CUDA → framework → model, and why every layer has to agree
• How to install a driver on Ubuntu 26.04 without hunting for version numbers
• How to verify driver health and confirm the GPU is actually usable
• How to read nvidia-smi — every field, not just the pretty table
• How to install the CUDA components an AI workload actually needs (which is often fewer than people think)
• The difference between the NVIDIA driver and the CUDA Toolkit
• The difference between the CUDA runtime and CUDA development tools
• How to verify CUDA is present and functional
• How to test GPU acceleration with a lightweight framework (PyTorch)
• How to troubleshoot the NVIDIA stack layer by layer
• When a reboot is required — and when it is not
• Why version compatibility between driver, CUDA, and framework matters

This is where the series stops being conceptual. By the end you will have a node that can run AI workloads, plus a mental model for diagnosing the whole stack when it breaks.

The NVIDIA Software Stack

Before installing anything, look at what you are assembling. GPU compute is a layered stack, and — exactly like the AI-infrastructure stack from Part 1 — each layer only talks to its immediate neighbors:

   AI Application
        │
  PyTorch / vLLM  (framework)
        │
   CUDA Runtime
        │
   NVIDIA Driver
        │
  Linux Kernel
        │
   NVIDIA GPU

Read it top to bottom. Your AI application calls a framework like PyTorch or an inference server like vLLM. The framework issues GPU work through the CUDA runtime. The runtime talks to the NVIDIA driver, which is the only thing on the system allowed to command the physical GPU. The driver lives partly inside the Linux kernel. And at the bottom is the GPU itself.

Sitting off to the side is the CUDA Toolkit — the developer kit, not part of the runtime path:

  ┌──────────────────────────────┐
  │        CUDA Toolkit          │
  │  nvcc (compiler)             │
  │  headers                     │
  │  libraries (cuBLAS, cuDNN…)  │
  │  developer / profiling tools │
  └──────────────────────────────┘

The Toolkit is what you use to build CUDA software; running pre-built AI software is a different job. This is the single most misunderstood thing about GPU setup, so hold onto it: not every inference server needs local CUDA compile tools. Many need only the driver, because the framework ships its own runtime.

NVIDIA GPU Detection

Start where Part 2 left off — confirming the hardware exists on the PCI bus. The GPU is a PCI Express device, so the kernel enumerates it whether or not any driver is loaded:

lspci | grep -i nvidia

lspci lists every device on the PCI bus; grep -i nvidia filters to NVIDIA (case-insensitive). Typical output looks like this:

01:00.0 VGA compatible controller: NVIDIA
  Corporation Device 2704 (rev a1)
01:00.1 Audio device: NVIDIA Corporation Device
  22bb (rev a1)

That first line is the GPU; the second is the audio device modern NVIDIA cards expose for HDMI/DisplayPort. If you see the GPU line, the hardware is physically present and the bus can see it.

To see which kernel driver is bound to the card, ask for the “kernel” details:

lspci -nnk | grep -iA3 nvidia

-nn prints numeric vendor and device IDs (useful for exact identification), -k shows the kernel driver in use, and -A3 includes three lines after each match. You are looking for a Kernel driver in use: line:

01:00.0 VGA compatible controller [0300]:
  NVIDIA Corporation Device [10de:2704] (rev a1)
  Kernel driver in use: nvidia
  Kernel modules: nvidia, nvidiafb, nouveau

Two states matter here, and confusing them causes most “my GPU doesn’t work” tickets:

• PCI-visible — lspci shows the card. The hardware and bus are fine.
• Driver-functioning — Kernel driver in use: nvidia (not nouveau, not blank). Only then can nvidia-smi and CUDA use the card.

A fresh Ubuntu install often shows Kernel driver in use: nouveau — the open-source community driver — or nothing at all. That is normal before you install NVIDIA’s driver. Use this flow to decide what to do next:

lspci sees the GPU?
   │
   ├─ no ──► check BIOS/UEFI (PCIe enabled?),
   │         reseat card / PCIe slot,
   │         or (in a VM) confirm passthrough
   │
   └─ yes ─► driver "nvidia" loaded?
               │
               ├─ no ──► install / troubleshoot
               │         the NVIDIA driver
               │
               └─ yes ─► run nvidia-smi

If lspci cannot see the GPU at all, no software fix will help — the problem is physical, firmware, or (inside a virtual machine) a passthrough that was never configured. We cover the VM case later.

How NVIDIA Drivers Work

The “NVIDIA driver” is not one file. It is a set of cooperating pieces:

• A kernel module — code loaded into the running Linux kernel that speaks the GPU’s low-level protocol. This is the privileged part; nothing else touches the hardware directly.
• User-space libraries — shared libraries (like the NVIDIA implementation of CUDA’s driver API) that applications link against to reach the kernel module.
• Device files — special files under /dev (for example /dev/nvidia0, /dev/nvidiactl) that user-space programs open to send work to the GPU.
• Utilities — command-line tools that ship with the driver, most importantly nvidia-smi.

The kernel side is actually several modules, each with a job. You do not need to memorize them, but recognizing them in lsmod output helps when diagnosing:

• nvidia — the core module; the heart of GPU access.
• nvidia_uvm — Unified Virtual Memory, used heavily by CUDA for managed memory. If this is missing, CUDA programs often fail even though nvidia-smi works.
• nvidia_drm — integrates with the kernel’s display (DRM) subsystem.
• nvidia_modeset — handles display mode setting.

Because the kernel module is compiled against a specific kernel, there is a hard rule you will meet in the troubleshooting section: when the kernel changes, the module has to match it again. Ubuntu’s packaged drivers handle this automatically, but a kernel update followed by a boot into a new kernel is the classic moment a working GPU suddenly “disappears.”

Installing the NVIDIA Driver on Ubuntu 26.04

Ubuntu ships a tool built for exactly this job: ubuntu-drivers. It inspects your hardware, knows which driver versions are packaged and signed for your system, and picks the recommended one. You do not hard-code a version, and you never download a raw .run installer.

First, see what is available for your card:

sudo ubuntu-drivers list

For a headless compute/AI node, list the general-purpose GPU (compute) options instead, which include NVIDIA’s Enterprise/Data Center “server” branches:

sudo ubuntu-drivers list --gpgpu

The output is a list of candidate driver packages; one is marked recommended. Now install the recommended driver:

sudo ubuntu-drivers install

For a server whose only job is compute — no desktop attached — prefer the compute-oriented install, which selects an appropriate -server (Enterprise Ready Driver) branch:

sudo ubuntu-drivers install --gpgpu

ubuntu-drivers resolves and installs the packages, including the matching kernel module. When it finishes, reboot so the module loads cleanly and any lingering nouveau driver is fully released:

sudo reboot

🛠️ DevOps Tip — On a headless AI node, choose the ERD -server branch (via --gpgpu). These “Enterprise Ready Driver” branches are tuned for datacenter/compute use and get a longer, more stable support lifecycle than the desktop branch — you get fewer surprise version jumps and better alignment with long-running inference services. The desktop branch optimizes for gaming and display features you do not need on a server.

Two things to internalize: the version you get is whatever ubuntu-drivers recommends today (so any version in this guide is an example), and you reboot after driver install and after any kernel update, because the module must bind to the running kernel.

Validating the Driver with nvidia-smi

After the reboot, the single most important verification command is nvidia-smi (NVIDIA System Management Interface). It ships with the driver, so if it runs at all, the driver is loaded:

nvidia-smi

Example output:

+-----------------------------------------------------+
| NVIDIA-SMI 5xx.xx   Driver: 5xx.xx  CUDA: 12.x     |
|-------------------------------+---------------------|
| GPU  Name         | Temp  Pwr | Memory        Util |
|   0  NVIDIA ...    |  41C  25W | 512/16384MiB   0%  |
+-------------------------------+---------------------+
| Processes:                                          |
|  GPU   PID   Process name              GPU Memory   |
|   0    1234  python                      480MiB     |
+-----------------------------------------------------+

Read the fields deliberately:

• Driver version — the installed NVIDIA driver. This is the number that must match your hardware and kernel.
• CUDA version — the maximum CUDA version this driver can support. See the tip below; this is the field everyone misreads.
• Name — the detected GPU model. Confirms the card matches what you expect.
• Temp / Pwr — current temperature and power draw. Useful for spotting thermal or power problems under load.
• Memory — VRAM used / total (here 512/16384MiB). Your primary capacity metric — an entire section below is devoted to it.
• Util — GPU compute utilization as a percentage. 0% at idle is normal.
• Processes — which processes currently hold GPU memory. Invaluable when a crashed job is still pinning VRAM.

🛠️ DevOps Tip — The CUDA version in nvidia-smi is the highest CUDA the driver can support — it is not proof that the CUDA Toolkit or nvcc is installed. A brand-new node with only the driver will happily show CUDA: 12.x here while nvcc --version returns “command not found.” Do not use this field to conclude the Toolkit is present.

What Is CUDA?

CUDA (Compute Unified Device Architecture) is NVIDIA’s platform for running general-purpose computation on the GPU. A GPU has thousands of small cores built for doing the same operation across huge amounts of data at once — exactly the math (large matrix multiplies) that neural networks are made of. CUDA is how software reaches that parallel hardware.

Conceptually, the call path is:

  Application
      │
  Framework (PyTorch, TensorFlow)
      │
  CUDA runtime
      │
  NVIDIA driver
      │
     GPU

CUDA is an umbrella over several components. You will hear these names constantly, so know what each is for:

• CUDA runtime — the libraries that let a program launch and manage GPU work at run time. This is what a running framework needs.
• CUDA libraries — pre-optimized building blocks. cuBLAS (linear algebra), cuDNN (deep-learning primitives), and NCCL (collective communication for multi-GPU) are the big three for AI.
• CUDA Toolkit — the full developer kit: compiler, headers, libraries, and profiling tools, used to build CUDA programs.
• nvcc — the CUDA compiler inside the Toolkit. Only needed if you compile GPU code yourself.

Here is the mindset shift for a DevOps engineer: you almost never write CUDA kernels. Your job is deployment and operations — making sure the right driver, the right CUDA runtime, and the right framework versions are present and agree; that services can access the GPU; and that the whole thing is monitored. You care about compatibility and dependencies, not about writing __global__ functions.

CUDA Toolkit vs NVIDIA Driver

These two are constantly conflated. They are different things with different jobs:

Aspect	NVIDIA Driver	CUDA Toolkit
What it does	Talks to the GPU	Develop + run CUDA software
Where it lives	Kernel module + user-space libs	Compiler, libraries, dev tools
Required for?	Always — no GPU access without it	Depends on the workload
Key command	Provides nvidia-smi	May provide nvcc
Must match…	Your GPU hardware + kernel	Your application / framework build

The operational impact: the driver is non-negotiable — install it once, keep it patched, reboot after kernel changes. The Toolkit is conditional — you add it only when a workload actually needs system CUDA (usually because it compiles GPU code). Confusing the two leads people to install a heavyweight Toolkit they never use, or to skip the driver and wonder why nothing works.

Installing CUDA on Ubuntu 26.04

Now apply the nuance. The right amount of CUDA to install depends entirely on your workload:

For PyTorch (and most inference), you usually need only the driver. The PyTorch pip wheel bundles its own CUDA runtime libraries. When you install PyTorch with a CUDA build (next sections), you get the runtime it needs inside the wheel. The system-level CUDA Toolkit is not part of that path. So on a node whose job is “serve models with PyTorch/vLLM,” a correctly installed driver is frequently the whole story — you can skip this section’s install steps entirely.

If you genuinely need nvcc or system CUDA — because you compile CUDA extensions, build a project from source, or a tool explicitly requires the Toolkit — you have two clean paths. Pick one; do not mix them.

1. Ubuntu’s packaged Toolkit (simplest, clean on 26.04):

   sudo apt install nvidia-cuda-toolkit

   This installs nvcc, the CUDA headers, and the core CUDA libraries from Ubuntu’s own repository. It integrates with the system package manager and puts nvcc on your PATH automatically — no manual environment variables needed. The version tracks Ubuntu’s repo, which may trail the very newest CUDA release.

2. NVIDIA’s CUDA repository (when you need the newest Toolkit). Do this by following NVIDIA’s official, current instructions rather than a copy-pasted .deb URL that may not exist for 26.04:

   Follow the NVIDIA CUDA Installation Guide for Linux at https://docs.nvidia.com/cuda/cuda-installation-guide-linux/ and select your distribution. It gives the exact, current repository setup for your Ubuntu release.

   This path can require adding NVIDIA’s CUDA directory to your PATH (the installer documents whether it does). Only add PATH/LD_LIBRARY_PATH entries if the method you followed tells you to — do not paste environment-variable exports “just in case.” Cargo-culted LD_LIBRARY_PATH lines cause more breakage than they fix.

Whichever path you take, reboot is not generally required for a Toolkit install (it is user-space) — but it is required after the earlier driver install. Confirm your choice installed what you expect in the next section.

Validating CUDA

If — and only if — you installed the Toolkit, confirm the compiler is present:

nvcc --version

Example output:

nvcc: NVIDIA (R) Cuda compiler driver
Cuda compilation tools, release 12.x, V12.x.xxx

If this prints a version, nvcc and the Toolkit are installed. If it says command not found, that is expected on a driver-only node — it is not an error, it just means you did not install the Toolkit (and probably do not need it).

To see the CUDA-related packages Ubuntu knows about:

apt list --installed 2>/dev/null | grep -i cuda

But the validation that actually matters for AI is functional, not package-level: does a framework reach the GPU? That is a two-minute PyTorch test, which needs a clean Python environment first.

A Clean Python Environment

Never install AI Python packages into the system Python. Ubuntu’s system Python is a dependency of the OS itself; pip-installing large, fast-moving packages into it can break system tools and is hard to undo. Instead, use a virtual environment (venv) — an isolated Python with its own packages that you can delete and rebuild at will.

Create a dedicated lab directory and a venv inside it:

sudo mkdir -p /opt/ai-lab
sudo chown "$USER" /opt/ai-lab
python3 -m venv /opt/ai-lab/venv
source /opt/ai-lab/venv/bin/activate

python3 -m venv builds the isolated environment; source .../activate switches your shell into it (your prompt gains a (venv) prefix). Everything you pip install now lands in /opt/ai-lab/venv, touching nothing system-wide. To leave it, run deactivate.

The full chain from OS to GPU now looks like this:

  Ubuntu 26.04
      │
  Python 3 (system)
      │
  venv (/opt/ai-lab/venv)
      │
  PyTorch (pip wheel)
      │
  CUDA runtime (bundled in wheel)
      │
     GPU (via NVIDIA driver)

Install PyTorch with a CUDA build. The install pattern is stable; the CUDA tag is not, so treat the tag shown here as an example:

pip install torch \
  --index-url https://download.pytorch.org/whl/cu126

The --index-url .../whl/cu126 part tells pip to fetch the build compiled for CUDA 12.6. Get the exact current tag (cu124, cu126, cu128, …) from the official selector at https://pytorch.org/get-started/locally/ — pick your OS, package (pip), and compute platform, and it hands you the precise command. Do not assume the tag above is current.

Now the moment of truth — a script that asks PyTorch whether it can see and use the GPU:

import torch
print(torch.__version__)
print("GPU available:", torch.cuda.is_available())
if torch.cuda.is_available():
    print("Device:", torch.cuda.get_device_name(0))
    x = torch.rand(3, 3, device="cuda")
    print((x @ x).sum().item())

Save it as /opt/ai-lab/gpu_check.py and run python /opt/ai-lab/gpu_check.py. torch.cuda.is_available() returning True, a real device name, and a printed number from the matrix multiply means the entire stack — driver, runtime, framework — works end to end. If it prints False, jump to the troubleshooting section; that specific failure is covered there.

Understanding GPU Memory

VRAM (video RAM) is the memory physically on the GPU, and it is almost always your tightest constraint. nvidia-smi reports it as total, used, and free, plus the per-process usage in the Processes table. Watch it live during a workload:

watch -n 1 nvidia-smi

Where does the VRAM go? For an AI model, three consumers dominate:

• Model weights — the parameters themselves, loaded into VRAM. A larger model, or a higher-precision copy of it, occupies more memory. This is a fixed cost the moment the model loads.
• KV cache (conceptual) — during text generation, the model caches intermediate results (“keys and values”) for tokens it has already processed so it does not recompute them. This cache grows with the context length and the number of concurrent requests. It is why a model that “fit” can still run out of memory mid-generation.
• Activations / batch working memory — scratch memory that scales with batch size (how many inputs you process at once) and sequence length.

So the practical levers are model size, batch size, and context length. Push any of them too far and you exceed VRAM.

🔍 Troubleshooting — CUDA out of memory. Problem: a job aborts with CUDA out of memory. Likely cause: the model weights + KV cache + batch working memory exceeded total VRAM — or a previous crashed process is still pinning VRAM. Check: run nvidia-smi and read the Processes table and used-memory figure. Is another process holding memory? Is used near total? Fix: if a stale process is holding VRAM, stop it. Otherwise reduce the load: smaller batch size, shorter context/max-length, a smaller or more compact model — or move to a GPU with more VRAM. Do not reach for random “clear the cache” tricks; they treat a symptom, not the cause. Validate: re-run under watch -n 1 nvidia-smi and confirm used memory now stays below total through the whole job.

GPU Utilization vs VRAM Utilization

These two numbers in nvidia-smi measure different things and routinely disagree:

• VRAM utilization — how much GPU memory is occupied.
• GPU (compute) utilization — how busy the GPU’s cores are.

A model can fill VRAM to 90% while GPU compute utilization sits near zero — the weights are resident in memory, but nothing is actively computing right now (idle between requests). The reverse also happens: heavy compute with modest memory use. High memory does not imply high compute, and low compute does not mean you have spare capacity to load another model.

Getting this distinction right is the foundation of GPU capacity planning and cost control — a full treatment belongs to the monitoring lesson later in this series (coming soon). For now, just never treat one number as a proxy for the other.

Troubleshooting the NVIDIA Stack

Every GPU problem is a broken layer. Diagnose top-down and you will find it fast:

  Linux sees the GPU?      (lspci)
      │ yes
  Kernel driver loaded?    (lsmod / nvidia-smi)
      │ yes
  CUDA sees the GPU?       (nvcc / runtime)
      │ yes
  Framework sees GPU?      (torch.cuda)
      │ yes
  Application issue        (your code / config)

Work through the common failures using Problem → Likely Cause → Check → Fix → Validate.

nvidia-smi: command not found Cause: the NVIDIA driver is not installed (the tool ships with it). Check: lspci -nnk | grep -iA3 nvidia — is the driver bound? Fix: install it with sudo ubuntu-drivers install (or --gpgpu) and reboot. Validate: nvidia-smi prints the device table.

NVIDIA-SMI has failed because it couldn't communicate with the driver Cause: the driver is installed but the kernel module is not loaded/talking — often after a kernel update, or module load failed. Check: lsmod | grep nvidia (is the module loaded?) and journalctl -b | grep -i nvidia for load errors. Fix: reboot; if it persists, reinstall the driver so the module rebuilds against the running kernel. Validate: lsmod | grep nvidia shows the modules and nvidia-smi works.

GPU appears in lspci but not in nvidia-smi Cause: hardware is present but no working NVIDIA driver is bound (often nouveau is loaded instead). Check: lspci -nnk | grep -iA3 nvidia — look at Kernel driver in use:. Fix: install the NVIDIA driver via ubuntu-drivers and reboot; it displaces nouveau. Validate: Kernel driver in use: nvidia and nvidia-smi lists the card.

Kernel module not loaded Cause: the nvidia module did not load at boot. Check: lsmod | grep nvidia; dmesg | grep -i nvidia and journalctl -b | grep -i nvidia for errors. Fix: sudo modprobe nvidia to load it manually and read any error; if it refuses, the cause is usually a kernel mismatch (below) or Secure Boot (further below). Validate: the module appears in lsmod and survives a reboot.

Driver broken after a kernel update Cause: you booted a new kernel and the module was built for the old one. Check: uname -r (running kernel) vs modinfo nvidia | grep vermagic (module’s kernel). A mismatch is the smoking gun. Fix: reinstall/rebuild the driver so DKMS compiles the module for the new kernel, then reboot. Validate: modinfo nvidia matches uname -r; nvidia-smi works.

Secure Boot blocking the module Cause: an unsigned kernel module is refused at load time. Check: mokutil --sb-state (is Secure Boot enabled?) and dmesg | grep -i -E 'nvidia|module verification' for signature errors. Fix: use Ubuntu’s signed driver (the default from ubuntu-drivers), or complete MOK enrollment if DKMS built the module. See the Secure Boot section below — do not disable Secure Boot as a reflex. Validate: the module loads and nvidia-smi works with Secure Boot still on.

A CUDA application says “no GPU found” Cause: the runtime cannot reach the driver, or nvidia_uvm is not loaded, or a version mismatch. Check: does plain nvidia-smi work? Is nvidia_uvm in lsmod | grep nvidia? Fix: if nvidia-smi fails, fix the driver first (above). If it works, ensure the CUDA runtime version the app expects matches your driver’s supported CUDA. Validate: the application enumerates the GPU.

torch.cuda.is_available() returns False Cause: usually a CPU-only PyTorch wheel was installed, or the driver is not working, or a driver/CUDA-build mismatch. Check: nvidia-smi works? python -c "import torch; print(torch.version.cuda)" — is it None? If None, you installed the CPU wheel. Fix: reinstall PyTorch from the CUDA index-url for your driver (see the PyTorch selector), inside the venv; confirm the driver works first. Validate: the gpu_check.py script prints GPU available: True and a device name.

GPU out of memory — see the dedicated Troubleshooting callout in Understanding GPU Memory.

Wrong CUDA-toolkit expectation from nvidia-smi Cause: assuming the CUDA version in nvidia-smi means the Toolkit/nvcc is installed. Check: nvcc --version — if it is “command not found,” the Toolkit is simply not installed. Fix: nothing is broken. Install the Toolkit only if the workload actually needs it (sudo apt install nvidia-cuda-toolkit); PyTorch does not. Validate: nvcc --version prints a version — if you needed it.

Conflicting driver packages Cause: a mix of driver install methods (a manual .run installer plus Ubuntu packages, or two driver branches) leaves inconsistent files. Check: dpkg -l | grep -i nvidia for overlapping packages; dmesg | grep -i nvidia for version-mismatch errors. Fix: settle on one method — Ubuntu packages. Purge the strays, then reinstall via ubuntu-drivers and reboot. Never layer a .run installer over apt-managed drivers. Validate: a single consistent driver set in dpkg -l; nvidia-smi works.

Secure Boot and Signed Modules

Secure Boot is a UEFI feature that only lets the system load kernel code carrying a trusted cryptographic signature. That is a good security property — it stops an attacker from slipping a malicious module into your kernel — but it also means an unsigned GPU module will be refused at load, and nvidia-smi will fail even though everything installed.

The symptom is a driver that installs without error yet never loads: nvidia-smi reports it cannot talk to the driver, and dmesg shows a module-verification / signature failure.

Check whether Secure Boot is on:

mokutil --sb-state

It prints SecureBoot enabled or SecureBoot disabled. Ubuntu’s default answer to this is designed in: the drivers ubuntu-drivers installs are pre-built and signed to work with Secure Boot out of the box. If instead a DKMS module is built locally (compiled on your machine), it may need MOK enrollment — you enroll a “Machine Owner Key” via a guided prompt on the next reboot so the kernel will trust your locally built module.

⚠️ Warning — Do not disable Secure Boot as your first move. It is a real defense, and the supported path (signed drivers, or MOK enrollment for DKMS modules) keeps it on. Turning it off should be a deliberate, last-resort decision, not a shortcut.

Bare Metal vs Virtual Machine GPUs

Where your Ubuntu runs changes how the GPU reaches it:

  Bare metal:
    Ubuntu ── direct ──► GPU

  Virtual machine:
    GPU
     │  PCI passthrough / vGPU
     ▼
   Hypervisor (KVM, VMware, cloud)
     │
   Ubuntu guest ──► NVIDIA driver ──► GPU

On bare metal, Ubuntu owns the hardware directly — everything in this lesson applies as written. In a virtual machine, the hypervisor sits between the GPU and your guest OS. The GPU must be explicitly handed to the guest, typically via PCI passthrough (the whole card is assigned to one VM) or vGPU (a card sliced into virtual GPUs). This is how KVM/OpenStack, VMware, and cloud GPU instances expose accelerators.

The trap to remember: seeing the GPU on the hypervisor is not the same as the guest seeing it. If passthrough or vGPU was not configured, lspci inside your Ubuntu VM simply will not show the card — which loops right back to the very first branch of the detection flow. Fix passthrough at the hypervisor layer before troubleshooting drivers inside the guest.

Hands-On Lab: Turn ai-node01 Into an NVIDIA CUDA AI Node

🧪 Hands-On Lab — Take the GPU you merely detected in Part 2 and make it a working compute device: driver, CUDA as needed, a clean Python environment, and a real GPU computation. Do these in order on ai-node01.

1. Verify Ubuntu. cat /etc/os-release and uname -r — confirm you are on Ubuntu 26.04 and note the kernel.
2. Verify PCI. lspci | grep -i nvidia — confirm the GPU is on the bus (from Part 2).
3. Identify the driver state. lspci -nnk | grep -iA3 nvidia — note whether nvidia, nouveau, or nothing is bound.
4. Install the driver. sudo ubuntu-drivers list --gpgpu, then sudo ubuntu-drivers install --gpgpu for this headless node.
5. Reboot if needed. After a driver install, reboot: sudo reboot. Reconnect over SSH.
6. Run nvidia-smi. Confirm the driver version, GPU name, and that VRAM/utilization read sanely.
7. Install CUDA components as needed. For a PyTorch node, usually skip the Toolkit. Only if you need nvcc: sudo apt install nvidia-cuda-toolkit.
8. Validate CUDA. If you installed the Toolkit, nvcc --version. Otherwise note that the driver alone is your CUDA foundation.
9. Create the venv. sudo mkdir -p /opt/ai-lab && sudo chown "$USER" /opt/ai-lab && python3 -m venv /opt/ai-lab/venv && source /opt/ai-lab/venv/bin/activate.
10. Install PyTorch. pip install torch --index-url https://download.pytorch.org/whl/<TAG> — get <TAG> from the pytorch.org selector.
11. Confirm the GPU. Run the gpu_check.py script; confirm GPU available: True and a device name.
12. Run a minimal tensor op. The script’s x @ x matrix multiply on device="cuda" printing a number is your GPU doing real work.
13. Inspect utilization. In another SSH session, run watch -n 1 nvidia-smi while a workload runs and watch memory and compute change.
14. Document the state. Record versions (next section) so you can reproduce and diagnose this node later.

  ┌────────────────────────────────────┐
  │  SUCCESS: ai-node01 is a CUDA node  │
  │                                     │
  │  Driver ............ installed ✓    │
  │  nvidia-smi ........ working  ✓     │
  │  CUDA runtime ...... present  ✓     │
  │  venv .............. /opt/ai-lab ✓  │
  │  PyTorch sees GPU .. True     ✓     │
  │  Tensor op on GPU .. ran      ✓     │
  └────────────────────────────────────┘

Capture Your Node’s Configuration

A GPU node’s exact versions are the first thing you will want when something breaks or when you rebuild. Capture them once into a plain text file (no secrets — versions only):

{
  echo "== OS =="
  cat /etc/os-release | grep PRETTY_NAME
  echo "== Kernel =="
  uname -r
  echo "== GPU =="
  lspci | grep -i nvidia
  echo "== Driver / CUDA (nvidia-smi) =="
  nvidia-smi | head -n 4
  echo "== nvcc (if installed) =="
  nvcc --version 2>/dev/null || echo "toolkit not installed"
  echo "== Python =="
  python3 --version
  echo "== PyTorch =="
  /opt/ai-lab/venv/bin/python -c \
    "import torch; print(torch.__version__, torch.version.cuda)" \
    2>/dev/null || echo "torch not installed"
} > ~/ai-node01-system-info.txt

cat ~/ai-node01-system-info.txt shows the record. When a future problem appears — a kernel update breaks the module, or a framework upgrade stops seeing the GPU — this file tells you exactly what the working baseline was, so you change one variable at a time instead of guessing.

What’s Next: Containerizing the GPU

Today you gave the host direct access to the GPU. That works, but production AI runs in containers, and containers need a bridge to the driver:

  Today:
    App ──► Host CUDA ──► GPU

  Next:
    Container ──► GPU runtime ──► Host driver ──► GPU

The NVIDIA Container Toolkit is that bridge — it lets a container reach the host’s GPU driver without bundling drivers inside the image. That is the subject of Part 5 (containerizing GPU workloads), which is coming soon and not linked yet.

NVIDIA GPUs to Explore for Your AI Lab

The recommendations below are grouped by use case — from a learning/local-AI card to a dedicated AI development system — and render beneath this lesson. They are starting points, not a shopping list. Before buying any GPU, verify it against your actual machine and workload:

• Physical dimensions — the card’s length, width, and slot height must fit your case.
• PSU capacity and connectors — enough wattage headroom, and the exact power connectors the card needs.
• PCIe slots — an available slot of the right generation, with clearance for the card’s width.
• Cooling and airflow — the case must move enough air to keep the card in its thermal range under sustained load.
• Motherboard compatibility — slot layout, BIOS/UEFI support, and (for multi-GPU) enough lanes.
• Current Linux driver support — confirm the model is supported by a current NVIDIA driver on Ubuntu 26.04 today.
• Your workload — match VRAM and compute to what you actually run; the biggest card is rarely the right answer for learning.

See the recommended GPUs below, grouped by tier, and cross-check each against the list above.

What You Learned

• How to identify your NVIDIA GPU with lspci and lspci -nnk, and how to tell PCI-visible apart from driver-functioning.
• The layered NVIDIA stack — application → framework → CUDA runtime → driver → kernel → GPU — and the separate CUDA Toolkit off to the side.
• How to install the driver the Ubuntu way with ubuntu-drivers (and why the -server/--gpgpu ERD branch fits a headless AI node), with no hard-coded versions.
• How to read every field of nvidia-smi, and the crucial fact that its CUDA number is the driver’s maximum — not proof the Toolkit is installed.
• The difference between the driver (always required) and the CUDA Toolkit (conditional), and between the CUDA runtime and dev tools.
• That PyTorch bundles its own CUDA runtime, so an inference node often needs only the driver — and how to install the Toolkit cleanly when you truly need nvcc.
• How to build an isolated /opt/ai-lab venv, install PyTorch from the correct CUDA index-url, and prove GPU acceleration with a tensor op.
• How VRAM is consumed (weights, KV cache, batch/context), why OOM happens, and why GPU utilization and VRAM utilization are different numbers.
• A layer-by-layer troubleshooting method for the whole stack, plus Secure Boot signed modules and bare-metal vs VM GPU access.

  ai-node01 build-along
  ------------------------------------
  Ubuntu ..................... [done]
  Networking ................. [done]
  Storage .................... [done]
  NVIDIA GPU (PCI) ........... [done]
  NVIDIA Driver .............. [done]
  CUDA ....................... [done]
  GPU Test ................... [done]
  ------------------------------------
  Docker ..................... [next]
  Kubernetes ................. [upcoming]
  Monitoring ................. [upcoming]

Next Lesson

AMD ROCm AI Infrastructure on Ubuntu 26.04 — the same job for AMD hardware: what ROCm is, how it differs from CUDA, how to check GPU and OS compatibility before you commit, and how PyTorch reaches AMD GPUs through the very same torch.cuda API. Continue to Part 4 →

If you would rather keep going on the NVIDIA path, the Docker Academy builds the container fundamentals Part 5 depends on, and the observability stack previews the GPU monitoring we add later. To shore up the fundamentals underneath all of this, the Linux admins guides pair well with the earlier Getting Started and first AI server lessons.