Running Local LLMs on Ubuntu 26.04

By the end of Part 5 you had ai-node01 running Ubuntu 26.04 with a GPU driver, CUDA (or ROCm), Docker, and the NVIDIA Container Toolkit — a host that can launch GPU containers. That is the platform. This lesson puts a real workload on it: you will operate a local large language model (LLM) as a proper service, using Ollama as the vehicle to learn inference, storage, networking, GPU usage, APIs, health, security, and lifecycle.

What You’ll Learn

What a local LLM actually is, and what “local” does and does not mean
The difference between training and inference, and why Part 6 is all inference
The LLM infrastructure stack — app → API → inference engine → model → GPU
The operational vocabulary — weights, parameters, tokens, context window, quantization, throughput, latency, tokens/sec, time-to-first-token
How model size and quantization drive memory and speed — without fake VRAM numbers
How to reason about VRAM planning and CPU offload
How to choose an inference engine (Ollama, llama.cpp, vLLM) for the job
How to deploy Ollama with Docker, persist models, and expose an API safely
How to run your first inference and verify the GPU is doing the work
How to test the native API and the OpenAI-compatible endpoint
How to add an optional Open WebUI and wire a multi-container Compose stack
How to keep the service private, healthy, and firewalled
How to troubleshoot the whole path — OOM, CPU fallback, unreachable API, lost models, restart loops, and slow inference
How to capture a clean, reproducible configuration for the node

This is where ai-node01 stops being a lab bench and starts being a service you operate.

What Is a Local LLM?

A large language model is a neural network trained to predict text. A local LLM is simply one you run on infrastructure you control, instead of calling someone else’s hosted API over the internet. The model weights live on your disk, the computation happens on your GPU or CPU, and the requests never leave your network unless you decide they should.

   User
    │
   App / client
    │  (HTTP request)
   Inference server
    │  (loads + runs)
   Model weights
    │
   GPU / CPU

“Local” is about ownership and location, not about a particular machine size. A local LLM can run on:

a workstation under your desk,
a homelab server in a closet,
a rack in your own datacenter,
a private-cloud VM you administer,
an on-prem node like ai-node01.

It does not have to be a laptop, and for most real work it should not be — LLMs are memory- and compute-hungry. The point is that you own the runtime. That ownership buys you privacy, predictable cost, offline capability, and full control over versions and data — and it hands you the operational responsibility that this lesson is about.

🤖 AI Infrastructure Tip — “Local” and “secure” are not the same word. Running the model on your own hardware keeps the weights and prompts off third-party servers, but a carelessly exposed local API can be as reachable to the internet as any cloud endpoint. We treat security as its own section later — do not skip it.

Training vs Inference, Revisited

Part 1 drew this line; it matters more now that you are running something. Training is the expensive, one-time-ish process of creating a model — feeding it enormous datasets and adjusting billions of parameters over days or weeks on large GPU clusters. Inference is using an already-trained model to answer a prompt. Training builds the weights; inference reads them.

  Training  ─►  produces  ─►  Model weights
                                  │
  Inference ─►  consumes  ◄───────┘

You are not training anything in this series. You download weights that someone else trained and you serve them. That reframes every requirement: your job is fast, reliable, well-monitored inference — loading weights into memory, answering requests, and keeping the service healthy — not gradient descent. When you plan VRAM, choose an engine, or debug a slowdown, think “inference,” not “training.”

The LLM Infrastructure Stack

Like the GPU stack from Part 3, an LLM service is layered, and each layer only talks to its neighbors. Read it top to bottom:

   Application / client
        │
    HTTP API   (e.g. :11434)
        │
  Inference engine
  (Ollama / llama.cpp / vLLM)
        │
     Model (weights)
        │
      GPU / CPU
        │
     Ubuntu 26.04

Your application — a script, a web UI, another service — makes an HTTP API call. The inference engine receives it, feeds the prompt through the loaded model, and runs the math on the GPU (or CPU), all on top of Ubuntu. When something breaks, you will walk this exact ladder: is the client reaching the API? Is the engine up? Is the model loaded? Is the GPU being used? Keeping the layers straight is the whole diagnostic discipline of this lesson.

Model Terminology You Actually Need

You do not need transformer math to operate an LLM. You do need a precise operational vocabulary, because every capacity and performance decision is phrased in these terms:

Model — the trained network you serve. In practice, a set of files on disk plus metadata.
Weights — the learned numbers (parameters) that are the model. Loading a model means reading these into memory.
Parameters — the count of those weights, usually given in billions: 7B, 8B, 70B. More parameters generally means more capability and more memory.
Tokenizer — the component that splits text into tokens and back. Models think in tokens, not characters.
Token — a chunk of text (often a word-piece). Prompts and responses are measured in tokens, and so is cost/speed.
Context window — the maximum number of tokens the model can consider at once (prompt + generated output). Exceed it and older tokens fall out of view.
Prompt — the input text you send for the model to continue or answer.
Quantization — storing weights at lower numeric precision to shrink memory use (more below).
Inference — the act of running the model to produce output.
Throughput — how much work per unit time, typically tokens per second across the whole system.
Latency — how long a single request takes.
Time-to-first-token (TTFT) — how long after you send a prompt before the first output token appears. This dominates how “snappy” a chat feels.

Keep these operational. When someone asks “will this model fit?” they mean weights + context + concurrency versus your VRAM. When they ask “is it fast enough?” they mean tokens/sec and TTFT under real load.

Model Parameters and Size

Model size is quoted in parameters: 7B, 8B, 14B, 32B, 70B, and larger. As a rough rule, more parameters raise both memory and compute requirements — a 70B model is far heavier than a 7B one. That rule is useful for a first sort, and dangerous if you treat it as a formula.

Parameter count is not a VRAM specification. The memory a model actually needs depends on:

architecture — different model families use memory differently at the same parameter count,
quantization — a 4-bit copy of a model is far smaller than a 16-bit one,
context window — longer contexts grow the KV cache (below),
batching / concurrency — more simultaneous requests need more working memory,
runtime — engines allocate and offload memory differently.

🛠️ DevOps Tip — Never promise that “a 7B model needs X GB.” Two 7B models at different quantization and context settings can have very different footprints. Size the specific model, at the specific quantization and context you will run, on the specific engine — then measure. Parameter count sorts candidates; it does not spec your hardware.

Quantization

Quantization stores the model’s weights at lower numeric precision to save memory (and often speed up inference). Instead of holding each weight as a 16-bit float, you hold it as an 8-bit or 4-bit integer:

FP16 / BF16 — 16-bit floating point. Higher precision, largest memory footprint, closest to the model’s original quality.
INT8 — 8-bit integers. Roughly half the memory, usually a small quality trade.
INT4 — 4-bit integers. Smallest and often fastest to run locally, with a larger (but frequently acceptable) quality trade.

In the local-LLM world you will constantly see GGUF — the quantized-file format used by llama.cpp and Ollama. A single model is typically published in several GGUF quantizations, and you pick the one that fits your hardware and quality bar.

There is no universally best quantization. Lower precision buys you the ability to run bigger models on smaller GPUs, or to run faster, at some cost to output quality — and how much that costs depends on the model and your task. Treat quantization as a dial you tune per model, not a global setting.

VRAM Planning

VRAM is the memory on the GPU, and it is almost always your tightest constraint for local inference. You cannot compute an exact figure from first principles, but you can reason about where it goes. Think of it conceptually:

  VRAM needed  ≈
       model weights
     + runtime overhead
     + KV cache (context × concurrency)
     + batching working memory
     + other GPU memory in use

This is a conceptual budget, not a formula — the point is to know the moving parts:

Model weights — the dominant, fixed cost once the model loads. Set largely by parameter count and quantization.
Runtime overhead — the engine’s own buffers and CUDA/ROCm context.
KV cache — during generation the model caches keys/values for tokens already processed so it need not recompute them. This grows with context length and with the number of concurrent requests. It is why a model that loaded fine can still run out of memory mid-conversation.
Batching working memory — scratch space that scales with how many requests you process at once.
Other GPU memory — anything else resident: a second model, a desktop session, a stray crashed process still pinning VRAM.

The operational levers, then, are: choose a smaller or more heavily quantized model, shorten the context, or reduce concurrency. Push any of them too far past your VRAM and you get an out-of-memory error — a failure we troubleshoot in detail later. The discipline is the same as Part 3: watch VRAM live during real load and size for the peak, not the idle.

CPU Offload

If a model does not fit entirely in VRAM, many runtimes can offload part of it to system RAM and run those parts on the CPU. This trades memory for speed: you fit a bigger model than the GPU alone could hold, but the CPU-resident layers are much slower, so overall tokens/sec drop.

  Fits in VRAM:   GPU only        → fast
  Partial offload: GPU + CPU/RAM  → fits, slower
  CPU only:        CPU + RAM      → most portable, slowest

How much you can offload, and how much it costs you, is runtime-dependent — engines differ in how gracefully they split a model. Offload is a legitimate tool for fitting a model you could not otherwise run, and a common accidental cause of “why is this so slow?” when a model silently spills out of VRAM. Know which is happening before you optimize.

Choosing an Inference Engine

The inference engine is the server that loads the model and answers API calls. Three come up constantly for local work. They overlap, but they optimize for different things:

Engine	Best at	Typical use
Ollama	Simple local model management + built-in API	Your first local LLM; dev + homelab
llama.cpp	Efficient local/quantized (GGUF) inference; lightweight; CPU or GPU	Squeezing models onto modest hardware
vLLM	Higher-throughput, server-oriented serving with batching + API	Production-style multi-user serving

A few honest caveats. These are not the only engines — the space moves quickly, and there are others. Not every engine runs every model or every quantization. And “best” depends on your goal: Ollama wins on ease, llama.cpp on lean local efficiency, vLLM on throughput under concurrency. We start with Ollama because it gets a working, observable service up fastest; Part 9 (production inference, coming soon) revisits higher-throughput serving.

The Primary Tool: Ollama

For the build-along, the engine is Ollama — a self-contained runtime that pulls models, loads them, uses the GPU if one is available, and exposes an HTTP API. It is the fastest way to a real, inspectable LLM service, which is exactly why it is a good teaching vehicle.

Be clear on the framing: this is not an Ollama tutorial. Ollama is how we make the abstract concrete — inference, storage, networking, GPU usage, APIs, health, security, lifecycle. Everything you learn here (persistent model storage, private networking, health gating, OOM diagnosis) transfers directly to llama.cpp, vLLM, and whatever you run next. The tool is incidental; the operations are the point.

Ollama Deployment Options

There are two honest ways to run Ollama on Ubuntu.

  Native install:
    Ubuntu ─► Ollama (systemd service) ─► GPU

  Containerized:
    Ubuntu ─► Docker ─► Ollama container ─► GPU

Native install exists and is one command:

curl -fsSL https://ollama.com/install.sh | sh

That downloads a script and pipes it straight into your shell. It works, but piping an internet script into a shell is a trust decision — you are executing whatever that URL serves, as your user, right now. Read scripts before you run them, especially with elevated rights.

Containerized is the path we take, because Part 5 already established Docker on ai-node01 and containers give you clean isolation, reproducible deployment, easy teardown, and a natural home for a multi-service stack. Everything below uses the container path.

🛠️ DevOps Tip — Preferring the container is not a knock on the native install; it is consistency. One deployment model (Docker) across your GPU workloads means one mental model for logs, restart policy, networking, and storage. That consistency is worth more operationally than shaving a layer.

Deploy Ollama with Docker

Here is the deployment. This is the verified command — use it as written:

docker run -d --gpus=all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama

Read it piece by piece, because every flag is doing operational work:

-d runs it detached (in the background) so it keeps serving after you close the shell.
--gpus=all grants the container access to all GPUs through the NVIDIA Container Toolkit you installed in Part 5. Drop this flag for CPU-only operation; it is the single switch between GPU and CPU inference.
-v ollama:/root/.ollama mounts a named volume called ollama at the path where the runtime stores models. This is what makes model data survive the container — more on it next.
-p 11434:11434 publishes the API port. 11434 is Ollama’s default.
--name ollama gives the container a stable name so docker exec, docker logs, and Compose can find it.
ollama/ollama is the image. For AMD hardware, use ollama/ollama:rocm and add --device /dev/kfd --device /dev/dri instead of --gpus=all, matching the AMD device model from Part 5.

Confirm it started:

docker ps

You should see the ollama container up with 11434 published. If it exited immediately, jump to the logs (docker logs ollama) and the troubleshooting section.

⚠️ Warning — -p 11434:11434 publishes the API on all host interfaces. On a shared network or anything internet-reachable, that exposes an unauthenticated LLM to the world. For local-only use, publish to loopback instead — -p 127.0.0.1:11434:11434 — so only the host itself can reach it. The security section expands on this; do not skip it before exposing anything.

For long-running services you also want a restart policy, so the container comes back after a reboot or crash. Add --restart unless-stopped (we build this into the Compose file later, where it belongs).

Model Storage That Survives

The most important line above is the volume. Containers are disposable — you will recreate this one to change flags, upgrade the image, or recover from a crash. If the model data lived inside the container, every recreate would re-download multiple gigabytes. It does not, because of the named volume:

  docker rm ollama       (container gone)
        │
  volume "ollama"        (still here)
        │
  docker run … -v ollama:/root/.ollama
        │
  model still present    (no re-download)

The mental model: the container is the process; the volume is the data. Delete and recreate the container freely — as long as you remount the ollama volume, the models are still there. Inspect what Docker is managing:

docker volume ls
docker volume inspect ollama

inspect shows the volume’s on-disk location under Docker’s data root. A few operational rules follow from this:

Put it on fast, roomy storage. Models are multi-gigabyte; NVMe makes load times bearable, and you need real capacity headroom.
Do not let models fill the root filesystem. If Docker’s data root is on / and you pull several large models, you can exhaust the OS disk. Check with df -h and, if needed, relocate Docker’s data root to a larger volume.
Back up deliberately. Models are re-downloadable, so they are not precious the way a database is — but re-pulling many large models over a slow link is painful. Know where the volume lives and whether it is in your backup scope.

Selecting a Model

With storage sorted, choose what to run. Ollama pulls from its model library, and the right choice is the one that fits your hardware and task — not the biggest one you can name.

🛠️ DevOps Tip — The DevOps Rule for model selection: the best model is the one that runs reliably on the hardware you actually have. A 7B or 8B model that answers fast on your GPU is worth more in production than a 70B model that swaps to CPU, crawls, or OOMs. Start small, confirm the whole pipeline works end to end, then scale the model up only if the hardware and the results justify it.

Pick a current, hardware-appropriate model from the library at ollama.com/library — the catalog changes over time, so treat any specific model name in this lesson as an example, not a requirement. For a starter GPU, choose a small, quantized model; you can always pull a larger one once the plumbing is proven.

Model Licenses

A model is software plus data, and it ships with a license. Before you build anything commercial on a model, read its license and confirm:

whether commercial use is permitted,
whether redistribution of the weights or derivatives is allowed,
any use restrictions or attribution requirements the license imposes.

Licenses vary widely between model families, and “open weights” does not automatically mean “unrestricted.” This is not legal advice — when it matters for a business, get a real review. The operational habit is simple: check the license the same way you would check any dependency’s before shipping it.

Download a Model and Run First Inference

Pull and run a model in one step. Use an example model name here — substitute a current one you chose from the library:

docker exec -it ollama ollama run <model>

docker exec -it ollama runs a command inside the already-running ollama container, interactively (-it). ollama run <model> downloads the model if it is not present, loads it, and drops you into a prompt. The first run includes the download, so it can take a while for a large model; subsequent runs load from the volume and start much faster. Type a question, read the answer, and type /bye to exit the interactive prompt — the model stays loaded and the API stays up.

That interaction is your first inference: a prompt in, tokens out, running on your own hardware. Now confirm where the work happened.

Verify the GPU Is Doing the Work

Do not assume the GPU is being used — verify it. In a second SSH session, watch the GPU while you send a prompt:

watch -n 1 nvidia-smi

watch -n 1 re-runs nvidia-smi every second so you see live movement. On AMD, use watch -n 1 rocm-smi. While a prompt is generating, read three fields (all covered in Part 3):

Memory used — should jump when the model loads and stay elevated while it is resident.
GPU utilization — should spike above idle while tokens are being generated.
Temperature / power — should rise under load, confirming real compute.

If VRAM barely moves and utilization stays at 0% while output streams, the model is running on the CPU, not the GPU — a specific, common failure we diagnose later. Seeing memory climb and utilization spike is your proof the whole GPU path works. This is spot-checking; proper dashboards (Prometheus/Grafana) arrive in Part 8 (monitoring, coming soon), previewed in the observability stack.

Test the LLM API

The interactive prompt is convenient, but the API is what real applications use. Ollama exposes an HTTP endpoint; test it directly:

curl http://localhost:11434/api/generate -d \
  '{"model":"<model>","prompt":"Say hello","stream":false}'

This POSTs a JSON body to the native generate endpoint: model picks a loaded model, prompt is the input, and stream:false asks for one complete JSON response instead of a token-by-token stream. You get back a JSON object containing the generated text and timing metadata. The request path is exactly the stack diagram in action:

  App (curl)
      │  HTTP POST /api/generate
   API :11434
      │
  Ollama engine
      │
    Model
      │
     GPU

If curl hangs or refuses the connection, the problem is on the client→network→API leg (is the container up? is the port published? are you hitting the right host?). If it returns JSON with text, your service is genuinely usable by software, not just by a human at a prompt.

OpenAI-Compatible API

Ollama also exposes an OpenAI-compatible endpoint at /v1/chat/completions. That matters operationally: a large ecosystem of tools and SDKs already speaks the OpenAI request/response shape, so you can point many existing clients at your local server by changing a base URL — no vendor lock-in, no code rewrite. You do not need it today, but knowing it exists is what makes a local model a drop-in backend later. Part 9 (production inference, coming soon) builds on this compatibility.

Optional: Add a Web UI

Everything so far is API-first, which is the right default for infrastructure. If you want a browser chat interface for testing, Open WebUI is a common choice. This section is entirely optional — the service is complete without it.

docker run -d -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://<ollama-host>:11434 \
  -v open-webui:/app/backend/data \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:main

The UI container listens on port 8080 internally, published on host port 3000. OLLAMA_BASE_URL tells it where the Ollama API lives — replace <ollama-host> with the address Ollama is reachable at. Its own data (users, chats, settings) persists in the open-webui named volume. The request path is one hop longer:

  Browser
    │  :3000
  Open WebUI
    │  OLLAMA_BASE_URL :11434
  Ollama
    │
   Model
    │
   GPU

A UI is convenient, but remember it is another exposed surface with its own auth and its own port — the same security rules apply to it as to the API.

A Multi-Container Compose Stack

Two docker run commands with hand-typed flags do not scale. Compose (installed in Part 5) lets you declare the whole stack — Ollama plus the UI — as one file, on a shared private network. Save this as compose.yaml:

services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    restart: unless-stopped
    volumes:
      - ollama:/root/.ollama
    ports:
      - "127.0.0.1:11434:11434"
    healthcheck:
      test: ["CMD", "curl", "-f",
             "http://localhost:11434/api/tags"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 60s
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    depends_on:
      ollama:
        condition: service_healthy
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - open-webui:/app/backend/data
    ports:
      - "3000:8080"

volumes:
  ollama:
  open-webui:

The important pieces:

The GPU block under deploy.resources.reservations.devices is the verified Compose GPU syntax from Part 5: driver: nvidia, count: 1 (use count: all for every GPU), and capabilities: [gpu] — which is mandatory; omit it and deployment errors.
OLLAMA_BASE_URL=http://ollama:11434 uses service-name DNS: on the shared Compose network, ollama resolves to the Ollama container. The LLM backend never needs a published port — the UI reaches it privately.
Ollama’s port is bound to 127.0.0.1, so the API is reachable from the host and from the UI, but not from the outside network.

Bring it up:

docker compose up -d

Later, when you add monitoring in Part 8 (coming soon), Prometheus and Grafana services join this same file on the same network — do not add them now; the point today is a clean two-service stack.

Network Isolation and LLM Security

An LLM API is an unauthenticated compute service by default. If it is reachable, anyone who finds it can run arbitrary inference on your GPU, read whatever it can access, and exhaust your resources. Treat exposure as a deliberate act.

⚠️ Warning — Do Not Accidentally Publish Your Local LLM to the Internet. Ollama’s API ships with no authentication. Publishing it on 0.0.0.0 (the default -p 11434:11434) on a machine with a public IP puts an open, unauthenticated GPU service on the internet. People scan for exactly this. Bind to 127.0.0.1 for local-only use, keep the port off your public interfaces, and never assume “it’s just my homelab” means no one will find it.

The controls, in roughly the order you should apply them:

Bind address — publish to 127.0.0.1 for local-only, or to a specific private interface. This is the single highest-impact control.
Firewall — use UFW to deny the port from untrusted networks (next section).
Reverse proxy — if you must expose the service, put a proxy (nginx, Caddy, Traefik) in front to add TLS and auth rather than exposing the engine directly.
Authentication — the engine has none built in; any real auth lives in the proxy or gateway layer.
TLS — terminate HTTPS at the proxy so prompts and responses are encrypted in transit.
Access control — restrict who and what can reach the API, by network and by identity.
Resource exhaustion — even authenticated, concurrent heavy requests can saturate the GPU; rate-limit and queue.

Remember the earlier point: local is not secure. The model running on your hardware protects the data’s location; it does nothing about who can reach the port. A full, hardened, public production API is the subject of Part 9 (coming soon). Today’s job is to keep the service private and correct. The Kali Linux networking material is a good companion for thinking about exposure and scanning.

A Basic Firewall

Ubuntu ships UFW (Uncomplicated Firewall). Use it to make sure the LLM port is not open to networks it should not be. First see what is actually listening:

ss -tulpn

ss -tulpn lists listening TCP/UDP sockets with the owning process and port. Confirm which address 11434 (and 3000, if you ran the UI) is bound to — 127.0.0.1:11434 is local-only; 0.0.0.0:11434 is every interface. Verify reality here rather than trusting your intent.

Then enable UFW with a sane default:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow OpenSSH
sudo ufw enable

This denies inbound connections by default while keeping SSH reachable so you do not lock yourself out. Check the result:

sudo ufw status verbose

⚠️ Warning — Do not blindly sudo ufw allow 11434. That opens the LLM port to every network UFW governs — the opposite of what you want. If a specific trusted host needs access, scope the rule to it (for example sudo ufw allow from 192.0.2.10 to any port 11434). Combined with binding Ollama to 127.0.0.1, the default-deny posture keeps the API private.

LLM Health and Startup Time

“The container is running” is not “the model is ready.” An LLM service becomes usable in stages:

  Container running
        │
  API responding        (port answers)
        │
  Model available       (weights loaded)
        │
  Inference works       (real response)

AI containers are slow to become ready — they may download a model, load multi-gigabyte weights into memory, and initialize the GPU before the first request can succeed. A plain curl against the API can fail for a minute after docker ps shows the container “up,” and that is normal, not broken.

This is why the Compose file has a healthcheck that curls /api/tags (the endpoint that lists available models) with a generous start_period. A container is only marked healthy once the API genuinely answers. And it is why open-webui uses depends_on: condition: service_healthy rather than a plain depends_on:

❗ Important — depends_on alone only waits for the container to start, not to be ready. Without a healthcheck and condition: service_healthy, the UI can come up before Ollama can serve, and its first calls fail. For AI services, always gate dependents on health, not on start.

Check health with docker ps (it shows healthy/unhealthy) and, when something is off, read the logs — covered below.

Basic Performance Testing

Before you optimize, measure — and measure on your own hardware, because published numbers rarely match your model, quantization, context, and GPU. The two metrics that matter most to users are time-to-first-token (how long until output starts) and tokens per second (how fast it streams once started).

A simple, honest way to get a feel is to time a request:

time curl http://localhost:11434/api/generate -d \
  '{"model":"<model>","prompt":"Say hello","stream":false}'

time reports how long the whole request took. The native API’s JSON response also includes timing fields you can read for load and generation durations. Run it a few times: the first call after a model loads is slower (load + warm-up); steady-state calls are more representative.

Record results in a table you fill in yourself — do not copy numbers from anywhere, including this guide:

Model	Quant	GPU	VRAM used	Context	TTFT	Tokens/sec	GPU util	RAM

Every cell here comes from your measurement under watch -n 1 nvidia-smi and time. Fabricated benchmarks are worse than none — they set false expectations and hide regressions. Measure before you optimize, and measure again after.

Concurrency: One User vs Many

A model that feels instant for one user can crawl for ten. Single-request behavior and multi-user behavior are different regimes:

  1 request  → low latency, some VRAM
  N requests → batching/queueing,
               more KV cache,
               higher throughput OR
               higher latency

Concurrency introduces batching (processing several requests together for better throughput) and queueing (requests waiting for GPU time). More concurrent requests grow the KV cache and working memory — so a context/concurrency combination that fit for one user can OOM under load. And there is a tension: batching can raise total throughput (tokens/sec across everyone) while raising individual latency (each user waits longer). Tuning that balance for real multi-user serving is a Part 9 topic (coming soon); for now, know that “works for me” is a single-user claim, and test at the concurrency you expect.

Troubleshooting: Out of Memory

The most common serious failure is CUDA out of memory (or the ROCm equivalent). Diagnose it, do not just throw hardware at it:

  OOM error
     │
  nvidia-smi: what holds VRAM?
     │
   ├─ stale process pinning memory? → stop it
   ├─ used ≈ total at load?         → model too big
   └─ OOM only mid-generation?      → KV cache
                                       (context/concurrency)

🔍 Troubleshooting — CUDA out of memory. Problem: inference aborts with an out-of-memory error, at load or mid-generation. Likely cause: model weights + KV cache + working memory exceeded VRAM — or a crashed process is still holding memory. Check: nvidia-smi (AMD: rocm-smi) — read the used-memory figure and the Processes table. Is another process pinning VRAM? Is used near total? Fix (in order): stop any stale process holding VRAM; then reduce load — a smaller or more heavily quantized model, a shorter context, or lower concurrency; consider CPU offload to fit at reduced speed. A bigger GPU is the last answer, not the first. Validate: re-run under watch -n 1 nvidia-smi and confirm used memory stays below total through the whole request.

The ordering matters. Engineers reflexively ask for a bigger GPU; more often the fix is a right-sized model, a shorter context, or evicting a zombie process — cheaper, faster, and it teaches you where the memory actually went.

CPU-Only Inference

You do not strictly need a GPU. Drop --gpus=all (or leave the GPU reservation out of Compose) and Ollama runs on the CPU, using system RAM instead of VRAM. It is slower — often much slower for large models — but for small or heavily quantized models it is entirely viable for development, testing, and light use.

This keeps the whole lesson accessible: you can learn the storage, API, networking, health, and security concepts on a CPU-only machine and move the exact same stack onto a GPU node later by adding one flag. It is also a useful diagnostic — if a model works on CPU but fails on GPU, the problem is in the GPU path, not the model.

NVIDIA vs AMD for Local LLMs

Both work. The honest comparison:

NVIDIA — CUDA is the most mature, widely-tested path; most engines and models are developed against it first, and driver/toolkit support on Ubuntu is well-trodden (Part 3). It is usually the smoothest road.
AMD — ROCm is a capable, improving platform; Ollama ships a ROCm image (ollama/ollama:rocm) and reaches AMD GPUs via /dev/kfd and /dev/dri (Part 4/5). Confirm GPU model and OS support first — ROCm’s supported-OS list may lag a brand-new Ubuntu release, so check the compatibility matrix before committing.

🤖 AI Infrastructure Tip — Judge GPUs on the technical facts — VRAM, driver/ROCm support for your model on your Ubuntu version, power, and price — not on which is easier to buy. The recommended GPUs below are grouped by use case; availability is not a measure of technical fit. Validate the actual requirements of the model you intend to run against the actual card.

Where llama.cpp and vLLM Fit

Ollama is your on-ramp, not your ceiling. Two engines you will meet as you grow:

llama.cpp is a lean, efficient inference engine centered on GGUF quantized models. It runs well on modest hardware, on CPU or GPU, and is a strong choice when you need to squeeze a model onto limited resources or want minimal overhead. (Ollama itself builds on this lineage of local, quantized inference.)

vLLM is a server-oriented engine built for throughput: efficient batching and memory management to serve many concurrent requests, with an OpenAI-compatible API. It is heavier to operate than Ollama, and it shines exactly where Ollama’s simplicity runs out — production multi-user serving.

Engine	Optimized for	Concurrency	Typical stage
Ollama	Simplicity, local management	Modest	First service; dev/homelab
llama.cpp	Lean local/quantized efficiency	Modest	Constrained hardware
vLLM	High-throughput serving	High	Production inference

Verify current capabilities before you choose — these engines evolve, and not every one runs every model or quantization. Part 9 (coming soon) returns to production serving with vLLM in view.

Reading the Logs

When inference misbehaves, the logs are your first stop — not a restart. For the container:

docker logs ollama
docker compose logs -f

docker logs ollama prints that container’s output; docker compose logs -f follows all services in the stack live. What to look for, mapped to layers:

Docker/runtime errors — the container failing to start, missing volume, port conflict.
GPU errors — messages about failing to find or initialize the GPU (a sign of the CPU-fallback problem).
Model-load errors — download failures, corrupt files, insufficient memory at load.
HTTP errors — bad requests, timeouts, or the API refusing connections.

The diagnostic path for a failed request follows the stack:

  Request fails
     │
  Reaches API?      → curl localhost:11434
     │ no  → port/bind/firewall
     │ yes
  Model loaded?     → /api/tags, logs
     │ no  → download/load error
     │ yes
  GPU used?         → nvidia-smi
     │ no  → CPU fallback / driver
     │ yes
  Read logs for the specific error

⚠️ Warning — A restart policy plus a fast crash loop can hide the real failure — the container keeps coming back and you never see why it died. When something restarts repeatedly, read the logs first; do not let auto-restart mask a configuration or GPU error.

Troubleshooting Reference

Work each failure with Problem → Likely Cause → Check → Fix → Validate. Where a symptom has several causes, the last few entries say so.

Model download fails Cause: no network egress, disk full, or an invalid model name. Check: docker logs ollama for the error; df -h for space; confirm the model exists at ollama.com/library. Fix: restore egress, free/relocate disk, or correct the model name. Validate: docker exec -it ollama ollama run <model> completes the pull.

Model won’t load Cause: insufficient memory at load, or a corrupt/partial download. Check: logs for an OOM or file error; nvidia-smi for available VRAM. Fix: choose a smaller/quantized model or free memory; re-pull to fix a corrupt file. Validate: the model appears in /api/tags and answers a prompt.

LLM uses CPU, not GPU Cause: --gpus=all (or the Compose GPU reservation) missing, or the container cannot reach the driver/toolkit. Check: watch -n 1 nvidia-smi during a prompt — no VRAM/util movement means CPU; check logs for GPU-init errors. Fix: run with --gpus=all / add the verified Compose GPU block; confirm the NVIDIA Container Toolkit from Part 5. Validate: VRAM rises and GPU utilization spikes during generation.

GPU detected but slow Cause: partial CPU offload (model spilled out of VRAM), or a heavily loaded GPU. Check: nvidia-smi — is VRAM near total (spilling) or is another process competing? Fix: use a smaller/quantized model or shorter context so it fits fully in VRAM; remove competing processes. Validate: the model fits in VRAM and tokens/sec improves.

CUDA out of memory — see the dedicated OOM callout above.

ROCm device unavailable Cause: /dev/kfd or /dev/dri not passed, missing render/video group membership, or an unsupported GPU/OS. Check: logs for device errors; confirm --device /dev/kfd --device /dev/dri and the compatibility matrix (Part 4/5). Fix: pass both devices, add the user to render,video, and verify GPU+OS support before expecting it to work. Validate: rocm-smi in the container sees the GPU and inference uses it.

API unreachable Cause: container down, wrong bind address, or firewall blocking the port. Check: docker ps; ss -tulpn for the bound address; curl http://localhost:11434/api/tags. Fix: start the container; publish/bind the correct address; adjust UFW for the intended source only. Validate: curl returns JSON from the API.

UI can’t reach Ollama Cause: wrong OLLAMA_BASE_URL, or the UI started before Ollama was ready. Check: the env value (in Compose it must be http://ollama:11434); docker ps for Ollama’s health. Fix: set the service-name URL; gate the UI with depends_on: condition: service_healthy. Validate: the UI lists models and returns a response.

Port conflict Cause: another process already owns 11434 (or 3000). Check: ss -tulpn | grep 11434 for the current owner. Fix: stop the conflicting process or remap the host port (for example -p 127.0.0.1:11435:11434). Validate: the container starts and docker ps shows the port published.

Model lost on recreate Cause: the container ran without the named volume, so models lived inside it. Check: docker volume ls for ollama; confirm -v ollama:/root/.ollama is present. Fix: always mount the volume; re-pull once into it. Validate: docker rm + docker run with the volume keeps the model — no re-download.

Continuous restarts Cause: the container crashes on startup and the restart policy relaunches it in a loop. Check: docker logs ollama for the crash reason (GPU init, bad config, OOM at load). Fix: address the underlying error; do not rely on restart to paper over it. Validate: the container stays up and reaches healthy.

Disk fills with models Cause: many/large models accumulating in the volume, possibly on the root filesystem. Check: df -h and docker volume inspect ollama for location; list models with /api/tags. Fix: remove unused models; relocate Docker’s data root / the volume to larger, faster storage. Validate: df -h shows healthy free space and pulls succeed.

Huge startup delay Cause: first-run model download plus multi-GB weight load and GPU init — expected, not a fault. Check: docker logs ollama shows download/load progress; the healthcheck is still in its start_period. Fix: wait for the load; use a healthcheck with a generous start_period so readiness is gated correctly. Validate: after load, the API answers promptly and docker ps shows healthy.

Slow with large context Cause: a long context grows the KV cache and per-token compute. Check: does latency scale with prompt/context length? Watch VRAM climb with context under nvidia-smi. Fix: shorten the context, or move to a model/quantization that handles your context within VRAM. Validate: latency and VRAM stay acceptable at your real context length.

Requests overwhelm the server Cause: concurrency beyond what the GPU/model can serve — queueing, rising latency, or OOM. Check: latency and errors under load; VRAM/util under nvidia-smi at peak. Fix: limit concurrency, queue/rate-limit at a proxy, or scale out (Part 9, coming soon). Validate: latency stays bounded and no OOM at your target concurrency.

GPU utilization ~0 during inference — multiple possible causes. Cause: CPU fallback (no GPU access), the model not actually loaded, or measuring between requests. Check: nvidia-smi timing vs a live prompt; logs for GPU-init errors; /api/tags for a loaded model. Fix: restore GPU access (as in “LLM uses CPU”), ensure the model is loaded, and observe during generation. Validate: utilization spikes while tokens stream.

GPU util high but tokens/sec low — multiple possible causes. Cause: partial CPU offload, an oversized/underquantized model for the card, a very long context, or memory-bandwidth limits. Check: is VRAM near total (spilling)? Is the context large? Is the model right-sized for this GPU? Fix: fit the model fully in VRAM (smaller/quantized), shorten context, or move to a more capable GPU only if the workload truly needs it. Validate: tokens/sec improves and VRAM stays below total.

Hands-On Lab: Build a Local LLM Server on ai-node01

🧪 Hands-On Lab — Turn the GPU container host from Part 5 into a running, private, healthy LLM service. Do these in order on ai-node01.

Confirm the platform. docker version and nvidia-smi (AMD: rocm-smi) — Docker and the GPU are both working from Part 5.
Prove Docker + GPU together. docker run --rm --gpus all ubuntu:24.04 nvidia-smi prints the GPU table from inside a container.
Prepare storage. Confirm space with df -h; plan for models to land in a named volume on fast disk.
Deploy Ollama. docker run -d --gpus=all -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 --name ollama ollama/ollama (loopback bind for safety).
Confirm it started. docker ps shows ollama up with the port published.
Check the volume. docker volume ls and docker volume inspect ollama — verify where model data will live.
Choose a model. Pick a current, hardware-appropriate model from ollama.com/library.
Pull and run it. docker exec -it ollama ollama run <model> — wait for the download, then ask a question.
Watch the GPU. In a second session, watch -n 1 nvidia-smi while generating — confirm VRAM and utilization move.
Test the native API. curl http://localhost:11434/api/generate -d '{"model":"<model>","prompt":"Say hello","stream":false}'.
List models via API. curl http://localhost:11434/api/tags returns the loaded model(s).
Time a request. time curl …/api/generate … to feel latency; run it a few times.
Verify listening sockets. ss -tulpn — confirm 11434 is bound to 127.0.0.1, not 0.0.0.0.
Enable the firewall. sudo ufw default deny incoming, sudo ufw allow OpenSSH, sudo ufw enable; sudo ufw status verbose.
(Optional) Add the UI. Run Open WebUI or the Compose stack; confirm it reaches Ollama over the private network.
Write the Compose file. Save compose.yaml (Ollama + optional UI) with volumes, healthcheck, GPU block, and loopback bind.
Bring up the stack. docker compose up -d; docker ps until Ollama shows healthy.
Read the logs. docker compose logs -f — confirm a clean model load and no GPU errors.
Record a benchmark. Fill one row of your own benchmark table from live measurement.
Save the configuration. Store compose.yaml, an .env.example, and a README under /srv/ai/llm-lab (next section).

  ┌──────────────────────────────────────┐
  │ SUCCESS: ai-node01 serves a local LLM │
  │                                       │
  │  Ollama container .... running  ✓     │
  │  Model in volume ..... persistent ✓   │
  │  GPU used ............ verified  ✓     │
  │  Native API .......... responds  ✓    │
  │  Bound to loopback ... private   ✓     │
  │  Firewall ............ default-deny ✓  │
  │  Healthcheck ......... healthy   ✓     │
  └──────────────────────────────────────┘

Save Your Configuration

Infrastructure you cannot rebuild from a repo is a liability. Capture this stack as version-controlled configuration under a clear layout:

  /srv/ai/llm-lab/
  ├── compose.yaml
  ├── .env.example
  ├── README.md
  └── data/        (volumes / bind mounts)

Create it with correct ownership, as in Part 5:

sudo mkdir -p /srv/ai/llm-lab/data
sudo chown -R "$USER" /srv/ai/llm-lab

compose.yaml — the stack definition, the source of truth.
.env.example — a template of the environment variables the stack expects, with placeholder values.
README.md — how to bring it up, which model to pull, and how to verify it.
data/ — where mounted data lives; keep it out of the repo.

Add a .gitignore so secrets and bulky data never get committed:

.env
data/

⚠️ Warning — Commit the .env.example, never the real .env. Secrets do not belong in a repository — not in Compose, not in env files, not in the README. If you later add auth or API keys at a proxy, they live in the ignored .env (or a secrets manager), and only the template is tracked.

Final Compose Quality Notes

A production-grade Compose file for AI services follows a checklist. The stack above already meets it:

Explicit container_name — stable names for logs, exec, and diagnostics.
Named volumes — model and app data persist across recreation.
A defined network — Compose’s default network gives service-name DNS; the backend stays private.
restart: unless-stopped — survives reboots without fighting your manual stops (unlike always).
A real healthcheck — tests the API, not just “is the process alive,” with a generous start_period for slow AI startup.
The verified GPU block — driver: nvidia, count, and the mandatory capabilities: [gpu].
Environment for wiring — OLLAMA_BASE_URL points services at each other by name.
Restrained ports — publish only what must be reachable, bound to 127.0.0.1 where possible.
Pinned images — pin explicit versions in production for reproducibility; latest moves under you.
No obsolete version: key — modern Compose does not use it; leave it out.

Where the Build-Along Stands

  ai-node01 build-along
  ------------------------------------
  Ubuntu ..................... [done]
  Networking ................. [done]
  Storage .................... [done]
  GPU driver + CUDA/ROCm ..... [done]
  Docker + GPU containers .... [done]
  Local LLM runtime .......... [done]
  Model storage (volume) ..... [done]
  LLM API .................... [done]
  ------------------------------------
  Kubernetes ................. [next]
  Monitoring ................. [upcoming]
  Production inference ....... [upcoming]

You now operate a real local LLM service — a runtime, persistent model storage, and an API — on infrastructure you built from a bare Ubuntu install. That is a genuine milestone.

Preview: Part 7

Everything so far runs on one node. That is exactly enough to learn on and to serve a small workload — and exactly what breaks when you need scheduling, failover, and scale:

  Today (single node):
    ai-node01 ─► Docker ─► LLM ─► GPU

  Tomorrow (orchestrated):
    Kubernetes
      ├─ schedules pods onto GPU nodes
      ├─ restarts failed workloads
      └─ scales services up/down

Docker solved packaging — one image, runs anywhere. Kubernetes adds scheduling, orchestration, and recovery — placing GPU workloads across nodes, restarting what dies, and scaling with demand. That is where the series goes next. The Kubernetes category is worth a look in the meantime.

🤖 AI Infrastructure Tip — Next: Kubernetes for AI Workloads on Ubuntu 26.04 — Coming Soon. The single-node stack you just built is the unit that Kubernetes will schedule and heal at scale. Everything you learned about volumes, health, and GPU access carries straight over.

Choosing GPU Hardware for Local LLMs

The GPUs recommended below (rendered as cards beneath this lesson) are grouped from a starter local-AI lab to a dedicated AI development system. They are starting points, not a shopping list — and for local LLMs specifically, choose against these criteria:

Model size and quantization — the models you intend to run, at the precision you will run them.
VRAM — enough to hold weights + KV cache + working memory for your context and concurrency, with headroom.
Context and concurrency — longer contexts and more simultaneous users need more memory.
Power and thermals — sufficient PSU headroom, connectors, and case airflow for sustained inference.
Budget — the biggest card is rarely the right first answer; a well-fit smaller GPU beats an over-bought one that sits idle.
Linux driver support — confirm the card is supported by a current NVIDIA driver (or ROCm) on Ubuntu 26.04 today.

See the recommended GPUs below and validate each against your actual models and machine. Do not assume a given model fits a given GPU without checking the VRAM math for that specific model and quantization.

What You Learned

What a local LLM is, why “local” means ownership and location (not a laptop), and why local is not automatically secure.
The inference framing of Part 6 — you serve trained weights; you do not train — and the layered app → API → engine → model → GPU stack.
The operational vocabulary — weights, parameters, tokens, context window, quantization, throughput, latency, tokens/sec, TTFT.
How parameter count and quantization drive memory and speed, without ever mapping a size to a fake VRAM number, plus the conceptual VRAM budget and CPU offload.
How to choose an engine — Ollama, llama.cpp, vLLM — for the job at hand.
How to deploy Ollama with Docker, persist models in a named volume, run inference, and verify the GPU is actually working.
How to test the native and OpenAI-compatible APIs, add an optional Open WebUI, and declare the whole stack in Compose on a private network.
How to keep the service private — loopback binding, UFW default-deny — and why AI containers need health gating, not just depends_on.
A layer-by-layer troubleshooting method covering OOM, CPU fallback, unreachable APIs, lost models, restart loops, disk exhaustion, and slow inference.
How to capture the stack as reproducible, secret-free configuration under /srv/ai/llm-lab.

Next Lesson

Kubernetes for AI Workloads on Ubuntu 26.04 — Coming Soon. The single-node LLM service you just built becomes a workload that Kubernetes schedules, restarts, and scales across GPU nodes. We take the exact same concerns — GPU access, persistent storage, health, and networking — and lift them from one Docker host to an orchestrated cluster.

Until then, revisit the Docker Academy to deepen the container fundamentals this stack rests on, preview GPU dashboards in the observability stack, and if you took the AMD path, keep the compatibility notes from Part 4 handy. New to the series? Start at Getting Started or the Ubuntu 26.04 AI Infrastructure overview.