Building an AI Inference Server on Ubuntu 26.04

By the end of Part 8 you had a GPU-aware Kubernetes cluster you can see: ai-node01 running an LLM behind a ClusterIP Service, with Prometheus and Grafana watching every GPU. It works, and it’s observable — but nothing outside the cluster can call it, there’s no TLS, no authentication, and no protection against a single client saturating your GPU. This lesson answers the question that turns a workload into a product: how do I turn this AI workload into a reliable inference service that real applications can consume?

What You’ll Learn

• The difference between a local LLM you experiment with and an inference server that serves clients, and why the second is an infrastructure problem
• The full inference-server architecture — client → HTTPS → gateway (auth + rate limit) → Service → GPU Pods → model storage — and what each layer is responsible for
• How to choose an inference engine, and why this build uses vLLM for server-style serving while Ollama stays fine for local use
• What an OpenAI-compatible API buys you, and where “compatible” is not “identical”
• How to deploy vLLM to Kubernetes with a GPU request, a model-cache PVC, a ConfigMap, and a Secret — and tune health probes for a slow model load
• Why ingress-nginx is retired and how to expose the service safely with the Gateway API, cert-manager, and Let’s Encrypt TLS
• How to authenticate the API and never expose an unauthenticated GPU, plus rate limiting and request-size limits that protect GPU capacity
• Why replicas don’t create GPU capacity, how vLLM handles concurrency, and how to think about GPU-aware scaling and load balancing across nodes
• How to reason about performance and SLOs, run a safe load test against your own lab, and log requests without leaking prompts
• A layer-by-layer method for troubleshooting the whole path from DNS to GPU

By the end you’ll have a secure, authenticated, TLS-terminated inference API that an application outside the cluster can call — the last piece before Part 10 turns it into a production platform.

From Local LLM to Inference Server

In Part 6 you ran a model on one host and talked to it yourself. That is a local LLM: a single operator, on the same machine or a trusted network, poking an API by hand. It is the right shape for experimenting, and nothing below is meant to disparage it.

  Local LLM (Part 6 experiment)
  ------------------------------
  You (admin)
     │  curl on localhost
     ▼
  model runtime
     │
   GPU (whole card)
  ------------------------------
  One user · trusted network
  No auth · no TLS · no limits

An inference server is a different job. It exists to serve other people’s applications — a web backend, a batch job, a mobile app — over a network you don’t fully trust, at the same time, without any of them being able to take the service down or read each other’s data.

  Inference server (this lesson)
  ------------------------------
  Application(s)
     │  HTTPS
     ▼
  Auth  (who are you?)
     │
  Load balancer
     │
  model runtime (batched)
     │
   GPU (shared across requests)
  ------------------------------
  Many clients · untrusted network
  Auth · TLS · rate limits · HA

Every box that appears in the second diagram is infrastructure work, not model work. The model is the same; what changes is that it now sits behind a front door with a lock, a queue, and a bouncer. That front door is what this lesson builds.

🤖 AI Infrastructure Tip — The temptation at this stage is to kubectl expose the Service as a LoadBalancer, hand out the IP, and call it done. Resist it. An unauthenticated GPU endpoint on the internet is discovered and abused within hours — someone else runs their inference on your electricity, or simply floods it until your real users get nothing. Exposure without auth, TLS, and rate limits is not “shipping fast,” it’s shipping a liability. The rest of this lesson is the difference.

Inference Server Architecture

Here is the whole path a production request travels, top to bottom. Read it once now; every section that follows builds one of these layers.

  Client application
      │  HTTPS (443)
      ▼
  Gateway  (the front door)
    ├─ TLS termination
    ├─ authentication  (API key / OIDC)
    ├─ rate limiting
    └─ request routing + logging
      │  HTTP (in-cluster)
      ▼
  Service  (stable name: vllm:8000)
      │  load-balances to ready Pods
      ▼
  Pod ... Pod   (vLLM runtime)
      │
   GPU  (nvidia.com/gpu reserved)
      │
  Model storage  (PVC / HF cache)

• Client — any application with a valid credential. It never talks to a Pod directly; it only knows the public hostname.
• Gateway — the single controlled entry point. It terminates TLS, checks the caller’s identity, enforces rate and size limits, routes the request, and logs it. This is where “public and dangerous” becomes “public and safe.”
• Service — the stable in-cluster name (vllm:8000) from Part 7. It load-balances across whichever Pods are currently Ready and hides their churn.
• Pods — one or more vLLM containers, each holding a GPU. This is where inference actually happens.
• GPU — reserved whole per Pod, exactly as in Part 7. The card is the scarce resource everything upstream is protecting.
• Model storage — a PersistentVolumeClaim caching the model weights so Pods don’t re-download gigabytes on every restart.

The mental model to carry: traffic flows down, trust is checked at the top. Nothing below the gateway is exposed to the outside world, and nothing reaches the GPU without passing the gateway’s checks first.

Choosing an Inference Engine

Parts 6–8 used Ollama, and for local use it is excellent — one binary, easy model pulls, a friendly API. But a server that must handle many concurrent clients, expose real metrics, and keep the GPU busy has different requirements. The main contenders:

Engine	Best for	Concurrency	Native metrics
Ollama	Local / simple serving	Basic	No Prometheus endpoint
llama.cpp	Lightweight / CPU + small GPU	Limited	Minimal
vLLM	Server-style serving	Continuous batching	Native /metrics

The deciding factor for an inference server is how the engine handles many requests hitting one GPU at once. vLLM was built for this: it uses continuous batching — as requests arrive, it merges them into the GPU’s in-flight batch instead of processing them strictly one at a time, which keeps an expensive accelerator busy and lifts total throughput under load. Just as important for everything you built in Part 8, vLLM exposes a native Prometheus /metrics endpoint with queue depth, KV-cache utilization, and running/waiting request counts — the exact signals a GPU-aware platform needs.

🤖 AI Infrastructure Tip — This is a “right tool for the job” choice, not a verdict on Ollama. Ollama remains a fine local runtime, and the Running Local LLMs lesson stands. You switch to vLLM here specifically for concurrency and native observability — the two things a multi-client server needs and a single-user experiment doesn’t. Pick the runtime by the workload, not by fashion. If you only ever serve yourself, Ollama was already enough.

vLLM runs a model with one command, and ships an official image:

vllm serve <model> --host 0.0.0.0 --port 8000 --api-key <YOUR_API_KEY>

vllm serve starts the server; --host 0.0.0.0 makes it listen on all interfaces (required inside a container); --port 8000 is its default; --api-key turns on a simple bearer-token check. The <model> is an example placeholder — you pick a current, license-checked model whose weights fit your GPU’s VRAM. Do not treat any specific model name as guaranteed to fit; verify its published requirements against your card first.

The OpenAI-Compatible API

vLLM serves an OpenAI-compatible API: the same request and response shapes the OpenAI SDKs use, on port 8000 under /v1. That single fact is why it’s worth choosing — a large ecosystem of clients, libraries, and tools that “speak OpenAI” can point at your server by changing only the base URL and the key. No custom client, no bespoke protocol.

The endpoints you’ll actually use:

• /v1/chat/completions — chat-style requests (the common one).
• /v1/completions — plain text completion.
• /v1/models — lists the served model(s); handy for a smoke test.
• /health — a liveness/readiness signal (not under /v1). Use this for probes; do not invent another health path.
• /metrics — the native Prometheus endpoint.

A minimal authenticated call:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $KEY" \
  -d '{"model":"<model>","messages":[{"role":"user","content":"hello"}]}'

A 200 with a JSON completion means the server is up, the key is accepted, and the model is loaded. A 401 means the key is wrong or missing; a connection refused means the server isn’t listening yet (often still loading the model).

❗ Important — “OpenAI-compatible” is a strong convenience, not a guarantee of feature parity. Compatible ≠ identical. Before you promise a client that some OpenAI feature works, verify it against the vLLM version you actually deployed: the exact endpoint paths, whether streaming behaves as expected, how model names are reported (the name you serve may differ from the name a client hard-codes), and whether tool/function calling and embeddings are supported for your model. Don’t assume every OpenAI feature is present just because the chat endpoint answers. Test the specific features your callers depend on.

Deploying vLLM to Kubernetes

You already know this shape from Part 7 — a Deployment that requests a GPU, with config, a secret, persistent storage, and a Service. Here you swap the runtime for vLLM and wire in the pieces a server needs.

The vLLM Container

The official image is vllm/vllm-openai:latest. Run standalone (outside Kubernetes) it looks like this — useful for a first local smoke test on the GPU node:

docker run --gpus all -p 8000:8000 --ipc=host \
  vllm/vllm-openai:latest --model <model>

--gpus all hands the container the GPU (the NVIDIA Container Toolkit from Part 5); -p 8000:8000 publishes the API; --model selects the weights. The --ipc=host flag matters specifically for vLLM: it needs a large shared-memory segment for its internal data movement, and the container default is too small. In Kubernetes you get the same effect with a Memory-backed emptyDir mounted at /dev/shm, shown below.

⚠️ Warning — latest is fine for a first experiment, but never run latest in anything you depend on. In Part 10 you’ll pin an immutable tag and scan the image. A moving latest means your inference server can change out from under you on the next Pod restart — a different vLLM version, different behavior, possibly a broken model load — with nothing in Git recording what changed. Pin the version once you’re past “does it start at all.”

ConfigMap, Secret, and the Model Cache

Non-secret settings — the model name and vLLM arguments — go in a ConfigMap so you can change them without rebuilding an image:

apiVersion: v1
kind: ConfigMap
metadata:
  name: vllm-config
data:
  MODEL: "<model>"          # example placeholder
  MAX_MODEL_LEN: "<context>"  # size to your GPU + model

The API key goes in a Secret. Create it from a value you generate — never a literal in a manifest you commit:

kubectl create secret generic vllm-api-key \
  --from-literal=api-key="<YOUR_API_KEY>"

⚠️ Warning — A Kubernetes Secret is base64-encoded, not encrypted — the same warning as Part 7, and it matters more now that the secret is an API key guarding a GPU. Anyone who can run kubectl get secret vllm-api-key -o yaml and base64 -d reads it in plaintext. A Secret is the correct place for the key (RBAC-restrictable, mountable, encryptable at rest by the cluster operator), but do not mistake the default for confidentiality. Never commit a Secret with a live value to Git; Part 10 covers encrypted-at-rest etcd, Sealed Secrets, and External Secrets/Vault for real protection.

The model weights need a PersistentVolumeClaim so a Pod restart doesn’t re-download gigabytes. vLLM caches downloaded models under the Hugging Face cache directory, so mount the PVC there:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vllm-model-cache
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: <size>Gi   # size to the model(s) you cache

Size storage to the models you actually cache — there is no universal model size, so measure, don’t guess.

The Deployment

The core Deployment: one replica, one GPU, the config and secret injected, the cache and shared memory mounted.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm
spec:
  replicas: 1
  selector:
    matchLabels: { app: vllm }
  template:
    metadata:
      labels: { app: vllm }
    spec:
      nodeSelector:
        accelerator: nvidia
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args: ["--model", "$(MODEL)",
                 "--host", "0.0.0.0", "--port", "8000",
                 "--api-key", "$(API_KEY)"]
          ports:
            - containerPort: 8000
          envFrom:
            - configMapRef: { name: vllm-config }
          env:
            - name: API_KEY
              valueFrom:
                secretKeyRef:
                  name: vllm-api-key
                  key: api-key
          resources:
            requests:
              cpu: "<n>"      # from Part 8 measurements
              memory: "<n>Gi" # from Part 8 measurements
            limits:
              nvidia.com/gpu: 1
          volumeMounts:
            - name: cache
              mountPath: /root/.cache/huggingface
            - name: shm
              mountPath: /dev/shm
      volumes:
        - name: cache
          persistentVolumeClaim:
            claimName: vllm-model-cache
        - name: shm
          emptyDir:
            medium: Memory

Read the important lines: nvidia.com/gpu: 1 reserves a whole card (AMD hardware uses amd.com/gpu: 1); the CPU and memory requests are placeholders you fill from the numbers you measured in Part 8 — do not paste arbitrary huge values, which only make the Pod harder to schedule. The /dev/shm emptyDir with medium: Memory is the Kubernetes equivalent of --ipc=host. replicas: 1 because you have one GPU; asking for two replicas on a one-GPU node leaves the second Pod Pending forever, exactly as in Part 7.

Model Startup Sequence

A vLLM Pod is Running long before it can answer a request. Knowing the sequence is what keeps you from misreading a slow start as a failure:

  Pod scheduled onto GPU node
      │
  Container starts (Running)
      │
  Download model → PVC cache
      │   (skipped if already cached)
  Load weights into VRAM
      │
  Init KV cache + warm up
      │
  /health returns OK  → Ready

The gap between “Running” and “Ready” can be minutes on a first, uncached run, and still tens of seconds when cached. Everything about the probes below exists to protect that gap.

Storage Options

Where the model cache lives is a real tradeoff, not a detail:

• Local NVMe — fastest load, simplest; but the data is tied to one node (node affinity), so a Pod rescheduled elsewhere can’t reach it.
• NFS — shared across nodes, so any node can pull from one cache; higher latency than local disk.
• Ceph / distributed storage — shared and resilient; more moving parts to operate.
• Object storage + local cache — weights in object storage, pulled to a fast local cache on first use; good for many nodes.
• A model registry — treat weights as versioned artifacts a Pod fetches on start (Part 10 expands this).
• Preloaded node storage — weights staged on the node ahead of time for the fastest possible cold start.

For a single GPU node, local NVMe is the right default. Reach for shared storage when several nodes must serve the same model without each re-downloading it.

Health Probes for a Slow-Loading Model

Three probes, three jobs — and for a model that takes minutes to load, getting them right is the difference between a stable service and a restart loop.

• startupProbe — guards the slow load. Until it passes, liveness and readiness are held off, so a long model load can’t be mistaken for a hang.
• readinessProbe — gates Service traffic. It hits vLLM’s /health, and the Pod joins the Service’s load-balancing pool only when the API can actually serve.
• livenessProbe — restarts a genuinely stuck process, and is deliberately non-aggressive.

          startupProbe:
            httpGet: { path: /health, port: 8000 }
            failureThreshold: 60
            periodSeconds: 10
          readinessProbe:
            httpGet: { path: /health, port: 8000 }
            periodSeconds: 10
          livenessProbe:
            httpGet: { path: /health, port: 8000 }
            periodSeconds: 20
            failureThreshold: 3

The startupProbe’s failureThreshold × periodSeconds budget (here up to 600 seconds) covers even a first uncached load before liveness starts counting. All three hit /health — the verified endpoint — not an invented path.

⚠️ Warning — Container running ≠ model ready, and an aggressive livenessProbe is the classic footgun here. If liveness starts checking immediately with a short timeout, it fails during the normal multi-minute model load, Kubernetes concludes the container is hung and restarts it — and the restart begins loading the model again, failing again, forever. That restart loop is caused entirely by the probe, not the app. Always give a slow model load a startupProbe, gate traffic with a readinessProbe on /health, and keep liveness patient. “Still loading” must never read as “hung.”

The Kubernetes Service

The Service is unchanged from Part 7 in shape, and remains ClusterIP — internal only — as the correct baseline. External exposure is the gateway’s job, not the Service’s.

apiVersion: v1
kind: Service
metadata:
  name: vllm
spec:
  selector:
    app: vllm
  ports:
    - port: 8000
      targetPort: 8000

The selector wires the Service to the vLLM Pods; anything inside the cluster reaches the API at vllm:8000, and the Service load-balances across Pods that pass the readiness probe. Confirm it has backends with kubectl get endpoints vllm — an empty list means no Pod is Ready yet. The gateway you build next targets this Service; it never targets Pods directly.

Ingress Is Retired — Use the Gateway API

To expose the Service outside the cluster with a hostname, TLS, and routing rules, the old instinct is to reach for an Ingress controller — usually ingress-nginx. Don’t.

❗ Important — ingress-nginx reached end-of-life in March 2026. It is retired and archived: no more releases, no more security patches. Do not stand up a new inference server on it. The Kubernetes project recommends the Gateway API (now GA) as its successor. The core Ingress API still exists but is feature-frozen — fine to recognize in older clusters, wrong to build on today. This lesson teaches the Gateway API, which is where new external-traffic work belongs.

The Gateway API splits what Ingress crammed into one object into a clear chain of resources with distinct owners:

  GatewayClass   (which implementation)
      │
  Gateway        (listeners: HTTPS/443 + TLS)
      │
  HTTPRoute      (host/path → Service)
      │
  Service        (vllm:8000)
      │
  vLLM Pods

• GatewayClass — names the implementation that will actually run the data plane (like a StorageClass names a storage provisioner).
• Gateway — declares listeners: for us, an HTTPS listener on port 443 referencing a TLS certificate. This is the public entry point.
• HTTPRoute — the routing rules: “requests for api.example.com on /v1 go to the vllm Service.”

Several implementations provide the GatewayClass — Envoy Gateway, NGINX Gateway Fabric, Traefik, and various cloud gateways. Pick one. This lab uses Envoy Gateway, a clean CNCF choice; install it per its current docs rather than pasting a manifest URL that will rot. Whichever you choose, it owns the responsibilities the reverse proxy always had:

• TLS termination — decrypt HTTPS at the edge.
• Routing — host/path to the right Service.
• Rate limiting — protect the backend (below).
• Authentication — verify the caller (below).
• Logging — a record of every request at the edge.

🛠️ DevOps Tip — Run one gateway implementation, not two. Layering gateways (or a gateway plus a legacy Ingress controller) produces exactly the kind of confusing, half-working routing that eats an afternoon. Choose Envoy Gateway (or your platform’s standard), learn its configuration, and keep the edge simple. The Nginx guides and Kubernetes & Helm guides are useful background on reverse-proxy and gateway concepts, but the resource model above is the current, supported path.

HTTPS and TLS with cert-manager

An AI API handles prompts and completions that are often sensitive; it must speak HTTPS, never plain HTTP. The standard way to get and auto-renew certificates in Kubernetes is cert-manager.

Install it with Helm, enabling its CRDs and Gateway API support:

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager \
  -n cert-manager --create-namespace \
  --set crds.enabled=true \
  --set config.enableGatewayAPI=true

crds.enabled=true installs cert-manager’s custom resources; config.enableGatewayAPI=true lets it watch Gateway listeners and provision their certificates. Next, a ClusterIssuer tells cert-manager how to get certificates — here, Let’s Encrypt over the ACME HTTP01 challenge:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    email: <YOUR_EMAIL>   # placeholder — use your own
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-account-key
    solvers:
      - http01:
          gatewayHTTPRoute: {}

The email is a placeholder — put your own address there, never a fabricated one. With the ClusterIssuer in place, cert-manager watches the Gateway’s TLS configuration, solves the ACME HTTP01 challenge (using a temporary HTTPRoute), and provisions the TLS Secret the Gateway references. The flow end to end:

  Client
    │  HTTPS (443)
    ▼
  Gateway (TLS listener)
    │  cert from Secret
    │  provisioned by cert-manager
    ▼
  Auth → Service → vLLM Pod

The public hostname api.example.com here is documentation only — you may not own a domain. If you don’t, you have two honest options: use a domain you control, or issue a self-signed / private CA certificate for a lab hostname and trust it on your clients. Let’s Encrypt needs a real, publicly resolvable domain to validate; a self-signed cert doesn’t, at the cost of clients having to trust your CA explicitly. Either way, the API is HTTPS — never expose it as plain HTTP.

Authentication: Never Expose an Unauthenticated GPU API

⚠️ Warning — Do not expose an unauthenticated GPU API. An open inference endpoint is found by scanners fast, and the abuse chain is direct: an attacker discovers the endpoint → sends unlimited requests → runs their inference on your GPU and electricity → your real users get slow responses or none, and your costs climb. Even “internal only for now” endpoints leak. Authentication is not optional polish; it is the load-bearing control that makes public exposure survivable. Build it before you route a single external request.

The simplest pattern, which you already wired above, is an API key: vLLM’s --api-key (sourced from the Secret) requires every request to carry Authorization: Bearer <key>. Requests with a valid key are authorized; requests without one get 401 Unauthorized:

# Authorized — 200 + completion
curl https://api.example.com/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"<model>","messages":[{"role":"user","content":"hi"}]}'

# No key — 401 Unauthorized
curl https://api.example.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"<model>","messages":[{"role":"user","content":"hi"}]}'

For a single team, a gateway-enforced or vLLM-enforced API key is enough. The more robust option for many users or organizations is an OAuth2/OIDC proxy at the gateway: callers authenticate against an identity provider and present short-lived tokens, so you get per-user identity, central revocation, and no shared secret to leak. Choose the key pattern for a lab and small team; reach for OIDC when you need real per-user identity.

Key Lifecycle

A key is only as safe as how you manage it. Treat the full lifecycle deliberately:

• Generation — generate keys from a strong random source; never a memorable string, and never password123.
• Rotation — rotate on a schedule and after any suspected exposure; support more than one valid key briefly so callers can migrate without downtime.
• Revocation — be able to invalidate a single key immediately when it leaks.
• Storage — keep keys in Secrets (encrypted at rest — Part 10), never in Git, never in an image, never in a ConfigMap.
• Audit — log which key made which request (never the key value itself) so you can trace and revoke.

❗ Important — Secrets management is a topic Part 10 expands, but internalize the core fact now: base64 is encoding, not encryption. A Kubernetes Secret protects a key through RBAC and optional encryption-at-rest, not by default confidentiality. The stronger patterns — encrypted etcd, Sealed Secrets, External Secrets Operator, Vault — exist precisely because “it’s in a Secret” is not “it’s encrypted.” For now: restrict who can read the Secret, keep it out of Git, and plan to encrypt it at rest.

Rate Limiting and Request-Size Limits

Authentication says who may call; rate limiting says how much, and it’s what actually protects a scarce GPU from a single misbehaving client. Without it, one authenticated caller in a retry loop can starve everyone else. Configure it at the gateway (each implementation has its own current syntax) to bound requests per client over a window — by API key or source.

Rate limiting counts requests; request-size limits bound how expensive each one is. An inference request’s cost scales with its input, so cap:

• Input tokens / context length — the biggest cost lever; an enormous prompt can occupy the GPU far longer than many small ones.
• Request body size — reject oversized payloads at the gateway before they reach a Pod.
• Max context — align the served context window with what your GPU’s VRAM and KV cache can actually hold.

🤖 AI Infrastructure Tip — There is no universal right number for any of these limits, and pasting one from a tutorial is how you either throttle legitimate users or leave the door open. Derive them from your stack: the GPU’s VRAM, the model’s context window, and the request sizes your real clients send (measured with the Part 8 monitoring). Set a defensible starting limit, watch queue depth and latency under real traffic, and adjust. Limits are a dial you tune with data, not a constant you copy.

Concurrency, Load Balancing, and Multiple GPU Nodes

A common misconception: “one GPU, so one request at a time.” Not with vLLM. Through continuous batching it serves many requests concurrently on one card — but concurrency is bounded, not unlimited. The ceiling is set by VRAM and the KV cache: each in-flight request consumes KV-cache memory, and when that fills, new requests queue rather than run. That queue depth is exactly the vLLM /metrics signal you watched in Part 8.

The dangerous instinct is to “add capacity” by raising replicas. It doesn’t work:

  replicas: 2   +   1 GPU
  ------------------------------
  Pod A → gets the GPU  (Running)
  Pod B → no GPU free   (Pending)
  ------------------------------
  Result: still 1 serving Pod.
  Replicas do NOT create GPUs.

A GPU is reserved whole per Pod, so a second replica needs a second GPU — on the same node or another. Real horizontal capacity means multiple GPU nodes, with the Service load-balancing across ready Pods on each:

  Gateway
     │
  Service (vllm:8000)
     ├──────────────┬──────────────┐
     ▼              ▼              ▼
  vLLM Pod       vLLM Pod       (ready
  ai-node01      ai-node02       Pods)
     │              │
   GPU            GPU

Now two GPUs serve in parallel, the Service spreads requests across both, and losing one node degrades capacity instead of taking the service down. The Service load-balances automatically across every Pod that passes readiness — you don’t configure per-Pod routing, you add GPU nodes and replicas that each land a real GPU.

GPU-Aware Scaling

Autoscaling inference is where beginners most often reach for the wrong tool. The default HorizontalPodAutoscaler on CPU is wrong for GPU inference: an overloaded vLLM Pod can show low CPU while its GPU is saturated and its queue is growing. Scaling on CPU would miss the overload entirely — or scale for a bottleneck that isn’t there.

Scale instead on the signals that reflect real inference load, all of which vLLM’s /metrics exposes:

• Queue depth — requests waiting for GPU time.
• Running vs waiting requests — how full the batch is.
• Latency — response time drifting past your target.

Two current, verified ways to drive scaling from those app metrics:

• prometheus-adapter / custom metrics — expose vLLM’s Prometheus metrics to the HPA as custom metrics, and scale on queue depth or latency instead of CPU.
• KEDA — an event-driven autoscaler whose Prometheus scaler reads vLLM’s /metrics directly and scales the Deployment on a PromQL expression.

Either way, remember the hard limit from the previous section: scaling adds Pods, and each new Pod needs a real GPU. An HPA that wants a third replica on a two-GPU cluster just produces a Pending Pod. GPU-aware scaling schedules onto GPUs that exist; it does not conjure them, and there is no autoscaling magic that changes that.

❗ Important — Advanced topic: scale-to-zero. KEDA can scale an inference Deployment down to zero Pods when idle and back up on the first request, which saves an idle GPU’s cost. The tradeoff is a cold start — the next caller waits through the full model-load sequence (potentially minutes) before getting a response. That is fine for spiky, latency-tolerant batch work and unacceptable for an interactive API. Scale-to-zero is a real option to know about, not a requirement for this lab; don’t add it unless the cost saving is worth the cold-start latency for your traffic.

Performance and SLOs

You already built the measurement layer in Part 8 — request rate, error rate, latency, time-to-first-token (TTFT), tokens/sec, concurrency, queue depth, GPU utilization, VRAM. A server is where you put those numbers to work.

Start with SLO thinking. A Service Level Objective is a target you commit to and measure against, built from a few Service Level Indicators (SLIs):

• Availability — the fraction of requests served successfully.
• Latency — response time (and TTFT) under a target.
• Error rate — the fraction of failed requests, kept under a ceiling.

⚠️ Warning — Do not adopt universal SLO numbers from a guide — including this one. The right latency target for an interactive chat feature and a nightly batch job are wildly different, and both depend on your model, GPU, and traffic. You set the targets, from what your users need and what your measured baseline can deliver. A number copied from a tutorial is a commitment you can’t defend.

For the same reason, a performance baseline is reader-filled. Never fabricate tokens/sec or latency — measure them on your own stack:

Metric	Baseline	Under load
Requests/sec
Latency (p50/p95)
Time-to-first-token
Tokens/sec
Queue depth
GPU utilization

Fill every cell from your dashboard, on your hardware. To generate load, ramp gradually against your own lab — a small script sending your curl call at rising concurrency, or an existing load tool — never against a service you don’t own, and never a sudden flood that just crashes your own server and teaches you nothing.

Then read the correlation while the load runs — this is the whole skill:

  Healthy under load
    requests ▲  GPU_UTIL ▲  latency ~flat
    → the GPU absorbs the work

  Saturated
    requests ▲  GPU_UTIL ~100%
    queue_depth ▲  TTFT ▲
    → past capacity; requests waiting

Rising throughput with stable latency means you have headroom. Rising queue depth and TTFT while GPU utilization pins at 100% means you’ve hit the GPU’s ceiling — the signal that you need another GPU node, not a bigger prompt limit. That distinction, read live, is what turns “it feels slow” into an evidence-based capacity decision.

Logging and Privacy

An inference server needs logs at several layers, each answering a different question:

• Gateway logs — who called, when, which route, response status, latency. The edge record of every request.
• Runtime logs — vLLM’s own output: model load, errors, batching behavior.
• Pod / system logs — Kubernetes events and the node’s view (Part 7’s kubectl logs / describe).

Thread a request ID through the gateway so one request can be traced across all three. That correlation is what lets you follow a slow or failed request from the edge to the GPU.

⚠️ Warning — Be careful what you log. Inference requests carry prompts and completions that routinely include proprietary and customer data, and users sometimes paste credentials into a prompt by accident. If your logging captures full request and response bodies by default, you’ve built a searchable archive of other people’s secrets. Minimize by default: log metadata (request ID, timing, status, token counts) rather than content; if you must log bodies for debugging, do it deliberately, briefly, and with access controls. This is an engineering practice for reducing risk — not legal advice, and no claim about any specific regulation.

Security Checklist

Before you consider the inference server exposed, walk this list. Every item is something this lesson or Part 7 built:

• HTTPS everywhere — TLS via cert-manager; no plain-HTTP endpoint.
• Authentication on every route — API key or OIDC; no unauthenticated path to the GPU.
• Key rotation — a real generation/rotation/revocation process.
• Rate limits — per-client request and size limits at the gateway.
• Restricted exposure — only the gateway is public; the Service stays ClusterIP.
• RBAC — least-privilege access to the namespace and Secrets.
• Non-root containers — run vLLM as a non-root user where the image allows.
• Read-only filesystem — where practical, with writable mounts only for the cache.
• Trusted images — pinned, scanned images (Part 10); not a moving latest.
• Resource limits — GPU limit set; CPU/memory requests sized from measurement.
• Secrets handled properly — out of Git, encrypted at rest, RBAC-restricted.
• Protected dashboards — Grafana/Prometheus from Part 8 not publicly exposed.
• Protected API — the whole chain above, verified end to end.

Troubleshooting

Diagnose the inference server top-down through this stack. A request that fails, fails at one layer — find it by walking down, not by guessing.

  Client
    │
  DNS        (name resolves?)
    │
  TLS        (cert valid?)
    │
  Gateway    (routing up?)
    │
  Auth       (key accepted?)
    │
  Service    (endpoints present?)
    │
  Pod        (Running?)
    │
  Runtime    (vLLM /health OK?)
    │
  GPU        (allocated + healthy?)

Work the failures with Problem → Likely Cause → Check → Fix → Validate.

🔍 Troubleshooting — API unreachable from outside. Problem: clients can’t reach api.example.com at all. Likely cause: DNS doesn’t point at the gateway, or the gateway isn’t listening. Check: resolve the hostname; kubectl get gateway for its address/status. Fix: point DNS at the gateway’s external address; confirm the Gateway is programmed. Validate: a request reaches the gateway (even a 401 proves connectivity).

🔍 Troubleshooting — TLS certificate invalid. Problem: clients get a certificate warning or handshake failure. Likely cause: cert-manager hasn’t issued the cert, or the Gateway references the wrong Secret. Check: kubectl get certificate; cert-manager logs; the Gateway’s TLS certificateRefs. Fix: resolve the ACME challenge (DNS/HTTP01), or point the listener at the right Secret; use a self-signed cert if you own no domain. Validate: the TLS Secret exists and clients get a valid chain.

🔍 Troubleshooting — Gateway returns 502/503. Problem: the gateway answers but with a bad-gateway error. Likely cause: the gateway can’t reach a ready backend — no endpoints, or all Pods unready. Check: kubectl get endpoints vllm; HTTPRoute backend reference; Pod readiness. Fix: align the HTTPRoute/Service selector; get a Pod to Ready (see model-load probes). Validate: a valid request returns a completion.

🔍 Troubleshooting — Authentication always fails (401). Problem: even a correct-looking key is rejected. Likely cause: the Secret value differs from the key clients send, or the header is malformed. Check: the Secret value vs the client key; the Authorization: Bearer header. Fix: align the key; send the correct header format; restart the Pod after a Secret change. Validate: a valid key returns 200, a missing key returns 401.

🔍 Troubleshooting — API key exposed / leaked. Problem: a key ended up in a log, repo, or client build. Likely cause: a key was hard-coded or logged. Check: where the key appears (Git history, logs, images). Fix: revoke that key immediately, issue a new one, rotate clients; scrub the exposure. Validate: the old key returns 401; clients work with the new key.

🔍 Troubleshooting — Pod stuck Pending. Problem: the vLLM Pod never schedules. Likely cause: no free GPU, nodeSelector/taint mismatch, or PVC unbound. Check: kubectl describe pod Events; kubectl describe node GPU allocatable; kubectl get pvc. Fix: free/add a GPU, fix the selector/toleration, provision the PVC. Validate: the Pod schedules onto a GPU node.

🔍 Troubleshooting — No GPU inside the Pod. Problem: the Pod runs but vLLM can’t see a GPU. Likely cause: the container didn’t request nvidia.com/gpu, or the host driver/plugin is broken. Check: the Pod’s resources.limits; nvidia-smi on the host; the device plugin. Fix: add the GPU limit; repair the host GPU stack (Part 3/Part 4). Validate: vLLM logs show the GPU and the model loads.

🔍 Troubleshooting — Model won’t load. Problem: the container runs but the model never becomes ready. Likely cause: wrong model name, missing weights in the cache, or a download failure. Check: kubectl logs for the load error; the PVC contents; network egress. Fix: correct the model name, pre-cache the weights, restore egress. Validate: /health returns OK and /v1/models lists the model.

🔍 Troubleshooting — CUDA out of memory (NVIDIA). Problem: load or inference fails with a CUDA OOM. Likely cause: the model + KV cache exceed VRAM, or a stale process holds memory. Check: nvidia-smi on the node for used VRAM; the model’s requirements; context length. Fix: use a smaller/quantized model, reduce max context, or free the stray process. Validate: the model loads and inference runs within VRAM.

🔍 Troubleshooting — ROCm fails on AMD. Problem: an AMD Pod can’t use the GPU. Likely cause: amd.com/gpu not requested, host ROCm broken, or 26.04 unsupported. Check: the resource request; host rocm-smi; the AMD compatibility matrix. Fix: request amd.com/gpu, repair host ROCm, confirm OS support (Part 4). Validate: vLLM sees the AMD GPU and serves.

🔍 Troubleshooting — Readiness never healthy. Problem: the Pod stays 0/1 Ready; the Service has no endpoints. Likely cause: the model is still loading, or the readiness probe targets the wrong path/port. Check: kubectl logs for load progress; the probe’s path: /health and port: 8000. Fix: give the load a startupProbe; point readiness at /health:8000. Validate: the Pod goes Ready and joins the Service.

🔍 Troubleshooting — Requests time out during startup. Problem: early requests hang or fail right after deploy. Likely cause: traffic hit the Pod before the model finished loading. Check: correlate request failures with the load window in logs. Fix: rely on the readiness probe so the Service withholds traffic until ready. Validate: requests succeed only after Ready, and clients retry cleanly.

🔍 Troubleshooting — Service has no endpoints. Problem: kubectl get endpoints vllm is empty. Likely cause: the Service selector doesn’t match Pod labels, or no Pod is Ready. Check: the Service selector vs Pod labels; Pod readiness. Fix: align labels; fix readiness so Pods enter the pool. Validate: endpoints list Pod IPs and requests route.

🔍 Troubleshooting — Gateway can’t reach the Service. Problem: the gateway is up but backend calls fail. Likely cause: the HTTPRoute points at the wrong Service/port, or a NetworkPolicy blocks it. Check: the HTTPRoute backendRefs; any NetworkPolicy on the namespace. Fix: correct the backend reference; allow gateway → Service traffic. Validate: the gateway routes to vllm:8000 and returns completions.

🔍 Troubleshooting — Slow under load. Problem: latency climbs as traffic rises. Likely cause: the GPU is saturated and requests are queuing. Check: vLLM /metrics queue depth and running/waiting; GPU utilization. Fix: add a GPU node/replica (each needs a real GPU); tune batching/limits. Validate: queue depth falls and latency returns to baseline.

🔍 Troubleshooting — TTFT spikes. Problem: time-to-first-token jumps intermittently. Likely cause: queueing at high concurrency, or very large prompts. Check: queue depth vs TTFT; input token sizes. Fix: enforce input-size limits; add capacity for sustained spikes. Validate: TTFT stabilizes within your SLO.

🔍 Troubleshooting — GPU at 100% but throughput poor. Problem: the GPU is pinned yet tokens/sec is low. Likely cause: thermal throttling, an oversized model spilling memory, or long contexts. Check: GPU temperature and clocks (Part 8), VRAM headroom, context length. Fix: improve cooling, right-size the model, cap context. Validate: clocks hold and tokens/sec recovers.

🔍 Troubleshooting — GPU low but requests slow. Problem: latency is bad while GPU utilization is low. Likely cause: a CPU bottleneck starving the GPU (Part 8 pattern), or gateway/network latency. Check: node CPU; gateway logs and latency; the request path. Fix: add CPU/tune pre-processing; resolve the edge/network latency. Validate: GPU utilization rises and latency drops under the same load.

🔍 Troubleshooting — Rate limiter blocks legitimate traffic. Problem: real clients get throttled (429). Likely cause: the limit is set below normal usage. Check: actual request rates vs the configured limit. Fix: raise the limit to a measured, defensible value; scope it per client. Validate: legitimate traffic passes; abuse is still capped.

🔍 Troubleshooting — Replicas can’t schedule. Problem: scaling up leaves Pods Pending. Likely cause: more replicas than GPUs — replicas don’t create GPUs. Check: kubectl describe node GPU allocatable vs requested. Fix: add a GPU node, or scale replicas to the number of free GPUs. Validate: every replica lands a GPU and runs.

🔍 Troubleshooting — Monitoring shows errors but logs are empty. Problem: the dashboard shows failures with nothing in the Pod logs. Likely cause: the failure is upstream of the Pod — gateway, TLS, or auth rejecting requests. Check: gateway logs and status codes; TLS/auth at the edge. Fix: resolve the edge-layer failure the gateway logs name. Validate: error rate drops and gateway logs show 200s.

Hands-On Lab: Build a Secure AI Inference API

🧪 Hands-On Lab — Turn the Part 8 observable cluster into a secure, authenticated, TLS-terminated inference API. Do these in order from a machine with kubectl/helm access to ai-control01.

1. Confirm the cluster and GPU. kubectl get nodes -o wide; nvidia-smi on ai-node01 (AMD: rocm-smi).
2. Smoke-test vLLM on the host. docker run --gpus all -p 8000:8000 --ipc=host vllm/vllm-openai:latest --model <model>; curl /v1/models.
3. Create the model-cache PVC. Apply vllm-model-cache; confirm it Bound.
4. Create the ConfigMap. Apply vllm-config with your model name and args.
5. Generate and store the API key. kubectl create secret generic vllm-api-key --from-literal=api-key="<YOUR_API_KEY>" from a strong random value.
6. Deploy vLLM. Apply the Deployment (1 replica, GPU limit, /dev/shm emptyDir, cache mount, config + secret).
7. Add the probes. Include the startupProbe, and readiness/liveness on /health:8000.
8. Watch it come up. kubectl get pods -w; kubectl logs -f to watch the model load to Ready.
9. Create the Service. Apply the ClusterIP vllm Service on 8000; confirm kubectl get endpoints vllm is populated.
10. Test in-cluster. From a temp Pod, curl http://vllm:8000/v1/models with the key.
11. Install a gateway. Install Envoy Gateway per its current docs; confirm the GatewayClass exists.
12. Install cert-manager. Helm install with crds.enabled=true and config.enableGatewayAPI=true.
13. Create the ClusterIssuer. Apply the Let’s Encrypt issuer with a placeholder email (or plan a self-signed cert if you own no domain).
14. Create the Gateway. Define an HTTPS/443 listener referencing the TLS cert for your hostname.
15. Verify the certificate. kubectl get certificate; confirm the TLS Secret is provisioned.
16. Create the HTTPRoute. Route your hostname’s /v1 to the vllm Service on 8000.
17. Test HTTPS end to end. Curl https://<host>/v1/chat/completions with the key → 200 + completion.
18. Prove auth works. Repeat with no key → 401; with a wrong key → 401.
19. Add rate limiting. Configure a per-client limit at the gateway; confirm excess requests get 429.
20. Add request-size limits. Cap body/context to a value sized from your VRAM.
21. Wire monitoring. Confirm Prometheus scrapes vLLM’s /metrics (queue depth, KV cache, running/waiting).
22. Record a baseline. Fill one row of the performance table from a light live request.
23. Run a gradual load test. Ramp concurrency against your own endpoint with a small script.
24. Observe the correlation. Watch requests, GPU utilization, queue depth, and TTFT together during the ramp.
25. Prove the GPU limit. Scale to replicas: 2 on one GPU; watch the second Pod sit Pending; scale back.
26. Check logging + privacy. Confirm gateway logs carry request IDs and status — and do not capture prompt bodies.
27. Walk the security checklist. HTTPS, auth, rate limits, ClusterIP-only Service, RBAC, sized limits.
28. Prove persistence. Delete the vLLM Pod; confirm the recreated Pod finds the cached model (no re-download).
29. Document the API. Record the hostname, endpoints, auth method, model, limits, and storage layout.
30. Save the configuration. Version all manifests (no secret values) under a kubernetes/inference directory.

  ┌──────────────────────────────────────┐
  │ SUCCESS: secure inference API         │
  │                                       │
  │  vLLM on GPU ......... Ready       ✓   │
  │  HTTPS (cert-manager)  valid cert  ✓   │
  │  Authentication ...... key + 401   ✓   │
  │  Rate limiting ....... 429 on abuse ✓  │
  │  Gateway API ......... routing     ✓   │
  │  Native /metrics ..... scraped     ✓   │
  │  Model persists ...... on delete   ✓   │
  └──────────────────────────────────────┘

Where the Build-Along Stands

  AI platform build-along
  ------------------------------------
  Ubuntu + GPU + Docker ...... [done]
  Local LLM service .......... [done]
  Kubernetes cluster ......... [done]
  Prometheus + Grafana ....... [done]
  Secure inference API ....... [done]
  HTTPS + TLS ................ [done]
  Authentication ............. [done]
  Rate limiting .............. [done]
  ------------------------------------
  Production automation ...... [next]

You now have what Part 8 could only describe: a real inference service. Applications outside the cluster reach it over HTTPS, authenticate with a key, are rate-limited so no one starves the GPU, and every request is load-balanced to a Ready vLLM Pod whose native metrics feed the monitoring you already built.

From Inference Server to Production Platform

There is still a gap between “it works” and “it’s a platform.” Ask yourself the honest operator questions:

  Can you rebuild this from scratch?
  Is any of it in version control?
  Does a Pod recover a node failure?
  Do updates roll out safely, and roll back?
  Is the whole thing reproducible?

Right now the answer to most of those is “because I set it up by hand.” That’s a lab, not a platform. Turning it into infrastructure you can consistently rebuild, update, recover, secure, and operate — with Infrastructure as Code, configuration management, CI/CD, GitOps, backups, and disaster recovery — is exactly the capstone. That is Part 10.

GPU Hardware to Consider for AI Inference

Sizing hardware for an inference server is a different question from sizing a learning node, because now real clients and concurrency drive the requirements. Work through these criteria before looking at any card — they point at the recommended GPUs below, grouped from a learning card up to a dedicated development platform:

• Model size and VRAM — the model and its KV cache must fit; concurrency eats VRAM, so a server needs headroom a single-user experiment doesn’t.
• Concurrency — how many simultaneous requests you must serve shapes VRAM and the number of GPUs, not just raw speed.
• Context length — longer contexts consume more KV-cache memory per request.
• Latency vs throughput — an interactive API optimizes for low latency; a batch service for total throughput. They pull against each other, and the right card differs.
• Power and cooling — a server GPU runs hot under sustained load; provision PSU headroom and airflow, or throttling will cap your throughput.
• Budget — more or bigger GPUs cost real money to buy and run; size to measured demand, not the biggest card available.

❗ Important — Do not assume a given model “fits” a given GPU without checking the model’s published requirements against the card’s VRAM, plus room for the KV cache at your target concurrency and context. There is no universal answer, and a model that fits for one user can OOM under load. Verify before you buy, and validate any card below against your measured workload rather than a spec sheet.

What You Learned

• The difference between a local LLM experiment and an inference server that serves untrusted clients — and that the second is an infrastructure problem, not a model one.
• The full inference-server architecture: client → HTTPS → gateway (auth + rate limit) → Service → GPU Pods → storage, with trust checked at the top and traffic flowing down.
• Why this build uses vLLM — continuous batching, high concurrency, and a native Prometheus /metrics endpoint — while Ollama stays fine for local use.
• What an OpenAI-compatible API (/v1/*, /health, /metrics on port 8000) buys you, and where compatible is not identical.
• How to deploy vLLM to Kubernetes with a GPU request, a model-cache PVC, a ConfigMap, a Secret (base64 is encoding, not encryption), /dev/shm, and probes tuned for a slow model load.
• Why ingress-nginx is retired and how to expose the service with the Gateway API (GatewayClass → Gateway → HTTPRoute), cert-manager TLS, and authentication that never leaves the GPU open.
• How rate limiting and request-size limits protect GPU capacity, why replicas don’t create GPU capacity, and how GPU-aware scaling works on queue depth and latency — not CPU.
• How to reason about SLOs, run a safe gradual load test, read the saturation correlation live, and log requests without leaking prompts.

Next Lesson

Build a Production AI Platform on Ubuntu 26.04 — Part 10 is the capstone: it takes everything from Parts 1–9 and makes it operate like production — Infrastructure as Code with Terraform or OpenTofu, configuration management with Ansible, CI/CD and GitOps, RBAC and NetworkPolicy, backups and disaster recovery, so the whole platform is reproducible, recoverable, and defensible. See it at /ubuntu-ai/production-ai-platform-ubuntu-26-04/.

Until then, deepen the layers this lesson rests on: the Kubernetes & Helm guides for the gateway and scheduling, the security hardening guides for auth and exposure, and the Nginx guides for reverse-proxy fundamentals. Revisit Part 7 — Kubernetes for AI Workloads for the cluster this exposes and Part 8 — Monitoring Ubuntu AI Infrastructure for the metrics that drive scaling. If the host GPU misbehaves, go back to the NVIDIA and AMD ROCm lessons. New to the series? Start at the Ubuntu 26.04 AI Infrastructure overview.