Monitoring Ubuntu AI Infrastructure

By the end of Part 7 you had a GPU-aware Kubernetes cluster — ai-control01 scheduling work onto ai-node01, an LLM Deployment behind a Service, models on a persistent volume, and health probes gating traffic. It runs. But “it runs” is a claim you cannot yet defend. This lesson makes the cluster observable, so you can answer the only question that matters in operations: is this infrastructure healthy, overloaded, failing, or underutilized?

What You’ll Learn

Why AI observability is different from traditional server monitoring — and what extra signals GPUs and inference add
The observability layers of an AI platform, and why an incident can start at any one of them
The difference between metrics, logs, and events, and which question each one answers
The Prometheus + Grafana architecture — pull-based scraping, time-series, PromQL, and dashboards
How to install the whole stack with the verified kube-prometheus-stack Helm chart
How to monitor the Ubuntu host — CPU, RAM, disk, network — and spot a CPU bottleneck starving the GPU
How to monitor Kubernetes — node status, Pod restarts, Deployments, and Pending pods
How to monitor NVIDIA GPUs with DCGM Exporter and the verified DCGM metric names
How the AMD path differs, and how to reason about it without inventing metric names
How to monitor AI inference — request rate, latency, time-to-first-token, tokens/sec, queue depth
How to build a Grafana dashboard, write basic PromQL, and set alerts on problems, not noise
How to correlate metrics to diagnose real incidents instead of guessing from a single number
How to secure the monitoring stack so it does not become an information leak

This is where the cluster stops being a black box and starts telling you the truth about itself.

Why AI Observability Is Different

Every server needs the classic four signals: CPU (is the processor busy?), RAM (is memory exhausted?), disk (is storage full or slow?), and network (is traffic flowing?). You already know these from running any Linux box. They still matter here — a full root filesystem or a saturated CPU will take an AI service down exactly like any other. But they are no longer sufficient.

An AI platform adds two whole domains the traditional four say nothing about. The first is the GPU: its utilization, VRAM usage, temperature, power draw, clock speeds, and hardware errors. A server can show 5% CPU and 20% RAM while a $2,000 accelerator sits either pinned at its thermal limit or completely idle — and the classic four would not tell you which. The second is inference: model-load time, request latency, time-to-first-token, tokens per second, concurrency, and queue depth. These describe whether users are being served well, which is the actual point of the platform.

  Traditional monitoring
      CPU · RAM · Disk · Net
            +
  GPU monitoring
      util · VRAM · temp · power · errors
            +
  Kubernetes monitoring
      nodes · pods · restarts · scheduling
            +
  Inference monitoring
      latency · TTFT · tokens/sec · queue
      = AI observability

Read that as an addition, not a replacement. AI observability is traditional + GPU + Kubernetes + inference. Drop any layer and you get a blind spot big enough to lose an incident in — a healthy host with a throttling GPU, a busy GPU serving nobody, a fast model behind a crash-looping Pod. This lesson builds the stack that watches all four at once.

The Observability Layers

An AI request passes through a tall stack, and a failure can originate at any level of it. Reading the stack top to bottom is the discipline that keeps you from “restart and hope.”

  Application / client
        │
  Inference metrics (API)
        │
  Pods (containers)
        │
  Kubernetes (scheduler)
        │
  GPU (device)
        │
  CUDA / ROCm (driver)
        │
  Ubuntu 26.04 (OS)
        │
  Hardware

CUDA is NVIDIA’s GPU compute platform; ROCm is AMD’s equivalent — the driver-level software that lets code actually run on the accelerator. When latency spikes, the cause could be a saturated GPU (device layer), a driver that lost the card after a reboot (CUDA/ROCm layer), a Pod that got OOM-killed (Kubernetes layer), a full disk (Ubuntu layer), or simply ten times the usual traffic (application layer). Observability’s job is to give you a signal at every one of these layers so you can tell them apart. A monitoring stack that only watches the top or only watches the bottom will leave you diagnosing by intuition.

Metrics vs Logs vs Events

Three different data types answer three different questions. Confusing them is a common early-career trap, so keep the roles distinct:

Metrics are numbers sampled over time — GPU utilization, request latency, memory used. They are cheap to store and fast to query, and they tell you that something is wrong (“latency tripled at 14:02”). Metrics are for detection and trend, not for the full story.
Logs are text lines a program emits — an Ollama model-load message, a CUDA out-of-memory error, an HTTP 500 stack trace. They may tell you why something is wrong, but only if the program bothered to log it. Logs are for the detail behind a metric.
Events are Kubernetes’ record of what the cluster did — “scheduled Pod,” “pulled image,” “killed container: OOMKilled,” “liveness probe failed.” They explain the orchestration layer’s decisions.

  Metric  →  "latency tripled at 14:02"
  Logs    →  "CUDA out of memory" in the pod
  Event   →  "OOMKilled; restarted container"

The workflow is almost always metric → log → event, in that order. A dashboard shows you the when and where; kubectl logs shows you the application’s side of the why; kubectl get events and kubectl describe show you Kubernetes’ side. This lesson focuses on metrics because that is what Prometheus and Grafana do — but the preview of centralized logging later, and the Part 7 kubectl commands, are the other two legs of the stool.

The Monitoring Architecture

The primary stack for this entire lesson is Prometheus (collects and stores metrics) plus Grafana (visualizes them), with Alertmanager (routes alerts) alongside. This is the de facto standard for Kubernetes and GPU monitoring, and it is what every dashboard, exporter, and alert below assumes.

  node-exporter ─┐
  kube-state-    ├─► Prometheus ─► Grafana
   metrics       │        │        (dashboards)
  GPU exporter  ─┘        │
                          └─► Alertmanager ─► notify

The shape is: many exporters each expose metrics, one Prometheus scrapes and stores them all, Grafana reads Prometheus to draw dashboards, and Alertmanager takes alerts Prometheus fires and turns them into notifications. Everything that follows plugs into this diagram — the host exporter, the Kubernetes exporter, and the GPU exporter are all just more boxes on the left feeding the same Prometheus.

🤖 AI Infrastructure Tip — Some teams swap Prometheus’s storage for VictoriaMetrics or Thanos when retention and cardinality grow beyond a single Prometheus. That is a scaling choice, not a different model — the scrape-and-store-and-query workflow is identical, and everything in this lesson transfers. Start with Prometheus; reach for a long-term store only when you have measured a real need.

How Prometheus Works

Prometheus is a time-series database with a scraper built in. Four ideas cover almost everything you need operationally:

Scrape targets and /metrics. Prometheus does not wait for data to be sent to it. On a fixed interval it makes an HTTP request to each target’s /metrics endpoint and reads whatever plain-text metrics that endpoint exposes. This is the pull model — Prometheus reaches out, exporters just publish.
Time-series. Each sample is a number tied to a metric name and a timestamp, appended to a series. That is what lets you ask “what was GPU utilization at 14:02?” and “what is its trend over the last hour?”
Labels. Metrics carry key/value labels that identify which thing they describe — gpu="0", node="ai-node01", pod="ollama-xxxx". Labels are how one metric name covers every GPU and pod, and how you filter to the one you care about.
PromQL. You query with PromQL, Prometheus’s query language, to select series, filter by label, and compute rates and aggregates.

  Exporter  ──►  /metrics  ──►  Prometheus
   (publish)     (HTTP)          (pull + store)

Service discovery is the last piece that makes this manageable in Kubernetes: instead of hand-listing every target, the Prometheus Operator watches for ServiceMonitor objects and automatically starts scraping any exporter that ships one. When you add the GPU exporter later, you do not edit Prometheus — its ServiceMonitor tells Prometheus to scrape it. Remember the direction of the arrow: Prometheus pulls. If a metric is missing, the first question is always “can Prometheus reach that target’s /metrics?”

How Grafana Works

Grafana is the visualization layer, and the single most important thing to understand about it is what it does not do: Grafana does not collect or store metrics. It queries a data source — here, Prometheus — and draws the results as graphs, gauges, and tables.

  Prometheus ──► Grafana ──► dashboards
   (data)         (query)     (you look)

That division matters when you debug. If a Grafana panel says “No Data,” the problem is almost never Grafana itself — it is that Prometheus does not have the series, or the panel’s PromQL query is wrong. Grafana is a window onto Prometheus; a clean window showing an empty room means the room is empty, not that the window is broken. Dashboards are just saved collections of PromQL queries with a chosen visualization, which is why you can import a community dashboard as a single JSON file and have it work instantly against your Prometheus.

Installing the Monitoring Stack

Rather than install Prometheus, Alertmanager, Grafana, node-exporter, kube-state-metrics, and the Operator one at a time, use the community Helm chart that bundles them: kube-prometheus-stack. Helm was introduced in Part 7; this is one command’s worth of it.

helm repo add prometheus-community \
  https://prometheus-community.github.io/helm-charts
helm repo update
helm install kube-prom \
  prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace

The first two lines register the chart repository and refresh the local index. The helm install line deploys the whole stack into a new monitoring namespace: Prometheus (with the Operator that manages ServiceMonitors), Alertmanager, Grafana (pre-wired to Prometheus as its data source), node-exporter (as a DaemonSet, one per node), and kube-state-metrics. Watch it come up:

kubectl get pods -n monitoring

example output
NAME                                    READY   STATUS
kube-prom-grafana-7d...                 3/3     Running
kube-prom-kube-state-metrics-...        1/1     Running
kube-prom-prometheus-node-exporter-a    1/1     Running
kube-prom-prometheus-node-exporter-b    1/1     Running
prometheus-kube-prom-...-0              2/2     Running
alertmanager-kube-prom-...-0            2/2     Running

Two node-exporter pods, one for ai-control01 and one for ai-node01, confirm the DaemonSet reached both nodes. To open Grafana without exposing it publicly, port-forward it to your workstation:

kubectl port-forward -n monitoring \
  svc/kube-prom-grafana 3000:80

Now http://localhost:3000 reaches Grafana over the tunnel only. The chart sets an initial admin password (retrievable from its Secret); change it immediately — we return to that in the security section.

⚠️ Warning — kubectl port-forward is for your access during setup, not a production exposure method. It runs only while the command runs and only for you. Never make Grafana or Prometheus reachable from an untrusted network by publishing them with a plain Service or Ingress and no authentication — the security section explains why metrics endpoints are sensitive.

Monitoring the Ubuntu Host

Underneath Kubernetes and the GPU sits plain Ubuntu, and its health is still the foundation. node-exporter — already running from the chart — exposes host metrics from each node’s kernel: CPU, load, memory, swap, filesystem, disk I/O, network, and uptime. These are the same fundamentals you would watch on any Linux server, now scraped automatically.

CPU and Load

Watch per-node CPU utilization and load average. The trap specific to AI infrastructure is assuming a slow model means a slow GPU. It often means the opposite:

  CPU: 100%   GPU: 30%
  → the GPU is STARVED, not broken

Inference is not pure GPU work. Tokenization, request handling, data movement, and pre/post-processing happen on the CPU, and if the CPU is pinned at 100% it cannot feed the GPU fast enough — so GPU utilization drops and latency rises. A dashboard that shows both side by side turns this from a mystery into an obvious diagnosis. High CPU with low GPU is a CPU bottleneck, and no bigger GPU will fix it.

System RAM and the OOM Killer

Distinct from VRAM (memory on the GPU), system RAM is what the host and containers use. When it runs out, Linux invokes the OOM (out-of-memory) killer, which terminates a process to reclaim memory — often your inference container, which then shows up in Kubernetes as an OOMKilled restart. Track memory used, available, and swap. Rising swap usage on a node meant to serve low-latency inference is itself a warning: swapping trades memory for crippling slowness.

Disk

Model downloads are large, and the classic failure is filling the root filesystem so Docker, containerd, or Kubernetes can no longer write. Monitor free space per mount and disk I/O. There is no universal model size to plan around — a quantized small model and a large full-precision one differ by orders of magnitude — so alert on free space remaining, not on an assumed footprint. When df on a node approaches full, image pulls fail, containers refuse to start, and the kubelet starts evicting pods.

Network

An AI platform generates several distinct traffic flows: model downloads pulling gigabytes from the internet, storage traffic to persistent volumes, API requests and responses to inference services, intra-cluster pod-to-pod traffic, and Prometheus’s own scrape traffic. Watch throughput and errors per interface. A saturated or erroring NIC shows up as slow model pulls, laggy API calls, or — subtly — scrape timeouts that make other metrics look like they vanished.

🛠️ DevOps Tip — Before you ever look at a GPU panel, glance at the host panels. A surprising share of “the GPU is slow” tickets are a full disk, a pinned CPU, or a swapping node. The cheapest layer to rule out is the one you already know how to read.

Monitoring Kubernetes

The cluster layer is watched by kube-state-metrics (the state of Kubernetes objects — nodes, pods, deployments) and the kubelet/cAdvisor metrics (actual container resource usage). Together they answer “is the cluster doing what I declared?”

Key things to put on a dashboard and understand:

Node status. A node going NotReady is a major event on a two-node cluster — if ai-node01 drops, you lose all your GPU capacity at once. This is not a slow degradation; it is a cliff.
Pod status and restarts. A steadily climbing restart count is the single most useful cluster signal. Repeated restarts almost always mean one of: an OOMKill (memory limit too low), a crash (bad config or missing dependency), a failing probe (the readiness/liveness lesson from Part 7), a GPU access problem, or a slow model load tripping an aggressive liveness probe.
Deployments and replicas. Track desired vs available replicas. If you asked for one inference replica and zero are available, the service is down regardless of what any host metric says.
Pending and Failed pods. A Pending GPU pod is the classic Kubernetes-AI failure — no GPU advertised, all GPUs already allocated, a nodeSelector mismatch, a taint without a toleration, insufficient CPU/RAM, or a Pending PVC. Part 7 walked this tree; the dashboard just makes it visible before a user complains.

Correlate these with the host metrics: an OOMKilled restart lines up with a system-RAM spike; a NotReady node lines up with a host that stopped reporting. That correlation is the whole game, and we return to it near the end.

Monitoring NVIDIA GPUs

The classic four signals cannot see the GPU. To fix that on NVIDIA hardware you deploy DCGM Exporter — a metrics endpoint built on NVIDIA’s DCGM (Data Center GPU Manager) telemetry library. It reads the GPU’s own counters and publishes them as Prometheus metrics.

  GPU ─► DCGM ─► DCGM Exporter ─► Prometheus ─► Grafana

The GPU Operator from Part 7 can deploy DCGM Exporter for you; you can also install it standalone with Helm:

helm repo add gpu-helm-charts \
  https://nvidia.github.io/dcgm-exporter/helm-charts
helm repo update
helm install dcgm-exporter \
  gpu-helm-charts/dcgm-exporter -n gpu-operator

The exporter ships a ServiceMonitor, so the Prometheus Operator discovers and scrapes it with no manual config edit. For visualization, NVIDIA publishes an official Grafana dashboard — ID 12239 — that you import by ID rather than building from scratch.

The metric names below are the verified DCGM names. Use them exactly, and know that they can vary by exporter version, so confirm against the exporter you deployed:

Metric	Meaning
`DCGM_FI_DEV_GPU_UTIL`	GPU utilization (%)
`DCGM_FI_DEV_FB_USED`	Framebuffer (VRAM) used (MiB)
`DCGM_FI_DEV_FB_FREE`	Framebuffer (VRAM) free (MiB)
`DCGM_FI_DEV_GPU_TEMP`	GPU temperature (°C)
`DCGM_FI_DEV_POWER_USAGE`	Power draw (W)
`DCGM_FI_DEV_SM_CLOCK`	Streaming-multiprocessor clock (MHz)

These fall into the categories you actually reason about: utilization, VRAM (used/free), temperature, power, clock, plus errors, PCIe throughput, and throttling state. Each answers a different operational question.

GPU Utilization vs VRAM — Two Different Numbers

The most common GPU misreading is treating utilization and VRAM as the same thing. They are independent, and the interesting states are the mismatches:

  FB_USED high + GPU_UTIL low
    → model loaded, barely working
      (loaded-but-idle)

  FB_USED low + GPU_UTIL high
    → small model, running hard

  FB_USED high + GPU_UTIL high
    → fully engaged

A GPU showing 95% VRAM used and 5% utilization is loaded but idle — the model occupies memory but almost no inference is happening. That is not “busy”; it is expensive memory holding a model nobody is querying, and it is exactly the pattern the cost section flags. Utilization tells you if the compute units are working; VRAM tells you how much memory the model and its KV cache occupy. Always read them together.

Temperature

There is no universal danger temperature — safe operating ranges differ by GPU model and generation. The right approach is to establish your card’s idle and loaded baseline, then use the vendor’s published thresholds for that specific GPU as the limit. Watch the trend and the delta from baseline, not a number you read in someone else’s guide. A card that normally sits at a given loaded temperature and suddenly runs much hotter is telling you something (dust, failed fan, worse airflow) regardless of the absolute value.

Power

Power draw (DCGM_FI_DEV_POWER_USAGE) is both an operational signal (is the card working?) and a capacity signal (are we near the card’s or the PSU’s limit?). Avoid universal wattage rules — like temperature, meaningful thresholds are per-card. Power that stays near the card’s rated maximum under sustained load is normal for a busy accelerator; power pinned at max while utilization is low is worth a look.

Thermal Throttling

Throttling is the mechanism that ties temperature to performance: when a GPU gets too hot, it reduces its clock speed to protect itself, which lowers throughput. So a chain of “high temperature → clock drops (DCGM_FI_DEV_SM_CLOCK falls) → tokens/sec falls” is a cooling problem masquerading as a performance problem. If inference slows and you see temperature high with clocks dropping, no software tuning will help until you fix airflow or reduce sustained load.

Monitoring AMD GPUs

The AMD path is separate and uses AMD’s device-metrics-exporter, which is built on amd-smi (AMD’s system-management interface). It exposes AMD GPU telemetry to Prometheus the same architectural way DCGM does for NVIDIA: GPU → exporter → Prometheus → Grafana.

Two honesty notes carry forward from Part 4. First, do not assume the old approach of scraping rocm-smi text output — the current, supported path is the device-metrics-exporter. Second, verify the exporter’s metric names against AMD’s own documentation rather than assuming they mirror DCGM’s; they do not. Because I will not invent AMD metric names, reason about them by category instead: AMD’s exporter provides the same kinds of signals — utilization, VRAM used/free, temperature, power, clock, and errors — and you build the same dashboards and alerts on them once you have confirmed the exact names for your exporter version.

❗ Important — The ROCm-on-26.04 caveat from Part 4 still applies. ROCm officially supports specific Ubuntu versions, and a brand-new release like 26.04 may not be on the supported list yet. Verify your GPU and your OS against the AMD compatibility matrix before you commit a node — and before you assume the exporter will run cleanly on it.

A Vendor-Neutral View

Notice that despite two entirely different exporters, the questions are identical. Is the GPU busy? How much VRAM is left? Is it too hot? Is it near its power limit? Are the clocks where they should be? Whether the metric is DCGM_FI_DEV_GPU_UTIL or an AMD utilization gauge, the dashboard asks the same thing. Build your GPU dashboard around those questions and it survives a hardware swap — you change the queries, not the way you think about the platform. That is the payoff of understanding categories over memorizing metric names.

AI Application and Inference Metrics

Everything so far is infrastructure — and infrastructure metrics are not enough. Here is the trap that catches teams who stop at the GPU dashboard:

  GPU_UTIL = 100%   ← looks great
  but latency = 30s, errors rising
  → users are NOT happy

100% GPU utilization does not mean users are being served well. It might mean the GPU is efficiently busy, or it might mean it is overwhelmed and every request is queued behind a wall of others. To know which, you must measure the inference layer — the application’s own view of the work:

Request rate — requests per second hitting the inference endpoint.
Error rate — failed requests and model errors.
Latency — total time to complete a request.
Time-to-first-token (TTFT) — how long until the first output token appears; this dominates perceived snappiness.
Tokens per second — generation throughput once streaming starts.
Queue depth — how many requests are waiting for GPU time.
Model load time — how long a cold model takes to become ready.
Concurrent requests — how many are in flight at once.

The critical constraint: use the metrics the runtime actually exposes. Do not fabricate an endpoint. Some runtimes ship a native Prometheus /metrics endpoint — vLLM, for example, exposes inference metrics directly. Others, like Ollama, do not natively expose a Prometheus endpoint, and pretending otherwise sets you up to chase a /metrics URL that returns nothing. When a runtime lacks native metrics, your honest options are:

run a sidecar or standalone exporter that measures requests (for example by proxying or probing the API),
put the service behind a gateway/proxy that emits request-rate, latency, and error metrics,
or choose a runtime (like vLLM) that exposes /metrics when inference observability is a hard requirement.

🤖 AI Infrastructure Tip — Tokens/sec is a genuinely useful number, but it is not comparable across systems. It depends on the model, the GPU, the quantization, the prompt, the context length, the batch size, the concurrency, and the runtime. A tokens/sec figure only means something against your own baseline on the same setup. Never compare it between unrelated systems, and never copy a number from a guide as an expectation.

Two more relationships worth internalizing. TTFT vs remaining tokens: a request’s total latency is roughly the time to the first token plus the time to stream the rest, and those are tuned differently — TTFT is dominated by prompt processing and scheduling, streaming by raw generation speed. And the latency vs throughput tradeoff: batching more requests together raises total throughput (tokens/sec across everyone) while raising each individual user’s latency. There is no single “fast” — there is fast-for-one-user and high-total-throughput, and they pull against each other. Never fabricate these numbers; measure them on your stack.

Golden Signals for AI

Google’s SRE “golden signals” map cleanly onto inference and give you a compact mental checklist for what a service dashboard must show:

Golden signal	For AI inference
Traffic	Request rate to the inference endpoint
Latency	Response time and time-to-first-token
Errors	Failed requests and model errors
Saturation	GPU utilization, VRAM, CPU

If your dashboard answers all four, you can characterize the service’s health in one glance: how much work is arriving, how fast it is served, how often it fails, and how close to full the resources are. Missing any one leaves a question you cannot answer during an incident.

Building a Grafana Dashboard

A good AI-infrastructure dashboard is organized top-down, from cluster summary to inference detail, so an on-call engineer reads it like a newspaper — headline first, then drill in. Group it into sections:

  ┌─────────────────────────────────────┐
  │ Cluster Overview  nodes up · alerts  │
  ├───────────────────┬─────────────────┤
  │ Ubuntu Servers    │ GPU Overview     │
  │ CPU RAM Disk Net  │ util VRAM temp   │
  ├───────────────────┼─────────────────┤
  │ K8s AI Workloads  │ LLM Inference    │
  │ pods restarts     │ rate TTFT tok/s  │
  │ replicas pending  │ errors queue     │
  └───────────────────┴─────────────────┘

Build it from the metrics you actually have — the node-exporter, kube-state-metrics, and DCGM series above, plus whatever the inference layer genuinely exposes. Do not add panels for invented metric names; an empty panel is worse than no panel. No custom Grafana plugin is required — the standard graph, gauge, and stat panels cover all of this against Prometheus.

PromQL Basics

You do not need advanced PromQL to build the dashboard. A handful of patterns cover almost every panel:

Selector — name a metric to get all its series:

DCGM_FI_DEV_GPU_UTIL

Label filter — narrow to one thing with a label matcher:

DCGM_FI_DEV_GPU_UTIL{gpu="0"}

Rate — convert an ever-increasing counter into a per-second rate over a window. This is the standard way to turn a request counter into a request rate:

rate(some_requests_total[5m])

Aggregation — collapse many series into one with sum, avg, or max, optionally grouped by a label:

avg(DCGM_FI_DEV_GPU_UTIL) by (node)

Read each one in words: the second is “GPU 0’s utilization,” the third is “requests per second averaged over five minutes,” the fourth is “average GPU utilization per node.” That vocabulary — select, filter by label, rate a counter, aggregate — is enough to write every panel on the dashboard above, using only metric names you have confirmed exist.

Alerting on Problems, Not Metrics

Dashboards are for when you are looking; alerts are for when you are not. The discipline that separates useful alerting from alert fatigue is simple: alert on problems, not on every metric. GPU utilization > 80% is usually normal — it is what you bought the GPU for — so alerting on it just trains you to ignore alerts.

The flow is: a Prometheus alert rule (a PromQL expression that must hold true for some duration) fires, Prometheus hands it to Alertmanager, and Alertmanager routes it to a notification channel.

  Prometheus rule ─► Alertmanager ─► notification
   (condition true    (dedupe,       (channel)
    for N minutes)      route)

Good alert categories, with thresholds you set from your own baselines rather than universal numbers:

Infrastructure — node down, root filesystem nearly full, memory exhaustion imminent.
Kubernetes — an AI Pod crash-looping, a Deployment with unavailable replicas, a GPU Pod stuck Pending too long.
GPU — the GPU or its exporter has gone missing (no metrics), temperature sustained abnormally high for this card, VRAM near exhaustion.
AI service — inference endpoint down, elevated error rate, latency degraded beyond baseline, queue depth growing.

Note what is not in that list: a hard-coded universal GPU temperature or power limit. Those are per-card; encode your baseline-derived threshold, or the vendor’s spec for that model, not a number from a tutorial. For the notification target, use a placeholder receiver — never put real email or Slack credentials in a dashboard or rule file:

receivers:
  - name: "team-placeholder"
    # Configure a real receiver out-of-band.
    # Do NOT commit tokens or webhook URLs here.

⚠️ Warning — Idle-GPU alerting needs a careful hand. A GPU that is allocated, holding VRAM, and sitting at ~0% utilization is worth investigating — it may be loaded-but-unqueried, poorly scheduled, or abandoned — but do not wire an alert that auto-kills it. A model mid-load or a service between traffic bursts looks identical for a moment. Flag it for a human; never let automation reap a running AI workload on a single-metric hunch.

Cost, Utilization, and Performance Engineering

GPU utilization is not just a technical metric — it is a business metric. An accelerator is expensive to buy and to power, so the question “are we using what we paid for?” is a real one, and your utilization dashboard answers it. A cluster of GPUs averaging low utilization is money sitting idle; the fix might be consolidation, better scheduling, or simply serving more traffic to hardware you already own.

The engineering method is the same loop every time, and it depends entirely on having the metrics this lesson builds:

  measure → baseline → change one thing
     → measure again → keep or revert

You cannot claim an optimization worked without a before number. So establish a baseline first — and fill it in with your measurements, never fabricated ones:

Metric	Baseline	After change
GPU utilization (%)
VRAM used
Request latency
Time-to-first-token
Tokens/sec
Concurrent requests

Every cell is something you read off the dashboard on your hardware. A performance change validated against a real baseline is engineering; a change made because it “should be faster” is guessing. Measure, change one variable, measure again.

Correlating Metrics and Incident Walkthroughs

The skill that separates real diagnosis from restart-and-hope is correlation: reading several metrics together instead of reacting to one. Two verified walkthroughs show why.

Incident 1 — inference suddenly slow. A single metric misleads you: latency is up, so you suspect the GPU. But correlate:

  latency ▲   TTFT ▲   GPU_UTIL ▼   CPU 100%
  → the GPU is starved by a CPU bottleneck

TTFT and latency climb while GPU utilization drops and CPU sits at 100%. A GPU that is slow because it is overloaded would show high utilization; falling utilization with a pinned CPU means the GPU is being starved — the CPU cannot pre-process fast enough to feed it. The fix is on the CPU side (more CPU, better batching, offloading pre-processing), not a bigger GPU.

Incident 2 — a Pod won’t load its model. The Pod restarts; kubectl logs shows a GPU out-of-memory error at load. A single-metric reader concludes “the model is too big.” But correlate with the VRAM dashboard, which shows another process already holding most of the framebuffer:

  logs: "CUDA out of memory" at load
  FB_USED dashboard: 90% held by
    a DIFFERENT process
  → not too big — memory not free

The model is not oversized; the VRAM was not free because a stale or competing process was pinning it. The fix is to free that memory, not to shrink the model. In both cases the log or single metric points one way and the correlation points the right way.

This reinforces the Part 7 diagnostic exactly: a Grafana restart spike leads you to kubectl describe pod, which shows the liveness probe failing, which kubectl logs explains as a 90-second model load tripping an aggressive probe — so the fix is tuning the readiness/startup probe, not restarting harder. Metric points you to the layer; logs and events explain the layer; correlation stops you fixing the wrong thing.

A Note on Centralized Logging

Metrics tell you that and where; for the why at scale you eventually want centralized logging. On one node kubectl logs is enough, but across many pods on many nodes — some of which have already restarted and lost their logs — you want the logs shipped somewhere searchable.

  Pods ─► logs ─► central store ─► search
                  (Loki /
                   OpenSearch)

Tools like Grafana Loki or OpenSearch collect logs from every pod into one place you can query alongside your metrics. That is the conceptual preview — do not over-build it now. Today’s stack is metrics-first; centralized logging is the natural next layer once the number of pods outgrows kubectl logs.

Troubleshooting

Work every monitoring failure along one model — the same Workload → Metric Source → Exporter → Prometheus → Query → Grafana chain that Part 8 is built on. Most “missing metric” problems are a broken link somewhere on that path.

  Metric source ─► Exporter ─► Network
     ─► Prometheus ─► Query ─► Grafana

🔍 Troubleshooting — Prometheus can’t scrape node-exporter. Problem: host metrics are missing; the node-exporter target shows Down. Likely cause: the exporter pod isn’t running on that node, or a network/port issue blocks the scrape. Check: kubectl get pods -n monitoring -o wide for a node-exporter on each node; Prometheus → Status → Targets for the error. Fix: ensure the DaemonSet scheduled onto the node (taints/tolerations); resolve the network path. Validate: the target flips to Up and host panels populate.

🔍 Troubleshooting — Kubernetes targets missing. Problem: no pod/node/deployment metrics. Likely cause: kube-state-metrics isn’t running, or its ServiceMonitor isn’t being picked up. Check: kubectl get pods -n monitoring for kube-state-metrics; confirm its ServiceMonitor exists. Fix: reinstall/repair the chart component; confirm the Operator watches that namespace. Validate: kube_pod_status_phase and friends appear in Prometheus.

🔍 Troubleshooting — Grafana shows No Data. Problem: a panel is empty. Likely cause: the panel’s PromQL is wrong, the metric name doesn’t exist, or the data source is misconfigured — not Grafana itself. Check: run the panel’s query directly in Prometheus; verify the exact metric name. Fix: correct the query/metric name; confirm the Prometheus data source URL. Validate: the same query returns series in Prometheus, and the panel fills.

🔍 Troubleshooting — GPU metrics missing entirely. Problem: no DCGM_FI_* series at all. Likely cause: DCGM Exporter isn’t deployed, or its ServiceMonitor isn’t scraped. Check: kubectl get pods -n gpu-operator for the exporter; Prometheus Targets for the DCGM job. Fix: deploy the exporter (Helm/GPU Operator); confirm the ServiceMonitor is in a watched namespace. Validate: DCGM_FI_DEV_GPU_UTIL returns series.

🔍 Troubleshooting — NVIDIA exporter running but no GPU metrics. Problem: the exporter pod is up, yet values are absent or zero. Likely cause: the exporter can’t reach the GPU/driver (host driver issue, device not exposed to the pod), or a version mismatch changed metric names. Check: the exporter pod logs; confirm the node’s driver works (nvidia-smi on the host from Part 3); verify actual metric names the exporter emits. Fix: repair GPU access on the node; align your queries to the exporter’s real metric names. Validate: utilization moves when you run an inference.

🔍 Troubleshooting — AMD metrics missing. Problem: no AMD GPU series. Likely cause: device-metrics-exporter not deployed, ROCm/OS unsupported on the node, or wrong metric names assumed. Check: the exporter pod status/logs; the AMD compatibility matrix for GPU+OS; the exporter’s documented metric names. Fix: deploy the AMD exporter on a supported node; use the verified AMD metric names. Validate: AMD utilization/VRAM series appear.

🔍 Troubleshooting — A target shows Down. Problem: Prometheus Targets lists a job as Down. Likely cause: the exporter is unreachable — pod not running, wrong port, or network policy blocking the scrape. Check: Status → Targets for the exact error (connection refused, timeout, 404 on /metrics). Fix: address the specific cause the error names. Validate: the target returns to Up.

🔍 Troubleshooting — A dashboard panel shows No Data (but others work). Problem: one panel is empty while the rest populate. Likely cause: that panel’s specific query/metric name is wrong or the label filter matches nothing. Check: copy the panel query into Prometheus; loosen the label filter. Fix: correct the metric name or label matcher. Validate: the panel fills.

🔍 Troubleshooting — GPU metrics vanish after a node reboot. Problem: GPU series were fine, then disappeared after the node rebooted. Likely cause: the GPU driver or exporter didn’t come back cleanly after reboot. Check: on the node, does the host see the GPU (Part 3)? Is the exporter pod running again? Fix: restore the driver/exporter on boot; ensure the exporter DaemonSet reschedules. Validate: DCGM_FI_* series return and move under load.

🔍 Troubleshooting — Inference metrics unavailable. Problem: no request-rate/latency/TTFT series for the model service. Likely cause: the runtime doesn’t expose native Prometheus metrics (e.g. Ollama), or the endpoint isn’t scraped. Check: does the runtime actually publish a /metrics endpoint? Confirm before expecting data. Fix: add a sidecar/exporter or gateway that emits the metrics, or use a runtime (e.g. vLLM) that exposes /metrics. Validate: inference series appear and move with traffic.

🔍 Troubleshooting — Excessive cardinality. Problem: Prometheus slows or uses huge memory. Likely cause: a label with too many unique values (high cardinality) explodes the number of series. Check: which metric/label has an unbounded value set (per-request IDs, timestamps as labels). Fix: drop or relabel the offending high-cardinality label at scrape time. Validate: series count and memory drop to sane levels.

🔍 Troubleshooting — Prometheus disk grows fast. Problem: the Prometheus PVC fills quickly. Likely cause: too many series, too-short a scrape interval, or a retention window too long for the volume. Check: series count, scrape interval, and configured retention vs disk size. Fix: reduce cardinality, lengthen the scrape interval where fine-grained data isn’t needed, tune retention, or grow the volume. Validate: disk usage stabilizes within the PVC.

🔍 Troubleshooting — An alert fires constantly. Problem: an alert is always firing (and thus ignored). Likely cause: the threshold is set on a normal condition (e.g. GPU util > 80%) or the for: duration is too short. Check: is the condition actually a problem, or expected behavior? Fix: re-baseline the threshold to a real problem state; add/extend the for: duration. Validate: the alert fires only during genuine incidents.

🔍 Troubleshooting — An alert never fires. Problem: a known bad condition happened but no alert came. Likely cause: wrong metric name in the rule, threshold unreachable, or Alertmanager routing/receiver misconfigured. Check: evaluate the rule’s PromQL in Prometheus; inspect Alertmanager routing and the (placeholder) receiver. Fix: correct the expression/threshold; fix the route to a working receiver. Validate: deliberately trip the condition and confirm a notification.

🔍 Troubleshooting — A Grafana query is slow. Problem: a panel takes a long time to render. Likely cause: a heavy query over a long range, or high-cardinality data. Check: the query’s range and how many series it touches. Fix: narrow the range, pre-aggregate, or reduce cardinality at the source. Validate: the panel renders quickly.

🔍 Troubleshooting — GPU busy but inference poor. Problem: GPU utilization is high yet latency/tokens-sec are bad. Likely cause: thermal throttling (clocks dropped), oversized model spilling memory, a long context, or a saturated queue. Check: correlate DCGM_FI_DEV_GPU_TEMP and DCGM_FI_DEV_SM_CLOCK (throttling?), VRAM near full, and queue depth. Fix: address the specific correlated cause — cooling, a right-sized model, shorter context, or more capacity. Validate: clocks hold and latency/tokens-sec return to baseline.

The recurring lesson across all of these: when a metric is missing, walk the chain source → exporter → network → Prometheus → query → Grafana, and remember that in this stack new exporters are wired in through their ServiceMonitor/scrape config — if Prometheus never learned to scrape a target, no amount of dashboard fiddling will conjure the data.

Hands-On Lab: Build an AI Infrastructure Monitoring Stack

🧪 Hands-On Lab — Make the Part 7 cluster observable end to end. Do these in order from a machine with kubectl/helm access to ai-control01.

Confirm the cluster. kubectl get nodes -o wide — both ai-control01 and ai-node01 are Ready.
Add the Helm repo. helm repo add prometheus-community … and helm repo update.
Install the stack. helm install kube-prom prometheus-community/kube-prometheus-stack -n monitoring --create-namespace.
Watch it come up. kubectl get pods -n monitoring until Prometheus, Alertmanager, Grafana, kube-state-metrics, and one node-exporter per node are Running.
Confirm host coverage. Verify a node-exporter pod on both nodes with -o wide.
Open Grafana. kubectl port-forward -n monitoring svc/kube-prom-grafana 3000:80, then browse http://localhost:3000.
Change the admin password. Retrieve the initial password from its Secret, log in, and rotate it immediately.
Check Prometheus targets. In Grafana’s Explore (or Prometheus directly), confirm node and kube-state-metrics targets are Up.
Read host metrics. Query CPU, memory, and filesystem-free for each node; confirm real values.
Read Kubernetes metrics. Query pod status and restart counts; find your inference Deployment.
Deploy the GPU exporter. Install DCGM Exporter via Helm (or confirm the GPU Operator deployed it) into gpu-operator.
Verify GPU scrape. Confirm the DCGM target is Up and DCGM_FI_DEV_GPU_UTIL returns series.
Import the GPU dashboard. Import Grafana dashboard ID 12239 and point it at Prometheus.
Generate GPU load. Send an inference request to the Part 7 LLM Service and watch DCGM_FI_DEV_GPU_UTIL and DCGM_FI_DEV_FB_USED move.
Read VRAM vs util together. Confirm you can distinguish loaded-but-idle from actively-working.
Add inference visibility. Determine whether your runtime exposes /metrics; if not, note the sidecar/exporter/vLLM option rather than inventing an endpoint.
Build a dashboard. Create the five-section layout (Cluster / Ubuntu / GPU / K8s Workloads / Inference) from metrics you confirmed exist.
Write PromQL panels. Use selector, label filter, rate(), and avg … by for each panel.
Add alert rules. Create rules for node-down, rootfs-nearly-full, AI-pod-crash-looping, and inference-endpoint-down, with baseline-derived thresholds.
Wire a placeholder receiver. Point Alertmanager at a placeholder — no real credentials committed.
Record your baseline. Fill one row of the baseline table from live measurement.
Save the configuration. Version the Helm values, dashboard JSON (no secrets), and alert rules under /srv/ai/monitoring.

  ┌──────────────────────────────────────┐
  │ SUCCESS: the cluster is observable    │
  │                                       │
  │  Prometheus + Grafana ... running ✓   │
  │  Host metrics ........... both nodes ✓│
  │  Kubernetes metrics ..... live    ✓   │
  │  GPU metrics (DCGM) ..... moving  ✓    │
  │  Inference visibility ... honest  ✓   │
  │  Dashboard (5 sections) . built   ✓   │
  │  Alerts (placeholder) ... firing  ✓   │
  │  Baseline recorded ...... yours   ✓   │
  └──────────────────────────────────────┘

Securing the Monitoring Stack

Monitoring is powerful and therefore sensitive. A metrics endpoint or a Grafana instance describes your infrastructure in detail — node names, GPU models, workload names, capacity, and where the gaps are. That is a gift to an attacker.

⚠️ Warning — Do not expose Prometheus, Grafana, or Alertmanager to untrusted networks. Concretely: put Grafana behind real authentication and change the default admin password immediately; keep Prometheus and its /metrics endpoints off the public internet (they leak infrastructure details even without login); never store credentials, tokens, or webhook URLs in dashboard JSON or alert-rule files (use secrets/out-of-band config); secure Alertmanager’s receivers the same way; and apply Kubernetes RBAC so only the accounts that need monitoring access have it. Metrics are not “just numbers” — they are a map of your platform. Treat the map like the territory.

Where the Build-Along Stands

  AI platform build-along
  ------------------------------------
  Ubuntu + GPU + Docker ...... [done]
  Local LLM service .......... [done]
  Kubernetes cluster ......... [done]
  Prometheus + Grafana ....... [done]
  GPU monitoring (DCGM) ...... [done]
  Inference monitoring ....... [done]
  Alerting ................... [done]
  ------------------------------------
  Production inference ....... [next]

You can now answer the question this lesson opened with. Healthy? The host, cluster, and GPU panels say so. Overloaded? Saturation and queue-depth show it. Failing? Restarts, target-down, and error-rate alerts catch it. Underutilized? The GPU-util-as-business-metric view exposes it. The cluster tells you the truth now.

The End-State Platform

Here is everything you have built across the series, watched by everything you built today:

  ┌─────────────────────────────────────┐
  │ Kubernetes cluster                   │
  │                                     │
  │  ai-control01        ai-node01 (GPU)│
  │   scheduler           LLM Pod        │
  │   api-server          + GPU + PVC    │
  │        │                  │          │
  │        └── monitoring ────┘          │
  │      Prometheus · Grafana ·          │
  │            Alertmanager              │
  └─────────────────────────────────────┘

A two-node GPU cluster running a persistent, health-gated LLM, fully instrumented. That is a real platform. But it is still a lab platform: the inference endpoint is a ClusterIP Service reachable only inside the cluster, with no public exposure, no authentication, no TLS, no load balancing, and no autoscaling. Which raises the next problem exactly: how do we turn this lab into a secure, scalable, authenticated, load-balanced inference API that real applications outside the cluster can call safely? That is Part 9.

Monitor the GPU You Already Own

One principle to carry out of this lesson before you reach for a purchase order: monitor the GPU you already own before buying another one. More than a few “we need more GPUs” decisions dissolve under a utilization dashboard that shows the current cards averaging low utilization, or loaded-but-idle, or throttling for want of airflow. Observability is the cheapest capacity you can add — it often finds the capacity you already paid for. When the metrics genuinely show sustained saturation with a real queue, then the case for more hardware is one you can defend with numbers.

For going deeper on that reasoning — baselining, finding bottlenecks, and reading saturation like an engineer instead of guessing — the recommended books below (AI Systems Performance Engineering especially) are the primary companion to this lesson. If and when the data does justify more capacity, the small “Exploring Additional GPU Capacity” cards beneath this lesson are grouped as starting points — validate any card against your measured workload, not a spec sheet.

What You Learned

Why AI observability is traditional + GPU + Kubernetes + inference, and why dropping any layer creates a blind spot.
The observability layers of an AI platform, and that an incident can begin at any one of them.
The distinct roles of metrics (that/where), logs (why, if logged), and events (what Kubernetes did).
The Prometheus + Grafana + Alertmanager architecture — pull-based scraping, time-series, labels, PromQL, and the fact that Grafana only visualizes.
How to install the whole stack with kube-prometheus-stack, and monitor the Ubuntu host (including a CPU bottleneck starving the GPU) and Kubernetes (nodes, restarts, Pending pods).
How to monitor NVIDIA GPUs with DCGM Exporter and the verified DCGM metric names, read utilization vs VRAM, and recognize thermal throttling — plus the honest AMD path.
How to measure inference — latency, TTFT, tokens/sec, queue depth — using metrics the runtime actually exposes, never fabricated ones.
How to build a dashboard, write basic PromQL, alert on problems not noise, correlate metrics to diagnose real incidents, and secure the whole stack.

Next Lesson

Building an AI Inference Server on Ubuntu 26.04 — The observable lab cluster becomes a real service: we take the internal ClusterIP LLM and turn it into a secure, scalable, authenticated, load-balanced inference API that applications outside the cluster can call — with the monitoring you built today watching every request.

Until then, deepen the two pillars this lesson rests on: browse the Prometheus monitoring guides and the Grafana category for query and dashboard patterns, and walk the broader observability stack for how metrics, logs, and traces fit together. Revisit Part 7 — Kubernetes for AI Workloads for the cluster this instruments, and Part 6 — Running Local LLMs for the inference service underneath it. New to the series? Start at the Ubuntu 26.04 AI Infrastructure overview.