GitHub Actions GPU Workflows: CI/CD for CUDA, AI, and Machine Learning

Every lesson in this academy so far has run its CI on ordinary CPU runners, and for the vast majority of software that is exactly right. But a specific class of AI and machine-learning work refuses to fit on a CPU box: validating that CUDA code compiles and runs on real hardware, exercising GPU-accelerated libraries, running local model inference, accelerating a test suite that would take hours on CPU, serving a model, doing computer vision on real image tensors, or confirming that a GPU container actually sees the device. For all of that, the runner itself needs a GPU. This lesson is about building GitHub Actions workflows that run on GPU hardware — safely, reproducibly, and without setting money on fire.

This is Part 14 of the GitHub AI Engineering Academy. It builds directly on two earlier lessons: Part 12, Deploying LLM Applications Using GitHub Actions, which established the build-once, promote-the-same-artifact deployment discipline, and Part 6, GitHub Copilot for Docker, whose container fundamentals every GPU example here depends on. Where Part 12 deployed applications that call a remote model API and need no GPU at all, this lesson handles the case where the compute is yours.

Here is the pipeline this lesson builds, from a commit to stored results:

   Git commit
       |
   GitHub Actions
       |
   GPU runner  (hosted T4  OR  self-hosted NVIDIA)
       |
   CUDA / driver visible  (nvidia-smi)
       |
   AI / ML test  (PyTorch, inference, CV)
       |
   GPU container  (--gpus all)
       |
   Artifacts  (reports, metrics, diagnostics)

One disclosure up front, because it governs how the examples in this lesson were validated. The environment used to author this lesson has no GPU. Every workflow, Dockerfile, and command below has been checked for syntax and configuration correctness, but no GPU workload was executed on GPU hardware here. Where the text says to run nvidia-smi or docker run --gpus all and observe success, treat those as documented instructions to run on your own GPU host — GPU runtime testing was not performed in this environment, only syntax and configuration were validated. That honesty matters more in GPU CI than almost anywhere else, because the whole point of GPU CI is to catch the gap between “it looks right” and “it actually runs on the device.”

Why CPU CI is not enough

A normal GitHub-hosted runner like ubuntu-latest is a CPU machine. It will happily lint your code, run unit tests that mock the model, and build a container image. What it cannot do is prove that a CUDA kernel launches, that PyTorch can allocate a tensor on a device, that your inference server loads a model into GPU memory, or that a GPU container can reach the physical accelerator. Those questions only have answers on hardware that has a GPU.

The failures you are trying to catch are specifically the ones that hide on CPU. Code that imports torch and runs on CPU can still be broken on GPU because of a CUDA version mismatch. A Dockerfile that builds fine can still fail to see the GPU at runtime because the container runtime is not configured. A model that loads in 8 GB of system RAM can exceed the memory of a T4. None of that surfaces until the job runs somewhere with a real GPU, which is exactly why GPU CI exists: to move those discoveries from production into a pull request.

What a GPU CI workflow actually is

A GPU CI workflow is not a special GitHub product. It is an ordinary workflow whose job happens to run on a runner that has a GPU. Compare the two:

CPU CI:   runs-on: ubuntu-latest
          -> checkout -> setup -> unit tests -> build -> done

GPU CI:   runs-on: [self-hosted, linux, x64, gpu, nvidia]
          -> checkout -> nvidia-smi -> GPU framework check
             -> GPU container test -> artifacts -> done

The YAML shape is identical. What changes is the runs-on target — you point the job at a runner that has a GPU — and the steps you add to exercise that GPU. Everything you already know about jobs, steps, secrets, and artifacts applies unchanged. This is the mental model to hold: GPU CI is CPU CI plus a GPU-bearing runner and a handful of GPU-aware steps.

The GPU stack

To reason about GPU CI failures you have to see the stack, because a failure at any layer looks like a failure at the top.

   Application / Model        (your code, your weights)
        |
   Framework                  (PyTorch / TensorFlow / JAX)
        |
   CUDA libraries             (CUDA runtime, cuDNN, ...)
        |
   NVIDIA Container Runtime   (exposes GPU to containers)
        |
   NVIDIA driver              (kernel <-> GPU)
        |
   GPU hardware               (T4, A10, ...)

The rule that governs this stack is compatibility between adjacent layers. A framework build targets a CUDA version; that CUDA runtime needs a driver at or above a minimum version; the container runtime bridges the host driver into the container. When people say “CUDA is broken,” they almost always mean two of these layers disagree about versions. This lesson deliberately does not hand you a fixed version matrix — those pairings change, and a stale matrix is worse than none. Instead, pin your CUDA userspace inside an immutable container image and keep only a compatible driver on the host, which collapses most of this stack into something reproducible.

GitHub GPU runner options

You have two families of GPU runner, and picking between them is the first real design decision.

GitHub-hosted GPU runners

GitHub-hosted GPU runners are generally available. They provide an NVIDIA Tesla T4 GPU on Linux and Windows, delivered as a class of larger runners on paid plans such as GitHub Team and Enterprise Cloud. They are billed per minute and require billing and payment setup on the account. Crucially, there is no automatic gpu label that appears for free: you configure a larger runner with a GPU-enabled image in your organization settings, assign it a custom label of your choosing, and then target that label in runs-on. GitHub supplies the T4 larger-runner class; it does not expose an open catalog of GPU SKUs the way a cloud provider does. If a T4 with per-minute billing fits your workload and you would rather not run hardware, this is the low-operations path.

Self-hosted GPU runners

Self-hosted GPU runners are runners you install on machines you own or rent: an on-prem workstation, a lab server, a private data appliance, or a cloud GPU VM you manage. You get any GPU you can obtain and full control over drivers and libraries. The trade is that you own maintenance and, above all, security — a self-hosted runner is trusted infrastructure. You assign it custom labels such as self-hosted, linux, x64, gpu, nvidia, and those labels only exist because you configured them; GitHub does not auto-supply GPU labels for self-hosted machines.

Hosted versus self-hosted

Neither option is universally better. The right choice depends on which of these dimensions dominate for you.

Dimension	GitHub-hosted GPU	Self-hosted GPU
Operational overhead	Low — GitHub runs the box	High — you run the box
Security responsibility	Mostly GitHub	Yours; it is trusted infra
Startup time	Provisioned per job	Fast if persistent; slower if ephemeral
Cost control	Per-minute billing, paid plans	You pay for hardware/VM; can idle-cost
GPU choice	T4 larger-runner class	Any GPU you can buy or rent
Network access	GitHub’s network	Your network, including private infra
Isolation	Fresh hosted environment	Depends on your design
Maintenance	GitHub patches	You patch drivers, OS, toolkit

A common outcome is a mix: hosted T4 runners for routine GPU checks, and self-hosted runners for a specific GPU you need or private data you cannot send to a hosted machine.

Self-hosted GPU runner architecture

When you do self-host, the architecture is a chain from GitHub down to the silicon:

   GitHub  ->  Actions job  ->  self-hosted runner agent
                                     |
                                  Ubuntu host
                                     |
                              NVIDIA driver
                                     |
                     CUDA + NVIDIA Container Toolkit
                                     |
                                    GPU

Recommended base OS

Ubuntu is the pragmatic default for GPU runners because NVIDIA’s driver, CUDA, and Container Toolkit packaging is best-trodden there. Do not read that as a mandate for one specific Ubuntu release — verify the supported combination of runner agent, NVIDIA driver, Container Toolkit, and CUDA against current documentation for the GPU and framework you target, and pick a release inside that supported set. The point is a supported, current combination, not a magic version number.

GPU hardware detection

The first diagnostic on any GPU host is nvidia-smi:

nvidia-smi

It reports the GPU model, driver version, total and used memory, utilization, and running processes. If it prints a table, the driver sees the GPU. But — and this is the recurring caution of this lesson — nvidia-smi succeeding does not validate the full CUDA and framework stack. It confirms the bottom two layers (driver and GPU). It says nothing about whether your CUDA runtime and PyTorch build agree. Note also that this command was not executed in the authoring environment, which has no GPU; run it on your host.

CUDA validation

Validate the stack top-down as a flow, not a single command:

   driver visible (nvidia-smi)
        -> CUDA runtime present
        -> framework detects device (torch.cuda.is_available())
        -> tiny GPU op actually executes

Only when a real operation runs on the device have you validated more than visibility.

Installing the self-hosted runner

Do not hardcode a runner binary version or download URL in your docs or scripts — GitHub updates the runner regularly and a pinned URL goes stale. Instead, open repository or organization Settings → Actions → Runners → New self-hosted runner, and GitHub generates the current download, checksum, and configuration commands for your OS and architecture. Run those. The flow they generate looks conceptually like this (use the exact generated commands, not these placeholders):

# GitHub's page generates the current download + checksum commands here.
# Then it generates a config command using a short-lived registration token:
./config.sh --url https://github.com/<org>/<repo> \
  --token <REGISTRATION_TOKEN_FROM_GITHUB> \
  --labels self-hosted,linux,x64,gpu,nvidia \
  --runnergroup gpu

# Run interactively:
./run.sh

# Or install as a service so it runs unattended:
sudo ./svc.sh install
sudo ./svc.sh start

The registration token is short-lived and issued by GitHub on that settings page. The --labels you choose here are the labels workflows will target. The --runnergroup places the runner in a group you can then restrict to specific repositories. Service mode (svc.sh) keeps the runner alive across reboots.

Runner labels

Labels are how a job finds this machine. A GPU runner might carry self-hosted, linux, x64, gpu, nvidia. Every one of those exists only because you assigned it — GitHub does not automatically tag a self-hosted machine as a GPU host. A workflow then requests the machine by listing labels in runs-on, and the job runs only on a runner that has all the listed labels.

Your first GPU workflow

Start with a manual, minimal workflow that proves the runner can see the GPU and the framework agrees. Trigger it by hand so it never fires accidentally:

name: gpu-smoke-test

on:
  workflow_dispatch:

jobs:
  gpu-check:
    runs-on: [self-hosted, linux, x64, gpu, nvidia]
    timeout-minutes: 15
    steps:
      - name: Check out code
        uses: actions/checkout@v4

      - name: Show GPU visibility
        run: nvidia-smi

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Framework device check
        run: |
          # Install a GPU build using PyTorch's current official selector;
          # do not paste a fixed pip line here — it may target the wrong CUDA.
          python -c "import torch; print('CUDA visible:', torch.cuda.is_available())"

The runs-on list must exactly match labels a real runner has, or the job queues forever. nvidia-smi confirms driver visibility; the Python line confirms the framework sees CUDA.

The Python GPU test

torch.cuda.is_available() returning True means the framework can see CUDA through the driver. It does not mean your model is fast, your batch size fits, or the whole stack is healthy — it is a visibility flag. Because PyTorch GPU builds are tied to a specific CUDA version and platform, install via PyTorch’s current official install selector rather than a hardcoded pip install torch line that may target the wrong CUDA build.

A safe GPU memory test

Confirm real device execution with a tiny operation, not a benchmark. On a shared runner, do not allocate large tensors or hammer the GPU:

import torch

assert torch.cuda.is_available(), "CUDA not visible to the framework"

# Tiny op on the device — proves execution, not performance.
device = torch.device("cuda")
a = torch.randn(256, 256, device=device)
b = torch.randn(256, 256, device=device)
c = a @ b
torch.cuda.synchronize()

print("Result device:", c.device)
print("Allocated MB:", round(torch.cuda.memory_allocated() / 1e6, 2))

This runs a real matrix multiply on the GPU and reports device and memory — enough to prove execution, small enough to be harmless on shared hardware.

Docker GPU workloads

Containers are the most reproducible way to run GPU work in CI, and they connect directly to Part 6. The chain is:

   Actions  ->  GPU runner  ->  Docker  ->  NVIDIA Container Runtime  ->  GPU container

The NVIDIA Container Toolkit

The NVIDIA Container Toolkit is what exposes the host GPU devices to containers — it bridges the host driver into the container so a containerized framework can reach the GPU. Install and wire it in with the current commands:

sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

nvidia-ctk runtime configure --runtime=docker registers the NVIDIA runtime with Docker; --set-as-default makes it the default runtime and --dry-run previews the change without applying it. Restarting Docker picks up the new configuration. Do not paste obsolete repository-setup commands from old blog posts — follow NVIDIA’s current install documentation for the repository steps that precede the apt-get install.

Validating Docker GPU access

Confirm a container can see the GPU:

docker run --rm --gpus all nvidia/cuda:<verified-tag> nvidia-smi

Use --gpus all to expose every GPU, or --gpus '"device=0"' to expose one. Choose a current nvidia/cuda image tag whose CUDA version is compatible with your host driver — do not blindly pin a stale tag from documentation. As with every GPU command here, this was not executed in the no-GPU authoring environment; run it on your host and confirm the container prints the GPU table.

Building a GPU-enabled Docker image

Here is a small, self-contained project to test a GPU workflow end to end:

gpu-ai-demo/
  .github/
    workflows/
      gpu-ci.yml
  app/
    gpu_test.py
  Dockerfile
  requirements.txt
  README.md

The Dockerfile starts from a current CUDA or framework base image with an immutable tag — never :latest for anything promoted — and installs the app:

# Pin an immutable, verified CUDA base tag compatible with your driver.
# Replace <verified-tag> with a current nvidia/cuda tag you have checked.
FROM nvidia/cuda:<verified-tag>

RUN apt-get update && apt-get install -y --no-install-recommends \
        python3 python3-pip \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app
COPY requirements.txt ./
RUN pip3 install --no-cache-dir -r requirements.txt

COPY app/ ./app/

# Use a non-root user in real images (see Part 6).
CMD ["python3", "app/gpu_test.py"]

# app/gpu_test.py
import torch


def main() -> int:
    if not torch.cuda.is_available():
        print("FAIL: CUDA not visible to the framework")
        return 1
    x = torch.randn(128, 128, device="cuda")
    y = (x @ x).sum().item()
    print(f"OK: GPU op executed, checksum={y:.2f}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

The requirements.txt should reference the framework, with the actual GPU build installed per the framework’s official selector rather than a stale pinned index URL.

A GPU CI workflow with Docker

This workflow verifies the GPU, builds the image, and runs the container against the GPU:

name: gpu-docker-ci

on:
  workflow_dispatch:

jobs:
  gpu-docker:
    runs-on: [self-hosted, linux, x64, gpu, nvidia]
    timeout-minutes: 30
    steps:
      - name: Check out code
        uses: actions/checkout@v4

      - name: Verify GPU is visible
        run: nvidia-smi

      - name: Set up Buildx
        uses: docker/setup-buildx-action@v3

      - name: Build GPU image (immutable tag)
        uses: docker/build-push-action@v6
        with:
          context: .
          load: true
          push: false
          tags: gpu-ai-demo:${{ github.sha }}

      - name: Run GPU test in container
        run: |
          docker run --rm --gpus all \
            gpu-ai-demo:${{ github.sha }}

      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: gpu-test-results
          path: results/
          if-no-files-found: ignore

The image is tagged with the immutable commit SHA, never :latest. The container runs with --gpus all so it can reach the device.

AI inference testing without the bill

The fastest way to make GPU CI expensive and flaky is to download a multi-gigabyte model on every run. Don’t. Make inference tests cost-aware by choosing one of these, in rough order of preference:

• A tiny model small enough to load in seconds.
• A cached model pulled once and reused via a version-keyed cache.
• A synthetic workload — random tensors shaped like real input.
• A small fixture — a handful of representative examples.

The CI question is “does inference run correctly on the GPU,” not “how fast is the production model.” Keep the workload small and the intent narrow.

Separating CPU and GPU tests

GPU minutes are the most expensive minutes you have, so never spend them on code that fails a cheap check. Stage the pipeline as a hierarchy and gate the GPU stage behind the CPU stages:

   Lint  ->  Unit tests  ->  CPU integration  ->  GPU integration
   (cheap, fast)                                  (expensive, gated)

jobs:
  cpu-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: |
          pip install -r requirements-dev.txt
          pytest tests/unit tests/cpu

  gpu-tests:
    needs: cpu-tests           # only runs if CPU tests pass
    runs-on: [self-hosted, linux, x64, gpu, nvidia]
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
      - run: nvidia-smi
      - run: pytest tests/gpu

The needs: cpu-tests dependency means a lint or unit failure never reaches the GPU runner.

Triggering GPU jobs selectively

Do not run GPU work on every commit. Fire it only when it is relevant, using path filters, labels, manual dispatch, and schedules:

on:
  pull_request:
    paths:
      - "model/**"
      - "inference/**"
      - "cuda/**"
      - "requirements*.txt"
      - "Dockerfile"
  workflow_dispatch:
  schedule:
    - cron: "0 3 * * *"   # nightly full GPU suite

jobs:
  gpu-tests:
    # Also gate on a label so contributors opt in explicitly.
    if: >
      github.event_name != 'pull_request' ||
      contains(github.event.pull_request.labels.*.name, 'gpu')
    runs-on: [self-hosted, linux, x64, gpu, nvidia]
    steps:
      - uses: actions/checkout@v4
      - run: nvidia-smi

Path filters restrict PR runs to changes that could affect the GPU. The label gate makes GPU testing an explicit choice. The schedule runs the full suite off the critical path.

GPU workflow cost controls

Layer several inexpensive guards:

• Cancel superseded runs with a concurrency group so a new push kills the old GPU job:

concurrency:
  group: gpu-${{ github.ref }}
  cancel-in-progress: true

• Bound every GPU job with timeout-minutes so a hung job cannot run forever.
• Shut down ephemeral runners when idle so you stop paying for a GPU you are not using.
• Keep tests small and representative rather than exhaustive.

Notice what is absent: any dollar figure. Prices change, so design for “spend GPU minutes only when they add information” rather than chasing a volatile per-minute number.

Ephemeral GPU runners

The cleanest cost and security posture for cloud GPUs is ephemeral: create the runner, use it once, destroy it.

   GitHub job queued
        -> provision GPU runner (fresh VM)
        -> register with GitHub
        -> run the job
        -> unregister
        -> destroy the VM

The benefits are a clean environment every time, no persistence of files or secrets, and no idle cost. The cost is implementation complexity — you must handle provisioning, authenticated registration, and reliable teardown, and a bug in teardown leaves an expensive VM running. It is worth it precisely because the alternative — a long-lived, privileged GPU box — accumulates risk.

Persistent GPU runner risks

A persistent GPU runner that survives between jobs collects hazards: leftover files and model caches that fill the disk, secrets or credentials lingering in the environment, state from other users’ jobs, the possibility that a compromised workflow established persistence, and driver or library drift over time. If you must run persistent runners, clean the workspace between jobs, isolate them, and monitor for drift — but prefer ephemeral where you can.

Runner security

This is the section to read twice. Self-hosted runners are trusted infrastructure, not sandboxes.

Warning: Never run untrusted pull requests on privileged self-hosted GPU runners. A self-hosted runner executes whatever the workflow tells it to on a machine that has your NVIDIA driver, your Docker socket, your local network, and any credentials present on the host. A fork pull request can propose arbitrary code and workflow changes, so running it on your GPU runner opens the door to secret theft, full host compromise, persistence across jobs, credential theft, and lateral movement into private infrastructure. For public repositories, keep sensitive self-hosted GPU runners entirely off fork PRs, and if you must test external contributions on a GPU, use isolated, ephemeral runners with no secrets, no production access, and aggressive teardown — never a persistent, privileged box.

Docker socket security

Access to /var/run/docker.sock on the runner is effectively root on the host — a job that can talk to the Docker socket can start a privileged container that mounts the host filesystem. Do not wave this away. Treat Docker socket access as a full-host trust boundary, and do not grant it to workflows you would not trust with root.

Runner network segmentation

Place GPU runners on a restricted subnet. Limit outbound network access where practical, remove any unnecessary route to production systems, and keep build runners separate from anything that can reach production. A GPU runner should be able to do its job and little else.

GPU runner secrets

Give GPU jobs only the secrets they need, scoped to the job, and prefer short-lived credentials over persistent cloud keys sitting on the GPU host. A long-lived cloud key on a shared GPU box is exactly the credential an attacker who lands on that box wants. Rotate, scope, and expire.

CUDA and dependency compatibility

Most GPU CI pain is version alignment, not logic bugs.

CUDA version compatibility

The driver, the CUDA runtime, and the framework build must align — the driver must be new enough for the CUDA runtime, and the framework must be built for that CUDA version. There is no static matrix in this lesson because those pairings change; consult the official compatibility documentation for the exact GPU, driver, and framework you use, and let a container pin the CUDA userspace so the host only needs a compatible driver.

Python dependency compatibility

GPU libraries — PyTorch, TensorFlow, JAX — ship platform-specific and CUDA-specific builds. A pip install line that worked last year may install a build for the wrong CUDA. Install using each framework’s current official instructions or selector rather than a copied, outdated command, and pin the versions you validated.

Caching, models, and artifacts

GPU caching

Cache pip downloads, Docker layers, and small models to avoid repeated fetches — but cache keys must include dependency and model versions, or you will serve a stale environment. GPU caches also get large fast, so watch for disk exhaustion and stale model files.

- name: Cache pip
  uses: actions/cache@v4
  with:
    path: ~/.cache/pip
    key: pip-${{ hashFiles('requirements*.txt') }}

Model caching

Cache a model through a registry-to-runner path so inference tests do not re-download weights, but never cache confidential weights in a public cache. A public cache is a public artifact; treat proprietary weights accordingly.

Artifact storage

Benchmark results, test reports, and diagnostics are fine to upload with actions/upload-artifact@v4. Do not upload proprietary weights, sensitive prompts, or private datasets without access controls — an artifact is retrievable by anyone who can read the run.

Benchmarking versus validation

Keep two questions apart. “Does it run correctly on the GPU?” is a CI question with a yes/no answer, and it belongs in every relevant pipeline. “How fast is it?” is a benchmarking question that needs controlled, dedicated hardware and a fixed environment, and shared CI runners cannot answer it reliably. If you do collect a basic performance number — execution time and GPU memory — label it clearly as environment-dependent and never present it as proof of universal GPU superiority. Conflating validation and benchmarking produces numbers no one can trust.

GPU test matrix

A matrix across Python version × framework × CUDA version gets expensive very fast — the combinations multiply and each one costs GPU minutes. Keep the matrix small and representative; test the combinations you actually support, not every theoretically possible one.

Multi-GPU and runner pools

Multi-GPU runners

A single runner may host several GPUs (device 0, 1, 2). Inside that host you select GPUs with CUDA_VISIBLE_DEVICES, and you are responsible for scheduling and isolating work across them. GitHub does not schedule individual GPUs within one runner — from GitHub’s side the runner is one machine, and how its GPUs are shared is your problem to manage.

- name: Pin to GPU 0
  run: python app/train_smoke.py
  env:
    CUDA_VISIBLE_DEVICES: "0"

One runner per GPU versus shared

Running one runner per GPU gives clean isolation and simple accounting at the cost of more runner processes; a shared multi-GPU runner is denser but requires you to prevent jobs from colliding on the same device. Choose based on how much isolation your workloads need.

GPU runner pools

Group several GPU machines into a runner group named, for example, gpu, with members gpu-runner-01, gpu-runner-02, gpu-runner-03, and use runner groups plus repository access policies to control which repositories may schedule onto the pool. This is how you scale GPU CI to many repositories without exposing the pool to all of them.

Cloud GPU runners and provisioning

Cloud GPU runners

You can run self-hosted runners on cloud GPU VMs, and the generic ephemeral pattern is provider-neutral:

   GitHub  ->  provision cloud GPU VM  ->  register runner
           ->  run job  ->  unregister  ->  destroy VM

This lesson deliberately avoids a cloud-vendor shootout — the shape is the same everywhere, and the details belong in each provider’s current documentation.

Terraform and GPU runner provisioning

Infrastructure-as-code makes provisioning repeatable. Connecting to Part 5, GitHub Copilot for Terraform, the flow is:

   Actions  ->  Terraform apply  ->  GPU VM  ->  self-hosted runner  ->  GPU tests
                                                          |
                                          Terraform destroy on completion

Require budget and cleanup controls: a spend cap, a guaranteed destroy even on failure, and authentication on runner registration. Never build unlimited auto-provisioning — an unbounded loop of GPU VMs is both a runaway bill and a security exposure.

Kubernetes GPU runners

Actions Runner Controller

Actions Runner Controller (ARC) can host self-hosted runners as pods on Kubernetes, and if the cluster has GPU nodes with the NVIDIA device plugin, those runner pods can request GPUs. This connects to Part 7, GitHub Copilot for Kubernetes.

   Actions  ->  runner controller (ARC)  ->  K8s pod (GPU node)  ->  GPU job

Only build on current, established projects here, and verify ARC and device-plugin specifics against their official documentation before committing.

Kubernetes GPU scheduling

A pod claims a GPU through resource limits, which the NVIDIA device plugin satisfies by scheduling onto a GPU node:

resources:
  limits:
    nvidia.com/gpu: 1

Verify the current behavior of the device plugin and your cluster’s GPU node setup — GPU scheduling semantics evolve, and GPU nodes are expensive, so size pools deliberately.

Workload patterns

Local LLM inference

For validating a local model server, run the cheap tests first, then on a GPU runner start a small inference service, evaluate a couple of prompts, and stop the service:

   PR  ->  CPU tests  ->  GPU runner: start local inference
        ->  evaluate a few prompts  ->  stop service

Keep the model small — the goal is to prove the integration starts, loads, and answers, not to benchmark a production model. This connects to the local-LLM material in the Ubuntu AI infrastructure series.

Computer vision

For a vision workflow, load a small test model, process a test image, and validate the output shape or classification, avoiding large datasets. A single fixture image and a shape assertion catch most integration breaks without downloading a dataset.

Training

Full training does not belong inside normal CI. Instead, use Actions to trigger, orchestrate, and validate training:

   Actions  ->  trigger training platform  ->  GPU training job
            ->  collect artifacts / metrics  ->  validate

The contrast is stark:

Aspect	CI validation	Real training
Duration	Minutes	Hours to days
Cost	Low	High
Data	Small fixture	Full dataset
State	Deterministic-enough	Checkpointing, sometimes distributed
Goal	Prove the path works	Produce a model

Let Actions be the control plane; let dedicated infrastructure be the compute.

Observability and health

Observability

Track job duration, GPU availability, GPU memory usage, failure rates, and queue time so you can see when GPU CI degrades. Use nvidia-smi carefully for point-in-time GPU state, but remember it reports visibility and utilization, not stack correctness.

Runner health checks

Before a runner accepts jobs, confirm the GPU is visible, disk space is adequate, Docker works, and the runner is connected. Do not overdo it — an excessive number of health jobs is itself wasted GPU time. A lightweight pre-flight check that fails fast is enough.

Troubleshooting common GPU CI failures

Symptom	Likely cause	First move
nvidia-smi fails	Driver missing or broken; GPU not visible	Reinstall/verify the NVIDIA driver on the host
CUDA unavailable	Runtime/framework mismatch	Compare framework’s required CUDA vs installed
Framework doesn’t detect GPU	Wrong framework build for this CUDA	Reinstall via official selector for your CUDA
Docker can’t access GPU	Container Toolkit not configured	Re-run nvidia-ctk runtime configure + restart Docker
Out of memory	Model/batch too large; other processes	Inspect nvidia-smi; reduce batch/model size
Incompatible CUDA runtime	Driver too old for CUDA version	Align driver/CUDA per compatibility docs
Missing driver	Host not provisioned	Install driver before registering the runner
Image arch mismatch	Wrong platform/arch image	Build/pull the correct architecture image
Disk full from model cache	Unbounded model/layer cache	Prune cache; add version-aware keys and cleanup

Diagnosing out of memory

Do not reflexively reach for a bigger GPU. Work the flow:

   Read the OOM error (what tried to allocate?)
        -> nvidia-smi (how much memory is used, by what?)
        -> check model / batch size
        -> check for other processes holding the GPU

Most OOMs in CI are an oversized batch or a leftover process, not a hardware limit.

Driver and runtime mismatch

When the framework and CUDA disagree, compare the framework’s stated CUDA requirement against the installed CUDA runtime and the driver’s supported version — conceptually, all three must line up. Do not hardcode a claim about which versions pair; verify against current documentation for your exact stack.

Reproducible GPU environments

Containerization is the strongest lever for reproducibility:

   host driver  ->  GPU container runtime  ->  pinned image  ->  framework + app

Pinning the CUDA userspace and framework inside an immutable image reduces drift — it does not eliminate it, since the host driver still varies, but it collapses most of the compatibility surface into an artifact you control.

Container image versioning

Avoid :latest for anything promoted; prefer immutable tags or digests so a rebuilt or re-pulled image is byte-for-byte the one you validated. A GPU image tagged my-gpu-image:latest is a moving target; my-gpu-image@sha256:... or a versioned tag is not.

GPU supply-chain security

Big CUDA images have a large vulnerability surface — base image provenance, framework versions, Python dependencies, and OS packages all matter. Scan GPU images with Trivy (aquasecurity/trivy-action) or Docker Scout as part of the pipeline, and treat a large CUDA base as something to inventory and patch, not to trust blindly.

An advanced Docker GPU workflow

This ties the pieces together — build, scan, run on GPU, integration-test, publish only if approved — and connects to Part 12’s container pipeline:

   Checkout -> build GPU image -> scan (Trivy) -> run with GPU
            -> integration test -> publish if approved

name: gpu-advanced

on:
  workflow_dispatch:

permissions:
  contents: read

jobs:
  build-and-test:
    runs-on: [self-hosted, linux, x64, gpu, nvidia]
    timeout-minutes: 45
    steps:
      - name: Check out code
        uses: actions/checkout@v4

      - name: Verify GPU
        run: nvidia-smi

      - name: Set up Buildx
        uses: docker/setup-buildx-action@v3

      - name: Extract image metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ghcr.io/${{ github.repository }}/gpu-ai-demo
          tags: |
            type=sha,format=long

      - name: Build image (load locally, immutable tag)
        uses: docker/build-push-action@v6
        with:
          context: .
          load: true
          push: false
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}

      - name: Scan image with Trivy
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{ steps.meta.outputs.tags }}
          format: table
          exit-code: "1"
          severity: CRITICAL,HIGH

      - name: Run GPU integration test
        run: |
          docker run --rm --gpus all \
            ${{ steps.meta.outputs.tags }}

      - name: Upload diagnostics
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: gpu-diagnostics
          path: results/
          if-no-files-found: ignore

  publish:
    needs: build-and-test
    runs-on: [self-hosted, linux, x64, gpu, nvidia]
    environment: gpu-images         # require a reviewer to approve publish
    permissions:
      contents: read
      packages: write
    steps:
      - name: Check out code
        uses: actions/checkout@v4

      - name: Log in to registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Extract image metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ghcr.io/${{ github.repository }}/gpu-ai-demo
          tags: |
            type=sha,format=long

      - name: Build and push (immutable tag only)
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}

The scan fails the job on CRITICAL/HIGH findings, the GPU test runs in a container with --gpus all, and publishing is gated behind a protected environment so nothing ships to the registry without approval — no unnecessary privilege anywhere.

30 GitHub Actions GPU Workflow Ideas

1. Validate that CUDA code compiles and runs on real GPU hardware.
2. Confirm torch.cuda.is_available() on every dependency bump.
3. Run a tiny tensor operation to prove device execution.
4. Smoke-test a GPU container with docker run --gpus all.
5. Check GPU memory allocation stays within a T4’s limits.
6. Validate a local LLM inference server starts and answers.
7. Run computer-vision inference on a single fixture image.
8. Verify a model loads into GPU memory without OOM.
9. Scan CUDA base images for vulnerabilities with Trivy.
10. Build and push an immutable-tagged GPU image on release.
11. Run GPU integration tests only on model/ path changes.
12. Nightly full GPU test suite on a schedule.
13. Label-gated GPU tests contributors opt into.
14. Multi-GPU pinning tests with CUDA_VISIBLE_DEVICES.
15. Validate NVIDIA Container Toolkit configuration after host updates.
16. Confirm driver/CUDA/framework alignment on a new base image.
17. Test a quantized model runs on GPU and returns valid output.
18. Regression-test inference output shape against a golden fixture.
19. Validate a Dockerfile’s GPU base image builds reproducibly.
20. Run a tiny training step to confirm the training code path works.
21. Trigger a training job on a dedicated platform and validate metrics.
22. Provision an ephemeral cloud GPU runner, test, and destroy it.
23. Health-check a self-hosted GPU runner before accepting jobs.
24. Cache a small model with a version-aware key and verify reuse.
25. Validate GPU-enabled ARC runner pods schedule on GPU nodes.
26. Compare CPU vs GPU test paths produce the same functional result.
27. Verify a batch size fits GPU memory before a larger run.
28. Test framework upgrade against the current CUDA base image.
29. Collect and upload GPU diagnostics as artifacts on failure.
30. Confirm nvidia-smi and docker info report a healthy GPU stack.

25 GPU CI Security and Reliability Rules

1. Never run untrusted fork PRs on privileged self-hosted GPU runners.
2. Treat every self-hosted runner as trusted infrastructure, not a sandbox.
3. Use runner groups and repo access policies to restrict GPU runners.
4. Prefer ephemeral runners destroyed after each job.
5. Give GPU jobs only job-scoped, short-lived secrets.
6. Keep no persistent cloud credentials on GPU hosts.
7. Remember Docker socket access equals root on the host.
8. Segment GPU runners onto a restricted network.
9. Limit outbound access and remove unnecessary production routes.
10. Separate build runners from anything touching production.
11. Never use :latest for promoted GPU images; pin immutable tags/digests.
12. Scan GPU/CUDA images for vulnerabilities before publishing.
13. Gate publishing behind a protected environment with reviewers.
14. Run cheap CPU tests before spending GPU minutes.
15. Trigger GPU jobs selectively with path filters, labels, and schedules.
16. Set timeout-minutes on every GPU job.
17. Cancel superseded runs with concurrency and cancel-in-progress.
18. Never download multi-GB models on every run; cache or use tiny models.
19. Version-key all caches to avoid stale environments.
20. Never cache confidential weights in a public cache.
21. Never upload proprietary weights or private data as open artifacts.
22. Clean the workspace between jobs on persistent runners.
23. Require budget caps and guaranteed teardown for provisioned GPUs.
24. Authenticate ephemeral runner registration; never register openly.
25. Verify driver/CUDA/framework compatibility against official docs, not a stale matrix.

Hands-On Lab: Build a Self-Hosted GPU CI Pipeline with GitHub Actions

This lab builds a complete self-hosted GPU CI pipeline on an Ubuntu host with an NVIDIA GPU. Because the authoring environment has no GPU, the GPU-runtime steps below were not executed here — they are validated for syntax and configuration only, and you run them on your own GPU host. Follow the steps in order.

1. Provision a GPU host. Start with an Ubuntu machine that has an NVIDIA GPU — a workstation, a lab server, or a cloud GPU VM you control.
2. Install the NVIDIA driver. Install a driver supported for your GPU and intended CUDA version, following NVIDIA’s current instructions.
3. Verify the driver. Run nvidia-smi and confirm it prints the GPU model, driver version, and memory.
4. Install Docker. Install Docker Engine using the official current instructions for Ubuntu.
5. Install the NVIDIA Container Toolkit. Run sudo apt-get install -y nvidia-container-toolkit.
6. Configure the Docker runtime. Run sudo nvidia-ctk runtime configure --runtime=docker.
7. Restart Docker. Run sudo systemctl restart docker.
8. Validate container GPU access. Run docker run --rm --gpus all nvidia/cuda:<verified-tag> nvidia-smi and confirm the container sees the GPU.
9. Create the runner in GitHub. In repository Settings → Actions → Runners, choose New self-hosted runner for Linux x64.
10. Download the runner. Use the current download and checksum commands GitHub generates — do not hardcode a version or URL.
11. Configure the runner. Run the generated ./config.sh command with the registration token, adding --labels self-hosted,linux,x64,gpu,nvidia and --runnergroup gpu.
12. Restrict the runner group. In organization settings, limit the gpu runner group to only the repositories that need it.
13. Install as a service. Run sudo ./svc.sh install then sudo ./svc.sh start so the runner survives reboots.
14. Confirm registration. Verify the runner shows as idle with its labels in the Runners list.
15. Create the project. Scaffold gpu-ai-demo/ with app/gpu_test.py, Dockerfile, requirements.txt, and .github/workflows/gpu-ci.yml.
16. Write the GPU test. Add the gpu_test.py that asserts torch.cuda.is_available() and runs a tiny GPU matrix multiply.
17. Write the Dockerfile. Base it on a verified immutable nvidia/cuda tag and install the app; never use :latest.
18. Pin dependencies. Add requirements.txt and install the GPU framework build via the framework’s official selector.
19. Write the workflow. Add a workflow_dispatch job on runs-on: [self-hosted, linux, x64, gpu, nvidia] that runs nvidia-smi, builds the image with an immutable SHA tag, and runs it with --gpus all.
20. Add a CPU gate. Add a cpu-tests job on ubuntu-latest and make the GPU job needs: cpu-tests.
21. Add cost controls. Add timeout-minutes and a concurrency group with cancel-in-progress: true.
22. Add path and label triggers. Restrict PR runs to GPU-relevant paths and gate on a gpu label.
23. Add scanning. Add a Trivy step that fails on CRITICAL/HIGH before any publish step.
24. Run and observe. Dispatch the workflow, watch it land on your GPU runner, and confirm nvidia-smi and the GPU test pass on your hardware.
25. Upload artifacts. Collect diagnostics with actions/upload-artifact@v4 and confirm they appear on the run.

The finished architecture:

   Git commit / dispatch
        |
   GitHub Actions
        |
   CPU gate (ubuntu-latest): lint + unit + CPU integration
        |
   Self-hosted GPU runner  [self-hosted, linux, x64, gpu, nvidia]
        |
   Ubuntu host -> NVIDIA driver -> Container Toolkit -> Docker
        |
   GPU container (--gpus all): nvidia-smi + tiny GPU test
        |
   Trivy scan -> gated publish (protected environment)
        |
   Artifacts: diagnostics + results

Bonus: Ephemeral GPU runner architecture

For the strongest isolation and cost posture, make the GPU runner disposable:

   provision  ->  register  ->  run  ->  unregister  ->  destroy

Each run gets a fresh machine that is torn down afterward, so no files, caches, secrets, or workflow persistence survive between jobs. Implement it only with authentication on registration and a guaranteed teardown path — dynamic runner registration without authentication and cleanup is an open door, not a convenience. Ephemeral is more work to build, and worth it wherever the runner would otherwise be long-lived and privileged.

What’s Next

You can now run GPU-accelerated AI and machine-learning workloads directly from GitHub automation — validating CUDA on real hardware, testing GPU containers, exercising local inference, and doing all of it with cost controls and a security posture that treats self-hosted runners as the trusted infrastructure they are. The last two lessons of this academy close the loop by putting AI into the review gate itself. Next up is Automated AI Code Review (Part 15, coming soon), which brings model-assisted review into the pull request. Until it publishes, revisit the GitHub AI Engineering Academy landing page to review the earlier lessons this one builds on — especially Part 6 for containers and Part 12 for the deployment pipeline these GPU workflows plug into.