Deploying LLM Applications Using GitHub Actions: Production AI CI/CD

Part 10 answered a narrower question than it looks: given an AI application, how do you verify it — lint it, run its deterministic tests, and grade its probabilistic output with evaluations. That pipeline ends with a green check and a scanned image. This lesson picks up exactly there and answers the next question: how do you take that verified image and actually ship it — package it as an immutable artifact, release it through staging, prove it works against a live model, get a human to approve it, promote it to production, watch it, and roll it back when it breaks.

This is Part 12 of the GitHub AI Engineering Academy. Part 10, GitHub Actions for AI Applications, built the CI side — build, test, evaluate. Part 12 builds the CD side — package, release, deploy, verify, and roll back. Everything here rests on three rules that do not bend:

• Build once, promote the same artifact. You build and scan one image, then deploy that exact image to staging and production. You never rebuild between environments, because a rebuild is a different artifact than the one you tested.
• Production requires human approval. A required reviewer on the production environment is the backstop. A passing evaluation is a signal, not authorization to ship.
• Deterministic controls, not the AI model, enforce production gates. Health probes, smoke tests, and error rates — signals that behave identically every run — decide whether a deploy succeeds or rolls back. The probabilistic model is never in that loop.

Here is the full path this lesson builds, from a commit to a running, observed service:

   Code
    |
   Tests            (deterministic)
    |
  AI Eval           (probabilistic)
    |
  Docker Build
    |
   Scan             (Trivy / Scout)
    |
  Registry          (immutable tag)
    |
  Staging
    |
  Smoke Test
    |
  Approval          <-- required human
    |
  Production        (same image)
    |
 Observability

Deploying an LLM application carries every concern a traditional service does — code correctness, container hygiene, networking, secrets, CI/CD, health checks, rollback — and then adds a set that exists only because there is a model in the loop: the model provider dependency, prompts as behavior, runtime configuration, output quality, latency, rate limits, cost per call, evaluation, and safety. A competent deployment pipeline takes both lists seriously. The traditional concerns keep the service running; the AI-specific concerns keep it correct while the thing at its center is probabilistic.

What You’ll Learn

• The deployment lifecycle for an LLM app — how packaging, release, staging, approval, production, and rollback map onto a probabilistic service, and the two lists of concerns a sound pipeline covers.
• The example FastAPI application — a small /health and /analyze service that returns structured JSON, and the llm-devops-app/ repository that wraps it in delivery machinery.
• Provider abstraction and prompt versioning — why the model lives behind an LLMClient and why prompts are Git-versioned source, not console configuration.
• Building and publishing an immutable container — a hardened Dockerfile, a build-scan-publish workflow to GHCR with verified actions, and why production tags are Git SHAs or semver, never latest.
• Environments, secrets, and OIDC — GitHub Environments for staging and production, a secrets strategy that separates model credentials from infrastructure credentials, and short-lived OIDC over static keys.
• Kubernetes deployment — an apps/v1 Deployment with probes, resource limits, a non-root context, and secrets via secretKeyRef, deployed by a workflow that authenticates, applies, waits, and smoke-tests.
• Staging, smoke tests, and the production approval gate — deploying the exact tested image, checking health and a low-cost AI smoke test, and requiring human approval before production.
• Rollback and deployment history — reversing a bad deploy with kubectl rollout undo or an image-ref revert, automated rollback on deterministic signals only, and the release metadata to keep.
• Testing model and prompt changes — evaluations before deployment, prompt regression testing, and treating a model swap as a dependency-like change.
• Reliability and observability — provider failure strategy, rate-limit handling, why liveness must never depend on inference, cost controls, custom metrics, and safe logging.

The Example LLM Application

Ground every example in one small service. The application is a FastAPI app with two endpoints and a deliberately narrow job:

• GET /health — a liveness/health endpoint that returns a simple status. It does not call the model. This independence is a design decision the whole lesson depends on.
• POST /analyze — accepts a block of text (say, a log excerpt or an incident description) and returns a structured JSON result with a summary, a severity, and a list of recommended_checks.

A representative response shape from /analyze:

{
  "summary": "Pod restarted repeatedly after an out-of-memory kill.",
  "severity": "high",
  "recommended_checks": [
    "inspect container memory limits",
    "review recent deployment for a leak"
  ]
}

❗ Important — This app is a demo that summarizes and suggests checks — not an autonomous repair system. It returns recommended_checks for a human to run; it does not execute commands, touch a cluster, or change infrastructure. The entire academy holds one line: AI proposes, humans and deterministic controls dispose. An app that returned “actions taken” instead of “recommended checks” would be a different, far riskier thing than what we deploy here.

The repository wraps that app in the delivery machinery, following the same separation of concerns Part 10 established:

llm-devops-app/
  .github/
    workflows/
      ci.yml           lint + tests + AI eval
      evaluation.yml   AI evaluations
      container.yml    build + scan + publish
      deploy.yml       staging, approve, prod
  app/
    main.py            FastAPI: /health, /analyze
    llm.py             LLMClient provider seam
    schemas.py         Pydantic response models
    prompts/
      troubleshoot.txt versioned prompt
  evaluations/
    cases.json         evaluation dataset
    evaluate.py        evaluation harness
  kubernetes/
    deployment.yaml    apps/v1 Deployment
    service.yaml       ClusterIP Service
    ingress.yaml       external routing
  Dockerfile
  compose.yaml
  pyproject.toml

Each area has a distinct role in the deployment story:

• app/ is the service. main.py wires the two endpoints; llm.py is the provider seam that everything else calls; schemas.py defines the response shape so the structured output is validated, not hoped for; prompts/troubleshoot.txt is the prompt as a source file.
• evaluations/ is the AI evaluation suite from Part 10 — the dataset and harness that must pass before an image is built.
• kubernetes/ holds the manifests this lesson deploys — a Deployment, a Service, and an Ingress. Part 7, GitHub Copilot with Kubernetes, built the Copilot-assisted versions of these.
• Dockerfile / compose.yaml package the app (Part 6, GitHub Copilot with Docker), and compose.yaml gives a local run.
• pyproject.toml pins dependencies so installs are reproducible and the tested artifact matches production.

Provider Abstraction and Prompt Versioning

Two design choices in app/ decide how safely the rest of the pipeline can move, so they come first.

The model lives behind an abstraction. Every part of the app that needs the model calls an LLMClient, never a provider SDK directly:

class LLMClient:
    """Provider-agnostic seam for the model call."""

    def __init__(self, api_key: str, model: str) -> None:
        self._api_key = api_key   # from env / secret
        self._model = model

    def analyze(self, text: str) -> dict:
        """Return a structured analysis for the text.

        Concrete provider call lives here, behind the
        interface. Callers depend on this method, not
        on any vendor SDK.
        """
        ...

The value of this seam is not academic. The credentials come from the environment (a secret), so no key is hardcoded. The provider call is confined to one place, so swapping vendors or updating an SDK changes llm.py and nothing else — main.py, the evaluation harness, and the tests all keep calling analyze(). That reduced coupling is what lets you treat a model change as a small, reviewable diff instead of a scattered rewrite.

🤖 AI Infrastructure Tip — Keep the current provider SDK shapes behind this seam and verify them against current docs, since they move: an OpenAI-style client uses client.chat.completions.create(...) (not the retired openai.ChatCompletion.create), an Anthropic-style client uses client.messages.create(...), and GitHub Models is reached through the azure-ai-inference client. Which one you use is an implementation detail of LLMClient — callers never see it, which is the entire point of the abstraction.

Prompts are versioned in Git. The prompt that steers /analyze lives in app/prompts/troubleshoot.txt, a source file, not a string literal buried in code and not a value typed into a provider console. This matters because a prompt is not documentation — a prompt is application behavior. A one-line edit to troubleshoot.txt can change the severity the app assigns or the checks it recommends, with zero change to any .py file.

Because the prompt is source, a change to it follows the same lifecycle as any code change:

edit troubleshoot.txt
      |
    commit
      |
  pull request
      |
  AI evaluation   (path filter triggers it)
      |
  human review
      |
    deploy

A prompt that lives only in a dashboard has no diff, no review, and nothing to trigger an evaluation — so a behavior change would ship with none of the controls the rest of the code gets. Version the prompt, and a behavior change becomes a reviewable pull request.

The Application CI Workflow

The deployment pipeline does not repackage what Part 10 already validates — it depends on it. The CI workflow (ci.yml) is the one built in GitHub Actions for AI Applications: a pull request runs lint, then a type check, then deterministic unit tests, then AI evaluations, and all of them must pass before anything gets packaged.

name: ci

on:
  push:
    branches: [main]
  pull_request:

permissions:
  contents: read

jobs:
  verify:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"

      - name: Install
        run: pip install -e ".[dev]"

      - name: Lint
        run: ruff check .

      - name: Type check
        run: mypy app

      - name: Unit tests (deterministic, no API)
        run: pytest -q

      - name: AI evaluations
        env:
          AI_API_KEY: ${{ secrets.AI_API_KEY }}
        run: python evaluations/evaluate.py

The order encodes the priorities. ruff and mypy are fast and free and catch obvious problems first. pytest runs the deterministic tests that mock the model — exact assertions on exact behavior. Only then do the AI evaluations spend tokens grading probabilistic output. If any stage fails, the pipeline stops before an image is ever built. The rest of this lesson assumes this gate is green; deployment is what happens after verification, never instead of it.

Building and Publishing the Container

A verified app becomes a deployable artifact by being built into a container image, scanned, and published to a registry. This is the pivot from CI to CD, and it is where the build once, promote the same artifact rule takes physical form: the image produced here is the only image that will ever reach production.

A Production Dockerfile

The image follows the container discipline from Part 6, GitHub Copilot with Docker. The properties that matter for a production LLM service:

• Non-root user — the container runs as an unprivileged user, not root, so a compromise is contained.
• Dependency caching and deterministic installs — dependencies are pinned and installed in a cached layer, so the build is reproducible and the artifact you scan is the artifact you run.
• Minimal base image — a slim base means fewer packages, fewer CVEs, a smaller attack surface.
• Explicit port — the app listens on a declared port the manifests reference.
• Health endpoint — the image exposes /health, which the orchestrator probes.
• No baked secrets — the API key is never in the image. It is supplied at runtime through the environment. An image is a distributable artifact; a key baked into a layer is a leaked key.

FROM python:3.12-slim AS base

# non-root user
RUN useradd --create-home appuser
WORKDIR /app

# deterministic dependency install (cached layer)
COPY pyproject.toml ./
RUN pip install --no-cache-dir .

# application code
COPY app/ ./app/

USER appuser
EXPOSE 8000

# health endpoint lives at /health; no secrets baked in
CMD ["uvicorn", "app.main:app", \
     "--host", "0.0.0.0", "--port", "8000"]

Every line here is a deployment decision: the non-root USER, the pinned base, the dependency layer that installs before code is copied so it caches, the explicit EXPOSE 8000, and — by their absence — no ENV AI_API_KEY=, no copied .env, nothing sensitive.

The Container Workflow

container.yml builds and publishes the image, and its shape enforces the safety rules from Part 10: build to prove it compiles, scan before publishing, and never publish from an untrusted pull request.

name: container

on:
  push:
    branches: [main]
    tags: ["v*"]

permissions:
  contents: read
  packages: write

jobs:
  build-publish:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: docker/setup-buildx-action@v3

      - name: Docker metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ghcr.io/${{ github.repository }}
          tags: |
            type=sha
            type=semver,pattern={{version}}

      - name: Log in to GHCR
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build image
        uses: docker/build-push-action@v6
        with:
          context: .
          push: false
          load: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}

      - name: Scan image with Trivy
        run: |
          trivy image --severity HIGH,CRITICAL \
            --exit-code 1 \
            ${{ fromJSON(steps.meta.outputs.json).tags[0] }}

      - name: Push image
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}

The verified pieces and the discipline around them:

• docker/setup-buildx-action@v3, docker/metadata-action@v5, docker/login-action@v3, docker/build-push-action@v6 — current major versions; confirm any uses: against its repository.
• Build, then scan, then push — the image is built and loaded locally, scanned, and only pushed if the scan passes. Publishing happens after review, not before.
• permissions: { contents: read, packages: write } — least privilege. The workflow reads the repo and writes packages, nothing more. GHCR authenticates with the built-in GITHUB_TOKEN, so no extra registry secret is needed.
• No publish from untrusted PRs — this workflow triggers on push to main and on tags, trusted refs. A pull-request build (in ci.yml) can prove the image builds without ever pushing it.

GHCR and Immutable Tags

The image is published to the GitHub Container Registry:

ghcr.io/example/llm-devops-app

❗ Important — Tag production images with immutable tags — a Git SHA or a semantic version — never latest. The chain that makes deployment auditable and rollback trivial is source commit → immutable image tag → deployment: a given running container maps back to exactly one commit, and “the last good version” is an unambiguous tag you can redeploy. latest is a moving pointer — it tells you nothing about what is running and gives rollback no fixed target. The metadata-action config above emits type=sha and type=semver tags for exactly this reason.

Scanning and Supply Chain

Scan the image before it ships. Two well-supported options:

• Trivy — trivy image <ref> scans OS packages and application dependencies for known CVEs; fail the build above a severity threshold (--severity HIGH,CRITICAL --exit-code 1), as the workflow does.
• Docker Scout — the same idea in a Build → Scan → Review → Publish flow, comparing against advisories and enforcing policy.

⚠️ Warning — A scan is a layer of assurance, not a proof of safety. Trivy and Docker Scout find known vulnerabilities in their databases; they cannot find an unknown one or a logic flaw. Fail the build on high-severity findings, but do not treat a clean scan as a guarantee the image is secure.

More broadly, an SBOM (software bill of materials) records exactly what is in the image, and provenance/attestation records how it was built — both raise supply-chain assurance and are worth adopting. They are layers of evidence, not a single guarantee. The security hardening guides go deeper on supply-chain patterns.

Environments, Secrets, and OIDC

Deployment introduces real credentials — a model API key and the credentials that deploy to a cluster — and GitHub Environments are where those live and where the approval gate attaches.

GitHub Environments. Define two environments, staging and production. Each provides:

• Environment-scoped secrets and variables — the production API key lives on the production environment, not the repo, so it is only usable by a job targeting that environment.
• Deployment history — a record of what was deployed where and when.
• Required reviewers — a job targeting the environment pauses until a designated person approves it. This is the human approval gate, covered below.

A secrets strategy. The credentials a deployment needs, and where they belong:

• AI_API_KEY — the model provider key, referenced as ${{ secrets.AI_API_KEY }}, injected into the container’s environment (and into Kubernetes via a Secret).
• CLOUD_DEPLOY_TOKEN / KUBECONFIG — the credentials that authenticate the deployment to the cloud or cluster, also stored as secrets — though OIDC, below, is the better option.

The rules that keep these safe:

• Never commit a .env with production credentials.
• Never bake a key into the Dockerfile.
• Never put a credential as a literal in workflow YAML — always ${{ secrets.X }}.
• Prefer avoiding long-lived credentials entirely.

OIDC — short-lived credentials. Instead of storing a long-lived cloud key, the workflow requests an OpenID Connect token from GitHub, exchanges it with the cloud provider for a short-lived credential scoped to the deployment, and uses that for the job’s duration:

GitHub Actions job
      |
   OIDC token   (id-token: write)
      |
  cloud / cluster trust
      |
short-lived credential
      |
    deployment

Nothing long-lived is stored, the credential expires quickly, and the trust relationship is cloud-neutral in concept — the same pattern works across providers, even though the exact configuration differs. Grant id-token: write only on the job that needs it, and verify the exact provider setup against current official documentation. OIDC removes the largest class of leaked-credential risk from the pipeline.

❗ Important — Keep the model/API credentials separate from the production infrastructure credentials, and never let the application inherit the CI/CD admin credentials that deploy it. The app needs to reach a model provider; it does not need — and must not have — the token that can change the cluster. A single blast-radius credential shared between “talk to the model” and “reconfigure production” is exactly the boundary an attacker or a bug would exploit.

Kubernetes Deployment

Kubernetes is the primary target for this lesson, and it connects the container from Part 6 with the orchestration from Part 7. The deployment is described by a manifest and executed by a workflow.

The Deployment Manifest

kubernetes/deployment.yaml uses the current GA apps/v1 API with the properties a production service needs — replicas, both probes, resource requests and limits, a non-root security context, and the API key injected from a Secret:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-devops-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: llm-devops-app
  template:
    metadata:
      labels:
        app: llm-devops-app
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
      containers:
        - name: app
          image: ghcr.io/example/llm-devops-app:sha-abc123
          ports:
            - containerPort: 8000
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 10
            periodSeconds: 15
          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"
            limits:
              cpu: "500m"
              memory: "256Mi"
          env:
            - name: AI_API_KEY
              valueFrom:
                secretKeyRef:
                  name: llm-api-secrets
                  key: api-key

❗ Important — The API key reaches the pod through valueFrom.secretKeyRef, which references a Secret by name and key — never a literal key value in the manifest. The manifest is committed to Git; a literal key in it is a leaked key in history. The referenced Secret (llm-api-secrets) is provisioned into the cluster securely and separately — by a sealed-secret controller, an external secrets operator, or a one-time kubectl apply from a credential the manifest never sees. The manifest says which secret to use, not what it is.

The rest of the manifest is deliberate: runAsNonRoot matches the non-root image; the readiness and liveness probes both hit /health (which does not call the model — see below); and the resource requests and limits size the pod. Note the image: is an immutable SHA tag, not latest — the deployment references one exact build.

Rolling Updates

The replicas: 3 plus Kubernetes’ default rolling-update strategy is what gives zero-downtime deploys. A rolling update replaces pods gradually, governed by two values worth understanding:

• maxUnavailable — how many pods may be down during the update. Keep it low (or zero) so capacity does not drop while deploying.
• maxSurge — how many extra pods may be created above the desired count during the update, so new pods come up before old ones go down.

The readiness probe is what makes this safe: a new pod receives traffic only once /health reports ready, so a broken new revision that never becomes ready does not take over from the working one. Rolling updates plus readiness probes mean a bad deploy degrades gracefully instead of causing an outage.

The Deploy Workflow

deploy.yml authenticates (via OIDC), updates the image, waits for the rollout, and smoke-tests it:

  deploy-staging:
    runs-on: ubuntu-latest
    environment: staging
    permissions:
      contents: read
      id-token: write        # for OIDC
    steps:
      - uses: actions/checkout@v4

      # authenticate to the cluster via OIDC
      # (provider-specific step omitted; short-lived cred)

      - name: Roll out the tested image
        run: |
          kubectl set image deployment/llm-devops-app \
            app=ghcr.io/example/llm-devops-app:${{ github.sha }}
          kubectl rollout status \
            deployment/llm-devops-app --timeout=120s

      - name: Smoke test
        run: ./scripts/smoke-test.sh staging

The flow is: authenticate with a short-lived OIDC credential, kubectl set image to the exact tested tag (or kubectl apply the manifest), kubectl rollout status to wait until the new pods are ready or the timeout trips, and then run a smoke test. The --timeout matters: without it, a rollout that never becomes ready would hang the job instead of failing it.

🛠️ DevOps Tip — Sizing an LLM application that calls a remote model API is undemanding: the heavy compute is in the provider’s cloud, so the pod needs CPU, memory, and outbound network — usually no GPU. The requests/limits above suit a modest API service. GPUs enter the picture only when you self-host a model, which is a later lesson (Part 14, coming soon).

Staging, Smoke Tests, and Production Approval

Staging is where the exact production artifact proves itself against a live model before a human is asked to approve it.

Deploy the exact image to staging. The staging deploy uses the same immutable image tag that will go to production — no rebuild, no “staging variant.” Staging exists to test the real artifact under realistic conditions, which only works if it is the real artifact.

The /health check. After the rollout, confirm /health returns an HTTP success and the expected JSON. This proves the process is up and serving — but it is a liveness check, not the only validation, because /health deliberately does not exercise the model.

The AI smoke test. Because /health says nothing about the model integration, a separate, low-cost smoke test sends one cheap prompt to the staging service and checks that:

• a valid structured response comes back,
• the required fields (summary, severity, recommended_checks) are present,
• there is no error, and
• the model integration works end to end (the app can reach the provider with the staging key).

POST /analyze  (one low-cost prompt)
      |
  200 OK ?
      |
  valid JSON with required fields ?
      |
  no provider error ?
      |
   model integration confirmed

⚠️ Warning — An AI smoke test validates structure and integration, not exact free-form text. It confirms the app returns a well-formed, non-error, schema-conformant response and can reach the model — it does not assert the summary equals a fixed string, because the output is probabilistic. Deep behavior belongs in the evaluation suite (which ran in CI); the smoke test is the shallow “is the live integration fundamentally alive?” check.

Require Human Approval Before Production

This is the gate the whole pipeline is built around. Every automated check can be green — deterministic tests, AI evaluations, the image scan, the staging health check, the AI smoke test — and a person still approves the promotion to production:

Staging
   |
Health Check
   |
AI Smoke Test
   |
Eval Results (reviewed)
   |
Human Approval    <-- required reviewer
   |
Production

Naming environment: production on the production job is what activates the required-reviewer rule — the run pauses at that job until a designated reviewer approves, and only then are the production secrets usable.

❗ Important — Deployment to production is not automatic just because an evaluation passed. A passing evaluation means the model’s output met a set of rules on a set of cases; it does not mean the change is correct for every input a probabilistic system will ever see. The required reviewer is the human judgment that a green pipeline can inform but never replace. AI-tested code does not get a bypass around the approval gate.

Build Once, Promote the Same Artifact

The single most important deployment rule in this lesson: deploy the exact image you tested, and never rebuild between staging and production.

Build (one immutable image)
   |
Scan
   |
Staging      (this image)
   |
Approval
   |
Production   (this SAME image)

❗ Important — A rebuild between staging and production is a different artifact than the one you validated. Dependencies may have shifted, a base image may have moved, the build may be non-reproducible in some subtle way — so the thing you ship to production would be something you never actually tested in staging. Build once, tag it immutably, scan it, promote that tag through every environment. The image that a reviewer approves is the image that runs.

Rollback and Deployment History

Deployments fail, and a plan to reverse them is not optional. Because every image carries an immutable tag, “the last good version” is an unambiguous, redeployable target.

Rollback mechanics. When a production deploy fails its smoke test, revert to the previous image:

Production Deploy
      |
  Smoke Test
      |
   Failure?
      |
Previous Image Tag (known-good)
      |
   Rollback

The concrete mechanisms:

• kubectl rollout undo deployment/llm-devops-app — reverts the Deployment to its previous revision, the fastest path on Kubernetes.
• Revert the desired image ref — kubectl set image (or re-apply the manifest) with the previous known-good immutable tag.
• GitOps revert — revert the desired-state commit and let the controller reconcile (see the GitOps section below).

❗ Important — Trigger automated rollback only on deterministic signals — a failed readiness probe, a failed smoke test, a spiking error rate — never on an AI-judged quality score. The signals that decide whether to reverse a production deploy must behave identically every run. A probabilistic model deciding to roll back would itself be a source of nondeterministic outages. The AI is graded by the pipeline; it never operates the pipeline.

Deployment history. Keep a record of every deployment tying together the pieces that let you answer “what is running and where did it come from”:

• the source commit,
• the image digest and immutable tag,
• the workflow run that deployed it,
• the environment, and
• the time.

Release metadata. A release record captures the same chain in one place — release identifier, commit, image, environment, evaluation result, scan result, and approver:

example release record (illustrative only)
release:   2026.04.0
commit:    abc123def
image:     ghcr.io/example/llm-devops-app:sha-abc123
env:       production
eval:      passed
scan:      passed
approver:  <reviewer>

❗ Important — The values above are illustrative placeholders, not real measurements. Never fabricate an evaluation score, a pass rate, a commit, or an approver to fill a template. A release record is only useful if every field is a real, recorded fact; an invented one is worse than a blank.

Testing Model and Prompt Changes

The model and the prompts are the parts of the app most likely to change behavior with the smallest diff, so they get their own testing discipline.

AI evaluation before deployment. Before any model or prompt change deploys, the evaluation suite from Part 10 grades it. The cases that matter most for this app assert:

• structured-response validity — the output conforms to the summary/severity/recommended_checks schema,
• expected safety — the app declines what it should decline,
• troubleshooting quality — the suggestions are relevant and useful,
• refusal of credential exposure — the app never reveals secrets or keys, and
• warning before destructive suggestions — a dangerous recommended check comes with a caution, not a bare command.

Prompt regression testing. A change to app/prompts/troubleshoot.txt triggers the evaluation suite through a path filter, and the useful comparison is candidate versus current — evaluate the main prompt and the proposed prompt against the same dataset and look at how behavior moved.

✅ Best Practice — Prompt regression testing is a comparison, not a simplistic exact pass/fail. You are asking “did this prompt change make behavior better or worse against our cases?” — a shift in pass rate and which cases moved — not whether one run happened to be byte-identical to a reference. Because the output is probabilistic, expect small run-to-run variation and design the suite so a guardrail regression fails the build while softer quality shifts inform the review.

Model changes are dependency-like. Swapping the provider or the model is like bumping a critical dependency: it triggers evaluation, deploys to staging, and gets reviewed before production.

⚠️ Warning — Do not silently swap the production model through environment configuration. Changing MODEL=... in a production environment variable is a behavior change with no commit, no evaluation, no staging, and no review — exactly the controls a model change most needs. Route model changes through the pipeline like any other change so the behavior shift is visible before it reaches users.

Reliability: Providers, Rate Limits, Latency, Cost

The model is an external dependency, which means the app must be built to survive the provider being slow, throttled, or down.

Provider failure strategy. When a model call fails, the app needs a deliberate response — a timeout so a call cannot hang, a bounded retry for transient errors, a fallback where one makes sense, and a degraded mode that returns a useful error rather than crashing.

⚠️ Warning — Do not blindly switch providers on a failure. Failing over to a different model provider silently changes behavior — different output, different guardrails, different structured-output reliability — none of which went through evaluation. A fallback path is fine when it is designed and evaluated; an automatic, unevaluated provider swap in the hot path is an untested behavior change shipping under load.

Rate limits. Providers throttle. Handle it with bounded retries and backoff, optional queueing, and user-visible errors when the limit genuinely cannot be met — not with an endless retry loop.

⚠️ Warning — Never retry endlessly. An unbounded retry loop against a rate-limited provider amplifies the problem — more calls, more throttling, more cost — and hides the real failure. Cap the retries, back off between them, and surface a clear error when the cap is reached.

Latency and the liveness probe. Model calls take seconds, not milliseconds, and this has a specific, critical consequence for Kubernetes:

❗ Important — The liveness probe must NOT depend on a successful model-provider inference. If /health (or a liveness check) called the model, a provider outage or a slow response would make healthy pods fail their liveness probe — so Kubernetes would kill and restart perfectly good pods during a provider incident, turning a degraded dependency into a self-inflicted outage. Keep the health/liveness path independent of inference: /health checks that the process is up and serving, nothing more. Test the model integration through a separate readiness or smoke path, apply a request timeout to every model call, and move heavy work to async jobs if a synchronous request would exceed sane latency.

Cost controls. Every model call spends money, so the deployment includes brakes: per-request limits, sensible model selection, bounded prompt size, controlled evaluation frequency (a small subset on PRs, the full suite nightly), and usage metrics so cost is visible.

🛠️ DevOps Tip — Manage cost with controls and metrics, not hardcoded dollar figures. Prices and limits change, so a number written into a lesson or a comment goes stale immediately. Instrument token usage, cap request rates, keep prompts tight, and check the provider’s current pricing when you need real figures — do not rely on a memorized amount.

Observability

A deployed LLM application needs monitoring that covers both the ordinary service signals and the model-specific ones. The observability stack guides cover the platform side; here is what to watch.

What to monitor:

• request rate and error rate — traffic and failure volume,
• latency — end-to-end and, where separable, model-call latency,
• model-provider failures — timeouts, 5xx, throttling from the provider,
• token usage — where the provider exposes it, for cost and drift,
• exceptions — application errors,
• evaluation regressions — a drop in pass rate across runs, and
• deployment events — what shipped when, correlated with the above.

Example custom metrics. Names an LLM service commonly exposes (labeled as illustrative custom metrics, not a standard):

example custom metric names
llm_requests_total
llm_request_errors_total
llm_request_duration_seconds

Logging. Log enough to trace a request, and nothing sensitive:

• Do log — a request ID, the provider/model identifier, latency, and success/failure.
• Never log — API keys, full sensitive prompts, or customer data.

⚠️ Warning — Never log API keys, full sensitive prompts, or customer data. Logs are stored, shipped, and widely readable; a key or a customer’s data in a log line is a leak that outlives the request. Log a request ID and metadata, not the payload.

Traceability. The request ID stitches the story together, so an incident can be followed from the user all the way to the model call:

User Request
     |
 Request ID
     |
  App Logs
     |
 Model Call
     |
  Response

Direct Deployment vs GitOps

There are two broad ways for GitHub Actions to get a new image running on the cluster, and the choice is about who talks to the cluster.

In direct deployment, the Actions workflow runs kubectl against the cluster itself. In GitOps, the workflow updates the desired state in a Git repository, and an in-cluster controller — Argo CD or Flux — notices the change and reconciles the cluster to match:

GitHub Actions
     |
 update deploy repo (desired state)
     |
 Argo CD / Flux  (in-cluster)
     |
   Kubernetes

The tradeoffs:

Aspect	Direct (Actions runs kubectl)	GitOps (controller deploys)
Mechanism	CI runs kubectl on the cluster	CI updates desired state; controller reconciles
Cluster access	CI holds cluster credentials	Cluster pulls; CI needs no cluster admin
Auditability	Workflow logs	Git history is the deploy record
Simplicity	Simpler, fewer moving parts	More components to run
Rollback	Redeploy previous tag	Revert the desired-state commit

Direct deployment is simpler and fine for many teams, at the cost of CI holding cluster credentials. GitOps adds components but gives strong auditability and lets the cluster pull its desired state rather than granting CI push access. This lesson is not a GitOps course — the point is only that both exist and that GitOps is the natural extension when you want the desired state, not the CI job, to be the source of truth.

Remote API vs Local Model

One architectural fork determines almost everything about a deployment’s footprint: does the app call a remote model API, or run a local/self-hosted model?

• Remote LLM API — the app is a Python API that needs the provider credentials and outbound connectivity to reach the model. No GPU, modest resources. This is what Part 12 deploys.
• Local / self-hosted model — the deployment additionally needs GPU hardware, model storage, an inference server, GPU scheduling, and much larger memory. This is a substantially heavier operation.

Part 12 focuses on the application layer around a remote model API — the common, lighter-weight case where the pipeline you have built is exactly what you need. The GPU and local-model material — hardware, inference servers, scheduling — is a later lesson (Part 14, coming soon).

Production Deployment Checklist

Before promoting an LLM application to production, confirm every item:

• Deterministic tests pass.
• AI evaluation passes its bar on the dataset.
• The image is scanned (Trivy or Docker Scout) with no high-severity findings.
• Secrets are configured — model key and deploy credentials as GitHub/Kubernetes secrets, none in code or images.
• Probes are configured — readiness and liveness, both independent of model inference.
• Resource limits reviewed — requests and limits sized for the workload.
• The image tag is immutable — a Git SHA or semver, not latest.
• Staging is deployed with the exact production image.
• Smoke tests pass — /health and a low-cost AI smoke test against staging.
• Monitoring exists — request/error/latency/provider metrics and safe logging.
• The rollback image is known — the previous known-good immutable tag.
• The approval is recorded — a required reviewer approved the production promotion.

30 GitHub Actions Ideas for LLM Deployment

Reusable deployment building blocks. Each is a starting point to adapt, review, and secure before you rely on it.

Build and package

1. Build the container image only after tests and AI evaluations pass.
2. Tag every image with an immutable Git SHA and a semantic version.
3. Never tag a production image latest.
4. Build a non-root, minimal-base image with no secrets baked in.
5. Use deterministic dependency installs so the artifact is reproducible.
6. Scan every image with Trivy or Docker Scout before publishing.
7. Generate an SBOM and provenance/attestation for built images.
8. Publish to GHCR with packages: write scoped to the publish job only.
9. Build on pull requests but push only from trusted refs.
10. Promote the exact same image from staging to production — never rebuild.

Secrets and access

11. Reference every secret as ${{ secrets.NAME }}, never a literal.
12. Scope production secrets to a protected GitHub Environment.
13. Use OIDC for short-lived cloud/cluster credentials over static keys.
14. Grant id-token: write only on the job that authenticates.
15. Separate the model API credential from the deployment credential.
16. Inject the API key via secretKeyRef in Kubernetes, never a literal.

Deploy to Kubernetes

17. Deploy with an apps/v1 Deployment, replicas, and both probes.
18. Keep the liveness probe independent of model inference.
19. Set resource requests and limits sized for a remote-API service.
20. Use kubectl rollout status with a timeout to wait for readiness.
21. Configure rolling updates with sensible maxUnavailable/maxSurge.
22. Deploy to a staging environment first, automatically.

Verify, approve, and observe

23. Run a /health check and a low-cost AI smoke test against staging.
24. Require a human reviewer on the production environment.
25. Record deployment history — commit, image digest, env, time.
26. Implement a rollback job to the previous known-good immutable tag.
27. Trigger automated rollback only on deterministic signals.
28. Emit custom metrics (llm_requests_total, error, duration).
29. Log request IDs and metadata — never keys, prompts, or customer data.
30. Trigger the evaluation suite on prompt and model-config changes.

25 Production Rules for LLM CI/CD

Rules specific to deploying probabilistic applications with real credentials. Treat these as widely-followed, documented guidance.

1. Verify — lint, type check, test, evaluate — before you package.
2. Build the image once and promote that exact artifact everywhere.
3. Never rebuild between staging and production.
4. Tag production images immutably; never use latest.
5. Bake no secrets into the image; supply them at runtime.
6. Run the container as a non-root user on a minimal base.
7. Scan the image before publishing and fail on high severity.
8. Store every credential as a secret; never a literal in YAML or a Dockerfile.
9. Scope production secrets to a protected environment.
10. Prefer OIDC short-lived credentials over long-lived static keys.
11. Separate model/API credentials from infrastructure credentials.
12. Inject Kubernetes secrets via secretKeyRef, never literal keys.
13. Keep the liveness probe independent of model-provider inference.
14. Set readiness probes so bad revisions never take traffic.
15. Set resource requests and limits on every workload.
16. Deploy to staging with the exact production image first.
17. Run a /health check and a low-cost AI smoke test on staging.
18. Validate structure and integration in smoke tests, not exact text.
19. Require human approval before production; no bypass for AI-tested code.
20. Automate rollback only on deterministic signals, never AI judgment.
21. Keep the previous known-good image ready as the rollback target.
22. Treat prompt and model changes as behavior changes through the pipeline.
23. Never swap the production model silently via environment config.
24. Never log API keys, full sensitive prompts, or customer data.
25. Record release metadata with real values; never fabricate a result.

Lab: Deploy a Python LLM Application to Kubernetes with GitHub Actions

Put the lesson together by deploying the demo app end to end. The point is the structure — build once, promote the same artifact, human-approve production — not the specific provider or cloud.

1. Scaffold llm-devops-app/ with the layout above (app/, evaluations/, kubernetes/, workflows, Dockerfile).
2. Build the FastAPI app with GET /health (no model call) and POST /analyze returning structured JSON (summary, severity, recommended_checks).
3. Put the model behind LLMClient in app/llm.py, reading the key from the environment.
4. Store the prompt as app/prompts/troubleshoot.txt and load it as a source file.
5. Define the response schema in app/schemas.py (Pydantic) so structured output is validated.
6. Reuse the CI workflow from Part 10 — lint, type check, unit tests, AI evaluations — and confirm it is green.
7. Write the Dockerfile — non-root, minimal base, deterministic install, explicit port, /health, no baked secrets.
8. Build locally with compose.yaml and confirm /health and /analyze work against a test key.
9. Write container.yml — buildx@v3, metadata@v5 emitting SHA and semver tags, login@v3 to GHCR, build-push@v6.
10. Build then scan the image with Trivy (--severity HIGH,CRITICAL --exit-code 1) before pushing.
11. Push to GHCR only from trusted refs, with packages: write scoped to the publish job.
12. Confirm the image is tagged immutably (SHA/semver), never latest.
13. Write kubernetes/deployment.yaml — apps/v1, replicas, both probes on /health, resource requests/limits, non-root securityContext.
14. Inject the API key via valueFrom.secretKeyRef, and provision the llm-api-secrets Secret into the cluster securely.
15. Add service.yaml and ingress.yaml for cluster networking and external routing.
16. Create staging and production GitHub Environments; scope the production key to production and add a required reviewer.
17. Configure OIDC so the deploy job requests a short-lived cluster credential; grant id-token: write only there.
18. Write the staging job in deploy.yml — authenticate, kubectl set image to the tested tag, kubectl rollout status --timeout.
19. Write the /health smoke check against staging (HTTP success + expected JSON).
20. Write the AI smoke test — one low-cost /analyze call, assert valid structured response and required fields, no error.
21. Gate production on environment: production so the run pauses for a required reviewer.
22. Deploy the exact same image to production after approval — confirm no rebuild happened.
23. Add a rollback job using kubectl rollout undo (or the previous immutable tag) triggered on smoke-test failure.
24. Instrument metrics — llm_requests_total, llm_request_errors_total, llm_request_duration_seconds.
25. Add safe logging — request ID, provider id, latency, success; never keys, prompts, or customer data.
26. Record release metadata — commit, image digest/tag, environment, eval result, scan result, approver (real values).
27. Test a prompt change — edit troubleshoot.txt, confirm the evaluation suite triggers via path filter.
28. Test a rollback — deploy a deliberately broken image, confirm the smoke test fails and rollback restores the last good tag.
29. Walk the production checklist and confirm every item.
30. Do a full dry run end to end and read the final architecture:

   Developer
      |
   GitHub
      |
     PR
      |
   Tests            (deterministic)
      |
   AI Eval          (probabilistic)
      |
 Docker Build
      |
  Image Scan        (Trivy / Scout)
      |
    GHCR            (immutable tag)
      |
 Staging K8s        (this image)
      |
 Smoke Tests
      |
Human Approval      <-- required
      |
Production K8s      (same image)

This is one of the most practical labs in the academy: at the end you have a real path from a commit to an approved, observed, reversible production deployment of a probabilistic application.

What’s Next

You now have the second half of the AI delivery story. Part 10 verified an AI application — tests and evaluations turning probabilistic behavior into a green check. Part 12 shipped it — an immutable image built once and scanned, promoted through staging to production as the same artifact, verified by a health check and an AI smoke test, gated behind a required human reviewer, deployed to Kubernetes with probes that stay independent of the model, watched with metrics and safe logs, and reversible with a known-good rollback tag. The through-line held the whole way: deterministic controls enforce the gates, the model is graded but never trusted to operate them, and a person approves production.

The next lesson, Part 13: GitHub Models Tutorial (coming soon), turns from deploying an application to experimenting with and comparing the models themselves in GitHub’s catalog and playground — the step before the abstraction and prompt you deployed here. When it publishes it will connect directly back to the LLMClient seam and the prompt-versioning discipline from this lesson.

To revisit the pieces this deployment rests on: Part 11, Building AI Agents with GitHub, builds the agents that generate proposed changes and send them through this very pipeline — the agent creates the change; this pipeline packages, tests, approves, and releases it. Part 10, GitHub Actions for AI Applications, is the CI foundation every stage here depends on. Part 6, GitHub Copilot with Docker, built the image; Part 7, GitHub Copilot with Kubernetes, built the manifests; and Part 9, GitHub Copilot with Python, built the app behind the abstraction. The GitHub AI Engineering Academy home has the full path, and the Docker guides, the Kubernetes and Helm guides, the security hardening guides, the hands-on Docker Academy, and the observability stack guides all go deeper on the tools this lesson wires into a production deployment.