Build an AI-Powered DevOps Pipeline with GitHub Actions and AI Agents

Across fifteen lessons this academy built the pieces of AI-assisted engineering one at a time. You learned to work in GitHub as an AI engineer, to drive Copilot in the CLI, the editor, and across Terraform, Docker, Kubernetes, Bash, and Python. You built AI-aware CI, then a real AI agent, then deployed an LLM application through GitHub Actions, and most recently made AI review a first-class part of the pull request. Each lesson solved a bounded problem. This capstone does something different: it assembles those pieces into one coherent, controlled, human-gated engineering system — and, just as importantly, it draws the lines that keep that system safe.

The temptation at the end of a journey like this is to imagine the finish line as an autonomous AI that takes an Issue and ships to production by itself. That is the wrong picture, and building it would be a mistake. The governing thesis of this entire lesson is simpler and far more durable:

AI generates, analyzes, and assists. Deterministic tooling validates. Humans retain authority over high-impact production decisions.

The pipeline is not “AI → production.” It is a loop with boundaries at every expensive step:

Issue → AI Agent → Plan → Branch → Code/Infra → Local Validation
      → Pull Request → GitHub Actions CI (lint / unit / IaC / scan / AI-eval / build)
      → AI Review (advisory) → Human Review → Merge
      → Immutable Artifact → Staging → Smoke/Integration
      → Human Production Approval → Production
      → Observability → Incident/Feedback → new Issue

Every arrow in that diagram is either an AI acceleration, a deterministic gate, or a human decision — never a place where a probabilistic model unilaterally changes production. This lesson walks that loop end to end, using a single realistic scenario the whole way through, and it is also the academy’s internal-link hub: each technology below links back to the lesson that teaches it in depth.

What an “AI DevOps Engineer” actually is

It helps to define the agent as a capability set rather than a personality. An AI DevOps Engineer, in this architecture, is the combination of task intake, repository understanding, AI reasoning, a set of constrained tools, local validation, the GitHub workflow, and — always — human oversight. Given those capabilities, it can:

• Interpret a GitHub Issue and extract acceptance criteria
• Inspect the repository to understand the existing code and infrastructure
• Propose a plan and a risk assessment
• Write application code, tests, and infrastructure on a feature branch
• Open a pull request with a full explanation
• Explain a failing CI job and propose a fix on its branch
• Respond to review comments

And it must not be able to:

• Merge its own pull request
• Modify production secrets
• Run a destructive Terraform apply
• Alter IAM or access policies
• Deploy arbitrary code
• Reach production systems at all

That second list is not a matter of the model choosing to behave. It is enforced by what tools the agent has, what credentials it holds, and what network it runs in. The capability set is deliberately shaped like a careful contributor who can propose anything through a PR and change nothing directly.

The nine-layer architecture

The system separates into nine layers, each with a single responsibility. Keeping them distinct is what lets you reason about security: a compromise or a bug in one layer is contained by the boundaries of the next.

Layer 1  Task Management     GitHub Issues — intent, acceptance criteria, risk, labels
Layer 2  AI Agent            Plan → branch → code/infra/tests → open PR (STOPS HERE)
Layer 3  Source Control      Branch protection / rulesets, CODEOWNERS, PR required
Layer 4  Validation          Actions: lint, types, unit, IaC validate, scans, AI eval
Layer 5  AI Review           Advisory diff review — risk, findings, missing tests
Layer 6  Human Review        A person reads code + AI + scans + plan, approves merge
Layer 7  Immutable Artifact  Build once: SHA/semver/digest, GHCR, scan, provenance
Layer 8  Deployment          Staging → validate → HUMAN prod approval → production
Layer 9  Observability       App/AI/dependency/K8s metrics → alerts → new Issue

Layers 1 through 6 get a change ready to merge. Layer 7 turns merged code into one artifact. Layers 8 and 9 promote and watch that artifact — with a human gate between staging and production. The agent lives only in Layer 2 and touches Layer 3 through a pull request; it has no reach into Layers 7, 8, or 9.

The capstone repository

Everything in this lesson lives in one repository whose structure encodes the architecture. The layout below is the shape of the capstone project; each directory maps to a layer above.

ai-devops-platform/
├── .github/
│   ├── workflows/
│   │   ├── agent.yml              # trusted trigger → run agent → open PR
│   │   ├── ci.yml                 # lint, types, unit, scans, path-filtered
│   │   ├── terraform.yml          # fmt, validate, TFLint, scan, plan
│   │   ├── container.yml          # build once, scan, push GHCR, attest
│   │   ├── ai-review.yml          # advisory AI review of the diff
│   │   ├── deploy-staging.yml     # deploy immutable artifact to staging
│   │   └── deploy-production.yml  # Environment-gated production promotion
│   ├── ISSUE_TEMPLATE/
│   │   └── agent-task.yml
│   ├── CODEOWNERS
│   └── AI_REVIEW.md               # reviewer guidance for AI findings
├── agent/
│   ├── github_client.py           # GitHub App auth → installation token
│   ├── planner.py                 # Issue → plan + risk assessment
│   ├── tools.py                   # constrained tools (NOT run_any_shell)
│   ├── executor.py                # orchestration loop
│   └── safety.py                  # policy, input sanitization boundaries
├── app/
│   ├── main.py                    # the service
│   ├── llm.py                     # provider-neutral model interface
│   └── prompts/
│       └── system.txt
├── tests/                         # deterministic unit tests (mock the model)
├── evaluations/                   # AI evaluations (probabilistic, graded)
├── terraform/
│   ├── modules/
│   └── environments/
│       ├── staging/
│       └── production/
├── kubernetes/
│   ├── base/
│   └── environments/
│       ├── staging/
│       └── production/
├── scripts/
├── Dockerfile
├── compose.yaml
├── pyproject.toml
├── .dockerignore
├── .gitignore
└── README.md

Notice what is not here: no secrets, no Terraform state, no long-lived credentials. Those live in GitHub Environments, a secure state backend, and a secrets manager — outside the repository the agent can read.

The scenario: Issue #147 — Add Redis-Based Response Caching

To keep the walkthrough concrete, every one of the twenty-five steps below follows a single change request. Here is the Issue:

Issue #147: Add Redis-Based Response Caching

The service calls the model provider on every request, which is slow and
expensive for repeated queries. Add Redis-backed caching:

- Cache successful responses only (never errors)
- TTL of 300 seconds
- Add unit tests for cache hit and cache miss
- Update compose.yaml for local Redis
- Update the Kubernetes manifests for the Redis dependency
- Update the README

Constraints:
- Do NOT expose Redis publicly
- Do NOT change production credentials
- Do NOT modify IAM or production secrets

Labels: agent-task, backend, docker, kubernetes, medium-risk

This is a medium-risk change: it touches application code, Compose, and Kubernetes, but not production secrets, IAM, or destructive infrastructure. That risk classification will shape how the pipeline treats it.

Walking the pipeline: twenty-five steps

Step 1 — Issue intake

A human writes Issue #147 with explicit acceptance criteria, an implied risk level, and labels. The labels (agent-task, backend, docker, kubernetes, medium-risk) are machine-readable routing hints, but the decisive act is human: a person has authorized this work and stated what “done” means and what the change must not touch.

Step 2 — Agent trigger

The agent begins only on a trusted trigger — a maintainer runs a manual workflow_dispatch, applies an approved label, or issues a trusted-collaborator slash command. It never starts from arbitrary public Issue content, because anyone can open an Issue and Issue text is untrusted. This single rule closes the most obvious abuse path: a stranger cannot get the agent to act simply by filing an Issue.

Step 3 — Agent authentication

The agent authenticates as a GitHub App, exchanging its credentials for a short-lived installation token scoped to this specific repository, with only the permissions it needs: read contents, write branches, open pull requests, comment on Issues. There is no broad organization token and no classic PAT tied to a human. This connects to the auth discipline from working in GitHub as an AI engineer: identity is scoped, short-lived, and least-privilege.

Step 4 — Reading the repository

The agent reads what it needs to understand the change: the Issue, the README, the architecture notes, the relevant Python in app/, the existing tests, compose.yaml, and the Kubernetes manifests. It does not read secrets, Terraform state, private keys, or unrelated large directories. Constraining what the agent reads limits both the tokens it burns and the blast radius of a leak.

Step 5 — Creating a plan

Before writing code, the agent produces an auditable, human-visible plan — posted as an Issue comment — describing the Redis abstraction it will add, the files it will change, and the tests it will write. A visible plan is reviewable before any code exists, which is far cheaper than reviewing a finished diff.

Step 6 — Risk assessment

The agent classifies the change. This one is medium: application code plus Compose plus Kubernetes, but no production secrets, no IAM, no destructive Terraform. The risk level determines how many gates apply later — a docs change and a production-secret change do not deserve the same ceremony, a point the risk-based automation table below makes explicit.

Step 7 — Creating a branch

The agent creates agent/issue-147-redis-cache. It never commits to main. The branch namespace (agent/) makes agent-authored work visible at a glance and easy to protect with rulesets.

Step 8 — The Python change

The agent adds a Redis abstraction with sensible timeouts, caches only successful responses, never logs secrets or full payloads, adds unit tests for hit and miss, and preserves the existing API surface. This is ordinary application engineering, the kind covered in driving Copilot for Python. A sketch of the caching wrapper:

# app/cache.py
import hashlib
import json
from typing import Optional

import redis

CACHE_TTL_SECONDS = 300


class ResponseCache:
    def __init__(self, client: redis.Redis) -> None:
        self._client = client

    def _key(self, prompt: str, model: str) -> str:
        digest = hashlib.sha256(f"{model}:{prompt}".encode()).hexdigest()
        return f"resp:{digest}"

    def get(self, prompt: str, model: str) -> Optional[dict]:
        raw = self._client.get(self._key(prompt, model))
        return json.loads(raw) if raw else None

    def set_success(self, prompt: str, model: str, response: dict) -> None:
        # Cache successful responses only; errors are never cached.
        self._client.setex(
            self._key(prompt, model),
            CACHE_TTL_SECONDS,
            json.dumps(response),
        )

Step 9 — The Compose update

The agent adds a Redis service to compose.yaml for local development, bound to the internal network and not published to a public port:

# compose.yaml (excerpt)
services:
  app:
    build: .
    environment:
      REDIS_URL: redis://cache:6379/0
      AI_API_KEY: ${AI_API_KEY}
    depends_on:
      - cache
  cache:
    image: redis:7-alpine
    # No published ports: Redis is reachable only on the internal network.
    networks:
      - internal
networks:
  internal:

This follows the container discipline from Copilot for Docker: minimal image, no needless exposure, secrets from the environment.

Step 10 — The Kubernetes update

The agent updates the manifests to express the Redis dependency and the new REDIS_URL configuration. If it adds an in-cluster Redis for the tutorial, it labels that clearly as a learning example, not a production data-tier recommendation — a managed cache is usually the right production choice. The application Deployment keeps the safe shape from Copilot for Kubernetes:

# kubernetes/base/deployment.yaml (excerpt)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ai-service
spec:
  replicas: 3
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: ai-service
          image: ghcr.io/example/ai-service@sha256:PLACEHOLDER
          ports:
            - containerPort: 8080
          env:
            - name: AI_API_KEY
              valueFrom:
                secretKeyRef:
                  name: ai-service-secrets
                  key: ai-api-key
            - name: REDIS_URL
              valueFrom:
                secretKeyRef:
                  name: ai-service-secrets
                  key: redis-url
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
          livenessProbe:
            httpGet:
              path: /health   # deliberately does NOT call the model provider
              port: 8080
          resources:
            requests:
              cpu: "100m"
              memory: "256Mi"
            limits:
              cpu: "500m"
              memory: "512Mi"
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]

The liveness probe hits /health, which checks the process — not the model provider. If liveness depended on inference, a provider outage would restart every pod and turn a degraded service into a dead one.

Step 11 — Terraform, only if required

Issue #147 does not require infrastructure changes, so the agent does not invent them — forcing an unnecessary Terraform change is its own anti-pattern. When infrastructure is required, the agent generates it and runs the deterministic safety chain from Copilot for Terraform: fmt, validate, TFLint, a security scan (Trivy trivy config or Checkov; tfsec is legacy and merged into Trivy), and terraform plan. It never runs a blind production apply.

Step 12 — Local validation

The agent validates before opening a PR: Ruff and mypy and pytest for Python, docker compose config for Compose, kubeconform for the manifests, and terraform fmt/validate/plan if infrastructure changed. This is the same local-first hygiene from Copilot for Bash — catch the cheap failures before they cost a CI run.

Step 13 — The agent reviews its own diff

The agent inspects its own git diff for unrelated changes, accidental secrets, noise, missing files, or — critically — accidental edits to .github/workflows/. An agent that quietly modifies the workflows that gate it is a red flag the self-review is designed to catch.

Step 14 — Commit

The agent commits with a conventional message referencing the Issue:

feat: add Redis response caching (#147)

Step 15 — Create the pull request

The agent opens a PR — and stops. The PR description is thorough: the Issue link, a summary, the architecture impact, the changed files, local test results, the risks, the deployment impact, and rollback notes. This is the agent’s authority ceiling. From here, deterministic CI, advisory AI review, and a human take over.

Step 16 — CI starts

Opening the PR triggers the CI workflow, which runs the full deterministic suite: lint → unit tests → type check → secret scan → dependency scan → Terraform validation → Kubernetes validation → Docker build → container scan → AI evaluation. Several disciplines shape this stage:

• Fast-fail ordering. Cheap checks first. You do not spend AI-evaluation tokens or GPU minutes before basic lint and unit tests have passed.
• Path-based CI. Jobs are conditional on what changed — terraform/**, kubernetes/**, app/**, Dockerfile — so a docs edit does not run the entire matrix.
• AI evaluation stage. Evaluations run only when app/prompts/**, app/llm.py, or evaluations/** change. Deterministic tests are not AI evaluations; the former assert exact behavior on mocked models, the latter grade probabilistic output against rules. This distinction comes straight from GitHub Actions for AI applications.
• GPU stage if required. Only when a GPU workload is affected does the pipeline reach for GPU runners, following GitHub Actions GPU workflows. Issue #147 touches neither prompts nor GPU code, so both stages are skipped.

A path-filtered skeleton:

# .github/workflows/ci.yml (excerpt)
name: CI
on:
  pull_request:
permissions:
  contents: read
concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: true
jobs:
  lint-and-test:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -e ".[dev]"
      - run: ruff check .
      - run: mypy app
      - run: pytest -q

Step 17 — Automated AI code review

A separate workflow runs the advisory AI review from automated AI code review. It takes the diff plus the test and scan summaries as input and produces a risk assessment, findings, missing-test observations, and operational-impact notes — posted as a PR comment. It is advisory. It is not a required check, and if it cannot run it reports “unavailable” rather than silently passing.

Step 18 — Human review

A human reviewer reads the code, the AI findings, the scanner output, the Terraform plan (if any), and the deployment effects, then decides. AI input is one signal among several, not the decision.

Two source-control controls back this up. CODEOWNERS routes sensitive paths to the right teams:

# .github/CODEOWNERS  (team handles are placeholders — replace with real teams)
/terraform/            @your-org/platform
/kubernetes/           @your-org/platform
/.github/workflows/    @your-org/devops
/app/                  @your-org/backend

Sensitive paths demand the review of the people accountable for them; an app-code change should not silently alter infrastructure or the workflows that gate the pipeline without the owning team seeing it.

Repository rulesets — the current recommended policy framework, with the branch-protection-to-rulesets migration generally available as of 2026-08-11 — enforce a required PR, required status checks (which must be success, skipped, or neutral to merge), required reviews, no force-pushes, and required workflows, layered at the repository, organization, and enterprise levels. Classic branch protection still exists, but rulesets are where new configuration should go. Verify the exact ruleset options against current GitHub documentation, as this area is actively evolving.

Step 19 — Merge

The PR merges only after CI passed, the AI review was considered, and a human approved. The agent cannot self-approve, and no configuration lets it bypass a failing required check.

Step 20 — Build the immutable artifact

On merge, the container workflow builds the image, scans it (Trivy or Docker Scout), tags it with an immutable reference — a Git SHA or semantic version, never latest for production — pushes it to GHCR, and produces a provenance attestation. Build once, deploy many: the same digest is what reaches every environment.

# .github/workflows/container.yml (excerpt)
permissions:
  contents: read
  packages: write
  id-token: write
  attestations: write
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - id: meta
        uses: docker/metadata-action@v5
        with:
          images: ghcr.io/${{ github.repository }}
          tags: |
            type=sha
            type=semver,pattern={{version}}
      - id: push
        uses: docker/build-push-action@v6
        with:
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
      - uses: actions/attest-build-provenance@v4
        with:
          subject-name: ghcr.io/${{ github.repository }}
          subject-digest: ${{ steps.push.outputs.digest }}
          push-to-registry: true

The actions/attest-build-provenance@v4 step generates SLSA provenance via Sigstore and in-toto, giving you a verifiable record of how and where the artifact was built. It needs id-token: write, attestations: write, and contents: read. Provenance attestation is available for public repositories on all plans; private repositories require GitHub Enterprise Cloud. Newer implementations may use actions/attest; verify the current action and major version against official documentation before adopting it. The point stands regardless of the exact action: the artifact is addressed by digest and its origin is attestable, so what runs in production is provably what CI built.

Step 21 — Deploy to staging

The deploy-staging workflow updates the staging Deployment to the exact immutable artifact — by digest — and waits on the rollout. Staging is a real GitHub Environment; it can require its own reviewers or run automatically, depending on your risk tolerance.

Step 22 — Staging validation

Staging is where you prove the change against a live model before a human is asked to approve production. Validation covers health, integration, and cache behavior:

• Health check. GET /health confirms the process is up — necessary but not sufficient, so it is never the only check.
• Integration test. Request 1 goes to the model and stores the response; request 2 is served from Redis with no model call. Mocks and a real staging Redis make this exact.
• AI smoke evaluation. A small, bounded evaluation confirms the live model integration still works — a handful of cases, not the full golden dataset.

# tests/integration/test_cache_behavior.py (excerpt)
def test_second_request_hits_cache(client, model_spy):
    r1 = client.post("/ask", json={"prompt": "what is a container?"})
    r2 = client.post("/ask", json={"prompt": "what is a container?"})
    assert r1.json() == r2.json()
    # The model provider was called exactly once; the second was cached.
    assert model_spy.call_count == 1

Step 23 — Human production approval

Staging passing is a signal, not authorization. Production is a GitHub Environment with required reviewers: the promotion job pauses until a designated human approves it, and only then are the production-scoped secrets usable. This is the second and most important human gate in the pipeline.

Step 24 — Production deployment

On approval, the deploy-production workflow promotes the exact staging artifact by digest — no rebuild — into the production Kubernetes environment, which carries the probes, limits, secure context, and immutable image reference from Copilot for Kubernetes. Deployment verification is deterministic: wait on kubectl rollout status, hit the health endpoint, run a bounded smoke test, and watch metrics and error rate before declaring success.

Step 25 — Observability and the feedback loop

Production is watched on four layers:

• Application: request rate, error rate, latency.
• AI: model-provider failures, inference latency, evaluation-quality trend, token usage and cost.
• Redis: connection failures, cache hit/miss ratio, memory.
• Kubernetes: pod restarts, readiness, resource saturation.

Example custom metric names (illustrative) might include ai_provider_request_failures_total, ai_inference_latency_seconds, cache_hits_total, and cache_misses_total. Whatever you name them, the observability layer is what closes the loop:

Production → metrics / logs / alerts → problem detected
          → automated Issue created → AI agent analyzes
          → proposed fix PR → CI → AI review → human review
          → merge → artifact → staging → HUMAN prod approval → Production

That loop is the whole system in miniature, and it never shortcuts the human gates.

The feedback loop in detail

Incident workflow. An alert fires, an Issue is created (verify the exact GitHub API against current docs; include the alert, environment, timestamp, sanitized logs, a dashboard reference, and the affected service — never secrets), the agent analyzes it and drafts a diagnosis, and it may open a proposed-fix PR. Alerts never trigger unreviewed production changes; they trigger proposals.

AI-assisted incident analysis summarizes logs, correlates recent deploys, suggests checks, and lists likely causes — always structured as three separate sections so a human is never handed a confident-sounding fabrication:

FACTS:               error rate rose from 0.2% to 4.1% at 14:03 UTC;
                     deploy of sha256:abc… completed 14:01 UTC.
HYPOTHESES:          the new cache key may collide across models;
                     Redis connection pool may be undersized.
RECOMMENDED CHECKS:  inspect cache-key construction for model scoping;
                     check redis_connection_failures_total since 14:00.

The agent never asserts a single root cause as fact. Facts are facts; everything else is labeled as a hypothesis or a check.

Rollback architecture

Deploy → health / metrics → regression detected
      → rollback → previous immutable artifact → verified healthy

Rollback comes in two forms. Automated rollback fires on deterministic signals only — a failed rollout, failing readiness, or an error rate over threshold — using kubectl rollout undo, reapplying the previous digest, or reverting a GitOps commit. Manual rollback is always available. An LLM’s quality judgment is never the sole authority to roll back production, because moving production must rest on signals that behave identically every run. The target artifact is known in advance; rollback is engineered, not improvised.

Safety boundaries

Terraform safety boundary

Agent generates TF → fmt / validate / TFLint / security scan → plan
   → PR → HUMAN review → SEPARATE approved apply workflow

The agent that writes Terraform is not the workflow that applies it, and a production apply never runs from an untrusted PR with real credentials.

Kubernetes safety boundary

Agent proposes manifests → kubeconform / policy (Kyverno/OPA)
   → PR → HUMAN review → approved deploy workflow → rollout

Coding and applying are separate identities with separate credentials.

Secret architecture

Four distinct credential domains, so no single compromise grants everything:

Agent credential      → read repo, write branch, create PR       (nothing else)
CI credential         → run checks, read repo                    (no deploy)
Deployment credential → deploy to an approved Environment        (scoped)
Application credential→ the model API key at runtime             (in the pod)

One credential does not equal all capabilities. This separation of duties is what makes a leaked agent token harmless to production.

OIDC, sandboxing, and network security

Prefer OIDC to exchange a GitHub identity for short-lived cloud credentials instead of storing static keys. Run the agent in a disposable sandbox — a throwaway container, isolated VM, or restricted runner — never on a production host. Ensure the agent’s network has no path to the production database, the management network, the cluster admin plane, or the secrets system. The agent should be technically unable to reach production, not merely instructed not to.

Model security and prompt injection

Every text the model reads — Issues, PRs, repository files, logs, external docs — is untrusted. Prompt injection is not solved by prompt wording; a determined instruction embedded in a file will sometimes influence the model. The defense is enforcement outside the model: permissions, sandboxing, tool restrictions, and credential boundaries. If the agent has no tool that can merge and no credential that can reach production, a successful injection produces, at worst, a PR a human rejects. This is the single most important security idea in the lesson — it comes from building AI agents with GitHub, and it is the reason the create-PR ceiling exists.

The tool-permission architecture

The agent’s tools are constrained by construction. It is given specific, auditable tools — not a general run_any_shell_command, which is an arbitrary escalation path.

Capability                       Allowed   Why
---------------------------------------------------------------------------
Read file                          ✓        needs to understand the repo
Search code                        ✓        needs to locate relevant code
Run tests                          ✓        validates its own change
Write to feature branch            ✓        makes the proposed change
Create pull request                ✓        the authority ceiling
---------------------------------------------------------------------------
Merge pull request                 ✗        humans + required checks decide
Terraform apply                    ✗        separate approved workflow
kubectl against production         ✗        deployment identity, not agent
Read production secrets            ✗        never in the agent's domain
Delete infrastructure              ✗        destructive; human-gated
Run arbitrary shell                ✗        arbitrary shell = arbitrary power

Autonomous versus controlled

It is worth stating the contrast plainly, because it is the crux of the whole lesson.

AI → Production  (high risk)         The controlled agentic flow (safe)
--------------------------------     -----------------------------------------
Model decides to ship                Human approves merge and production
No deterministic gates               Deterministic CI gates every merge
Model judgment moves prod            Deterministic signals move prod
Injection can reach production       Injection reaches, at most, a PR
No accountable party                 Named humans accountable at each gate
Rollback improvised                  Rollback engineered to a known artifact

The left column is faster to build and impossible to trust. The right column is the point of the last sixteen lessons.

Progressive autonomy: Levels 0–5

You do not deploy the top of this ladder on day one. You climb it, earning trust with evidence, and every rung keeps the same non-negotiables.

Level 0  Explain            AI explains code and CI failures; changes nothing.
Level 1  Suggest            AI suggests changes; a human applies them.
Level 2  Draft PR           Agent opens draft PRs; humans finish and merge.
Level 3  Fix CI on branch   Agent iterates on its own branch until CI is green.
Level 4  Deployment candidate  Agent produces a candidate; humans approve deploy.
Level 5  Deterministic deploy  A deterministic system deploys AFTER human approval.

Note that even Level 5 is not “AI deploys.” It is “a deterministic system deploys after a human approves.” The human production gate never disappears; higher autonomy only changes how much the agent drafts.

Risk-based automation

Not every change deserves the same ceremony. Match the gates to the stakes.

Risk       Example                       Automation
---------------------------------------------------------------------------
Low        Docs, comments                Agent → PR
Medium     App code (Issue #147)         Agent → PR + tests + human review
High       Terraform / IAM               PR + tests + review + EXTRA approval
Critical   Prod secrets / destructive    Manual — no agent authority at all

Issue #147 sits at Medium: a PR with tests and human review. A change to production secrets would be Critical and outside the agent’s reach entirely.

Workflow architecture

Do not put everything in one workflow. The seven files each own a responsibility:

agent.yml              trusted trigger → run agent → open PR
ci.yml                 lint, types, unit, scans (path-filtered, fast-fail)
terraform.yml          fmt, validate, TFLint, scan, plan
container.yml          build once, scan, push GHCR, attest provenance
ai-review.yml          advisory AI review of the diff
deploy-staging.yml     deploy immutable artifact to staging
deploy-production.yml  Environment-gated production promotion

Share logic with reusable workflows (workflow_call) rather than copy-paste. Avoid circular triggers, document the dependencies between workflows, and put concurrency limits, timeouts, and cost controls on every job — an AI stage without a timeout is an open-ended bill.

Failure handling, resilience, and audit

Each stage fails safely: tests fail → the PR is blocked; AI review fails → it is marked unavailable, never silently approved; the build fails → no artifact exists to deploy; staging fails → production is unavailable; the production smoke test fails → rollback or escalate. When the AI provider is down, deterministic CI keeps running, AI review shows as unavailable, and high-risk changes still cannot bypass required human review. The pipeline degrades to “slower and more manual,” never to “unguarded.”

Everything leaves an audit trail: the Issue, the workflow runs, the commits, the PR, the reviews, the approvals, and the deployment history. An illustrative end-to-end trace (sample IDs):

Issue #147 → branch agent/issue-147-redis-cache → commit abc123
→ PR #152 → CI green → AI review (advisory) → human approval → merge
→ image ghcr.io/…@sha256:def456… → staging validated
→ HUMAN production approval → production → observed healthy

For compliance and governance, this architecture naturally produces approval records, evidence of least privilege, separation of duties, and full traceability — though it offers no legal guarantees, and you should map it to your own obligations. The human roles are explicit: a developer authors and owns the change, a platform engineer owns infrastructure and the pipeline, a security reviewer owns sensitive paths, and a production approver owns go-live. The agent does not replace any of them; it accelerates their work while they remain accountable.

AI DevOps anti-patterns

Avoid all of these:

1. Letting an AI agent deploy directly to production.
2. Giving the agent a general run_any_shell_command tool.
3. Granting the agent a broad organization or classic PAT.
4. Letting the agent hold or read production secrets.
5. Allowing the agent to merge its own pull requests.
6. Running untrusted PR code with pull_request_target and secrets.
7. Trusting Issue or PR text as instructions the model must obey.
8. Making AI code review a required, blocking merge check.
9. Treating a passing AI evaluation as authorization to ship.
10. Using latest image tags for production instead of immutable references.
11. Rebuilding the artifact between staging and production.
12. Making the liveness probe depend on model-provider inference.
13. Letting an AI quality score be the sole rollback trigger.
14. Running the agent on a production host instead of a disposable sandbox.
15. Skipping the human approval gate on production promotion.

Production-readiness checklist

GitHub

• Repository rulesets require PRs, status checks, and reviews.
• CODEOWNERS routes terraform/, kubernetes/, and .github/workflows/.
• Production is a GitHub Environment with required reviewers.

Agent

• Authenticates as a scoped GitHub App with a short-lived token.
• Runs only on a trusted trigger, never arbitrary Issue content.
• Uses constrained tools; no run_any_shell_command.
• Stops at create-PR; cannot merge, deploy, or reach production.
• Runs in a disposable sandbox with no production network path.

Code

• Ruff, mypy, and pytest pass; secrets never logged.
• AI evaluations gate prompt and model changes.

Terraform

• fmt, validate, TFLint, and a security scan run in CI.
• Apply is a separate, human-approved workflow.

Docker

• Non-root, minimal base, no secrets baked in.
• Image scanned with Trivy or Docker Scout.

Kubernetes

• apps/v1 with readiness and liveness probes (liveness ≠ inference).
• Resource requests/limits and non-root securityContext set.
• Secrets via secretKeyRef, validated with kubeconform and policy.

AI

• Model interface is provider-neutral and mockable.
• AI review is advisory; deterministic checks are required.

Deployment

• Build once; promote the same digest through environments.
• Provenance attestation produced and verifiable.
• Production promotion requires human approval.

Operations

• Four-layer observability (app / AI / dependency / Kubernetes).
• Rollback is engineered to a known artifact on deterministic signals.
• Full audit trail from Issue to production.

40 AI-Powered DevOps Pipeline Ideas

1. Agent that triages new Issues and proposes labels and risk levels.
2. Agent that drafts acceptance criteria for under-specified Issues.
3. Agent that opens a plan comment before writing any code.
4. Agent that adds missing unit tests to an existing PR.
5. Agent that raises test coverage on a targeted module.
6. Agent that updates a dependency and runs the suite on its branch.
7. Agent that migrates a workflow from a deprecated action version.
8. Agent that generates a Dockerfile that passes a container scan.
9. Agent that adds readiness and liveness probes to a Deployment.
10. Agent that adds resource requests and limits to manifests.
11. Agent that converts a plaintext env value to a secretKeyRef.
12. Agent that drafts a Terraform module and a plan summary.
13. Agent that summarizes a Terraform plan for human review.
14. Agent that flags a security group opening to the world in a plan.
15. Agent that adds path filters to a bloated CI workflow.
16. Agent that splits a monolithic workflow into responsibility files.
17. Agent that adds concurrency and timeout controls to jobs.
18. Agent that converts a static cloud key to an OIDC exchange.
19. Advisory AI review that annotates diffs with risk and missing tests.
20. AI evaluation stage gated on prompt and model changes.
21. AI smoke test after a staging deploy.
22. Agent that drafts a rollback runbook for a service.
23. Agent that proposes a canary or blue-green strategy as a PR.
24. Agent that generates a golden dataset skeleton for evaluations.
25. Agent that adds structured JSON output to make a feature testable.
26. Agent that creates an Issue automatically from an alert.
27. Agent that produces a Facts/Hypotheses/Checks incident summary.
28. Agent that correlates an error spike with a recent deploy.
29. Agent that drafts a fix PR for a recurring, well-understood failure.
30. Agent that updates a README to match code changes in the same PR.
31. Agent that generates provenance-attestation steps for a build.
32. Agent that adds a secret scan to a pipeline missing one.
33. Agent that proposes CODEOWNERS entries for sensitive paths.
34. Agent that drafts a repository ruleset from a template.
35. Agent that standardizes reusable workflows across repositories.
36. Agent that reports agent-PR acceptance rate as a metric.
37. Agent that measures AI-review agreement with human decisions.
38. Agent that tracks deployment frequency and rollback rate.
39. Agent that flags workflows running untrusted code with secrets.
40. Agent that audits the org for latest tags in production manifests.

Every one of these stops at a PR or a proposal. None deploys.

30 Rules for Safe AI-Powered DevOps

1. AI generates and analyzes; humans decide what ships to production.
2. The agent stops at create-PR — never merge, never deploy.
3. Give the agent constrained tools, never a general shell.
4. Authenticate the agent as a scoped GitHub App, not a broad PAT.
5. Use short-lived tokens and OIDC; avoid static long-lived keys.
6. Separate agent, CI, deployment, and application credentials.
7. Keep production secrets in a protected Environment the agent cannot reach.
8. Run the agent in a disposable sandbox with no production network path.
9. Trigger the agent only from trusted actions, never arbitrary Issue text.
10. Treat all Issue, PR, and repository text as untrusted input.
11. Enforce security outside the model, not in the prompt.
12. Make AI code review advisory; keep required checks deterministic.
13. Never let AI review silently pass when it is unavailable.
14. Require human review before any merge to the default branch.
15. Require human approval before any production promotion.
16. Green CI is a signal, not authorization to ship.
17. Build once and promote the same immutable artifact by digest.
18. Never use latest for production images.
19. Scan every image and attest its provenance.
20. Validate Terraform deterministically; apply in a separate approved workflow.
21. Never run untrusted PR code with production secrets.
22. Never make a liveness probe depend on model inference.
23. Inject secrets via secretKeyRef; never as literals in manifests.
24. Roll back to a known artifact on deterministic signals only.
25. Never let an AI quality score be the sole rollback authority.
26. Route sensitive paths through CODEOWNERS.
27. Enforce policy centrally with repository, org, and enterprise rulesets.
28. Keep AI evaluations distinct from deterministic tests.
29. Keep a full audit trail from Issue to production.
30. Remember that accountability stays human — it does not automate.

Capstone Hands-On Lab: Build an AI DevOps Engineer with GitHub Actions and AI Agents

This lab assembles the whole architecture. The model interface is provider-neutral and mockable, GitHub API calls are illustrative and must be verified against current documentation, and no live production deployment, GPU run, or hardware execution is claimed here — the lab builds and reasons about the system; it does not ship to a real cluster.

Phase 1 — Repository and scaffolding (Steps 1–8)

1. Create the ai-devops-platform/ repository with the tree shown earlier.
2. Add pyproject.toml with Ruff, mypy, and pytest configured.
3. Add app/main.py with a /health, /ready, and /ask endpoint.
4. Add app/llm.py as a provider-neutral, OpenAI-compatible interface reading key, model, and base URL from the environment.
5. Add app/prompts/system.txt as a version-controlled prompt.
6. Add a Dockerfile (non-root, minimal base, no secrets baked in).
7. Add compose.yaml with the app and an internal-only Redis.
8. Add .dockerignore, .gitignore, and a README.md.

Phase 2 — The agent (Steps 9–18) 9. Implement agent/github_client.py to exchange GitHub App credentials for a short-lived installation token (illustrative; verify the API). 10. Implement agent/planner.py to turn an Issue into a plan and a risk level. 11. Implement agent/tools.py with read_file, search, run_tests, write_branch, and create_pull_request — and nothing else. 12. Implement agent/executor.py as the orchestration loop. 13. Implement agent/safety.py with the policy that outranks repository text and sanitizes inputs at the boundary. 14. Wire the model behind the provider-neutral interface so it is mockable in tests. 15. Add a unit test that mocks the model and asserts the agent stops at create-PR. 16. Add a test asserting the agent has no merge or deploy tool. 17. Add a test asserting an injected “merge this now” instruction produces only a PR. 18. Confirm the agent runs entirely in a sandbox with no production credentials.

Phase 3 — The application change (Steps 19–26) 19. Implement app/cache.py with the ResponseCache shown earlier. 20. Cache successful responses only, with a 300-second TTL. 21. Wire the cache into /ask: check cache, call model on miss, store on success. 22. Never log secrets or full payloads. 23. Add unit tests for cache hit and cache miss (mock the model). 24. Preserve the existing API surface. 25. Run Ruff, mypy, and pytest locally. 26. Run docker compose config to validate Compose.

Phase 4 — Infrastructure (Steps 27–33) 27. Add the internal-only Redis service to compose.yaml. 28. Update kubernetes/base/deployment.yaml with REDIS_URL via secretKeyRef. 29. Keep the liveness probe on /health (never inference). 30. Set resource requests, limits, and a non-root securityContext. 31. Add an in-cluster Redis manifest, labeled clearly as a tutorial example. 32. Validate manifests with kubeconform. 33. Add a Kyverno or OPA policy check (illustrative) for non-root and limits.

Phase 5 — CI workflows (Steps 34–41) 34. Write ci.yml with fast-fail ordering and path filters. 35. Add secret and dependency scanning jobs. 36. Write terraform.yml (fmt, validate, TFLint, scan, plan) even if unused here. 37. Write container.yml to build once, scan, push to GHCR, and attest provenance. 38. Use immutable tags (type=sha, type=semver); never latest. 39. Set least-privilege permissions: on every job. 40. Add concurrency groups and timeouts. 41. Add an AI-evaluation job gated on prompt and app/llm.py changes.

Phase 6 — Review and merge gates (Steps 42–48) 42. Write ai-review.yml as an advisory, non-blocking annotation. 43. Ensure AI review reports “unavailable” instead of passing silently. 44. Add CODEOWNERS for terraform, kubernetes, and workflows. 45. Configure a repository ruleset requiring PRs, checks, and reviews (verify options). 46. Confirm required status checks must be success/skipped/neutral to merge. 47. Confirm the agent cannot self-approve or bypass a failing check. 48. Merge only after CI passes, AI review is considered, and a human approves.

Phase 7 — Deployment (Steps 49–56) 49. Write deploy-staging.yml to deploy the exact digest to a staging Environment. 50. Wait on kubectl rollout status and hit /health. 51. Add the cache-behavior integration test (request 2 served from Redis). 52. Add a small, bounded AI smoke evaluation. 53. Configure the production Environment with required reviewers. 54. Write deploy-production.yml to promote the same digest after approval. 55. Scope production secrets to the production Environment only. 56. Add deterministic deployment verification (rollout, health, smoke, error rate).

Phase 8 — Observability and feedback (Steps 57–62) 57. Emit application metrics (requests, errors, latency). 58. Emit AI metrics (provider failures, inference latency, eval trend, cost). 59. Emit cache metrics (cache_hits_total, cache_misses_total, memory). 60. Emit Kubernetes signals (restarts, readiness, saturation). 61. Wire an alert to auto-create an Issue (verify the API; include no secrets). 62. Have the agent produce a Facts/Hypotheses/Checks incident summary.

Phase 9 — Safety hardening and review (Steps 63–66) 63. Verify credential separation across agent, CI, deploy, and app. 64. Verify the rollback path targets a known artifact on deterministic signals. 65. Walk the production-readiness checklist end to end. 66. Confirm every automated step stops at a PR or a proposal — none deploys.

The finished system, drawn in full:

        ┌──────────────────────────────────────────────────────────────┐
        │                     GitHub Issue #147                          │
        │            (intent, acceptance criteria, risk, labels)         │
        └───────────────────────────────┬──────────────────────────────┘
                          trusted trigger │
                                          ▼
        ┌──────────────────────────────────────────────────────────────┐
        │   AI AGENT  (GitHub App, short-lived token, sandbox)           │
        │   plan → branch → code / tests / infra → local validate        │
        │   constrained tools · policy > repo text · STOPS AT CREATE-PR  │
        └───────────────────────────────┬──────────────────────────────┘
                                          ▼  pull request
        ┌──────────────────────────────────────────────────────────────┐
        │   GITHUB ACTIONS CI  (deterministic, fast-fail, path-filtered) │
        │   lint · types · unit · secret/dep scan · IaC validate         │
        │   docker build · container scan · AI eval (gated)              │
        └───────────────────────────────┬──────────────────────────────┘
                                          ▼
        ┌───────────────────────────┐   ┌──────────────────────────────┐
        │  AI REVIEW (advisory)     │──▶│  HUMAN REVIEW  (decides merge)│
        │  risk · findings · gaps   │   │  CODEOWNERS · rulesets        │
        └───────────────────────────┘   └───────────────┬──────────────┘
                                                          ▼  merge
        ┌──────────────────────────────────────────────────────────────┐
        │   IMMUTABLE ARTIFACT  (build once)                             │
        │   ghcr.io/…@sha256:… · scan · provenance attestation           │
        └───────────────────────────────┬──────────────────────────────┘
                                          ▼  same digest
        ┌──────────────────────────────────────────────────────────────┐
        │   STAGING  → health · integration (cache) · AI smoke            │
        └───────────────────────────────┬──────────────────────────────┘
                                          ▼
                          ┌───────────────────────────────┐
                          │   HUMAN PRODUCTION APPROVAL    │  ◀── the gate
                          │   (Environment + reviewers)    │
                          └───────────────┬───────────────┘
                                          ▼  same digest, promoted
        ┌──────────────────────────────────────────────────────────────┐
        │   PRODUCTION  (K8s: probes · limits · non-root · secretKeyRef) │
        │   deterministic verify · rollback on deterministic signals     │
        └───────────────────────────────┬──────────────────────────────┘
                                          ▼
        ┌──────────────────────────────────────────────────────────────┐
        │   OBSERVABILITY  (app · AI · dependency · Kubernetes)          │
        │   alert → new Issue → agent analysis → proposed fix PR ────────┼──┐
        └──────────────────────────────────────────────────────────────┘  │
                          ▲                                                 │
                          └─────────────────  feedback loop  ──────────────┘

Every arrow that touches production passes through a human. That is the design.

The internal-link hub: where each piece was taught

This capstone stands on the whole academy. Each technology links back to the lesson that teaches it in depth:

• Working in GitHub as an AI engineer — Part 1
• GitHub Copilot for DevOps — Part 2
• Copilot in the CLI — Part 3
• Copilot in VS Code — Part 4
• Copilot for Terraform — Part 5
• Copilot for Docker — Part 6
• Copilot for Kubernetes — Part 7
• Copilot for Bash — Part 8
• Copilot for Python — Part 9
• GitHub Actions for AI applications — Part 10
• Building AI agents with GitHub — Part 11
• Deploying LLM applications with GitHub Actions — Part 12
• GitHub Models (retired 2026-07-30; provider-neutral interfaces and the Copilot coding agent are the path forward) — Part 13
• GitHub Actions GPU workflows — Part 14
• Automated AI code review — Part 15

If you name the books that support this journey in prose — Learning GitHub Actions, GitHub Copilot Unleashed, and The Ultimate Git and GitHub Guide — they map cleanly onto the Actions, Copilot, and Git foundations the pipeline rests on.

GitHub AI Engineering Academy Complete

You completed all 16 lessons.

You began by learning to work inside GitHub as an AI engineer, then drove Copilot across the command line, the editor, Terraform, Docker, Kubernetes, Bash, and Python. You built AI-aware continuous integration, then a real AI agent, deployed an LLM application through GitHub Actions, understood the retirement of GitHub Models and the move to provider-neutral interfaces, ran GPU workflows, and made AI review a first-class part of the pull request. This capstone assembled all of it into one controlled, human-gated engineering system — from a GitHub Issue, through an agent that stops at create-PR, through deterministic CI and advisory AI review and human approval, to an immutable artifact promoted across staging and production behind a human gate, and back again through observability into the next Issue.

The governing principle is worth carrying with you long after the syntax fades:

The future of DevOps is not an AI with unrestricted production access. It is an engineering system where AI accelerates planning, coding, troubleshooting, testing, review, and automation — while GitHub, deterministic tooling, security controls, observability, and humans provide the boundaries.

AI generates and analyzes. Deterministic tooling validates. Humans retain authority over the decisions that are expensive to get wrong. Build systems that reflect that, and AI makes your engineering faster without making it less accountable.

Continue from the pillars of the academy:

• Academy home — all 16 lessons
• Start again from the foundations — Part 1: GitHub for AI engineers
• Containers with AI — Copilot for Docker
• Orchestration with AI — Copilot for Kubernetes
• Infrastructure with AI — Copilot for Terraform
• Deploying AI applications — Part 12: LLM applications with GitHub Actions

Thank you for building it the right way.