GitHub Copilot with Kubernetes: AI-Assisted Kubernetes for DevOps Engineers

Kubernetes is where containers stop being a single artifact and become a running system — scheduled, scaled, networked, and kept alive across failures. It is also almost entirely YAML: Deployments, Pods, Services, ConfigMaps, Secrets, Ingress, Jobs, CronJobs, RBAC, resource requests, probes, affinity and tolerations, storage claims, Helm values, plus the kubectl commands, logs, and events you use to operate it all. That verbosity is exactly where an AI pair programmer helps most — and exactly where an unreviewed draft does the most damage. A manifest that parses is not automatically secure, correct, or production-ready.

This is Part 7 of the GitHub AI Engineering Academy. Part 6, GitHub Copilot with Docker, built and hardened the container image; Part 5, GitHub Copilot with Terraform, provisioned the infrastructure those clusters run on. This lesson takes the image to the cluster and keeps one frame throughout: Copilot proposes → you read and understand → schema and policy tools validate → a test cluster confirms → a human approves → it ships.

The path a Kubernetes change travels is worth picturing before we start:

Requirement
     |
   Copilot
     |
    YAML
     |
schema validation  (kubeconform)
     |
 lint / policy     (helm lint, Kyverno)
     |
 test cluster
     |
engineer review    <-- required
     |
  production

Copilot assists at the first two steps — turning a requirement into YAML. It does not decide whether that YAML is safe to apply. That decision comes from the deterministic checks and the human review that follow. The two review-shaped steps — policy validation and an engineer reading the result — are what make Copilot’s speed safe to use on a cluster.

What You’ll Learn

• Why Kubernetes is a strong AI-assist use case — the manifests, kubectl, and troubleshooting workflows where Copilot saves the most time, and the caveat that frames all of it.
• Generating core workloads — Deployments, Services, ConfigMaps, Secrets, Ingress, Jobs, and CronJobs with current GA API versions and matching labels.
• Production hardening — resource requests and limits, readiness/liveness/startup probes, and a locked-down securityContext.
• Kubernetes RBAC with Copilot — least-privilege ServiceAccounts, Roles, and bindings, and how to catch AI’s tendency toward over-broad access.
• Troubleshooting — a repeatable diagnostic ladder plus focused walkthroughs of CrashLoopBackOff, ImagePullBackOff, Pending, OOMKilled, and Service connectivity.
• Validation tooling — kubectl --dry-run, kubeconform, helm lint, and policy engines as the source of truth.
• Copilot with Helm — extracting hardcoded values, generating values files, and rendering locally before deploy.
• CI with GitHub Actions — validating manifests on every PR with least-privilege permissions, plus a brief look at GitOps.
• 30 reusable prompts and a hands-on lab that builds, breaks, and fixes an AI API on a real cluster.

Kubernetes Concepts Copilot Can Help With

This lesson assumes you already work with Kubernetes; here is a fast refresher on the objects Copilot generates most, so the examples land. If you want to build the fundamentals or go deeper, the Kubernetes and Helm guides cover these in depth — treat this lesson as a complement, not a beginner’s introduction.

• Pod — the smallest deployable unit, one or more containers sharing a network and storage namespace. You rarely create Pods directly.
• Deployment — manages a replicated, self-healing set of Pods and handles rolling updates. The workhorse for stateless services.
• StatefulSet — like a Deployment but for workloads needing stable identities and storage (databases, brokers).
• DaemonSet — runs one Pod per node (log shippers, node agents).
• Service — a stable virtual endpoint that load-balances to a set of Pods selected by labels.
• ConfigMap — non-secret configuration injected as env vars or mounted files.
• Secret — configuration for sensitive values; base64-encoded, not encrypted.
• Ingress — HTTP/S routing rules that an ingress controller turns into external access.
• Job / CronJob — run-to-completion and scheduled batch work.
• Namespace — a scope for names and a boundary for quotas and policy.
• ServiceAccount / Role / RoleBinding — the identity a Pod runs as and the permissions granted to it.
• PersistentVolumeClaim — a request for durable storage that outlives a Pod.

Copilot can generate every one of these. Whether what it generates is correct for your cluster is a separate question this lesson keeps returning to.

Create a Kubernetes Demo Repository

Ground the examples in a small project. This continues directly from Part 6, GitHub Copilot with Docker — the image you built and pushed to GHCR there is the workload you deploy here:

copilot-kubernetes-demo/
  app/
    main.py            FastAPI app, /health route
  kubernetes/
    namespace.yaml
    deployment.yaml
    service.yaml
    configmap.yaml
    ingress.yaml
    rbac.yaml
    job.yaml
  helm/
    ai-api/
      Chart.yaml
      values.yaml
      templates/
  .github/
    workflows/
      k8s-validate.yml
  README.md

The application is the same minimal FastAPI service exposing a /health route. The container journey from Part 6 hands directly to the cluster:

App Code
   |
  Image
   |
Registry  (ghcr.io/example/ai-api)
   |
Deployment
   |
 Service
   |
 Ingress

Docker produced the image and the registry entry. Kubernetes owns everything from the Deployment down — and that is where Copilot goes to work next.

Generate a Kubernetes Deployment with Copilot

Open kubernetes/deployment.yaml and describe the goal in a comment, or ask in Copilot Chat:

“Write a Kubernetes Deployment for a service called ai-api, 3 replicas, container image ghcr.io/example/ai-api:1.0.0, listening on port 8000.”

A first draft typically looks like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ai-api
  namespace: ai-app
  labels:
    app: ai-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: ai-api
  template:
    metadata:
      labels:
        app: ai-api
    spec:
      containers:
        - name: ai-api
          image: ghcr.io/example/ai-api:1.0.0
          ports:
            - containerPort: 8000

This is valid apps/v1, the selector matches the Pod template labels, and it will apply cleanly. It is also a good example of why “it applies” is not the finish line. Ask Copilot the follow-up that turns a draft into a review: “What production features is this Deployment missing?” A careful reading finds several:

• No resource requests or limits — the scheduler cannot place it well, and one Pod can starve its neighbors.
• No probes — Kubernetes will route traffic to a Pod that is still starting and will not restart one that has deadlocked.
• No securityContext — the container runs as root with more privileges than it needs.
• No explicit rolling-update strategy — you are trusting the defaults for how updates roll out.
• Thin labels — beyond app, there is nothing to select on for versioning or ownership.

None of these stop the apply. All of them matter in production. The next several sections add them back one at a time — reading each addition rather than trusting it.

Resource Requests and Limits

Every container should declare what it needs and what it may use. Requests inform scheduling — Kubernetes places a Pod on a node with enough free capacity. Limits cap consumption — a container exceeding its memory limit is OOMKilled, and one exceeding its CPU limit is throttled, not killed.

          resources:
            requests:
              cpu: "250m"
              memory: "256Mi"
            limits:
              cpu: "1"
              memory: "1Gi"

Ask Copilot to add resources and it will produce numbers like these. Read them as illustrative placeholders, not measurements: 250m CPU is a quarter of a core, 256Mi is the request, 1Gi the ceiling. The right values come from observed usage — run the workload under realistic load, watch kubectl top pods, and set requests near the steady state with limits above the peaks you are willing to tolerate.

⚠️ Warning — Do not ship Copilot’s default resource numbers as if they were correct. A memory limit set too low OOMKills the app under normal load; a CPU limit set too low throttles it into latency. Copilot cannot know your workload’s real footprint. Measure with kubectl top (or your metrics stack), then set values from evidence.

The Prometheus monitoring guides go deeper on measuring the usage these values should be based on.

Readiness, Liveness, and Startup Probes

Probes are where AI-generated manifests most often look right and behave wrong, because the fields are simple but the semantics are not. The three probes answer different questions:

• readinessProbe — “should this Pod receive traffic right now?” A failing readiness probe removes the Pod from the Service endpoints without restarting it. Use it for temporary unreadiness — warming a cache, waiting on a dependency.
• livenessProbe — “is this process broken and in need of a restart?” A failing liveness probe kills and restarts the container. Use it for deadlocks and unrecoverable states.
• startupProbe — “has this slow-starting app finished booting?” It holds off the liveness and readiness checks until the app is up, so a slow boot is not mistaken for a failure.

A readiness probe against the real health route:

          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 15
            periodSeconds: 20
          startupProbe:
            httpGet:
              path: /health
              port: 8000
            failureThreshold: 30
            periodSeconds: 5

Copilot generates probes readily, and makes a predictable set of mistakes:

• Wrong path or port — pointing at / or 8080 when the app serves health on /health:8000. A probe against the wrong endpoint fails or, worse, passes for the wrong reason.
• Over-aggressive timings — a short initialDelaySeconds with a low failureThreshold restarts a healthy app that simply boots slowly.
• Reusing one endpoint for all three probes without thinking — the same /health can serve all three here, but only if that route reflects both liveness and readiness. If readiness should also check the database and liveness should not, they need different endpoints.

Container started is not the same as ready. The lifecycle the probes govern:

Container Starts
      |
 Startup Probe    (holds liveness/readiness)
      |
 Initialized
      |
 Readiness Probe --> Traffic
      |
 Liveness Probe  --> detects deadlock,
                     restarts container

❗ Important — Ask Copilot to explain what each probe actually tests before you trust it, and confirm the httpGet path and port match the app. A liveness probe pointed at a slow or dependency-coupled endpoint will restart-loop a working app; a readiness probe that always passes routes traffic to a Pod that cannot serve. The probe fields are easy; the semantics are the review.

Kubernetes SecurityContext

By default a container can run as root with more Linux capabilities than it needs. A securityContext hardens it. Ask Copilot to “add a locked-down securityContext to this container,” and read the result:

          securityContext:
            runAsNonRoot: true
            runAsUser: 10001
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
            seccompProfile:
              type: RuntimeDefault

What each field buys you:

• runAsNonRoot / runAsUser — refuse to run as root; run as a fixed unprivileged UID. The single highest-value hardening step.
• allowPrivilegeEscalation: false — a process cannot gain more privileges than it started with (no setuid escalation).
• readOnlyRootFilesystem: true — the container filesystem is immutable; an attacker cannot write a payload to it. Apps that need scratch space get an explicit emptyDir volume.
• capabilities: drop: ["ALL"] — remove every Linux capability, then add back only any the app genuinely needs (usually none for a web service).
• seccompProfile: RuntimeDefault — apply the runtime’s default syscall filter.

The right way to arrive at this is not to paste the strictest possible context and hope:

Secure Default
      |
  Test App
      |
Adjust to Minimum Necessary

A too-strict context breaks working apps in ways that look like unrelated bugs — readOnlyRootFilesystem turns a temp-file write into a crash; dropping a capability an app quietly relied on causes a permission error at startup. Start locked down, run the app on a test cluster, and loosen only the specific field the failure points to.

At the namespace level, Pod Security Admission enforces baseline or restricted standards by labeling the namespace (for example pod-security.kubernetes.io/enforce: restricted). It complements the per-container securityContext by making the cluster reject Pods that do not meet the bar. Treat it as the cluster-side backstop for what the manifest declares.

✅ Best Practice — Have Copilot generate the hardened securityContext, then verify it on a test cluster before rolling it anywhere real. Secure-by-default plus adjust-to-fit beats permissive-by-default plus hope. The manifest declaring runAsNonRoot is only proof of intent until a Pod actually starts under it.

Kubernetes Services

A Deployment runs Pods; a Service gives them a stable address and load-balances across them. There are three types you reach for:

• ClusterIP (default) — an internal-only virtual IP. Most services should be ClusterIP; internal callers reach them by name.
• NodePort — opens a port on every node. Useful for development and specific integrations, rarely the right production front door.
• LoadBalancer — provisions an external load balancer (on a supporting cloud). For the handful of services that genuinely face the internet.

Not everything should be externally exposed. The API might be reachable from the internet; its database and cache should not be. The typical shape:

Internet
   |
Ingress / LoadBalancer
   |
 Service   (ClusterIP)
   |
  Pods

A ClusterIP Service for the API:

apiVersion: v1
kind: Service
metadata:
  name: ai-api
  namespace: ai-app
spec:
  selector:
    app: ai-api
  ports:
    - port: 80
      targetPort: 8000
  type: ClusterIP

The load-bearing line is selector: app: ai-api. A Service finds its Pods by label match — the selector here must equal the labels on the Deployment’s Pod template. Get this wrong and you hit the single most common Service bug:

🔍 Troubleshooting — “Service has no endpoints.” If a Service routes to nothing, the selector does not match any Pod’s labels. Run kubectl get endpoints ai-api (or kubectl get endpointslices); an empty result means selector-mismatch. Compare kubectl get svc ai-api -o yaml (the selector) against kubectl get pods --show-labels (the actual Pod labels). Copilot often generates a Service and a Deployment with subtly different labels — app: ai-api on one, app: ai_api or app.kubernetes.io/name: ai-api on the other. The label strings must be identical.

ConfigMaps

Configuration does not belong baked into the image — the same image should run in dev, staging, and production with different settings. A ConfigMap holds non-secret configuration and injects it into Pods:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ai-api-config
  namespace: ai-app
data:
  LOG_LEVEL: "info"
  MAX_WORKERS: "4"
  FEATURE_FLAG_BETA: "false"

Consume the whole ConfigMap as environment variables with envFrom:

          envFrom:
            - configMapRef:
                name: ai-api-config

Two decisions Copilot will not make for you:

• ConfigMap vs Secret — ConfigMaps are for non-sensitive values (log levels, feature flags, URLs). Anything sensitive — passwords, tokens, keys — belongs in a Secret (next section), never a ConfigMap.
• Env vars vs mounted files — envFrom is convenient for flat key/value config. For structured config (a full config.yaml, TLS certs), mount the ConfigMap as a volume so the app reads it as a file.

There is a reload caveat worth knowing: values injected as environment variables are read once at container start — changing the ConfigMap does not update a running Pod’s env; you must restart the Pods (kubectl rollout restart deployment/ai-api). Values mounted as files are updated in place by the kubelet, but the app still has to re-read the file to notice. Ask Copilot to explain which mechanism a given manifest uses before you assume a config change took effect.

Kubernetes Secrets

Secrets carry sensitive configuration, and they carry a dangerous misconception with them.

⚠️ Warning — Kubernetes Secrets are base64-encoded, which is NOT encryption. echo aGVsbG8= decodes in one command. Anyone who can read a Secret manifest, or read the object from the API, can read the value. Never commit a plaintext production credential to Git just because it will become a Kubernetes Secret. In every committed manifest, use placeholders only.

A Secret manifest, with a placeholder value:

apiVersion: v1
kind: Secret
metadata:
  name: ai-api-secrets
  namespace: ai-app
type: Opaque
stringData:
  DATABASE_PASSWORD: "REPLACE_ME_FROM_SECRETS_MANAGER"

The stringData field takes plaintext that Kubernetes encodes for you — handy for examples, and exactly why the value here must be a placeholder, not a real password. For real credentials, the value should never live in Git at all. Source it from one of:

• An external secrets operator — syncs values from AWS Secrets Manager, Vault, GCP Secret Manager, or Azure Key Vault into Kubernetes Secrets at runtime.
• A cloud secret manager referenced via workload identity — the Pod’s ServiceAccount is bound to a cloud identity that reads the secret directly, so nothing sensitive is stored in the cluster.
• GitHub Actions secrets — for values injected during a deploy pipeline, referenced as ${{ secrets.NAME }} and never printed.

Consume the Secret the same way as a ConfigMap, but keep it separate so access can be controlled independently:

          envFrom:
            - secretRef:
                name: ai-api-secrets

Copilot will happily generate a Secret with a literal password if your prompt includes one. Replace it with a placeholder before you commit, and let a real secrets system supply the value at deploy time.

Ingress

An Ingress defines HTTP/S routing rules that an ingress controller (NGINX, Traefik, a cloud controller) turns into actual external access. The current API is networking.k8s.io/v1, which requires ingressClassName and a pathType per path:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ai-api
  namespace: ai-app
spec:
  ingressClassName: nginx
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: ai-api
                port:
                  number: 80

The parts Copilot must get right on the current API:

• ingressClassName — names which controller handles this Ingress. Do not assume every cluster runs nginx; set it to the class your cluster actually has.
• pathType — Prefix (match by path prefix) or Exact; it is required, and older examples in Copilot’s training data omit it.
• backend.service.name + backend.service.port.number — the modern nested form. The old serviceName/servicePort flat fields are from the removed extensions/v1beta1 API; reject them if Copilot emits them.

Host rules route by domain; paths route by URL within a host; TLS is configured with a tls: block referencing a Secret holding the certificate. The concepts are stable, but the controller is not universal — an annotation that works for the NGINX controller means nothing to Traefik. Ask Copilot which controller a given annotation targets, and match it to your cluster.

Jobs and CronJobs

Not every workload is a long-running server. A Job runs a Pod to completion; a CronJob runs one on a schedule. Both use batch/v1.

A one-shot Job for a database maintenance task:

apiVersion: batch/v1
kind: Job
metadata:
  name: db-maintenance
  namespace: ai-app
spec:
  backoffLimit: 3
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: maintenance
          image: ghcr.io/example/ai-api:1.0.0
          command: ["python", "-m", "app.maintenance"]

A CronJob for a periodic report:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-report
  namespace: ai-app
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 300
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: report
              image: ghcr.io/example/ai-api:1.0.0
              command: ["python", "-m", "app.report"]

Common uses are database maintenance, periodic reports, and cache cleanup. The risks Copilot will not warn you about:

• Concurrency — concurrencyPolicy: Forbid stops a slow run from overlapping the next scheduled one; the default Allow can stampede a database. Choose deliberately.
• Retries and failed Jobs — backoffLimit caps retries; failed Jobs and their Pods accumulate unless you set history limits, quietly consuming resources.
• Resources — batch Pods need requests/limits too, or a heavy report can starve the cluster.
• Time zones — the schedule is interpreted in the cluster’s time zone (UTC on many clusters). 0 2 * * * is 02:00 UTC, not necessarily your local 2 a.m. Verify the effective time zone before relying on it.

Kubernetes RBAC with Copilot

RBAC controls what an identity may do. A Pod runs as a ServiceAccount; a Role (namespaced) or ClusterRole (cluster-wide) lists allowed verbs on resources; a RoleBinding or ClusterRoleBinding ties them together. All use rbac.authorization.k8s.io/v1.

Ask Copilot for exactly the access needed: “Create a ServiceAccount and Role that can get, list, and watch Pods only in the ai-app namespace, and bind them.”

apiVersion: v1
kind: ServiceAccount
metadata:
  name: ai-api-reader
  namespace: ai-app
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: ai-app
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ai-api-reader-binding
  namespace: ai-app
subjects:
  - kind: ServiceAccount
    name: ai-api-reader
    namespace: ai-app
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

This grants read-only access to Pods in one namespace and nothing else — the least-privilege ideal. The trouble is that least privilege is not Copilot’s default instinct. Generated RBAC skews broad because broad rules “just work”: a ClusterRole where a namespaced Role would do, verbs: ["*"] instead of the three verbs actually used, resources: ["*"] to avoid enumerating.

⚠️ Warning — AI-generated RBAC must be reviewed for excess. Never accept cluster-admin, verbs: ["*"], or resources: ["*"] from a generated manifest without asking whether the workload truly needs them — it almost never does. Prefer a namespaced Role over a ClusterRole, list explicit verbs, and delete any rule you cannot justify. Ask Copilot to explain why each rule exists, then remove the ones the app does not use.

The security hardening guides cover RBAC and least-privilege patterns in more depth.

Copilot for kubectl

Copilot is excellent at recalling kubectl syntax and field selectors you would otherwise look up. Describe the goal in plain language and read the command it returns. A set that earns its place:

# Pods that are not Running, across all namespaces
kubectl get pods -A \
  --field-selector=status.phase!=Running

# Pods sorted by restart count (most restarts last)
kubectl get pods -A \
  --sort-by='.status.containerStatuses[0].restartCount'

# Recent events, newest last
kubectl get events -A \
  --sort-by=.metadata.creationTimestamp

# Node resource pressure and allocation
kubectl top nodes
kubectl describe nodes | grep -A5 "Conditions"

# Which Deployment owns a Pod (walk the ownerReferences)
kubectl get pod <pod> -o \
  jsonpath='{.metadata.ownerReferences[0].name}'

# Services that have no endpoints (selector mismatch)
kubectl get endpoints -A

# Pods using a specific ServiceAccount
kubectl get pods -A -o \
  jsonpath='{range .items[?(@.spec.serviceAccountName=="ai-api-reader")]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}'

Read-only commands like these are safe to run and verify. The habit to keep is reading before running anything that mutates — a generated kubectl delete, scale, or patch is a proposal, not a conclusion. Copilot recalls the syntax; you decide whether to execute it.

🛠️ DevOps Tip — When a kubectl one-liner Copilot gives you uses a jsonpath or --field-selector you do not fully understand, ask it to explain the selector before you rely on the output. A field selector that silently matches nothing looks identical to one that matches nothing because everything is healthy — and only one of those is good news.

Kubernetes Troubleshooting with Copilot

Troubleshooting is where Copilot is strongest and where the discipline matters most, because it is so easy to accept a confident hypothesis as a conclusion. Gather evidence from the cluster first, then let Copilot interpret it. The ladder:

Symptom
   |
kubectl get         (state)
   |
kubectl describe    (events, config)
   |
kubectl logs        (app output)
   |
kubectl logs --previous   (last crash)
   |
kubectl get events  (scheduling, pulls)
   |
inspect YAML
   |
Copilot HYPOTHESIS
   |
engineer verification    <-- decides

Copilot generates hypotheses; the cluster and the engineer decide. Feed it the actual output — “here is the describe and the logs, what is the likely cause?” — and verify its answer against the evidence rather than adopting it. The rest of this section walks the failures you will hit most.

CrashLoopBackOff

A container starts, exits, and Kubernetes restarts it with growing backoff. Causes range from a bad start command to a missing config value, a failing migration, or an over-eager liveness probe. Gather the evidence:

kubectl get pod <pod> -n ai-app
kubectl describe pod <pod> -n ai-app
kubectl logs <pod> -n ai-app
kubectl logs <pod> -n ai-app --previous

get shows the status and restart count; describe shows recent events and the last exit reason; logs shows the current attempt; logs --previous shows the output of the crashed container before its restart — often the only place the real stack trace lives. Give Copilot the previous logs and describe output; it will usually name a plausible cause (a missing env var, an unreachable dependency, a probe killing a healthy-but-slow app). Verify against the evidence before changing anything.

ImagePullBackOff

The Pod cannot pull its image, so it never starts. kubectl describe pod shows the pull error in the events. Walk the usual causes:

• Wrong image name or tag — a typo, or a tag that does not exist in the registry.
• Registry authentication — a private registry (like GHCR) needs an imagePullSecret; without it the pull is denied.
• Network or registry reachability — the nodes cannot reach the registry.
• Rate limits — anonymous pulls from a public registry can be throttled.

Ask Copilot to explain the exact message from describe and it will map it to one of these; confirm by checking whether the image and tag exist and whether the Pod’s imagePullSecrets are set.

Pending

A Pending Pod has been accepted but not scheduled onto a node. The reason is always in the scheduler’s events:

kubectl describe pod <pod> -n ai-app

Common causes: not enough free CPU or memory on any node, an unsatisfiable nodeSelector or affinity rule, a taint the Pod does not tolerate, or an unbound PersistentVolumeClaim. Paste the events and ask: “Explain why this Pod is Pending from these scheduler events.” Copilot reads Insufficient cpu or node(s) had untolerated taint and translates it — but confirm against kubectl top nodes and the Pod’s own affinity/tolerations before acting.

OOMKilled

A container that exceeds its memory limit is killed with reason OOMKilled (often surfacing as exit code 137). The reflex to raise the limit is usually wrong:

Inspect Limit
     |
Inspect Actual Usage
     |
Understand the App
     |
    Tune

Check the limit (kubectl get pod <pod> -o yaml), check real usage (kubectl top pod <pod>), and understand why the app used that memory. A limit set too low needs raising; a genuine memory leak needs fixing, not a bigger ceiling that just delays the kill. Ask Copilot to help interpret the usage pattern, not to reflexively bump the number.

🔍 Troubleshooting — When Copilot suggests “increase the memory limit” for an OOMKilled Pod, treat it as one hypothesis, not the fix. If usage climbs steadily until the kill, that is a leak a higher limit only postpones. Compare kubectl top pod over time against the limit before you change anything.

Service Connectivity

“The API can’t reach PostgreSQL” is a chain of specific checks, each ruling out a layer. Walk it in order rather than guessing:

Pod running?          kubectl get pod
   |
Service exists?       kubectl get svc
   |
Endpoints populated?  kubectl get endpoints
   |
Labels match selector?  compare labels
   |
DNS resolves?         nslookup from a Pod
   |
TCP reachable?        connect to host:port
   |
NetworkPolicy allows? kubectl get netpol
   |
App config correct?   env / connection string

Most “can’t connect” failures are one of the first four: the Pod is not running, the Service has no endpoints because of a selector mismatch, or the labels simply do not line up. If DNS and TCP work but traffic is still blocked, a NetworkPolicy may be denying it. Give Copilot the output from each rung — it is good at spotting which layer breaks the chain — and verify the fix at that specific layer.

Kubernetes YAML Review with Copilot

Beyond generating manifests, Copilot is a useful reviewer. Give it a role and ask for risks:

“Review this manifest as a senior platform engineer. Look at reliability, security, scalability, and operational risks.”

Copilot will typically flag:

• Missing resource requests and limits — no scheduling guidance, no protection against noisy neighbors.
• Missing probes — traffic to unready Pods, no restart for deadlocked ones.
• Root containers — no securityContext, runAsNonRoot absent.
• Floating image tags — latest or an unpinned tag that changes under you.
• No rolling-update strategy — defaults trusted implicitly, replicas: 1 with no disruption budget.
• Over-exposed Services — a LoadBalancer or NodePort where ClusterIP would do.
• Broad RBAC — wildcard verbs or resources, a ClusterRole where a Role fits.
• Missing PodDisruptionBudget — nothing protecting availability during voluntary disruptions.
• Insecure capabilities — capabilities not dropped, allowPrivilegeEscalation left on.
• Missing labels — nothing to select on for versioning or ownership.

This review is valuable and it is not authoritative. Copilot reasons about the text of your manifest; it does not enforce anything. That is what validation and policy tooling do — Copilot’s review complements them, it does not replace them. Use it to catch obvious gaps early, then let the deterministic tools in the next section decide what is actually allowed.

Kubernetes Validation Tooling

A manifest that parses is not a manifest that is correct. The source of truth is a chain of deterministic checks, run locally and in CI:

# Client-side: does it parse and pass basic checks?
kubectl apply --dry-run=client -f kubernetes/

# Server-side: would the API server accept it?
kubectl apply --dry-run=server -f kubernetes/

# Schema validation against the Kubernetes API schema
kubeconform -strict -summary kubernetes/

# Helm chart structure and rendering
helm lint helm/ai-api

What each contributes:

• kubectl apply --dry-run=client parses and validates locally without contacting the cluster’s admission chain — a fast first gate.
• kubectl apply --dry-run=server sends the manifest through the real API server (including admission webhooks) without persisting it — it catches what client-side cannot.
• kubeconform validates every resource against the Kubernetes JSON schema for your target version, catching wrong field names and types that --dry-run=client may miss. It is fast enough to run on every commit.
• Policy engines — Kyverno and OPA-Gatekeeper enforce organizational rules (no root containers, resource limits required, approved registries only) as admission policy. They are the cluster-side backstop that makes a standard mandatory rather than advisory.

The full pipeline for anything Copilot generates:

Copilot YAML
     |
   parse
     |
  schema        (kubeconform)
     |
  policy        (Kyverno / OPA)
     |
 test cluster
     |
   review

GitHub Copilot with Helm

Raw manifests do not scale across environments — dev, staging, and production duplicate 90% of the same YAML with a few values changed. Helm packages manifests into a chart with parameterized values. A chart is a directory:

helm/ai-api/
  Chart.yaml       chart metadata and version
  values.yaml      default configuration values
  templates/       templated manifests
    deployment.yaml
    service.yaml
    ingress.yaml

Copilot is genuinely useful here. Its best Helm trick is refactoring: give it a working manifest and ask it to “extract the hardcoded values — image, tag, replicas, port, resources — into values.yaml and template the Deployment.” It pulls the literals into values.yaml:

# values.yaml
replicaCount: 3
image:
  repository: ghcr.io/example/ai-api
  tag: "1.0.0"
service:
  port: 80
  targetPort: 8000
resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  limits:
    cpu: "1"
    memory: "1Gi"

and rewrites the template to reference them ({{ .Values.replicaCount }}, {{ .Values.image.repository }}:{{ .Values.image.tag }}). It can also generate environment-specific values files (values-staging.yaml, values-prod.yaml) and explain an unfamiliar template block when you inherit a chart.

Helm Troubleshooting

Templating adds a class of errors raw YAML does not have. The frequent ones:

• Template render error — a malformed {{ }} action, or a pipeline that references a function that does not exist.
• Missing value — a template references .Values.something that no values file defines, rendering an empty string or failing.
• Indentation — a templated block inserted at the wrong indent produces invalid YAML even though the template looks fine.
• Wrong type — a value expected as a number supplied as a string (or vice versa), which the API server rejects on apply.

Two commands catch nearly all of it before a cluster ever sees the chart:

# Structural and best-practice checks
helm lint helm/ai-api

# Render the templates to plain YAML locally
helm template helm/ai-api -f helm/ai-api/values-prod.yaml

helm lint catches structural problems; helm template renders the chart to the exact manifests Helm would apply, so you read the real output before deploying. Render locally, read the YAML, then deploy — never deploy a chart you have not rendered.

Copilot with Kubernetes and GitHub Actions

Manifests should be validated the same way on every change, which is what CI is for. Ask Copilot to “write a GitHub Actions workflow that validates Kubernetes manifests with kubeconform and a client-side dry run on every pull request, with least-privilege permissions.” The verified shape:

name: k8s-validate

on:
  push:
    branches: [main]
  pull_request:

permissions:
  contents: read

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - name: Check out code
        uses: actions/checkout@v4

      - name: Install kubeconform
        run: |
          curl -sSL -o kubeconform.tar.gz \
            https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz
          tar xzf kubeconform.tar.gz kubeconform
          sudo mv kubeconform /usr/local/bin/

      - name: Validate manifests (schema)
        run: kubeconform -strict -summary kubernetes/

      - name: Helm lint
        run: helm lint helm/ai-api

      - name: Render Helm chart
        run: helm template helm/ai-api > /dev/null

The safety-relevant points in any Copilot-generated Kubernetes workflow:

• Pinned, real actions — actions/checkout@v4 is a current major version; confirm any uses: against its repository and reject invented ones.
• Least-privilege permissions: — this workflow only reads and validates, so contents: read is all it needs. Copilot often omits the block; add it.
• Validation, not deployment — the pipeline validates and renders; it does not apply anything to a cluster.

The full PR pipeline to build toward: PR → YAML validate → helm lint → render → security/policy → human review. The one rule that overrides convenience:

❗ Important — Do NOT auto-deploy production manifests from untrusted pull request contexts. A fork PR that can run kubectl apply with cluster credentials, or that can read a kubeconfig secret, is an attack surface. Validate and render on PRs; deploy only from trusted refs (like main) after human approval, using environment protection rules and required reviewers.

GitOps Concepts

Validation in CI answers “is this manifest valid?” GitOps answers “is the cluster what Git says it should be?” The model, briefly:

Git  (desired state)
   |
GitOps Controller   (Argo CD / Flux)
   |
 Cluster  (reconciled to match)

A GitOps controller — Argo CD or Flux, named here only — continuously reconciles the cluster to the manifests in Git, so Git becomes the single source of truth and deploys become merges. This lesson does not build a GitOps setup; the point is that the validated, reviewed manifests you produce here are exactly what a GitOps controller consumes, which prepares the pipeline and agent lessons later in the academy.

30 GitHub Copilot Prompts for Kubernetes Engineers

Reusable starting prompts. Each produces a draft to read, validate, test, and review before you apply it.

Workloads

1. “Write a Deployment for a service named ai-api, 3 replicas, image ghcr.io/example/ai-api:1.0.0, port 8000.”
2. “What production features is this Deployment missing?”
3. “Add resource requests and limits to this container and explain how to size them.”
4. “Add readiness, liveness, and startup probes hitting /health on port 8000.”
5. “Add a rolling-update strategy with maxUnavailable and maxSurge and explain the tradeoff.”
6. “Convert this Deployment to a StatefulSet and explain what changes and why.”
7. “Add a locked-down securityContext: non-root, drop all capabilities, read-only root filesystem.”

Networking and config

8. “Write a ClusterIP Service for this Deployment and make the selector match its labels.”
9. “Explain the difference between ClusterIP, NodePort, and LoadBalancer for this service.”
10. “Write a networking.k8s.io/v1 Ingress with ingressClassName and pathType for api.example.com.”
11. “Create a ConfigMap for LOG_LEVEL and MAX_WORKERS and inject it with envFrom.”
12. “Write a Secret manifest with placeholder values and explain why base64 is not encryption.”
13. “Show how to source these Secret values from an external secrets manager instead of Git.”

RBAC and security

14. “Create a ServiceAccount and Role that can get, list, watch Pods only in the ai-app namespace, and bind them.”
15. “Review this RBAC for least privilege and remove any excessive verbs or resources.”
16. “Explain what cluster-admin grants and why a namespaced Role is safer here.”
17. “Review this manifest as a senior platform engineer for reliability and security risks.”
18. “Add Pod Security Admission labels to enforce the restricted standard on this namespace.”

Batch

19. “Write a batch/v1 Job for a database maintenance task with a backoffLimit.”
20. “Write a CronJob that runs a nightly report at 02:00 with concurrencyPolicy Forbid.”
21. “Explain the time-zone and concurrency risks in this CronJob schedule.”

kubectl and troubleshooting

22. “Give me a kubectl command to list all non-running Pods across every namespace.”
23. “Give me a kubectl command to find the Pods with the most restarts.”
24. “Explain why this Pod is Pending from these scheduler events.”
25. “This Pod is in CrashLoopBackOff; here is the describe and previous logs — what is the likely cause?”
26. “Explain this ImagePullBackOff message and list what to check.”
27. “This Pod was OOMKilled; help me decide whether to raise the limit or fix the app.”
28. “The API can’t reach PostgreSQL; walk me through Service, endpoints, labels, DNS, and NetworkPolicy.”

Helm and CI

29. “Extract the hardcoded values from this Deployment into values.yaml and template it.”
30. “Write a GitHub Actions workflow that validates these manifests with kubeconform on every PR, least-privilege permissions.”

Lab: Build and Troubleshoot an AI API on Kubernetes with GitHub Copilot

Put the whole lesson together by deploying the AI API to a real (test) cluster — a local kind/minikube cluster is ideal. The point is the cycle, not the specific files. Work each step through the same loop: Copilot proposes → you read → validation and the cluster confirm → you approve.

1. Namespace — ask Copilot for a namespace.yaml for ai-app; apply it and confirm with kubectl get ns.
2. Deployment — ask for a Deployment (ai-api, 3 replicas, ghcr.io/example/ai-api:1.0.0, port 8000); read the naive draft and note what is missing.
3. Service — ask for a ClusterIP Service; confirm the selector matches the Pod labels exactly.
4. ConfigMap — add a ConfigMap for LOG_LEVEL and MAX_WORKERS; wire it with envFrom.
5. Resources — add requests and limits; note that the numbers are placeholders to replace with measured usage.
6. Probes — add readiness, liveness, and startup probes hitting /health:8000; have Copilot explain what each tests.
7. securityContext — add runAsNonRoot, allowPrivilegeEscalation: false, dropped capabilities, and a seccomp profile.
8. Deploy — apply the manifests to the test cluster and watch kubectl get pods -n ai-app -w.
9. Inspect rollout — use kubectl rollout status deployment/ai-api and kubectl get endpoints ai-api to confirm the Service has endpoints.
10. Simulate a failure — deliberately break something (point a probe at the wrong port, or set a memory limit far too low) and re-apply.
11. Troubleshoot with kubectl — use get, describe, logs, and logs --previous to gather evidence of the failure.
12. Copilot explains — give Copilot the describe and logs output and ask for a hypothesis; verify it against the evidence.
13. Correct the YAML — fix the one thing that is wrong (right port, sane limit), and re-apply.
14. Validate — run kubectl apply --dry-run=client -f kubernetes/ and kubeconform -strict kubernetes/.
15. Actions validation — add .github/workflows/k8s-validate.yml that runs kubeconform and helm lint on every PR with permissions: { contents: read }; confirm the action versions.
16. Document — ask Copilot for a README covering deploy commands, the manifests, and the troubleshooting steps; verify every command against the actual files.

The end-to-end shape you have practiced:

GitHub
   |
Copilot YAML
   |
validation   (kubeconform / dry-run)
   |
test cluster
   |
troubleshooting
   |
   PR
   |
human approval

Commit each piece only after you have read and validated it. By the end you will have used Copilot to generate, harden, deploy, break, and fix a Kubernetes workload — while keeping schema validation, policy, and a real cluster as the source of truth.

🛠️ DevOps Tip — Add a repo-level custom instructions file so Copilot defaults to your Kubernetes conventions — resource limits required, probes present, non-root securityContext, namespaced RBAC, least-privilege Actions permissions — across the whole project. Verify the current custom-instructions setup in the VS Code docs, since the mechanism evolves.

What’s Next

You now have Copilot working across the Kubernetes lifecycle: generating Deployments, Services, ConfigMaps, Secrets, Ingress, Jobs, and RBAC with current GA API versions; hardening with resources, probes, and a locked-down securityContext; troubleshooting the failures that actually happen; validating with kubeconform, helm lint, and policy engines; and wiring it into GitHub Actions — all under the same discipline that a manifest which parses is not automatically secure, correct, or production-ready.

The next lesson, Part 8: GitHub Copilot for Bash, covers the operational automation that surrounds both containers and clusters — strict-mode scripts, ShellCheck, and the hard rule to never blindly run AI-generated shell. It is worth holding the three together: Docker packages the workload; Kubernetes orchestrates it — and Bash (Part 8) glues the operational automation around both.

To revisit the image these manifests deploy, return to Part 6, GitHub Copilot with Docker; for the infrastructure that provisions the cluster itself, see Part 5, GitHub Copilot with Terraform. The GitHub AI Engineering Academy home has the full path, and the Kubernetes and Helm guides, the Docker guides, the security hardening guides, the Prometheus monitoring guides, the hands-on Docker Academy, and the Ubuntu AI Infrastructure series all go deeper on the tools here.