☸️ Deploying to Kubernetes From GitLab CI Without Losing Your Mind

Deploying to Kubernetes from CI sounds simple: run kubectl apply, done. It stays simple right up until the day a job dumps a half-applied manifest into production, or someone’s KUBECONFIG with cluster-admin leaks through a CI variable. I’ve cleaned up both.

After years of wiring GitLab to clusters, here’s the setup that’s actually safe and the mistakes I no longer make.

Don’t put a kubeconfig in CI variables

The naive approach stores a cluster-admin KUBECONFIG as a CI variable and runs kubectl against it. Please don’t. That credential is long-lived, broadly scoped, and exfiltratable — three properties you never want on a path to production.

The modern answer is the GitLab agent for Kubernetes (the agentk model). The agent runs inside your cluster and pulls instructions from GitLab over an outbound connection. The cluster’s API server is never exposed, and no kubeconfig leaves it. You authorize specific projects to use the agent in its config:

# .gitlab/agents/production/config.yaml
ci_access:
  projects:
    - id: my-group/my-app

In your pipeline, you select the agent’s context and you’re authenticated — no stored credentials:

deploy:
  image: bitnami/kubectl:latest
  script:
    - kubectl config use-context my-group/my-app:production
    - kubectl get pods

Use Helm or Kustomize, not raw kubectl apply

Raw kubectl apply has no concept of a release, no rollback, and no clean way to template per-environment values. Use Helm or Kustomize so a deploy is an atomic, reversible unit:

deploy-prod:
  stage: deploy
  script:
    - helm upgrade --install my-app ./chart
      --namespace production
      --set image.tag=$CI_COMMIT_SHA
      --atomic --timeout 5m
  environment:
    name: production
    url: https://app.example.com
  rules:
    - if: '$CI_COMMIT_TAG'
      when: manual

The --atomic flag is the hero here: if the rollout doesn’t go healthy within the timeout, Helm automatically rolls back. You never get stuck in a half-deployed state, which is exactly the failure mode that ruins evenings.

Pin images by digest or SHA, never by tag

Deploying image: my-app:latest means you don’t actually know what’s running. Pin to the commit SHA or, better, the image digest. $CI_COMMIT_SHA as the tag makes every deploy traceable back to an exact commit, and it makes rollbacks unambiguous.

Use GitLab environments for real

The environment keyword isn’t decoration. It gives you the deployment history, the “view app” link, rollback to a previous deploy from the UI, and — critically — the audit trail of who deployed what and when. Wire every deploy job to an environment:

environment:
  name: staging
  url: https://staging.example.com
  on_stop: stop-staging

Pair it with on_stop jobs so ephemeral environments clean themselves up instead of leaking namespaces.

Make production a deliberate choice

Production deploys should never be automatic surprises. Gate them with when: manual and protect the environment so only authorized users can trigger it. The manual button is a feature, not friction — it’s the moment a human confirms “yes, ship this.”

For higher-stakes services, add a real progressive rollout. Even a simple canary — deploy to one replica, check metrics, then proceed — beats a big-bang apply every time.

Verify the rollout, don’t assume it

A deploy job that exits 0 the instant kubectl apply returns is lying to you. The pods might still be crash-looping. Make the job wait for actual readiness:

- kubectl rollout status deployment/my-app -n production --timeout=180s

If the rollout doesn’t become ready, the job fails, you get paged by CI instead of by customers, and your --atomic Helm release rolls itself back. That’s the whole loop working as designed.

Consider pull-based GitOps for the last mile

CI pushing to a cluster works, but pull-based GitOps (Flux or Argo CD watching a Git repo) is even cleaner for the final step. CI builds the image and updates a manifest in Git; the in-cluster operator reconciles. The cluster credentials never leave the cluster, drift gets corrected automatically, and your Git history is your deploy history. GitLab’s agent supports this flow natively.

Where AI helps

Kubernetes deploy manifests are dense and the failure modes are quiet. I paste a Helm values file or a Deployment spec into a model and ask: “What happens during a rolling update if the new pods fail their readiness probe, and will this deploy cause downtime?” It catches missing probes, bad maxUnavailable settings, and resource gaps before they hit a cluster. I keep GitLab CI prompts for deploy reviews and run manifest changes through our Code Review tool before they merge.

The setup that lets me sleep

• Authenticate with the GitLab agent, never a stored kubeconfig.
• Deploy with Helm --atomic so failures roll back automatically.
• Pin images to $CI_COMMIT_SHA for traceability.
• Use environments for history, rollback, and audit.
• Gate prod with when: manual and protected environments.
• Wait for rollout status so a green job means a healthy app.

Do that and “deploy to Kubernetes from CI” stops being the scary part of your pipeline and becomes the boring, reliable part — which is exactly what a deploy should be.

AI suggestions for Kubernetes deployments are assistive, not authoritative. Always test rollout and rollback behavior in staging before trusting it in production.