GitLab CI/CD → Kubernetes Deploy Patterns Prompt

You are a senior DevOps engineer who has built GitLab → Kubernetes CD pipelines using kubectl, Helm, Kustomize, and pull-based GitOps (Argo/Flux). You know the trade-offs and the security boundaries.

I will provide:
- The application architecture (single service, monorepo with N services, microservices)
- Current deployment tooling (none, kubectl raw, Helm, Kustomize, GitOps)
- The target environments (dev, staging, prod) and promotion model
- Auth to cluster: kubeconfig stored as GitLab variable, GitLab Agent for Kubernetes, OIDC

Your job:

1. **Choose tooling**:
   - **`kubectl` raw**: simple, explicit; harder for multi-environment customization
   - **Helm**: powerful templating; chart per app; values per env; rollback built-in
   - **Kustomize**: declarative overlays; closer to YAML; no rollback abstraction
   - **GitOps (Argo/Flux)**: pull-based; CI just commits to a deploy repo; CD is owned by the GitOps tool
2. **Auth approach**:
   - **kubeconfig as GitLab variable**: simple, but the token in the variable is high-privilege; rotate
   - **GitLab Agent for Kubernetes (KAS)**: bidirectional connection; finer RBAC; preferred for new setups
   - **OIDC federation**: cluster trusts GitLab as OIDC issuer; short-lived tokens; modern best
3. **Promotion model**:
   - **Branch-per-env**: `main` → dev; PR to `staging` branch → staging; PR to `prod` branch → prod
   - **Tag-based**: tags promote to higher envs
   - **GitOps**: deploy repo's directories per env; CI moves image tags between dirs
4. **Pipeline shape** (Helm example):
   - Build → Lint chart → Push image → Deploy dev (auto) → Deploy staging (auto on main) → Deploy prod (manual, tag-gated)
5. **For secrets**:
   - Don't pass secrets via Helm values — they go into release secrets
   - Use External Secrets Operator + cloud secret manager + IRSA / Workload Identity
   - Or sealed-secrets (committed encrypted secrets)
6. **For drift detection**:
   - GitLab CI imperative deploy + manual `kubectl edit` = invisible drift
   - GitOps (Argo/Flux) detects and reverts drift
   - For imperative deploys, periodic `helm diff` check job
7. **For multi-cluster**:
   - Per-cluster GitLab Agent
   - Or per-cluster kubeconfig stored as scope-named variable
   - For multi-region, parallelize per cluster with `parallel: matrix:`
8. **For rollback**:
   - Helm: `helm rollback <release> <revision>`
   - Kustomize: revert git, redeploy
   - GitOps: revert git in deploy repo
   - kubectl raw: `kubectl rollout undo`

Mark DESTRUCTIVE: pushing image with `:latest` tag in production (silent updates), `kubectl apply -f` without diff preview (can wipe in-cluster state), GitOps repo with auto-prune turned on while in-cluster resources exist outside git.

---

Application + service count: [DESCRIBE]
Current tooling: [none / kubectl / Helm / Kustomize / GitOps]
Environments + promotion: [DESCRIBE]
Cluster auth method: [kubeconfig / GitLab Agent / OIDC]
Goal: [design new / migrate / improve]

Why this prompt works

GitLab → Kubernetes CD has many viable approaches, each with security and operational trade-offs. Picking one without considering rollback, drift, and secrets leads to operational debt. This prompt forces a structured choice.

How to use it

1. Pick the tooling per project size — kubectl OK for tiny; Helm for templated; GitOps for multi-team.
2. Use GitLab Agent for new clusters — better security model than kubeconfig variables.
3. Define promotion model upfront — branch-based or tag-based; don’t ad-hoc.
4. Decide drift policy — fully GitOps or scheduled drift check.

Useful commands

# Verify cluster access from CI
kubectl auth can-i list pods --namespace=myapp
kubectl auth can-i create deployments --namespace=myapp
kubectl version --short

# Helm
helm list -n <ns>
helm history <release> -n <ns>
helm get values <release> -n <ns> > current.yaml
helm rollback <release> <revision> -n <ns>

# Kustomize
kustomize build overlays/prod
kubectl apply -k overlays/prod
kubectl diff -k overlays/prod          # preview

# Drift check
helm diff upgrade <release> ./chart -f values.yaml --no-color    # plugin

# GitLab Agent verification
kubectl -n gitlab-agent get pods
kubectl -n gitlab-agent logs deploy/gitlab-agent --tail=200

Pattern 1: kubectl + kubeconfig variable (simple)

deploy-staging:
  stage: deploy
  image: bitnami/kubectl:1.30
  variables:
    KUBECONFIG: kubeconfig.yaml
  before_script:
    - echo "$KUBECONFIG_STAGING" > $KUBECONFIG
    - chmod 600 $KUBECONFIG
  script:
    - kubectl set image deploy/web -n myapp web="$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA"
    - kubectl rollout status deploy/web -n myapp --timeout=10m
  environment:
    name: staging
    url: https://staging.example.com
    deployment_tier: staging
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Pattern 2: Helm with multi-environment values

.deploy-helm: &deploy-helm
  image: alpine/helm:3.15.0
  script:
    - helm upgrade --install $RELEASE_NAME ./chart \
        --namespace $K8S_NAMESPACE \
        --create-namespace \
        -f chart/values.yaml \
        -f chart/values-$ENV.yaml \
        --set image.tag=$CI_COMMIT_SHORT_SHA \
        --wait --timeout 10m

deploy-dev:
  <<: *deploy-helm
  variables:
    RELEASE_NAME: web-dev
    K8S_NAMESPACE: dev
    ENV: dev
  environment: { name: dev, url: https://dev.example.com, deployment_tier: development }
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

deploy-staging:
  <<: *deploy-helm
  variables:
    RELEASE_NAME: web-staging
    K8S_NAMESPACE: staging
    ENV: staging
  environment: { name: staging, url: https://staging.example.com, deployment_tier: staging }
  rules:
    - if: $CI_COMMIT_TAG =~ /^staging-/

deploy-production:
  <<: *deploy-helm
  variables:
    RELEASE_NAME: web-prod
    K8S_NAMESPACE: production
    ENV: production
  environment: { name: production, url: https://example.com, deployment_tier: production }
  rules:
    - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
      when: manual

Pattern 3: Kustomize overlays

deploy:
  image: alpine/k8s:1.30.0
  script:
    - cd k8s/overlays/$ENV
    - kustomize edit set image web="$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA"
    - kubectl apply -k . --record
    - kubectl rollout status deploy/web -n $NAMESPACE --timeout=10m

k8s/
├── base/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
└── overlays/
    ├── dev/
    │   ├── patch.yaml
    │   └── kustomization.yaml
    ├── staging/
    └── prod/

Pattern 4: GitOps (CI commits to deploy repo, Argo/Flux applies)

update-deploy-repo:
  stage: deploy
  image: alpine/git
  before_script:
    - git config user.email "ci@example.com"
    - git config user.name "GitLab CI"
  script:
    - git clone https://$DEPLOY_REPO_TOKEN_USER:$DEPLOY_REPO_TOKEN@gitlab.example.com/deploy/cluster-prod.git
    - cd cluster-prod/apps/web
    - sed -i "s|image:.*|image: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA|" deployment.yaml
    - git add -A
    - git commit -m "deploy: web $CI_COMMIT_SHORT_SHA from $CI_PROJECT_PATH@$CI_COMMIT_SHA"
    - git push
  rules:
    - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
  environment:
    name: production
    deployment_tier: production

Argo / Flux watches cluster-prod and applies changes automatically.

Pattern 5: GitLab Agent for Kubernetes (kubectl via CI)

# In agent config (cluster-side): gitlab-agent-config
ci_access:
  projects:
  - id: my-group/my-project
    default_namespace: web-app

# In .gitlab-ci.yml — no kubeconfig needed!
deploy:
  image: bitnami/kubectl:1.30
  script:
    - kubectl config use-context my-group/cluster-prod:my-agent
    - kubectl set image deploy/web web="$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA"
    - kubectl rollout status deploy/web --timeout=10m

Comparison

Aspect	kubectl	Helm	Kustomize	GitOps
Templating	Manual	Built-in (Go)	Overlay patches	Whatever you commit
Rollback	rollout undo	rollback by rev	git revert	git revert
Multi-env	Many manifests	Per-env values	Overlays	Per-env folders
Drift detect	Manual / cron	Manual / helm-diff	Manual / kubectl diff	Automatic
Best for	Small / one-off	Moderate complexity	Stable specs	Multi-team / scale

Common findings this catches

• Production deploy without explicit approval → add when: manual AND protected env.
• Kubeconfig token granting cluster-admin → scope down to namespace-specific SA.
• helm install instead of helm upgrade --install → second deploy fails.
• Missing --wait → pipeline marks “deployed” before pods are ready.
• No image digest pinning → mutable tag means dev/prod could diverge.
• GitOps repo has prune: true but cluster has resources outside git → drift cleanup deletes them.
• Multiple environments using the same Helm release name → collisions; use namespace + release prefix.

When to escalate

• New cluster onboarding — coordinate Agent setup with cluster admin.
• Production change-management policy requires external approval — integrate with ServiceNow / external ticket.
• Multi-cluster fan-out at scale — consider Argo CD ApplicationSet or Flux KustomizationController over GitLab CI.