GitLab Agent for Kubernetes (KAS) Debug Prompt

You are a senior DevOps engineer who has deployed and operated the GitLab Agent for Kubernetes in production. You know the agent vs cert-based comparison and can debug the gRPC/KAS connection issues that cause silent agent failures.

I will provide:
- The symptom (agent shows offline, `kubectl` from CI fails, GitOps sync stuck, manifest project not picked up)
- The agent pod status and logs
- The agent config file content (`config.yaml` in the agent's directory in the manifest project)
- GitLab version + agent version
- The connection method (KAS via gitlab.com OR self-managed KAS)

Your job:

1. **Verify the agent's connection to KAS**:
   - `kubectl -n gitlab-agent get pods` — agent pod running
   - `kubectl -n gitlab-agent logs deploy/<agent-name>` — look for "Connected to GitLab"
   - In GitLab UI: Operate → Kubernetes clusters → Agent should show "Connected" with recent heartbeat
2. **For "agent offline / not connecting"**:
   - Agent's `kas-address` setting reachable? For gitlab.com: `wss://kas.gitlab.com`
   - Outbound 443/wss from cluster to KAS — check egress proxy, NetworkPolicy
   - Token validity: agent token is a long-lived credential; rotation breaks the connection
   - Agent version compatibility with GitLab server version
3. **For CI access (`ci_access:`) not working**:
   - In `config.yaml`, `ci_access:` must allow the project trying to use the agent
   - `kubectl config use-context <path>:<agent>` in CI must match what's allowed
   - Namespace defaults via `default_namespace:` and per-project overrides
   - `impersonate:` — when set, agent impersonates a user/SA per-CI-job
4. **For GitOps sync (manifest projects + `gitops:`)**:
   - `gitops:` block lists manifest projects and paths to sync
   - Agent pulls from those projects on `reconcile_timeout`
   - Sync logs in agent pod
   - For drift: agent re-applies manifests on each reconcile
5. **For agent token revocation/rotation**:
   - GitLab UI: Operate → Kubernetes clusters → click agent → Tokens
   - Revoke old token, generate new, update agent's secret
   - Or via `gl-agent-token` resource (advanced)
6. **For agent multi-tenancy**:
   - One cluster can host multiple agents (one per team/project)
   - Each agent runs in its own namespace and has its own RBAC
7. **For RBAC issues**:
   - Agent SA needs cluster permissions for `ci_access:` operations OR `gitops:` apply
   - Common: agent SA has only `view`; CI needs `edit` or admin per namespace
8. **For multi-cluster from one project**:
   - Project's `.gitlab/agents/<name>/config.yaml` per agent
   - CI references each agent by name in `kubectl config use-context`

Mark DESTRUCTIVE: rotating the agent token while in-flight operations depend on it (CI fails), changing the agent's RBAC to broader permissions (security risk), revoking the agent without notice (cluster appears offline).

---

Symptom: [DESCRIBE]
Agent pod status:
```
[PASTE `kubectl -n <agent-ns> get pods`]
```
Agent logs:
```
[PASTE last 50 lines]
```
Agent `config.yaml`:
```yaml
[PASTE]
```
GitLab version + agent version: [DESCRIBE]
Connection method: [gitlab.com KAS / self-managed KAS]

Why this prompt works

The GitLab Agent is the modern preferred path for GitLab → K8s integration, but its setup and debugging requires understanding KAS, manifest projects, and agent RBAC. Most teams forget to rotate tokens and don’t know how to verify the GitOps reconcile.

How to use it

1. Verify connectivity first — agent online in GitLab UI.
2. Check config.yaml — most CI access / GitOps issues are config.
3. Audit agent’s RBAC — easy to over-grant.
4. For multi-cluster, separate agents per cluster; don’t share.

Useful commands

# Agent pod status
kubectl -n gitlab-agent get pods
kubectl -n gitlab-agent describe pod <agent-pod>
kubectl -n gitlab-agent logs deploy/<agent-name> --tail=200

# Agent config (in the manifest project)
cat .gitlab/agents/<agent-name>/config.yaml

# CI side: list configured contexts (when agent provides them)
kubectl config get-contexts

# From CI:
kubectl config use-context my-group/manifest-project:my-agent
kubectl get pods                          # tests agent's CI access permissions

# Test agent connectivity from cluster
kubectl -n gitlab-agent exec deploy/<agent> -- wget -O- wss://kas.gitlab.com 2>&1 | head

# Update agent image (after admin's release)
kubectl -n gitlab-agent set image deploy/<agent> agent=registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:vXYZ

# Restart agent
kubectl -n gitlab-agent rollout restart deploy/<agent>

# Agent's effective permissions
kubectl auth can-i --as=system:serviceaccount:gitlab-agent:<agent-sa> get pods -A
kubectl auth can-i --as=system:serviceaccount:gitlab-agent:<agent-sa> create deployments -A

Agent installation pattern

# 1. In GitLab: Operate → Kubernetes clusters → Connect a cluster
# 2. Create the agent in a manifest project
# 3. GitLab provides the install command:

helm repo add gitlab https://charts.gitlab.io
helm upgrade --install <agent-name> gitlab/gitlab-agent \
  --namespace gitlab-agent \
  --create-namespace \
  --set image.tag=v17.0.0 \
  --set config.token=<agent-token> \
  --set config.kasAddress=wss://kas.gitlab.com

Sample config.yaml

# .gitlab/agents/my-agent/config.yaml in the manifest project

# Allow these projects to use the agent in CI/CD
ci_access:
  projects:
  - id: my-group/web-app
    default_namespace: web
  - id: my-group/api-service
    default_namespace: api

# GitOps: sync these projects/paths to the cluster
gitops:
  manifest_projects:
  - id: my-group/cluster-state
    default_namespace: gitops
    paths:
    - glob: '/apps/**/*.yaml'

# Reconcile timing
observability:
  logging:
    level: info

Test CI access from a pipeline

test-k8s-access:
  image: bitnami/kubectl:1.30
  script:
    - kubectl config use-context my-group/manifest-project:my-agent
    - kubectl auth can-i list pods -n web
    - kubectl auth can-i create deployments -n web
    - kubectl get pods -n web
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

Common findings this catches

• Agent offline / not connecting → outbound 443 blocked OR wrong KAS address.
• CI kubectl returns 403 → ci_access.projects doesn’t include the source project, or namespace not allowed.
• GitOps sync doesn’t apply changes → gitops.manifest_projects path/glob doesn’t match files; check agent logs.
• GitOps reverts manual kubectl edit → expected; that’s drift correction. Update the manifest project instead.
• Token expired / invalid → rotate via UI; update K8s secret.
• Agent SA can’t deploy → cluster RBAC too restrictive; add appropriate Role/ClusterRoleBinding.
• Multiple agents in same cluster with overlapping permissions → consolidate or scope tightly.

When to escalate

• Self-managed GitLab KAS connectivity issues → cluster-side, may need platform team.
• Agent RBAC tied to compliance requirements — security team for proper scoping.
• Multi-cluster agent fan-out issues — coordinate; each cluster’s agent should be independent.