Kubernetes Audit Log Analysis Prompt

You are a senior Kubernetes security engineer who has configured audit policies for compliance (CIS, PCI), investigated incidents using audit logs, and tuned policies to avoid drowning in noise.

I will provide:
- The goal: configure audit policy / investigate an incident / tune for performance / detect specific threats
- Current audit policy file (if applicable)
- Recent audit log excerpts (sanitized)
- Apiserver log backend config (`--audit-log-path` / `--audit-webhook-config-file`)
- The K8s version and managed-vs-self-managed cluster

Your job:

1. **For managed clusters (EKS/GKE/AKS)**:
   - Most providers enable audit logging at the control plane and forward to cloud logging
   - Customizing the policy may be limited
   - EKS: `--enable-control-plane-log-types audit api authenticator`
   - GKE: Cloud Logging audit logs auto-collected; policy levels configurable on Cluster
   - AKS: Diagnostic Settings for audit log forward
2. **For self-managed**: kube-apiserver flags
   - `--audit-policy-file=<path>` — policy
   - `--audit-log-path=<path>` — file backend
   - `--audit-log-maxsize=<MB>`, `--audit-log-maxage=<days>`, `--audit-log-maxbackup=<files>`
   - `--audit-webhook-config-file` — for remote shipping
3. **Audit policy levels** (per rule):
   - **`None`** — don't log
   - **`Metadata`** — request metadata only (no body)
   - **`Request`** — metadata + request body
   - **`RequestResponse`** — metadata + request + response body
   - Higher levels = more disk; tune per resource
4. **Common policy patterns**:
   - **Log all changes** at `Metadata` level
   - **Log secret reads** at `Metadata` (don't capture secret content)
   - **Log RBAC changes** at `RequestResponse`
   - **Log `exec`/`portforward`** at `Metadata` — high-signal
   - **Omit noisy reads** (e.g., `system:kube-scheduler` reading pods) at `None`
5. **For query / investigation**:
   - Look for `kubectl exec` events: `auditID`, `user`, `verb=create`, `subresource=exec`
   - Look for secret reads: `verb=get`, `objectRef.resource=secrets`
   - Look for impersonation: `impersonatedUser` field
   - Look for failed auth: `responseStatus.code=401/403`
   - Suspicious patterns: `system:anonymous`, `system:unauthenticated`
6. **For performance**:
   - High-cardinality logging (e.g., all pod list calls) overwhelms storage
   - Webhook backend can become a bottleneck — apiserver blocks on webhook response
   - Rate-limit at the policy level (omit high-volume reads)
7. **For detecting common attacks**:
   - **Reconnaissance**: unusual `list secrets`, `list serviceaccounts`, `get namespaces` from non-admin users
   - **Privilege escalation**: `bind` on roles/clusterroles, `create serviceaccounts/token`, `impersonate`
   - **Persistence**: new `cronjobs`, `mutatingwebhookconfigurations`, `validatingwebhookconfigurations`
   - **Lateral movement**: `kubectl exec` into unrelated pods, `port-forward` to in-cluster services
   - **Data exfil**: large `get` calls, repeated `download from pod`

Mark DESTRUCTIVE: changing audit policy without testing (may drown logs OR miss compromise), audit webhook misconfigured (blocks apiserver), audit log path on small disk (apiserver halt under disk pressure).

---

Goal: [configure / investigate / tune / detect]
Managed/self-managed: [DESCRIBE]
Current audit policy:
```yaml
[PASTE]
```
Recent log sample:
```
[PASTE]
```
Investigation context: [DESCRIBE]

Why this prompt works

Kubernetes audit logs are the forensic record of “who did what” but they’re verbose and easy to misconfigure. Most clusters either have no policy (drowning) or a too-restrictive one (missing the compromise). This prompt walks policy design and query patterns.

How to use it

1. Define a clear policy goal: compliance, IR, threat detection?
2. Start with a published policy (CIS, upstream sample) and customize.
3. Ship logs off-host — local files are limited in retention and an attacker target.
4. For investigations, narrow by user, verb, and resource.

Useful commands

# View current policy (self-managed)
sudo cat /etc/kubernetes/audit-policy.yaml
sudo grep audit /etc/kubernetes/manifests/kube-apiserver.yaml

# View audit log (self-managed file backend)
sudo tail -f /var/log/kubernetes/audit.log
sudo grep '"verb":"create"' /var/log/kubernetes/audit.log | head

# Parse JSON-line audit log
jq -r 'select(.verb=="create" and .objectRef.subresource=="exec") | "\(.requestReceivedTimestamp) \(.user.username) -> \(.objectRef.namespace)/\(.objectRef.name)"' /var/log/kubernetes/audit.log

# Top users by call count
jq -r '.user.username' /var/log/kubernetes/audit.log | sort | uniq -c | sort -nr | head

# Failed auth attempts
jq 'select(.responseStatus.code==401 or .responseStatus.code==403)' /var/log/kubernetes/audit.log

# Managed cluster (EKS example)
aws logs tail /aws/eks/<cluster>/cluster --filter-pattern "kubectl exec"
aws logs tail /aws/eks/<cluster>/cluster --filter-pattern '{ $.verb = "create" && $.objectRef.subresource = "exec" }'

# GKE
gcloud logging read 'resource.type="k8s_cluster" AND protoPayload.methodName="io.k8s.core.v1.pods.exec.create"' --limit 50

Sample audit policy (CIS-aligned)

# /etc/kubernetes/audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
# Don't log read-only system component requests
- level: None
  users: ["system:kube-proxy"]
  verbs: ["watch"]
  resources:
  - group: ""
    resources: ["endpoints", "services"]

- level: None
  users: ["system:unsecured"]
  namespaces: ["kube-system"]
  verbs: ["get"]

# Log secret access at Metadata (don't capture content)
- level: Metadata
  resources:
  - group: ""
    resources: ["secrets", "configmaps"]

# Log RBAC changes at RequestResponse
- level: RequestResponse
  verbs: ["create", "update", "patch", "delete"]
  resources:
  - group: "rbac.authorization.k8s.io"
    resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]

# Log exec / portforward / proxy explicitly
- level: Metadata
  resources:
  - group: ""
    resources: ["pods/exec", "pods/portforward", "pods/proxy", "services/proxy"]

# Log all changes to workload
- level: Metadata
  verbs: ["create", "update", "patch", "delete"]
  resources:
  - group: ""
  - group: "apps"
  - group: "batch"
  - group: "networking.k8s.io"

# Default: log everything else at Metadata
- level: Metadata
  omitStages:
  - RequestReceived

Detection patterns

Suspicious: kubectl exec to a critical pod

jq -r 'select(.verb=="create" and .objectRef.subresource=="exec") | "\(.requestReceivedTimestamp) \(.user.username) -> \(.objectRef.namespace)/\(.objectRef.name)"' /var/log/kubernetes/audit.log | \
    grep -E "kube-system|production"

Suspicious: list secrets across all namespaces

jq 'select(.verb=="list" and .objectRef.resource=="secrets" and (.objectRef.namespace // "" == "")) | {time:.requestReceivedTimestamp, user:.user.username}' /var/log/kubernetes/audit.log

Suspicious: impersonation

jq 'select(.impersonatedUser) | {time:.requestReceivedTimestamp, actor:.user.username, impersonated:.impersonatedUser.username, verb, objectRef}' /var/log/kubernetes/audit.log

Privilege escalation: creating a token for another SA

jq 'select(.verb=="create" and .objectRef.resource=="serviceaccounts" and .objectRef.subresource=="token") | {time:.requestReceivedTimestamp, user:.user.username, target:.objectRef.namespace + "/" + .objectRef.name}' /var/log/kubernetes/audit.log

New webhook configuration (persistence)

jq 'select(.verb=="create" and (.objectRef.resource=="mutatingwebhookconfigurations" or .objectRef.resource=="validatingwebhookconfigurations")) | {time:.requestReceivedTimestamp, user:.user.username, name:.objectRef.name}' /var/log/kubernetes/audit.log

Common findings this catches

• High volume of list pods from kube-controller-manager → normal; mark None to reduce noise.
• system:anonymous making API calls → anonymous auth enabled; disable.
• A user impersonating system:admin without that being their normal role → suspicious.
• pods/exec calls into kube-system → potential intrusion or privileged debugging; audit.
• serviceaccounts/token create from unexpected source → token theft.
• Webhook config changes outside expected admins → persistence; investigate.
• Audit log gaps (timestamps with no entries) → audit pipeline dropped; check backend.

When to escalate

• Suspected active compromise — preserve logs immediately; engage IR team; rotate credentials.
• Policy tuning that needs cluster-admin coordination — staged rollout.
• Audit webhook performance issues blocking apiserver — switch to batch mode or file backend.