Kubernetes Secrets Management Review Prompt

You are a senior Kubernetes security engineer with deep experience operating Secrets across managed and self-managed clusters — including KMS envelope encryption, sealed-secrets, external-secrets-operator, and Vault integration.

I will provide:
- A representative set of Secrets, Pods that use them, and SA bindings
- Cluster context: managed (EKS/GKE/AKS) or self-managed; etcd encryption-at-rest config (`--encryption-provider-config` on kube-apiserver)
- Inventory of secret-related tools installed (sealed-secrets, ESO, vault-injector, etc.)
- The audit scope (one namespace, all, just CI/CD-related secrets, etc.)

Your job:

1. **Storage layer hygiene**:
   - Is etcd encryption-at-rest enabled? On managed services, this is often default (EKS, GKE, AKS) but verify
   - On self-managed: `--encryption-provider-config` set on `kube-apiserver` with a non-`identity` provider (aescbc, secretbox, kms)
   - KMS provider used? (Strongest: keys live in cloud KMS, not on disk)
   - Default Secret type usage — most should be `Opaque`; `kubernetes.io/dockerconfigjson` for registry; `kubernetes.io/tls` for TLS; SA tokens for SAs
2. **Access layer**:
   - Run RBAC audit on `get/list/watch secrets` — who can read all secrets? (See [kubernetes-rbac-audit](/prompts/kubernetes-rbac-audit/))
   - Are there `default` ServiceAccounts mounted into pods that get `secrets get`?
   - Service-account-token automounting: `automountServiceAccountToken: false` on pods that don't talk to the API server
3. **Application layer**:
   - **`env` from secret** vs **volume mount**: env vars are visible in `/proc/<pid>/environ` (any other process in the container's PID namespace) AND show up in pod descriptions. Prefer file mounts.
   - **`subPath` mount of a secret** doesn't auto-update when the secret changes; without subPath, kubelet syncs ~60s. Apps may need a sighup or restart.
   - **Hardcoded secret data in ConfigMaps**: ConfigMaps are NOT secrets. Move to a Secret.
   - **Base64 is encoding, not encryption.** Secret data being base64'd in YAML is not security; it's the on-the-wire format.
4. **Rotation & lifecycle**:
   - How are secrets rotated? Manual `kubectl edit` is fragile. Pod doesn't auto-restart on Secret change (unless using a controller like Reloader or pod template hash).
   - Are app credentials short-lived (Vault dynamic, IRSA, Workload Identity) or long-lived?
   - Stale secrets (last modified > 90 days)? Risk if compromised long ago.
5. **Tool selection** — recommend or critique:
   - **SealedSecrets** (Bitnami): seals at apply time using controller's public key; lives in git; controller decrypts at runtime. Good for GitOps. Doesn't help with rotation.
   - **External Secrets Operator (ESO)**: pulls from Vault/AWS Secrets Manager/etc. into K8s Secret. Centralized rotation source.
   - **Vault Agent Injector / CSI Secrets Store**: bypass K8s Secret entirely; mount directly. Strongest.
   - **kubeseal+sops+git** vs **ESO**: choose based on rotation needs.
6. **For each finding** produce: **severity**, **target Secret / pod / SA**, **specific issue**, **suggested remediation**.
7. **Common dangerous patterns** to surface explicitly:
   - Secret name in a public ConfigMap or in image labels
   - Secret committed to git (even base64'd)
   - Multi-namespace secret sharing via copy-paste rather than sync tool
   - Secret with `creationTimestamp` > 1 year, never rotated
   - Service account token with no expiration (legacy v1 token; modern uses BoundServiceAccountToken)
   - Pod mounts secret as env AND prints env on startup (env vars in app logs)

Mark anything DESTRUCTIVE: rotating a production secret without coordinating with consumers, deleting Secrets that are referenced.

---

Cluster context: [DESCRIBE]
Etcd encryption status: [enabled with provider X / unknown / disabled]
Secret tooling installed: [sealed-secrets / ESO / Vault / none]
Sample Secret(s) to review:
```yaml
[PASTE one or more]
```
Pods consuming them:
```yaml
[PASTE relevant pod spec excerpts]
```
SA + RBAC for those pods:
```yaml
[PASTE]
```
Inventory question: [DESCRIBE the broader audit goal]

Why this prompt works

Kubernetes Secrets are misunderstood: they’re base64-encoded blobs, not encrypted blobs. Without etcd encryption-at-rest, they’re plaintext on disk. Without RBAC discipline, anyone with secrets get can read every credential. Most secret incidents trace to “we forgot one of these layers.” This prompt covers all four: storage, access, app, and lifecycle.

How to use it

1. Always check etcd encryption first. On managed services, it’s usually enabled — but verify with provider docs. Self-managed almost always needs explicit config.
2. Cross-reference with an RBAC audit. Secret strength matters less if every pod can read every secret.
3. For app-layer review, prefer volume mounts over env vars for anything that could leak into logs (DB passwords, API tokens).
4. Score by blast radius: a CI/CD token that can deploy is more dangerous than a read-only DB credential.

Useful commands

# Inventory
kubectl get secrets -A
kubectl get secrets -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name} type={.type} age={.metadata.creationTimestamp}{"\n"}{end}'

# Find secrets that haven't been touched in a year
kubectl get secrets -A -o json | jq -r --arg cutoff "$(date -u -d '1 year ago' +%Y-%m-%d)" \
  '.items[] | select(.metadata.creationTimestamp < $cutoff) | "\(.metadata.namespace)/\(.metadata.name) \(.metadata.creationTimestamp)"'

# Which pods mount each secret?
kubectl get pods -A -o json | jq -r \
  '.items[] | . as $p | (.spec.volumes // [])[] | select(.secret) | "\($p.metadata.namespace)/\($p.metadata.name) volume=\(.name) secret=\(.secret.secretName)"'

# Which pods consume secret env vars?
kubectl get pods -A -o json | jq -r \
  '.items[] | . as $p | (.spec.containers // [])[] | . as $c |
    ((.env // [])[] | select(.valueFrom.secretKeyRef) |
       "\($p.metadata.namespace)/\($p.metadata.name) container=\($c.name) env=\(.name) secret=\(.valueFrom.secretKeyRef.name)")'

# Etcd encryption (self-managed; you need the kube-apiserver flags)
ssh control-plane-host \
  'sudo grep encryption-provider-config /etc/kubernetes/manifests/kube-apiserver.yaml'
# Then read the file referenced; look for `providers:` and make sure first is NOT `identity`

# Test etcd encryption is working (self-managed)
sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/<secret-name> | hexdump -C | head
# If you see `k8s:enc:aescbc:v1:` prefix → encrypted
# If you see plaintext → encryption broken or `identity` provider

# RBAC: who can read secrets?
kubectl rbac-tool who-can list secrets        # krew plugin
kubectl rbac-tool who-can get secrets -A

# SA token automounting status
kubectl get pods -A -o json | jq -r \
  '.items[] | "\(.metadata.namespace)/\(.metadata.name) automount=\(.spec.automountServiceAccountToken // true)"'

# Sealed-secrets verify
kubectl get sealedsecrets -A
kubectl get pods -n kube-system -l name=sealed-secrets-controller

# ESO verify
kubectl get externalsecrets -A
kubectl get secretstores -A
kubectl get clustersecretstores

Recommended patterns

Volume mount over env

# Bad — env var with secret value
env:
- name: DB_PASSWORD
  valueFrom:
    secretKeyRef:
      name: db-creds
      key: password

# Better — file mount; nothing in env or `kubectl describe pod`
volumes:
- name: db-creds
  secret:
    secretName: db-creds
    defaultMode: 0400
containers:
- name: app
  volumeMounts:
  - name: db-creds
    mountPath: /etc/secrets
    readOnly: true
# App reads /etc/secrets/password

Automount=false where possible

# Pod doesn't talk to API server → don't mount SA token
spec:
  automountServiceAccountToken: false

Reloader for auto-restart on secret change

# https://github.com/stakater/Reloader
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    secret.reloader.stakater.com/reload: "db-creds"

ESO + AWS Secrets Manager

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-sm
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-creds
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-sm
    kind: SecretStore
  target:
    name: db-creds
  data:
  - secretKey: password
    remoteRef:
      key: prod/db/password

Common findings this catches

• Etcd --encryption-provider-config references a file that has identity provider listed first → secrets stored unencrypted; the YAML “looks configured” but isn’t.
• SAs in default namespace bound to view — view includes get/list secrets. Limit further with custom Role.
• Pod mounts a Secret as env AND logs printenv on startup → secrets in stdout, persisted in log aggregation.
• kubectl describe pod shows secret env var names but values are hidden — names alone can leak structure (“DB_HOST=…”, “STRIPE_KEY=…”).
• TLS Secret with private key in Opaque type instead of kubernetes.io/tls → harder to validate; the typed Secret is checked by some controllers.
• automountServiceAccountToken: true on a static-site pod — pod has API access it doesn’t need.
• Same DB credential mounted in 12 namespaces via kubectl create secret copies — rotation requires touching all 12.
• Secrets older than 1 year, never rotated in a namespace owned by a former team.

When to escalate

• Discovery of a leaked secret in git history or logs — coordinate with security; rotate the underlying credential before cleaning up the K8s secret.
• Major rotation projects (rotate every prod DB password) — staged rollout, application change management.
• Etcd-at-rest encryption migration from identity to KMS — requires reading and rewriting every secret in the cluster; plan a window.