🔐 Managing Secrets in Production: Vault, Sealed Secrets, and the Patterns That Actually Hold

The fastest way to turn a minor breach into a catastrophic one is to store your secrets badly. A leaked database password is bad. A leaked database password that’s also the root credential, hardcoded in a git repo, with no rotation and no audit trail — that’s the breach that makes the news.

In 25 years I’ve cleaned up enough plaintext-secret messes to have strong opinions. Here’s how I actually manage secrets in production, the two tools I reach for most, and how I use AI to catch the credentials that leak despite all of it.

The three rules

Before any tooling, internalize these:

1. Secrets never live in git in plaintext. Not in .env, not in a values file, not “temporarily.”
2. Secrets are short-lived and rotatable. A credential you can’t rotate in minutes is a liability, not an asset.
3. Every access is audited. You want to answer “who read the prod DB password, and when?” without guessing.

Everything below exists to enforce those three.

HashiCorp Vault: dynamic secrets and a real audit trail

Vault’s superpower isn’t storing static secrets — it’s generating them on demand and expiring them automatically. Instead of a long-lived database password baked into your app config, the app authenticates to Vault and gets a credential that’s valid for an hour.

# App requests a short-lived DB credential
vault read database/creds/app-readonly
# Returns username/password with a 1h TTL; Vault revokes it automatically

The blast radius of a leaked credential collapses from “forever” to “the next 60 minutes.” That alone justifies the operational cost.

Key things I always configure:

• Audit devices on day one. vault audit enable file file_path=/var/log/vault/audit.log. Without this you have no record of access, which defeats half the point.
• Least-privilege policies per app. Each workload gets a policy scoped to exactly the paths it needs — never a wildcard.
• Auto-unseal with a cloud KMS so a node restart doesn’t require a human with key shares at 3am.
• Auth via Kubernetes service accounts or cloud IAM, not static tokens. The identity the platform already trusts becomes the identity Vault trusts.

# A scoped policy — read-only, single path
path "database/creds/app-readonly" {
  capabilities = ["read"]
}

Sealed Secrets: GitOps-safe secrets for Kubernetes

Vault is the right answer for dynamic, app-fetched secrets. But sometimes you genuinely need a secret to live in git — a TLS cert, a webhook token — because your whole deployment flow is GitOps and “out of band” secrets break the model.

Bitnami’s Sealed Secrets solves this. A controller in the cluster holds a private key; you encrypt secrets with the matching public key using kubeseal. The encrypted SealedSecret is safe to commit because only the in-cluster controller can decrypt it.

# Encrypt a secret so it's safe to commit to git
kubectl create secret generic db-creds \
  --dry-run=client --from-literal=password=s3cr3t -o yaml \
  | kubeseal --format yaml > sealed-db-creds.yaml
git add sealed-db-creds.yaml   # safe to commit

The cluster decrypts it into a normal Secret at apply time. Your git history stays clean, your GitOps flow stays intact, and a repo leak exposes nothing usable.

The catch worth flagging: the sealing key is now your crown jewel. Back it up, restrict access to it, and have a documented recovery plan — if you lose it, every SealedSecret in the repo becomes undecryptable.

Don’t forget Kubernetes Secret hygiene

Whichever tool feeds them, the resulting Secret objects still need basic hygiene:

# Encrypt secrets at rest in etcd (EncryptionConfiguration on the API server)
# Restrict access with RBAC — secrets are not readable by default to everyone

• Enable encryption at rest for etcd so a disk image isn’t a credential dump.
• Lock down RBAC so only the service accounts that need a secret can get it.
• Mount secrets as files with restrictive permissions rather than environment variables where you can — env vars leak into crash dumps, child processes, and logs.

Using AI to hunt for leaked secrets

Here’s where AI is genuinely useful, and where I’m careful. Models are excellent at pattern-spotting credentials that dumb regex misses — a private key split across lines, a base64-encoded token, a connection string that looks like a comment.

I run it as a reviewer over diffs and manifests:

“Review this pull request diff for any committed secrets, credentials, private keys, or hardcoded tokens — including base64-encoded or obfuscated ones. For each finding, give the file, line, and what kind of secret it is. Do not output the full secret value; redact it.”

That last instruction matters: you don’t want the secret echoed into a chat log, you want to know it exists so you can rotate it. Pair AI scanning with a deterministic pre-commit scanner — the regex tool catches the obvious cases fast, the model catches the clever ones. Our Code Review tool runs exactly this kind of static pre-scan with an AI layer on top.

You can keep your standard secret-audit prompts alongside other security hardening prompts so the whole team scans consistently.

The rotation reflex

A secret that’s never rotated is just a long-lived liability with extra steps. Whatever you store, automate rotation:

• Vault dynamic secrets rotate themselves — that’s the point.
• Static secrets (API keys, certs) get a scheduled rotation job, not a calendar reminder.
• When anyone with access leaves, rotate everything they could have touched. “They probably didn’t copy it” is not a security control.

The short version

Keep secrets out of plaintext git, make them short-lived, and audit every access. Use Vault for dynamic, app-fetched credentials; use Sealed Secrets when GitOps forces secrets into the repo; lock down etcd encryption and RBAC underneath both. Then point AI at your diffs and manifests as a leak detector — redacting values, flagging existence, leaving rotation to your automation. The goal is simple: when a credential leaks, it’s already expired or already rotated, and you can prove exactly who touched it.

AI-generated secret scans are assistive, not authoritative. Always confirm findings and rotate any exposed credential immediately rather than assuming it was unused.