AI for Kubernetes & Helm Difficulty: Advanced ClaudeChatGPTCursor

Kubernetes Encryption-at-Rest KMS Provider Design Prompt

Design and roll out etcd encryption-at-rest with an EncryptionConfiguration and a KMS v2 provider — provider ordering, key rotation, and re-encrypting existing Secrets without downtime.

Target user: Kubernetes platform and security engineers hardening self-managed control planes
Difficulty: Advanced
Tools: Claude, ChatGPT, Cursor

The prompt

You are a senior Kubernetes platform-security engineer who has enabled encryption-at-rest on self-managed clusters and migrated fleets from the legacy `aescbc` provider to KMS v2. You understand exactly how the kube-apiserver `EncryptionConfiguration` resolves providers, why provider *order* determines read vs. write behavior, and how to re-encrypt existing objects without an outage or a lost key.

I will provide:
- Kubernetes version and how the control plane is managed (kubeadm, static pods, managed offering with a customizable apiserver, etc.)
- Which resources must be encrypted (at minimum `secrets`; often `configmaps`, custom resources holding tokens)
- The intended key backend (cloud KMS via a KMS plugin socket, or local `aescbc`/`secretbox` as an interim step)
- Current state: no encryption, legacy `aescbc`, or partially migrated
- Constraints: HA apiserver count, allowed restart windows, compliance requirement (FIPS, key rotation cadence)

Your job:

1. **Explain the model precisely.** The apiserver reads `--encryption-provider-config`. For each resource, the provider list is ordered: the **first** provider is used to **write (encrypt)**; **all** providers are tried in order to **read (decrypt)**. `identity` means plaintext. This ordering is the key to every safe migration and rotation.

2. **Produce the `EncryptionConfiguration`** (apiVersion `apiserver.config.k8s.io/v1`) for the target state. For KMS v2 use `kind: KMS`, `apiVersion: v2`, a unique `name`, the plugin `endpoint` (unix socket), and `timeout`. Show the resource list, and be explicit that `secrets` should list the KMS provider first and `identity` last only during migration.

3. **Give the rollout sequence for an HA control plane**, one apiserver at a time:
- Stage the config file and the KMS plugin (static pod / systemd) on every control-plane node.
- Add `--encryption-provider-config=<path>` (and `--encryption-provider-config-automatic-reload=true` where supported) to each apiserver, rolling node by node so at least one apiserver stays up.
- Verify each apiserver comes back healthy before moving to the next.

4. **Re-encrypt existing data.** New writes use the first provider, but objects already in etcd stay in their old form until rewritten. Provide the re-encrypt step:
`kubectl get secrets --all-namespaces -o json | kubectl replace -f -`
and explain that this rewrites every Secret through the current write provider. Note the same must be done for any other resource type you added.

5. **Verify encryption actually happened** by reading the raw value straight from etcd:
`etcdctl get /registry/secrets/<ns>/<name> | hexdump -C`
and interpreting the prefix (`k8s:enc:kms:v2:<name>:` for KMS v2, `k8s:enc:aescbc:v1:` for aescbc, or readable plaintext meaning it is NOT encrypted).

6. **Design key rotation.** For KMS v2, rotation is driven by the plugin's key ID; explain observing `apiserver_encryption_config_controller_automatic_reloads_total` and the KMS key-ID change, then re-encrypting. For local providers, rotation means generating a new key, placing it FIRST in the provider list, reloading, and re-running the re-encrypt loop so old keys can later be removed.

7. **Call out the failure modes** that cause an outage or data loss, and how to avoid each: removing a provider whose keys still encrypt live data (Secrets become undecryptable), KMS plugin socket down at apiserver start (apiserver fails to serve), and mismatched configs across HA members (intermittent decrypt failures depending on which apiserver serves the read).

Mark DESTRUCTIVE / IRREVERSIBLE clearly: deleting or reordering a provider before re-encrypting, and losing the KMS key material or local key file (permanently unreadable Secrets).

---

Kubernetes version + control-plane management: [DESCRIBE]
Resources to encrypt: [secrets / configmaps / CRDs — LIST]
Key backend: [KMS plugin / aescbc / secretbox]
Current state: [none / aescbc / partial]
Constraints (HA count, restart window, compliance): [DESCRIBE]
Current EncryptionConfiguration (if any):
```yaml
[PASTE or "none"]
```

Run this prompt with AI

Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.

Safety notes

Provider ORDER is load-bearing: the first provider encrypts writes; all providers are tried for reads. Putting `identity` first silently writes NEW Secrets as plaintext.
Never remove a provider until you have re-encrypted every object that could still be sealed with its key. `kubectl get secrets -A -o json | kubectl replace -f -` rewrites Secrets through the current write provider — run it and verify before deleting the old provider.
Losing KMS key material (or a local `aescbc`/`secretbox` key file) makes every Secret encrypted with it PERMANENTLY undecryptable. Back up local key files as sensitively as the data they protect; for cloud KMS, protect the key from deletion/disable.
In HA, every apiserver must load an equivalent config. A mismatched member will fail to decrypt objects written by another, producing intermittent 500s on Secret reads depending on which apiserver answers.
If the KMS plugin socket is unavailable when the apiserver starts, the apiserver will not serve. Deploy the plugin as a static pod / early systemd unit and confirm the socket before adding the apiserver flag.
Verify with raw etcd reads (`etcdctl get /registry/secrets/<ns>/<name> | hexdump -C`), not by trusting that the config was applied — a readable plaintext prefix means the object is NOT encrypted yet.

Why this prompt works

Encryption-at-rest is deceptively easy to turn on and easy to get subtly wrong. The apiserver happily starts with a config that writes new Secrets as plaintext, and existing Secrets stay in their old encoding until something rewrites them. This prompt forces the model to reason about provider ordering, the read-vs-write asymmetry, and the re-encrypt step — the three things people miss.

How to use it

State your control-plane management model — the rollout steps differ between kubeadm static pods and a managed apiserver you can only pass flags to.
Always include secrets in the resource list; add configmaps and token-bearing CRDs if they hold sensitive data.
Do the re-encrypt loop after enabling — turning on the config is not enough.
Verify from etcd, not from the config, before you consider it done.

Useful commands

# Point the apiserver at the config (kubeadm static pod manifest excerpt)
# --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
# --encryption-provider-config-automatic-reload=true

# Re-encrypt all Secrets through the current write provider
kubectl get secrets --all-namespaces -o json | kubectl replace -f -

# Re-encrypt another resource you added (example: configmaps)
kubectl get configmaps --all-namespaces -o json | kubectl replace -f -

# Verify the on-disk encoding straight from etcd
ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/my-secret | hexdump -C | head

# KMS v2 reload / health signals
kubectl get --raw /metrics | grep apiserver_encryption_config_controller_automatic_reloads_total

Pattern: KMS v2 EncryptionConfiguration (target state)

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - configmaps
    providers:
      - kms:
          apiVersion: v2
          name: cluster-kms
          endpoint: unix:///var/run/kmsplugin/socket.sock
          timeout: 3s
      - identity: {}   # only during migration; remove after re-encrypt

Pattern: interim local provider (no cloud KMS yet)

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key-2026-07
              secret: <base64-32-byte-key>   # keep OUT of git; sensitive
      - identity: {}

Pattern: key rotation for a local provider

providers:
  - aescbc:
      keys:
        - name: key-2026-10       # NEW key first => used for writes
          secret: <new-base64-key>
        - name: key-2026-07       # OLD key still present for reads
          secret: <old-base64-key>
  - identity: {}

Reload the config, re-run the Secret re-encrypt loop so everything is rewritten with key-2026-10, then drop key-2026-07 on the next change.

Common findings this catches

New Secrets are plaintext → identity (or a passthrough) is listed first in the provider list.
Old Secrets still readable in etcd after enabling → the re-encrypt loop was never run.
Intermittent 500 on Secret reads in HA → one apiserver has a different or missing config.
Apiserver won’t start → KMS plugin socket not present at boot, or wrong endpoint path.
Rotation “done” but old key still required → objects not re-encrypted; removing the old key would break decryption.

When to escalate

KMS plugin crash-looping or high timeout latency — coordinate with the team owning the cloud KMS integration.
Suspected unreadable Secrets after a botched provider removal — stop, restore the removed provider/key from backup before any further change.
Compliance-driven rotation cadence or FIPS requirements — involve security/compliance before finalizing the key policy.

Related prompts

More Kubernetes & Helm prompts & error guides

Browse every Kubernetes & Helm prompt and troubleshooting guide in one place.

Free download · 368-page PDF

Reading prompts? Get all 500 in one free PDF

500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.

500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
Instant PDF download — yours free, forever
Plus one practical AI-workflow email a week (no spam)

Single opt-in · unsubscribe anytime · no spam.