Kubernetes Cluster Upgrade Pre-Flight Planning Prompt

You are a senior Kubernetes platform engineer who has shepherded multiple production clusters across consecutive minor versions. You know which deprecations bite (in-tree CSI removal, PSP removal, removed beta APIs) and the right order to upgrade.

I will provide:
- Current Kubernetes version and target version (`kubectl version --short`)
- Cluster type: managed (EKS/GKE/AKS) or self-managed (kubeadm/kops/k3s/RKE)
- Topology: control-plane count, worker count, node pools, CNI, CSI drivers, ingress controllers
- Output of `kubectl get apiservices` (sanity for aggregated API servers)
- List of installed Helm releases (`helm list -A`)
- List of installed operators / CRDs (`kubectl get crd`)
- Workload inventory at a high level (Deployments, StatefulSets, CronJobs, DaemonSets counts and notable ones)
- Whether `kube-no-trouble` (kubent) or `pluto` has been run; if yes, paste output

Your job:

1. **Map the version jumps**. For each minor version between current and target:
   - List **removed APIs** (these will break manifests/clients that still reference them)
   - List **removed features and deprecations** (e.g., dockershim, PSP, in-tree storage drivers, beta APIs going GA-only)
   - List **default changes** (CRI socket location, kubelet config, feature gates flipping)
   - Note **kubeadm / kops / managed-service-specific** changes
2. **Per managed service**, surface known gotchas:
   - EKS: control-plane separately from node groups; AWS-VPC-CNI vs Cilium; aws-iam-authenticator → IAM auth v2
   - GKE: auto-upgrade windows; surge upgrade behavior; node-pool image-type changes
   - AKS: API-server vNet integration; kubenet vs Azure CNI; managed identity vs SP
3. **Enumerate workload risks**:
   - Manifests/Helm charts using removed APIs (run `kubectl-no-trouble` or `pluto detect-helm`)
   - In-tree CSI drivers being removed → migrate to out-of-tree CSI (cloud-provider PVs need migration)
   - PodSecurityPolicy (PSP) removed in 1.25 → replace with Pod Security Admission or third-party (Kyverno, Gatekeeper)
   - Removed flags on kubelet/kube-apiserver that your config still references
   - Webhook controllers compiled against old API client-go (Kyverno, cert-manager, ingress-nginx versions)
4. **Recommend upgrade ordering**:
   - Pre-flight: backup etcd (self-managed), snapshot PVs, freeze deployments
   - Upgrade control plane first
   - Upgrade nodes / node pools one pool at a time (canary first)
   - Upgrade addons LAST (CNI, CSI, ingress, cert-manager) — but verify the new versions support the new K8s
5. **For each addon**: check the compatibility matrix.
6. **List the smoke-test checklist** for post-upgrade validation:
   - All nodes Ready, version matches
   - All system pods Running
   - DNS resolution works (`busybox nslookup kubernetes.default`)
   - Ingress serves a known endpoint
   - PVCs still bound and pods can mount
   - HPAs still scaling
   - cert-manager issuing certs
   - Metrics-server / Prometheus still scraping
7. **Identify the rollback path** and mark the point of no return (typically: etcd migration on self-managed, or node-pool replacement on managed).

Mark all DESTRUCTIVE steps clearly.

---

Current K8s version: [e.g., v1.27.10]
Target K8s version: [e.g., v1.30.0]
Cluster type: [EKS / GKE / AKS / kubeadm / kops / k3s / RKE]
Topology summary:
```
[DESCRIBE — control-plane count, node pools, CNI, CSI, ingress, namespaces]
```
Installed CRDs:
```
[PASTE kubectl get crd -o name]
```
Helm releases:
```
[PASTE helm list -A]
```
Pre-existing audit (kubent / pluto):
```
[PASTE if available]
```

Why this prompt works

Kubernetes deprecates and removes APIs on a known schedule, but most clusters drift behind. The deprecations that bite the hardest are the ones that have been “deprecated for 3 versions” — your manifests still work in 1.27, then on day one of 1.29 every controller using policy/v1beta1 PodDisruptionBudget breaks. This prompt forces an inventory-first walk: what’s currently in use, what gets removed, what depends on what.

How to use it

1. Run kubent or pluto detect-helm BEFORE writing the plan. They scan live manifests for deprecated/removed APIs. Their output is your pre-flight punch list.
2. For each Helm release, check its release notes for K8s version compatibility. Old chart versions may not support your target.
3. Capture the addon list (CNI, CSI, ingress, cert-manager, kyverno, prometheus-operator, etc.) and their versions — these need coordinated upgrades.
4. Treat managed and self-managed differently. EKS/GKE/AKS each have their own upgrade quirks documented in provider docs.

Useful commands

# Versions
kubectl version --short
kubectl get nodes -o wide                  # node versions vs API server
kubectl get apiservices                    # detect aggregated APIs

# CRD inventory
kubectl get crd
helm list -A

# Find use of deprecated APIs
# Install kubent: https://github.com/doitintl/kube-no-trouble
kubent
kubent --helm3

# Pluto (alternative)
# https://github.com/FairwindsOps/pluto
pluto detect-helm -o wide
pluto detect-files -d ./manifests

# What versions does the API server know about?
kubectl api-resources -o wide
kubectl api-versions

# Discover in-tree CSI usage (migration target)
kubectl get storageclass -o yaml | grep -E "provisioner:|kubernetes.io/"
# Anything with provisioner=kubernetes.io/* is in-tree → migrate

# PodSecurityPolicy (removed 1.25)
kubectl get psp
# If present, plan migration to Pod Security Admission OR Kyverno/Gatekeeper

# Webhook controllers (cert-manager, kyverno, ingress-nginx etc.)
kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations

# etcd backup (self-managed)
sudo ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /backup/etcd-$(date +%F).db
sudo ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-$(date +%F).db

Notable removals to plan around (cumulative)

Version	Removed	What to do
1.22	extensions/v1beta1 Ingress, networking.k8s.io/v1beta1 Ingress	Move to networking.k8s.io/v1
1.25	PodSecurityPolicy	Pod Security Admission OR Kyverno/Gatekeeper
1.25	cronjob v1beta1, endpointslice v1beta1	Use v1 versions
1.26	flowcontrol v1beta1, horizontalpodautoscaler v2beta2	Use v1 / v2
1.27	In-tree volume plugins (most) being migrated to CSI	Migrate to CSI drivers
1.29	Several CSI in-tree drivers fully removed	Migrate; verify storage works
1.30	Various beta flags removed	Audit kubelet/apiserver flags
1.32+	(check current upstream notes)	—

(Always verify against upstream changelogs; this list ages.)

Addon compatibility matrix (verify per release)

Addon	Verify	Common breakage
cert-manager	Compat matrix in their docs	Webhook timeout if version too old
ingress-nginx	Compat in chart README	Annotations changed across major bumps
Cilium / Calico	Vendor compat docs	CRD bumps required
Kyverno	Webhook + new policy types	May reject API objects on new K8s
metrics-server	Args changed between releases	TLS args; insecure-skip-tls-verify
prometheus-operator	CRDs change between major releases	scrape configs
velero	Backup driver compatibility	CSI snapshots

Upgrade ordering (high-level)

1. Pre-flight
   - Back up etcd (self-managed)
   - Snapshot PVs (or verify backups exist)
   - Freeze deployments / merges
   - kubent / pluto clean OR migrate findings first

2. Upgrade addons that NEED upgrading BEFORE cluster jump
   (e.g., cert-manager < new K8s minor)

3. Upgrade control plane
   - Managed: trigger via provider; usually irreversible
   - Self-managed: kubeadm upgrade plan; kubeadm upgrade apply

4. Smoke test control plane only
   - kubectl get nodes  (control-plane shows new version; nodes still old)
   - kubectl get all -A | head
   - Webhook health

5. Upgrade nodes (one pool at a time)
   - Cordon + drain a canary node
   - Replace / upgrade
   - Verify pods reschedule and run

6. Upgrade addons that pair with new K8s
   - CNI
   - CSI
   - ingress controller

7. Post-upgrade smoke tests (see below)

8. Unfreeze

Post-upgrade smoke test checklist

# All nodes Ready, version matches
kubectl get nodes -o wide

# System pods running
kubectl get pods -n kube-system

# DNS works
kubectl run dnscheck --rm -it --image=busybox:1.28 --restart=Never -- \
  nslookup kubernetes.default

# Ingress serves
curl -v https://<known-endpoint>/

# PVCs still bound, pod can mount
kubectl get pvc -A
kubectl run pvc-test --rm -it --image=alpine --restart=Never \
  --overrides='{"spec":{"volumes":[{"name":"v","persistentVolumeClaim":{"claimName":"<pvc>"}}],"containers":[{"name":"c","image":"alpine","stdin":true,"tty":true,"volumeMounts":[{"name":"v","mountPath":"/data"}]}]}}' -- sh

# Metrics
kubectl top node
kubectl top pod -A

# HPAs scaling
kubectl get hpa -A

When to escalate

• Self-managed cluster: anything involving etcd recovery from a failed control-plane upgrade — engage etcd-experienced engineer.
• Managed cluster: an upgrade in progress that’s stuck for >2× the expected window — open a provider support ticket; don’t fight the upgrade controller.
• Addon vendor compatibility unclear — pin to a known-good version pair rather than guessing.
• Discovered after upgrade that a webhook is broken and all creates fail — temporarily delete the webhook (after disabling the workload that uses it) to recover the cluster; re-enable when fixed.