Kubernetes VolumeSnapshot & CSI Snapshot Prompt

You are a senior Kubernetes engineer who has used CSI snapshots for backup and restore — VolumeSnapshotClass, VolumeSnapshot, VolumeSnapshotContent across CSI drivers (EBS, GCE PD, Ceph RBD).

I will provide:
- The CSI driver
- The snapshot use case (point-in-time backup, clone for testing)
- Symptom (snapshot stuck, restore fails)

Your job:

1. **CSI snapshot objects**:
   - **VolumeSnapshotClass** — defines snapshot driver + retention policy
   - **VolumeSnapshot** — user-facing object (in namespace)
   - **VolumeSnapshotContent** — cluster-scoped, backed by actual snapshot
   - Like PVC/PV pair
2. **For snapshot stuck**:
   - Snapshotter sidecar in CSI driver not running
   - Driver doesn't support snapshots (verify CSIDriver)
   - Backend issue (no space, IAM)
3. **For restore**:
   - Create new PVC with `dataSource: kind: VolumeSnapshot, name: <snap>`
   - Provisioner creates volume from snapshot
   - Backend-specific copy time (instant for some, slow for others)
4. **For VolumeSnapshotClass**:
   - `driver` (must match installed CSI)
   - `deletionPolicy: Delete` or `Retain`
   - Optional parameters
5. **For snapshotter sidecars**:
   - `csi-snapshotter` sidecar in CSI driver pod
   - Runs alongside `csi-provisioner` and `csi-attacher`
   - Needs RBAC
6. **For backup tooling integration**:
   - Velero supports CSI snapshots
   - Stash, Kasten use them
   - Coordinated backup of multiple PVCs

Mark DESTRUCTIVE: setting deletionPolicy: Delete on important snapshots (lost when VS deleted), snapshotting without quiescing app (crash-consistent only), restoring over running volume.

---

CSI driver: [DESCRIBE]
Use case: [DESCRIBE]
VolumeSnapshot state:
```
[PASTE]
```
Symptom: [DESCRIBE]

Why this prompt works

CSI snapshots solve real backup needs but the object model is confusing. This prompt walks it.

How to use it

1. Verify CSI driver supports snapshots.
2. Check snapshotter sidecar running.
3. For restore, use dataSource.
4. For backups, use Retain policy.

Useful commands

# CSI capabilities
kubectl get csidriver
kubectl get csidriver <name> -o yaml | yq '.spec'

# VolumeSnapshot
kubectl get volumesnapshot -A
kubectl get volumesnapshot <name> -o yaml

# VolumeSnapshotContent (cluster-scoped)
kubectl get volumesnapshotcontent
kubectl describe volumesnapshotcontent <name>

# VolumeSnapshotClass
kubectl get volumesnapshotclass

# CSI driver pods
kubectl get pods -n <csi-ns>
kubectl logs <csi-controller-pod> -c csi-snapshotter

# Create snapshot
kubectl apply -f - <<EOF
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: mydata-snap-1
spec:
  volumeSnapshotClassName: ebs-snapshot-class
  source:
    persistentVolumeClaimName: mydata
EOF

# Restore from snapshot
kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mydata-restored
spec:
  storageClassName: gp3
  dataSource:
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
    name: mydata-snap-1
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 50Gi
EOF

Patterns

VolumeSnapshotClass

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: ebs-csi-snapshot-class
driver: ebs.csi.aws.com
deletionPolicy: Retain                # IMPORTANT for backups
parameters:
  tagSpecification_1: "team=platform"

Periodic snapshot via CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
  name: db-snapshot
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: snapshotter
          containers:
          - name: snap
            image: bitnami/kubectl
            command:
            - sh
            - -c
            - |
              cat <<EOF | kubectl apply -f -
              apiVersion: snapshot.storage.k8s.io/v1
              kind: VolumeSnapshot
              metadata:
                name: db-$(date +%F-%H%M)
                namespace: production
              spec:
                volumeSnapshotClassName: ebs-csi-snapshot-class
                source:
                  persistentVolumeClaimName: db-data
              EOF
          restartPolicy: OnFailure

Common findings this catches

• VolumeSnapshot stuck READYtoUSE: false → snapshotter sidecar not running.
• Restore PVC stuck Pending → provisioner can’t fulfill from snapshot (cross-region, wrong class).
• deletionPolicy: Delete caused backup loss → use Retain for backups.
• CSI driver doesn’t support snapshots → check CSIDriver capability.
• Snapshot quota exceeded in cloud → manage retention.
• Cross-AZ restore slow → backend may not support directly.
• App not quiesced during snapshot → crash-consistent only.

When to escalate

• Backend snapshot service issues → cloud / storage team.
• Cross-region snapshot architecture — strategic.
• Velero / backup tool integration — coordinate.