Kubernetes Init Containers Design Prompt

You are a senior Kubernetes engineer who has used init containers for dependency checks, config injection, schema migrations, and image construction. You know how they differ from regular containers and how to debug "pod stuck in Init:0/2."

I will provide:
- The pod spec with init containers
- The use case (dependency wait, secret pull, migration, build/copy)
- Symptom (init failing, slow startup, init succeeds but main fails)

Your job:

1. **Init container semantics**:
   - Run sequentially BEFORE app containers
   - Each must complete successfully (exit 0) before next runs
   - If one fails: retry per `restartPolicy` (Pod-level, applies)
   - Logs available via `kubectl logs <pod> -c <init-container>`
   - Pod status: `Init:N/M` (running init N of M)
2. **Common use cases**:
   - **Wait for dependency** (e.g., wait for postgres to be reachable)
   - **Pull secrets** from external source into emptyDir
   - **Database migration** (run alembic / flyway before app starts)
   - **Fetch data** (download model file, config)
   - **Permission fix** (chown emptyDir for non-root app)
3. **For "Init:0/N stuck"**:
   - First init container running but not exiting; check its logs
   - Wait scripts in tight loop without backoff
   - Network reach issue
4. **For "Init failed and pod CrashLoopBackOff"**:
   - Init returned non-zero; Pod retried per restartPolicy
   - Fix init logic; redeploy
5. **For "Init succeeds but main fails to find what init left"**:
   - emptyDir or projected volume mounted in both
   - Path differences
   - Permissions (init ran as root, main as non-root)
6. **For dependency wait patterns**:
   - Better: use service mesh / readiness probe ordering
   - Init for "this MUST exist" pre-start checks only
7. **For migration init**:
   - Idempotent migration (no-op if applied)
   - Locking to prevent concurrent migrations from multiple pods
   - Don't run on every pod start — better as Job before deploy
8. **For sidecar patterns vs init** (1.28+):
   - Native sidecars are `initContainers` with `restartPolicy: Always`
   - Run alongside main; reverse shutdown order

Mark DESTRUCTIVE: idempotent-only migrations (don't run destructive in init), removing init containers that other containers depend on, long-running init causing slow pod startup.

---

Pod spec excerpt:
```yaml
[PASTE init containers]
```
Use case: [DESCRIBE]
Symptom: [DESCRIBE]

Why this prompt works

Init containers solve real problems but are often misused — running migrations on every start, waiting forever for dependencies. This prompt walks the patterns.

How to use it

1. Pick use case carefully — Init for “must succeed before start” only.
2. For migrations, prefer Job unless idempotent + locked.
3. For permissions, verify chown matches main UID.
4. For waits, use timeouts.

Useful commands

# Pod state
kubectl get pod <pod>                # Status: Init:0/2 = running first of 2 inits

# Init logs
kubectl logs <pod> -c <init-container>
kubectl logs <pod> -c <init-container> --previous

# Describe
kubectl describe pod <pod>

# Force re-run init (delete pod)
kubectl delete pod <pod>

Patterns

Wait for DB

initContainers:
- name: wait-for-db
  image: busybox:1.36
  command:
  - sh
  - -c
  - |
    until nc -z postgres-svc 5432; do
      echo "Waiting for postgres..."
      sleep 2
    done
    echo "Postgres ready"

(Better with a timeout; the above can hang forever.)

Pull secrets from Vault

initContainers:
- name: vault-secrets
  image: vault:1.15
  env:
  - name: VAULT_ADDR
    value: https://vault.example.com
  - name: VAULT_TOKEN
    valueFrom:
      secretKeyRef: { name: vault-token, key: token }
  command:
  - sh
  - -c
  - |
    vault kv get -format=json secret/myapp > /vault/secrets.json
  volumeMounts:
  - name: secrets
    mountPath: /vault
containers:
- name: app
  image: myapp
  volumeMounts:
  - name: secrets
    mountPath: /etc/secrets
    readOnly: true
volumes:
- name: secrets
  emptyDir:
    medium: Memory

Fix permissions

initContainers:
- name: fix-perms
  image: busybox
  command: ["sh", "-c", "chown -R 1000:1000 /data"]
  securityContext:
    runAsUser: 0          # need root to chown
  volumeMounts:
  - name: data
    mountPath: /data
containers:
- name: app
  image: myapp
  securityContext:
    runAsUser: 1000        # non-root main
  volumeMounts:
  - name: data
    mountPath: /data

Database migration (NOT recommended unless idempotent)

# Better: run as a Job before deploy
# This init runs on EVERY pod start, including HPA scale-up

initContainers:
- name: migrate
  image: myapp:v1
  command: ["alembic", "upgrade", "head"]
  env:
  - name: DATABASE_URL
    valueFrom:
      secretKeyRef: { name: db, key: url }

Native sidecar (K8s 1.28+)

initContainers:
- name: log-sidecar
  image: fluentbit
  restartPolicy: Always           # makes it a sidecar
  volumeMounts:
  - name: logs
    mountPath: /logs
containers:
- name: app
  image: myapp
  volumeMounts:
  - name: logs
    mountPath: /logs

The log-sidecar starts BEFORE main, runs alongside, terminates AFTER.

Common findings this catches

• Init hangs forever waiting on missing dependency → add timeout + exit non-zero.
• Migration init running on every pod start → race condition; use Job.
• Main container can’t read what init wrote → permissions; chown in init.
• emptyDir not shared properly — mount in both containers correctly.
• Init image too large — slow startup; build minimal.
• Init logs not captured → log shipper not aware of init container names.
• Many sequential inits → cumulative latency; parallelize if possible.

When to escalate

• Persistent migration races → architecture review.
• Sidecar 1.28+ adoption — coordinate cluster upgrade.
• Slow init due to upstream service — engage owners.