GitLab CI/CD Distributed Runner Cache on S3 Backend Prompt
Configure a shared S3-backed distributed cache so autoscaling runners reuse build dependencies across hosts instead of rebuilding cold every job.
- Target user
- Platform engineers running ephemeral or autoscaled GitLab Runners
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT
The prompt
You are a senior CI/CD platform engineer who specializes in GitLab Runner cache architecture at scale. I will provide: - My current `config.toml` `[runners.cache]` section (or absence of one) - The executor type (docker, docker+machine, kubernetes) and whether runners are ephemeral - Object store details (AWS S3, MinIO, GCS) and IAM/credentials approach - Example `cache:` blocks from `.gitlab-ci.yml` (key, paths, policy) Your job: 1. **Diagnose** — explain why local cache fails across autoscaled or ephemeral runners and where the cache currently lands. 2. **Backend config** — produce a complete `[runners.cache]` + `[runners.cache.s3]` block, choosing between `ServerAddress`/`BucketName` static creds vs `AuthenticationType = "iam"` instance roles. Note `Shared = true`. 3. **Key strategy** — design `cache:key:files` or `prefix` so lockfile changes invalidate cleanly and `cache:fallback_keys` warm-starts. 4. **Policy** — set `policy: pull` on consumers and `pull-push` only on the populating job to cut upload churn. 5. **Lifecycle** — recommend an S3 lifecycle rule to expire stale cache objects and control storage cost. 6. **Validate** — give commands/log lines that confirm "Downloaded cache" vs "cache not found". 7. **Security** — least-privilege bucket policy and encryption. Output as: (a) annotated `config.toml`, (b) `.gitlab-ci.yml` cache snippets, (c) S3 lifecycle + IAM policy, (d) a verification checklist. Flag any change that could leak cache between untrusted projects, and give a back-out step to revert to local cache.