GitLab CI/CD parallel:matrix needs Cross-Job Wiring Prompt
Wire needs: between parallel:matrix-expanded jobs so a downstream matrix job depends only on the matching upstream matrix instance — not all of them — using needs:parallel:matrix to keep your DAG narrow and fast across architectures, regions, or test shards.
- Target user
- Engineers building matrix DAGs that fan out and re-converge
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT
The prompt
You are a GitLab CI DAG engineer who connects parallel:matrix jobs with surgical needs: so each downstream matrix instance waits only for its matching upstream instance. I will provide: - My matrix dimensions (e.g., ARCH: [amd64, arm64], REGION: [us, eu]) and the jobs in each stage - The dependency I want (e.g., `deploy[us]` should need `build[us]` only, not `build[eu]`) - Where artifacts pass between matrix instances Your job: 1. **Matrix recap** — show the upstream job's `parallel:matrix:` so the expanded job names are unambiguous (GitLab names them by the variable values). 2. **Targeted needs (CRITICAL)** — write `needs:` using `job:` + `parallel:matrix:` so a downstream instance depends on exactly the matching upstream instance. Show how to pin one or more matrix variables in the `needs:` selector, and contrast with a naive `needs: [build]` that would wait on the entire build matrix. 3. **Artifact scoping** — confirm `artifacts: true/false` per need so each downstream instance pulls only its matching upstream's artifacts (e.g., the arm64 image manifest, not amd64). 4. **Partial fan-in** — handle the case where a downstream dimension is smaller (e.g., one `publish-manifest` job that needs ALL `build` matrix instances) and show that wiring too. 5. **Limits & footguns** — note the matrix size cap, that `needs:` has a per-job limit, and that mismatched variable names between the matrix and the `needs:` selector silently break the link (job waits on nothing or everything). 6. **Verify** — describe how the pipeline graph should look (parallel lanes that stay separate, not a fully-connected mesh) and how to confirm each instance pulled the right artifacts. Output: (a) the upstream `parallel:matrix:` block, (b) the downstream `needs:` with `parallel:matrix:` selectors, (c) the artifact-scoping flags, (d) the fan-in-all variant, (e) the footgun list, (f) the graph verify. Bias toward: instance-to-instance needs that keep lanes independent, explicit artifact scoping, and matching variable names exactly.
Why this prompt works
The whole point of a parallel:matrix is to run independent lanes — amd64 next to arm64, us next to eu — without one waiting on the other. But the moment you add a downstream stage, a naive needs: [build] collapses that independence: every deploy instance now waits on the entire build matrix, so your arm64 deploy is blocked until the amd64 build finishes even though they share nothing. That re-serializes exactly the work you parallelized. The fix is needs:parallel:matrix, which lets a downstream instance depend on only its matching upstream instance, and it’s underused mostly because the selector syntax is unfamiliar. This prompt makes the model produce the precise instance-to-instance wiring, which is the difference between independent lanes and an accidental mesh.
Artifact scoping rides along with the dependency. Once deploy[arm64] needs only build[arm64], it should pull only the arm64 artifact, not download every architecture’s output. Getting the artifacts: true/false flags right per need keeps each lane lean and avoids the subtle bug where a job grabs the wrong architecture’s binary because the artifact wiring didn’t match the dependency wiring. The prompt ties these together so they can’t drift apart.
It also covers the two shapes real pipelines need: clean instance-to-instance fan-out, and asymmetric fan-in where a single publish-manifest job legitimately needs all the build instances. Knowing when to keep lanes separate versus when to converge is the actual design skill. And because the failure mode here is silent — a mismatched variable name in the needs selector makes the job wait on nothing or everything with no error — the prompt insists on verifying the resolved DAG graph rather than trusting the YAML, which is the only way to catch the mistake before it ships a half-built release.