🕸️ Optimizing GitLab Pipeline DAGs with needs: Using AI

The default GitLab pipeline model is a sequence of stages: every job in build must finish before any job in test starts, even if a particular test job only depends on one build artifact. On a pipeline with a slow integration build sitting in the build stage, your fast unit tests sit there twiddling their thumbs for ten minutes waiting for something they don’t even need. That’s wasted wall-clock time on every single run, multiplied across your whole team.

The fix is the needs keyword, which turns the pipeline into a directed acyclic graph (DAG): a job starts the moment its specific dependencies finish, ignoring stage boundaries. Figuring out the right needs graph by hand on a 40-job pipeline is genuinely hard — you have to reason about which artifacts each job actually consumes. This is where AI earns its keep, as long as you verify the graph it produces.

Why stages alone leave time on the table

Here’s the classic slow shape:

stages:
  - build
  - test
  - deploy

build_backend:
  stage: build
  script: ["make backend"]

build_frontend:
  stage: build
  script: ["npm run build"]

unit_backend:
  stage: test
  script: ["make test-backend"]

e2e:
  stage: test
  script: ["npm run e2e"]

unit_backend only needs build_backend, but because it’s in the test stage it waits for build_frontend too. If the frontend build is the slow one, every backend test run pays for it. Multiply that across dozens of pipelines a day and it’s hours of engineer waiting time.

Have AI infer the real dependency graph

I paste the pipeline into Claude with a precise ask: “For each job, list which other jobs produce an artifact it consumes, based on the artifacts:paths it produces and the files its script reads. Then propose a needs: clause for each job.” The key is anchoring the model on artifacts, not stages — the whole point is to break free of stage ordering.

The model is good at the mechanical part: matching artifacts: paths: ["dist/"] in one job to a script that reads dist/ in another. Where it gets sloppy is implicit dependencies — a job that needs a database migration to have run, or a login step, without any artifact passing between them. Those don’t show up in the YAML, so the model can’t see them. That’s the gap I fill as the human in the loop.

The refactored version:

unit_backend:
  stage: test
  needs: ["build_backend"]
  script: ["make test-backend"]

e2e:
  stage: test
  needs: ["build_backend", "build_frontend"]
  script: ["npm run e2e"]

Now unit_backend starts the instant build_backend is done, regardless of the frontend.

Make artifact dependencies explicit

needs does double duty: it controls ordering and which artifacts get downloaded. By default a needs job pulls artifacts from all jobs it needs. If you want the ordering dependency without the download, use the expanded form:

deploy:
  needs:
    - job: build_backend
      artifacts: true
    - job: lint
      artifacts: false

This matters for speed: pulling a 500MB artifact you don’t use just to enforce ordering is wasteful. I ask the model to mark artifacts: false on any needs entry that’s purely about ordering. It won’t volunteer this — you have to prompt for it — but it’s an easy win once you do.

Pro Tip: A job with needs: [] (empty array) starts immediately at pipeline creation, ignoring all stages. It’s perfect for fast linters or a “post a started message to Slack” job. Ask AI to identify which of your jobs have zero real dependencies and could run on needs: [] — it’s often more than you’d guess, and those jobs finishing first gives developers faster signal.

Watch for the cross-stage download trap

needs lets a job depend on a job in a later stage’s position in the file, but a job can only need jobs that run before it in the DAG — you can’t create a cycle, and you can’t needs a job that GitLab would schedule after it. The model occasionally proposes a graph with a subtle cycle (A needs B, B needs C, C needs A through some chain). GitLab will reject this at lint time with the job ... needs ... but it does not exist, but the error message points at one edge, not the cycle, so it’s annoying to debug.

When I get a proposed DAG back, I ask the model a follow-up: “Trace this graph and confirm there are no cycles, listing the topological order.” Making it show its work catches most cycle bugs before I ever push. I still validate in the pipeline editor, which renders the DAG visually — a picture of the graph is the fastest way to spot a missing or wrong edge.

Cap parallelism so you don’t starve the runners

A fully-parallelized DAG can try to start 30 jobs at once. If your runner fleet only has 10 concurrent slots, the other 20 queue anyway — and worse, a wide DAG can saturate runners that other teams share. There’s a real tension between “start everything ASAP” and “don’t melt the shared runner pool.”

I have the AI flag the maximum width of the DAG (how many jobs could run simultaneously) and compare it to my known runner concurrency. If the graph is wider than my capacity, parallelizing further is theater — the jobs just queue. Sometimes the right move is to keep a couple of stage boundaries to throttle. The model can do this math; it just needs you to tell it the runner count, because it has no way to know your infrastructure. If runner saturation is a recurring pain, the Kubernetes executor tuning guide covers scaling the fleet to match a wide DAG.

Measure before and after — don’t trust vibes

A DAG refactor feels faster, but feel is not data. Before merging, I note the pipeline’s wall-clock duration from a few recent runs, apply the change on a branch, and compare. GitLab’s pipeline duration and the per-job timing chart tell you whether the critical path actually got shorter. Sometimes the DAG looks beautiful but the critical path (the longest dependency chain) didn’t change at all, so total time is identical — you just rearranged the queue.

I ask the AI to identify the critical path explicitly: “Given these job durations, what’s the longest dependency chain in the proposed DAG?” That number is your real floor. No amount of parallelism beats the critical path, so if you want a faster pipeline, that’s the chain to attack — usually by splitting the slowest job or caching its inputs.

Keep the human in the loop

To be clear about the division of labor: the AI is a fast junior engineer for graph-building. It reads the YAML, matches artifacts to consumers, and drafts a needs graph far faster than I would. What it cannot do is know your hidden runtime dependencies, your runner capacity, or whether a “behavior-preserving” graph actually preserved behavior. So every proposed DAG gets the same treatment: validate in the pipeline editor, confirm no cycles, check the critical path, and run it on a throwaway branch before it touches main. And as always, no real CI variables or tokens go into the chat — the DAG structure is all the model needs.

Conclusion

The needs keyword is one of the biggest free speedups in GitLab CI, and the only reason people don’t use it more is that building the graph by hand is tedious. Hand the tedium to AI: let it infer the artifact graph and draft the needs clauses, then verify the cycles, the critical path, and the runner math yourself. Review before merge, measure before and after, and you’ll cut real minutes off every run. Browse more in the GitLab CI/CD category, or grab ready-made DAG prompts from the prompt packs.