🧬 Dynamic Child Pipelines in GitLab: Generating YAML on the Fly

There’s a class of pipeline that a static .gitlab-ci.yml simply cannot express cleanly. A monorepo where the set of services changes weekly. A matrix whose dimensions come from a config file. A “deploy every region in this manifest” job where the regions aren’t known until the manifest is read. You can fake these with grotesque rules and copy-pasted job blocks, or you can do what GitLab actually built for this: generate the pipeline YAML at runtime and hand it back as a child pipeline.

I reached for this the third time I found myself maintaining forty near-identical job definitions by hand. It’s a sharp tool — powerful and easy to cut yourself on. Here’s how to wield it.

How dynamic child pipelines work

The mechanism is two stages. A generate job runs a script that writes a valid .gitlab-ci.yml and saves it as an artifact. A trigger job then runs that artifact as a child pipeline:

stages: [generate, run]

generate-config:
  stage: generate
  image: python:3.12-slim
  script:
    - python scripts/generate_pipeline.py > generated-config.yml
  artifacts:
    paths:
      - generated-config.yml

run-child:
  stage: run
  trigger:
    include:
      - artifact: generated-config.yml
        job: generate-config
    strategy: depend

The trigger.include.artifact pulls the generated file from the generate-config job, and GitLab runs it as a full pipeline. strategy: depend again ties the parent’s status to the child’s, so a failure in a generated job turns the whole thing red.

Generate from a manifest, not from cleverness

The best dynamic pipelines are boring: they read a declarative source of truth and emit jobs mechanically. The worst ones embed business logic in a thousand-line Bash heredoc that nobody can debug.

Keep a manifest — a YAML or JSON file checked into the repo — that lists the things to act on:

# services.yml
services:
  - name: auth
    path: services/auth
  - name: billing
    path: services/billing
  - name: gateway
    path: services/gateway

Then your generator is a dumb template expander:

import yaml

manifest = yaml.safe_load(open("services.yml"))
jobs = {"stages": ["test", "build"]}

for svc in manifest["services"]:
    name, path = svc["name"], svc["path"]
    jobs[f"test:{name}"] = {
        "stage": "test",
        "script": [f"make test DIR={path}"],
        "rules": [{"changes": [f"{path}/**/*"]}],
    }
    jobs[f"build:{name}"] = {
        "stage": "build",
        "needs": [f"test:{name}"],
        "script": [f"make build DIR={path}"],
        "rules": [{"changes": [f"{path}/**/*"]}],
    }

print(yaml.dump(jobs))

Anyone can read that. The pipeline’s shape is a pure function of the manifest, which means a code review of a service addition is a one-line manifest change, not a forty-line YAML diff.

The monorepo “only build what changed” pattern

This is the killer application. In a monorepo, you don’t want to test every service on every push — you want to test the ones that changed. Generate jobs conditionally based on the diff:

import subprocess, yaml

changed = subprocess.check_output(
    ["git", "diff", "--name-only", "origin/main...HEAD"]
).decode().splitlines()

manifest = yaml.safe_load(open("services.yml"))
jobs = {"stages": ["test"]}

for svc in manifest["services"]:
    if any(f.startswith(svc["path"]) for f in changed):
        jobs[f"test:{svc['name']}"] = {
            "stage": "test",
            "script": [f"make test DIR={svc['path']}"],
        }

# Empty pipeline guard
if len(jobs) == 1:
    jobs["noop"] = {"stage": "test", "script": ["echo nothing changed"]}

print(yaml.dump(jobs))

Note the empty-pipeline guard. A child pipeline with zero jobs is an error in some configurations and a silent no-op in others — either way, add a noop job so the merge request always has a green check to require.

Validate before you trigger

The scariest failure mode is a generator that emits invalid YAML, because the failure surfaces in the child pipeline as a cryptic parse error far from the bug. Validate the generated config in the generate job before it ever becomes an artifact:

generate-config:
  stage: generate
  script:
    - python scripts/generate_pipeline.py > generated-config.yml
    - python -c "import yaml,sys; yaml.safe_load(open('generated-config.yml'))"
  artifacts:
    paths: [generated-config.yml]

If you’re on a tier that supports it, GitLab’s CI Lint API can validate the generated YAML against the full schema, catching invalid keywords the YAML parser would happily accept. Wiring that into the generate job turns a confusing downstream failure into an obvious upstream one.

Keep the generator output as an artifact you can read

When a dynamic pipeline misbehaves, the first question is “what did it actually generate?” Because the config is an artifact, you can download generated-config.yml from the generate job and read exactly what ran. Always keep that artifact for a reasonable retention window — it’s your only window into what the pipeline was, and debugging blind is miserable.

When NOT to use this

Dynamic child pipelines add a layer of indirection. Don’t reach for them when:

• A static pipeline with rules:changes and a parallel:matrix already does the job. Generated YAML is overkill for a fixed three-service repo.
• The generation logic is so complex it needs its own tests. If your generator is a small application, that’s a smell — simplify the manifest.
• You can’t articulate why the job set isn’t knowable ahead of time. “Knowable but tedious” is a job for matrices and includes, not codegen.

Where to go from here

Dynamic child pipelines turn an unmaintainable wall of repeated YAML into a small generator reading a declarative manifest. Keep the generator boring, validate its output before triggering, guard against empty pipelines, and always retain the generated config as a readable artifact. Used with that discipline, it’s one of the most powerful patterns in GitLab CI.

For more on scaling pipelines and monorepo CI, see our GitLab CI/CD guides. And when reviewing a generator script before it ships, our AI code review assistant is good at spotting the edge cases that produce malformed YAML.

Generated pipeline behavior depends on your GitLab version and tier. Validate generated config against your own setup before relying on it.