GitLab CI/CD Monorepo Strategy Prompt

You are a senior platform engineer who has built monorepo CI/CD at scale — selective builds based on changed paths, child pipelines per service, shared library dependency detection, fan-out/fan-in across many services.

I will provide:
- The monorepo structure (top-level dirs per service, shared libraries, generated code)
- Service count and complexity
- Current pipeline approach (build everything, custom path filters, manual decision)
- The goal: design new / optimize existing / fix selective-build bugs

Your job:

1. **Choose the strategy**:
   - **`rules:changes:`** — declarative, simple; one job per service with `rules:changes:` for that service's paths
   - **Dynamic child pipelines** — parent detects changes via script, emits child YAML triggering only changed services
   - **Per-service triggers** — services live in subdirs each with own `.gitlab-ci.yml`; trigger via `include:project:`
2. **For `rules:changes:` approach**:
   ```yaml
   test-service-a:
     script: cd services/a && npm test
     rules:
       - changes:
           - services/a/**/*
           - shared/**/*           # also rebuild if shared changes
           - package-lock.json
   ```
   - Pros: simple, declarative
   - Cons: many jobs to define; one per service
3. **For dynamic child pipeline approach**:
   ```yaml
   generate-pipeline:
     script:
       - ./scripts/detect-changes.sh > child.yml
     artifacts: { paths: [child.yml] }

   trigger-children:
     trigger:
       include:
         - artifact: child.yml
           job: generate-pipeline
       strategy: depend
   ```
   - Pros: scales to many services; logic in code
   - Cons: complex generator; debugging harder
4. **For shared library impact**:
   - When `shared/auth-lib/` changes, every consuming service must rebuild
   - Track dependencies explicitly (e.g., metadata file per service listing libs used)
   - Or use language-specific tools: `nx affected` (Node), `bazel query` (Bazel), `pants` (Python)
5. **For fan-out + fan-in (e.g., monolith deploy after all services pass)**:
   - Use `needs:` to gate downstream on dynamic child pipeline status (`trigger:strategy: depend`)
   - Or post-stage that runs after all triggers
6. **For shared base images** in monorepo:
   - Build common base image first
   - Service builds use the base
   - Cache via registry; tag base with content hash
7. **For monorepo CI execution time**:
   - First-class path filter prevents 60-min full-build for a one-line README change
   - Parallel execution per service
   - Per-service `parallel: matrix:` for further parallelism
8. **For language-specific patterns**:
   - **Nx (Node)**: `nx affected --target=test --base=$CI_MERGE_REQUEST_DIFF_BASE_SHA`
   - **Bazel**: `bazel query` for affected targets
   - **Go workspaces**: `go list -m all` with `go.work`
   - **Java/Maven multi-module**: `mvn --also-make`

Mark DESTRUCTIVE: deploying only "changed" services when shared lib changed (silent regression), `rules:changes:` that misses indirect dependencies, child pipeline that fails-open on errors.

---

Monorepo structure: [DESCRIBE]
Service count: [N]
Current approach: [DESCRIBE]
Goal: [design / optimize / debug]

Why this prompt works

Monorepo CI is uniquely tricky: simple “build everything” doesn’t scale; “build only changed” misses transitive deps. This prompt walks the strategies and trade-offs.

How to use it

1. Start with the smallest strategy that works — rules:changes: if you have < 10 services.
2. For 10-50 services, dynamic child pipelines.
3. For 50+ services, language-specific affected-target tools.
4. Always include shared paths in change detection.

Useful commands

# Detect changed files since target branch
git diff --name-only $CI_MERGE_REQUEST_DIFF_BASE_SHA..HEAD

# Group by service (assume services/* layout)
git diff --name-only $CI_MERGE_REQUEST_DIFF_BASE_SHA..HEAD | \
    awk -F/ '/^services\// {print $2}' | sort -u

# With language-specific tools
nx affected --target=test --base=$CI_MERGE_REQUEST_DIFF_BASE_SHA --head=HEAD
bazel query 'rdeps(//..., set('$(git diff --name-only $CI_MERGE_REQUEST_DIFF_BASE_SHA | tr '\n' ' ')'))'

Patterns

Pattern A: rules:changes: per service

.service-template: &service-template
  image: node:20

test-service-a:
  <<: *service-template
  stage: test
  script:
    - cd services/a
    - npm ci && npm test
  rules:
    - changes:
        - services/a/**/*
        - shared/**/*
        - package-lock.json
        - .gitlab-ci.yml

test-service-b:
  <<: *service-template
  stage: test
  script:
    - cd services/b
    - npm ci && npm test
  rules:
    - changes:
        - services/b/**/*
        - shared/**/*
        - package-lock.json
        - .gitlab-ci.yml

# ... one per service

Pattern B: Dynamic child pipeline

stages: [generate, build]

generate-pipeline:
  stage: generate
  image: alpine
  script:
    - apk add --no-cache git jq
    - chmod +x ./scripts/detect-changes-and-emit-yml.sh
    - ./scripts/detect-changes-and-emit-yml.sh > children.yml
    - cat children.yml
  artifacts:
    paths: [children.yml]
    expire_in: 1 hour

trigger-children:
  stage: build
  needs: [generate-pipeline]
  trigger:
    include:
      - artifact: children.yml
        job: generate-pipeline
    strategy: depend

# scripts/detect-changes-and-emit-yml.sh
#!/bin/bash
set -euo pipefail

BASE="${CI_MERGE_REQUEST_DIFF_BASE_SHA:-$(git rev-parse origin/main)}"
CHANGED=$(git diff --name-only "$BASE"..HEAD)

# Detect changed services
SERVICES=$(echo "$CHANGED" | awk -F/ '/^services\// {print $2}' | sort -u)

# Detect shared lib changes (rebuilds all consumers)
if echo "$CHANGED" | grep -q '^shared/'; then
    SERVICES=$(ls services/)        # rebuild all
fi

cat <<EOF
stages: [test, build, deploy]

EOF

for s in $SERVICES; do
    cat <<EOF
test-$s:
  stage: test
  image: node:20
  script:
    - cd services/$s
    - npm ci
    - npm test

build-$s:
  stage: build
  needs: [test-$s]
  script:
    - cd services/$s
    - docker build -t \$CI_REGISTRY_IMAGE/$s:\$CI_COMMIT_SHORT_SHA .
    - docker push \$CI_REGISTRY_IMAGE/$s:\$CI_COMMIT_SHORT_SHA

EOF
done

Pattern C: Nx-style affected (Node monorepo)

test-affected:
  image: node:20
  script:
    - npm ci
    - npx nx affected --target=test --base=$CI_MERGE_REQUEST_DIFF_BASE_SHA --head=HEAD
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

build-affected:
  image: node:20
  script:
    - npm ci
    - npx nx affected --target=build --base=$CI_MERGE_REQUEST_DIFF_BASE_SHA --head=HEAD
  artifacts:
    paths: [dist/]
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

Pattern D: Per-service child via include

# Top-level .gitlab-ci.yml
include:
- local: 'services/a/.gitlab-ci.yml'
  rules:
    - changes:
        - services/a/**/*
        - shared/**/*
- local: 'services/b/.gitlab-ci.yml'
  rules:
    - changes:
        - services/b/**/*
        - shared/**/*

# services/a/.gitlab-ci.yml
test-service-a:
  script: cd services/a && npm test

Common findings this catches

• Selective build misses shared lib change → add shared/**/* to all services’ changes:.
• Trigger fans out to 50 services on every push → effective only if changes always span all; rare.
• Dynamic child generator has logic bug → emits no children for valid changes; tests skipped silently.
• rules:changes: doesn’t include .gitlab-ci.yml → CI config changes don’t retrigger tests.
• Cache shared across services but different lockfiles → cache invalidates more than needed.
• Fan-in deploy waits for stale services → only deploy changed services; track which.
• needs: graph across services creates cross-service coupling → unwanted; isolate via child pipelines.

When to escalate

• 100+ services scale issues — adopt Bazel/Nx/Pants with proper affected detection.
• CI bills exploding — audit; per-MR run cost; consider deduplicated execution.
• Shared library team owns CI for monorepo — coordinate ownership.