GitLab CI/CD GPU Runner ML Training Pipeline Prompt

You are a senior ML-platform engineer who runs GPU training and evaluation jobs in GitLab CI. You know how to tag and target GPU runners, pin CUDA images to the node driver, mount large datasets without abusing GitLab cache, verify the GPU is actually present before training, and — above all — keep GPU-hours from leaking through timeouts, retries, and idle scheduling.

I will provide:
- The training job (framework, GPU count/type needed, dataset size, expected duration)
- Runner setup (Kubernetes executor with GPU nodes / self-hosted GPU host / cloud)
- Current CI (YAML) or none
- Dataset/weights source and size
- The goal (run on GPU / control cost / speed data loading / verify GPU)

Your job:

1. **Target GPU runners deterministically**: tag GPU runners (e.g., `tags: [gpu, a100]`) and require the tag; on Kubernetes set `nvidia.com/gpu` resource requests and a node selector/toleration for the GPU pool.
2. **Pin a compatible CUDA image**: choose a `nvidia/cuda:<toolkit>-*` or framework image whose toolkit works with the node driver; add a pre-flight `nvidia-smi` that FAILS the job if no GPU is visible.
3. **Handle large data efficiently**: mount datasets/weights from a persistent volume or object store (S3/GCS) rather than GitLab cache; cache only small, reusable pip/conda layers. Stream or pre-stage big data.
4. **Enforce cost guardrails**:
   - Hard `timeout` sized to expected duration + margin.
   - `retry: 0` on GPU jobs (retrying a 4-hour train doubles the bill).
   - Run GPU jobs `when: manual` or gated to protected refs/schedules so an MR can't accidentally launch a training run.
   - Alert/observe GPU utilization; a job at 5% GPU util is wasting the hardware.
5. **Right-size resources**: request only the GPUs the job uses; set `resource_group` so concurrent expensive jobs don't stack on one node.
6. **Emit artifacts/metrics**: checkpoints to object store, metrics to the MR (JUnit/metrics report), logs of `nvidia-smi` for evidence.
7. **Clean shutdown**: ensure the job releases the GPU (no lingering processes) and scale-to-zero on the node pool where supported.

Deliverables:
- A `.gitlab-ci.yml` GPU training job with tag/resource targeting, CUDA pin, and a `nvidia-smi` pre-flight
- The cost guardrails (timeout, retry:0, gating, resource_group)
- The dataset-mount approach (volume/object store, not cache) for large data
- GPU-utilization/verification evidence steps

Mark DESTRUCTIVE or RISKY: auto-retry on GPU jobs, no timeout, MR-triggerable training, caching multi-GB data through GitLab cache, and untagged GPU jobs.

---

Training job: [framework, GPU type/count, dataset size, duration]
Runner setup: [k8s GPU nodes / self-hosted / cloud]
Current CI (YAML): [PASTE or none]
Dataset/weights source + size: [DESCRIBE]
Goal: [run on GPU / cost / data speed / verify]

Why this prompt works

GPU CI fails in two expensive ways: jobs that don’t actually get a GPU (untagged, mismatched CUDA) and jobs that get one and never let go (no timeout, auto-retry, idle scheduling). This prompt makes GPU targeting explicit and verified with a pre-flight nvidia-smi, then wraps the job in cost guardrails so a stuck training run can’t quietly burn a month’s GPU budget.

How to use it

1. Tag GPU runners and require the tag (plus k8s GPU resource requests).
2. Pin a CUDA image compatible with the node driver; verify with nvidia-smi.
3. Mount large datasets from a volume/object store, not GitLab cache.
4. Set a hard timeout, retry: 0, and gate the job so MRs can’t launch it.

Useful commands

# Pre-flight: fail fast if no GPU is visible
nvidia-smi || { echo "No GPU visible — refusing to run training on CPU"; exit 1; }
nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv

# Check CUDA toolkit vs. driver compatibility inside the image
nvcc --version
python -c "import torch; print(torch.cuda.is_available(), torch.version.cuda)"

# Stage a large dataset from object store (not GitLab cache)
aws s3 sync s3://ml-datasets/imagenet /data/imagenet --only-show-errors

GitLab CI patterns

Guarded GPU training job

train:
  image: nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04   # pin toolkit to node driver
  tags: [gpu, a100]                        # require GPU runner
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule"'   # scheduled/protected only
    - if: '$CI_COMMIT_BRANCH == "main"'
      when: manual                         # never auto-launch from an MR
  timeout: 6h                              # hard cap sized to expected duration
  retry: 0                                 # retrying a long train doubles the bill
  resource_group: gpu-a100                 # serialize expensive jobs on the pool
  before_script:
    - nvidia-smi || { echo "No GPU visible"; exit 1; }   # pre-flight
    - pip install -r requirements.txt
    - aws s3 sync s3://ml-datasets/$DATASET /data --only-show-errors
  script:
    - python train.py --data /data --epochs 30 --checkpoint-dir /ckpt
    - aws s3 sync /ckpt s3://ml-checkpoints/$CI_PIPELINE_ID   # artifacts to object store
  artifacts:
    when: always
    paths: [metrics.json]
    reports:
      metrics: metrics.json

Kubernetes executor GPU targeting

[[runners]]
  executor = "kubernetes"
  [runners.kubernetes]
    [runners.kubernetes.node_selector]
      "cloud.google.com/gke-accelerator" = "nvidia-tesla-a100"
    [[runners.kubernetes.tolerations]]
      key = "nvidia.com/gpu"
      operator = "Exists"
      effect = "NoSchedule"

train-k8s:
  tags: [k8s-gpu]
  variables:
    KUBERNETES_GPU_LIMIT: "1"              # request exactly the GPUs used
  timeout: 4h
  retry: 0
  script:
    - nvidia-smi
    - python train.py

Utilization evidence

gpu-util-evidence:
  extends: train
  after_script:
    - nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv > gpu-util.csv
  artifacts:
    paths: [gpu-util.csv]
    expire_in: 30 days

Common findings this catches

• Untagged GPU jobs landing on CPU runners and failing CUDA init, or CPU jobs idling GPU hardware.
• CUDA image toolkit mismatched with the node driver, causing runtime CUDA errors.
• No timeout / auto-retry enabled, letting a stuck train burn hours of GPU time.
• Training launchable from any MR instead of scheduled/protected refs.
• Multi-GB datasets pushed through GitLab cache, saturating cache storage.

When to escalate

• GPU pool capacity/quota — a platform/finance conversation, not a pipeline tweak.
• Persistently low GPU utilization — profile the data pipeline; the bottleneck is usually I/O, not compute.
• Multi-node distributed training — needs a scheduler (Kubeflow/Slurm/Ray) beyond a single CI job.