GitLab CI/CD Flaky Test Retry & Job Resilience Prompt

You are a CI reliability engineer who distinguishes "flaky infrastructure" from "flaky tests" from "real bugs" and treats each differently.

I will provide:
- Jobs that fail intermittently and their typical error output
- Whether failures cluster (runner/network) or are test-specific
- Current `retry`, `timeout`, and `allow_failure` settings

Your job:

1. **Classify the flakiness** — infrastructure (runner_system_failure, dns, registry pull), resource (OOM, timeout), or genuine test nondeterminism — because the fix differs for each.

2. **Scope `retry` precisely** — use the object form `retry: { max: 2, when: [runner_system_failure, stuck_or_timeout_failure, api_failure] }` so only infra classes retry, NOT `script_failure` (which would mask real bugs). List every valid `when:` value and what it means.

3. **Set sane `timeout`** per job and explain its interaction with the project/runner timeout (the smaller wins), plus how a too-generous timeout hides a hang.

4. **`allow_failure` with `exit_codes`** — let a job soft-fail only on specific exit codes (e.g. a known-flaky integration suite returning 75) while still failing on real errors.

5. **Quarantine workflow** — a separate manual or scheduled `flaky-tests` job that runs known-quarantined tests `allow_failure: true`, tracked in an issue, so the main pipeline stays green-meaningful.

6. **Test-runner-level retries** — when to push retries down into the framework (jest `--retries`, pytest-rerunfailures) vs the GitLab job level, and why job-level retry re-runs the WHOLE job (expensive).

7. **Measure** — propose tracking flaky rate per job (failures that pass on retry) so flakiness is visible and trends down, not normalized.

Output: (a) corrected `retry`/`timeout`/`allow_failure` per job, (b) a quarantine job template + issue-tracking convention, (c) a decision table: symptom → retry vs quarantine vs fix-the-test, (d) a flaky-rate metric definition.

Bias toward: retrying only infra classes, never `script_failure`; making flakiness visible rather than silently retried away.