🔎 Using AI to Turn GitLab Pipeline Failures Into Clear Summaries

It is 11pm. The deploy pipeline is red, the on-call channel is quiet because everyone has gone to bed, and I am scrolling through four thousand lines of job log looking for the one line that actually broke. Somewhere in that wall of ANSI color codes, npm deprecation warnings, and retry chatter is a single stack frame that matters. I find it eventually — a missing environment variable, three screens up from the bottom — but I have burned twenty minutes and most of my patience getting there.

That ritual is the worst part of running CI at scale, and it is almost entirely automatable. A GitLab pipeline failure is just text, and distilling a root cause out of noisy text is exactly the kind of thing a language model is good at. In this guide I will wire up a .gitlab-ci.yml that captures the failed job log, sends it to an AI summarizer, and posts a clean one-paragraph root-cause back to the merge request — so the next person who hits a red pipeline reads a summary instead of scrolling logs.

The mental model: AI as a fast junior engineer

Before any YAML, set expectations. The AI here is a fast, tireless junior engineer. It is excellent at reading a thousand lines of log and telling you, in plain English, “the build failed because DATABASE_URL was unset during the migration step.” It is genuinely good at this — faster than you, and never bored.

What it is not is a source of truth. It will occasionally invent a plausible cause that is wrong, or confidently blame the last error line when the real failure happened earlier. So treat its summary as a lead, not a verdict. Read the one cited line it points to before you act. And the rule that matters most: never let it see your secrets. CI logs are a notorious place for tokens to leak, and you are about to pipe those logs to an API. Scrub first.

Step 1: Capture the tail of the log as an artifact

The cleanest signal usually lives in the last chunk of output. An after_script runs even when the job fails, so it is the right place to snapshot the tail of the log into an artifact that later jobs can read.

.capture-log:
  after_script:
    # The running job's own stdout isn't a file, so capture from the
    # commands we care about. Tee build output into a log we can tail.
    - tail -n 400 build.log > failure-tail.log || true
  artifacts:
    when: on_failure
    paths:
      - failure-tail.log
    expire_in: 3 days

build:
  stage: build
  extends: .capture-log
  script:
    - npm ci 2>&1 | tee build.log
    - npm run build 2>&1 | tee -a build.log

The when: on_failure on the artifact means we only pay the storage cost when something actually breaks. Four hundred lines is usually plenty — enough context for the model, small enough to stay cheap.

Step 2: A .post stage job that runs only on failure

GitLab ships a built-in .post stage that always runs last. Put a summarizer job there, gated with rules so it only fires when the pipeline failed.

ai-summary:
  stage: .post
  image: alpine:3.20
  rules:
    - when: on_failure
  variables:
    GIT_STRATEGY: none
  before_script:
    - apk add --no-cache curl jq
  script:
    - echo "Summarizing failed pipeline $CI_PIPELINE_ID"
    - ./ci/summarize-failure.sh

when: on_failure at the rules level is the GitLab-native way to say “only run this job if an earlier job in the pipeline failed.” GIT_STRATEGY: none skips the checkout — this job does not need your source tree, just the API and the log.

Step 3: Pull the failed job log via the GitLab API

If you would rather not rely on artifacts, you can fetch the raw trace of any failed job directly from the GitLab API using the job-scoped CI_JOB_TOKEN. List the pipeline’s jobs, find the failed one, and grab its trace.

#!/usr/bin/env sh
set -eu

API="$CI_API_V4_URL/projects/$CI_PROJECT_ID"
AUTH="JOB-TOKEN: $CI_JOB_TOKEN"

# Find the first failed job in this pipeline.
FAILED_ID=$(curl -sf --header "$AUTH" \
  "$API/pipelines/$CI_PIPELINE_ID/jobs?scope=failed&per_page=1" \
  | jq -r '.[0].id')

# Pull its raw trace, keep the last 400 lines.
curl -sf --header "$AUTH" \
  "$API/jobs/$FAILED_ID/trace" | tail -n 400 > raw-trace.log

Pro Tip: CI_JOB_TOKEN can read traces in the same project without you minting a personal access token, and it expires the moment the job ends. Prefer it over a long-lived PAT whenever the job only needs to read within its own project.

Step 4: Scrub secrets, then summarize

This is the non-negotiable step. Before a single byte leaves your runner, redact anything that looks like a token. GitLab masks variables in the UI, but masking is best-effort and does not cover secrets your app prints itself. Strip them.

# Mask common secret shapes before sending anywhere.
sed -E \
  -e 's/(gl(pat|cbt|ptt)-)[A-Za-z0-9_-]+/\1REDACTED/g' \
  -e 's/(AKIA)[0-9A-Z]{16}/\1REDACTED/g' \
  -e 's/(ghp_)[A-Za-z0-9]+/\1REDACTED/g' \
  -e 's/(eyJ[A-Za-z0-9_-]+)\.[A-Za-z0-9_.-]+/JWT-REDACTED/g' \
  raw-trace.log > clean-trace.log

PROMPT='You are a senior CI engineer. Read this failed GitLab job log
and reply in ONE short paragraph: the single root cause, the exact line
or step that proves it, and the most likely fix. Do not speculate beyond
the log. If the cause is unclear, say so.'

SUMMARY=$(curl -sf https://your-ai-endpoint/summarize \
  --header "Authorization: Bearer $AI_API_KEY" \
  --header "Content-Type: application/json" \
  --data "$(jq -n --arg p "$PROMPT" --rawfile log clean-trace.log \
            '{system: $p, log: $log}')" \
  | jq -r '.summary')

Define AI_API_KEY as a masked, protected CI/CD variable in your project settings, never inline in the YAML. Masked means GitLab redacts it from job logs; protected means it is only exposed on protected branches. Both matter when the thing reading your pipeline is itself an automated job.

Step 5: Post the summary as a merge request note

A summary nobody reads is useless. Push it where the failure already has eyes — the merge request. The Notes API attaches a comment to the MR for the current branch.

# Resolve the MR for this branch (if any).
MR_IID=$(curl -sf --header "$AUTH" \
  "$API/merge_requests?source_branch=$CI_COMMIT_REF_NAME&state=opened" \
  | jq -r '.[0].iid // empty')

if [ -n "$MR_IID" ]; then
  BODY=$(printf '### 🔎 AI failure summary\n\n%s\n\n_Auto-generated. Verify before acting — do not trust blindly._' "$SUMMARY")
  curl -sf --request POST --header "$AUTH" \
    --data-urlencode "body=$BODY" \
    "$API/merge_requests/$MR_IID/notes"
fi

The same SUMMARY string can fan out to Slack or your incident tooling with another curl. The signpost in the comment body — “verify before acting” — is there on purpose. It keeps readers in the right mindset.

What it actually produces

Here is the raw tail of a failed job — the kind of thing you would otherwise scroll through:

$ npm run db:migrate
> migrate up
[14:02:11] connecting to database...
npm warn deprecated inflight@1.0.6: This module is not supported
npm warn deprecated rimraf@2.7.1: Rimraf versions prior to v4
[14:02:12] ERROR: getaddrinfo ENOTFOUND ${DB_HOST}
    at GetAddrInfoReqWrap.onlookup (node:dns:107:26)
[14:02:12] retry 1/3...
[14:02:14] ERROR: getaddrinfo ENOTFOUND ${DB_HOST}
[14:02:16] retry 2/3...
[14:02:18] ERROR: getaddrinfo ENOTFOUND ${DB_HOST}
Cleaning up project directory and file based variables
ERROR: Job failed: exit code 1

And the AI summary that lands on the MR:

The migration step failed because DB_HOST was never substituted — the log shows the literal string ${DB_HOST} in the DNS lookup, so the variable was empty at runtime. The npm deprecation warnings and the three retries are noise. Most likely fix: define DB_HOST as a CI/CD variable for this environment, or check that the environment: block on the migrate job is set.

That is the twenty-minute scroll compressed into four sentences, with the deprecation warnings correctly dismissed as noise. But notice the model made a claim — “the variable was empty.” Before you go define a variable, glance at the cited line yourself. Ninety percent of the time it is right; the verification costs you ten seconds and saves you from chasing a confident wrong answer.

Pro Tip: Ask the model to name the specific line or step it based its conclusion on, like in the prompt above. A summary you can trace back to evidence is a summary you can verify in seconds instead of re-reading the whole log.

Picking the model and prompt

You do not need a frontier model for this; log summarization is well within reach of smaller, cheaper options, and a self-hosted model like Gemma keeps the logs on your own infrastructure if data residency is a concern. If you would rather iterate on the prompt interactively before baking it into CI, paste a sample log into ChatGPT or Claude and tune the wording until the output is consistently tight. Reusable prompt scaffolds for exactly this kind of log-triage live in our prompts library and the curated prompt packs.

If this pattern earns its keep, it generalizes. The same capture-scrub-summarize loop powers the incident-response dashboard, where the AI triages a failure and drafts the first update while a human confirms the root cause. And the broader GitLab CI/CD guides cover the surrounding pipeline patterns this job plugs into.

Conclusion

Red pipelines are not going away, but the 11pm scroll-and-squint can. Capture the tail of the log, scrub the secrets, hand it to a fast junior engineer who turns four thousand lines into one honest paragraph, and post that paragraph where the failure already lives. Just remember what the junior is: quick and clear, occasionally wrong, and never — ever — trusted with a real secret. Scrub first, verify second, and let the summary do the scrolling for you.