🚨 GitLab CI Error Guide: 'aborted: terminated' — Fix Signal-Killed Jobs

Overview

A running GitLab CI job dies mid-script when the runner process receives a termination signal from the operating system or orchestrator. The trace ends abruptly with:

ERROR: Job failed (system failure): aborted: terminated

You may also see the build-side variant just before the job stops:

WARNING: Received signal: terminated. Shutting down...
ERROR: Job failed: execution took longer than ... OR canceled

This is not your script returning a bad exit code — an external force killed the job. GitLab often marks it a system failure and may auto-retry it.

Symptoms

• A job that was progressing normally stops with aborted: terminated and no application-level error.
• The failure is marked “system failure” (a broken-plug icon), distinct from a script exit 1.
• It recurs on long-running jobs (builds, training, large test suites) but not on short ones.
• Retrying the same job sometimes succeeds, pointing at an environmental cause.
• On Kubernetes/cloud, the failure coincides with a node drain, spot/preemptible reclaim, or a pod eviction.
• dmesg/node logs show an OOM kill or the runner service restarting around the failure time.

Common Root Causes

• Spot/preemptible instance reclaimed — the cloud provider reclaimed the runner node mid-job; the runner got SIGTERM.
• Runner process restarted/upgraded — gitlab-runner was restarted (config reload, upgrade, autoscaler scale-in) while the job ran.
• Kubernetes pod evicted or node drained — resource pressure, node autoscaler scale-down, or a rolling node upgrade evicted the job pod.
• OOM at the node/cgroup level — the container or node ran out of memory; the kernel OOM-killer terminated processes (often paired with exit code 137).
• Job timeout / auto-cancel — the project/job timeout elapsed, or interruptible: true let a newer pipeline cancel this one.
• Ungraceful shutdown of DinD/services — a services: container died, taking the job with it.
• Manual or API cancellation — someone (or automation) canceled the pipeline.

Diagnostic Workflow

First confirm whether the signal came from the platform (eviction/reclaim) or a timeout. Check runner and node events:

# Emit environment context at job start so post-mortems have data
diagnostics:
  before_script:
    - echo "Runner: $CI_RUNNER_DESCRIPTION  Node: $(hostname)"
    - date -u +"job start %H:%M:%S"
  script:
    - ./long-task.sh
  after_script:
    - date -u +"job end %H:%M:%S"     # compare against timeout window

On a self-hosted/Kubernetes runner, inspect the platform for the real cause:

# Run these on the runner host / cluster, not inside the job:
# journalctl -u gitlab-runner --since '20 min ago'      # runner restart/scale-in
# kubectl get events --sort-by=.lastTimestamp | grep -i 'evict\|preempt\|drain\|OOM'
# dmesg -T | grep -i 'killed process\|out of memory'    # OOM kill

Make the job report memory pressure so OOM is distinguishable from reclaim:

mem-aware:
  script:
    - ( while true; do cat /sys/fs/cgroup/memory.current 2>/dev/null; sleep 15; done ) &
    - ./build.sh

Example Root Cause Analysis

A nightly build job on an autoscaling spot-runner fleet failed roughly one night in three with aborted: terminated, always as a system failure, and always succeeded on retry. The pipeline ran on cheap spot instances:

build:
  tags: [spot-runner]
  timeout: 90m
  script:
    - make -j"$(nproc)" release        # ~50 minute build

Cluster events at the failure times showed preempted node events — the cloud provider was reclaiming the spot instance mid-build. Because the whole ~50-minute build ran in one job with no checkpointing, a reclaim in the last 10 minutes wasted the entire run. The fixes applied:

1. Enabled retry scoped to system failures so a reclaim auto-retries on a fresh node:

build:
  tags: [spot-runner]
  timeout: 90m
  retry:
    max: 2
    when:
      - runner_system_failure
      - stuck_or_timeout_failure
  script:
    - make -j"$(nproc)" release

2. Split the long build so a reclaim loses less work, and cached intermediate artifacts so a retry resumed rather than restarting cold.
3. Routed the truly non-interruptible release build to a small pool of on-demand (non-spot) runners, keeping spot for stages that tolerate restarts.

After scoping retries to system failures and moving the critical build off spot, the nightly pipeline became reliable while keeping most of the spot cost savings.

Prevention Best Practices

• Add retry: scoped to runner_system_failure and stuck_or_timeout_failure so platform-caused terminations auto-recover instead of failing the pipeline.
• Run genuinely non-interruptible, long jobs on on-demand runners; reserve spot/preemptible for restart-tolerant stages.
• Checkpoint and cache long jobs so a mid-run kill loses minutes, not the whole run.
• Right-size memory requests/limits to avoid OOM kills; treat frequent exit-137 alongside terminated as a memory problem.
• Configure the runner’s graceful shutdown / drain timeout so an upgrade or scale-in lets in-flight jobs finish.
• Set realistic timeout values and understand interruptible: — an interruptible job will be canceled by a newer pipeline by design.
• Cordon/drain nodes gracefully during maintenance so jobs aren’t hard-killed.

Quick Command Reference

# Was it a runner restart or scale-in?
journalctl -u gitlab-runner --since '30 min ago' | grep -i 'signal\|shutdown\|terminated'

# Kubernetes eviction / preemption / OOM events
kubectl get events --sort-by=.lastTimestamp | grep -iE 'evict|preempt|drain|oomkill'

# Node-level OOM kills
dmesg -T | grep -i 'out of memory\|killed process'

# Cloud spot/preemption notices (example: AWS metadata)
curl -s http://169.254.169.254/latest/meta-data/spot/instance-action || echo "no reclaim notice"

Conclusion

aborted: terminated is an external signal, not your script failing — the runner was killed by a spot reclaim, a runner restart or scale-in, a Kubernetes eviction, an OOM kill, or a timeout/cancel. Distinguish the cause from runner and node events, then make jobs resilient: retry on system failures, checkpoint long work, right-size memory, and keep non-interruptible builds off preemptible hardware. Done well, the same platform churn that caused the failure becomes a transparent auto-retry.