🚨 GitLab CI Error Guide: 'image pull failed: ImagePullBackOff' Kubernetes Executor

Exact Error Message

A job on the Kubernetes executor never starts; the build pod is stuck pulling its image:

Preparing the "kubernetes" executor
Using Kubernetes namespace: gitlab-runner
ERROR: Job failed (system failure): prepare environment: waiting for pod running:
  pull image "registry.example.com/team/builder:1.4": failed to pull and unpack image:
  failed to resolve reference: pulling from host registry.example.com failed:
  401 Unauthorized
ERROR: Job failed (system failure): image pull failed: rpc error

kubectl describe pod on the build pod shows the underlying Kubernetes condition:

  Warning  Failed     kubelet  Failed to pull image "registry.example.com/team/builder:1.4":
                               401 Unauthorized
  Warning  Failed     kubelet  Error: ErrImagePull
  Normal   BackOff    kubelet  Back-off pulling image "registry.example.com/team/builder:1.4"
  Warning  Failed     kubelet  Error: ImagePullBackOff

What the Error Means

With the Kubernetes executor, each job runs in a pod whose containers use your image: and services:. If the kubelet can’t pull one of those images, the pod sits in ErrImagePull and then ImagePullBackOff (exponential back-off between retries). The runner waits for the pod to become Running, times out, and fails the job as a system failure — your script never runs.

The cause is on the pull side: wrong image name/tag, a private registry with no credentials available to the pod, a registry rate limit, or the node being unable to reach the registry. It is distinct from a pod that schedules fine but exceeds the start timeout.

Common Causes

1. Private registry without imagePullSecrets — the pod has no credentials to authenticate.
2. Wrong image name or tag (:1.4 doesn’t exist, typo in the path) → manifest unknown / not found.
3. Docker Hub / registry rate limiting the node’s anonymous pulls → 429 Too Many Requests.
4. Node can’t reach the registry (network policy, proxy, DNS) → dial tcp ... i/o timeout.
5. imagePullSecrets in the wrong namespace or referencing a malformed .dockerconfigjson secret.

How to Reproduce the Error

Point a job at a private image without giving the runner pull credentials:

build:
  image: registry.example.com/team/builder:1.4
  script:
    - echo "never reached"

With no imagePullSecrets configured for the runner’s pods:

ERROR: Job failed (system failure): prepare environment: waiting for pod running:
  pull image ... 401 Unauthorized

Diagnostic Commands

Read-only Kubernetes inspection in the runner namespace:

# Find the stuck build pod and its phase
kubectl -n gitlab-runner get pods | grep runner-

# The decisive evidence: events at the bottom of describe
kubectl -n gitlab-runner describe pod <build-pod> | tail -30

# Confirm an imagePullSecret exists and is the right type
kubectl -n gitlab-runner get secret
kubectl -n gitlab-runner get secret registry-cred -o jsonpath='{.type}'

# Does the runner config reference the secret?
grep -n -A2 image_pull_secrets /etc/gitlab-runner/config.toml

runner-abc-project-42-... 0/2 ImagePullBackOff
  Warning Failed kubelet ... 401 Unauthorized
kubernetes.io/dockerconfigjson

A 401/manifest unknown in the events plus a missing or wrong-namespace pull secret confirms the cause.

Step-by-Step Resolution

1. Create a registry pull secret in the runner namespace

kubectl -n gitlab-runner create secret docker-registry registry-cred \
  --docker-server=registry.example.com \
  --docker-username="$REG_USER" \
  --docker-password="$REG_TOKEN"

The secret must live in the same namespace the runner schedules pods into.

2. Reference the secret from the runner config

In /etc/gitlab-runner/config.toml (or the Helm values.yaml), tell the executor to attach it:

[[runners]]
  executor = "kubernetes"
  [runners.kubernetes]
    namespace = "gitlab-runner"
    image_pull_secrets = ["registry-cred"]

Restart the runner so new pods get the secret.

3. Verify the image name and tag

Confirm the reference actually exists (a manifest unknown event means it doesn’t):

# Read-only: list tags the way the kubelet would resolve them
crane ls registry.example.com/team/builder 2>/dev/null | grep 1.4

Fix typos in the registry host, project path, or tag. Avoid latest for reproducibility.

4. Handle rate limits and node networking

For 429 Too Many Requests from Docker Hub, authenticate node pulls or mirror images into your own registry. For dial tcp ... timeout, check the node’s egress, proxy, and any NetworkPolicy blocking the registry.

5. Re-run and watch the pod reach Running

After fixing credentials/name, re-run the job and confirm:

kubectl -n gitlab-runner get pods -w | grep runner-

The pod should move from ContainerCreating to Running without ImagePullBackOff.

Prevention and Best Practices

• Provision image_pull_secrets for every private image a pipeline can reference, in the runner’s namespace.
• Pin specific image tags (never latest) so a mistyped or moved tag fails loudly at review, not in CI.
• Mirror frequently pulled public images into your own registry to dodge Docker Hub rate limits on nodes.
• Keep the pull secret’s .dockerconfigjson current; rotate tokens before they expire to avoid sudden 401s.
• Pasting kubectl describe pod output into the free incident assistant tells you whether it’s auth, a bad name, or a rate limit. More patterns live in the GitLab CI/CD guides.

Related Errors

• ERROR: Job failed: timed out waiting for pod to start — the pod schedules but never becomes ready for other reasons (resources, scheduling); ImagePullBackOff is one specific cause of that timeout.
• ERROR: Preparation failed: failed to pull image — the Docker executor analogue of an image-pull failure.
• 401 Unauthorized from the registry — the auth-specific root cause behind many ImagePullBackOff events.

Frequently Asked Questions

Why is this a system failure, not a normal job failure?

The pull fails during pod preparation, before your script runs. The runner can’t start the build container, so it reports a system failure. Retrying without fixing credentials or the image name will fail identically.

Where exactly does the pull secret need to live?

In the namespace the runner schedules build pods into (commonly gitlab-runner), and it must be listed under image_pull_secrets in the runner config. A secret in the wrong namespace is invisible to the pod and the pull still fails.

How do I tell auth from a missing image?

Read the kubelet events in kubectl describe pod. 401 Unauthorized / no basic auth credentials means missing or wrong credentials; manifest unknown / not found means the image name or tag doesn’t exist. They need different fixes.

My pods get 429 Too Many Requests. What now?

The node is hitting Docker Hub’s anonymous pull limit. Authenticate the node’s pulls with a pull secret, or mirror the images into your own registry so CI pulls don’t depend on Docker Hub’s shared limits.