🚨 GitLab CI Error Guide: 'smudge filter lfs failed' — Fix Git LFS Checkout

Overview

A GitLab CI job fails during checkout when Git tries to replace an LFS pointer with the real object and the LFS download fails. Git aborts the checkout with:

Downloading assets/model.bin (412 MB)
Error downloading object: assets/model.bin (a1b2c3): Smudge error: Error downloading
assets/model.bin: batch response: This repository is over its data quota.

error: external filter 'git-lfs filter-process' failed
fatal: assets/model.bin: smudge filter lfs failed

The missing-binary variant appears when git-lfs isn’t installed in the job image at all:

git: 'lfs' is not a git command. See 'git --help'.
error: external filter 'git-lfs filter-process' failed

The clone/checkout never completes, so the job stops before running any of your script.

Symptoms

• Checkout fails with smudge filter lfs failed and often a preceding Error downloading object.
• The job image is a minimal image (alpine/scratch-based) without git-lfs.
• The error mentions data quota, batch response, 401, or 403 — an auth or quota problem.
• Files that should be large binaries are instead 130-byte pointer text files after checkout.
• The same repo checks out fine locally (where git-lfs is installed and auth is cached).

Common Root Causes

• git-lfs not installed in the image — the custom image: lacks the git-lfs binary, so the smudge filter can’t run.
• LFS bandwidth/storage quota exceeded — the namespace is over its LFS data quota; the server refuses the download.
• LFS auth failure — the job token or credentials can’t authenticate to the LFS endpoint (401/403), often on forks or with restricted tokens.
• Network/proxy blocking the LFS endpoint — the LFS object server is unreachable (firewall, proxy, self-signed cert).
• Corrupt or missing object — the referenced LFS object isn’t present on the server (bad push, cleanup).
• Smudge attempted when unnecessary — a job that only needs source is forced to download gigabytes of LFS at checkout and fails on one object.

Diagnostic Workflow

First confirm whether git-lfs even exists in the job image, then check what LFS is doing:

lfs-debug:
  image: alpine:3.20
  before_script:
    - apk add --no-cache git git-lfs
    - git lfs version                     # prove the binary is present
    - git lfs install --skip-repo
  script:
    - git lfs env                         # shows endpoint + credentials source
    - GIT_TRACE=1 GIT_TRANSFER_TRACE=1 git lfs pull --include="assets/**" 2>&1 | tail -40

To isolate a single job from LFS while you fix auth/quota, skip smudge and pull only what’s needed:

variables:
  GIT_LFS_SKIP_SMUDGE: "1"        # checkout pulls pointers only — never fails on smudge

build:
  before_script:
    - git lfs install --skip-repo
    - git lfs pull --include="assets/models/**" --exclude=""   # fetch only this job's objects
  script:
    - ./build.sh

Distinguish quota/auth from network by reading the batch response:

lfs-check:
  script:
    - git lfs install --skip-repo
    - git lfs pull 2>&1 | grep -iE 'quota|401|403|batch response|could not resolve' || echo "no auth/quota error"

Example Root Cause Analysis

A pipeline that had worked for months began failing every job at checkout with smudge filter lfs failed, and the batch response said This repository is over its data quota. Investigation showed each of the pipeline’s six jobs cloned the repo and smudged the full LFS set — hundreds of megabytes — on every run:

# Every job used the default checkout, smudging ALL LFS objects
test:      { script: [ "./test.sh" ] }
lint:      { script: [ "./lint.sh" ] }
build:     { script: [ "./build.sh" ] }   # only THIS one needs the binaries
package:   { script: [ "./package.sh" ] }

Six jobs × full LFS download per pipeline, across dozens of pipelines a day, had exhausted the namespace’s monthly LFS bandwidth quota — after which the server refused all downloads and every smudge failed. The fix removed unnecessary LFS traffic:

variables:
  GIT_LFS_SKIP_SMUDGE: "1"        # global: jobs get pointers, not binaries

# Jobs that only need source do nothing special — pointers are fine
test:  { script: [ "./test.sh" ] }
lint:  { script: [ "./lint.sh" ] }

# Only the job that needs binaries pulls them, scoped to what it uses
build:
  before_script:
    - git lfs install --skip-repo
    - git lfs pull --include="assets/**" --exclude=""
  script:
    - ./build.sh

Cutting six full downloads per pipeline to one scoped download brought bandwidth back under quota, and the smudge failures stopped. The team also raised the quota as a longer-term measure and cached .git/lfs/objects so repeated pipelines reused fetched assets.

Prevention Best Practices

• Ensure every image that checks out LFS content ships git-lfs (the GitLab helper image does; minimal custom images may not) and run git lfs install --skip-repo.
• Set GIT_LFS_SKIP_SMUDGE: "1" globally and git lfs pull --include=... only in the jobs that actually need binaries — most jobs are fine with pointers.
• Scope every git lfs pull with --include/--exclude so a job downloads only its objects, not the whole repo.
• Cache .git/lfs/objects keyed on .gitattributes so unchanged assets are fetched once, not per pipeline.
• Monitor LFS storage/bandwidth quota and alert before exhaustion; a full quota fails every smudge.
• On fork MRs, verify the LFS endpoint is reachable with the available token; restrict LFS-dependent jobs to protected refs if fork tokens can’t authenticate.

Quick Command Reference

# Is git-lfs installed in the image?
git lfs version || (apk add --no-cache git-lfs || apt-get install -y git-lfs)
git lfs install --skip-repo

# Show LFS endpoint and credential source
git lfs env

# Skip smudge, then fetch only what a job needs
export GIT_LFS_SKIP_SMUDGE=1
git lfs pull --include="assets/**" --exclude=""

# Trace an LFS download to see quota/auth/network cause
GIT_TRACE=1 GIT_TRANSFER_TRACE=1 git lfs pull 2>&1 | tail -40

Conclusion

smudge filter lfs failed means Git could not turn an LFS pointer into its real object at checkout — because git-lfs isn’t in the image, the namespace is over its LFS quota, auth to the LFS endpoint failed, or the object is unreachable. Install git-lfs, skip smudge globally and git lfs pull --include only where binaries are needed, scope and cache those fetches to protect quota, and read the batch response to tell quota from auth from network. Most jobs never need the binaries at all — let them keep the pointers.