🚨 GitLab CI Error Guide: 'shell executor job failed' — Fix Runner Host Setup

Overview

When a GitLab Runner is configured with the shell executor, jobs run as commands directly on the runner host — there is no container isolation. That makes shell-executor failures highly dependent on the state of the host, and the job log usually ends with a bare non-zero exit code rather than a descriptive message:

$ terraform --version
/bin/bash: line 42: terraform: command not found
Cleaning up project directory and file based variables
ERROR: Job failed: exit status 127

exit status 127 means “command not found”; exit status 126 means “found but not executable” (a permissions problem); any other non-zero code comes straight from a script command that failed. Because the shell executor inherits the gitlab-runner user’s environment — not your interactive shell — a job that works when you SSH in can still fail in CI.

Symptoms

• A job passes on your laptop or in a Docker-executor pipeline but fails on a specific shell runner.
• The log ends with ERROR: Job failed: exit status 127 (or 126, or 1).
• command not found for a tool that is clearly installed on the host.
• “Permission denied” writing to the build directory, the cache, or a system path.
• The job hangs or exits immediately after the Preparing environment or Getting source stage.

Common Root Causes

• A tool is missing from the gitlab-runner user’s PATH, even though it exists for your login shell (installed under /usr/local/bin or a version manager the runner never sources).
• Wrong file or directory permissions — the gitlab-runner user cannot write to the build directory or read a needed file.
• A sudo command with no passwordless rule, which blocks in a non-interactive shell.
• .bash_logout or a shell profile interfering — the shell executor sources login-shell files, and a clear, exit, or interactive-only command there can break jobs.
• An unclean runner host — leftover files from a previous job, a full disk, or a stale lock in the build directory.
• The runner is unhealthy — a stale registration or a mismatched runner/GitLab version.

Diagnostic Workflow

First read the exit code at the very bottom of the job log — it categorises the whole failure. 127 = not found, 126 = not executable, anything else = a real command failure.

Reproduce the environment the runner actually uses. The shell executor runs as the gitlab-runner user, so become that user with a login shell and check PATH and tool availability the way CI sees them:

sudo -iu gitlab-runner            # login shell as the runner user
echo "$PATH"                      # is /usr/local/bin present?
which terraform || echo "not on runner PATH"
env | sort                        # compare against your own shell's env

Verify the runner itself is registered and healthy, and inspect its config to confirm the executor and any environment overrides:

sudo gitlab-runner verify         # checks each registered runner is reachable
sudo gitlab-runner list           # shows executor type per runner
sudo cat /etc/gitlab-runner/config.toml   # executor = "shell", environment = [...]

Check permissions and disk on the build directory the runner writes into (default ~gitlab-runner/builds):

sudo ls -ld /home/gitlab-runner/builds
sudo du -sh /home/gitlab-runner/builds/* 2>/dev/null | sort -rh | head
df -h /home/gitlab-runner         # a full disk fails jobs at the source stage

Rule out interfering profile files, which the login shell sources on every job:

sudo -u gitlab-runner cat /home/gitlab-runner/.bash_logout
sudo -u gitlab-runner cat /home/gitlab-runner/.bashrc

Example Root Cause Analysis

A Terraform pipeline failed on a self-managed shell runner with terraform: command not found and exit status 127, even though terraform version worked fine over SSH.

Running sudo -iu gitlab-runner reproduced it immediately: the interactive admin login had /usr/local/bin on PATH via /etc/profile.d/, but the gitlab-runner user’s non-interactive job shell resolved a shorter PATH that omitted /usr/local/bin, where the Terraform binary lived. which terraform as the runner user returned nothing.

There were two clean fixes. The durable one was to add the directory to the runner’s environment in config.toml, which the shell executor injects into every job:

[[runners]]
  executor = "shell"
  environment = ["PATH=/usr/local/bin:/usr/bin:/bin"]

After sudo gitlab-runner restart, the job resolved terraform correctly. A quick verification confirmed the tool and permissions were now consistent between the login shell and the job shell, and the pipeline passed.

Prevention Best Practices

• Install CI tooling into a directory that is on the gitlab-runner user’s PATH, or set PATH explicitly via the environment key in config.toml.
• Never rely on sudo in shell-executor jobs unless you have configured a passwordless, tightly scoped sudoers rule for the gitlab-runner user.
• Keep the gitlab-runner home directory free of interactive-only .bashrc/.bash_logout commands.
• Run sudo gitlab-runner verify after upgrades and monitor disk on the builds directory so jobs never fail at the source stage.
• Prefer the Docker executor for reproducibility where possible; reserve the shell executor for tasks that genuinely need host access.

Quick Command Reference

sudo -iu gitlab-runner            # become the runner user (login shell)
echo "$PATH"; which <tool>        # confirm tools are resolvable in CI
sudo gitlab-runner verify         # check runners are healthy
sudo gitlab-runner list           # show executor per runner
sudo cat /etc/gitlab-runner/config.toml   # inspect executor + environment
df -h /home/gitlab-runner         # rule out a full disk
# exit 127 = not found, 126 = not executable, other = script failure

Conclusion

Shell-executor failures are almost always host-state problems: the gitlab-runner user’s environment, PATH, permissions, or disk differs from your interactive shell. The fastest path to a fix is to become the runner user with sudo -iu gitlab-runner, reproduce the failing command, and read the exit code — 127 and 126 point straight at PATH and permission issues. Pin PATH in config.toml, avoid sudo, and keep the host clean. For more pipeline fixes, see the GitLab CI/CD guides.