🚨 GitLab CI Error Guide: 'Permission denied (publickey)' — Fix SSH Auth

Overview

A GitLab CI job fails whenever it tries an SSH operation — cloning a private repo over git@, scp/rsync to a deploy host, or ssh into a server — without a usable private key the remote will accept:

git@gitlab.example.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

The deploy-host variant looks the same but names the server:

Permission denied (publickey).
lost connection

The job’s script never reaches the intended command because SSH authentication fails first.

Symptoms

• SSH clone, submodule fetch, rsync, or ssh deploy@host fails immediately with Permission denied (publickey).
• The same command works from a developer laptop but not in CI.
• ssh -T git@gitlab.example.com in the job prints the permission-denied line.
• Verbose SSH (ssh -vvv) shows no mutual signature algorithm or offers no identity file.
• The job hangs then fails on host-key verification the first time it contacts a new host.

Common Root Causes

• No private key in the job — the runner has no key loaded; SSH has nothing to offer.
• Key not added to the agent — the key variable exists but ssh-add was never run, or ssh-agent isn’t started.
• Public key not registered — the corresponding public key isn’t a Deploy Key on the target repo, or not in the deploy user’s authorized_keys.
• Masked/protected variable not available — the private key CI variable is protected but the branch isn’t, so it’s empty at runtime.
• Mangled key formatting — pasting the key into a variable stripped newlines; OpenSSH rejects a single-line key.
• Missing/incorrect known_hosts — strict host-key checking blocks the connection to an untrusted host.
• Read-only deploy key used for push — a deploy key without write access fails on push operations.
• Legacy key type disabled — newer OpenSSH refuses ssh-rsa with SHA-1; the key algorithm is no longer accepted.

Diagnostic Workflow

Add a debug step that proves whether a key is loaded and which host is being contacted:

ssh-debug:
  image: alpine:3.20
  before_script:
    - apk add --no-cache openssh-client git
    - eval "$(ssh-agent -s)"
    - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -    # load key; strip CR
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - ssh-keyscan gitlab.example.com >> ~/.ssh/known_hosts  # trust host
  script:
    - ssh-add -l                                  # list loaded keys — must not be empty
    - ssh -T git@gitlab.example.com || true       # expect a welcome, not permission-denied
    - ssh -vvv git@gitlab.example.com 2>&1 | tail -30   # inspect offered identities

Verify the private-key variable is intact (correct header and multi-line):

check-key:
  script:
    - 'echo "$SSH_PRIVATE_KEY" | head -1'   # should be -----BEGIN OPENSSH PRIVATE KEY-----
    - 'echo "$SSH_PRIVATE_KEY" | wc -l'      # more than 1 line — not collapsed

Confirm the CI variable is available on this ref (a protected variable is empty on unprotected branches) and that the matching public key is registered as a Deploy Key (repo → Settings → Repository → Deploy keys) or in the host’s authorized_keys.

Example Root Cause Analysis

A deploy job cloned a second private repo during the build and failed with Permission denied (publickey) on every MR, but succeeded on the default branch. The pipeline used:

variables:
  # SSH_PRIVATE_KEY set as a PROTECTED CI variable
build:
  before_script:
    - eval "$(ssh-agent -s)"
    - echo "$SSH_PRIVATE_KEY" | ssh-add -
  script:
    - git clone git@gitlab.example.com:team/shared-lib.git

Two problems compounded: the key variable was marked protected, so on unprotected MR branches $SSH_PRIVATE_KEY was empty and ssh-add loaded nothing; and the key had been pasted as a single line, so even on the default branch OpenSSH rejected it intermittently. The fix:

1. Re-added the key as a File-type masked variable (preserving newlines), or piped it through tr -d '\r' and ensured a trailing newline.
2. Decided the shared-lib clone genuinely needed to run on MRs, so the variable was left protected but the job was moved to run only on protected refs — OR, where MRs needed it, a read-only Deploy Key scoped to just shared-lib was used instead of a broadly-scoped key.
3. Added ssh-keyscan to populate known_hosts so the first contact didn’t fail host verification.

After loading a well-formed key available on the ref and trusting the host, the clone succeeded.

Prevention Best Practices

• Store the private key as a File-type or properly-newlined masked CI variable; verify the first line is the BEGIN ... PRIVATE KEY header.
• Register the matching public key as a scoped, read-only Deploy Key on exactly the repos a job needs (write only when pushing).
• Always start ssh-agent and ssh-add the key in before_script; confirm with ssh-add -l.
• Populate known_hosts with ssh-keyscan (or a pinned known-hosts value) rather than disabling host-key checking.
• Be deliberate about protected variables: protected keys are empty on unprotected branches — either mark the branch protected or scope the job to protected refs.
• Prefer ed25519 keys; if you must use RSA, ensure the server accepts the signature algorithm (PubkeyAcceptedAlgorithms).
• Never bake private keys into images or echo them to logs; keep them in masked variables only.

Quick Command Reference

# Load a key safely in a job (strip CR, keep newlines)
eval "$(ssh-agent -s)"
echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
ssh-add -l                                   # verify a key is loaded

# Trust the host without disabling verification
mkdir -p ~/.ssh && chmod 700 ~/.ssh
ssh-keyscan gitlab.example.com >> ~/.ssh/known_hosts

# Test auth (expect a welcome banner, not permission-denied)
ssh -T git@gitlab.example.com

# Diagnose which identities SSH offers
ssh -vvv git@gitlab.example.com 2>&1 | grep -i 'offering\|identity\|denied'

Conclusion

Permission denied (publickey) means SSH could not present a private key the remote accepts. In CI the usual causes are a missing or agent-less key, a private-key variable that’s malformed or unavailable on the branch (protected-variable trap), a public key that isn’t registered as a Deploy Key or authorized_keys entry, or an untrusted host. Load a well-formed key into ssh-agent, trust the host with ssh-keyscan, register a correctly-scoped public key, and confirm the variable is present on the ref — SSH operations then authenticate cleanly.