🚨 Linux Error Guide: 'Host key verification failed' — fix the SSH known_hosts mismatch

Overview

SSH refuses to connect and prints a warning that the remote host’s key does not match the one it remembers. The classic full form is loud and alarming on purpose:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
Add correct host key in /home/user/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/user/.ssh/known_hosts:42
Host key verification failed.

It can also appear in a shorter form (for example when StrictHostKeyChecking is on but no full banner is printed, or in Git-over-SSH and Ansible/rsync-over-SSH runs):

Host key verification failed.
fatal: Could not read from remote repository.

SSH remembers each server’s public host key in known_hosts. On every connection it compares the key the server presents against the stored one. When they differ, SSH aborts before authenticating — by design, because a changed key could mean an attacker is impersonating the server. The job is to determine whether the change is legitimate (far more common) or genuinely suspicious, then act accordingly.

Symptoms

• ssh, git clone/git pull over SSH, scp, rsync, or Ansible all fail against one specific host with Host key verification failed.
• The error names an “Offending” key and a line number in a known_hosts file.
• The failure started right after a server was rebuilt, reimaged, reinstalled, or migrated to new hardware/cloud instance.
• Connecting to other hosts works fine — only this one host is affected.
• In automation, jobs that previously succeeded now fail at the SSH handshake with no authentication attempt logged on the server.

Common Root Causes

• The server was rebuilt or reimaged. A fresh OS install generates new host keys, so the fingerprint legitimately changed. This is the overwhelmingly common cause.
• IP address reuse. A cloud instance was destroyed and a different machine got the same IP (or the same hostname now points at a new box). Your old entry belongs to the previous server.
• A host behind a load balancer or floating IP where different backends present different host keys.
• Host keys were regenerated manually or by a config-management run (ssh-keygen -A, package reinstall of openssh-server).
• You connected via a name/IP that now resolves elsewhere (DNS change, VPN routing, /etc/hosts edit).
• A genuine man-in-the-middle — rare, but the reason the check exists. Treat it as possible until you can prove the key change was expected.

Diagnostic Workflow

First, read the error: it tells you the exact file and line of the offending entry.

# The warning names the file and line, e.g. known_hosts:42
ssh -v user@server 2>&1 | grep -i 'known_hosts\|host key\|offending'

Look at the stored entry before removing anything:

# Show the exact stored line the warning referenced
sed -n '42p' ~/.ssh/known_hosts

# Find every entry for this host (entries may be hashed)
ssh-keygen -F server.example.com
ssh-keygen -F 203.0.113.10

Compare fingerprints. Get the key the server is presenting now:

# Fingerprint the key the remote is currently offering
ssh-keyscan -t ecdsa,ed25519,rsa server.example.com 2>/dev/null | ssh-keygen -lf -

Then verify it against a trusted source — this is the security-critical step. Get the expected fingerprint out-of-band (from the server’s console, your cloud provider’s instance metadata/console log, or your config-management record) and confirm they match:

# Run ON the server (via console/other trusted path) to get its real fingerprints
for f in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf "$f"; done

If the presented fingerprint matches the server’s actual key, the change is legitimate and you can safely update known_hosts. If it does NOT match any key on the real server, stop — do not connect, and investigate the network path.

Once confirmed legitimate, remove the stale entry and re-add the correct one:

# Remove the old entry (works for both plain and hashed known_hosts)
ssh-keygen -R server.example.com
ssh-keygen -R 203.0.113.10

# Reconnect — you'll be prompted to accept the new key
ssh user@server.example.com

Example Root Cause Analysis

A CI pipeline that deploys via rsync over SSH started failing overnight with Host key verification failed. The engineer ran the pipeline’s SSH command manually with -v and saw Offending ED25519 key in /home/deploy/.ssh/known_hosts:17.

ssh-keyscan -t ed25519 deploy-target produced a fingerprint that did not match line 17. Rather than blindly deleting the entry, the engineer checked the cloud console: the deploy target instance had been replaced by an autoscaling event at 02:14, and the replacement reused the same DNS name via the launch template. The cloud provider’s console log showed the new instance’s host-key fingerprints, which matched exactly what ssh-keyscan now returned.

Confirmed legitimate, the fix was ssh-keygen -R deploy-target followed by a controlled re-add. The durable fix was to stop relying on interactive trust in CI: the deploy host key is now pinned in a managed known_hosts file distributed to the runners, and instance replacements publish their host key into that file automatically. No more overnight breakage, and the trust decision is no longer “type yes.”

Prevention Best Practices

• Manage known_hosts centrally for automation. Distribute a curated, verified known_hosts (or use ssh-keyscan gated by a fingerprint check) instead of StrictHostKeyChecking=no, which disables the protection entirely.
• Prefer SSH host certificates. With an SSH Certificate Authority, clients trust the CA (@cert-authority in known_hosts) and never need per-host entries — rebuilt servers just present a new cert signed by the same CA.
• Persist host keys across rebuilds where appropriate. Back up /etc/ssh/ssh_host_* and restore them on reprovision so the fingerprint stays stable (only where the identity is meant to persist).
• Record fingerprints out-of-band. Publish each host’s fingerprints from a trusted channel (provider console, config-management inventory) so you can verify changes instead of trusting blindly.
• Never blanket-disable checking. StrictHostKeyChecking=no and wiping known_hosts on every run turns off exactly the defense that catches interception.
• Use ssh-keygen -R / -F, not manual editing, to avoid corrupting hashed known_hosts files.

Quick Command Reference

# Inspect the stored entry the warning pointed to
ssh-keygen -F <host>                 # find entry (handles hashed files)
sed -n '<N>p' ~/.ssh/known_hosts     # show a specific line

# Get and fingerprint the key the server presents now
ssh-keyscan -t ed25519,ecdsa,rsa <host> 2>/dev/null | ssh-keygen -lf -

# Get the real fingerprints ON the server (trusted path)
for f in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf "$f"; done

# Remove a stale/verified-wrong entry, then reconnect
ssh-keygen -R <host>
ssh-keygen -R <ip>
ssh <user>@<host>

# Add a verified key non-interactively (only after out-of-band check)
ssh-keyscan -t ed25519 <host> >> ~/.ssh/known_hosts

Conclusion

Host key verification failed is SSH doing its job: the server’s identity does not match what you trusted before. In practice it almost always means the host was rebuilt or its IP was reused, and the fix is a one-line ssh-keygen -R followed by re-adding the key. But the warning exists to catch interception, so make the verification real — compare the presented fingerprint against the server’s actual key from a trusted channel before you clear the entry. For fleets, move past interactive trust altogether with a managed known_hosts or SSH host certificates, and you eliminate both the security risk and the recurring breakage.