🚨 TLS & Security Error Guide: 'Host key verification failed' (SSH known_hosts)

Overview

Host key verification failed means the SSH client refused to connect because the server’s host key did not match what it has recorded in known_hosts, or it could not confirm the key at all. SSH uses Trust On First Use: the first connection records the server’s public host key, and every later connection must present the same key. If the presented key differs, SSH assumes the host may be impersonated (a man-in-the-middle) and aborts before authentication.

The classic alarming form:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
Offending ECDSA key in /home/deploy/.ssh/known_hosts:42
Host key verification failed.

It also appears more quietly in batch/automation when StrictHostKeyChecking blocks an unknown host:

Host key verification failed.

The fix is not to blindly delete the entry — that defeats the protection. The correct response is to determine why the key changed (legitimate rotation/rebuild vs an actual MITM) and verify the new fingerprint out-of-band before trusting it.

Symptoms

• SSH/SCP/rsync/git-over-SSH aborts with “Host key verification failed”.
• A “REMOTE HOST IDENTIFICATION HAS CHANGED” banner names an offending line in known_hosts.
• Automation fails on a new host because the key is unknown and StrictHostKeyChecking=yes.
• Connecting by IP works but by hostname fails (or vice versa) due to separate entries.

ssh -v deploy@app-03.internal.example.com 2>&1 | grep -iE 'host key|known_hosts|offending'

debug1: load_hostkeys: fopen /home/deploy/.ssh/known_hosts: ...
Offending ECDSA key in /home/deploy/.ssh/known_hosts:42
Host key verification failed.

Common Root Causes

1. The server was rebuilt / reprovisioned (new host keys)

A reinstall, re-image, or cloud instance replacement regenerates host keys, so the recorded key no longer matches.

ssh-keygen -F app-03.internal.example.com
ssh-keyscan -t ecdsa app-03.internal.example.com 2>/dev/null | ssh-keygen -lf -

# Host app-03.internal.example.com found: line 42
256 SHA256:OldFingerprintAAA... app-03.internal.example.com (ECDSA)
256 SHA256:NewFingerprintBBB... app-03.internal.example.com (ECDSA)

The stored fingerprint differs from the live one — expected after a rebuild.

2. Stale known_hosts after a deliberate key rotation

Host keys were rotated for hygiene/compliance, but clients still cache the old key.

ssh-keygen -l -F app-03.internal.example.com

# Host app-03.internal.example.com found: line 42
256 SHA256:OldFingerprintAAA...

If the rotation is documented/expected, this is a stale-cache problem, not an attack.

3. IP address reused by a different host (DHCP/cloud churn)

A known_hosts entry keyed by IP now points at a different machine because the IP was reassigned.

ssh-keygen -F 10.0.2.55
ssh-keyscan -t ed25519 10.0.2.55 2>/dev/null | ssh-keygen -lf -

# Host 10.0.2.55 found: line 17
256 SHA256:HostXFingerprint...
256 SHA256:HostYFingerprint...

Same IP, different fingerprint — the address was recycled.

4. Connecting through a different name/alias/port with no entry

The host answers on multiple names or a non-default port and only one variant is recorded.

grep -E 'app-03|10\.0\.2\.55|\[.*\]:2222' ~/.ssh/known_hosts

app-03.internal.example.com ecdsa-sha2-nistp256 AAAA...

A connection to [app-03.internal.example.com]:2222 has no entry, so verification cannot proceed.

5. A wrong or shared known_hosts file in automation

CI/automation uses a checked-in or shared known_hosts that is outdated or for a different fleet.

ssh -G app-03.internal.example.com | grep -iE 'userknownhostsfile|globalknownhostsfile'

userknownhostsfile /opt/ci/known_hosts ~/.ssh/known_hosts

A stale /opt/ci/known_hosts pins the wrong key for the whole pipeline.

6. Genuine man-in-the-middle / wrong host behind a proxy

A real interception, or a bastion/port-forward pointing at the wrong backend, presents an unexpected key.

ssh-keyscan -t ed25519 app-03.internal.example.com 2>/dev/null | ssh-keygen -lf -
# compare against the fingerprint published by the host's owner / config management

256 SHA256:UnexpectedZZZ... app-03.internal.example.com (ED25519)

If the fingerprint matches no record from a trusted source, treat it as hostile and do not connect.

Diagnostic Workflow

Step 1: Read the offending entry and which file holds it

ssh -v <user>@<host> 2>&1 | grep -iE 'offending|known_hosts'
ssh -G <host> | grep -i userknownhostsfile

This names the exact file and line of the cached key.

Step 2: Get the stored fingerprint and the live fingerprint

ssh-keygen -l -F <host>
ssh-keyscan -t ed25519,ecdsa,rsa <host> 2>/dev/null | ssh-keygen -lf -

Compare SHA256:... values. A difference confirms the key changed; the question is whether the change is legitimate.

Step 3: Verify the new fingerprint out-of-band

# On the server itself (console / config management), print its host key fingerprint:
for f in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf "$f"; done

256 SHA256:NewFingerprintBBB... root@app-03 (ECDSA)
256 SHA256:NewEd25519CCC... root@app-03 (ED25519)

The live fingerprint from Step 2 must match what the server reports here (or your config-management record). Only then is the change trustworthy.

Step 4: Remove the stale entry (after verification)

ssh-keygen -R <host>
ssh-keygen -R 10.0.2.55     # also remove any IP-keyed entry

ssh-keygen -R surgically removes only the offending host, leaving the rest of known_hosts intact.

Step 5: Re-add the verified key and reconnect

ssh-keyscan -t ed25519,ecdsa <host> 2>/dev/null >> ~/.ssh/known_hosts
ssh <user>@<host> 'echo connected'

connected

Example Root Cause Analysis

A nightly deploy job that SSHes to app-03.internal.example.com started failing:

@@@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @@@
Offending ECDSA key in /opt/ci/known_hosts:42
Host key verification failed.

The first instinct — “we got hacked” — needs verification, not a blind delete. Pulling both fingerprints:

ssh-keygen -l -F app-03.internal.example.com -f /opt/ci/known_hosts
ssh-keyscan -t ecdsa app-03.internal.example.com 2>/dev/null | ssh-keygen -lf -

256 SHA256:OldFingerprintAAA...
256 SHA256:NewFingerprintBBB...

Cross-checking with the cloud team’s change log shows app-03 was re-imaged the previous evening as part of an OS upgrade, which regenerated its host keys. To confirm it is the real host and not an impostor, the new fingerprint is compared against the value the provisioning automation recorded:

# from config management output for app-03:
# ECDSA SHA256:NewFingerprintBBB...

The new key matches the provisioned fingerprint exactly — a legitimate rotation, not an attack. Fix: remove the stale entry from the CI known_hosts and add the verified key:

ssh-keygen -R app-03.internal.example.com -f /opt/ci/known_hosts
ssh-keyscan -t ecdsa,ed25519 app-03.internal.example.com 2>/dev/null >> /opt/ci/known_hosts

The deploy connects, and the pipeline still rejects any unverified key change in the future.

Prevention Best Practices

• Never blindly ssh-keygen -R and reconnect without verifying the new fingerprint out-of-band — that is exactly the check the warning exists to enforce.
• Distribute host keys at provision time: have config management write a managed known_hosts (or use SSH certificates) so rebuilds update clients automatically.
• Prefer SSH host certificates signed by a trusted CA (@cert-authority in known_hosts) so a rebuilt host signed by the same CA is trusted without per-host churn.
• In automation, pin a managed known_hosts and update it through the same pipeline that reprovisions hosts; avoid StrictHostKeyChecking=no, which silently trusts any key.
• Record host-key fingerprints in your inventory so an unexpected change is immediately distinguishable from a planned rotation. See the security hardening guides for an SSH host-key management baseline.
• When a key change appears during an incident, the free incident assistant can help confirm whether it lines up with a logged rebuild/rotation before you trust it.

Quick Command Reference

# What entry is offending, and from which file?
ssh -v <user>@<host> 2>&1 | grep -iE 'offending|known_hosts'
ssh -G <host> | grep -i userknownhostsfile

# Stored vs live fingerprint
ssh-keygen -l -F <host>
ssh-keyscan -t ed25519,ecdsa,rsa <host> 2>/dev/null | ssh-keygen -lf -

# Verify on the server itself (out-of-band)
for f in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf "$f"; done

# Remove stale entry (host and any IP-keyed copy)
ssh-keygen -R <host>
ssh-keygen -R <ip>

# Re-add the verified key
ssh-keyscan -t ed25519,ecdsa <host> 2>/dev/null >> ~/.ssh/known_hosts

Conclusion

Host key verification failed means the server’s host key does not match known_hosts, and SSH is protecting you from a possible impersonation. Diagnose the cause before clearing it:

1. The server was rebuilt/re-imaged, regenerating its host keys.
2. Keys were deliberately rotated and clients still cache the old one.
3. An IP-keyed entry now points at a different, recycled host.
4. You are connecting via a name/alias/port that has no entry.
5. Automation uses a stale or wrong shared known_hosts.
6. A genuine MITM or a misrouted proxy is presenting an unexpected key.

Always pull both fingerprints, verify the new one out-of-band against the host or your inventory, and only then ssh-keygen -R and re-add it — and prefer SSH certificates or managed known_hosts so legitimate rebuilds never look like attacks.