🚨 Automation Error Guide: '401 invalid signature' Webhook HMAC Verification Failed

Overview

A webhook signature failure happens when your receiver recomputes the HMAC of the incoming payload and the result does not match the signature the sender put in the request header. The receiver rejects the request with 401 Unauthorized to prevent forged or replayed events. The delivery never reaches your handler, so downstream automation silently stops.

You will see this in your receiver log:

WARN webhook.verify signature mismatch: expected=sha256=7d2f... received=sha256=9b14... path=/hooks/github
INFO http "POST /hooks/github HTTP/1.1" 401 31 "-" "GitHub-Hookshot/abc123"

And the sender’s delivery dashboard records the failed response:

Response 401  Unauthorized
{"message":"signature verification failed"}

It occurs the moment a signed event is delivered — most commonly after rotating a secret, changing a body parser, or deploying a new receiver. A signature that verified yesterday can start failing the instant any byte of the signed body or the shared secret changes.

Symptoms

• Sender’s delivery log shows repeated 401 responses with signature verification failed.
• Receiver logs signature mismatch or HMAC verification failed.
• Events stop processing even though the endpoint is reachable (200 on health checks).
• Manual replays from the provider UI also return 401.

# Tail the receiver and watch for the rejection
journalctl -u webhook-receiver --no-pager | grep -i "signature" | tail -5

WARN webhook.verify signature mismatch path=/hooks/stripe expected=t=1718... v1=4a9c...

# Confirm the request actually carries a signature header
curl -s -D - -o /dev/null https://hooks.example.com/health

HTTP/1.1 200 OK

Common Root Causes

1. The shared secret does not match

The most common cause: the secret configured on the sender differs from the one your receiver uses (often after a rotation that updated only one side).

# Compare the secret the receiver loaded (length/fingerprint, not the value)
printf '%s' "$WEBHOOK_SECRET" | wc -c
printf '%s' "$WEBHOOK_SECRET" | sha256sum

44
b91c3f...  -

If the fingerprint here does not match what you set in the provider, the HMAC can never agree. A trailing newline in a secret file (echo vs printf) is a frequent culprit.

2. Signing the wrong bytes (re-serialized body)

The HMAC must be computed over the exact raw bytes received. If a JSON middleware parses and re-serializes the body before you verify, key order and whitespace change and the digest no longer matches.

grep -RnE "express.json|bodyParser.json|c.req.json\(\)" ./src | head

src/app.ts:14:app.use(express.json())   // parses BEFORE verify -> raw body lost

You must capture the raw body (e.g., express.raw({type:'*/*'})) and verify against that, not against JSON.stringify(req.body).

3. Wrong digest encoding (hex vs base64)

Providers differ: some send sha256=<hex>, others base64. Comparing a hex digest to a base64 signature always fails even with the right secret.

# What does the header actually look like?
grep -i "x-hub-signature\|stripe-signature\|x-signature" /var/log/webhook/raw.log | tail -1

X-Hub-Signature-256: sha256=7d2f1a9c4b...   <- hex, lowercase

If your code produces hmac.digest('base64'), switch to 'hex' (or strip the sha256= prefix before comparing).

4. Timestamp outside the tolerance window

Stripe-style signatures sign timestamp.payload and reject deliveries older than a tolerance (default 300s). A slow queue, clock skew, or a delayed replay produces a valid-looking but expired signature.

# Check host clock drift against NTP
chronyc tracking | grep -E "System time|Last offset"
date -u

System time     : 0.000004 seconds slow of NTP time

Large drift, or buffering events for minutes before verifying, trips the t= tolerance check.

5. Header name or scheme parsing mismatch

The signature lives in a provider-specific header (X-Hub-Signature-256, Stripe-Signature, X-Signature). Reading the wrong header, or not splitting the t=...,v1=... scheme, yields an empty/garbled expected value.

grep -RniE "x-hub-signature|stripe-signature|x-signature|headers\[" ./src | head

src/verify.ts:8: const sig = req.headers['x-signature']  // provider sends X-Hub-Signature-256

6. Non-constant-time comparison flagged or truncated

Using == on digests of different lengths (one prefixed, one not) returns false even when the underlying bytes match. The prefix or casing must be normalized before a timingSafeEqual.

grep -RnE "timingSafeEqual|crypto.timingSafe|hmac.compare" ./src | head

src/verify.ts:12: if (expected !== received) return res.sendStatus(401)

Diagnostic Workflow

Step 1: Confirm a signature actually arrived

# Log the raw header and body for one delivery
journalctl -u webhook-receiver --no-pager | grep -iE "signature|x-hub|stripe-sig" | tail -3

If no signature header is present, the sender isn’t configured to sign — fix the sender, not the verifier.

Step 2: Recompute the HMAC by hand for one captured payload

# Given the exact raw body saved to body.bin and the shared secret
openssl dgst -sha256 -hmac "$WEBHOOK_SECRET" -hex body.bin

HMAC-SHA256(body.bin)= 7d2f1a9c4b...

Compare this to the sha256= value in the captured header. Match means the receiver’s body/encoding is wrong; mismatch means the secret is wrong.

Step 3: Verify the secret fingerprint on both sides

printf '%s' "$WEBHOOK_SECRET" | sha256sum

Set the same secret string in the provider UI and confirm there is no trailing newline or surrounding quotes.

Step 4: Confirm you verify the raw body, not a re-serialized one

grep -RnE "express.raw|express.json|getRawBody|req.rawBody" ./src | head

Ensure the body parser preserves raw bytes for the webhook route and that verification runs before any JSON parsing.

Step 5: Check encoding, prefix handling, and clock

grep -RnE "digest\('(hex|base64)'\)|replace\('sha256='|timingSafeEqual" ./src | head
chronyc tracking | grep "Last offset"

Align the digest encoding to the provider, strip the scheme prefix, and ensure host time is within tolerance.

Example Root Cause Analysis

A GitHub push webhook into /hooks/github starts returning 401 for every delivery right after a deploy. The endpoint health check is green, so it is not a connectivity problem.

The receiver log shows the mismatch:

WARN webhook.verify signature mismatch expected=sha256=7d2f1a9c received=sha256=7d2f1a9c

The two digests look identical, which is suspicious. Recomputing by hand against the saved raw body:

openssl dgst -sha256 -hmac "$WEBHOOK_SECRET" -hex body.bin

HMAC-SHA256(body.bin)= 7d2f1a9c4b8e...   (full 64 hex chars)

The recomputed digest matches the header. The bug is in the comparison: the new deploy added express.json() ahead of the verify middleware, so the handler was hashing JSON.stringify(req.body) for everything except the first 8 truncated characters that happened to align in the log.

Fix: mount a raw body parser for the webhook route and verify before parsing:

# app.ts
# app.post('/hooks/github', express.raw({ type: '*/*' }), verifyAndHandle)
sudo systemctl restart webhook-receiver

Verifying against the raw bytes makes the digests agree and deliveries return 200.

Prevention Best Practices

• Always verify the HMAC against the raw request body, captured before any JSON middleware re-serializes it.
• Store secrets without trailing newlines and validate the fingerprint on both sender and receiver after every rotation.
• Support dual secrets during rotation: accept either the old or new secret for a short overlap so no deliveries 401 mid-rollout.
• Match the provider’s digest encoding (hex vs base64) and strip the scheme prefix before a constant-time compare.
• Keep hosts on NTP and honor the timestamp tolerance window; alert on signature-failure rate so a rotation mistake is caught in minutes, not days.
• For ad-hoc triage, the free incident assistant can turn a batch of 401 verification logs into the likely secret-or-body cause. More patterns in the automation guides.

Quick Command Reference

# Recompute the expected HMAC for a captured raw body
openssl dgst -sha256 -hmac "$WEBHOOK_SECRET" -hex body.bin

# Fingerprint the loaded secret (detect mismatch / trailing newline)
printf '%s' "$WEBHOOK_SECRET" | sha256sum
printf '%s' "$WEBHOOK_SECRET" | wc -c

# Inspect the actual signature header and scheme
journalctl -u webhook-receiver | grep -iE "signature|x-hub|stripe-sig" | tail -3

# Confirm raw-body handling and digest encoding in code
grep -RnE "express.raw|express.json|digest\('(hex|base64)'\)|timingSafeEqual" ./src

# Check clock drift for timestamp-tolerant schemes
chronyc tracking | grep -E "System time|Last offset"

# Restart the receiver after a fix
sudo systemctl restart webhook-receiver

Conclusion

A 401 invalid signature means the receiver’s recomputed HMAC did not equal the sender’s signature. The usual root causes:

1. The shared secret differs between sender and receiver (often a one-sided rotation or trailing newline).
2. The HMAC is computed over a re-serialized body instead of the raw bytes.
3. The digest encoding (hex vs base64) or sha256= prefix is mishandled.
4. The signed timestamp is outside the tolerance window due to delay or clock skew.
5. The wrong signature header is read or the t=,v1= scheme isn’t parsed.
6. The comparison fails on length/prefix differences before a constant-time check.

Recompute the HMAC by hand for one captured payload first — that single test tells you whether the secret or the signed bytes are wrong, and the fix follows directly.