🚨 GCP Error Guide: 'invalid_grant: Invalid JWT Signature' — Fix Service Account Auth

Overview

When an application authenticates to Google as a service account, the client library builds a signed JWT and exchanges it at the OAuth token endpoint for an access token. If Google cannot verify that signature, the token exchange fails and you get:

google.auth.exceptions.RefreshError:
 ('invalid_grant: Invalid JWT Signature.',
  {'error': 'invalid_grant', 'error_description': 'Invalid JWT Signature.'})

The raw HTTP response from oauth2.googleapis.com/token looks like this:

HTTP/1.1 400 Bad Request
{
  "error": "invalid_grant",
  "error_description": "Invalid JWT Signature."
}

The error surfaces at the moment your code first tries to get a token — not at startup — so it often shows up mid-run or on the first API call rather than when credentials load.

Symptoms

• invalid_grant: Invalid JWT Signature (or the sibling Invalid JWT: ... token expired) from any Google client library at token-refresh time.
• Authentication that worked yesterday suddenly fails everywhere using the same key file.
• CI jobs, cron tasks, or backends that use a JSON key stop working while gcloud on your laptop (using your user creds) still works fine.
• The failure is total for that identity — every Google API returns it — which distinguishes it from a per-API 403 PERMISSION_DENIED.

Common Root Causes

• The signing key was deleted or rotated — the private key in your JSON file no longer exists on the service account, so Google has no matching public key to verify the signature.
• Stale or wrong key file — the app is loading an old key (baked into an image, cached in a secret, or a leftover file) that has since been deleted.
• Corrupted or mangled key — the private_key PEM lost its newlines (\n) during copy/paste, env-var injection, or JSON re-encoding, so the signature is malformed.
• Serious client clock skew — the VM/container clock is far enough off that the JWT iat/exp are invalid, producing signature/invalid_grant failures.
• Mismatched client_email and key — the JSON was hand-assembled and the private key does not belong to the named service account.
• Service account disabled or deleted — the identity itself is gone, so no key on it can be verified.
• Domain-wide delegation misconfig — for delegated (impersonation) flows, the sub or scopes aren’t authorized, and the grant is rejected.

Diagnostic Workflow

First, identify which service account and key the failing process is actually using:

echo "$GOOGLE_APPLICATION_CREDENTIALS"
python -c "import json,os;d=json.load(open(os.environ['GOOGLE_APPLICATION_CREDENTIALS']));print(d['client_email'], d['private_key_id'])"

List the keys that currently exist on that service account and check whether your private_key_id is still there:

gcloud iam service-accounts keys list \
  --iam-account=SA_NAME@PROJECT_ID.iam.gserviceaccount.com

If your private_key_id is not in the list, the key was deleted or rotated — that is the root cause.

Confirm the service account itself is enabled and exists:

gcloud iam service-accounts describe \
  SA_NAME@PROJECT_ID.iam.gserviceaccount.com

Check the client clock, since large skew alone triggers invalid_grant:

date -u
timedatectl status   # look for "System clock synchronized: yes"

Verify the PEM in the JSON is intact (real newlines, proper header/footer):

python -c "import json,os;print(json.load(open(os.environ['GOOGLE_APPLICATION_CREDENTIALS']))['private_key'][:40])"
# should print: -----BEGIN PRIVATE KEY-----\n... with actual line breaks when written out

Look at key-creation/deletion history in the audit log:

gcloud logging read 'protoPayload.serviceName="iam.googleapis.com"
  AND protoPayload.methodName:"ServiceAccountKey"' \
  --project=PROJECT_ID --limit=20 \
  --format='table(timestamp, protoPayload.methodName, protoPayload.authenticationInfo.principalEmail)'

Example Root Cause Analysis

A batch pipeline running in Cloud Run started failing every run with invalid_grant: Invalid JWT Signature. The same code path had worked for months. Because the token exchange failed, the team first suspected an OAuth outage.

The engineer pulled the private_key_id out of the mounted key file and ran gcloud iam service-accounts keys list — the ID was not in the results. The IAM audit log showed a DeleteServiceAccountKey event from a well-meaning security automation that had rotated keys older than 90 days the previous night. The Cloud Run service was still mounting the old key from Secret Manager, which the rotation job hadn’t updated.

The fix was to create a fresh key, store it in Secret Manager, and redeploy:

gcloud iam service-accounts keys create key.json \
  --iam-account=SA_NAME@PROJECT_ID.iam.gserviceaccount.com
gcloud secrets versions add pipeline-sa-key --data-file=key.json

The durable fix was to stop shipping long-lived keys entirely: the pipeline moved to Workload Identity Federation / attached service accounts so there is no JSON key to rotate or leak.

Prevention Best Practices

• Prefer keyless auth — use Workload Identity (GKE), attached service accounts (Cloud Run/Compute), or Workload Identity Federation instead of downloaded JSON keys; there is nothing to expire or mismatch.
• Coordinate key rotation — if you must rotate keys, update every consumer (Secret Manager version, CI secret, image) in the same change; never delete an old key before the new one is live everywhere.
• Store keys in a secret manager — not baked into images or committed to config, so there is one source of truth to rotate.
• Keep clocks synced — ensure NTP/chronyd is running on every host; clock drift is a silent cause of invalid_grant.
• Handle env-var newlines carefully — when injecting private_key through an env var, preserve \n escaping or base64-encode the whole JSON to avoid PEM corruption.
• Alert on auth failures — treat a spike of invalid_grant in logs as a page-worthy signal, since it means an identity is fully broken.

Quick Command Reference

# Which SA / key is the process using?
python -c "import json,os;d=json.load(open(os.environ['GOOGLE_APPLICATION_CREDENTIALS']));print(d['client_email'], d['private_key_id'])"

# Do the keys on this SA still include mine?
gcloud iam service-accounts keys list \
  --iam-account=SA_NAME@PROJECT_ID.iam.gserviceaccount.com

# Is the service account enabled?
gcloud iam service-accounts describe SA_NAME@PROJECT_ID.iam.gserviceaccount.com

# Check for clock skew
date -u && timedatectl status

# Create a replacement key
gcloud iam service-accounts keys create key.json \
  --iam-account=SA_NAME@PROJECT_ID.iam.gserviceaccount.com

# Audit key create/delete events
gcloud logging read 'protoPayload.methodName:"ServiceAccountKey"' \
  --project=PROJECT_ID --limit=20

Conclusion

invalid_grant: Invalid JWT Signature means Google could not verify the signature on your service account’s JWT — almost always because the signing key was rotated or deleted, the key file is stale or corrupted, or the client clock has drifted. Find the private_key_id the process is really using, confirm it still exists with gcloud iam service-accounts keys list, and either point the app at a live key or (better) move to keyless Workload Identity so there is no long-lived key to break in the first place.