🚨 GCP Error Guide: '504 Gateway Timeout' on Cloud Load Balancing — Fix Backend Timeout

Overview

A Google Cloud external Application Load Balancer returns 504 Gateway Timeout when the backend does not send a full response within the backend service’s configured timeout. The client sees:

HTTP/1.1 504 Gateway Timeout

In the load balancer’s request logs the entry carries a statusDetails that names the timeout cause:

jsonPayload.statusDetails: "backend_timeout"

A related but different value appears when the backend closed the connection first:

jsonPayload.statusDetails: "response_sent_by_backend" / "backend_connection_closed_before_data_sent_to_client"

504 is distinct from 502: a 502 usually means the backend was unhealthy or reset the connection, while a 504 means the backend was reachable but too slow — it exceeded the proxy’s patience. Reading statusDetails is what tells the two apart.

Symptoms

• Long-running requests (reports, uploads, streaming) fail at a suspiciously round time — often the default backend timeout.
• statusDetails: backend_timeout in the load balancer logs.
• 504s that correlate with request duration, not with backend health checks (backends show healthy).
• WebSocket or streaming connections dropped after a fixed interval.
• Fast endpoints work; only slow ones time out.

Common Root Causes

• Backend service timeout too short — the default backend-service timeout (30s for many configs) is shorter than a legitimately long request.
• Genuinely slow backend — a slow query, downstream dependency, or overloaded instance that can’t respond in time.
• Keepalive/idle mismatch — the backend closes idle connections sooner than the LB expects, or the backend’s own timeout is shorter than the LB’s.
• Long-lived connections without config — WebSockets/streaming need the backend timeout raised because it also acts as the max stream duration.
• Slow health vs. slow response confusion — backends pass health checks (a cheap endpoint) but the real endpoint is slow, so it’s a 504, not a 502.
• Client or CDN timeout shorter than backend — a proxy in front giving up before the LB does.

Diagnostic Workflow

Confirm it is a timeout, not an unhealthy backend, by reading statusDetails:

gcloud logging read \
  'resource.type="http_load_balancer" AND httpRequest.status=504' \
  --limit=20 --freshness=1h \
  --format='value(timestamp, jsonPayload.statusDetails, httpRequest.requestUrl)'

Check the backend service’s configured timeout — compare it to how long the request actually needs:

gcloud compute backend-services describe BACKEND_SERVICE --global \
  --format='value(timeoutSec, connectionDraining.drainingTimeoutSec)'

Confirm the backends are actually healthy (rules out 502-style causes):

gcloud compute backend-services get-health BACKEND_SERVICE --global

Measure real backend latency to see if the backend is slow or the timeout is just too tight:

gcloud logging read 'resource.type="http_load_balancer" AND httpRequest.status=504' \
  --limit=20 --format='value(httpRequest.latency, jsonPayload.statusDetails)'

Example Root Cause Analysis

A reporting endpoint behind an external Application Load Balancer returned 504 Gateway Timeout for large date ranges but worked for small ones.

Diagnosis: the LB logs showed statusDetails: backend_timeout, and every failure had a latency very close to 30 seconds — the default backend-service timeout. get-health reported all backends healthy, ruling out a 502-style unhealthy-backend problem. The report generation for large ranges legitimately took 45–60 seconds, so the backend was fine but the proxy gave up at 30s.

Root cause: the backend-service timeout (30s) was shorter than the legitimate duration of a heavy report, so the LB cut the connection while the backend was still working.

Fix: raise the backend-service timeoutSec to cover the real p99 of the report endpoint with headroom, and — better long-term — move heavy reports to an async job with a status poll so no single request has to stay open that long. The 504s stopped and the synchronous timeout became a safety net rather than the primary limit.

Prevention Best Practices

• Set the backend-service timeout from the real p99 latency of the slowest legitimate endpoint, with headroom — don’t leave the default on a service that has long requests.
• Prefer async patterns (job + poll, or streaming with heartbeats) for anything that can run tens of seconds, instead of holding a synchronous request open.
• Keep the backend’s own timeout and keepalive longer than the LB’s, and any front proxy/CDN timeout longer than the LB’s, so the layers agree on who gives up last.
• For WebSockets/streaming, raise the backend timeout deliberately — it doubles as the max stream duration.
• Health-check a path representative of real work, or at least know your health endpoint is cheaper than your real endpoints.
• Alert on statusDetails=backend_timeout rate to catch creeping backend latency before users do.

Quick Command Reference

# Find 504s and their timeout cause
gcloud logging read 'resource.type="http_load_balancer" AND httpRequest.status=504' \
  --limit=20 --format='value(timestamp, jsonPayload.statusDetails, httpRequest.latency)'

# Show the backend service timeout
gcloud compute backend-services describe BACKEND_SERVICE --global --format='value(timeoutSec)'

# Confirm backends are healthy (rule out 502)
gcloud compute backend-services get-health BACKEND_SERVICE --global

# Raise the backend timeout
gcloud compute backend-services update BACKEND_SERVICE --global --timeout=120

Conclusion

A 504 Gateway Timeout from Cloud Load Balancing means the backend was reachable but too slow — the proxy hit the backend-service timeout while the backend was still working. Read statusDetails to separate it from a 502 (unhealthy backend), confirm backends are healthy, then decide whether the backend is genuinely slow or the timeout is simply tighter than the request needs. Raise the timeout from real p99 latency for legitimate long requests, but treat any endpoint that routinely runs tens of seconds as a candidate for an async pattern rather than a longer synchronous hold.