🚨 GCP Error Guide: 'Memory limit exceeded' on Cloud Run — Fix Container OOM

Overview

Cloud Run terminates a container instance when it exceeds its configured memory limit, logging the event and returning 503 to in-flight requests. The literal log entry reads:

Memory limit of 512 MiB exceeded with 517 MiB used. Consider increasing the memory limit, see https://cloud.google.com/run/docs/configuring/memory-limits

You will also see the request that triggered it fail:

The request failed because the instance could not start successfully, or exceeded its memory limit.

Cloud Run enforces the limit hard: the instance is killed (OOM), not throttled. Because a killed instance drops its in-flight requests, a memory problem shows up as intermittent 503s and a restart loop rather than a clean error, which is what makes it easy to misdiagnose as a networking or cold-start issue.

Symptoms

• Memory limit of N MiB exceeded entries in the Cloud Run service logs.
• Intermittent 503 responses, especially on large payloads or concurrent requests.
• Instances restarting frequently; container instance count churning in metrics.
• Latency spikes as requests are retried against fresh (cold) instances.
• The failure correlates with request size or concurrency, not with a specific endpoint’s logic.

Common Root Causes

• Limit set too low for real usage — the default or a copied-over 256/512 MiB that the runtime plus workload genuinely exceeds.
• High concurrency multiplying memory — Cloud Run sends multiple concurrent requests to one instance; per-request memory times concurrency blows the limit.
• Buffering large payloads in memory — reading an entire upload, response, or file into memory instead of streaming.
• In-memory caches or leaks — an unbounded cache, or a leak that grows until the next request tips it over.
• Writing to the in-memory filesystem — Cloud Run’s local filesystem is a tmpfs backed by memory; writing temp files counts against the memory limit.
• Runtime baseline underestimated — the language runtime, framework, and libraries consume a fixed baseline before your code allocates anything.

Diagnostic Workflow

Confirm the OOM and read the actual usage number from the log:

gcloud run services logs read SERVICE --region=REGION --limit=50 \
  | grep -i 'memory limit'

Check the current limit and concurrency — the two settings that interact:

gcloud run services describe SERVICE --region=REGION \
  --format='value(spec.template.spec.containers[0].resources.limits.memory, spec.template.spec.containerConcurrency)'

Chart real memory utilization to size the limit from evidence, not guesswork:

# In Metrics Explorer, chart:
#   run.googleapis.com/container/memory/utilizations  (per revision)
# Look at the peak, not the average, and note the concurrency at that peak.
gcloud monitoring dashboards list

Rule out tmpfs writes and large in-memory buffers by checking what the code does with temp files and payloads (any write under a non-persistent path counts as memory):

gcloud run services logs read SERVICE --region=REGION --limit=100 \
  | grep -iE 'oom|out of memory|killed'

Example Root Cause Analysis

An image-processing service on Cloud Run began throwing 503s only under load, with Memory limit of 512 MiB exceeded in the logs.

Diagnosis: memory utilization in Metrics Explorer sat comfortably at ~40% for single requests but spiked past 100% whenever the instance handled several at once. The service had containerConcurrency set to 8, and each request decoded a full image into memory. 8 concurrent decodes times the per-request footprint exceeded 512 MiB even though a single request was fine.

Root cause: not a leak and not an undersized single-request footprint — the memory limit was being multiplied by concurrency, and the copied-over 512 MiB limit had never been sized against concurrent load.

Fix: two changes. Lower containerConcurrency to a value the memory budget supports and raise the memory limit to cover peak concurrent usage with headroom, then stream/resize images instead of holding the full decode in memory. 503s stopped and instance churn flattened.

Prevention Best Practices

• Size the memory limit from peak utilization at your real concurrency, with ~25% headroom — never copy a default across services.
• Tune containerConcurrency and memory together; remember total memory ≈ per-request footprint × concurrency + runtime baseline.
• Stream large uploads/downloads instead of buffering them fully in memory.
• Treat any temp-file writes as memory (tmpfs); write to a bucket or stream instead when files are large.
• Bound in-memory caches and add a memory profile to CI so regressions surface before production.
• Alert on memory/utilizations crossing ~85% so you resize before the OOM, not after the 503s.

Quick Command Reference

# Find memory-limit OOM events
gcloud run services logs read SERVICE --region=REGION --limit=50 | grep -i 'memory limit'

# Show current memory limit and concurrency
gcloud run services describe SERVICE --region=REGION \
  --format='value(spec.template.spec.containers[0].resources.limits.memory, spec.template.spec.containerConcurrency)'

# Raise the memory limit (size from peak utilization)
gcloud run services update SERVICE --region=REGION --memory=1Gi

# Adjust concurrency to fit the memory budget
gcloud run services update SERVICE --region=REGION --concurrency=4

Conclusion

Memory limit exceeded on Cloud Run is an OOM kill, not a throttle, which is why it presents as flaky 503s and instance churn rather than a clean error. The fix is to size the limit from evidence — peak memory utilization at your real concurrency — and to remember that Cloud Run multiplies per-request memory by containerConcurrency and counts tmpfs writes against you. Tune memory and concurrency together, stream large payloads instead of buffering them, and alert before 85% utilization so you resize ahead of the outage instead of after it.