🚨 Pulumi Error: 'rpc error: code = Unavailable desc = transport is closing' — Cause, Fix, and Troubleshooting Guide

Overview

error: waiting for RPCs: rpc error: code = Unavailable desc = transport is closing means a provider plugin process died in the middle of a deployment. Pulumi runs providers as gRPC subprocesses; transport is closing is gRPC’s way of saying the connection was torn down while the engine still had in-flight calls it was waiting for. The provider didn’t return an error — it vanished.

The most common trigger is the operating system killing the provider for using too much memory (OOM). It can also be a provider panic/crash or the host terminating the process. Unlike a connection refused (which means the process never came up or already exited before a call), transport is closing means the process was alive and serving, then dropped mid-flight.

error: waiting for RPCs: rpc error: code = Unavailable desc = transport is closing

Symptoms

• pulumi up runs for a while, processes several resources, then fails with transport is closing.
• The failure is intermittent — it happens on big updates or busy CI and passes on retry with less load.
• dmesg shows the provider (or node/python language host) was OOM-killed.
• A pending operations warning appears on the next command because an operation was interrupted.
• Larger stacks, high --parallel, or memory-hungry providers (kubernetes with many manifests, aws with many resources) are involved.

Common Root Causes

1. Out-of-memory kill (most common)

The kernel OOM-killer terminates the provider or language host when the machine runs out of memory. The gRPC channel closes abruptly, and every pending RPC fails with transport is closing.

# from dmesg
Out of memory: Killed process 12345 (pulumi-resource) ...

2. A provider panic or crash mid-operation

A bug in the provider (or a custom/dynamic provider) panics during Create/Update, taking the process down while requests are outstanding.

3. Too much parallelism

High concurrency multiplies memory use and open connections. On a small runner this pushes the host over its limit and the provider gets killed under load.

4. Host or container resource limits

CI containers and Kubernetes pods with tight memory.limit cgroups will kill the process at the limit — the same OOM story, just enforced by the container runtime.

5. The machine went to sleep or the network host dropped

A laptop suspending, or an SSH/session host being terminated during a long run, cuts the transport mid-deployment.

How to Diagnose

Confirm an OOM kill on the host — this is the fastest way to prove the cause:

dmesg -T | grep -i -E "killed process|out of memory|oom-killer"

Re-run with gRPC and verbose logging to capture the provider’s final output before it died:

PULUMI_DEBUG_GRPC="$(pwd)/grpc.log" \
  pulumi up --logtostderr -v=9 2> pulumi-debug.log
grep -iE "panic|signal|killed|transport is closing" pulumi-debug.log

Watch memory while the update runs (in another terminal) to see it climb toward the limit:

# Linux
watch -n1 'free -m; echo; ps aux --sort=-%mem | head'

Check for interrupted work left behind:

pulumi stack export | jq '.deployment.pending_operations'

Fixes

Reduce parallelism to lower peak memory. This resolves the majority of OOM-driven cases:

pulumi up --parallel 1
# or a moderate value if the host has headroom
pulumi up --parallel 4

Give the run more memory. Increase the CI runner/container size or the pod’s memory limit, and free memory before starting. More RAM plus lower --parallel is the durable combination for large stacks.

Split a very large stack. If a single stack manages thousands of resources, break it into smaller stacks (by service or layer) and wire them together with stack references so no single update needs so much memory at once.

Retry — completed steps are saved. State from finished operations is persisted, so a retry resumes rather than starting over:

pulumi up

Clear any pending operations first. If the interrupted run left a pending op, reconcile it before the next update:

pulumi refresh          # re-reads real cloud state and clears stale pending ops
# or, if you know the op did not take effect, cancel the stale lock/op:
pulumi cancel

Fix a crashing provider. If the debug log shows a panic rather than an OOM, update the provider plugin to a fixed version, or for a custom/dynamic provider add error handling so it returns a gRPC error instead of exiting.

What to Watch Out For

• transport is closing = process died while serving; connection refused = process never answered — the OOM check (dmesg) is the same first step for both.
• After the crash, always check pending_operations and run pulumi refresh so state matches reality before the next up.
• Don’t just keep retrying at high --parallel; if it’s OOM, lower parallelism or add memory or it will fail again.
• Container/pod memory limits cause the same kill as host OOM — raise the cgroup limit, not just the node size.
• Long interactive runs are fragile to laptop sleep and dropped SSH; run big deployments from CI or a stable host.

Related Guides

• Pulumi Error: ‘rpc error: code = Unavailable … connection refused’ — Troubleshooting Guide
• Pulumi Error: ‘pending operations’ — Troubleshooting Guide
• Pulumi Error: ‘could not load plugin’ — Troubleshooting Guide