🚨 Jenkins Error: 'FATAL: command execution failed' — Cause, Fix, and Troubleshooting Guide

Overview

A build dies partway through a step and the console ends with a blunt remoting failure. Jenkins tried to launch a process on an agent and the communication channel to that agent collapsed:

FATAL: command execution failed
java.io.IOException: Backing channel 'agent-01' is disconnected.
	at hudson.remoting.Channel$1.handle(Channel.java:...)
Caused by: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call ... failed.
	The channel is closing down or has closed down
	at hudson.remoting.Channel.call(Channel.java:...)

This is a remoting error, not a build-script error. Jenkins runs steps on agents through a bidirectional channel; when that channel drops mid-step — because the agent died, ran out of memory, lost network connectivity, or was terminated — the running command cannot report back and Jenkins marks it FATAL: command execution failed. It is frequently paired with ChannelClosedException, hudson.remoting.RequestAbortedException, or a java.io.IOException: Backing channel ... is disconnected / broken pipe.

Symptoms

• A step (often a long sh/bat, a stash, or an archive) stops abruptly with FATAL: command execution failed.
• The stack trace names ChannelClosedException, RequestAbortedException, or Backing channel '<agent>' is disconnected.
• The agent shows as offline in Manage Jenkins > Nodes right after the failure, sometimes reconnecting seconds later.
• Failures cluster on one agent, on memory-heavy steps, or on the longest-running stages.
• The controller log or agent log shows Ping failed, Connection was aborted, or a broken pipe near the failure time.
• Retrying the build on a different, healthy agent succeeds.

Common Root Causes

• Agent process died or was OOM-killed — the JVM or a child process on the agent was terminated (kernel OOM-killer, container eviction, kill), taking the channel with it. Check for an OOM alongside java.lang.OutOfMemoryError.
• Network instability between controller and agent — a dropped TCP connection, firewall idle-timeout, VPN blip, or proxy reset severs remoting; ping/keep-alive then fails.
• Agent host resource exhaustion — the node ran out of memory, file descriptors, or PIDs, so it could no longer fork the requested command.
• Container/pod agent evicted or scaled in — Kubernetes or a cloud autoscaler reclaimed the ephemeral agent mid-build (node drain, spot reclaim, pod eviction).
• Launch method / JVM crash — a mismatched Java version, a bad JVM flag, or an inbound-agent that lost its connection back to the controller.
• Controller under heavy load or restarting — the controller GC-paused or restarted (safe restart, upgrade), and every in-flight agent channel closed at once.

How to diagnose

Start by correlating the failure time with the agent’s availability. In Manage Jenkins > Nodes, open the failing agent and read its Log — a clean shutdown, a Terminated event, or an abrupt gap tells you whether the agent left voluntarily.

On the agent host, look for an OOM kill or a JVM crash around the failure timestamp:

# On the agent host
dmesg -T | grep -i 'killed process\|out of memory'   # kernel OOM-killer
journalctl --since '15 min ago' | grep -i 'oom\|jenkins\|agent'
ls -1 hs_err_pid*.log 2>/dev/null                     # JVM fatal crash logs

For Kubernetes-based agents, check whether the pod was evicted or the node scaled in:

kubectl get events --sort-by=.lastTimestamp | grep -iE 'evict|preempt|drain|oomkill'
kubectl describe pod <agent-pod> | sed -n '/Events/,$p'

On the controller, the remoting channel state is visible in the system log:

# Manage Jenkins > System Log, or the controller service journal
journalctl -u jenkins --since '15 min ago' | grep -iE 'ping failed|channel|remoting|disconnected'

You can also confirm a suspected agent is memory-starved from the Script Console (Manage Jenkins > Script Console) — but run read-only checks; the console executes on the controller unless you target a node.

Fixes

Fix 1: Give the agent (and its JVM) enough memory

If diagnosis shows an OOM, raise the container/VM memory and cap the agent JVM so it fails gracefully instead of being killed. For a Kubernetes pod template, set requests/limits so the scheduler reserves headroom:

# Kubernetes plugin pod template (excerpt)
spec:
  containers:
    - name: jnlp
      resources:
        requests:
          memory: "1Gi"
          cpu: "500m"
        limits:
          memory: "2Gi"
      env:
        - name: JAVA_OPTS
          value: "-Xmx512m"   # keep agent JVM well under the pod limit

Also right-size the build itself — a mvn/gradle/node step with a huge heap can OOM the node even when the agent JVM is fine.

Fix 2: Add retry and a health check around fragile steps

Wrap steps that suffer transient channel drops so a lost agent is retried on a fresh one instead of failing the pipeline:

pipeline {
  agent { label 'linux' }
  options { retry(count: 2) }        // whole-build retry for infra flakiness
  stages {
    stage('Build') {
      steps {
        retry(3) {                   // step-level retry for the flaky command
          sh './build.sh'
        }
      }
    }
  }
}

Fix 3: Harden the remoting connection

For inbound (JNLP) or SSH agents dropping on idle firewalls, enable TCP keep-alive and increase ping tolerances so a brief network stall does not close the channel. On the agent launch command add -webSocket or ensure keep-alive is on; on the controller, review Manage Jenkins > Nodes > (agent) > Configure launch settings and, for SSH agents, add JVM/connection options. Also confirm the agent’s Java version matches what the controller requires — a mismatched JDK can crash the remoting layer.

Fix 4: Keep non-interruptible work off preemptible agents

If ephemeral/spot agents are being reclaimed mid-build, route long, non-restartable stages to a small pool of stable, on-demand nodes via a dedicated label, and reserve autoscaled/spot capacity for restart-tolerant stages.

Fix 5: Reduce controller pressure

If every agent channel closed simultaneously, the controller likely GC-paused or restarted. Raise the controller heap, offload all builds to agents (keep the built-in node at 0 executors), and avoid triggering builds during upgrades or safe restarts.

What to watch out for

• Treat FATAL: command execution failed as an infrastructure signal, not a script bug — the command may have been running fine when the channel died.
• Always cap the agent JVM heap below the container/VM memory limit, or the OOM-killer will target the agent and reproduce this error.
• Idle-timeout firewalls and NAT gateways silently kill long-lived remoting connections; enable keep-alive on both ends.
• When failures follow autoscaler activity, correlate timestamps with scale-in/eviction events before blaming the pipeline.
• Don’t mask a real OOM with blind retries — retrying an under-provisioned agent just fails again more slowly.
• Try the free incident assistant to triage a noisy remoting incident, and browse the full Jenkins guides hub for related failures.

Related

• Jenkins Error: ChannelClosedException — the underlying remoting exception most often paired with this failure.
• Jenkins: agent went offline during build — when the agent disconnect itself is the primary symptom.
• Jenkins Error: java.lang.OutOfMemoryError: Java heap space — the memory exhaustion that frequently kills the agent behind this error.