🚨 Jenkins Error: 'Agent went offline during the build' — Cause, Fix, and Troubleshooting Guide

Overview

A build fails partway through, not because a step returned an error, but because the controller lost its connection to the agent running the job. Jenkins aborts the build and records:

Agent went offline during the build
ERROR: Connection was broken: java.io.IOException: Unexpected termination of the channel
Finished: FAILURE

On older versions and legacy terminology you may instead see:

slave went offline during the build

The controller and agent talk over a single channel (SSH or JNLP/inbound). When that channel dies mid-build — because the agent crashed, ran out of memory, lost the network, filled its disk, or was terminated by a cloud provider — the build cannot continue and is marked FAILURE.

Symptoms

• A build that was running normally stops with Agent went offline during the build and no step-level error.
• The node shows as offline or “disconnected” in Manage Jenkins → Nodes immediately after the failure.
• Cloud/ephemeral agents (EC2, Kubernetes pods) disappear entirely from the node list.
• The agent reconnects a minute later, so the node looks healthy by the time you investigate — but the build is already dead.
• Recurs on memory-heavy or long builds, rarely on short ones.
• Agent log shows OutOfMemoryError, channel stopped, disk-full errors, or an abrupt end with no shutdown message.

Common Root Causes

• Agent JVM or process OOM — the agent host ran out of memory; the kernel OOM-killer killed the agent JVM or the build process, dropping the channel. Often paired with java.lang.OutOfMemoryError in the agent log.
• Network interruption / JNLP inbound drop — a firewall idle-timeout, load balancer, VPN, or transient packet loss cut the TCP channel; the inbound agent couldn’t hold the connection.
• Cloud agent terminated — an EC2 spot instance was reclaimed, an autoscaler scaled the node in, or a Kubernetes agent pod was evicted/OOMKilled/preempted mid-build.
• Disk full on the agent — the agent’s workspace or temp volume filled, so the agent process or its channel buffer failed. This can also trip the node monitor’s disk threshold.
• Ping/response timeout — the controller’s agent-to-controller ping timed out (default response threshold), so Jenkins declared the agent dead even though it was just slow/GC-paused.
• Agent JVM crash — a native crash (hs_err_pid file), a bad plugin, or an incompatible Java version killed the agent JVM.
• Controller restart / heavy GC — the controller itself paused long enough (GC or restart) to drop channels.

How to diagnose

Read the agent-side log first — it usually contains the real cause that the controller can’t see. In Manage Jenkins → Nodes → agent-01 → Log, or on the agent host:

# Agent service / launcher log (path varies by launch method)
journalctl -u jenkins-agent --since '20 min ago'
tail -n 200 /var/log/jenkins-agent/agent.log
# JVM crash dump if the agent process died natively
ls -1 /home/jenkins/hs_err_pid*.log 2>/dev/null

Check the OS for OOM kills and disk pressure on the agent:

dmesg -T | grep -i 'killed process\|out of memory'   # OOM-killer activity
df -h /home/jenkins /tmp                              # disk full?
free -m                                               # available memory

For Kubernetes agents, inspect why the pod vanished:

kubectl get events -n jenkins --sort-by=.lastTimestamp | grep -iE 'evict|oomkill|preempt|kill'
kubectl describe pod <agent-pod> -n jenkins | grep -iA3 'Last State\|Reason'

Check controller-side connectivity and timeout settings in the system log:

# Manage Jenkins → System Log — look for:
#   "Ping response time out" / "Terminated" / "Connection was broken"

To rule out a ping-timeout false positive, review the ping thread settings under Manage Jenkins → System (or the hudson.slaves.ChannelPinger properties) and correlate the failure time with GC pauses in the controller/agent GC logs.

Fixes

1. Give the agent enough memory (and cap build memory)

If the agent log shows OOM, either add RAM to the agent or constrain what the build allocates. For a Kubernetes agent, set requests/limits so the pod isn’t evicted or OOMKilled:

podTemplate(containers: [
  containerTemplate(
    name: 'build', image: 'maven:3.9-eclipse-temurin-21',
    resourceRequestMemory: '2Gi', resourceLimitMemory: '4Gi',
    resourceRequestCpu: '1', resourceLimitCpu: '2'
  )
]) {
  node(POD_LABEL) {
    stage('build') { sh 'mvn -B verify' }
  }
}

Cap JVM-based build tools so the total stays under the limit (e.g. MAVEN_OPTS=-Xmx2g). Leave headroom for the agent JVM itself.

2. Make the connection survive network blips

For inbound (JNLP) agents behind a firewall or load balancer that idle-times out connections, keep the channel warm and reconnect automatically. Launch the agent with reconnect enabled:

java -jar agent.jar \
  -url https://jenkins.example.com/ \
  -name agent-01 \
  -secret @secret-file \
  -webSocket        # WebSocket transport survives proxies/LBs better than raw TCP

-webSocket (or the WebSocket option in the node config) avoids the fixed inbound TCP port and tolerates proxies far better. Also raise firewall idle timeouts on the path between agent and controller.

3. Don’t run critical builds on reclaimable cloud capacity

If EC2 spot reclaim or node scale-in is terminating agents mid-build, either move long/critical jobs to on-demand nodes, or make them resumable with retry:

options { retry(2) }   // re-run the build on a fresh agent after an offline failure

Combine retry with idempotent stages so a mid-build agent loss costs a re-run, not a broken deploy. Configure the cloud plugin’s idle/retention so agents aren’t torn down while builds are queued on them.

4. Keep the agent disk clear

If disk-full is dropping agents, add workspace cleanup and build discarders so volumes don’t fill. See the disk-space guide below for the full approach:

post { always { cleanWs() } }   // Workspace Cleanup plugin

5. Tune the ping timeout for GC-heavy agents

If healthy agents are being dropped during long GC pauses, raise the response threshold instead of masking a real problem. As a controller JVM flag:

-Dhudson.slaves.ChannelPinger.pingTimeoutSeconds=300
-Dhudson.slaves.ChannelPinger.pingIntervalSeconds=60

Prefer fixing the GC pressure first; only relax the ping timeout when the pauses are genuinely unavoidable.

What to watch out for

• The controller’s error is a symptom. The root cause is almost always in the agent log or the OS/cloud event stream — start there, not in the build console.
• Ephemeral agents delete themselves on failure, taking their logs with them. Ship agent logs off-box (or to pod logs collection) before they vanish.
• Frequent offline-drops with disk errors will also trip the Disk space is below threshold node monitor — treat them together.
• A channel that closes cleanly vs. one that drops mid-stream look different: a clean close often surfaces as ChannelClosedException instead. Check which you actually have.
• The free incident assistant can help correlate an agent drop with OOM and cloud eviction events across logs.

Related

• Jenkins Error: ChannelClosedException — the related channel failure when the agent connection closes rather than the node going offline.
• Jenkins Error: Disk space is below threshold — the node monitor that takes agents offline when their disk fills.
• GitLab CI: exit code 137 (OOMKilled) — the equivalent OOM/eviction mid-job failure on GitLab runners.