🚨 Ansible Error Guide: 'A worker was found in a dead state' — Fix OOM-Killed Forks

Overview

Ansible runs each host through a forked worker process. When one of those worker processes dies unexpectedly — killed by the kernel, crashed, or terminated — the parent detects the missing child and aborts the run:

ERROR! A worker was found in a dead state

There is usually no clean per-task traceback, because the worker was killed from the outside rather than failing a task. The most common killer, on the Ansible controller itself, leaves its fingerprint in the kernel log:

Out of memory: Killed process 21874 (ansible-playbook) total-vm:... anon-rss:...

Because the failure is a dead child process rather than a failed module, the fix lives at the controller and run-configuration level — memory, forks, and how much data each worker carries — not in any single task.

Symptoms

• The run dies with A worker was found in a dead state, often after fanning out to many hosts at once.
• No task-level error or stack trace precedes it; the play just stops.
• dmesg / journalctl -k on the controller shows the OOM killer terminating an ansible-playbook or Python process.
• Failure correlates with high forks, large inventories, or heavy fact gathering / large registered variables.
• Reducing --forks makes the run succeed, which strongly implicates controller memory.
• Occasionally triggered by a segfault in a C extension the module loads, not memory.

Common Root Causes

• Controller OOM from high forks: each fork is a Python process holding facts and variables; forks = 50 on a memory-thin controller can exhaust RAM and invite the OOM killer.
• Large fact and variable footprint: gathering all facts on thousands of hosts, or register-ing huge command outputs, multiplies memory per worker.
• Big loops and templated data in memory: massive loop lists or large inventory variables held across the run.
• Undersized CI runner or bastion: running Ansible from a small container/VM (512 MB–1 GB) with default forks.
• A crashing dependency: a module’s underlying C library or Python extension segfaulting inside the worker, killing it without an OOM entry.
• Swap disabled with tight RAM: no swap headroom, so a transient spike triggers an immediate kill instead of slowdown.

Diagnostic Workflow

First confirm whether the kernel killed the worker for memory — this decides the whole fix path. Check the controller’s kernel log right after a failure:

dmesg -T | grep -iE 'killed process|out of memory|oom'
journalctl -k --since '10 minutes ago' | grep -iE 'oom|killed process'

Watch controller memory while re-running to see it climb toward the ceiling:

# In a second terminal on the controller during the run
watch -n1 'free -m; echo; ps -C ansible-playbook -o pid,rss,cmd --sort=-rss | head'

Test the hypothesis directly by lowering the fork count — if it now completes, memory (or a per-worker resource) is the cause:

ansible-playbook site.yml --forks 5

Reduce the per-worker data footprint in the play and re-run:

- name: Memory-lean run
  hosts: all
  gather_facts: true
  gather_subset:
    - "!all"
    - "!min"
    - network            # collect only what you actually use
  tasks:
    - name: Avoid holding huge output in memory
      ansible.builtin.command: some-report
      register: report
      no_log: false
      changed_when: false
      # process/stream large output instead of registering it whole where possible

If there is no OOM line, capture whether a specific module kills its worker by isolating it with high verbosity:

ansible-playbook site.yml --limit one_host --tags suspect -vvv

Example Root Cause Analysis

A nightly compliance play ran against roughly 1,200 hosts from a 2 GB CI runner with forks = 40 in ansible.cfg. It failed intermittently with A worker was found in a dead state, always partway through the fact-gathering phase, and never on the same host twice.

journalctl -k on the runner showed the OOM killer terminating a Python child during the run. free -m under watch confirmed memory climbed to zero available as 40 workers each gathered the full fact set simultaneously. The failure wasn’t any host or task — it was 40 concurrent fact-heavy workers exceeding 2 GB.

Two changes fixed it. First, forks was lowered to 15, capping concurrent worker memory. Second, gather_subset was narrowed to only the fact groups the compliance checks actually referenced, cutting per-worker memory sharply. The play then completed reliably. As a durable fix, the CI runner was resized to 4 GB so the same inventory had headroom, and a free -m check was added at job start to fail fast if memory was already low.

Prevention Best Practices

• Right-size forks to controller RAM, not to host count — high parallelism multiplies memory per worker. Start conservative and raise while watching free -m.
• Narrow fact gathering with gather_subset or gather_facts: false where facts aren’t needed; enable fact caching so you don’t re-gather every run.
• Avoid registering huge outputs; stream or write large data to files instead of holding it in memory across the play.
• Give the controller swap headroom so a transient spike degrades performance instead of triggering an instant OOM kill.
• Size CI runners and bastions deliberately — default forks on a 1 GB runner is a common trap.
• Monitor controller memory during large runs and alert before it hits the ceiling; add a pre-run memory check in CI.

Quick Command Reference

# Confirm the OOM killer was involved
dmesg -T | grep -iE 'oom|killed process'
journalctl -k --since '15 min ago' | grep -i oom

# Watch memory and top Ansible processes during a run
watch -n1 'free -m; ps -C ansible-playbook -o pid,rss,cmd --sort=-rss | head'

# Lower parallelism to test the memory hypothesis
ansible-playbook site.yml --forks 5

# Check the configured fork count in effect
ansible-config dump --only-changed | grep -i forks

# Isolate a suspected crashing module on one host
ansible-playbook site.yml --limit one_host -vvv

Conclusion

“A worker was found in a dead state” means a forked worker process died out from under Ansible, and on the controller that almost always means the OOM killer reaped it. Check dmesg/journalctl -k for an OOM line, drop --forks to confirm memory is the cause, then reduce the per-worker footprint with gather_subset and fact caching and give the controller more RAM or swap. When there’s no OOM entry, isolate the run to one host with -vvv to catch a crashing module instead. Tune parallelism to available memory, and this failure disappears.