🚨 Ansible EDA Error: Rulebook Source Plugin Crash Stops Event Automation

Overview

Ansible Event-Driven Automation (EDA) runs a rulebook whose sources (webhook listener, Kafka, url_check, alertmanager) feed events to rules. If a source plugin raises an unhandled exception, that source’s event stream dies and ansible-rulebook logs the crash — often taking the rulebook activation down with it:

ERROR - Source plugin ansible.eda.kafka failed
Traceback (most recent call last):
  ...
kafka.errors.NoBrokersAvailable: NoBrokersAvailable
ERROR - Shutting down rulebook due to source error

Once the source thread has crashed, no new events arrive, so rules stop firing and the automation silently goes idle even though the process may still appear “up.”

Symptoms

• ansible-rulebook logs a source plugin traceback and Source plugin ... failed.
• Events stop triggering rules; the automation appears to hang with no actions.
• In AWX/EDA controller, the rulebook activation shows Failed or repeatedly restarting.
• A specific source (Kafka, webhook, url_check) is named in the error, not the rules.
• The process may still be running while producing no output.

Common Root Causes

• Upstream source unavailable — the broker, webhook publisher, or endpoint the source connects to is down (NoBrokersAvailable, ConnectionRefused).
• Bad source configuration — wrong host/port/topic, missing TLS/auth, or a malformed source argument.
• Plugin exception on a bad event — the plugin fails to parse a malformed message and does not handle it gracefully.
• Missing collection or dependency — the source plugin’s collection or a Python dependency is not installed in the EDA environment.
• Credential/permission failure — the source cannot authenticate to Kafka/webhook/cloud and raises on connect.
• No restart policy — a transient source failure is not retried, so a blip permanently stops the stream.

Diagnostic Workflow

Run the rulebook verbosely to surface the plugin traceback:

ansible-rulebook --rulebook rulebook.yml --inventory inventory.yml -vv 2>&1 | tail -60

Identify which source and plugin failed from the rulebook:

grep -nA4 'sources:' rulebook.yml

Confirm the source plugin’s collection is installed in the EDA environment:

ansible-galaxy collection list | grep -iE 'ansible.eda|kafka'
python -c "import aiokafka; print(aiokafka.__version__)" 2>&1

Test connectivity to the upstream the source depends on (Kafka example):

nc -zv broker.internal 9092
kcat -b broker.internal:9092 -L 2>&1 | head

For the EDA controller, inspect the activation status and logs:

# In AWX/EDA controller
# Rulebook Activations -> <activation> -> Instances -> Logs
grep -iE 'source plugin|shutting down|traceback' activation.log | tail

Example Root Cause Analysis

An EDA rulebook that auto-remediated alerts stopped acting on anything overnight. The activation still showed “running,” but no rules fired. Running the rulebook with -vv revealed the Kafka source had crashed:

ERROR - Source plugin ansible.eda.kafka failed
kafka.errors.NoBrokersAvailable: NoBrokersAvailable
ERROR - Shutting down rulebook due to source error

nc -zv broker.internal 9092 failed to connect. The Kafka cluster had rolled during a maintenance window and its advertised listener address changed; the source plugin lost the brokers, raised NoBrokersAvailable, and — with no restart policy — the source thread exited and took the event stream with it. The rulebook process lingered but was deaf.

The immediate fix was to point the source at the correct bootstrap servers and restart the activation. The durable fix was to enable the EDA controller’s restart policy (restart_policy: on-failure) so a source crash relaunches the activation instead of leaving it silently idle, and to add a heartbeat/liveness event so a stream that goes quiet is detected rather than assumed healthy. A second, malformed-message crash later was handled by validating events in the rule condition instead of letting the plugin choke.

Prevention Best Practices

• Set a restart policy on rulebook activations (restart_policy: on-failure/always) so a source crash relaunches the stream instead of going silently idle.
• Monitor for source liveness — emit or expect a periodic heartbeat event and alert when the stream goes quiet, since “process up” does not mean “receiving events.”
• Pin and pre-install the source plugin’s collection and Python dependencies in the EDA execution environment.
• Validate source configuration (brokers, topics, auth) in a smoke test before deploying the activation.
• Handle malformed events defensively in rule conditions rather than relying on the plugin to survive bad input.
• Make source upstreams (Kafka, webhook publisher) themselves resilient, and expect reconnects across their maintenance windows.

Quick Command Reference

# Reproduce with the traceback
ansible-rulebook --rulebook rulebook.yml -i inventory.yml -vv 2>&1 | tail -60

# Which source failed?
grep -nA4 'sources:' rulebook.yml

# Is the plugin/collection installed?
ansible-galaxy collection list | grep -i ansible.eda

# Test the upstream the source needs
nc -zv <broker-host> 9092
kcat -b <broker-host>:9092 -L | head

# Restart the activation after fixing (controller) or rerun locally
ansible-rulebook --rulebook rulebook.yml -i inventory.yml

Conclusion

An EDA source plugin crash is uniquely deceptive: the rulebook process can stay “up” while its event stream is dead, so the automation goes quietly idle. Run the rulebook verbosely to get the plugin traceback, and treat the named source — not the rules — as the fault. Fix the upstream or the source config, then make the activation self-healing with a restart policy and a liveness heartbeat, so the next transient broker or endpoint blip reconnects automatically instead of silently stopping every rule you depend on.