⚡ SaltStack Reactors: Event-Driven Automation Without the Meltdown

SaltStack’s event bus is the feature that elevates it from configuration management to an automation platform. Every meaningful thing that happens — a minion booting, a beacon firing on a changed file, a job completing, a custom event from your application — lands on the bus, and the reactor system lets you turn any of those events into automated action across the fleet. It’s genuinely powerful. It’s also the part of Salt most likely to take down your master if you get it wrong.

The failure mode is specific and brutal: a reactor with a too-loose tag match fires on unrelated events, or a reaction that emits an event matching its own trigger creates an infinite loop. Either way, a single minion event fans out into thousands of jobs, the master’s job queue floods, and the whole control plane grinds. This guide shows how to write reactors that react precisely, can’t loop, and stay bounded.

Match the event tightly

The first rule is that a reactor should fire on exactly the events you mean and nothing else. Salt matches reactors against event tags, and a sloppy match like salt/minion/*/start plus a broad reaction is how one rolling reboot turns into a fleet-wide stampede. Be specific about the tag and key on the event data:

# /etc/salt/master.d/reactor.conf  (provided separately; enable after testing)
reactor:
  - 'myapp/deploy/requested':
      - /srv/reactor/handle_deploy.sls

# /srv/reactor/handle_deploy.sls
{% set service = data['data']['service'] %}
{% set version = data['data']['version'] %}

run_deploy_orchestration:
  runner.state.orchestrate:
    - args:
        - mods: orch.deploy
        - pillar:
            target_service: {{ service }}
            target_version: {{ version }}

Notice the custom tag myapp/deploy/requested rather than a broad built-in event. The reactor fires only when something deliberately emits that exact tag, and it reads service and version from the event payload. This precision is the single biggest factor in keeping a reactor safe.

Use orchestrate for anything multi-step

Beginners reach for local.state.apply to run a state directly on the minion that triggered the event. That’s fine for a one-step reaction, but the moment your response needs to coordinate multiple steps — drain the node, deploy, health-check, re-add to the pool — you want runner.state.orchestrate, which runs the workflow from the master with proper ordering:

# /srv/salt/orch/deploy.sls
{% set service = pillar['target_service'] %}

deploy_in_batches:
  salt.state:
    - tgt: 'role:{{ service }}'
    - tgt_type: grain
    - sls: apps.{{ service }}
    - batch: 25%        # roll the deploy, don't hit every node at once
    - require:
        - salt: drain_from_lb

drain_from_lb:
  salt.function:
    - name: lb.deregister
    - tgt: 'role:{{ service }}'
    - tgt_type: grain

The require ordering makes the orchestration deterministic, and batch: 25% bounds the blast radius so a fleet-wide event doesn’t fan out to every node simultaneously. Orchestrate states are also idempotent when written this way, so a re-fired event re-runs safely instead of corrupting state.

Guard against reaction loops

The nastiest reactor bug is the self-trigger. A reactor reacts to an event, the reaction changes state, the state change emits an event, and that event matches the same reactor. Now you have a loop that the master happily executes forever. The defense is to make the trigger and the reaction’s side effects distinguishable. If your reaction must emit an event, give it a different tag namespace than the one the reactor matches, and add an explicit guard:

# Only react if this wasn't itself triggered by a prior reaction.
{% if data['data'].get('source') != 'reactor' %}
run_deploy_orchestration:
  runner.state.orchestrate:
    - args:
        - mods: orch.deploy
{% endif %}

Tagging reaction-emitted events with source: reactor and skipping those in the reactor breaks the cycle. Trace the loop scenario explicitly when you design any reactor that triggers something which itself produces events — it’s the failure that takes down masters.

Drafting reactors with AI, keeping the off switch human

Reactors are subtle enough that AI drafts help, and dangerous enough that you keep a human kill switch. A prompt:

You are a senior SaltStack engineer. Write a reactor that responds to the custom event myapp/deploy/requested by running an orchestrate state to deploy a service in 25% batches. Add a loop guard so reaction-emitted events can’t re-trigger it, and bound the blast radius. Provide the master config separately.

The model produced a tight reactor and added the operational caveat that matters:

Here’s the reactor and orchestrate state with the loop guard and batching. One strong recommendation: add an off switch — a pillar flag like reactor_enabled: false that the orchestrate state checks before acting. Auto-remediation reactors have a habit of fighting a human mid-incident, restarting a service someone just stopped on purpose. You want to disable the reactor without editing master config and reloading.

That’s the human-in-the-loop value made concrete. The model surfaced the classic operational hazard — automation overriding an operator during an incident — and the fix is a flag a person can flip. Test every new reactor against a single targeted minion before enabling it fleet-wide, and never enable a reactor in master config as part of the same change that introduces it.

Power with brakes

The reactor system is what makes Salt feel alive — infrastructure that responds to events instead of waiting for a human to run a command. But that responsiveness is only safe with brakes: precise tag matching so it fires on the right events, orchestrate with batching so reactions stay bounded, loop guards so a reaction can’t trigger itself, and an off switch so humans stay in control.

For generating reactors and orchestration, see our SaltStack reactor and orchestrate prompt, and pair it with the SaltStack states and pillar design prompt for the states they invoke. The Infrastructure as Code category covers the rest of the configuration-management toolchain. Build the loop guard and the off switch before you enable anything fleet-wide — they’re the brakes that let you use the power safely.