🔁 Writing Idempotent Automation Scripts You Can Re-Run Safely

The first time a deploy script half-fails and you are afraid to run it again, you learn what idempotency is worth. A non-idempotent script that dies at step 4 leaves you in an unknown state: did step 3 happen? Will re-running it duplicate something? You end up manually unwinding the mess.

An idempotent script you can run a hundred times and the system ends up in the same state. Run it after a partial failure and it picks up cleanly. In 25 years of automating infrastructure, idempotency is the single property that turns a fragile script into something you trust on a schedule.

What idempotent actually means

Idempotent means the result of running the operation once is the same as running it N times. The script describes a desired end state and makes only the changes needed to reach it. It does not blindly perform actions; it checks first.

The mental shift is from “do this action” to “ensure this state.” Not “create the user” but “ensure the user exists.” Not “append the line” but “ensure the line is present.” That one reframing is most of the work.

Check-then-act in bash

The core pattern is to test current state before changing it. Compare these two:

# Not idempotent — fails the second run, or duplicates
mkdir /opt/app/data
echo "export PATH=/opt/app/bin:\$PATH" >> /etc/profile.d/app.sh
useradd appuser

# Idempotent — safe to re-run
mkdir -p /opt/app/data

grep -qxF 'export PATH=/opt/app/bin:$PATH' /etc/profile.d/app.sh 2>/dev/null \
  || echo 'export PATH=/opt/app/bin:$PATH' >> /etc/profile.d/app.sh

id appuser &>/dev/null || useradd appuser

Three small changes:

• mkdir -p succeeds whether or not the directory exists.
• grep -qxF ... || echo appends the line only if it is not already there. No more duplicate PATH entries after five runs.
• id appuser || useradd creates the user only if it is missing.

The pattern is always the same: test for the desired state, and only act if it is absent.

Idempotent file edits

The append-if-missing trick is the most common need. For more structured edits, write the whole file from a template rather than mutating it in place:

cat > /etc/myapp/config.ini <<EOF
host = ${DB_HOST}
port = ${DB_PORT}
EOF

Rendering the entire file every run is inherently idempotent — the end state does not depend on what was there before. In-place edits with sed -i are where idempotency goes to die, because a second run edits the already-edited line. When you can render the whole file, do.

Idempotency in Python

The same discipline applies in Python, just with cleaner conditionals:

from pathlib import Path

data_dir = Path("/opt/app/data")
data_dir.mkdir(parents=True, exist_ok=True)   # idempotent mkdir

profile = Path("/etc/profile.d/app.sh")
line = 'export PATH=/opt/app/bin:$PATH\n'
existing = profile.read_text() if profile.exists() else ""
if line not in existing:
    with profile.open("a") as f:
        f.write(line)

exist_ok=True is the mkdir -p of Python. The line not in existing check is the grep || echo. Same pattern, more readable.

For anything that talks to a remote system — a cloud API, a database — idempotency usually means using upsert semantics or checking for existence first: get then create only on 404, or a PUT to a known resource ID rather than a POST that generates a new one each call.

Make operations resumable

True idempotency also means a partial run leaves no half-done damage. Two techniques:

Use atomic writes. Write to a temp file, then mv it into place. The mv is atomic on the same filesystem, so a reader never sees a half-written file and a crash mid-write leaves the old file intact:

tmp="$(mktemp)"
generate_config > "$tmp"
mv "$tmp" /etc/myapp/config.ini

Mark completion explicitly. For multi-step migrations, drop a marker once a step finishes and skip it on re-run:

if [[ ! -f /var/lib/app/.migrated-v2 ]]; then
  run_migration_v2
  touch /var/lib/app/.migrated-v2
fi

Now a script that dies at step 5 re-runs and skips steps 1 through 4 cleanly.

Test it the honest way

There is exactly one test for idempotency, and it is brutal: run the script twice and diff the system state. The second run should report no changes and alter nothing. If the second run does work, your script is not idempotent yet. I also like to kill the script partway through with Ctrl-C and re-run it — a resumable script recovers; a fragile one corrupts.

Where AI helps

When I am converting an old fire-once script into something I can schedule, I hand it to AI with a sharp prompt:

“Rewrite this bash script to be idempotent. For every command that creates, appends, or modifies state, add a check so the script is a no-op on a second run. Use atomic writes for file generation. Don’t change what the script ultimately produces.”

The model is reliably good at spotting the bare mkdir, the unconditional >>, the useradd with no guard. I review each change against the check-then-act rule. I keep these conversion prompts in my prompt library because I do this conversion constantly.

The payoff

Idempotent scripts are the ones you can put in a systemd timer or a config-management run and forget about. They recover from partial failures, they do not accumulate duplicate state, and they let you re-run after a scare without holding your breath. That trust is the whole point of automation.