What actually breaks, and what to check first
The dangerous automation script is not the one that crashes. It is the one that keeps going after a step failed, and reports success. Bash defaults are actively unhelpful here: an unset variable expands to an empty string, a failing command in a pipeline is invisible unless you ask, and the script continues to the next line regardless. `rm -rf "$DIR/"` with `DIR` unset is the canonical result.
So the first thing worth fixing in almost any Bash script is not its logic but its failure behaviour. `set -euo pipefail` converts three classes of silent corruption into immediate, visible stops, and it is a one-line change.
Python automation fails differently. It is much harder to lose a variable, but easy to build shell commands by string concatenation, to ignore a subprocess return code, or to write a file non-atomically so that an interrupted run leaves a half-written config in place of a working one.
Triage order
Work down this list in order. Each step either finds the fault or rules out a whole class of cause — a negative result is progress, not a wasted step.
-
Does the script stop when a step fails?
head -5 script.sh — look for set -euo pipefailWithout it, a failed command is ignored and the script proceeds with wrong state. `pipefail` matters independently: without it a pipeline reports the exit status of its last command only, so a failure upstream is invisible.
-
Is every expansion quoted?
shellcheck script.shUnquoted expansions word-split on whitespace and glob unexpectedly. This is why scripts work in testing and fail on the one path containing a space. ShellCheck finds these reliably and takes seconds.
-
Which command actually failed?
bash -x script.sh 2>&1 | tail -40Trace mode prints each command with expansions resolved, so you see the real values rather than the source text. Usually faster than adding echo statements, and it shows what the variable actually contained.
-
Is it safe to run twice?
Run it twice in a scratch environment and compare the result.Automation that is only safe once cannot be re-run during an incident — precisely when you most need it. Idempotency is a reliability property, not a style preference.
-
For Python: are subprocess results checked?
grep -n "subprocess\.\(run\|call\|Popen\)" *.py`subprocess.run` without `check=True` returns a result object that is easy to discard, so the command fails and the script continues. `shell=True` with interpolated input is a command-injection defect.
Diagnostic commands
Every command is labelled by what it can do to the system. Read-only commands are safe to run during an incident; the others are not, and are marked accordingly.
set -euo pipefail Exit on error, on unset variables, and on any failure within a pipeline.
How to read it: The single highest-value line in a Bash script. Add `IFS=$'\n\t'` alongside it when handling filenames that may contain spaces.
shellcheck -S warning script.sh Static analysis for real Bash defects.
How to read it: Prioritise quoting and word-splitting findings — those are genuine bugs. Style suggestions can wait; SC2086 (unquoted expansion) usually cannot.
trap 'rc=$?; echo "FAILED line $LINENO (exit $rc)" >&2' ERR Report where a script failed instead of ending silently.
How to read it: Turns "the script did not work" into a line number. Combine with a cleanup trap on EXIT to remove temporary files on every exit path.
mktemp -d Create a temporary directory with a race-free unique name.
How to read it: Predictable paths under /tmp are a symlink-attack vector on shared hosts. Pair with `trap 'rm -rf "$tmp"' EXIT` so cleanup happens even on failure.
python3 -c "import ast,sys; ast.parse(open(sys.argv[1]).read())" script.py Confirm a Python file parses without executing any of it.
How to read it: Useful in a pipeline gate for scripts that have side effects at import time, where simply importing to check is itself dangerous.
bash -n script.sh Syntax-check a script without running it.
How to read it: Catches unclosed quotes and missing `fi`/`done`. It does not catch logic or quoting errors — that is what ShellCheck is for.
Failure modes
These are distinct problems, not variations of one. Matching the symptom to the right cause is most of the work.
The script reports success but did nothing.
- Cause:
- A command failed, Bash continued, and the final command succeeded — so the exit status is 0.
- Fix:
- `set -e` plus an ERR trap. Without them the exit status reflects only the last command, which is why "it exited 0" is weak evidence that anything worked.
Works on most inputs, breaks on one filename.
- Cause:
- An unquoted expansion word-split on a space, or a glob character was expanded.
- Fix:
- Quote every expansion. ShellCheck flags these exhaustively, which makes this one of the few bug classes you can eliminate rather than manage.
A pipeline reports success although a stage failed.
- Cause:
- Bash returns the exit status of the last command in the pipeline by default.
- Fix:
- `set -o pipefail`. Without it, `generate | tee out.txt` succeeds whenever `tee` succeeds, no matter what happened upstream.
A destructive command runs against the wrong path.
- Cause:
- A variable was unset or empty, so the path collapsed to something dangerous.
- Fix:
- `set -u` makes an unset variable a fatal error. Use `"${DIR:?DIR is required}"` for the specific ones, so the failure names the missing variable.
An interrupted run leaves a corrupt configuration file.
- Cause:
- The file was written in place, so a partial write replaced valid content.
- Fix:
- Write to a temporary file on the same filesystem, then `mv` it over the target. Rename is atomic; a direct write is not.
A Python script continues after a shell command failed.
- Cause:
- `subprocess.run` was called without `check=True`, so the non-zero return code was never inspected.
- Fix:
- Pass `check=True`, or test `returncode` explicitly. Also prefer a list of arguments over `shell=True`, which removes an entire injection class.
Common mistakes
- Omitting `set -euo pipefail` and relying on each command to fail loudly. Most do not.
- Parsing `ls` output. Filenames may contain spaces and newlines; use globs or `find -print0`.
- Building shell commands by string concatenation in Python. Pass an argument list instead — it is both safer and clearer.
- Using `/tmp/myscript.$$` as a temporary path on a shared host. Use `mktemp`, which is race-free.
- Catching every exception and continuing. A script that swallows errors is indistinguishable from one that works.
- Testing only the success path. The failure path is the one that runs at 3am, and it is usually the one nobody exercised.
Frequently asked questions
What does set -euo pipefail actually do?
Three separate things. `-e` exits when a command fails instead of continuing with wrong state; `-u` treats an unset variable as an error rather than expanding it to an empty string; and `-o pipefail` makes a pipeline fail if any stage fails, not just the last one. Each closes a way for a script to report success after doing something wrong, and together they are a one-line change with a large effect.
Why does my script break on filenames with spaces?
Because an unquoted expansion is word-split by the shell, so one filename becomes several arguments. Quoting every expansion fixes it — `"$file"` rather than `$file` — and ShellCheck identifies every instance. This is the rare bug class you can eliminate entirely rather than merely reduce.
When should automation be Python instead of Bash?
When it needs data structures, error handling with context, or calls to APIs. Bash is excellent at running programs and connecting them together; it is poor at anything resembling a data model. A useful rule is that once you are parsing JSON or nesting more than a couple of conditionals, the script has outgrown the shell.
How do I make a script safe to re-run?
Check state before changing it, and write files atomically. Create the directory only if it does not exist, add the line only if it is absent, and write to a temporary file that you then rename over the target — rename is atomic, so an interrupted run leaves the old file intact rather than a half-written one. Then actually run it twice and confirm the second run changes nothing.
Is shell=True in Python subprocess ever acceptable?
Only with a fully static command string containing no interpolated input. As soon as a variable is inserted, you have a command-injection vector — and the argument-list form is usually clearer anyway, because it makes the boundaries between arguments explicit rather than leaving them to shell parsing.