What actually breaks, and what to check first
Infrastructure as code fails differently from application code, because the failure usually is not a crash. It is a *successful* run that produced the wrong state — a playbook that reports "ok" while having changed nothing useful, a chart that renders valid YAML describing a broken Deployment, a module that applies cleanly and quietly widens a security group.
That is why IaC review has to check behaviour rather than syntax. A linter tells you the file parses. It does not tell you the task is not idempotent, that the template silently dropped a value because the variable was undefined, or that a default in a module three levels down opened a port.
The unifying discipline across Ansible, Helm and Terraform is the same: render before you apply, diff against what exists, and make the destructive paths loud.
Triage order
Work down this list in order. Each step either finds the fault or rules out a whole class of cause — a negative result is progress, not a wasted step.
-
Does the rendered output match what you think you wrote?
helm template . -f values.yaml | ansible-playbook --check --diff | terraform planEvery IaC tool has a render-without-applying mode. Most "mysterious" IaC bugs are visible in the rendered output — a value that resolved to empty, an indentation error that reparented a key, a conditional that evaluated the other way.
-
Is the run idempotent?
Run it twice. The second run should report zero changes.A second run that still reports changes means a task is not converging — usually a `command`/`shell` task with no `creates`/`changed_when` guard. Non-idempotent IaC is unsafe to re-run, which removes your main recovery tool during an incident.
-
Did an undefined variable render as empty rather than failing?
Search the rendered output for empty values: `helm template . | grep -nE ":\s*(\"\"|null|~)?$"`Templating engines are permissive by default. An undefined value becomes an empty string, producing valid YAML that describes something wrong. Ansible has `--check` plus strict undefined; Helm has `required`. Use them.
-
Are secrets in the repository?
git grep -nEi "(password|secret|token|api[_-]?key)\s*[:=]" -- ":!*.example"A hit that is not a placeholder is an incident, not a code-review comment. Rotate the credential first — it is in history and history is distributed — then move it to a secret manager.
-
What does this change do to blast radius?
Diff the rendered output against the current live state, not against the previous file version.File diffs hide the impact of changed defaults and inherited values. Rendered diffs show what will actually change in the cluster or cloud, which is the only diff that predicts an outage.
Diagnostic commands
Every command is labelled by what it can do to the system. Read-only commands are safe to run during an incident; the others are not, and are marked accordingly.
ansible-playbook site.yml --check --diff Dry-run a playbook and show the file-level changes it would make.
How to read it: Tasks reporting `changed` in check mode are the ones to review. Modules that cannot predict changes are skipped in check mode — those are your blind spots and deserve a manual read.
ansible-lint site.yml Catch non-idempotent patterns and deprecated syntax.
How to read it: Prioritise the idempotency rules over style rules. A `command` task with no `creates` guard is a real defect; a naming convention warning is not.
helm template release . -f values.yaml --debug Render a chart to final YAML without touching the cluster.
How to read it: This is the manifest Kubernetes will receive. Read it. Most chart bugs are obvious here and invisible in the templates.
helm diff upgrade release . -f values.yaml Show what an upgrade changes against what is actually deployed.
How to read it: Requires the helm-diff plugin, and is the single highest-value addition to a Helm workflow. Watch for changes to volume claims and stateful sets — those are the ones that cause data loss.
kubectl apply --dry-run=server -f manifest.yaml Validate a manifest against the live API server, including admission control.
How to read it: Server-side dry run catches what client-side validation cannot: admission webhooks, quota rejections and CRD schema violations.
helm rollback release <revision> Return a release to a previous revision.
How to read it: Rolls back the manifest, not the data. A rollback across a schema migration will leave the application pointing at a database it no longer understands.
Failure modes
These are distinct problems, not variations of one. Matching the symptom to the right cause is most of the work.
A playbook reports "changed" on every run, forever.
- Cause:
- A `command` or `shell` task with no idempotency guard, so Ansible cannot tell whether the work was already done.
- Fix:
- Add `creates:`/`removes:` to the task, or a `changed_when:` expression derived from the output. Prefer a purpose-built module over `shell` wherever one exists.
Helm renders valid YAML that Kubernetes rejects.
- Cause:
- A value was undefined and rendered as empty, producing a structurally valid but semantically invalid manifest.
- Fix:
- Wrap required values in `{{ required "message" .Values.x }}` so rendering fails loudly at template time rather than silently at apply time.
A module bumps a version and unrelated resources are replaced.
- Cause:
- A changed default inside the module altered a ForceNew attribute.
- Fix:
- Read the rendered plan, not the module changelog. Pin module versions and treat a module upgrade as a change requiring the same plan review as a resource change.
The pipeline applies successfully but the change has no effect.
- Cause:
- The wrong target — wrong Ansible inventory, wrong Helm namespace, wrong Terraform workspace.
- Fix:
- Echo the resolved target at the start of every pipeline run. This is a two-line change that eliminates an entire category of confusing incidents.
Secrets appear in CI logs.
- Cause:
- A debug task or `--diff` printed a templated file containing a credential.
- Fix:
- Mark the task `no_log: true`, and register the CI variable as masked. Then rotate whatever was printed — CI logs are retained and frequently readable by more people than the secret store.
Common mistakes
- Treating "the linter passes" as "the change is safe". Linters check form; only a rendered diff checks effect.
- Running Ansible with `--check` and assuming full coverage. Modules that cannot predict changes are silently skipped.
- Storing environment differences in duplicated files rather than in values/variables, so environments drift apart one copy-paste at a time.
- Helm-rolling back across a database migration and expecting the data layer to follow.
- Writing IaC that is only safe to run once. If it is not safe to re-run, it is not usable during an incident, which is exactly when you need it.
Frequently asked questions
What makes an Ansible task idempotent?
That running it twice produces the same end state and reports no change the second time. Purpose-built modules are idempotent by design; `command` and `shell` are not, because Ansible has no way to know what your command did. Guard them with `creates:` or `changed_when:` so they can report accurately.
Why does my Helm chart render fine but fail to deploy?
Because `helm template` only validates that the output is YAML — it never contacts the cluster. Admission webhooks, resource quotas, CRD schemas and RBAC all reject at apply time. `kubectl apply --dry-run=server` runs the manifest through the real API server and catches those.
How should secrets be handled in infrastructure code?
They should not be in it. Reference them from a secret manager or inject them as environment variables at run time, and keep only the reference in the repository. If a secret has already been committed, rotation comes first: Git history is distributed, so removing the file does not un-share the credential.
Should each environment have its own copy of the code?
No — that is how environments drift. Keep one code path and vary the inputs, so a fix applied in staging is structurally the same change that reaches production. Duplicated directories per environment guarantee that they diverge, and the divergence is discovered during an incident.