Ansible Drift Detection with Check Mode Prompt

You are a senior automation engineer who has implemented drift detection — periodic check runs, diff comparisons, alerting on changes outside Ansible.

I will provide:
- Drift detection use case
- Current verification approach
- Goal

Your job:

1. **Check mode (`--check`)**:
   - Dry run; reports what would change
   - Module-by-module
   - Doesn't modify
2. **Diff mode (`--diff`)**:
   - Shows actual content changes (templates, files)
   - Combine with --check
3. **For periodic drift runs**:
   - Cron / systemd timer / Ansible Tower schedule
   - Output to log / monitoring
   - Alert if changed > 0
4. **For drift sources**:
   - Manual SSH edits
   - Out-of-band tools
   - Failed previous runs
5. **For drift response**:
   - Apply Ansible to reconcile
   - Investigate why drift occurred
   - Block manual access if needed
6. **For per-host drift**:
   - Run per inventory
   - Track drift trends
7. **For policy enforcement**:
   - Drift = policy violation
   - Alert security on critical drift
8. **For compliance reporting**:
   - Aggregate drift over time
   - Per-control basis (CIS, etc.)

Mark DESTRUCTIVE: drift "reconcile" wiping intentional manual changes, drift alerts so noisy they're ignored, drift detection in production without check mode.

---

Use case: [DESCRIBE]
Current verification: [DESCRIBE]
Goal: [DESCRIBE]

Why this prompt works

Drift undermines IaC. This prompt walks detection.

How to use it

1. Periodic —check runs.
2. Alert on changed > 0.
3. Per-host trends.
4. Block manual access if drift persistent.

Useful commands

# Check + diff
ansible-playbook site.yml --check --diff

# Output JSON for parsing
ansible-playbook site.yml --check --diff -i inventory > /tmp/drift.txt 2>&1

# Per-host
for HOST in $(ansible-inventory --list | jq -r 'keys[]'); do
    ansible-playbook --check --diff --limit $HOST site.yml > /tmp/drift-$HOST.log 2>&1
done

# Count changes
grep -c "changed=" /tmp/drift.txt

Patterns

Cron-driven drift check

# /etc/cron.daily/ansible-drift
#!/bin/bash
cd /opt/ansible
ansible-playbook site.yml --check --diff -i inventories/production > /var/log/ansible-drift-$(date +%F).log 2>&1

# Count changes
CHANGES=$(grep "changed=" /var/log/ansible-drift-$(date +%F).log | \
    grep -v "changed=0" | wc -l)

if [ "$CHANGES" -gt 0 ]; then
    # Send alert
    curl -X POST -H "Content-Type: application/json" \
        -d "{\"text\":\"Drift detected: $CHANGES hosts\"}" \
        $SLACK_WEBHOOK
fi

Systemd timer

# /etc/systemd/system/ansible-drift.service
[Unit]
Description=Ansible drift check

[Service]
Type=oneshot
User=ansible
WorkingDirectory=/opt/ansible
ExecStart=/usr/bin/ansible-playbook site.yml --check --diff -i inventories/production
StandardOutput=file:/var/log/ansible-drift.log

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/ansible-drift.timer
[Unit]
Description=Ansible drift check daily

[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true

[Install]
WantedBy=timers.target

Prometheus metrics from drift

#!/bin/bash
# Parse output and write Prometheus textfile

OUTPUT=$(ansible-playbook --check site.yml 2>&1)
CHANGES=$(echo "$OUTPUT" | grep "changed=" | awk '{print $2}' | cut -d= -f2 | paste -sd+ | bc)
FAILURES=$(echo "$OUTPUT" | grep "failed=" | awk '{print $4}' | cut -d= -f2 | paste -sd+ | bc)

cat > /var/lib/node_exporter/textfile/ansible_drift.prom <<EOF
# HELP ansible_drift_changes_total Number of tasks that would change
# TYPE ansible_drift_changes_total gauge
ansible_drift_changes_total $CHANGES

# HELP ansible_drift_failures_total Number of task failures
# TYPE ansible_drift_failures_total gauge
ansible_drift_failures_total $FAILURES

# HELP ansible_drift_last_run_timestamp Last drift check
# TYPE ansible_drift_last_run_timestamp gauge
ansible_drift_last_run_timestamp $(date +%s)
EOF

Per-task drift detail

- name: Configure nginx
  template:
    src: nginx.conf.j2
    dest: /etc/nginx/nginx.conf
  register: nginx_drift
  check_mode: true                      # always check mode

- name: Report drift
  debug:
    msg: "Nginx config drifted: {{ nginx_drift.changed }}"
  when: nginx_drift.changed

Common findings this catches

• Drift in custom shell tasks not detected → improve idempotency.
• Drift with secrets in diff output → sanitize.
• No alerting on drift → integrate with notification.
• Drift reconciled wiping intentional changes → audit changes before apply.
• Drift counts unstable → false positives in idempotency.
• Per-host drift untracked → split log per host.
• Drift trend rising → manual access issue.

When to escalate

• Drift policy across org — coordinate.
• Compliance evidence collection — security.
• Drift response automation — engineering.