📊 Ansible Callback Plugins for Logging and Observability

Last month I sat watching a playbook crawl for forty minutes. It rolled out a fleet of app servers, and the terminal just dribbled green ok lines while I sipped cold coffee and wondered which of the thirty-odd tasks was eating my afternoon. I had no idea. The default Ansible stdout gives you a flat wall of identical-looking output, and “it’s slow” is not a data point you can act on. Then I enabled profile_tasks, re-ran, and there it was in black and white: one apt update task was burning 31 of those 40 minutes because of a misconfigured proxy. Five minutes of config gave me an answer that staring at the screen never would.

That is the whole pitch for callback plugins. Ansible already emits a stream of events for every play, task, and result. Callback plugins are the hooks that decide what happens with that stream. Out of the box you get a readable terminal. With a little configuration you get profiling, structured logs, and run results piped to wherever your team actually looks. Here is how I set them up, and how I let AI scaffold the custom ones without ever handing it the keys.

Turning callbacks on in ansible.cfg

Everything starts in ansible.cfg. There are two flavors of callback. The stdout callback owns your terminal output, and you can only have one. Notification and aggregate callbacks run alongside it, and you can enable as many as you like.

[defaults]
stdout_callback = yaml
callbacks_enabled = profile_tasks, timer, ansible.posix.profile_roles

[ansible_runner]
# nothing here yet, just noting the section exists

A couple of gotchas worth knowing. The setting was renamed: older docs and tutorials use callback_whitelist, but on any modern Ansible you want callbacks_enabled. They do the same thing; pick the new name. The other one bites everyone at least once. By default, enabled callbacks only fire when you run a playbook, not when you run an ad-hoc ansible command. If you want callbacks active for ad-hoc runs too, you need:

[defaults]
bin_ansible_callbacks = true

Pro Tip: Keep ansible.cfg in version control next to your playbooks and let Ansible pick it up from the project directory. A callback config that lives only on one engineer’s laptop is a debugging session nobody else can reproduce.

Profiling with profile_tasks and timer

profile_tasks is the one I reach for first, every time. It prints a running timestamp next to each task and, at the end of the run, a sorted leaderboard of the slowest tasks. The timer callback is its tiny sibling; it just prints total wall-clock time at the bottom. Enable both:

[defaults]
callbacks_enabled = profile_tasks, timer

Now the bottom of every run looks like this:

Monday 16 June 2026  09:14:52 +0000 (0:00:31.402)  0:38:11.908 ********
===============================================================================
apt : update package cache ------------------------------------- 1860.21s
nginx : install nginx ------------------------------------------- 12.04s
app : deploy release artifact ----------------------------------- 9.88s

That first line told me immediately my “slow playbook” was one cache update. No guessing, no bisecting. You can tune how many tasks it lists with the ANSIBLE_PROFILE_TASKS_TASK_OUTPUT_LIMIT environment variable if the default twenty is too noisy.

Readable diffs with the yaml stdout callback

The default stdout callback crams multi-line strings, registered variables, and --diff output onto escaped single lines. It is unreadable for anything non-trivial. The yaml stdout callback (built in, part of ansible.builtin) re-renders all of that as proper block YAML.

[defaults]
stdout_callback = yaml

Run with --diff and instead of \n-littered noise you get a clean before/after block you can actually read in a review. If you want even more control over the format, community.general.diy lets you template the output yourself with Jinja2, down to per-event strings. I rarely need that level of customization, but it is there when a compliance team wants output in a specific shape.

A persistent run log with log_plays

Terminal output scrolls away. For anything you might need to explain later, the built-in log_plays callback writes every task result to a file. It is the cheapest audit trail you will ever set up.

[defaults]
callbacks_enabled = log_plays

[callback_log_plays]
log_folder = /var/log/ansible/hosts

It writes one log file per host under that folder. One caveat that matters: log_plays records task results verbatim, which means it will happily write whatever a module returned, secrets included. Make sure any task that touches credentials is marked no_log: true so its results never reach the log. More on that below, because it is the single most important habit on this page.

Writing a custom callback that posts failures to a webhook

The built-ins cover profiling and logging. What they do not do is push a failure into Slack or PagerDuty the moment it happens. That is a custom callback, and it is genuinely small. Drop a Python file in a callback_plugins/ directory next to your playbook (or any path on ANSIBLE_CALLBACK_PLUGINS), and Ansible loads it automatically.

# callback_plugins/webhook_notify.py
from __future__ import annotations

import json
import os
import urllib.request

from ansible.plugins.callback import CallbackBase


class CallbackModule(CallbackBase):
    """Post task failures and a run summary to a webhook."""

    CALLBACK_VERSION = 2.0
    CALLBACK_TYPE = "notification"
    CALLBACK_NAME = "webhook_notify"
    # requires callbacks_enabled in ansible.cfg
    CALLBACK_NEEDS_ENABLED = True

    # keys we never forward, even if a module returns them
    REDACT = ("password", "token", "secret", "ansible_password", "key")

    def __init__(self):
        super().__init__()
        self.url = os.environ.get("ANSIBLE_WEBHOOK_URL")
        self.failures = 0

    def _scrub(self, result):
        clean = {}
        for k, v in result.items():
            if any(s in k.lower() for s in self.REDACT):
                clean[k] = "********"
            else:
                clean[k] = v
        return clean

    def _post(self, payload):
        if not self.url:
            return
        data = json.dumps(payload).encode("utf-8")
        req = urllib.request.Request(
            self.url, data=data, headers={"Content-Type": "application/json"}
        )
        try:
            urllib.request.urlopen(req, timeout=5)
        except Exception as exc:  # never let the notifier break the run
            self._display.warning(f"webhook_notify failed: {exc}")

    def v2_runner_on_failed(self, result, ignore_errors=False):
        if ignore_errors:
            return
        self.failures += 1
        task = result._task
        # respect no_log: do not forward results the author marked secret
        if task.no_log:
            payload = {"task": task.get_name(), "result": "redacted (no_log)"}
        else:
            payload = {
                "task": task.get_name(),
                "host": result._host.get_name(),
                "result": self._scrub(result._result),
            }
        self._post({"event": "task_failed", **payload})

    def v2_playbook_on_stats(self, stats):
        summary = {}
        for host in sorted(stats.processed.keys()):
            summary[host] = stats.summarize(host)
        self._post({"event": "run_complete", "failures": self.failures, "stats": summary})

Three things make this a real plugin and not a toy. The class must be named CallbackModule and subclass CallbackBase. The CALLBACK_VERSION, CALLBACK_TYPE, and CALLBACK_NAME attributes are how Ansible classifies and loads it; the name must match what you put in callbacks_enabled. And the v2_runner_on_failed / v2_playbook_on_stats hooks are the event names Ansible calls. There are dozens more (v2_runner_on_ok, v2_playbook_on_task_start, and so on), but these two give you per-failure alerts plus a final summary, which is most of what an on-call rotation actually wants.

Enable it like any other:

[defaults]
callbacks_enabled = profile_tasks, timer, webhook_notify

Scrub secrets before they leave the building

Read the plugin above again and notice what it does not do. It checks task.no_log and refuses to forward those results. It runs every other result through _scrub to mask anything that looks like a credential. And it swallows its own exceptions so a flaky webhook can never fail your deployment.

This is not optional polish. A callback plugin sees every result of every task, in full, before you do. If a module returns a generated password or a registered variable holds a token, your naive logger or notifier will broadcast it to a log file, a chat channel, or a third-party endpoint. The defense is layered: mark sensitive tasks no_log: true at the playbook level and redact defensively inside the callback. Belt and suspenders, because the cost of getting this wrong is a secret in a log you can never fully claw back.

Pro Tip: Test secret scrubbing by deliberately running a task that returns a fake token, then grep your log file and webhook payloads for it. If you can find the fake secret, an attacker can find a real one.

Where AI fits, and where it does not

I let AI write the first draft of every custom callback I build. It is a fast junior engineer: it knows the CallbackBase interface, remembers the exact hook names I always forget, and scaffolds the urllib boilerplate in seconds. Paste the requirement into Claude or your editor’s assistant in Cursor, and you get a working skeleton far faster than typing it yourself. For the Jinja2 templating in community.general.diy, a quick prompt from a saved prompt pack beats hunting through docs.

But the junior engineer does not get the vault keys, and it does not get my trust by default. Every AI-generated callback goes through the same gate: I read every line, then run the playbook with --check first so a broken plugin or a leaky payload shows up against nothing real. I specifically verify the no_log handling and the redaction logic myself, because that is exactly the kind of safety detail an LLM will cheerfully omit when it is focused on “make the webhook fire.” Treat the output like any other untrusted change. A second pair of eyes, human or an automated code review pass, catches the leak the first draft missed. If you are systematizing this, the IaC category collects more of these patterns.

Wrapping up

Callback plugins turn Ansible from a thing that runs into a thing you can observe. profile_tasks and timer tell you where the time goes, yaml makes diffs readable, log_plays gives you an audit trail, and a thirty-line custom plugin pushes failures to wherever your team lives. Let AI scaffold the custom ones, then do the part it cannot: read it, dry-run it, and make absolutely sure no secret ever rides out on the event stream. The forty-minute mystery playbook is a problem you only have to solve once.