Python Structured Logging for Automation Scripts Prompt
Add production-grade structured logging to a Python automation script — JSON logs, correlation IDs, levels, redaction of secrets, and rotation — so cron and systemd runs are debuggable after the fact.
- Target user
- Engineers making batch scripts observable in shared log pipelines
- Difficulty
- Intermediate
- Tools
- Claude, ChatGPT
The prompt
You are a senior engineer who has spent too many on-call shifts grepping `print()` output that lost the one detail you needed. Add structured, production-grade logging to a Python automation script. I will provide: - The script and how it currently logs (prints, ad-hoc logging, nothing) - Where logs land (stdout for systemd/journald, a file, a shipper like Loki/ELK) - What sensitive values flow through it (tokens, PII, connection strings) - Whether multiple instances or threads run concurrently Your job: 1. **Choose the setup** — configure the stdlib `logging` module via `dictConfig`; emit JSON when shipped to an aggregator and human-readable lines for interactive runs, selected by a `--log-format` flag or TTY detection. Recommend `structlog` only if the gains justify the dependency, and explain the tradeoff. 2. **Make every line queryable** — include timestamp (ISO 8601 UTC), level, logger name, message, and structured fields (not interpolated into the message string). For systemd, log to stdout and let journald capture it rather than managing files. 3. **Correlation IDs** — generate a run ID at startup and attach it (plus relevant entity IDs) to every record via a `LoggerAdapter` or `contextvars`, so all lines from one run — and one work item — are trivially filterable. 4. **Levels with intent** — DEBUG for wire details, INFO for lifecycle milestones, WARNING for handled-but-notable, ERROR for failed work items, CRITICAL for aborting. Show how `--verbose`/`-v` flags map to levels. 5. **Redact secrets** — install a logging filter that masks known sensitive keys and pattern-matches tokens/passwords before they hit any handler, so a careless `logger.info(config)` cannot leak credentials. This is non-negotiable. 6. **Rotation (file path)** — if writing files, use `RotatingFileHandler`/`TimedRotatingFileHandler` with size/age caps, or hand off to `logrotate`; never let an automation log grow unbounded on a small disk. 7. **Exceptions** — log tracebacks with `logger.exception` inside handlers, attach the run/item context, and ensure a top-level handler logs a structured fatal record with a non-zero exit. 8. **Bash interop** — for mixed pipelines, show a matching JSON log helper in Bash so shell and Python lines share fields and the run ID propagates via env var. Output: (a) the `dictConfig` and JSON formatter, (b) the redaction filter with tests proving secrets never appear, (c) the correlation-ID adapter, (d) before/after of one noisy script, (e) a query cheatsheet for the chosen aggregator. Be opinionated: structured fields over string interpolation, stdout for systemd, and redaction enabled by default.