Python Log Parser & Anomaly Extractor Prompt
Build a Python tool that streams large or rotated log files, parses structured and semi-structured lines, aggregates error patterns, and emits a summary report without loading everything into memory.
- Target user
- DevOps engineers and SREs analyzing log data
- Difficulty
- Intermediate
- Tools
- Claude, ChatGPT
The prompt
You are a senior Python engineer who writes memory-efficient log analysis tools. The input may be gigabytes, gzip-rotated, and partly malformed — handle that gracefully. I will provide: - A representative sample of the log format(s), including any timestamp and severity convention - What I want extracted (error rates, top exceptions, slow requests, a specific event) - Output target (terminal table, JSON, CSV) and whether the source is a file, a glob of rotated files, or stdin Your job: 1. **Stream, don't slurp** — iterate line by line (transparently opening `.gz` via `gzip`), so memory stays flat regardless of file size. Support reading from stdin so it composes in a pipeline. 2. **Parse defensively** — use a compiled regex or a structured (JSON-line) parser, and count/skip unparseable lines into a "malformed" bucket instead of crashing. 3. **Aggregate efficiently** — use `collections.Counter`/`defaultdict` for top-N patterns, normalizing variable parts (IDs, timestamps, numbers) so similar messages group together. 4. **Add time + severity filters** — let the user scope by `--since/--until` and minimum level via CLI flags. 5. **Emit clean output** — a human table by default, `--json` for machines; include totals, the malformed count, and the time span actually covered. 6. **Be honest about gaps** — surface when the requested window exceeds the data present. Output: the full script with `argparse`, the regex/parse strategy explained, a sample run against the provided log, and a note on how it stays O(1) in memory.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Asyncio TaskGroup Structured-Concurrency Python Script Prompt
Write a Python 3.11+ automation script that fans work out with asyncio.TaskGroup so failures cancel siblings cleanly, errors aggregate into an ExceptionGroup, and no task is ever silently orphaned.
-
Convert a Fragile Bash Pipeline to Python Prompt
Rewrite an overgrown, fragile Bash script full of pipes, subshells, and string parsing into a maintainable Python program with proper error handling, types, and tests.
-
Add Pytest Coverage to an Automation Script Prompt
Wrap an existing Python automation script in a pytest suite that mocks subprocess, network, filesystem, and clock side effects so the logic is tested fast and deterministically without touching real systems.
-
Idempotent Bulk File Rename & Migrate Tool Prompt
Write a Python tool that safely renames or relocates large batches of files using a rule set, with mandatory dry-run, collision detection, atomic moves, and an undo manifest so a bad run is reversible.
More Bash & Python Automation prompts & error guides
Browse every Bash & Python Automation prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.