📋 Parsing Logs with Bash and Python: A Practical Guide

Half of operations work is reading logs, and most of that is extracting a few useful numbers from a mountain of lines. Over 25 years I have written more log parsers than I can count, and the main lesson is knowing which tool to reach for: a one-line grep for a quick answer, awk for columns, and Python when the format gets gnarly or you need real data structures.

Here is the progression I actually use, from fastest-to-type to most robust.

Level 1: grep for the quick answer

When you just need to find or count something, nothing beats grep:

# count errors in the last log
grep -c 'ERROR' app.log

# show errors with 2 lines of context
grep -B2 -A2 'ERROR' app.log

# errors but not the noisy known-benign one
grep 'ERROR' app.log | grep -v 'connection reset'

grep -c for counts, -A/-B for context, and a grep -v to filter known noise. This answers “is it happening and how often” in seconds.

Level 2: awk for columns and math

The moment you need a specific field or a calculation, switch to awk. For space-delimited logs it is unbeatable:

# extract the 7th field (e.g. response time) and average it
awk '{ sum += $7; n++ } END { print sum/n }' access.log

# count requests per status code (status in field 9)
awk '{ count[$9]++ } END { for (c in count) print c, count[c] }' access.log

# requests slower than 1000ms
awk '$7 > 1000 { print $1, $7 }' access.log

That status-code histogram is something I run constantly during an incident. awk does the field splitting, the accumulation, and the math in one pass over the file — fast even on multi-gigabyte logs.

Level 3: sort | uniq for top-N

The classic pipeline for “what are the most common X” deserves its own mention because you will use it weekly:

# top 10 IPs by request count
awk '{ print $1 }' access.log | sort | uniq -c | sort -rn | head

# most common error messages
grep ERROR app.log | sed 's/[0-9]//g' | sort | uniq -c | sort -rn | head

That second one strips digits with sed first, so messages that differ only by an ID or timestamp collapse together and the real frequency leaders surface. This pipeline finds the noisy error or the abusive client in seconds.

Level 4: Python when the format fights back

Multi-line stack traces, JSON logs, fields with embedded spaces, or anything where you need real data structures — that is where bash runs out of road and Python takes over.

For structured JSON logs, Python is the obvious choice:

import json
from collections import Counter

status_counts = Counter()
slow = []

with open("app.jsonl") as f:
    for line in f:
        try:
            rec = json.loads(line)
        except json.JSONDecodeError:
            continue  # skip malformed lines, don't crash
        status_counts[rec["status"]] += 1
        if rec.get("duration_ms", 0) > 1000:
            slow.append(rec)

print(status_counts.most_common())
print(f"{len(slow)} slow requests")

The try/except around json.loads is the detail people forget — real log files always contain at least one truncated or malformed line, and a parser that crashes on line 40,000 is useless. Skip bad lines, keep going.

Regex parsing for unstructured logs

When the format is text but consistent, a named-group regex turns each line into a dict:

import re

pattern = re.compile(
    r'(?P<ip>\S+) \S+ \S+ \[(?P<time>[^\]]+)\] '
    r'"(?P<method>\S+) (?P<path>\S+)[^"]*" '
    r'(?P<status>\d+) (?P<size>\d+)'
)

for line in open("access.log"):
    m = pattern.match(line)
    if not m:
        continue
    rec = m.groupdict()
    if rec["status"].startswith("5"):
        print(rec["time"], rec["path"])

Named groups (?P<name>) keep the regex readable and give you a clean dict per line. Once you are in Python you can aggregate, join against other data, or emit metrics — things bash makes painful.

Where AI genuinely shines

Writing the regex for a log format is the single most tedious part of log parsing, and it is where I lean on AI hardest. I paste two or three sample lines and ask:

“Here are three sample log lines. Write a Python regex with named groups that captures timestamp, level, service, and message. Handle the case where the message itself contains brackets. Show how to use it to count messages per level.”

Giving it real sample lines is the trick — the model matches the actual format instead of guessing. I always test the generated regex against a handful of real lines, including the weird ones, before trusting it. I keep a couple of these log-parsing prompts in my prompt library.

AI is also excellent at the reverse: paste a confusing log excerpt and ask it to explain what the entries mean and what is likely wrong. That is reading comprehension, which models do well.

Picking the right level

• Need to find or count something? grep.
• Need a field or a calculation? awk.
• Need top-N? sort | uniq -c | sort -rn.
• Multi-line, JSON, or needs data structures? Python.

Start at the cheapest level that answers the question. Most of the time grep or awk is enough, and you save Python for the genuinely structured work. For more parsing patterns and AI prompts, see our automation guides.