🕵️ Linux auditd: Tracking Who Did What on Your Servers

Sooner or later you hit the question that /var/log can’t answer: who deleted that file, who edited sudoers, who read the customer database export at 3am. Shell history lies, gets cleared, and only covers interactive sessions. The kernel audit subsystem — auditd — doesn’t. It records syscall-level events straight from the kernel, before any userspace tool can hide them. If you’re under any kind of compliance regime (PCI, HIPAA, SOC 2), it’s not optional. Even if you’re not, it’s the difference between knowing and guessing.

The catch is that auditd, misconfigured, generates a firehose of useless events that buries the one you needed. The skill is rules, not installation.

How auditd is wired

There are three pieces: the kernel audit subsystem (always present), the auditd daemon that writes events to /var/log/audit/audit.log, and the rules that tell the kernel what to record. You manage rules with auditctl (live, ephemeral) and /etc/audit/rules.d/*.rules (persistent). You read events with ausearch and summarize them with aureport.

sudo dnf install audit   # or: sudo apt install auditd
sudo systemctl enable --now auditd
sudo auditctl -s         # status: enabled, backlog, lost events

Two kinds of rules

Watches monitor a file or directory for access:

# -w path -p permissions(read/write/execute/attribute) -k key
sudo auditctl -w /etc/sudoers -p wa -k sudoers_changes
sudo auditctl -w /etc/passwd -p wa -k identity
sudo auditctl -w /etc/ssh/sshd_config -p wa -k sshd_config

The -k key is a label you’ll search by later — choose names you’ll recognize at 2am.

Syscall rules trigger on a system call matching filters:

# Log every use of these privilege-changing syscalls by non-system users
sudo auditctl -a always,exit -F arch=b64 -S execve -F euid=0 -k root_exec

A starter ruleset that isn’t noise

Put this in /etc/audit/rules.d/hardening.rules. It covers the things auditors and incident responders actually ask about, without logging every read() on the box:

# Identity and auth files
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/sudoers -p wa -k privilege
-w /etc/sudoers.d/ -p wa -k privilege

# SSH and PAM config
-w /etc/ssh/sshd_config -p wa -k sshd
-w /etc/pam.d/ -p wa -k pam

# Scheduled jobs
-w /etc/cron.d/ -p wa -k cron
-w /etc/crontab -p wa -k cron

# Privilege escalation
-a always,exit -F arch=b64 -S execve -F euid=0 -F auid>=1000 -F auid!=4294967295 -k priv_esc

# Loading/unloading kernel modules
-a always,exit -F arch=b64 -S init_module -S delete_module -k modules

# Lock the config so rules can't be changed without an audit gap
-e 2

That -e 2 at the end makes the rules immutable until reboot — an attacker can’t quietly disable auditing without a reboot, which is itself a loud event. Load it:

sudo augenrules --load
sudo auditctl -l   # confirm rules are active

Reading the log without going blind

Raw audit.log is barely human-readable. Use the tools:

# Everything tagged 'privilege', resolved to readable names
sudo ausearch -k privilege -i

# All events by a specific login user (audit UID survives su/sudo)
sudo ausearch -ua 1001 -i

# A summary report of authentication events
sudo aureport -au

# Failed events in the last day
sudo ausearch --start yesterday -m all -sv no -i

The key concept is auid — the audit/login UID. It’s set at login and follows the user through su and sudo, so even after someone becomes root, the events still trace back to the human who logged in. That’s the property that makes auditd worth the trouble.

Where AI earns its keep

I’ll be blunt: auditd’s filter syntax and the structure of audit.log are genuinely unpleasant. This is one of the best places I know to put a language model to work. Two patterns I use constantly:

1. Writing rules from intent. “Give me an audit rule that logs any write to files under /etc/kubernetes/” — the model produces the -w line and explains the -p flags.
2. Explaining a raw event. Paste a multi-line type=SYSCALL ... type=PATH record and ask what actually happened, in English. The model decodes the a0/a1/a2 argument fields and the syscall name faster than I can look them up.

Scrub hostnames and usernames before pasting log lines anywhere, the same as you would with any security data. Keep a few auditd analysis prompts ready and you turn an opaque log format into plain answers.

Operational gotchas

• Watch the backlog. Under heavy load the audit buffer can fill and the kernel loses events. Check auditctl -s for a rising lost counter and raise --backlog_limit if needed.
• Don’t audit everything. A rule on open syscalls system-wide will generate gigabytes an hour and hide the signal. Audit the sensitive paths, not the world.
• Ship the logs off-box. An attacker with root can fill or rotate local logs. Forward to a remote collector so the record survives the host.
• Disk space is real. Configure max_log_file and max_log_file_action in /etc/audit/auditd.conf so a full audit partition doesn’t take the box down.

Configured with restraint, auditd answers the “who did what” question definitively, and it’s the log an attacker can’t quietly rewrite. Start with the starter ruleset, learn ausearch -k, and you’ll never again have to guess who touched the box. More server-hardening guides are in the AI for Linux Admins collection.

Audit configuration affects security posture and disk usage. Validate rules in a test environment and monitor for lost events before relying on auditd for compliance evidence.