🕵️ Auditing an Inherited Linux Server with AI: A Recon Playbook

At some point you’ll SSH into a server nobody understands. The person who built it left, there’s no documentation, and it’s running something important enough that nobody dares touch it. Your job is to figure out what it does, what depends on it, and what’s quietly on fire — all without breaking the thing while you poke at it. This is archaeology, and the cardinal rule is: read everything, change nothing, until you understand it.

This is one of my favorite uses of AI, because it’s pure read-only pattern-matching. I run a battery of inventory commands, dump the output, and let the model synthesize “here’s what this box appears to be and here’s what looks risky.” It’s a fast junior engineer reading a pile of evidence. It never gets credentials and never runs anything on the box — every command below, I run myself and feed it the output. The recon is non-destructive by design; keeping the AI strictly in the read-only seat keeps it that way.

Start with identity and uptime

Before anything, establish what you’re standing on:

cat /etc/os-release          # distro and version
uname -r                     # kernel
uptime                       # how long it's been up, load average
hostnamectl                  # hostname, virtualization, machine-id

A 900-day uptime tells you this box has never been patched or rebooted and is fragile. The OS version tells you whether you’re on something supported or a relic past end-of-life. Hand this to your assistant along with the question that frames the whole audit. I keep this recon prompt with my other linux admin prompts:

Here’s the os-release, kernel, and uptime for a server I just inherited with no documentation. Flag anything concerning about support status or patch hygiene, and tell me what to inventory next.

Inventory what’s actually running

The heart of the audit is figuring out what services exist and which are doing real work:

systemctl list-units --type=service --state=running
ss -tulpn                    # listening ports + owning process
systemctl list-unit-files --state=enabled

ss -tulpn is the most revealing single command — every listening port and the process behind it tells you what this box serves to the network. A mystery service on port 8443, a database listening on 0.0.0.0 (exposed!), an old telnetd — they all surface here. Pro Tip: Diff “enabled at boot” against “currently running.” A service enabled but not running may have crash-looped and given up; a service running but not enabled will vanish on the next reboot. Both are landmines on an inherited box, and AI is good at spotting the mismatch when you give it both lists.

Feed the ss and systemctl output to the AI: “Here are the listening ports and running services. Identify what each likely is, flag anything exposed to the network that shouldn’t be, and call out anything deprecated or suspicious.” It’ll recognize the common daemons instantly and flag the weird ones for you to investigate.

Find the hidden automation

The work a server does on a schedule is invisible until you go looking, and it’s where the real surprises hide:

for u in $(cut -f1 -d: /etc/passwd); do crontab -l -u "$u" 2>/dev/null; done
ls -la /etc/cron.* /etc/cron.d/
systemctl list-timers --all

Per-user crontabs, system cron, and systemd timers — you have to check all three because a job can live in any of them. This is how you discover the nightly script that rsyncs to a host that no longer exists, or the cron job that’s been emailing errors to a dead mailbox for two years. The incident response helper is handy when one of these jobs turns out to be the cause of a recurring 3 a.m. alert nobody traced.

Map users, sudo, and SSH access

Who can get in, and who can become root, is a security question you answer before you trust anything:

awk -F: '$3 >= 1000 || $3 == 0 {print $1, $3, $7}' /etc/passwd
getent group sudo wheel
sudo grep -rE '^[^#]' /etc/sudoers /etc/sudoers.d/
sudo ls -la /home/*/.ssh/authorized_keys

Look for UID-0 accounts other than root (a backdoor), shared accounts, and authorized_keys files full of keys belonging to people who left. Hand the AI the combined picture and ask it to flag the stale and the surprising. It reads the access list faster than you and names the outliers; you decide what to revoke — and you revoke it deliberately, never letting the model do it.

Check the health and the risks

Now assess what’s actually wrong. Read-only diagnostics first:

df -h && df -i               # disk and inode pressure
free -h                      # memory and swap
journalctl -p err -b --no-pager | tail -50   # recent errors
sudo dmesg -T | grep -iE 'error|fail|oom|i/o' | tail -30
last -20                     # recent logins

That journal and dmesg grep surfaces the quiet failures — OOM kills, disk I/O errors, failed services. Dump it all to the AI for a synthesis: “Here’s disk, memory, recent errors, and dmesg. Summarize the health of this box and rank the top three risks.” This is exactly the pattern-matching it excels at, and it’ll produce a prioritized list far faster than you’d assemble it by hand. The code review tool and monitoring alerts helper help you turn the findings into reviewed remediation scripts and alert rules once you know what you’re dealing with.

Write it down before you touch anything

The output of an audit is documentation — the thing that was missing in the first place. As you go, have the AI draft a server profile from your collected output: what it runs, what listens, what’s scheduled, who has access, and the ranked risks. That draft becomes the runbook the next person (or future you) actually has. Refine it in the prompt workspace and save the recon command battery in the prompt packs and prompts library so the next mystery box is a checklist, not a cold start.

Only after you understand the machine do you start changing it — and every change is a human-reviewed, deliberate action, ideally on a snapshot first. The whole audit succeeds precisely because AI stayed in the read-only seat: it synthesized the evidence, it never held credentials, and it never ran a command that could break the very thing you were trying to understand.

Conclusion

Inheriting an undocumented server is detective work, and the rule is read everything before you change anything. Inventory identity, running services, hidden automation, access, and health — all with non-destructive commands — then let AI synthesize the pile of output into a clear picture and a ranked risk list. It’s superb at that read-only pattern-matching. Keep it out of the credential and execution path entirely, document what you find, and only then start fixing. The first artifact of a good audit is the documentation that should have existed all along.