🗄️ Linux Backup and Restore with rsync and Borg (Done Right)

Everybody has backups until the day they need one. Then they discover the cron job had been failing silently for eight months, or the backup ran fine but nobody ever tested a restore, or the encryption passphrase died with the person who set it up. A backup you’ve never restored from isn’t a backup — it’s a hopeful guess. I’ve watched smart teams lose data not because they didn’t back up, but because they never closed the loop.

AI has made the building of a solid backup system much faster for me. It drafts the rsync and Borg invocations, the retention math, the systemd timer — all the fiddly parts I’d otherwise crib from old scripts. It’s a fast junior engineer for the scaffolding. What it can’t do, and must never do, is run the backup against my production data with credentials of its own. Backups touch your most sensitive data; the model drafts, a human reviews and runs.

rsync for mirrors, Borg for history

These solve different problems and people conflate them. rsync makes a copy — great for mirroring a directory to another host or a cheap second disk, fast and simple. But a plain rsync mirror has no history: if a file gets corrupted and you sync, you’ve now mirrored the corruption. Borg (BorgBackup) gives you deduplicated, compressed, encrypted, versioned snapshots — you can restore yesterday’s version even after today’s got mangled.

A solid rsync mirror:

rsync -aAX --delete --info=progress2 /srv/ /mnt/backup/srv/

-a preserves permissions and timestamps, -A ACLs, -X extended attributes, --delete keeps the mirror in sync. For versioned backups, Borg:

borg init --encryption=repokey /mnt/backup/borgrepo
borg create --stats --compression zstd \
  /mnt/backup/borgrepo::'{hostname}-{now}' /srv /etc /home

If you’re unsure which fits your case, ask. I keep these backup prompts with my other linux admin prompts:

I need to back up /srv (50GB, mostly static), /etc, and a Postgres dump nightly, keeping 7 daily / 4 weekly / 6 monthly snapshots, encrypted, to a remote host. Should I use rsync, Borg, or both, and why?

Retention is a policy, not a guess

The point of versioned backups is keeping the right history without the repo growing forever. Borg’s prune enforces a retention policy cleanly:

borg prune --list \
  --keep-daily=7 --keep-weekly=4 --keep-monthly=6 \
  /mnt/backup/borgrepo
borg compact /mnt/backup/borgrepo

That keeps 7 daily, 4 weekly, 6 monthly snapshots and reclaims space with compact. Getting the retention math right — and making sure prune runs after create, never before — is exactly the kind of detail AI is good at drafting and exactly the kind of thing you must review, because a misordered prune can delete the snapshot you just made. Pro Tip: Always run borg prune with --dry-run the first time and read what it intends to delete. A retention policy that looks right in your head can silently keep zero monthlies because of an off-by-one. Verify against the actual prune output, not your mental model.

Encrypt, and don’t lose the key

Borg’s repokey mode encrypts the repo and stores the (passphrase-protected) key inside it; keyfile mode keeps the key separate. Either way, the passphrase is now the single point of failure for your entire backup. Lose it and the backups are unrecoverable noise — that’s the whole point of encryption.

Store the passphrase in a real secret manager, document where it lives, and make sure more than one human can reach it. Never paste it into an AI prompt, never embed it in a script the model can see, never put it in the same repo as your code. This is the sharpest edge of the general rule: AI helps you write the backup logic, but the secrets that protect your data stay entirely on the human side and never enter a model’s context.

Automate with a timer — and alert on failure

A backup that fails silently is worse than no backup, because it breeds false confidence. Run it from a systemd timer with proper logging and failure alerting:

# /etc/systemd/system/borg-backup.service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/borg-backup.sh
# OnFailure= a unit that pages you

Wire OnFailure= to something that actually notifies a human, or pipe the script’s exit status into your monitoring. The monitoring alerts helper can draft an alert rule that fires when a backup hasn’t completed in 26 hours — which catches the “cron died months ago” failure mode before it costs you. Have the AI draft the timer and the wrapper script, then run it through the code review tool before it goes live; a backup script bug is the one you find at the worst possible moment.

Test the restore — this is the whole job

Here is the step everyone skips and the only one that actually matters. Schedule a real restore test:

borg list /mnt/backup/borgrepo                    # what snapshots exist
borg extract --dry-run /mnt/backup/borgrepo::archive-name
borg mount /mnt/backup/borgrepo::archive-name /mnt/restore   # browse it

Mount a snapshot, pull a known file, diff it against the original, confirm it’s byte-identical. Do this on a calendar cadence — quarterly at minimum — not just once at setup. If you’re restoring a database, restore it into a throwaway instance and actually start it, because a Postgres backup that won’t replay is not a backup. The incident response helper is useful for drafting a restore runbook so the test (and the real thing) follows a checklist instead of improvisation.

Off-site and immutable, or it didn’t happen

A backup on the same machine survives a deleted file, not a dead disk, a ransomware event, or a fat-fingered rm -rf. You need at least one copy on a different host, ideally append-only so a compromised production box can’t reach back and delete its own backups:

borg create ... ssh://borg@backuphost/./repo::'{now}'   # remote repo
# enforce append-only on the server side in ~/.ssh/authorized_keys:
# command="borg serve --append-only --restrict-to-path /repo"

That --append-only restriction is the difference between a backup and a liability — it means even root on the production host can’t prune or delete the remote repo. Have the AI draft the SSH authorized_keys restriction and explain each flag, then review it yourself, because this is the line that protects you from your own compromised server. I keep these vetted backup scripts and the restore runbook in the prompt packs and prompts library.

Conclusion

Backups fail in three ways: they break silently, the key gets lost, or nobody ever tested a restore. Use rsync for mirrors and Borg for encrypted versioned history, get retention right with a dry-run prune, alert loudly on failure, keep an append-only off-site copy, and — above all — test the restore on a schedule. AI makes building all of this dramatically faster as a drafter and reviewer, but the passphrase, the credentials, and the act of running it against real data stay with a human. A backup you’ve restored from is the only kind that counts.