👁️ Watching Filesystem Events with inotify on Linux

For years I solved “do something when a file changes” with a cron job that polled a directory every minute, diffed a checksum, and felt vaguely ashamed every time I looked at it. It worked, until it didn’t: a one-minute latency window on a config reload turned into a forty-second outage, and the polling itself started showing up in my I/O graphs on a busy NFS-adjacent box. The fix had been sitting in the kernel the whole time. inotify lets the kernel tell you the instant something happens to a file or directory, so you stop guessing and start reacting. This is a walk through the tools I actually reach for now, the knobs you will eventually have to turn, and the sharp edges that will waste your afternoon if nobody warns you first.

What inotify actually is

inotify is a Linux kernel subsystem that delivers filesystem events through a file descriptor. You register watches on paths, and the kernel queues events like modify, create, delete, move, and close_write as they occur. It is local-filesystem only and it does not recurse on its own, two facts that explain most of the confusion people hit later. The userspace tooling lives in the inotify-tools package on Debian, Ubuntu, RHEL, and friends:

sudo apt-get install inotify-tools   # Debian/Ubuntu
sudo dnf install inotify-tools       # Fedora/RHEL

That gives you two binaries: inotifywait for blocking until events happen, and inotifywatch for gathering statistics over a window.

Watching a tree in real time with inotifywait

The workhorse invocation runs in monitor mode (-m, stay alive instead of exiting on the first event), recurses (-r), and filters to the events you care about (-e):

inotifywait -m -r -e modify,create,delete /etc/myapp/

Each line of output is a path, a comma-separated event mask, and the affected filename. For scripting you almost never want the default human format. Pin it down with --format and a custom timestamp via --timefmt:

inotifywait -m -r \
  -e modify,create,delete,close_write \
  --timefmt '%F %T' \
  --format '%T %w%f %e' \
  /etc/myapp/

Here %T is the timestamp, %w is the watched directory, %f is the filename within it, and %e is the event list. The output becomes trivially greppable and parseable, which matters the moment you pipe it into anything.

Pro Tip: Prefer close_write over modify when you want to act on a finished file. modify fires on every write() syscall, so a single large copy can hammer your handler dozens of times; close_write fires once, when the writer closes the descriptor.

Reacting to events in a bash loop

The pattern that replaced my polling cron is a while read loop fed directly by inotifywait. Drop the -m here and use -q so each event is one clean line, then loop:

#!/usr/bin/env bash
set -euo pipefail

WATCH_DIR="/etc/myapp"

inotifywait -m -r -q \
  -e close_write,moved_to,delete \
  --format '%e %w%f' \
  "$WATCH_DIR" |
while read -r events file; do
  case "$events" in
    *CLOSE_WRITE*|*MOVED_TO*)
      echo "config changed: $file -> reloading"
      systemctl reload myapp || logger -t myapp-watch "reload failed for $file"
      ;;
    *DELETE*)
      logger -t myapp-watch "config removed: $file"
      ;;
  esac
done

A few non-obvious details earn their keep. Running inotifywait in monitor mode and piping into the loop keeps a single persistent watch set, instead of re-arming after every event and missing changes in the gap. Matching MOVED_TO alongside CLOSE_WRITE is what makes this survive real editors, which we will get to. And debouncing matters in production: if a deploy rewrites ten files at once you do not want ten reloads, so in busier setups I buffer events and reload at most once per second with a timestamp guard.

This is exactly the kind of glue I now hand to an AI assistant first. Describe the directory, the events, and the action, and a tool like Claude or Cursor drafts the loop in seconds. Treat that draft like work from a fast junior engineer: it gets the skeleton right and the edge cases wrong. You still review the signal handling, confirm the case patterns, and test it before it touches anything real. And it goes without saying, but the assistant gets nothing resembling production credentials or SSH access. It writes the script; a human runs it.

Gathering statistics with inotifywatch

When the question is “what is churning this directory” rather than “react now,” inotifywatch aggregates events over a window and prints a table:

inotifywatch -v -r -t 60 /var/log/

-t 60 collects for sixty seconds, -r recurses, and -v is verbose. The output ranks event types by count, which is a quick way to discover that some forgotten log shipper is rewriting a file four hundred times a minute. I keep this in my back pocket for capacity questions before deciding whether a real-time watch is even the right tool. If the churn surprises you, that signal is worth surfacing in your broader monitoring and alerting workflow rather than leaving it in a terminal scrollback.

Persistent rules with incron

inotifywait is great for ad hoc and supervised work, but if you want a declarative, persistent “when this path changes, run this command” rule, that is what incron is for. It is the inotify analogue of cron, with a per-user table edited via incrontab:

sudo apt-get install incron
sudo incrontab -e

Each line is a path, a space-separated event mask, and a command, where incron substitutes $@ for the watched path, $# for the filename, and $% for the event flags:

/etc/myapp/  IN_CLOSE_WRITE,IN_MOVED_TO  /usr/local/bin/reload-myapp.sh $@/$#

Two operational gotchas: you must add allowed users to /etc/incron.allow, and incron does not recurse, so each subdirectory you care about needs its own line. For a handful of well-known config paths it is clean and survives reboots through the incrond service. For deep, dynamic trees I stick with a supervised inotifywait script under its own systemd unit.

The systemd alternative: .path units

If you are already living in systemd, you may not need inotify-tools at all. A .path unit watches a path and activates a partner .service when it changes. Create /etc/systemd/system/myapp-reload.path:

[Unit]
Description=Watch myapp config

[Path]
PathChanged=/etc/myapp/app.conf
Unit=myapp-reload.service

[Install]
WantedBy=multi-user.target

Pair it with a oneshot myapp-reload.service that runs your reload command, then enable the path unit:

sudo systemctl daemon-reload
sudo systemctl enable --now myapp-reload.path

systemd path units use inotify under the hood but give you journald logging, restart policy, and dependency ordering for free. The trade-off is they watch specific paths rather than whole recursive trees, and the directives (PathExists, PathChanged, PathModified, DirectoryNotEmpty) are coarser than raw event masks. For single-file config reloads they are my default; for crawling a tree, raw inotify wins.

Tuning the kernel limits

inotify is cheap but not free. Every watch consumes a small slice of kernel memory, and the limits are per user, not per process. The three you will meet are exposed under /proc/sys/fs/inotify/:

cat /proc/sys/fs/inotify/max_user_watches    # total watches per user
cat /proc/sys/fs/inotify/max_user_instances  # total inotify fds per user
cat /proc/sys/fs/inotify/max_queued_events   # per-instance event queue depth

The classic failure is inotifywait exiting with “Failed to watch …; upper limit on inotify watches reached” when you point -r at a giant tree, or your IDE silently stops noticing file changes. Because watches are not recursive, a deep -r registers one watch per directory, and a node_modules or a sprawling log root blows through the default fast. Raise it persistently:

echo 'fs.inotify.max_user_watches=524288' | sudo tee /etc/sysctl.d/60-inotify.conf
echo 'fs.inotify.max_user_instances=512' | sudo tee -a /etc/sysctl.d/60-inotify.conf
sudo sysctl --system

Pro Tip: Watch counts are per real UID across every process that user runs. If a CI agent, a dev IDE, and your watcher script all share an account, they draw from the same pool, so size max_user_watches for the busiest combined case, not for one tool in isolation.

The atomic-save pitfall that bites everyone

Here is the trap that sends people to forums convinced inotify is broken. Most serious editors, including vim, emacs, and VS Code, do not write your file in place. They write a temporary file, fsync it, then rename() it over the original. From inotify’s perspective the original inode is gone and a new file has appeared. If you watched only modify or close_write on the original filename, you will see exactly nothing when the editor saves.

The robust fix is to watch the directory rather than the file, and to include the move events:

inotifywait -m -e close_write,moved_to,create \
  --format '%e %f' \
  /etc/myapp/

moved_to catches the atomic rename, close_write catches in-place writers like >> and printf, and watching the parent directory means you keep getting events even after the original inode is replaced. The same logic explains why watching a single file with a .path unit can miss editor saves: point it at PathModified on the directory, or restart the watcher after a swap. Once you internalize “watch the directory, match the moves,” ninety percent of “inotify isn’t firing” bug reports evaporate.

This rename behavior is also a great litmus test for AI-generated watchers. Ask an assistant for a file watcher and it will frequently hand you a naive modify-only loop, because that is the most common pattern in its training data, not the most correct one. Knowing the failure mode yourself is what lets you catch it in review. If you want a head start, our prompts library and prompt packs include hardened starting points, and you can iterate on them in the prompt workspace before anything runs on a real host. The assistant accelerates the boring parts; you stay the engineer who signs off.

Wrapping up

inotify turns “poll and pray” into “react the instant it happens,” and the toolkit around it scales from a one-off inotifywait in a terminal to declarative incron rules and systemd path units. Reach for the lightweight loop when you are supervising, incron or path units when you want persistence, inotifywatch when you are diagnosing, and remember the two rules that save the most time: tune the per-user watch limits before they ambush you, and always watch the directory and the move events so atomic saves do not slip past. Let an AI draft the glue, keep a human in the loop on every line that touches production, and never hand the model your credentials. For more in this vein, browse the rest of the Linux admins category.