systemd Timer Debugging Prompt
Diagnose systemd timers that don't fire, fire at wrong times, miss runs after sleep, or run with wrong environment — `OnCalendar`, `Persistent=`, `AccuracySec=`, time zones.
- Target user
- Linux sysadmins migrating from cron or running systemd timer-based jobs
- Difficulty
- Intermediate
- Tools
- Claude, ChatGPT
The prompt
You are a senior Linux sysadmin who has migrated cron jobs to systemd timers and debugged the subtle differences — `OnCalendar=` syntax, `Persistent=true`, timezone handling, missed runs, environment differences from interactive shells.
I will provide:
- The timer unit file (`/etc/systemd/system/<name>.timer` or `/etc/systemd/user/...`)
- The corresponding service unit file
- `systemctl status <name>.timer`
- `systemctl list-timers --all`
- `journalctl -u <name>.service` for recent runs
- The symptom (didn't fire, fired late, missed after suspend, ran but app error)
Your job:
1. **Verify the timer is enabled AND started**:
- `systemctl is-enabled <name>.timer` → must be `enabled`
- `systemctl is-active <name>.timer` → must be `active`
- **A timer can be enabled but inactive** (never `systemctl start`ed AND no reboot since enable). Common surprise.
2. **Read `OnCalendar=` carefully**:
- Syntax: `DayOfWeek Year-Month-Day Hour:Minute:Second`
- **Examples** (and what they mean):
- `*-*-* 03:00:00` — every day at 03:00
- `Mon..Fri 09:00` — weekdays at 09:00
- `*-*-01 04:00` — first of every month at 04:00
- `hourly`, `daily`, `weekly`, `monthly`, `yearly` — convenience aliases
- `systemd-analyze calendar "<spec>"` — validates and shows next 3 firing times
- **Time zone**: defaults to system local time. Use `OnCalendar=*-*-* 03:00 UTC` or set `WorkingDirectory=` / unit timezone.
3. **`Persistent=true` for missed-run handling**:
- When timer fires while system is off (e.g., laptop), `Persistent=true` runs ONCE after boot to catch up
- Without `Persistent=`, the missed run is just skipped
- Stored state: `/var/lib/systemd/timers/<name>.timer`
4. **`AccuracySec=`** — slack window for the timer:
- Default `1min` — timer can fire anywhere in a 1-minute window
- Lower for precise timing (`AccuracySec=1s`); raise for less wake-ups (`AccuracySec=1h`)
- systemd batches timers within their accuracy windows to save power
5. **`OnBootSec=` / `OnUnitActiveSec=` / `OnUnitInactiveSec=` monotonic timers**:
- Useful for "run N minutes after boot" or "run N minutes after the last run"
- Combined with `OnCalendar=` is allowed; either trigger fires the service
6. **For "service ran but failed"**:
- `systemctl status <name>.service` shows last exit
- Service unit must have correct `Type=`, `ExecStart=`, working dir, user, environment
- Service runs in a clean environment — NOT your shell env. No `.bashrc`, no `PATH` aliases. Set `Environment=` explicitly.
7. **For drift / late firing**:
- `journalctl -u <name>.service` shows actual fire times
- Compare to `next_elapse` in `systemctl list-timers`
- High system load can defer timer firing
- `OnCalendar=hourly` fires at minute 0; if exact-half-past needed, use `*-*-* *:30:00`
8. **For interactive testing**:
- `systemd-analyze calendar "<spec>"` — shows next runs
- `systemctl start <name>.service` — run the service manually right now
- `systemctl status` and `journalctl -u <name>.service -n 50`
Mark DESTRUCTIVE: replacing cron with timers while cron is still scheduled (jobs run twice), Persistent=true on a job that's NOT idempotent (catches up may cause damage), running service manually that mutates state (may collide with the next scheduled run).
---
Timer unit:
```ini
[PASTE <name>.timer]
```
Service unit:
```ini
[PASTE <name>.service]
```
`systemctl status <name>.timer`:
```
[PASTE]
```
`systemctl list-timers <name>.timer` and `journalctl -u <name>.service -n 50`:
```
[PASTE]
```
Symptom: [DESCRIBE]
Why this prompt works
systemd timers are subtly different from cron — different env, different syntax, different missed-run behavior, different precision. Engineers comfortable with cron often hit unexpected gotchas. This prompt forces unit-by-unit verification.
How to use it
- Always run
systemd-analyze calendarto verify the OnCalendar spec before committing. - Include both
.timerand.servicewhen debugging — they’re a pair. - For drift complaints, capture
list-timersoutput AND actual run timestamps from journalctl. - Check the timer is
enabledANDstarted— common to forget the start.
Useful commands
# Timer inventory
systemctl list-timers --all
systemctl list-timers <name>.timer
# Verify enabled + active
systemctl is-enabled <name>.timer
systemctl is-active <name>.timer
# Validate calendar spec
systemd-analyze calendar "Mon..Fri 09:00"
systemd-analyze calendar "*-*-* 03:00"
systemd-analyze calendar "*-*-1 04:00 UTC"
# Service runs
systemctl status <name>.service
journalctl -u <name>.service --since "7 days ago" --no-pager
# Run service manually (test)
sudo systemctl start <name>.service
# Persistent state
ls -la /var/lib/systemd/timers/
cat /var/lib/systemd/timers/stamp-<name>.timer
# Edit with drop-in
sudo systemctl edit <name>.timer
# (Adds /etc/systemd/system/<name>.timer.d/override.conf)
sudo systemctl daemon-reload
sudo systemctl restart <name>.timer
Common patterns
Daily backup at 03:00 with catch-up
# /etc/systemd/system/backup.timer
[Unit]
Description=Daily backup
[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true
AccuracySec=1min
[Install]
WantedBy=timers.target
# /etc/systemd/system/backup.service
[Unit]
Description=Backup script
[Service]
Type=oneshot
User=backup
Group=backup
ExecStart=/usr/local/bin/backup.sh
Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
WorkingDirectory=/var/backups
Enable:
sudo systemctl daemon-reload
sudo systemctl enable --now backup.timer
Every 5 minutes (monotonic)
[Timer]
OnBootSec=5min
OnUnitActiveSec=5min
AccuracySec=10s
Specific weekdays
[Timer]
OnCalendar=Mon,Wed,Fri 09:00
Persistent=true
Random delay (avoid stampede)
[Timer]
OnCalendar=hourly
RandomizedDelaySec=10min
Common findings this catches
- Timer enabled but never fired → not started; need
systemctl start <name>.timerOR reboot. - Missed runs after laptop sleep → no
Persistent=true; add it (if idempotent). - Timer fires but service exits 217/USER → service
User=doesn’t exist. - Daily timer at 03:00 fires at 03:00 local, not 03:00 UTC → add
UTCto spec or run in UTC. - Cron entry AND timer both exist → job runs twice. Remove cron.
- Job’s
PATHis wrong → service has minimal env; setEnvironment=PATH=...or full paths in script. - High
AccuracySec=causes spread → if exact firing matters, lower it (cost: less power efficiency).
Migration from cron
# Crontab entry
0 3 * * * /usr/local/bin/backup.sh
Becomes:
- A
.serviceunit running the script - A
.timerunit withOnCalendar=*-*-* 03:00:00 Persistent=true - Comment out the crontab entry to avoid double-execution
# /etc/cron.daily/* and /etc/cron.weekly/* are also worth checking
ls /etc/cron.{hourly,daily,weekly,monthly}/
When to escalate
- Timer drift correlating with kernel timekeeping issues — check NTP/chrony status.
- Container-based timer jobs hitting cgroup CPU limits and missing windows — adjust container resources.
- Persistent issues with
Persistent=trueand non-idempotent jobs — redesign the job to be idempotent before solving with timers.
Related prompts
-
Linux Server Troubleshooting Prompt
Help diagnose CPU, memory, disk, network, and service issues on Ubuntu or RHEL servers from raw command output.
-
Sudoers & Systemd Services Review Prompt
AI review of /etc/sudoers (and /etc/sudoers.d/*) and systemd service unit files for privilege escalation, unsafe defaults, and hardening gaps.
-
systemd Unit Failure Debugging Prompt
Diagnose systemd unit failures — dependency cycles, mount/target failures, exit codes, journalctl filtering, drop-in overrides, and silent service flapping.