Skip to content
CloudOps
Newsletter
All prompts
AI for Linux Admins Difficulty: Intermediate ClaudeChatGPT

systemd Timer Debugging Prompt

Diagnose systemd timers that don't fire, fire at wrong times, miss runs after sleep, or run with wrong environment — `OnCalendar`, `Persistent=`, `AccuracySec=`, time zones.

Target user
Linux sysadmins migrating from cron or running systemd timer-based jobs
Difficulty
Intermediate
Tools
Claude, ChatGPT

The prompt

You are a senior Linux sysadmin who has migrated cron jobs to systemd timers and debugged the subtle differences — `OnCalendar=` syntax, `Persistent=true`, timezone handling, missed runs, environment differences from interactive shells.

I will provide:
- The timer unit file (`/etc/systemd/system/<name>.timer` or `/etc/systemd/user/...`)
- The corresponding service unit file
- `systemctl status <name>.timer`
- `systemctl list-timers --all`
- `journalctl -u <name>.service` for recent runs
- The symptom (didn't fire, fired late, missed after suspend, ran but app error)

Your job:

1. **Verify the timer is enabled AND started**:
   - `systemctl is-enabled <name>.timer` → must be `enabled`
   - `systemctl is-active <name>.timer` → must be `active`
   - **A timer can be enabled but inactive** (never `systemctl start`ed AND no reboot since enable). Common surprise.
2. **Read `OnCalendar=` carefully**:
   - Syntax: `DayOfWeek Year-Month-Day Hour:Minute:Second`
   - **Examples** (and what they mean):
     - `*-*-* 03:00:00` — every day at 03:00
     - `Mon..Fri 09:00` — weekdays at 09:00
     - `*-*-01 04:00` — first of every month at 04:00
     - `hourly`, `daily`, `weekly`, `monthly`, `yearly` — convenience aliases
   - `systemd-analyze calendar "<spec>"` — validates and shows next 3 firing times
   - **Time zone**: defaults to system local time. Use `OnCalendar=*-*-* 03:00 UTC` or set `WorkingDirectory=` / unit timezone.
3. **`Persistent=true` for missed-run handling**:
   - When timer fires while system is off (e.g., laptop), `Persistent=true` runs ONCE after boot to catch up
   - Without `Persistent=`, the missed run is just skipped
   - Stored state: `/var/lib/systemd/timers/<name>.timer`
4. **`AccuracySec=`** — slack window for the timer:
   - Default `1min` — timer can fire anywhere in a 1-minute window
   - Lower for precise timing (`AccuracySec=1s`); raise for less wake-ups (`AccuracySec=1h`)
   - systemd batches timers within their accuracy windows to save power
5. **`OnBootSec=` / `OnUnitActiveSec=` / `OnUnitInactiveSec=` monotonic timers**:
   - Useful for "run N minutes after boot" or "run N minutes after the last run"
   - Combined with `OnCalendar=` is allowed; either trigger fires the service
6. **For "service ran but failed"**:
   - `systemctl status <name>.service` shows last exit
   - Service unit must have correct `Type=`, `ExecStart=`, working dir, user, environment
   - Service runs in a clean environment — NOT your shell env. No `.bashrc`, no `PATH` aliases. Set `Environment=` explicitly.
7. **For drift / late firing**:
   - `journalctl -u <name>.service` shows actual fire times
   - Compare to `next_elapse` in `systemctl list-timers`
   - High system load can defer timer firing
   - `OnCalendar=hourly` fires at minute 0; if exact-half-past needed, use `*-*-* *:30:00`
8. **For interactive testing**:
   - `systemd-analyze calendar "<spec>"` — shows next runs
   - `systemctl start <name>.service` — run the service manually right now
   - `systemctl status` and `journalctl -u <name>.service -n 50`

Mark DESTRUCTIVE: replacing cron with timers while cron is still scheduled (jobs run twice), Persistent=true on a job that's NOT idempotent (catches up may cause damage), running service manually that mutates state (may collide with the next scheduled run).

---

Timer unit:
```ini
[PASTE <name>.timer]
```
Service unit:
```ini
[PASTE <name>.service]
```
`systemctl status <name>.timer`:
```
[PASTE]
```
`systemctl list-timers <name>.timer` and `journalctl -u <name>.service -n 50`:
```
[PASTE]
```
Symptom: [DESCRIBE]

Why this prompt works

systemd timers are subtly different from cron — different env, different syntax, different missed-run behavior, different precision. Engineers comfortable with cron often hit unexpected gotchas. This prompt forces unit-by-unit verification.

How to use it

  1. Always run systemd-analyze calendar to verify the OnCalendar spec before committing.
  2. Include both .timer and .service when debugging — they’re a pair.
  3. For drift complaints, capture list-timers output AND actual run timestamps from journalctl.
  4. Check the timer is enabled AND started — common to forget the start.

Useful commands

# Timer inventory
systemctl list-timers --all
systemctl list-timers <name>.timer

# Verify enabled + active
systemctl is-enabled <name>.timer
systemctl is-active <name>.timer

# Validate calendar spec
systemd-analyze calendar "Mon..Fri 09:00"
systemd-analyze calendar "*-*-* 03:00"
systemd-analyze calendar "*-*-1 04:00 UTC"

# Service runs
systemctl status <name>.service
journalctl -u <name>.service --since "7 days ago" --no-pager

# Run service manually (test)
sudo systemctl start <name>.service

# Persistent state
ls -la /var/lib/systemd/timers/
cat /var/lib/systemd/timers/stamp-<name>.timer

# Edit with drop-in
sudo systemctl edit <name>.timer
# (Adds /etc/systemd/system/<name>.timer.d/override.conf)
sudo systemctl daemon-reload
sudo systemctl restart <name>.timer

Common patterns

Daily backup at 03:00 with catch-up

# /etc/systemd/system/backup.timer
[Unit]
Description=Daily backup

[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true
AccuracySec=1min

[Install]
WantedBy=timers.target
# /etc/systemd/system/backup.service
[Unit]
Description=Backup script

[Service]
Type=oneshot
User=backup
Group=backup
ExecStart=/usr/local/bin/backup.sh
Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
WorkingDirectory=/var/backups

Enable:

sudo systemctl daemon-reload
sudo systemctl enable --now backup.timer

Every 5 minutes (monotonic)

[Timer]
OnBootSec=5min
OnUnitActiveSec=5min
AccuracySec=10s

Specific weekdays

[Timer]
OnCalendar=Mon,Wed,Fri 09:00
Persistent=true

Random delay (avoid stampede)

[Timer]
OnCalendar=hourly
RandomizedDelaySec=10min

Common findings this catches

  • Timer enabled but never fired → not started; need systemctl start <name>.timer OR reboot.
  • Missed runs after laptop sleep → no Persistent=true; add it (if idempotent).
  • Timer fires but service exits 217/USER → service User= doesn’t exist.
  • Daily timer at 03:00 fires at 03:00 local, not 03:00 UTC → add UTC to spec or run in UTC.
  • Cron entry AND timer both exist → job runs twice. Remove cron.
  • Job’s PATH is wrong → service has minimal env; set Environment=PATH=... or full paths in script.
  • High AccuracySec= causes spread → if exact firing matters, lower it (cost: less power efficiency).

Migration from cron

# Crontab entry
0 3 * * * /usr/local/bin/backup.sh

Becomes:

  • A .service unit running the script
  • A .timer unit with OnCalendar=*-*-* 03:00:00 Persistent=true
  • Comment out the crontab entry to avoid double-execution
# /etc/cron.daily/* and /etc/cron.weekly/* are also worth checking
ls /etc/cron.{hourly,daily,weekly,monthly}/

When to escalate

  • Timer drift correlating with kernel timekeeping issues — check NTP/chrony status.
  • Container-based timer jobs hitting cgroup CPU limits and missing windows — adjust container resources.
  • Persistent issues with Persistent=true and non-idempotent jobs — redesign the job to be idempotent before solving with timers.

Related prompts

Newsletter

Free: the DevOps AI Incident-Triage Cheat Sheet

Subscribe and we’ll send you the one-page cheat sheet — plus weekly AI prompts, automation ideas, and tool reviews for infrastructure engineers. One email a week. No spam, unsubscribe anytime.

  • AI Incident-Triage Cheat Sheet (PDF)
  • Access to 1,603 DevOps AI prompts
  • One practical workflow email per week