🚨 Automation Error: Overlapping Cron Runs Cause a Race Condition

Overview

A job scheduled every few minutes starts taking longer than its interval, so the scheduler launches a new copy before the previous one has finished. Two or more instances then run concurrently against the same data, and the symptom shows up downstream as duplicated or corrupted work:

$ pgrep -af sync-orders.sh
20144 /bin/bash /opt/jobs/sync-orders.sh
20361 /bin/bash /opt/jobs/sync-orders.sh
20588 /bin/bash /opt/jobs/sync-orders.sh

ERROR: duplicate key value violates unique constraint "orders_external_id_key"
DETAIL: Key (external_id)=(ORD-88213) already exists.

Three copies of the same “every 5 minutes” job are alive at once, and the database is rejecting the duplicate rows they are racing to insert.

Symptoms

• Multiple processes with the same job name appear in ps/pgrep simultaneously.
• Duplicate rows, duplicate emails/notifications, or double-counted metrics.
• Unique-constraint violations, deadlocks, or SELECT ... FOR UPDATE lock waits during the job.
• Load and memory climb steadily through the day as overlapping runs stack up.
• The job “works fine” in staging (small dataset, finishes fast) but races in production (large dataset, overruns its interval).
• Partially written output files, or a temp file clobbered mid-write by a second run.

Common Root Causes

• Runtime exceeds the interval. The dataset grew, a downstream call got slower, or a lock stalls the job past its schedule period, so the next tick starts before this one ends.
• No mutual exclusion. The job assumes it is the only instance and was never wrapped in a lock (flock, a lock row, a distributed lease).
• Non-idempotent work. Each run inserts rather than upserts, appends rather than replaces, so concurrent runs collide instead of converging.
• Multiple schedulers. The same cron entry is deployed to several hosts, or duplicated across a systemd timer and crontab, so copies fire in parallel by design.
• Retries layered on a slow run. A wrapper retries the job while the original is still running, multiplying instances.
• Shared mutable temp paths. Two runs write the same /tmp/export.csv or lock-free working directory and corrupt each other’s output.

Diagnostic Workflow

First confirm that overlap is actually happening — count live instances:

pgrep -af sync-orders
ps -eo pid,etimes,cmd | grep '[s]ync-orders' | sort -k2 -n

etimes (elapsed seconds) tells you whether an instance has been running longer than the schedule interval — the smoking gun for overrun.

Measure how long the job actually takes versus how often it is scheduled:

# What the schedule promises
crontab -l | grep sync-orders
systemctl list-timers --all | grep sync-orders

# What it actually costs (wrap a manual run)
/usr/bin/time -v /opt/jobs/sync-orders.sh 2>&1 | grep -i 'Elapsed\|wall'

Check the scheduler logs to see runs starting while a prior one is still active:

journalctl -u sync-orders.service --since '2 hours ago' | grep -i 'Started\|Deactivated\|Finished'
grep CRON /var/log/syslog | grep sync-orders

Confirm there is no lock protecting the critical section:

grep -n 'flock\|lockfile\|LOCK\|advisory' /opt/jobs/sync-orders.sh || echo "NO LOCK FOUND"

Look at the downstream damage to classify the race (duplicates vs corruption):

# Example: duplicate detection in Postgres
psql -c "SELECT external_id, count(*) FROM orders GROUP BY 1 HAVING count(*) > 1 LIMIT 20;"
# Example: who holds the lock right now
psql -c "SELECT pid, state, wait_event_type, query FROM pg_stat_activity WHERE query ILIKE '%orders%';"

Example Root Cause Analysis

An orders sync ran from crontab every 5 minutes:

*/5 * * * *  /opt/jobs/sync-orders.sh >> /var/log/sync-orders.log 2>&1

For months it finished in ~90 seconds. After a marketing push, the upstream order feed grew tenfold and a single run began taking 7-8 minutes. Now at every :05, :10, :15 tick, cron launched a fresh copy while the previous one was still paging through the feed. Three concurrent runs fetched overlapping pages and each tried to INSERT the same orders, producing duplicate key value violates unique constraint and, worse, a growing pile-up that pushed load average past the node’s core count.

Two problems compounded: the job overran its interval, and it had no mutual exclusion and used raw INSERT instead of upsert. The fix addressed both.

First, wrap the job in flock so only one instance can run, and non-blocking -n so a late tick exits immediately instead of queueing behind the running copy:

#!/usr/bin/env bash
set -euo pipefail

exec 9>/var/lock/sync-orders.lock
if ! flock -n 9; then
  echo "$(date -Is) previous run still active; skipping this tick" >&2
  exit 0
fi

# --- critical section: guaranteed single instance ---
/opt/jobs/_sync-orders-impl.sh

Second, make the write idempotent so that even if isolation ever fails, the work converges instead of colliding:

INSERT INTO orders (external_id, status, total_cents)
VALUES ($1, $2, $3)
ON CONFLICT (external_id)
DO UPDATE SET status = EXCLUDED.status, total_cents = EXCLUDED.total_cents;

After deploying, pgrep -af sync-orders never showed more than one process, the duplicate-key errors stopped, and the skipped ticks (logged as “previous run still active”) made the overrun visible so the team could later split the feed into smaller batches.

Prevention Best Practices

• Wrap every periodic job in a lock. flock -n on a single host; a Postgres advisory lock, a Redis lease, or a Kubernetes Lease object across multiple hosts.
• Make the work idempotent. Upsert instead of insert, replace instead of append, derive stable keys — so a race degrades to wasted effort, not corruption.
• Choose skip-vs-queue on purpose. Use non-blocking flock -n to skip an overlapping tick (freshness matters) or blocking flock to serialize (every run must eventually happen).
• In Kubernetes, set concurrencyPolicy: Forbid on the CronJob so the controller itself refuses to start an overlapping Job.
• Alarm on runtime approaching the interval. If p95 runtime exceeds ~70% of the schedule period, the job is heading for overlap — shorten the work or lengthen the interval before it races.
• Have exactly one scheduler own each job. Audit for the same job defined in both crontab and a systemd timer, or deployed to multiple hosts without a distributed lock.

Quick Command Reference

# Detect overlap and how long instances have run
pgrep -af my-job
ps -eo pid,etimes,cmd | grep '[m]y-job' | sort -k2 -n

# Compare schedule interval to actual runtime
crontab -l | grep my-job
/usr/bin/time -v /opt/jobs/my-job.sh 2>&1 | grep Elapsed

# Confirm whether a lock exists
grep -n 'flock\|advisory\|LOCK' /opt/jobs/my-job.sh || echo NO_LOCK

# Single-instance wrapper pattern
flock -n /var/lock/my-job.lock /opt/jobs/my-job.sh || echo "skipped: already running"

# Kubernetes: forbid overlapping runs
kubectl patch cronjob my-job --type merge -p '{"spec":{"concurrencyPolicy":"Forbid"}}'

Conclusion

Overlapping cron runs are a race condition hiding behind a schedule: the job was correct when it finished inside its interval, and became dangerous the moment it didn’t. The two-layer fix is durable — enforce single-instance execution with flock or a distributed lease so copies can’t run concurrently, and make the underlying work idempotent so a race can never corrupt data even if the lock is bypassed. Then alert when runtime creeps toward the interval, because the only thing worse than a job that overran is one that has been quietly duplicating work for a week.