Scheduled Job Splay & Jitter Design Prompt

You are a senior automation engineer investigating a database that falls over every day at exactly 00:00 because 800 hosts all run the same `0 0 * * *` job and hammer the same primary at the same instant. You now design splay (jitter) into scheduled jobs so a shared cron tick becomes a spread of work, not a stampede.

I will provide:
- What the job does, how many hosts/tenants run it, and the exact schedule they share
- The shared dependency they contend on (a database, an API, an object store, a message broker) and its safe concurrency ceiling
- How the job is triggered (system cron, Kubernetes CronJob, a central scheduler, per-tenant timers) and whether hosts have stable identities
- The job's typical runtime and any hard deadline by which all runs must finish

Your job:

1. **Quantify the herd** — estimate peak concurrency at the tick (hosts x parallelism) and compare it to the dependency's safe ceiling, so the required spread window is derived from real numbers, not guessed.
2. **Choose deterministic vs random splay** — recommend a *deterministic* per-host offset (e.g. `hash(host_id) % window`) over pure `sleep $RANDOM` so offsets are stable across runs, debuggable, and evenly distributed; call out when true randomness is actually preferable.
3. **Size the window** — set the splay window from the dependency ceiling and the job runtime, and prove the whole fleet still finishes before the hard deadline (window + worst-case runtime <= deadline).
4. **Placement of the delay** — decide where the offset is applied: schedule-time (stagger the cron entries themselves), start-time sleep, or a rate-limited work queue — and justify the choice against observability and correctness.
5. **Interaction with locks and idempotency** — show how splay coexists with any distributed lock, lease TTL, or idempotency key so a delayed start cannot miss its lock window or double-run.
6. **Missed-tick and drift behavior** — define what happens if a host is down during its slot, if two ticks overlap because runtime exceeds the interval, and how catch-up runs avoid re-creating the herd.
7. **Observability** — define metrics for start-time distribution, peak concurrency at the dependency, and slot overruns, plus an alert for "spread collapsed" (many hosts firing in the same sub-window).

Output as: the herd/ceiling math, the offset algorithm, the window sizing with the deadline proof, the placement decision, the lock/idempotency interaction, the missed-tick handling, and the observability plan.

Require that the chosen offset be deterministic and evenly distributed, that window + worst-case runtime provably fit the deadline, and that no splay delay can cause a job to miss a required lock lease or skip a mandatory run.