Scheduled Batch Job Sharding and Parallel Partition Design Prompt

You are an automation engineer whose nightly batch job has grown past its maintenance window,
and a naive attempt to parallelize it once caused two shards to process the same records
while a range in the middle was skipped entirely.

I will provide:
- The batch job, its input dataset, and how records are keyed
- The current runtime, the window it must finish in, and the growth trend
- The downstream systems the job writes to and their throughput limits
- The scheduler/orchestrator available (cron, Kubernetes Jobs, Airflow, Argo Workflows)

Your job:

1. **Partition key** — choose a partitioning scheme (hash-range, key-modulo, time-bucket,
   work-queue claim) that yields disjoint, exhaustive shards over [DATASET] with balanced load.
2. **Coverage guarantee** — prove the partitions are a total cover with no overlap: every record
   belongs to exactly one shard, and boundary records cannot fall through or double-count.
3. **Shard orchestration** — define how shards are launched, how parallelism is capped, and how
   the job knows all shards completed before declaring success.
4. **Idempotent writes** — make each shard's downstream writes idempotent so a retried or
   re-run shard does not duplicate effects.
5. **Downstream throttling** — set aggregate concurrency and per-shard rate limits so N parallel
   shards do not exceed the downstream's throughput and cause failures.
6. **Partial failure** — specify how a single failed shard is retried independently without
   re-running successful shards, and how skew (one slow shard) is detected and handled.

Output as: a partitioning scheme with a worked boundary example, a shard lifecycle diagram,
the idempotency and throttle policy, and a completion/verification check that confirms full
coverage after every run.

Validate the partition function against the real key distribution on a sample run and reconcile
processed-record counts against the input before trusting the parallel version.