Python Paginated REST API Extractor Prompt
Build a Python script that walks a paginated REST API, handles rate limits and retries, and writes normalized records to JSONL incrementally and resumably.
- Target user
- Data and platform engineers pulling bulk data from APIs
- Difficulty
- Advanced
- Tools
- Claude, Copilot
The prompt
You are a senior data integration engineer who writes API extractors that are polite to upstream services and safe to interrupt and resume. I will provide: - The base URL, auth method, and the pagination style (cursor, page number, or Link header) - The fields I need and the output path - Any documented rate limits Your job: 1. **CLI** — `argparse` for base URL, output file, page size, and a `--resume` flag. 2. **Authenticate from env** — read tokens from environment variables; never log full credentials. 3. **Paginate generically** — implement a generator that yields pages until exhausted, abstracting the pagination style behind one function. 4. **Respect limits** — honor `Retry-After` and rate-limit headers; back off exponentially with jitter on 429 and 5xx, but fail fast on 4xx like 401/403. 5. **Stream output** — append each normalized record as a JSONL line and flush, so memory stays bounded and partial output survives a crash. 6. **Resume** — record the last successful cursor/page to a checkpoint file so `--resume` continues without duplicating already-fetched records. 7. **Dry-run** — `--dry-run` fetches only the first page and prints the shape. Output as: (a) the typed script, (b) a sample checkpoint file, (c) a cron example. Always honor Retry-After and checkpoint progress so an interrupted extract resumes without re-hammering the API.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Asyncio TaskGroup Structured-Concurrency Python Script Prompt
Write a Python 3.11+ automation script that fans work out with asyncio.TaskGroup so failures cancel siblings cleanly, errors aggregate into an ExceptionGroup, and no task is ever silently orphaned.
-
Convert a Fragile Bash Pipeline to Python Prompt
Rewrite an overgrown, fragile Bash script full of pipes, subshells, and string parsing into a maintainable Python program with proper error handling, types, and tests.
-
Add Pytest Coverage to an Automation Script Prompt
Wrap an existing Python automation script in a pytest suite that mocks subprocess, network, filesystem, and clock side effects so the logic is tested fast and deterministically without touching real systems.
-
Idempotent Bulk File Rename & Migrate Tool Prompt
Write a Python tool that safely renames or relocates large batches of files using a rule set, with mandatory dry-run, collision detection, atomic moves, and an undo manifest so a bad run is reversible.
More Bash & Python Automation prompts & error guides
Browse every Bash & Python Automation prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.