🔌 Calling APIs from Bash and Python Scripts Without the Footguns

Most automation eventually has to call an API — a cloud provider, an internal service, a webhook, a status page. The call itself is one line of curl. The footguns are everything around it: silent failures, leaked tokens, missing timeouts, ignored error codes, and pagination you forgot existed.

After 25 years of gluing systems together, here is how I call APIs from scripts so they fail loudly, never leak secrets, and behave under load.

The bare-minimum safe curl

A naive curl https://api.example.com/thing will happily print an HTML error page and exit zero, so your script marches on as if it succeeded. The flags that fix this:

curl --fail-with-body \
     --silent --show-error \
     --max-time 10 \
     --retry 3 --retry-delay 2 \
     -H "Authorization: Bearer ${API_TOKEN}" \
     "https://api.example.com/v1/status"

What each one buys you:

• --fail-with-body makes curl exit non-zero on HTTP 4xx/5xx and still show the response body, so your error handling triggers and you can see why.
• --silent --show-error hides the progress bar but keeps real errors.
• --max-time 10 caps the whole request. Without it, a hung connection hangs your script forever.
• --retry 3 handles transient network failures.

That combination turns curl from “fire and hope” into something a script can actually depend on.

Never put secrets on the command line

A token passed as a curl argument shows up in ps, in shell history, and in process logs. Anyone on the box can see it. Read it from the environment or a file instead:

# from environment (set by systemd, vault, etc.)
curl -H "Authorization: Bearer ${API_TOKEN}" ...

# from a file, never echoed
curl -H @<(printf 'Authorization: Bearer %s' "$(cat /etc/secrets/token)") ...

And scrub tokens from any debug output. The number of credentials I have seen leaked into a CI log because someone added set -x to a script with an inline token is too high.

The Python version with httpx

For anything beyond a single call, Python with httpx (or requests) is cleaner and gives you real error handling:

import httpx

def get_status():
    headers = {"Authorization": f"Bearer {os.environ['API_TOKEN']}"}
    try:
        resp = httpx.get(
            "https://api.example.com/v1/status",
            headers=headers,
            timeout=10.0,
        )
        resp.raise_for_status()
    except httpx.TimeoutException:
        raise RuntimeError("API timed out")
    except httpx.HTTPStatusError as e:
        raise RuntimeError(f"API returned {e.response.status_code}: {e.response.text}")
    return resp.json()

Two non-negotiables here: an explicit timeout (httpx and requests default to no timeout, which is a hang waiting to happen), and raise_for_status() so a 500 becomes an exception instead of a silently-parsed error body.

Handle pagination, don’t pretend it isn’t there

The classic bug: you test against an account with 12 resources, the API returns one page, everything works. In production there are 4,000 resources across 80 pages and your script silently processes only the first 50. Always follow pagination:

def list_all(url):
    items = []
    while url:
        resp = httpx.get(url, headers=headers, timeout=10.0)
        resp.raise_for_status()
        data = resp.json()
        items.extend(data["items"])
        url = data.get("next")  # cursor/link style
    return items

Whether the API uses cursors, link headers, or page numbers, the rule is the same: loop until there is no next page. Never assume one page is all of it.

Respect rate limits

When an API returns 429, it usually tells you how long to wait in a Retry-After header. Honor it instead of hammering:

if resp.status_code == 429:
    wait = int(resp.headers.get("Retry-After", 5))
    time.sleep(wait)
    continue

Ignoring Retry-After and retrying immediately is how you get your token throttled harder or banned. Combine this with the exponential-backoff-and-jitter pattern for anything that runs across many machines.

Validate the response shape

APIs change and return surprises. A script that blindly does data["items"][0]["id"] crashes cryptically when the response is shaped differently than you expect. Check before you index, and fail with a useful message:

data = resp.json()
if "items" not in data:
    raise RuntimeError(f"unexpected response shape: {list(data)}")

Where AI helps

API integration is full of small, easy-to-forget details, which makes a focused review prompt valuable. When I have a working call, I ask:

“Review this API client for production use. Check for: missing timeout, unhandled non-2xx responses, secrets on the command line or in logs, missing pagination, and no rate-limit handling. List each issue and the fix.”

The model reliably catches the missing timeout and the ignored pagination — exactly the bugs that pass testing and fail in production. I also use AI to turn an API’s docs into a first-draft client: paste the endpoint description and ask for an httpx function with proper error handling. I keep these prompts in my prompt library.

The checklist

• Always set a timeout. Always.
• --fail-with-body / raise_for_status() so errors are loud.
• Keep tokens out of argv and logs; read from env or files.
• Follow pagination to the last page.
• Honor Retry-After and back off on 429.
• Validate the response shape before indexing into it.

Get these right and your API-calling scripts stop being the flaky part of your automation. For more patterns and AI prompts, see our automation guides.