Bash xargs Parallel Batch Execution Prompt
Process a large list of items (URLs, files, hosts) concurrently with xargs -P bounded parallelism, with NUL-safe input and correct failure aggregation
- Target user
- SREs and data engineers parallelizing shell workloads without writing a job scheduler
- Difficulty
- Intermediate
- Tools
- Claude, ChatGPT
The prompt
You are a senior automation engineer who specializes in shell parallelism with `xargs -P` and safe input handling. I will provide: - The list of work items and where it comes from (file, command output, find results) - The command to run per item and how long each roughly takes - My target CPU/IO budget and whether I can tolerate partial failures Your job: 1. **Make input NUL-safe** — pipe with `-print0`/`-z` and `xargs -0` so filenames with spaces or newlines do not split incorrectly. 2. **Bound parallelism** — set `-P "$(nproc)"` (or an explicit cap) and `-n 1` (or a batch size) and explain the throughput vs. memory tradeoff. 3. **Wrap the per-item command** — move logic into a small exported function or helper script invoked as `xargs ... bash -c '...' _` so each item gets isolated error handling. 4. **Aggregate exit status** — explain that `xargs` returns 123 if any invocation fails, and capture per-item success/failure into a results file. 5. **Interleave output cleanly** — note that parallel stdout interleaves and show buffering per item (write to per-item temp logs, then concatenate). 6. **Handle signals** — ensure Ctrl-C stops spawning new jobs and propagate to children. Output as: one annotated pipeline plus the helper function, using `set -euo pipefail`. Caution explicitly that parallel jobs writing to the same file or rate-limited API will corrupt data or get throttled — call out where I must add per-item isolation.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Parallel Job Execution with xargs and GNU Parallel Prompt
Turn a serial shell loop into a controlled parallel pipeline using xargs -P or GNU parallel, with bounded concurrency, per-job logging, fail-fast vs collect-errors modes, and correct quoting.
-
Bash Multi-Repo Git Bulk Command Runner Prompt
Build a Bash script that runs the same git or shell command across many local repositories with per-repo isolation, dirty-tree guards, and an aggregated pass/fail summary
-
Bash Safe In-Place sed Edit Wrapper Prompt
Build a Bash wrapper around sed -i that backs up, validates, and atomically applies in-place edits so a bad expression can never corrupt or truncate config files
-
Bash Health-Gated Service Restart Orchestrator Prompt
Build a Bash script that restarts services one at a time, waits for each to pass a health check before moving on, and rolls back or aborts on failure to avoid taking a whole fleet down at once
More Bash & Python Automation prompts & error guides
Browse every Bash & Python Automation prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.