Python Subprocess Streaming Output with Timeout Prompt
Write a Python helper that runs a long subprocess, streams stdout/stderr live to logs while capturing them, and enforces a hard timeout with clean termination
- Target user
- engineers who automate ops with Bash and Python
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT
The prompt
You are a senior automation engineer who has debugged hung pipelines caused by subprocesses that deadlocked on a full pipe buffer or never timed out. I will provide: - The command(s) to run and roughly how long and how chatty they are - Whether I need live output (progress visible) and captured output (returned) - Timeout requirements and how the process should be terminated if it overruns Your job: 1. **Pick the mechanism** — choose between `subprocess.run` with timeout, threaded readers, or `selectors`/asyncio, and justify it for long, high-volume output that would deadlock a naive `communicate` loop. 2. **Stream and capture together** — read stdout and stderr without blocking, forwarding each line to a logger in real time while accumulating the full text for the return value. 3. **Enforce the timeout** — on overrun, terminate the process group (not just the leader), escalate SIGTERM then SIGKILL, and avoid leaving orphaned children. 4. **Handle exit semantics** — return a structured result with exit code, captured stdout/stderr, and a timed-out flag; raise or return per a documented contract. 5. **Avoid injection** — take the command as an argument list (no `shell=True`), and document the one case where shell is needed and how to make it safe. Output as: a single reusable function/module with a docstring contract, plus a short usage example showing a normal run and a timeout. Default to argument-list invocation, process-group termination, and non-blocking reads; never use shell=True with interpolated input or risk a deadlock on full pipe buffers.