OpenTelemetry Collector Pipeline Design Prompt
Design a production OpenTelemetry Collector configuration with well-ordered receiver, processor, and exporter pipelines for traces, metrics, and logs, tuned for reliability and backpressure.
- Target user
- Observability and platform engineers running the OTel Collector
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a senior observability engineer who designs OpenTelemetry Collector deployments for high-throughput production environments. I will provide: - The current Collector version and deployment mode (agent DaemonSet, gateway Deployment, or both) - Expected ingest volume per signal (spans/sec, datapoints/sec, log lines/sec) and peak-to-average ratio - Source receivers in use (OTLP gRPC/HTTP, prometheus, filelog, hostmetrics, kafka, etc.) - Destination backends and their auth/protocol requirements - Any current config YAML and observed problems (drops, OOMs, latency, gaps) Your job: 1. **Topology** — recommend agent vs gateway split, whether to fan-in through a gateway tier, and where load balancing (loadbalancing exporter by trace ID) belongs for tail sampling. 2. **Receivers** — configure each receiver with sane limits (max_recv_msg_size, keepalive, read buffers) and TLS/auth where the source requires it. 3. **Processor ordering** — lay out each pipeline with the correct order: memory_limiter first, then any transform/attributes/resource/k8sattributes/filter processors, then batch last. Explain why the order matters. 4. **Exporters** — configure exporters with retry_on_failure, sending_queue (with persistent storage extension for durability), and timeout tuning per backend. 5. **Separation of signals** — define distinct pipelines for traces, metrics, and logs, and note where a shared processor is safe vs where it must be per-signal. 6. **Resilience** — add the health_check, pprof, and zpages extensions, plus file_storage for queue persistence across restarts. 7. **Capacity** — size memory_limiter (limit_mib, spike_limit_mib) and sending_queue depth against the ingest numbers I gave, and state the CPU/memory request/limit you'd set. Output as: (a) annotated full Collector YAML, (b) a table explaining each processor's placement, (c) capacity/resource recommendations, (d) a rollout and rollback plan with the exact validation command. Call out any place where my current config risks silent telemetry loss or unbounded memory growth.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
OpenTelemetry Collector Gateway Scaling & HA Prompt
Design a horizontally scalable, highly available OpenTelemetry Collector gateway tier — autoscaling, load balancing, queue persistence, and zero-loss rolling upgrades — so telemetry survives replica churn and traffic spikes.
-
OpenTelemetry Collector Routing & Multi-Backend Fan-out Prompt
Design a routing-connector-based OpenTelemetry Collector topology that fans telemetry out to multiple backends or tenants by attribute — with per-route processors, failover, and cost tiers — without duplicating data or dropping unmatched signals.
-
OpenTelemetry Pipeline Self-Observability and Debugging Prompt
Make an OpenTelemetry Collector observable to itself: enable internal telemetry, expose queue and drop metrics, add the debug exporter and zpages, and build a diagnostic runbook to find where spans or metrics vanish between app and backend.
-
OpenTelemetry OTTL Transform & PII Redaction Prompt
Design OTTL-based transform, redaction, and filter processors in the OpenTelemetry Collector to scrub PII, drop noisy telemetry, and reshape attributes fleet-wide — safely, without breaking correlation or silently deleting signal.
More OpenTelemetry prompts & error guides
Browse every OpenTelemetry prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.