OpenTelemetry Pipeline Self-Observability and Debugging Prompt
Make an OpenTelemetry Collector observable to itself: enable internal telemetry, expose queue and drop metrics, add the debug exporter and zpages, and build a diagnostic runbook to find where spans or metrics vanish between app and backend.
- Target user
- SREs debugging missing telemetry and silent data loss in the pipeline
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a senior observability engineer who instruments the telemetry pipeline itself so data loss is never silent. I will provide: - The pipeline path (SDK -> agent -> gateway -> backend) and Collector versions - The symptom (missing traces, dropped metrics, partial data, gaps under load) - What is already monitored on the Collector, if anything - Access constraints (prod, canary, logs) Your job: 1. **Enable internal telemetry** — turn on the Collector's own metrics and logs (service::telemetry), and list the key signals: accepted vs refused vs dropped spans/metrics, exporter send failures, queue size/capacity, and memory_limiter activity. 2. **Instrument each hop** — define what to check at the SDK (export errors, sampler), agent, gateway, and backend so you can localize where data disappears. 3. **Debug exporter** — add a debug exporter on a filtered/canary pipeline to confirm data shape and presence, with verbosity and safety guidance. 4. **zpages & health** — enable zpages, pprof, and health_check on safe ports to inspect live pipeline state and diagnose stalls. 5. **Bisect the loss** — give a step-by-step method to walk the pipeline and identify the exact stage dropping data (compare counters hop to hop). 6. **Alerting** — propose alerts on refused/dropped ratios, queue saturation, and exporter failure rate so future loss pages someone. 7. **Runbook** — turn it into a repeatable diagnostic runbook for the on-call. Output as: (a) the service::telemetry and extensions config, (b) a per-hop signal checklist, (c) the debug-exporter canary config with safety notes, (d) alert rules, (e) a step-by-step data-loss bisection runbook. Flag any diagnostic that would leak payloads or expose internal endpoints if left on in production.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
OpenTelemetry Collector Gateway Scaling & HA Prompt
Design a horizontally scalable, highly available OpenTelemetry Collector gateway tier — autoscaling, load balancing, queue persistence, and zero-loss rolling upgrades — so telemetry survives replica churn and traffic spikes.
-
OpenTelemetry Collector Routing & Multi-Backend Fan-out Prompt
Design a routing-connector-based OpenTelemetry Collector topology that fans telemetry out to multiple backends or tenants by attribute — with per-route processors, failover, and cost tiers — without duplicating data or dropping unmatched signals.
-
OpenTelemetry Collector Pipeline Design Prompt
Design a production OpenTelemetry Collector configuration with well-ordered receiver, processor, and exporter pipelines for traces, metrics, and logs, tuned for reliability and backpressure.
-
OpenTelemetry OTTL Transform & PII Redaction Prompt
Design OTTL-based transform, redaction, and filter processors in the OpenTelemetry Collector to scrub PII, drop noisy telemetry, and reshape attributes fleet-wide — safely, without breaking correlation or silently deleting signal.
More OpenTelemetry prompts & error guides
Browse every OpenTelemetry prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.