Skip to content
DevOps AI ToolKit
Newsletter
All prompts
AI for OpenTelemetry Difficulty: Advanced ClaudeChatGPTCursor

OpenTelemetry Pipeline Self-Observability and Debugging Prompt

Make an OpenTelemetry Collector observable to itself: enable internal telemetry, expose queue and drop metrics, add the debug exporter and zpages, and build a diagnostic runbook to find where spans or metrics vanish between app and backend.

Target user
SREs debugging missing telemetry and silent data loss in the pipeline
Difficulty
Advanced
Tools
Claude, ChatGPT, Cursor

The prompt

You are a senior observability engineer who instruments the telemetry pipeline itself so data loss is never silent.

I will provide:
- The pipeline path (SDK -> agent -> gateway -> backend) and Collector versions
- The symptom (missing traces, dropped metrics, partial data, gaps under load)
- What is already monitored on the Collector, if anything
- Access constraints (prod, canary, logs)

Your job:

1. **Enable internal telemetry** — turn on the Collector's own metrics and logs (service::telemetry), and list the key signals: accepted vs refused vs dropped spans/metrics, exporter send failures, queue size/capacity, and memory_limiter activity.
2. **Instrument each hop** — define what to check at the SDK (export errors, sampler), agent, gateway, and backend so you can localize where data disappears.
3. **Debug exporter** — add a debug exporter on a filtered/canary pipeline to confirm data shape and presence, with verbosity and safety guidance.
4. **zpages & health** — enable zpages, pprof, and health_check on safe ports to inspect live pipeline state and diagnose stalls.
5. **Bisect the loss** — give a step-by-step method to walk the pipeline and identify the exact stage dropping data (compare counters hop to hop).
6. **Alerting** — propose alerts on refused/dropped ratios, queue saturation, and exporter failure rate so future loss pages someone.
7. **Runbook** — turn it into a repeatable diagnostic runbook for the on-call.

Output as: (a) the service::telemetry and extensions config, (b) a per-hop signal checklist, (c) the debug-exporter canary config with safety notes, (d) alert rules, (e) a step-by-step data-loss bisection runbook.

Flag any diagnostic that would leak payloads or expose internal endpoints if left on in production.

Run this prompt with AI

Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.

Related prompts

More OpenTelemetry prompts & error guides

Browse every OpenTelemetry prompt and troubleshooting guide in one place.

Free download · 368-page PDF

Reading prompts? Get all 500 in one free PDF

500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.

  • 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
  • Instant PDF download — yours free, forever
  • Plus one practical AI-workflow email a week (no spam)

Single opt-in · unsubscribe anytime · no spam.