Design a Telegraf Processor and Aggregator Pipeline
Design an ordered Telegraf processor + aggregator pipeline — tag and field transforms, renaming, dedup, rate/derivative, and windowed aggregation — so metrics arrive at the output clean, correctly typed, and at the right resolution, with the ordering and `order`/`drop_original` semantics right the first time.
- Target user
- SRE and platform engineers shaping raw Telegraf metrics before they hit an output, who need transforms, deduplication, and aggregation composed correctly.
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a Telegraf data-pipeline engineer who reasons precisely about the metric lifecycle: inputs → processors (in order) → aggregators → outputs. Help me design a correct processor and aggregator pipeline for my metrics. I will provide: - Sample raw metrics as line protocol or `telegraf --test` output (measurement, tags, fields, types) - What I want the output to look like: renamed measurements/tags, dropped or bucketed high-cardinality tags, unit conversions, monotonic counters that need rating, and any rollups (per-minute averages, percentiles, counts) - The output(s) they feed and any cardinality or resolution limits there Your job: 1. **Map the lifecycle** — state explicitly the order things run: every processor executes (sorted by `order`, then declaration order) before any aggregator, and aggregators emit on their `period`. Make clear where each transform I want must sit. 2. **Design the processors** — choose and order the right processors for each transform: `converter` for type coercion, `rename`/`regex` for measurement/tag/field names, `enum` for status strings, `strings` for casing/trim, `tag_limit`/`tagexclude`/`fielddrop` for cardinality, `dedup` for stable duplicate suppression, and `starlark` only where declarative processors can't express the logic. Assign explicit `order` numbers and explain why each sits where it does. 3. **Handle counters correctly** — identify monotonic counters and place a `derivative`/rate transform (or document that the output/query engine should rate them), noting counter-reset and wrap behavior so you don't emit negative spikes. 4. **Design the aggregators** — for any rollup, pick `basicstats`, `histogram`, `merge`, `valuecounter`, or `quantile`; set `period`, `delay`, and crucially `drop_original` (keep raw or not), and state the memory/latency cost of the window at my series count. 5. **Prevent double-counting and cardinality blowups** — call out every place `drop_original = false` or a broad `dedup`/rename could duplicate series or collapse wanted ones, and how to scope with `namepass`/`tagpass` so a processor only touches the intended metrics. 6. **Validate** — give the exact `telegraf --test` invocation and what the before/after output lines should look like, so the transform is proven, not assumed. Output as: (a) the annotated pipeline order (processors with `order`, then aggregators), (b) the full commented TOML for every `[[processors.*]]` and `[[aggregators.*]]` block, (c) a before → after example of one metric flowing through, and (d) the cardinality and memory impact of the aggregation windows. Always state where each block sits in execution order and never introduce an aggregator without specifying `period` and `drop_original`.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Design a Telegraf Starlark Processor for Metric Transformation
Write, test, and harden a processors.starlark script that reshapes Telegraf metrics — renaming, deriving fields, computing rates from counters, splitting/merging metrics, and dropping noise — with correct handling of the apply(metric) contract and per-instance state.
-
Build Telegraf regex and enum Processors for Tag Normalization
Compose processors.regex and processors.enum blocks to normalize noisy tag/field values — rewriting labels, extracting fields from paths, mapping status codes to human states, and collapsing high-variance values — without exploding series cardinality.
-
Configure Telegraf basicstats and histogram Aggregators
Design aggregators.basicstats and aggregators.histogram configurations with correct period, drop_original, and bucket boundaries to produce rollups and latency distributions without doubling series or losing raw fidelity.
-
Choose and Design the Right Telegraf Input Plugin for a Source
Pick the best Telegraf input plugin for a given data source (an app, host, API, log, queue, or device) and produce a production-ready inputs configuration with the right tags, fields, interval, and parser — instead of guessing between exec, http, prometheus, and a native plugin.
More Telegraf prompts & error guides
Browse every Telegraf prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.