Design a Telegraf outputs.kafka Pipeline with Topic Routing
Configure the outputs.kafka plugin with dynamic topic routing, partition keys, serialization, compression, acks, and TLS/SASL so Telegraf publishes metrics to Kafka reliably and with sane partition distribution.
- Target user
- Data-platform and observability engineers streaming Telegraf metrics into Kafka for downstream processing or fan-out.
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a Telegraf-to-Kafka pipeline engineer who has tuned producers for durability, ordering, and partition balance in production. I will provide: - The Kafka target: broker list, whether topics are pre-created or auto-created, partition counts, and the security mode (PLAINTEXT / TLS / SASL_SSL) - The metrics and how consumers want them split: single topic, per-measurement topic, per-tenant topic, and any ordering/partition-key needs - Durability requirements: can metrics tolerate loss, and expected throughput Your job: 1. **Design topic routing** — decide between a static `topic`, `topic_tag`/`topic_suffix` dynamic routing, or `routing_tag`/`routing_key` for partitioning; align this with whether brokers auto-create topics and warn about unbounded topic creation from an unbounded routing tag. 2. **Set the partition key** — choose `routing_key` (e.g. `random`, a tag, or host) to balance partitions vs preserve per-source ordering, and explain the hot-partition risk of a low-cardinality or skewed key. 3. **Choose serialization** — pick a `data_format` (influx line protocol, json, carbon2, etc.) that matches the consumer, and note schema-registry/Avro limitations if relevant. 4. **Tune durability and throughput** — set `required_acks` (recommend `-1`/all for no-loss with `min.insync.replicas`), `compression_codec`, `max_message_bytes`, and batch/flush at the agent level; explain the loss modes of acks=0/1 and of buffer overflow. 5. **Secure the connection** — TLS (`tls_ca`/`tls_cert`/`tls_key`, `insecure_skip_verify=false`) and SASL (`sasl_username`/`sasl_password`/`sasl_mechanism`) with the password from a secret store, never inline. 6. **Verify** — how to confirm messages land on expected topics/partitions (console consumer) and how to watch producer errors via internal metrics. Output as: (a) topic + partition routing design, (b) the full outputs.kafka config, (c) durability/throughput tuning with loss-mode analysis, (d) the TLS/SASL security block with secret handling, (e) a consumer-side verification step. Recommend the highest-durability setting the throughput budget allows, and call out every configuration that can lose data or create unbounded topics.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Design a Telegraf outputs.influxdb_v2 Output for Reliable Writes
Configure the outputs.influxdb_v2 plugin with correct org/bucket/token handling, batching, retries, timeouts, and content-coding so writes stay durable and efficient against InfluxDB 2.x / Cloud without dropping metrics under backpressure.
-
Telegraf Kafka Consumer Input Prompt
Configure inputs.kafka_consumer to ingest metrics/events from Kafka topics — consumer group balancing, offset handling, TLS/SASL auth, message parsing, and backpressure — so Telegraf becomes a reliable stream consumer, not a lag generator.
-
Configure Telegraf outputs.prometheus_client for Scrape Exposure
Stand up the outputs.prometheus_client plugin as a scrape target with correct metric_version, expiration, TLS, and string/collector handling so Prometheus reads stable, correctly-typed metrics without stale or duplicated series.
-
Choose and Design the Right Telegraf Input Plugin for a Source
Pick the best Telegraf input plugin for a given data source (an app, host, API, log, queue, or device) and produce a production-ready inputs configuration with the right tags, fields, interval, and parser — instead of guessing between exec, http, prometheus, and a native plugin.
More Telegraf prompts & error guides
Browse every Telegraf prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.