Skip to content
DevOps AI ToolKit
Newsletter
All prompts
AI for Telegraf Difficulty: Advanced ClaudeChatGPTCursor

Design a Telegraf outputs.kafka Pipeline with Topic Routing

Configure the outputs.kafka plugin with dynamic topic routing, partition keys, serialization, compression, acks, and TLS/SASL so Telegraf publishes metrics to Kafka reliably and with sane partition distribution.

Target user
Data-platform and observability engineers streaming Telegraf metrics into Kafka for downstream processing or fan-out.
Difficulty
Advanced
Tools
Claude, ChatGPT, Cursor

The prompt

You are a Telegraf-to-Kafka pipeline engineer who has tuned producers for durability, ordering, and partition balance in production.

I will provide:
- The Kafka target: broker list, whether topics are pre-created or auto-created, partition counts, and the security mode (PLAINTEXT / TLS / SASL_SSL)
- The metrics and how consumers want them split: single topic, per-measurement topic, per-tenant topic, and any ordering/partition-key needs
- Durability requirements: can metrics tolerate loss, and expected throughput

Your job:

1. **Design topic routing** — decide between a static `topic`, `topic_tag`/`topic_suffix` dynamic routing, or `routing_tag`/`routing_key` for partitioning; align this with whether brokers auto-create topics and warn about unbounded topic creation from an unbounded routing tag.

2. **Set the partition key** — choose `routing_key` (e.g. `random`, a tag, or host) to balance partitions vs preserve per-source ordering, and explain the hot-partition risk of a low-cardinality or skewed key.

3. **Choose serialization** — pick a `data_format` (influx line protocol, json, carbon2, etc.) that matches the consumer, and note schema-registry/Avro limitations if relevant.

4. **Tune durability and throughput** — set `required_acks` (recommend `-1`/all for no-loss with `min.insync.replicas`), `compression_codec`, `max_message_bytes`, and batch/flush at the agent level; explain the loss modes of acks=0/1 and of buffer overflow.

5. **Secure the connection** — TLS (`tls_ca`/`tls_cert`/`tls_key`, `insecure_skip_verify=false`) and SASL (`sasl_username`/`sasl_password`/`sasl_mechanism`) with the password from a secret store, never inline.

6. **Verify** — how to confirm messages land on expected topics/partitions (console consumer) and how to watch producer errors via internal metrics.

Output as: (a) topic + partition routing design, (b) the full outputs.kafka config, (c) durability/throughput tuning with loss-mode analysis, (d) the TLS/SASL security block with secret handling, (e) a consumer-side verification step.

Recommend the highest-durability setting the throughput budget allows, and call out every configuration that can lose data or create unbounded topics.

Run this prompt with AI

Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.

Related prompts

More Telegraf prompts & error guides

Browse every Telegraf prompt and troubleshooting guide in one place.

Free download · 368-page PDF

Reading prompts? Get all 500 in one free PDF

500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.

  • 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
  • Instant PDF download — yours free, forever
  • Plus one practical AI-workflow email a week (no spam)

Single opt-in · unsubscribe anytime · no spam.