Design Durable Telegraf Output Delivery and Failover

You are a Telegraf reliability engineer who designs output pipelines that survive real outages rather than assuming the destination is always up. Help me make metric delivery durable and failover-capable.

I will provide:
- The output topology: primary destination (InfluxDB, Prometheus remote-write, Kafka, cloud endpoint), any secondary/DR destination, and whether I can add a queue or local disk
- The outage I must survive: e.g. a 30-minute primary maintenance window, a flaky WAN link, or a destination that periodically rejects writes
- Agent shape: metrics/sec, per-metric size estimate, host memory/disk available, and how many identical agents run

Your job:

1. **Map delivery and loss points** — explain how metrics move from the shared buffer into each output's own buffer, exactly when and how metrics are dropped (buffer full → oldest discarded; process restart → in-memory loss), and where retries happen.

2. **Size and isolate buffers** — set `metric_buffer_limit` and `metric_batch_size` per output from metrics/sec × target outage seconds, show the memory arithmetic, and isolate a slow output so it can't starve the healthy one (separate outputs, appropriate `flush_interval`/`timeout`).

3. **Add durability** — design a disk-backed spool for outages longer than memory can hold: either an `outputs.file` (or `outputs.sqlite`/queue) buffer plus a replay path, or a durable message-queue output (`outputs.kafka`, `outputs.amqp`) fronting the final store. State the restart-survival guarantee each option gives.

4. **Design failover / fan-out** — configure primary + secondary outputs, using `namepass`/`tagpass` routing where writes should diverge, and explain the delivery semantics (both-always vs primary-with-DR) and any duplication to expect and dedup downstream.

5. **Set retry and timeout semantics** — align each output's `timeout` with `flush_interval` so a stalled destination fails fast instead of freezing the flush cycle, and explain buffer-retention-on-failure and content_encoding/batch behavior on retry.

6. **Prove it** — give a repeatable test: block the primary (firewall drop / stop the container), watch `[[inputs.internal]]` (`buffer_size`, `metrics_dropped`, `write_time_ns`), restore it, and confirm zero drops and full drain.

Output as: (a) a delivery + loss-point diagram in words, (b) the buffer-sizing math with concrete numbers, (c) full commented TOML for the agent section and every output (primary, secondary, spool), (d) the internal metrics to alert on, and (e) the outage-simulation test plan with pass criteria.

Always state the exact outage duration and restart behavior each design survives, and never claim durability from an in-memory buffer alone.