OpenTelemetry Tail Sampling Strategy Prompt

You are a senior observability engineer who designs cost-effective trace sampling without losing signal on incidents.

I will provide:
- Current trace volume (spans/sec, traces/sec), retention target, and backend cost per span/trace
- The services involved and which are latency- or error-sensitive
- Current sampling approach (head, none, vendor default) and the pain (cost, missing errors, blind spots)
- Collector deployment mode and replica count

Your job:

1. **Head vs tail** — decide what belongs at head (cheap, per-request, e.g. debug/health filtering) vs tail (error/latency/rare-path retention) and justify the split.
2. **Load balancing** — specify the loadbalancing exporter routed by traceID in front of a gateway tier so every span of a trace lands on one instance; show the two-tier config.
3. **Policy design** — build an ordered tail_sampling policy set: status_code=ERROR, latency threshold, specific attribute matches (tenant, route), rate_limiting, and a low probabilistic base rate as the catch-all.
4. **Decision tuning** — set decision_wait, num_traces, and expected_new_traces_per_sec against my volume, and explain the memory implication of each.
5. **Guardrails** — ensure error and high-latency traces are always kept (100%) regardless of the base probabilistic rate, and add a rate limit so a storm can't blow the budget.
6. **Cost model** — estimate retained spans/sec and monthly cost after the policy, versus today.
7. **Validation** — describe how to confirm sampling behaves as intended (synthetic error/slow requests, checking they survive) before full rollout.

Output as: (a) two-tier Collector YAML (LB tier + sampling tier), (b) the ordered policy table with rationale, (c) memory/decision_wait sizing math, (d) a cost-before/after estimate, (e) a validation checklist.

Flag any policy ordering or topology gap that would cause errors or slow traces to be dropped.