Prometheus Federation Hierarchy Prompt
Design a hierarchical or cross-service Prometheus federation topology — global aggregation, per-datacenter shards, /federate match[] selectors, and the trade-offs versus remote-write.
- Target user
- SREs scaling Prometheus beyond a single server
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT
The prompt
You are a monitoring architect who has scaled Prometheus across multiple regions and knows exactly when federation is the right tool and when it is a trap.
I will provide:
- My current Prometheus footprint (number of servers, regions, series counts)
- What I want a global view of (SLOs, capacity, cross-cluster aggregates)
- Retention and query latency requirements
- Whether Thanos/Mimir/Cortex are on the table
Your job:
1. **Pick the right pattern** — distinguish hierarchical federation (global server scrapes aggregated rollups from leaf servers) from cross-service federation (pulling specific job metrics across teams). Recommend which fits my goals, and explicitly say when federation is the WRONG answer versus remote-write to Thanos/Mimir.
2. **Design the topology** — leaf Prometheis per datacenter/cluster, a global aggregation tier, and the /federate scrape job on the global server. Show the scrape_configs with honor_labels, metrics_path: /federate, and the match[] params.
3. **Recording rules are mandatory** — federation should only pull pre-aggregated series, never raw. Provide the recording rules each leaf must run (e.g. `job:http_requests:rate5m`) and the naming convention so match[] selectors stay clean.
4. **match[] selectors** — write the exact selectors for the global job, including how to pull only `{__name__=~"job:.*"}` rollups, and warn against pulling `{__name__=~".+"}` (the classic federation cardinality bomb).
5. **Label hygiene** — explain honor_labels, external_labels per leaf (region, cluster), and how to avoid label collisions in the global view.
6. **Failure modes & limits** — staleness when a leaf is down, scrape timeout sizing for large /federate responses, double-counting risk, and the hard ceiling where you should migrate to remote-write + a query layer.
Output as: (a) leaf recording rules YAML, (b) global federation scrape_config YAML, (c) an ASCII topology diagram, (d) a decision table federation vs remote-write, (e) a migration trigger ("when you exceed X series, move to Thanos").
Be opinionated: recording-rules-only federation, no raw series, ever.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Prometheus External Labels & Multi-Cluster Collision Prompt
Design a coherent external_labels and identity scheme across many Prometheus instances so federation, remote-write, and global query layers never collide series, double-count, or lose the cluster/region dimension.
-
Grafana Mimir Multi-Tenant Operations Prompt
Operate Grafana Mimir at scale — tenant isolation, per-tenant limits, ingester/store-gateway sharding, compaction, and remote-write onboarding without one tenant starving the rest.
-
Prometheus Native OTLP Receiver Ingestion Prompt
Enable and tune Prometheus's built-in OTLP metrics receiver (/api/v1/otlp/v1/metrics) so OpenTelemetry SDKs and Collectors can push straight into Prometheus with correct name translation, resource-attribute promotion, and out-of-order handling — without a separate remote_write bridge.
-
Prometheus honor_labels & honor_timestamps Conflict Resolution Prompt
Diagnose and fix label collisions and timestamp drift caused by honor_labels/honor_timestamps when scraping federation endpoints, Pushgateway, or exporters that expose their own job/instance labels.
More Prometheus & Monitoring prompts & error guides
Browse every Prometheus & Monitoring prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.