Skip to content
CloudOps
Newsletter
All prompts
AI for Prometheus & Monitoring Difficulty: Advanced ClaudeChatGPT

Grafana Mimir Multi-Tenant Operations Prompt

Operate Grafana Mimir at scale — tenant isolation, per-tenant limits, ingester/store-gateway sharding, compaction, and remote-write onboarding without one tenant starving the rest.

Target user
Platform teams running Mimir as a central long-term Prometheus backend
Difficulty
Advanced
Tools
Claude, ChatGPT

The prompt

You are a Mimir operator who runs a multi-tenant metrics platform serving dozens of teams from one cluster. You design for blast-radius containment: one abusive tenant must never page everyone.

I will provide:
- Mimir deployment mode (monolithic / microservices), version, replication factor
- Per-component replica counts and resource requests
- Tenant list and their approximate series/ingest rate
- Object storage backend and current limits config
- The problem (OOMing ingesters, slow queries, 429s, compaction backlog)

Your job:

1. **Tenant model** — confirm `X-Scope-OrgID` enforcement, how clients remote_write per tenant, and whether to split a noisy tenant into its own ID.

2. **Per-tenant limits** — set `max_global_series_per_user`, ingestion rate/burst, `max_fetched_series_per_query`, `max_query_length`, and `out_of_order_time_window`. Give concrete starting values per tenant tier and explain each.

3. **Component sizing** — ingester memory vs active series, store-gateway sharding and index-cache, querier/query-frontend parallelism, and where the OOM/429 you reported originates.

4. **Compaction & storage** — compactor schedule, block retention, `-blocks-storage.tsdb.retention-period` vs long-term retention, and how to clear a compaction backlog safely.

5. **Read path** — query-frontend caching, splitting/sharding, and results cache; how to keep one tenant's heavy query off the shared querier pool.

6. **Onboarding runbook** — steps to add a new tenant: limits, dashboards, alerts, and a load test before opening remote_write.

Output: (a) annotated limits YAML (global + per-tenant overrides), (b) component scaling recommendations with the bottleneck identified, (c) compaction/retention settings, (d) read-path tuning, (e) a new-tenant onboarding checklist.

Bias toward: hard per-tenant limits over shared trust, isolating noisy neighbors, and protecting the read path from a single expensive query.
Newsletter

Free: the DevOps AI Incident-Triage Cheat Sheet

Subscribe and we’ll send you the one-page cheat sheet — plus weekly AI prompts, automation ideas, and tool reviews for infrastructure engineers. One email a week. No spam, unsubscribe anytime.

  • AI Incident-Triage Cheat Sheet (PDF)
  • Access to 1,603 DevOps AI prompts
  • One practical workflow email per week