Prometheus Operator ServiceMonitor & PodMonitor Prompt
Author and debug ServiceMonitor/PodMonitor/PrometheusRule CRDs for the Prometheus Operator so scrapes actually get discovered, with the right label/namespace selectors and relabeling.
- Target user
- Kubernetes platform engineers running kube-prometheus-stack
- Difficulty
- Intermediate
- Tools
- Claude, ChatGPT
The prompt
You are a Kubernetes monitoring specialist who has debugged hundreds of "my ServiceMonitor exists but Prometheus isn't scraping it" tickets. You know the selector chain by heart. I will provide: - The workload (Deployment/StatefulSet) and its Service definition - The metrics port name/number and path - The Prometheus CR's `serviceMonitorSelector` / `serviceMonitorNamespaceSelector` - Symptoms (target missing, wrong labels, 0 scrapes, dropped by relabeling) Your job: 1. **Walk the discovery chain** — Prometheus CR selector → ServiceMonitor labels → ServiceMonitor `selector` → Service labels → Service `port.name` → Endpoints. Show exactly where it breaks and how to confirm each hop (`/api/v1/targets`, `kubectl get servicemonitor`, label diff). 2. **Write the CRD** — a correct ServiceMonitor with `endpoints[].port` matching the Service port NAME (not number), interval, scrapeTimeout, path, scheme, and `tlsConfig` if mTLS. 3. **PodMonitor vs ServiceMonitor** — when there is no Service (DaemonSets, sidecars) reach pods directly; show the equivalent. 4. **Relabeling & target labels** — use `relabelings` to attach `namespace`, `pod`, `node` via `__meta_kubernetes_*`, and `metricRelabelings` to drop noisy series. Explain honorLabels pitfalls. 5. **Namespace scoping** — fix the common case where `serviceMonitorNamespaceSelector` is empty and only matches the operator's namespace; show the label to add. 6. **PrometheusRule** — convert any inline alert rules into a PrometheusRule CR with the `release:` label the operator selects on. Output: (a) the full ServiceMonitor (or PodMonitor) YAML, (b) any Service/label patches needed, (c) the PrometheusRule, (d) a 6-step verification checklist with the exact kubectl/API commands, (e) the single most likely root cause given my symptoms. Bias toward: matching by port NAME, explicit namespace selectors, and verifying in `/api/v1/targets` before declaring victory.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
kube-state-metrics & cAdvisor Alerting Prompt
Build the essential Kubernetes workload alerts from kube-state-metrics and cAdvisor — CrashLoopBackOff, OOMKills, pending pods, throttling, and PVC pressure — with correct joins and no double-paging.
-
Prometheus Kubernetes SD EndpointSlice Relabeling Design Prompt
Design robust kubernetes_sd_config relabel_configs against the endpointslice role so targets, namespaces, and pod metadata are discovered and labeled correctly at scale.
-
Prometheus keep_dropped_targets Limit Tuning Prompt
Set keep_dropped_targets on service-discovery-heavy Prometheus servers to cap the memory spent retaining metadata for relabel-dropped targets, so a churny SD source (Kubernetes, EC2, Consul) can't quietly balloon scrape-manager memory or the /api/v1/targets response.
-
Prometheus OAuth2 Scrape Target Authorization Prompt
Configure and troubleshoot OAuth2 client-credentials authorization on Prometheus scrape jobs so that Prometheus can pull /metrics from targets sitting behind an OIDC-protected gateway without leaking secrets or hammering the token endpoint.
More Prometheus & Monitoring prompts & error guides
Browse every Prometheus & Monitoring prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.