SRE Prompts: Force SLO Math, Keep Agents From Changing State

This page gives you production-ready SRE prompt patterns you can copy, paste, and adapt today: discovery, diagnostics, SLO and error-budget templates, incident triage, automation, and the guardrails that keep an agent safe. Treat every output as analysis first, require human approval before any state change, and expect faster triage, clearer runbooks, and prompt assets you can reuse instead of rebuilding from scratch each incident.

---

TL;DR:

• For diagnostics, collect timestamped raw logs first, rank three hypotheses next, and request one check that can distinguish the leading causes before proposing a fix.
• For SLO calculations, define the SLI, measurement window, and historical baseline first; require the formula, since targets without these inputs invite invented numbers.
• During incidents, assign a commander, operations lead, communications lead, and scribe, then timestamp each observation, action, approval, and expected outcome in the living document.
• Schedule a daily 9 AM check for production pods, and alert when any pod exceeds three restarts in an hour.
• Version prompts, review changes, and test them against known incidents; adapt names, severity definitions, and output formats to match your own infrastructure before rollout.

---

Table of Contents

• General starter prompts and agent-scope questions
• Resource discovery prompts for Kubernetes and cloud resources
• Diagnostics prompts for logs, 500s, OOMs, and crashloops
• SLO and error-budget prompts that force real arithmetic
• Incident triage prompts and the living incident document
• Automation and scheduling prompts that turn toil into tickets
• Agent guardrails and human-approval workflows
• How we build and maintain our prompt packs
• Prompts for root cause analysis and post-incident review
• Prompts for capacity planning and resource optimization
• Prompts for automated alert enrichment and contextualization
• Integrating AI prompts with your existing DevOps tools and CI/CD pipelines
• Tuning prompts for your specific infrastructure environment
• Prompts for security incident detection and response
• Why prompts deserve the same rigor as production code
• Try our battle-tested prompts and incident response tools
• FAQ
• Sources

General starter prompts and agent-scope questions

Before you hand an agent a real incident, you need to know what it can actually see and touch. A scope prompt forces that answer out loud, which saves you from discovering mid-incident that your agent can’t read the namespace that’s on fire.

Start every new integration with prompts that map capability before you trust output:

• “List every system, cluster, and observability source you currently have access to, and note which ones are read-only.”
• “Is this request analysis or execute? If execute, state exactly what permissions you’d need and wait for approval.”
• “Give me a one-page inventory of the resources you can see right now, grouped by environment, as a markdown table.”

Microsoft’s Azure SRE Agent documentation frames this same idea as a starter prompt pattern, with general-scope questions sitting ahead of any App Service, Container Apps, or AKS specific request. That sequencing matters: an agent that can’t state its own boundaries shouldn’t be trusted with a rollback.

Resource discovery prompts for Kubernetes and cloud resources

Once scope is confirmed, discovery prompts turn a sprawling environment into a short, readable list of facts. The goal isn’t insight yet, just evidence you can scan in under a minute.

Useful discovery prompts to keep in your library:

• “Which node pools are configured for my AKS cluster, and what’s the current utilization on each?”
• “List every pod in crashloop state across all namespaces in the last hour, with restart counts.”
• “Show me the image tag running on each workload in the payments namespace.”
• “What liveness and readiness probes are configured on this deployment, and what are their thresholds?”

Microsoft’s own starter prompts for Azure SRE Agent include near-identical phrasing, such as asking which node pools exist for a given AKS cluster, which confirms this is the right level of specificity for a discovery prompt. Scope these by namespace, a time window, or a correlation ID when the environment is large, and always ask for compact CSV or markdown output. A firehose of raw JSON defeats the point of asking a model to summarize anything.

Diagnostics prompts for logs, 500s, OOMs, and crashloops

Diagnostics prompts fail when they ask for a conclusion before they ask for facts. The better pattern separates the two steps explicitly: gather raw evidence first, then rank hypotheses, then propose a discriminating check that would confirm or rule one out.

1. Ask for raw facts: “Pull the last 200 log lines for pod api-7f4d with timestamps, and quote the exact error text without summarizing.”
2. Ask for ranked hypotheses: “Given these logs and the fact that memory usage spiked at 14:02, list the three most likely causes, ordered by probability, with your reasoning for each.”
3. Ask for a discriminating check: “What single check would tell us definitively whether this is an OOM kill versus an application-level crash?”
4. Ask for verification before action: “Before suggesting a rollback, what evidence would confirm the previous image version doesn’t have this bug?”

Microsoft’s Container Apps troubleshooting guide shows this exact shape in practice: the agent identifies an invalid image tag as the likely cause, recommends a rollback to the last working version, and explicitly flags that the recommendation needs verification because model output can vary from the documented example.

Pro Tip: Never let a diagnostic prompt skip straight to “fix it.” Ask for the hypothesis and the check first, every time, even when you’re confident you already know the answer.

SLO and error-budget prompts that force real arithmetic

A prompt that asks “what should our uptime target be?” invites a made-up number. A better prompt forces the model to show its work: the SLI definition, the measurement window, the historical baseline, and the arithmetic connecting an incident to the error budget it consumed.

Templates worth keeping on hand:

• “Define a candidate SLI for checkout latency, state the measurement window you’re assuming, and show the formula you’d use to calculate the error budget.”
• “Given a 99.9% availability target over a 30-day window, how much error budget remains after this incident, and show the calculation.”
• “Based on error-budget consumption this month, should we pause the next feature launch? State your reasoning and what data would change your answer.”

The error-budget math only means something when the SLI and window are explicit first, a point Google’s SRE Workbook makes directly: SLO work should force a decision conversation among stakeholders, not substitute a model’s guess for that trade-off. Treat any SLO prompt that skips the window or baseline as incomplete, and push back on arbitrary percentage targets the same way you’d push back on a teammate pulling a number out of the air.

Incident triage prompts and the living incident document

A good incident triage prompt resists the urge to diagnose immediately. It collects the shape of the problem first: what was expected, what actually happened, how wide the blast radius is, what changed recently, and where the telemetry lives. Only after that does it ask for hypotheses.

1. Collect the baseline: “What was the expected behavior, what’s actually happening, which services and regions are affected, and what changed in the last two hours? Link the relevant dashboards.”
2. Assign roles explicitly: “Confirm who is incident commander, ops lead, comms lead, and scribe for this incident, and note the assignment in the living doc with a timestamp.”
3. Maintain the living document: “Update the incident doc with this new observation, the action taken, who approved it, and what we expect to see next.”
4. Rank mitigations with approval checkpoints: “List three possible mitigations ranked by risk and reversibility, and flag which ones require commander approval before execution.”

Assigning named roles instead of leaving them implicit cuts confusion fast, and Google’s incident management guidance treats the living document itself as the highest-leverage habit in the whole process: update it after every observation, with timestamps, owners, and the reasoning behind each action. Our own breakdown of incident response roles goes deeper on how commander, scribe, comms, and ops split responsibility during a live incident.

Pro Tip: If your incident doc doesn’t have a timestamp on every entry, it’s a chat log, not a living document. The timestamp is what lets a scribe or a later reviewer reconstruct the decision tree.

Automation and scheduling prompts that turn toil into tickets

Recurring checks are where prompts earn their keep quietly, in the background, without anyone needing to remember to run them. The template matters more than the specific check: name the cadence, tie it to a real query, and define what counts as a pass.

• “Create a health check task that runs every day at 9 AM, checks pod restart counts across production namespaces, and alerts if any pod restarted more than three times in an hour.”
• “Classify the last week’s alerts into actionable versus non-actionable, and estimate how many engineer-hours were spent on the non-actionable ones.”
• “Write a ticket description for automating this manual check, including acceptance criteria and the observability query it should run.”

Microsoft’s starter prompts include this exact scheduling pattern, creating a recurring health check task at a fixed cadence tied to a specific query. Our alert fatigue reduction prompt extends this into a repeatable classification workflow you can run weekly.

Agent guardrails and human-approval workflows

The single most important habit in this entire prompt library is refusing to let an agent cross from analysis into execution without a named human saying yes. Every prompt pattern above assumes that boundary; this section makes it explicit.

• Require the agent to label every response as “analysis” or “execute,” with execute responses blocked until an approval token is logged.
• Use a confinement wrapper that separates the agent’s observability interface from anything with write access, so a bad instruction can’t accidentally touch production state.
• Keep summary telemetry available to the agent, but route raw write access through a separate, audited path a human controls.

Agentic reliability work only holds up when read-only investigation is the default, not the exception, according to the Stratus paper on agentic cloud reliability, which recommends confinement wrappers and treating model output as analysis until a designated human approves execution. Our post on building an AI ops copilot with guardrails walks through what that confinement layer looks like in a real pipeline.

How we build and maintain our prompt packs

We maintain our prompt libraries the same way we’d maintain any production code: reviewed, versioned, and tested against real failure scenarios before they ship. Our incident response prompt library holds over a hundred copy-paste prompts organized by triage stage, and our validators catch config errors before a prompt-driven change ever reaches a cluster.

A short adoption checklist we recommend to any team starting out:

• Review a prompt pack against your own observability stack before trusting it blind.
• Sandbox-test diagnostic prompts on a known incident before using them live.
• Adapt variable names, namespaces, and alert thresholds to match your environment.
• Add an explicit approval gate for any prompt that touches state.
• Monitor the first few uses closely, then widen adoption once the pattern holds.

Prompts for root cause analysis and post-incident review

Root cause prompts fail for the same reason diagnostic prompts fail: they get asked too early, before the facts are settled. The sequence that works treats root cause analysis as a distinct phase that starts only after the incident is mitigated, not during the scramble.

A useful sequence looks like this:

• “Summarize the timeline of this incident from the living doc, including every action taken and its outcome.”
• “Based on this timeline, what is the most likely root cause, and what evidence directly supports it versus what’s still a hypothesis?”
• “Were there any contributing factors beyond the immediate trigger, such as alerting gaps, missing runbook steps, or delayed escalation?”
• “Draft a post-incident review document with sections for summary, impact, timeline, root cause, contributing factors, and follow-up actions with owners.”

Google’s troubleshooting methodology frames this as the hypothetico-deductive method applied to systems: collect expected versus actual behavior and scope first, then move to ranked hypotheses, then to the discriminating check that confirms the real cause. That same structure belongs in your post-incident review, not just the live incident.

The follow-up action list is where most reviews lose their value, because vague items like “improve monitoring” never get done. Push the prompt further: “For each follow-up action, assign an owner, a target date, and a specific, testable definition of done.” A review that doesn’t produce a ticket with an owner is a document nobody will reread.

Prompts for capacity planning and resource optimization

Capacity prompts work best when they’re anchored to a concrete growth assumption instead of asking the model to predict the future unprompted. Give it the inputs, and let it do the arithmetic you’d otherwise do by hand.

Templates worth testing against your own metrics:

• “Given current CPU and memory utilization across this node pool over the last 30 days, and assuming 20% month-over-month traffic growth, when do we hit 80% utilization?”
• “Which workloads in this namespace are over-provisioned relative to their actual usage, and by how much?”
• “Propose a right-sized resource request and limit for this deployment based on its p95 usage over the last two weeks, and explain your reasoning.”
• “If we migrated these three services to a shared node pool, what’s the expected utilization, and what’s the risk of noisy-neighbor contention?”

These prompts work because they ask for a specific, checkable number rather than a vague recommendation. If the model’s arithmetic doesn’t match your own spreadsheet, that’s a signal to recheck its assumptions before trusting the recommendation. Always ask it to show the formula, not just the answer, so you can verify the math in thirty seconds rather than trusting a number that sounds plausible.

Prompts for automated alert enrichment and contextualization

Raw alerts tell you something crossed a threshold. Enriched alerts tell you what that threshold crossing means for the system around it, which is the difference between a two-minute triage and a twenty-minute one spent gathering context by hand.

• “Enrich this alert with the deployment history for the affected service over the last 24 hours.”
• “Add the current error budget status and recent related alerts to this notification before it’s sent.”
• “Given this alert, pull the three most relevant dashboards and summarize what each one shows right now.”
• “Correlate this alert with any other alerts firing in the same namespace in the last 15 minutes, and note if they share a root cause.”

The value here is compounding: an alert enriched with deployment history, budget status, and correlated signals turns a page that used to require five separate dashboard checks into a single message a responder can act on immediately. Our piece on reducing alert fatigue with the USE and RED methods covers the metric selection that makes enrichment prompts like these useful instead of noisy, since enrichment is only as good as the underlying SLIs it pulls from.

Integrating AI prompts with your existing DevOps tools and CI/CD pipelines

A prompt library that only works in a chat window is a toy. The real value shows up when the same patterns plug into your CI/CD pipeline, your ticketing system, and your existing observability stack without requiring a separate workflow.

Patterns worth building toward:

• “Review this pull request’s diff against our Terraform state and flag any change that would modify a production security group.”
• “Given this failed CI/CD pipeline run, pull the relevant logs and classify the failure as test flake, infrastructure issue, or code defect.”
• “Generate a rollback plan for this deployment that references our existing runbook format, and open it as a draft ticket.”
• “Check this manifest against our validation rules before it merges, and flag anything that would fail in production.”

The integration work matters more than the prompt wording here. An agent that can read your GitLab pipeline logs, open a draft PR, and attach context to a ticket is doing real work, while the same prompt run in isolation just produces a paragraph nobody acts on. Our AI tools for SRE teams guide covers which copilots and agents plug into common DevOps toolchains without a custom integration layer.

Tuning prompts for your specific infrastructure environment

A prompt written for a generic Kubernetes cluster will underperform on your actual environment until you tune it with the specifics that make your setup different: naming conventions, namespace structure, and the quirks of your particular stack.

A few tuning moves that consistently help:

• Swap generic resource names for your actual naming convention, so the agent references payments-prod-eastus instead of a placeholder.
• Add your organization’s severity definitions directly into the prompt instead of letting the model guess what “critical” means to your team.
• Include a short glossary of internal shorthand (service names, team abbreviations) the model otherwise has no way to know.
• Adjust output format to match what your team actually reads, whether that’s a Slack-ready summary or a markdown table for a wiki page.

Environments built on OpenStack and Kolla-Ansible need extra tuning because the terminology and failure modes differ sharply from a managed Kubernetes service. A prompt that works well against AKS often needs rewriting before it produces anything useful against a Kolla-Ansible deployment, since the resource model, networking layer, and common failure signatures aren’t the same. Start with the general template, run it against a known-good and a known-bad case in your own environment, and adjust the phrasing until the output matches what an experienced operator would actually say.

Prompts for security incident detection and response

Security prompts need a tighter approval loop than general operational prompts, because the cost of a false negative (a missed compromise) and a false positive (an unnecessary lockout) are both high. Treat every security prompt as analysis-only by default.

• “Review the last 24 hours of authentication logs for this service and flag any pattern that looks like credential stuffing or unusual geographic access.”
• “Given this suspicious process running on a production host, what evidence would confirm or rule out a compromise, without taking any containment action yet?”
• “Draft a containment plan for this suspected incident, listing each step and which ones require security team approval before execution.”
• “Check this CI/CD pipeline configuration for exposed secrets or overly broad permissions.”

Our audit and logging policy design prompt and CI/CD secret exposure review prompt are built around this same analysis-first structure. For teams that want frameworks mapped to established standards, the incident response playbooks mapped to NIST, CISA, and AWS are a useful third-party reference for aligning your prompt-driven containment steps with recognized security frameworks.

Why prompts deserve the same rigor as production code

Most teams treat prompts as disposable, something you type once and forget. I think that’s backwards. A prompt that catches an OOM kill correctly today should catch it correctly next month, and the only way to guarantee that is to version it, review changes to it, and test it against past incidents the way you’d test a regression suite.

The payoff isn’t abstract: teams that maintain a reviewed prompt library handle incidents more consistently, repeat fewer old mistakes, and free up time for project work instead of re-deriving the same diagnostic steps every time someone new joins on-call. Start with two or three prompts you use constantly, put them under version control, and let the library grow from there.

— James

Try our battle-tested prompts and incident response tools

We built our prompt libraries, validators, and audit services around the exact patterns covered above: analysis-first prompts, explicit approval gates, and templates reviewed by a senior systems engineer against real production failures.

Our incident response AI prompts give you over a hundred free, copy-paste prompts for triage, root cause analysis, and post-incident reviews, ready to adapt to your own stack. If you’d rather have us look directly at your environment, our Kubernetes Health Check runs for $300 one-off, and our AI Incident Response service helps teams wire these prompts into a real approval workflow. Pro plans for ongoing access to our prompt packs run a monthly or yearly subscription plan on our pricing page.

FAQ

Will AI replace the SRE role?

No single source supports that claim, and the practical pattern across current guidance is the opposite: agents handle discovery and diagnostics faster, but human approval remains required for any state-changing action. SRE work shifts toward reviewing, approving, and maintaining these prompt-driven workflows rather than disappearing.

What does SRE stand for?

SRE stands for Site Reliability Engineering, a discipline that applies software engineering practices to operations and reliability work. It originated at Google and is documented extensively in the Google SRE Book.

What are the main pillars of SRE monitoring?

Definitions vary across teams, but a commonly cited version covers latency, traffic, errors, and saturation as the core signals worth monitoring. These map closely to the SLI categories Google’s SRE Workbook uses when defining service-level objectives.

Is working as an SRE a difficult job?

The role is demanding because it combines software engineering, systems troubleshooting, and on-call incident response, often under time pressure. Teams that maintain structured prompts, living incident documents, and clear role assignments tend to reduce that pressure by making incident response more repeatable.

How much do prompt packs and audits cost?

Our Pro plan for ongoing prompt pack access costs a monthly or yearly subscription plan, and a Kubernetes Health Check audit runs $300 one-off through our consulting page. Other services, including our OpenStack and Kolla-Ansible review, are priced individually on the same page.

Sources

• Starter prompts for Azure SRE Agent | Microsoft Learn
• Continuous improvement to get reliability (SRE Workbook)
• Stratus paper (agentic cloud reliability)

Recommended

• SLOs and Error Budgets With Prometheus, the Practical Way
• Reconciliation Loops for Self-Correcting Systems: Power
• Stop Alert Noise with 4 Prometheus Prompts for Production
• Turning Plain-English SLO Requirements Into PromQL With AI