Automation Observability and Metrics Design Prompt

You are a senior automation/platform engineer who has been burned by automation that failed silently for weeks before anyone noticed. Design an observability layer so every automated workflow is measurable, debuggable, and trustworthy.

I will provide:
- The automated workflows we run (triggers, actions, frequency)
- Our current telemetry stack (metrics, logs, tracing, eventing tools)
- What "good" looks like per workflow (success criteria, expected duration)
- Past incidents where automation failed undetected

Your job:

1. **Instrumentation per workflow** — define the events each workflow must emit: start, decision points, action attempts, success, failure, skip, and back-out, each with structured context (trigger source, target, dry-run flag, correlation ID).
2. **Core metrics** — specify the golden signals for automation: invocation rate, success/failure rate, action duration, skip rate, manual-override rate, and time-to-detect-failure.
3. **SLOs and burn alerts** — set per-workflow SLOs (e.g. success rate, latency) and define alerts on SLO burn plus on the silent-failure case (no invocations when some were expected).
4. **Dashboards** — describe the panels an on-call engineer needs to answer "is automation healthy, and if not, which workflow and why" in under a minute.
5. **Audit and traceability** — ensure every automated action is traceable end-to-end (who/what triggered it, what changed, the back-out path) via correlation IDs across logs/metrics/traces.
6. **Failure-mode coverage** — map each past silent-failure incident to a signal that would now catch it.

Output as: (a) the per-workflow instrumentation schema, (b) the metrics catalog with types and labels, (c) the SLO + alert-rule table, (d) the dashboard layout, (e) an incident-to-signal coverage matrix.

Default to over-instrumenting detection of failure: an automated action you cannot observe, trace, or alert on must be treated as not safe to run unattended, and every workflow must have an explicit back-out path that is itself logged and gated by approval where blast radius warrants.