Terraform Datadog Monitors as Code Prompt

You are a senior SRE building a reusable Terraform module to manage Datadog monitors, dashboards, and SLOs as code with the `DataDog/datadog` provider (>= 3.40). The goal is a single typed `monitors` map driving `for_each`, with safe rollout that never floods on-call.

Work through these steps in order:

1. Configure the provider with `api_key` and `app_key` sourced from variables (never inline), plus `api_url` for EU/US region correctness.

2. Model a typed variable `monitors` as `map(object({...}))` carrying `type`, `query`, `message`, `thresholds`, `tags`, `notify_handles`, `priority`, and `renotify_interval`. Iterate with `for_each = var.monitors` so each monitor has a stable address keyed by its map key, NOT by list index.

3. Template the notification handles into `message` using `join`/`format` so `@slack-`, `@pagerduty-`, and `@team-` handles are consistent. Enforce a tag taxonomy (`service`, `env`, `team`, `managed_by:terraform`).

4. Prevent alert storms on `apply`: set `no_data_timeframe` sanely, `notify_no_data = false` for volatile metrics, and warn that recreating a monitor (changing its key, or changing an immutable field) DESTROYS and recreates it — which drops mute state, existing downtimes, and can fire a burst of notifications the instant it is recreated.

5. Recommend wrapping risky rollouts with a Datadog downtime (`datadog_downtime_schedule`) or `@notify` suppression during the first apply, and using `terraform state mv` when renaming keys to avoid recreation.

6. Add `datadog_service_level_objective` tied to the monitors and a `datadog_dashboard` summarizing them.

7. Explicitly enumerate DESTRUCTIVE operations: changing a monitor's `for_each` key, deleting a monitor referenced by an SLO, and mass `-target`-less applies that recreate many monitors at once.

Input template:
```
Datadog region: <US1 | EU1 | ...>
Services to cover: <svc names>
Alert routing: <slack/pagerduty/team handles>
SLO targets: <99.9 over 30d, ...>
Existing monitors to import: <ids or none>
```

Why this prompt works

Datadog monitors carry runtime state Terraform does not manage: mute status, active downtimes, and alert history. The naive mistake is treating monitors as disposable resources and letting for_each keys drift, which silently destroys and recreates them, dropping that runtime state and firing a burst of pages. This prompt centers stable keying, a typed monitors map, and downtime-gated rollout so codifying alerts does not become an incident of its own.

How to use it

Provide your Datadog region, the services in scope, routing handles, and SLO targets. Ask the model to emit the typed variable "monitors", the for_each resource, the SLO, and a dashboard. Before the first apply, schedule a downtime window. Use terraform plan to confirm zero unexpected -/+ (destroy-then-create) lines.

Useful commands

# Region-correct provider auth
export DD_API_KEY=... DD_APP_KEY=...
export DD_SITE=datadoghq.eu   # or datadoghq.com

# Plan and grep for destructive recreation of monitors
terraform plan -no-color | grep -E 'must be replaced|# .*will be destroyed'

# Import an existing monitor instead of recreating it
terraform import 'datadog_monitor.this["api-latency-p99"]' 1234567

# Rename a monitor key WITHOUT destroy/recreate
terraform state mv \
  'datadog_monitor.this["old-key"]' \
  'datadog_monitor.this["new-key"]'

# Schedule a downtime before a risky bulk apply (Datadog CLI/API)
curl -X POST "https://api.${DD_SITE}/api/v2/downtime" \
  -H "DD-API-KEY: $DD_API_KEY" -H "DD-APPLICATION-KEY: $DD_APP_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"data":{"type":"downtime","attributes":{"scope":"managed_by:terraform","schedule":{"start":null}}}}'

Patterns

Typed monitors map driven by for_each:

variable "monitors" {
  type = map(object({
    type              = string
    query             = string
    message           = string
    priority          = number
    renotify_interval = optional(number, 0)
    notify_handles    = list(string)
    tags              = list(string)
  }))
}

resource "datadog_monitor" "this" {
  for_each = var.monitors

  name    = each.key
  type    = each.value.type
  query   = each.value.query
  message = "${each.value.message}\n\n${join(" ", each.value.notify_handles)}"

  priority          = each.value.priority
  renotify_interval = each.value.renotify_interval
  notify_no_data    = false
  no_data_timeframe = 20

  tags = concat(each.value.tags, ["managed_by:terraform"])
}

SLO wired to the managed monitors plus a summary dashboard widget:

resource "datadog_service_level_objective" "api" {
  name        = "API availability"
  type        = "monitor"
  monitor_ids = [datadog_monitor.this["api-5xx-rate"].id]

  thresholds {
    timeframe = "30d"
    target    = 99.9
    warning   = 99.95
  }
  tags = ["service:api", "managed_by:terraform"]
}

resource "datadog_dashboard" "overview" {
  title       = "Service Overview (managed by terraform)"
  layout_type = "ordered"

  widget {
    slo_definition {
      slo_id      = datadog_service_level_objective.api.id
      view_type   = "detail"
      time_windows = ["30d"]
      view_mode   = "overall"
    }
  }
}