HCP Terraform Agent Pool Design Prompt

You are a platform engineer who runs HCP Terraform (Terraform Cloud) agent pools in production and knows how token rotation and pool deletion break running plans.

I will provide:
- Why remote agents are needed (private VPC endpoints, on-prem targets, air-gapped state)
- Org name, which workspaces need private access, and how they run today
- Where agents will run (EC2 ASG, Kubernetes, on-prem VMs)

Your job:

1. **Pool topology** — decide how many `tfe_agent_pool`s to create. Recommend scoping by blast radius / network segment (e.g. one pool per VPC or per environment) rather than a single global pool. Explain org-scoped vs project-scoped pools.

2. **Allowed-workspaces scoping** — use `tfe_agent_pool_allowed_workspaces` to restrict which workspaces may target a pool, so a dev workspace can't run agents that sit in the prod VPC. Show the resource and the `organization_scoped = false` pairing.

3. **Agent tokens** — create `tfe_agent_token` per pool, and describe secure delivery to hosts (never commit; inject via secrets manager / instance profile). Note the token is shown ONCE at creation.

4. **Self-hosted agent deployment** — how to run the `hashicorp/tfc-agent` container/binary with `TFC_AGENT_TOKEN` and `TFC_AGENT_NAME`, as an EC2 ASG, a Kubernetes Deployment, or systemd. Cover graceful drain so an agent finishes its current run before terminating.

4b. **Workspace execution mode** — set the workspace to `execution_mode = "agent"` and bind `agent_pool_id` via `tfe_workspace_settings` (execution mode moved off the `tfe_workspace` resource). Show it.

5. **Scaling** — agents are single-run at a time; size the pool to peak concurrent runs. Describe autoscaling on queue depth and idle scale-in with drain.

6. **Host hardening** — agents hold cloud credentials and run arbitrary Terraform: minimal base image, no inbound ports (agents dial out to HCP over 443), least-privilege instance role, egress allowlist, patch cadence, and isolating pools by trust level.

7. **DESTRUCTIVE operations** — call out clearly: (a) revoking/rotating a `tfe_agent_token` immediately kills agents using it and FAILS any in-flight plan/apply on those agents — drain first, rotate with overlap; (b) deleting a `tfe_agent_pool` that workspaces still reference breaks their runs — repoint workspaces before destroy.

Output: (a) pool topology, (b) the `tfe` HCL (pool, allowed-workspaces, token, workspace settings), (c) the agent deployment manifest, (d) scaling + drain plan, (e) hardening checklist, (f) a safe token-rotation runbook.

Bias toward: least-privilege pools scoped to workspaces, drain-before-terminate, and overlapping token rotation.

---
Input template:
```
Why agents (private endpoints / on-prem / air-gap):
Org + workspaces needing private access:
Agent runtime (EC2 ASG / K8s / on-prem VM):
Peak concurrent runs:
Network egress constraints:
```

Why this prompt works

HCP Terraform agent pools look simple until a token rotation kills every in-flight apply or a terraform destroy on a pool takes out three workspaces that still pointed at it. This prompt front-loads the two destructive failure modes — token revocation and pool deletion — and forces a drain-before-terminate discipline. It also pins down the details people get wrong: execution mode now lives on tfe_workspace_settings, tokens are shown exactly once, and agents run single-run-at-a-time so the pool must be sized to peak concurrency.

How to use it

1. Explain why you need agents (private VPC endpoints are the usual answer) and which workspaces need them.
2. Run the prompt; apply the tfe HCL to create the pool, scope it to workspaces, and mint a token.
3. Deploy the tfc-agent runtime and confirm the agent registers as idle in the pool.
4. Flip the target workspaces to execution_mode = "agent" and run a plan to verify connectivity.
5. Before any token rotation or pool teardown, follow the drain runbook — never rotate under a running apply.

Useful commands

# Run a self-hosted agent (container) pointed at your pool token
docker run -d --name tfc-agent \
  -e TFC_AGENT_TOKEN="$TFC_AGENT_TOKEN" \
  -e TFC_AGENT_NAME="agent-vpc-prod-1" \
  hashicorp/tfc-agent:latest

# Gracefully drain: agent finishes its current run, then exits (no new runs)
docker kill --signal=INT tfc-agent

# List agents in a pool via the HCP Terraform API
curl -s \
  --header "Authorization: Bearer $TFE_TOKEN" \
  "https://app.terraform.io/api/v2/agent-pools/${POOL_ID}/agents" | jq '.data[].attributes'

# Apply the agent-pool infra itself
terraform init && terraform apply

Patterns

The tfe provider HCL — a pool scoped to specific workspaces, a token, and the workspace bound to agent execution:

resource "tfe_agent_pool" "vpc_prod" {
  name                = "vpc-prod"
  organization        = var.tfe_org
  organization_scoped = false          # restrict to allowed workspaces only
}

resource "tfe_agent_pool_allowed_workspaces" "vpc_prod" {
  agent_pool_id         = tfe_agent_pool.vpc_prod.id
  allowed_workspace_ids = [tfe_workspace.prod_network.id]
}

resource "tfe_agent_token" "vpc_prod" {
  agent_pool_id = tfe_agent_pool.vpc_prod.id
  description   = "vpc-prod pool token (rotate with overlap)"
}

# Execution mode lives on tfe_workspace_settings, not tfe_workspace.
resource "tfe_workspace_settings" "prod_network" {
  workspace_id   = tfe_workspace.prod_network.id
  execution_mode = "agent"
  agent_pool_id  = tfe_agent_pool.vpc_prod.id
}

output "agent_token" {
  value     = tfe_agent_token.vpc_prod.token
  sensitive = true   # shown once — deliver to hosts via a secrets manager
}

A Kubernetes Deployment running the agents, with a preStop drain hook so a terminating pod finishes its current run:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tfc-agent-vpc-prod
spec:
  replicas: 3            # size to peak concurrent runs; one run per agent
  selector:
    matchLabels: { app: tfc-agent-vpc-prod }
  template:
    metadata:
      labels: { app: tfc-agent-vpc-prod }
    spec:
      terminationGracePeriodSeconds: 3600   # allow a long apply to finish
      containers:
        - name: tfc-agent
          image: hashicorp/tfc-agent:latest
          env:
            - name: TFC_AGENT_TOKEN
              valueFrom:
                secretKeyRef: { name: tfc-agent-vpc-prod, key: token }
            - name: TFC_AGENT_NAME
              valueFrom:
                fieldRef: { fieldPath: metadata.name }
          lifecycle:
            preStop:
              exec:
                # SIGINT = graceful drain: no new runs, finish the current one
                command: ["/bin/sh", "-c", "kill -INT 1"]