> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bbrands.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Infrastructure as Code (Terraform)

> Every Better Stack resource is defined as code under infra/betterstack. Understand the layout, per-environment state, and the exact meaning of plan, apply and import.

<Info>
  All Better Stack configuration — status pages, monitors, heartbeats, log
  labels, dashboards and alerts — is declared as code under
  `infra/betterstack/`. The UI is never the source of truth; Terraform is.
</Info>

## Why infrastructure as code

Configuring dozens of monitors, sections and alerts by hand does not scale and
drifts silently between environments. Terraform gives us:

| Manual UI                       | Terraform                                  |
| ------------------------------- | ------------------------------------------ |
| Dozens of objects created once  | `terraform apply` per environment          |
| Drift between dev / cert / prod | Same modules, different `*.tfvars`         |
| No review trail                 | PR-reviewed `.tf` files                    |
| "Who changed this?"             | Git history + state, reproducible anywhere |

## How Terraform thinks

Terraform continuously reconciles **three** pictures of the world. Understanding
this is the key to everything else on this page:

```mermaid theme={null}
flowchart LR
    code["Declared config<br/>.tf + adopted/*.json"]
    state["Tracked state<br/>terraform.tfstate"]
    real["Real world<br/>Better Stack API"]

    code -->|"desired"| engine((Terraform))
    state -->|"known"| engine
    real -->|"actual"| engine
    engine -->|"reconcile"| result["plan / apply"]
```

* **Declared config** — what the `.tf` files and adoption manifests say should
  exist.
* **Tracked state** (`terraform.tfstate`) — Terraform's inventory of what it
  already manages, including each resource's Better Stack id.
* **Real world** — what the Better Stack API returns right now.

## `plan` vs `apply` — the core distinction

This is the concept every operator must internalize. The simplest analogy:
**`plan` is the diff, `apply` is the commit.**

<CardGroup cols={2}>
  <Card title="terraform plan — dry run, read-only" icon="magnifying-glass">
    Refreshes state from the Better Stack API, compares it against the declared
    config, and prints the actions it *would* take: `+ create`, `~ update`,
    `- destroy`. It changes **nothing**. A plan showing
    `0 to add, 0 to change, 0 to destroy` proves there is no drift.
  </Card>

  <Card title="terraform apply — executes the change" icon="play">
    Takes that same diff and runs the real API calls (POST / PATCH / DELETE),
    then writes the resulting ids back into the tfstate. This is the **only**
    operation that mutates infrastructure.
  </Card>
</CardGroup>

<Note>
  You never `apply` without first reading the `plan`. In CI the plan is printed
  in the job log and `apply` additionally requires typing `APPLY` — see
  [CI/CD for Monitoring](/monitoring/github-actions).
</Note>

### A third verb: `import` (adoption)

When a resource already exists in Better Stack (created by hand before Terraform
managed it), `import` attaches it to the state **without recreating it**. This
is how the existing monitors were adopted: the first `apply` in development
imported **68** resources, and the follow-up `plan` reported **0 changes**.

### Idempotency — why re-running is safe

Terraform is declarative: applying the same config twice does not duplicate
anything, as long as the state is preserved.

| Run                              | What happens                                            |
| -------------------------------- | ------------------------------------------------------- |
| First `apply`                    | Creates resources; stores their ids in the tfstate      |
| Second `apply` (no changes)      | `0 to add, 0 to change, 0 to destroy` — nothing dupes   |
| `apply` after editing a `.tf`    | Updates only the changed attributes in place            |
| `apply` after adding one monitor | Creates just that one; existing resources are untouched |

<Warning>
  Duplicates only happen if the state is **lost or bypassed** — for example,
  applying against empty local state while the objects already exist in the UI,
  or deleting `terraform.tfstate`. This is exactly why CI refuses to run without
  a remote backend, and why existing resources are imported before the first
  apply.
</Warning>

## Repository layout

```text theme={null}
infra/betterstack/
  main.tf, providers.tf, variables.tf, outputs.tf, locals.tf, versions.tf
  environments/
    development.tfvars      # feature flags per environment (no secrets)
    certification.tfvars
    production.tfvars
  adopted/
    development.json        # desired state (ids + attributes)
    development.imports.tf  # one-time import blocks
    certification.json
    production.json
  modules/
    uptime-environment/     # status page, sections, monitors, heartbeats
    notification-policies/  # Slack escalation policies + Slack-only urgencies
    api-log-alerts/         # log labels, duration_ms metric, dashboard, charts, alerts
    idp-oauth-alerts/       # IdP JWKS monitor + OAuth log alerts
    infrastructure-charts/  # PostgreSQL/Redis charts (metrics-push cron)
    platform-log-alerts/    # platform-wide log alerts
  backend_override.tf.example  # template for the remote state backend
```

| Module                  | Manages                                                                                | Enabled by                                           |
| ----------------------- | -------------------------------------------------------------------------------------- | ---------------------------------------------------- |
| `uptime-environment`    | Status page, sections, 31 monitors, 3 heartbeats                                       | `enable_uptime_monitors = true`                      |
| `notification-policies` | Slack escalation policies + Slack-only urgencies (no email/SMS)                        | `slack_integration_ids` non-empty                    |
| `api-log-alerts`        | Log labels, `duration_ms` metric, dashboard, 5xx + latency charts, per-category alerts | `enable_api_log_alerts` + `logtail_enabled` + source |
| `idp-oauth-alerts`      | IdP JWKS monitor + OAuth error log alerts                                              | `enable_idp_oauth_alerts` / `idp_issuer_url`         |
| `infrastructure-charts` | PostgreSQL/Redis dashboard charts                                                      | `enable_infrastructure_charts` + `logtail_enabled`   |
| `platform-log-alerts`   | Platform-wide log alerts                                                               | `enable_platform_log_alerts` + `logtail_enabled`     |

### Providers

| Provider  | Package                       | Manages                                       |
| --------- | ----------------------------- | --------------------------------------------- |
| Uptime    | `BetterStackHQ/better-uptime` | Monitors, heartbeats, status pages, resources |
| Telemetry | `BetterStackHQ/logtail`       | Sources, metrics/labels, dashboards, alerts   |

## Adoption manifests

Every environment converges to the same catalogue — **31 monitors** (12 API,
including the cross-folder Digital Contract journey, + 9 integrations +
10 platforms), **4 sections** and **3 heartbeats** — each
pointing at its own hosts. The `adopted/<env>.json` manifest is the desired
state; after adoption you **edit it via PR**. The generator synthesizes entries
that do not exist yet (with `monitor_id = null`) so Terraform **creates** them on
apply instead of importing.

| Environment   | Status page id | Imported (already exist)                 | Created on apply                            |
| ------------- | -------------- | ---------------------------------------- | ------------------------------------------- |
| development   | `252747`       | all 30 + 4 + 3                           | —                                           |
| certification | `252750`       | 10 platforms + 3 sections + 3 heartbeats | 20 API/integration monitors + Infra section |
| production    | `251528`       | 10 platforms + 3 sections + 3 heartbeats | 20 API/integration monitors + Infra section |

Regenerate only to audit drift, and always review the diff before committing:

```bash theme={null}
node scripts/status-monitoring/export-betterstack-inventory.mjs development
node scripts/status-monitoring/generate-adoption-manifest.mjs --all
git diff infra/betterstack/adopted/
```

## Per-environment state (workspaces)

State is isolated per environment using Terraform **workspaces**: the workspace
name equals the environment name, so development, certification and production
each keep an independent state under the `env:/` prefix of the backend.

<Warning>
  Local state is gitignored and embeds the `x-health-token` value. Losing it
  causes duplicate creates on the next apply. Before any team or CI use, migrate
  to a shared **S3-compatible remote backend**.
</Warning>

```bash theme={null}
cd infra/betterstack
cp backend_override.tf.example backend_override.tf   # fill in bucket/endpoint
terraform init -migrate-state
```

The same block is stored in the `TF_BACKEND_OVERRIDE` GitHub secret so the
workflow writes it before `terraform init`.

## Running it locally

`terraform-apply.sh` wraps `init` + workspace selection + `plan`/`apply`, and
loads tokens automatically from `backend/horizon-api/.env.<env>` (mapping
`BETTERSTACK_UPTIME_API_TOKEN` to the provider's `BETTERUPTIME_API_TOKEN`).

```bash theme={null}
# Dry run, then apply, then confirm no drift
./scripts/status-monitoring/terraform-apply.sh plan development
./scripts/status-monitoring/terraform-apply.sh apply development
./scripts/status-monitoring/terraform-apply.sh plan development   # expect: No changes

# Shortcuts
pnpm tf:betterstack:plan
pnpm tf:betterstack:apply
```

<Note>
  `destroy` is deliberately guarded: it requires `CONFIRM_DESTROY=<env>` so it
  can never run by accident.
</Note>

## Outputs

After `apply`, Terraform exposes:

* `status_page_resources_json` — the component-scope → status-page-resource id
  map used to fill `BETTERSTACK_STATUS_PAGE_RESOURCES`.
* `uptime_heartbeat_ids` — the three cron heartbeat ids.

## Related source

* IaC deep dive: `docs/initiatives/betterstack/06-terraform-iac.md`
* Module README: `infra/betterstack/README.md`
* Runner script: `scripts/status-monitoring/terraform-apply.sh`