> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bbrands.io/llms.txt
> Use this file to discover all available pages before exploring further.

# CI/CD for Monitoring

> The Better Stack Terraform workflow: an automatic validate lane on PRs and a manual, gated plan/apply lane per environment — with the safety decisions behind each guardrail.

<Info>
  Monitoring changes ship through the **`Better Stack Terraform`** GitHub Actions
  workflow (`.github/workflows/betterstack-terraform.yml`). It has two clearly
  separated lanes: a static **validate** lane that runs automatically on pull
  requests, and a **plan/apply** lane that only runs when a human dispatches it.
</Info>

## Two lanes at a glance

```mermaid theme={null}
flowchart TD
    subgraph pr [Pull request]
        A["PR touches infra/betterstack/**"] --> V["validate job<br/>fmt · init -backend=false · validate · JSON check"]
        V --> G1{Green?}
        G1 -->|yes| M["Safe to merge"]
    end
    subgraph dispatch [Manual dispatch]
        D["workflow_dispatch<br/>environment + action"] --> GATE["gate job<br/>requires APPLY for apply"]
        GATE --> TF["terraform job<br/>in GitHub Environment"]
        TF --> OUT["Outputs summary (apply)"]
    end
```

## Validate lane — automatic, safe by construction

Triggered on every pull request to `master`, `release` or `develop` that touches
`infra/betterstack/**`, the runner script, or the workflow file itself. It runs
with **no secrets and no state**, so it physically cannot change Better Stack.

| Step                 | Command                            | Purpose                        |
| -------------------- | ---------------------------------- | ------------------------------ |
| Format check         | `terraform fmt -check -recursive`  | Enforce canonical formatting   |
| Init without backend | `terraform init -backend=false`    | Resolve modules/providers only |
| Validate             | `terraform validate`               | Type-check the configuration   |
| Manifest JSON check  | `JSON.parse` each `adopted/*.json` | Catch malformed adoption files |

This is the lane you see turn green on a monitoring PR. It proves the config is
well-formed; it does **not** talk to Better Stack.

## Plan/apply lane — manual, gated, per environment

This lane runs via **`workflow_dispatch`** (Actions tab → Run workflow) or as
the final stage of the **Release Train** orchestrator
(`.github/workflows/release-train.yml`). It takes two inputs — the target
`environment` (a fixed choice list: `development`, `certification`,
`production`) and the `action` (`plan` or `apply`) — plus a confirmation
string.

Dispatch runs are **tag-only**: pick a CalVer release tag in "Use workflow
from". The shared guard (`.github/scripts/validate-release-tag.sh`) rejects
branch refs and enforces the channel matrix (alpha → development, rc →
development + certification, stable → all three), and requires the tag to have
a GitHub Release.

```mermaid theme={null}
sequenceDiagram
    participant U as Operator
    participant GH as GitHub Actions
    participant Env as GitHub Environment
    participant BS as Better Stack

    U->>GH: Run workflow (env, action, confirm_apply)
    GH->>GH: gate — action=apply requires "APPLY"
    GH->>Env: terraform job (env secrets + protection rules)
    Env->>Env: require TF_BACKEND_OVERRIDE (else fail fast)
    Env->>Env: write backend_override.tf, terraform init
    Env->>BS: terraform plan / apply
    BS-->>Env: diff / applied ids
    Env->>GH: outputs summary (apply)
```

### The confirmation gate

The `gate` job runs first. If `action = apply` and the `confirm_apply` input is
not exactly `APPLY`, it fails before anything touches infrastructure. `plan`
needs no confirmation — it is read-only. When the workflow is called by the
Release Train, the train's single `RELEASE` confirmation replaces the typed
`APPLY` (via a `workflow_call`-only `confirmed` input that manual dispatches
cannot set). The gate then validates the release tag with the shared guard.

### Environment isolation and secrets

The `terraform` job runs **inside the selected GitHub Environment**, so that
environment's protection rules (required reviewers, wait timers) and its scoped
secrets apply automatically.

<CardGroup cols={2}>
  <Card title="Per-environment secrets" icon="key">
    `BETTERSTACK_UPTIME_API_TOKEN`, `HEALTH_MONITOR_TOKEN`, and optional
    `LOGTAIL_API_TOKEN`. Defined on each of `development`, `certification` and
    `production`.
  </Card>

  <Card title="Repository secrets (remote state)" icon="database">
    `TF_BACKEND_OVERRIDE` (the full `backend "s3"` block), `TF_STATE_ACCESS_KEY`
    and `TF_STATE_SECRET_KEY`. Shared across environments; state stays isolated
    by workspace.
  </Card>
</CardGroup>

### Fail-fast on missing remote state

Before `init`, the job checks that `TF_BACKEND_OVERRIDE` exists. If it does not,
it errors out immediately — CI must never plan against an empty local state,
which would try to recreate every resource. When present, the block is written
to `backend_override.tf` and `terraform init` picks up the remote backend.

### Concurrency lock

A concurrency group keyed by environment
(`betterstack-terraform-<environment>`) with `cancel-in-progress: false`
guarantees two operations never run against the same environment at once; a
second run queues behind the first.

### Outputs

On `apply`, the final step appends `terraform output` to the job summary so the
resulting `status_page_resources_json` and `uptime_heartbeat_ids` are visible
without re-running anything.

## How to run it

<Steps>
  <Step title="Open the workflow">
    GitHub → **Actions** → **Better Stack Terraform** → **Run workflow**, and
    pick the release **tag** in "Use workflow from" (branch refs are rejected).
  </Step>

  <Step title="Run a plan first">
    Pick the target `environment`, set `action = plan`, and run. Read the diff
    in the job log. A `0 to add, 0 to change, 0 to destroy` plan means no drift.
  </Step>

  <Step title="Apply when the plan is what you expect">
    Run again with `action = apply` and type `APPLY` in `confirm_apply`. The gate
    passes, the job runs inside the environment, and the outputs summary is
    posted.
  </Step>

  <Step title="Confirm convergence">
    Run one more `plan` — it should report no changes.
  </Step>
</Steps>

## Safety decisions

<AccordionGroup>
  <Accordion title="Why apply is never automatic">
    Monitoring changes affect the public status page and can page on-call
    engineers. A human must read the plan and explicitly type `APPLY`, so an
    infrastructure change is always a deliberate, reviewed action — never a side
    effect of merging.
  </Accordion>

  <Accordion title="Why CI refuses to run without remote state">
    Terraform tracks resource ids in its state. Running against empty local
    state would make Terraform believe nothing exists and recreate every
    monitor, duplicating resources on the status page. Requiring
    `TF_BACKEND_OVERRIDE` makes that failure impossible.
  </Accordion>

  <Accordion title="Why one concurrency group per environment">
    Two overlapping applies against the same state can corrupt it or race on the
    Better Stack API. The per-environment lock serializes operations while still
    allowing different environments to run in parallel.
  </Accordion>

  <Accordion title="Why the validate lane has no secrets">
    PR validation should be safe to run on any change, including from forks. By
    using `init -backend=false` and no tokens, the lane can check correctness
    without any ability to reach or mutate Better Stack.
  </Accordion>
</AccordionGroup>

## Related source

* Workflow: `.github/workflows/betterstack-terraform.yml`
* Runner: `scripts/status-monitoring/terraform-apply.sh`
* Backend template: `infra/betterstack/backend_override.tf.example`