> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bbrands.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Operations Runbook

> Day-2 monitoring tasks with the exact commands: sync environment variables, add or change a monitor, rotate the health token, regenerate status-page resources, and resolve drift.

<Info>
  This runbook covers the recurring operational tasks for the monitoring
  platform. Each task pairs the **functional trigger** ("when do I need this")
  with the **exact commands** to run. No task here ever prints or stores secret
  values.
</Info>

## Sync environment variables

The environment variables live in **1Password Environments** (one per
environment) and flow to two destinations: local `.env.<env>` files for
development, and Vercel for deployed apps.

<CardGroup cols={2}>
  <Card title="1Password → local files" icon="laptop-code">
    Run `pnpm env:sync` to mirror every 1Password Environment into each app's
    `.env.<environment>` file. With no arguments it syncs **all** apps and **all**
    environments; environments your token cannot access are skipped with a
    warning instead of failing.
  </Card>

  <Card title="1Password → Vercel" icon="cloud-arrow-up">
    Run the **Sync Env** workflow (`workflow_dispatch`) to push a 1Password
    Environment into Vercel, then **redeploy** `horizon-api` so the new values
    take effect.
  </Card>
</CardGroup>

```bash theme={null}
# Everything (all apps, all accessible environments)
pnpm env:sync

# Only the API app
pnpm env:sync api

# A single environment
pnpm env:sync --env certification
```

<Warning>
  Editing a variable in 1Password is not enough for deployed environments: after
  the Sync Env workflow updates Vercel, you must **redeploy** `horizon-api` for
  the runtime to read the new values.
</Warning>

## Add or change a monitor

Monitoring is code — never click-ops in the Better Stack UI.

<Steps>
  <Step title="Edit the definition">
    Change the module under `infra/betterstack/modules/` or the environment's
    `adopted/<env>.json` desired-state manifest.
  </Step>

  <Step title="Open a PR">
    The **validate** lane runs automatically (`fmt`, `validate`, JSON checks).
    See [CI/CD for Monitoring](/monitoring/github-actions).
  </Step>

  <Step title="Plan">
    Dispatch the workflow with `action = plan` for the target environment and
    read the diff. Locally: `pnpm tf:betterstack:plan`.
  </Step>

  <Step title="Apply">
    Dispatch with `action = apply` and type `APPLY`, or run
    `./scripts/status-monitoring/terraform-apply.sh apply <env>` locally. Confirm
    a follow-up `plan` shows no changes.
  </Step>
</Steps>

## Rotate `HEALTH_MONITOR_TOKEN`

The health token authenticates Better Stack's monitors against the component
endpoints. Rotate it if it may have leaked, or on a routine schedule. Order
matters to avoid a monitoring gap.

<Steps>
  <Step title="Generate a new token">
    `openssl rand -hex 32`.
  </Step>

  <Step title="Store it in the vault">
    Update `HEALTH_MONITOR_TOKEN` in the environment's 1Password Environment.
  </Step>

  <Step title="Propagate to the runtime">
    Run the **Sync Env** workflow, then redeploy `horizon-api` so the endpoints
    accept the new token. The endpoints fail closed, so old and new must not be
    live at once for long.
  </Step>

  <Step title="Update the monitors">
    Run `terraform apply <env>` so Better Stack sends the new `x-health-token`
    on every monitor (the token feeds `TF_VAR_health_monitor_token`).
  </Step>

  <Step title="Verify">
    Smoke-test the endpoints (below); all should return `200`.
  </Step>
</Steps>

## Regenerate status-page resources after an apply

`BETTERSTACK_STATUS_PAGE_RESOURCES` maps each component scope to its status-page
resource id, which the degraded sync uses to open/close Status Reports. After an
`apply` that creates or changes resources, regenerate it:

```bash theme={null}
node scripts/status-monitoring/fetch-status-page-resources.mjs <env>
```

Copy the resulting JSON into the environment's `BETTERSTACK_STATUS_PAGE_RESOURCES`
variable in 1Password, then run **Sync Env** and redeploy. Terraform also exposes
this as the `status_page_resources_json` output.

## Detect and resolve drift

Drift is any difference between the committed config and the live Better Stack
state (usually from a manual UI change).

<Steps>
  <Step title="Detect">
    Run a `plan`. A `0 to add, 0 to change, 0 to destroy` result means no drift.
    Any `~`/`+`/`-` lines are drift.
  </Step>

  <Step title="Decide the direction">
    **Code wins** (the norm): `apply` to push the committed config back over the
    manual change. **Adopt the change**: update the `.tf`/manifest to match
    reality, then confirm `plan` is clean.
  </Step>

  <Step title="Audit manifests (optional)">
    Regenerate and review the diff before committing:

    ```bash theme={null}
    node scripts/status-monitoring/export-betterstack-inventory.mjs <env>
    node scripts/status-monitoring/generate-adoption-manifest.mjs --all
    git diff infra/betterstack/adopted/
    ```
  </Step>
</Steps>

## Simulate degraded in development

To exercise the yellow path without a real incident:

```bash theme={null}
node scripts/status-monitoring/simulate-degraded-development.mjs
```

Then confirm the component turns yellow via a Status Report (not red) on the
development status page.

## Smoke-test the endpoints

```bash theme={null}
./scripts/status-monitoring/smoke-component-endpoints.sh
```

It loads `HEALTH_MONITOR_TOKEN` from `backend/horizon-api/.env.development` and
checks all component pull endpoints. Override the host or token inline:

```bash theme={null}
HOST=https://certification.api.bbrands.io \
HEALTH_MONITOR_TOKEN=<token> \
  ./scripts/status-monitoring/smoke-component-endpoints.sh
```

## Environment variable reference

Names only — never commit or paste values.

| Variable                                         | Purpose                                       |
| ------------------------------------------------ | --------------------------------------------- |
| `HEALTH_MONITOR_TOKEN`                           | `x-health-token` shared secret for endpoints  |
| `BETTERSTACK_UPTIME_API_TOKEN`                   | Uptime provider / status page management      |
| `BETTERSTACK_STATUS_PAGE_ID`                     | Numeric status page id per environment        |
| `BETTERSTACK_STATUS_PAGE_RESOURCES`              | Component scope → resource id map (JSON)      |
| `BETTERSTACK_HEARTBEAT_CRON_HEALTH_API`          | Heartbeat for the health/api cron             |
| `BETTERSTACK_HEARTBEAT_CRON_HEALTH_INTEGRATIONS` | Heartbeat for the health/integrations cron    |
| `BETTERSTACK_HEARTBEAT_CRON_METRICS_PUSH`        | Heartbeat for the metrics-push cron           |
| `BETTERSTACK_METRICS_URL`                        | Telemetry metrics ingestion URL (optional)    |
| `BETTERSTACK_METRICS_TOKEN`                      | Telemetry metrics token (optional)            |
| `LOGTAIL_API_TOKEN`                              | Enables the Terraform log-alerts modules      |
| `TF_BACKEND_OVERRIDE`                            | Remote state `backend "s3"` block (CI secret) |
| `TF_STATE_ACCESS_KEY` / `TF_STATE_SECRET_KEY`    | Remote state credentials (CI secrets)         |

## Related source

* Operational runbook: `docs/initiatives/betterstack/02-betterstack-runbook.md`
* Status page operations: `docs/operations/status-page.md`
* Env sync script: `scripts/env/sync-local.mjs`