Skip to main content
This runbook covers the recurring operational tasks for the monitoring platform. Each task pairs the functional trigger (“when do I need this”) with the exact commands to run. No task here ever prints or stores secret values.

​
Sync environment variables

The environment variables live in 1Password Environments (one per environment) and flow to two destinations: local .env.<env> files for development, and Vercel for deployed apps.

1Password → local files

Run pnpm env:sync to mirror every 1Password Environment into each app’s .env.<environment> file. With no arguments it syncs all apps and all environments; environments your token cannot access are skipped with a warning instead of failing.

1Password → Vercel

Run the Sync Env workflow (workflow_dispatch) to push a 1Password Environment into Vercel, then redeploy horizon-api so the new values take effect.
Editing a variable in 1Password is not enough for deployed environments: after the Sync Env workflow updates Vercel, you must redeploy horizon-api for the runtime to read the new values.

​
Add or change a monitor

Monitoring is code — never click-ops in the Better Stack UI.
1

Edit the definition

Change the module under infra/betterstack/modules/ or the environment’s adopted/<env>.json desired-state manifest.
2

Open a PR

The validate lane runs automatically (fmt, validate, JSON checks). See CI/CD for Monitoring.
3

Plan

Dispatch the workflow with action = plan for the target environment and read the diff. Locally: pnpm tf:betterstack:plan.
4

Apply

Dispatch with action = apply and type APPLY, or run ./scripts/status-monitoring/terraform-apply.sh apply <env> locally. Confirm a follow-up plan shows no changes.

​
Rotate HEALTH_MONITOR_TOKEN

The health token authenticates Better Stack’s monitors against the component endpoints. Rotate it if it may have leaked, or on a routine schedule. Order matters to avoid a monitoring gap.
1

Generate a new token

openssl rand -hex 32.
2

Store it in the vault

Update HEALTH_MONITOR_TOKEN in the environment’s 1Password Environment.
3

Propagate to the runtime

Run the Sync Env workflow, then redeploy horizon-api so the endpoints accept the new token. The endpoints fail closed, so old and new must not be live at once for long.
4

Update the monitors

Run terraform apply <env> so Better Stack sends the new x-health-token on every monitor (the token feeds TF_VAR_health_monitor_token).
5

Verify

Smoke-test the endpoints (below); all should return 200.

​
Regenerate status-page resources after an apply

BETTERSTACK_STATUS_PAGE_RESOURCES maps each component scope to its status-page resource id, which the degraded sync uses to open/close Status Reports. After an apply that creates or changes resources, regenerate it:
Copy the resulting JSON into the environment’s BETTERSTACK_STATUS_PAGE_RESOURCES variable in 1Password, then run Sync Env and redeploy. Terraform also exposes this as the status_page_resources_json output.

​
Detect and resolve drift

Drift is any difference between the committed config and the live Better Stack state (usually from a manual UI change).
1

Detect

Run a plan. A 0 to add, 0 to change, 0 to destroy result means no drift. Any ~/+/- lines are drift.
2

Decide the direction

Code wins (the norm): apply to push the committed config back over the manual change. Adopt the change: update the .tf/manifest to match reality, then confirm plan is clean.
3

Audit manifests (optional)

Regenerate and review the diff before committing:

​
Simulate degraded in development

To exercise the yellow path without a real incident:
Then confirm the component turns yellow via a Status Report (not red) on the development status page.

​
Smoke-test the endpoints

It loads HEALTH_MONITOR_TOKEN from backend/horizon-api/.env.development and checks all component pull endpoints. Override the host or token inline:

​
Environment variable reference

Names only — never commit or paste values.
  • Operational runbook: docs/initiatives/betterstack/02-betterstack-runbook.md
  • Status page operations: docs/operations/status-page.md
  • Env sync script: scripts/env/sync-local.mjs