Skip to main content
The status page is only as good as the signal behind it. B Brands does not rely on a synthetic ping saying “the server is up”; it derives each component’s health from real production traffic and publishes three distinct states.

​
The tri-state verdict

Every component (an API category or an integration) is continuously classified into one of three states.

operational

Everything is within the healthy band.

degraded

A stale cron, a single failing route, or a sustained low-level error rate.

down

A short-window error spike that means a real outage.

​
How the verdict is computed

Functionally: the API constantly counts successful and failed requests per component, looks at two time windows, and picks the worst state that applies. Technically, precedence is evaluated top-down:
  1. down — short-window error rate >= 50% over >= 5 requests, or (for integrations) a failed active probe.
  2. degraded — a stale folded cron, an individual route failing, or a sustained long-window error rate >= 1% over >= 5 requests.
  3. operational — everything else.
Hysteresis prevents flapping. Once a component enters degraded, it only returns to operational when the long-window error rate drops below the recovery threshold (0.5%). The last verdict is cached in Redis (health:verdict:{component}).

​
Traffic windows

Counters live in 5-minute Redis buckets (traffic:{scope}:{ok|err}:{windowStart}, TTL ~70 min). Two spans are read:

​
Thresholds

All thresholds are exported from lib/endpoint/health/integration-traffic.ts, so tuning is a one-line code change reviewed in a PR.

​
Per-route tracking

recordApiTraffic records both a category counter and a per-route counter. The path is normalized (normalizeRoutePath) so dynamic segments — UUIDs, numeric ids, long opaque tokens — collapse to [id], which bounds cardinality. Each route key is {METHOD}:{normalized-path}. This catches a single endpoint failing hard (for example one broken MaihueGO route) inside an otherwise healthy category: the route signal degrades the whole category even when the aggregate error rate stays low and would otherwise hide the problem.

​
Component pull endpoints

Better Stack does not read Redis or internal state. It polls one HTTP endpoint per component:
Both require the header x-health-token, matched against HEALTH_MONITOR_TOKEN:
  • 401 when the header does not match.
  • 503 when the token is unset (fail closed — never expose health without auth).
  • 503 only when the verdict is down. Both operational and degraded return 200; the yellow layer is published through Status Reports, never as downtime. The full verdict is always in the response body.

​
Degraded sync (Status Reports)

lib/endpoint/health/degraded-sync.ts reconciles each verdict with the published Status Reports on Better Stack:
  • Entering degraded → createDegradedReport, storing the report id in health:degraded-report:{component}.
  • Leaving degraded → resolveDegradedReport publishes a resolved status update on the report (never deletes it, so the status-page entry and its Slack mirror both show the recovery) and clears the marker.
It reads the component → status-page-resource id map from the BETTERSTACK_STATUS_PAGE_RESOURCES JSON environment variable. It runs from both health crons after verdicts are computed and is fully best-effort (it never throws). Any component not present in the map is simply skipped.

​
Cron heartbeats

Heartbeats confirm the scheduler itself is alive — a separate concern from component health. pingHeartbeat retries once with a short backoff inside its timeout, absorbing transient network blips.

​
Infrastructure metrics

The metrics-push cron (/api/v3/cron/metrics/push) measures database and Redis latency for the Infrastructure charts. Because a single cold-start sample produced false spikes, the collector warms up the connection with one untimed probe, takes three timed samples, and reports the median — keeping the 5-minute cadence while smoothing one-off outliers.

​
End-to-end signal path

  • Verdict engine and thresholds: backend/horizon-api/lib/endpoint/health/
  • Status Report client: backend/horizon-api/lib/integration/betterstack/status-report.client.ts
  • Deep dive: docs/initiatives/betterstack/01-architecture.md