> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bbrands.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Health Architecture

> The tri-state verdict engine, the component pull endpoints, degraded sync via Status Reports and cron heartbeats — how horizon-api turns real traffic into a status page.

The status page is only as good as the signal behind it. B Brands does not rely
on a synthetic ping saying "the server is up"; it derives each component's health
from **real production traffic** and publishes three distinct states.

## The tri-state verdict

Every component (an API category or an integration) is continuously classified
into one of three states.

<CardGroup cols={3}>
  <Card title="operational" icon="circle-check">
    Everything is within the healthy band.
  </Card>

  <Card title="degraded" icon="triangle-exclamation">
    A stale cron, a single failing route, or a sustained low-level error rate.
  </Card>

  <Card title="down" icon="circle-xmark">
    A short-window error spike that means a real outage.
  </Card>
</CardGroup>

### How the verdict is computed

Functionally: the API constantly counts successful and failed requests per
component, looks at two time windows, and picks the worst state that applies.

Technically, precedence is evaluated top-down:

1. **`down`** — short-window error rate `>= 50%` over `>= 5` requests, or (for
   integrations) a failed active probe.
2. **`degraded`** — a stale folded cron, an individual route failing, or a
   sustained long-window error rate `>= 1%` over `>= 5` requests.
3. **`operational`** — everything else.

<Note>
  **Hysteresis** prevents flapping. Once a component enters `degraded`, it only
  returns to `operational` when the long-window error rate drops below the
  recovery threshold (`0.5%`). The last verdict is cached in Redis
  (`health:verdict:{component}`).
</Note>

### Traffic windows

Counters live in 5-minute Redis buckets
(`traffic:{scope}:{ok|err}:{windowStart}`, TTL \~70 min). Two spans are read:

| Window | Buckets | Span     | Drives     |
| ------ | ------- | -------- | ---------- |
| Short  | 2       | \~10 min | `down`     |
| Long   | 12      | \~60 min | `degraded` |

### Thresholds

All thresholds are exported from `lib/endpoint/health/integration-traffic.ts`,
so tuning is a one-line code change reviewed in a PR.

| Constant                       | Value | Meaning                                    |
| ------------------------------ | ----- | ------------------------------------------ |
| `MAX_TRAFFIC_ERROR_RATE`       | 0.50  | Short-window rate that means `down`.       |
| `MIN_TRAFFIC_VOLUME`           | 5     | Minimum requests before a rate is trusted. |
| `DEGRADED_ERROR_RATE`          | 0.01  | Long-window rate to enter `degraded`.      |
| `DEGRADED_RECOVERY_ERROR_RATE` | 0.005 | Long-window rate to leave `degraded`.      |
| `MIN_ROUTE_VOLUME`             | 5     | Minimum requests for a route to count.     |
| `ROUTE_DOWN_ERROR_RATE`        | 0.50  | Per-route rate that degrades its category. |

## Per-route tracking

`recordApiTraffic` records both a category counter and a per-route counter. The
path is normalized (`normalizeRoutePath`) so dynamic segments — UUIDs, numeric
ids, long opaque tokens — collapse to `[id]`, which bounds cardinality. Each
route key is `{METHOD}:{normalized-path}`.

This catches a single endpoint failing hard (for example one broken MaihueGO
route) inside an otherwise healthy category: the route signal degrades the whole
category even when the aggregate error rate stays low and would otherwise hide
the problem.

## Component pull endpoints

Better Stack does not read Redis or internal state. It polls one HTTP endpoint
per component:

```text theme={null}
GET /api/v3/health/component/api/<category>
GET /api/v3/health/component/integration/<name>
```

Both require the header `x-health-token`, matched against `HEALTH_MONITOR_TOKEN`:

* `401` when the header does not match.
* `503` when the token is unset (**fail closed** — never expose health without
  auth).
* `503` **only** when the verdict is `down`. Both `operational` and `degraded`
  return `200`; the yellow layer is published through Status Reports, never as
  downtime. The full verdict is always in the response body.

## Degraded sync (Status Reports)

`lib/endpoint/health/degraded-sync.ts` reconciles each verdict with the
published Status Reports on Better Stack:

* Entering `degraded` → `createDegradedReport`, storing the report id in
  `health:degraded-report:{component}`.
* Leaving `degraded` → `resolveDegradedReport` publishes a `resolved` status
  update on the report (never deletes it, so the status-page entry and its
  Slack mirror both show the recovery) and clears the marker.

It reads the component → status-page-resource id map from the
`BETTERSTACK_STATUS_PAGE_RESOURCES` JSON environment variable. It runs from both
health crons after verdicts are computed and is fully best-effort (it never
throws). Any component not present in the map is simply skipped.

## Cron heartbeats

Heartbeats confirm the scheduler itself is alive — a separate concern from
component health.

| Cron route                         | Heartbeat env var                                | Expect / grace  |
| ---------------------------------- | ------------------------------------------------ | --------------- |
| `/api/v3/cron/health/api`          | `BETTERSTACK_HEARTBEAT_CRON_HEALTH_API`          | 5 min / \~3 min |
| `/api/v3/cron/health/integrations` | `BETTERSTACK_HEARTBEAT_CRON_HEALTH_INTEGRATIONS` | 5 min / \~3 min |
| `/api/v3/cron/metrics/push`        | `BETTERSTACK_HEARTBEAT_CRON_METRICS_PUSH`        | 5 min / \~3 min |

`pingHeartbeat` retries once with a short backoff inside its timeout, absorbing
transient network blips.

## Infrastructure metrics

The metrics-push cron (`/api/v3/cron/metrics/push`) measures database and Redis
latency for the Infrastructure charts. Because a single cold-start sample
produced false spikes, the collector warms up the connection with one untimed
probe, takes three timed samples, and reports the **median** — keeping the
5-minute cadence while smoothing one-off outliers.

## End-to-end signal path

```mermaid theme={null}
sequenceDiagram
    participant Client
    participant API as horizon-api
    participant Redis
    participant Monitor as Better Stack Monitor
    participant Reports as Status Reports
    participant Page as Status page

    Client->>API: Normal requests
    API->>Redis: recordApiTraffic (ok/err buckets)
    Monitor->>API: GET /health/component/... (x-health-token)
    API->>Redis: read short + long windows
    API-->>Monitor: 200 (operational/degraded) or 503 (down)
    Monitor->>Page: component red on 503
    Note over API,Reports: Every 5 min (health crons)
    API->>Reports: create/resolve degraded report
    Reports->>Page: component yellow while report is open
```

## Related source

* Verdict engine and thresholds: `backend/horizon-api/lib/endpoint/health/`
* Status Report client: `backend/horizon-api/lib/integration/betterstack/status-report.client.ts`
* Deep dive: `docs/initiatives/betterstack/01-architecture.md`