> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bbrands.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Retries and failures

> How each response is classified, the backoff schedule, the exhausted state, automatic disabling after consecutive failures, and how operators replay.

## Delivery states

Every delivery is one row in `webhook_dispatch`. Its `status` moves through:

```mermaid theme={null}
stateDiagram-v2
    [*] --> pending: routed
    pending --> delivering: worker picks it up
    delivering --> success: 2xx
    delivering --> error: retryable failure (next_retry_at set)
    delivering --> error: terminal failure (no retry)
    error --> delivering: retry due
    delivering --> exhausted: retryable failure on attempt 5
    success --> pending: replay (confirm_duplicate)
    error --> pending: replay
    exhausted --> pending: replay
```

| `status`     | Meaning                                                                                                   |
| ------------ | --------------------------------------------------------------------------------------------------------- |
| `pending`    | Routed, waiting for the deliver worker.                                                                   |
| `delivering` | A `POST` is in flight.                                                                                    |
| `success`    | Subscriber answered `2xx`. `delivered_at` is set.                                                         |
| `error`      | Last attempt failed. If `next_retry_at` is set, a retry is scheduled; otherwise the failure was terminal. |
| `exhausted`  | The fifth attempt failed with a retryable error. No further automatic attempts.                           |

`attempts`, `last_http_status` and `last_error` are updated on every
attempt so the ledger explains itself. `attempts` is the lifetime POST
count for that row: operator replay does not zero it.

## Outcome matrix

| Subscriber response                                           | Classification | Effect                                                                                                         |
| ------------------------------------------------------------- | -------------- | -------------------------------------------------------------------------------------------------------------- |
| `2xx`                                                         | Success        | `status = success`; webhook `consecutive_failures` reset to `0`.                                               |
| `408`, `429`, any `5xx`                                       | Retryable      | `status = error`, `next_retry_at` scheduled; `exhausted` on the last attempt.                                  |
| Timeout (10 s), DNS/TCP/TLS failure                           | Retryable      | Same as above.                                                                                                 |
| Any other `4xx` (`400`, `401`, `403`, `404`, `410`, `422`, …) | Terminal       | `status = error` with no retry. Counts toward auto-disable.                                                    |
| Any `3xx`                                                     | Terminal       | Redirects are never followed (a signed body must not be replayed to a third host). Counts toward auto-disable. |

<Note>
  Only **terminal** and **exhausted** outcomes increment
  `consecutive_failures`. A `503` that succeeds on the next retry is not a
  lost delivery and does not move the breaker.
</Note>

## Backoff schedule

Retryable failures are redelivered on a fixed, subscriber-facing schedule
(`WEBHOOK_OUT_MAX_ATTEMPTS = 5`):

| Attempt | Waits before it                                                      | Cumulative |
| ------- | -------------------------------------------------------------------- | ---------- |
| 1       | —                                                                    | 0          |
| 2       | 1 minute                                                             | 1 m        |
| 3       | 5 minutes                                                            | 6 m        |
| 4       | 30 minutes                                                           | 36 m       |
| 5       | 2 hours                                                              | 2 h 36 m   |
| after 5 | `exhausted` (a 6-hour slot is reserved but never used automatically) | —          |

`next_retry_at` on the ledger row shows the exact time of the upcoming
attempt. The body's `attempt` field increments accordingly while `id` stays
the same.

## Automatic disable

A dead endpoint should not consume queue capacity indefinitely. When a
webhook accumulates **20 consecutive** terminal or exhausted deliveries:

* `is_actived` becomes `false`,
* `disabled_at` is stamped and `disabled_reason = "consecutive-failures"`,
* the routing stage stops writing dispatch rows for it,
* the health component `webhook-out` and the dashboard summary count it under
  `disabled_auto`.

Deliveries already in flight finish their current attempt; nothing new is
scheduled.

### Re-enabling

1. Fix the endpoint (or its URL via `PATCH`).
2. Send a test delivery: `POST /api/v3/webhook/webhook/{id}/test`.
3. Activate: `PATCH /api/v3/webhook/webhook/{id}/active` with
   `{ "is_actived": true }`. This clears `consecutive_failures`,
   `disabled_at` and `disabled_reason`.
4. Replay what was missed (below).

There is one other `disabled_reason`: `legacy-secret-rotation-required`.
Webhooks that existed before this release were switched off by migration
`0237` because their secrets predate the signing contract; rotate the secret,
then activate.

## Replay

`POST /api/v3/webhook/dispatch/{id}/replay` re-enqueues one ledger row
(`status = pending`, last error and `next_retry_at` cleared) **without
resetting `attempts`**. The next POST increments the same counter, so the
row keeps a full delivery history. The subscriber receives the same `id`,
`event` and `data`; `attempt` is the next sequential number.

* Replaying an `error` or `exhausted` row needs no body.
* Replaying a `success` row returns `409 Conflict` unless the body includes
  `{ "confirm_duplicate": true }`, because the subscriber will see the event
  twice. Endpoints that deduplicate on `id` handle this transparently.

Automatic retries still stop at five POSTs (`WEBHOOK_OUT_MAX_ATTEMPTS`). An
operator replay of an `exhausted` row performs one additional POST; if that
POST is retryable it is marked `exhausted` again (no new automatic schedule).

Horizon Enterprise exposes the same action per row on the webhook detail
screen, filtered by status, event and date.

<Warning>
  Replay does not resurrect the source resource. If the row was purged since,
  `data.document_id` still points at the old id and your API read will
  return `404` — treat that as "already gone".
</Warning>