Retries and failures

​

Delivery states

Every delivery is one row in webhook_dispatch. Its status moves through: attempts, last_http_status and last_error are updated on every attempt so the ledger explains itself. attempts is the lifetime POST count for that row: operator replay does not zero it.

​

Outcome matrix

​

Backoff schedule

Retryable failures are redelivered on a fixed, subscriber-facing schedule (WEBHOOK_OUT_MAX_ATTEMPTS = 5): next_retry_at on the ledger row shows the exact time of the upcoming attempt. The body’s attempt field increments accordingly while id stays the same.

​

Automatic disable

A dead endpoint should not consume queue capacity indefinitely. When a webhook accumulates 20 consecutive terminal or exhausted deliveries:

• is_actived becomes false,
• disabled_at is stamped and disabled_reason = "consecutive-failures",
• the routing stage stops writing dispatch rows for it,
• the health component webhook-out and the dashboard summary count it under disabled_auto.

Deliveries already in flight finish their current attempt; nothing new is scheduled.

​

Re-enabling

1. Fix the endpoint (or its URL via PATCH).
2. Send a test delivery: POST /api/v3/webhook/webhook/{id}/test.
3. Activate: PATCH /api/v3/webhook/webhook/{id}/active with { "is_actived": true }. This clears consecutive_failures, disabled_at and disabled_reason.
4. Replay what was missed (below).

There is one other disabled_reason: legacy-secret-rotation-required. Webhooks that existed before this release were switched off by migration 0237 because their secrets predate the signing contract; rotate the secret, then activate.

​

Replay

POST /api/v3/webhook/dispatch/{id}/replay re-enqueues one ledger row (status = pending, last error and next_retry_at cleared) without resetting attempts. The next POST increments the same counter, so the row keeps a full delivery history. The subscriber receives the same id, event and data; attempt is the next sequential number.

• Replaying an error or exhausted row needs no body.
• Replaying a success row returns 409 Conflict unless the body includes { "confirm_duplicate": true }, because the subscriber will see the event twice. Endpoints that deduplicate on id handle this transparently.

Automatic retries still stop at five POSTs (WEBHOOK_OUT_MAX_ATTEMPTS). An operator replay of an exhausted row performs one additional POST; if that POST is retryable it is marked exhausted again (no new automatic schedule). Horizon Enterprise exposes the same action per row on the webhook detail screen, filtered by status, event and date.