Webhook delivery and retries

The retry schedule, what to build your receiver against, and how to inspect, replay and debug deliveries.

A delivery is retried until your server answers 2xx.

Retry schedule

After failed attemptWait
15 seconds
230 seconds
32 minutes
410 minutes
51 hour
66 hours
712 hours

That is eight attempts over about 19 hours.

What counts as a failure:

  • Retried: a timeout, a connection error, and the statuses 408, 425, 429 and every 5xx.
  • Final, not retried: any other status, such as 400, 401, 404 or 410.
  • Done: any 2xx.

An endpoint that fails eight attempts in a row is paused for 5 minutes, and the pause doubles each time it happens again, up to 6 hours. Deliveries for a paused endpoint wait instead of using up their retries, your organization's owners and admins get one email, and the first success clears the pause.

So if your endpoint is down for an hour you lose nothing. If your endpoint answers 404 because the URL is wrong, Dialgem stops after one attempt, so fix the URL and use Resend.

What to build against

  • At least once. The same event can arrive twice, for example when your server processed it but its reply was lost. Deduplicate on the X-Dialgem-Delivery header, which is the same on every retry of one delivery.
  • Not strictly ordered. A retried earlier event can arrive after a later one. Use the ids and timestamps in the body, not arrival order, to decide what is newest.
  • Answer fast. Return 2xx as soon as you have stored the event, and do the work afterwards. A receiver that does slow work before replying will time out and be retried while it is still busy.
  • Be idempotent. Processing the same call_id and event twice must be harmless.

A simple pattern is a table keyed by delivery id:

sql
create table processed_webhooks (delivery_id text primary key, received_at timestamptz default now());

-- in the handler: insert, and if it already exists, return 200 without doing the work again
insert into processed_webhooks (delivery_id) values ($1) on conflict do nothing;

The delivery log

Settings, Webhooks, Deliveries lists what was sent, newest first. Open one to see:

  • the exact body that was sent;
  • each attempt's status and timing.

Deliveries are kept for 30 days.

If one failed, choose Resend to send it again. A resend is a new delivery, not a retry of the old one.

Debugging checklist

  1. Open the delivery in the log and read the status of each attempt.
  2. A 4xx other than 408, 425 or 429 means your server rejected it. Check the URL and your signature code. See Verify signatures.
  3. A timeout means your handler is too slow. Reply first, work later.
  4. A connection error usually means a firewall, an expired TLS certificate or a wrong hostname. The endpoint must be public HTTPS.
  5. No deliveries at all? Check that the endpoint is subscribed to the event and, if you limited it to some agents, that the call's agent is one of them.

Next