Webhook delivery and troubleshooting#

This reference describes how phone.systems™ delivers webhook events to your server: what counts as a successful delivery, how failed deliveries are retried, in which order events arrive, and when events are dropped. It also lists common problems and how to fix them.

Successful and failed deliveries#

phone.systems™ sends one HTTPS POST request for each event.

Result	Description
Success	Your server responds with any `2xx` status code. The response body is ignored.
Failure	Your server responds with any other status code. A redirect (`3xx`) is not followed and counts as a failure. A `4xx` response is retried in the same way as a `5xx` response. A timeout, a connection error, or a TLS error is also a failure.
Timeouts	phone.systems™ waits up to 3 seconds to connect to your server and up to 10 seconds for the response.

Respond quickly. Store the event, respond with 2xx, and process the event afterwards.

Retries#

If a delivery attempt fails, phone.systems™ retries the event. An event is attempted up to 10 times in total. The waits between attempts are:

Attempt	Wait before the attempt
1	None. The event is sent as soon as it is ready.
2	Up to 1 second
3	Up to 2 seconds
4	Up to 5 seconds
5	Up to 10 seconds
6	Up to 30 seconds
7	Up to 1 minute
8	Up to 2 minutes
9	Up to 5 minutes
10	Up to 5 minutes

Each wait is a random time between half of the listed value and the listed value. If the 10th attempt fails, the event is dropped, about 7 to 14 minutes after the first attempt.

When your server fails for more than one call at a time, phone.systems™ treats it as down. New events then wait in the queue instead of using up their attempts, and they are sent as soon as your server responds again. Events that are already being retried keep their schedule. The X-PS-Attempt header shows the number of the attempt.

If a failed response includes a Retry-After header, phone.systems™ waits for that time before the next attempt, up to 5 minutes. Use it, for example with a 429 or 503 response, when your server needs to slow down.

Ordering and concurrency#

Events of the same call, identified by global_tag, are delivered one at a time, in the order phone.systems™ received them. This covers the live events and the events of all CDRs of the call together, also across retries. The next event of the call is sent only after the previous one was delivered or dropped.
Events of different calls are delivered independently, in any order, and can arrive at the same time. Up to 8 requests can be in progress at the same time for your webhook, but never two of the same call.
The CDR events of a call normally arrive after its live events. When the end of a call is noticed late ("reason": "lost", up to 3 minutes later), its CDR events can arrive before call.ended. Use cdr_number to place a CDR within its call.
ts is when the event was produced, by more than one system. Do not sort events by ts. Process the events of a call in the order they arrive.
A slow or failing server delays only its own events. It does not affect other accounts.

Duplicate events#

Delivery is at least once. Your server can receive the same event more than once, for example when your response was lost or arrived after the timeout.

A retried event has the same id and X-PS-Event-Id, but a new X-Request-Id and X-PS-Attempt.
Store the id of processed events and skip an event whose id you already processed.
Respond with 2xx to a duplicate, so that it is not retried again.

Dropped events#

An event is dropped, and counted in Dropped on the Webhooks page, in these cases:

All 10 delivery attempts failed.
Too many events are waiting for your server. When the queue of your webhook is full, the oldest waiting events are dropped. An event that is being sent is never dropped this way.

Dropped events are not sent again. An event that is larger than 1 MiB is not sent. In practice, this happens only with a very long transcript. When you delete the webhook, events stop being sent within 30 seconds and waiting events are discarded.

If the delivery service restarts, events that were waiting to be delivered are lost and are not counted in Dropped. The Webhooks page then shows Event bus restarted at and the time. After the restart, calls in progress are reported again with "catch_up": true.

A call.connected or call.ended for a call whose call.started you never received means that call.started was dropped.

Limits#

Limit	Value
Webhooks per account	1
URL length	2,048 characters
Event size	1 MiB (1,048,576 bytes)
Requests in parallel	8
Attempts per event	10, over about 7 to 14 minutes
Response time	3 seconds to connect, 10 seconds for the response
Signature age	Your choice. 5 minutes is recommended.

Best practices#

Verify, then respond fast. Check the signature, store the event, respond with 2xx, and process the event in the background.
Skip duplicates by id, or by the X-PS-Event-Id header before you parse the body. Keeping processed IDs for one day is enough.
Key your data by what it describes. Identify a call by global_tag and a CDR by its uuid (cdr_uuid in follow-up events). Do not use the event id, which is different for every event. global_tag can be null. Fall back to the CDR uuid for such events.
Accept parallel requests. Up to 8 arrive at the same time, but never two of the same call.
Process the events of a call in the order they arrive. Do not sort them by ts. For when things happened, use time_start, time_connect, and time_end in data.
Do not wait for "final": true before you process a call. The last CDR can arrive much later, or not at all.
Ignore fields and event types that you do not recognize. New ones can be added.

Troubleshooting#

Check the status and the counters on the Webhooks page first. When Last attempt failed is shown, the page also shows the details of the last failure, such as the HTTP status, the error, the request ID, and the event ID.

If the problem continues, contact DIDWW Support and include the time of the problem, the event ID, and the request ID from your logs or the Webhooks page. Never send the webhook secret.