# Webhook delivery and troubleshooting This reference describes how phone.systems(TM) delivers webhook events to your server: what counts as a successful delivery, how failed deliveries are retried, in which order events arrive, and when events are dropped. It also lists common problems and how to fix them. ## Successful and failed deliveries phone.systems(TM) sends one HTTPS `POST` request for each event. | Result | Description | | --- | --- | | Success | Your server responds with any `2xx` status code. The response body is ignored. | | Failure | Your server responds with any other status code. A redirect (`3xx`) is not followed and counts as a failure. A `4xx` response is retried in the same way as a `5xx` response. A timeout, a connection error, or a TLS error is also a failure. | | Timeouts | phone.systems(TM) waits up to 3 seconds to connect to your server and up to 10 seconds for the response. | Respond quickly. Store the event, respond with `2xx`, and process the event afterwards. ## Retries If a delivery attempt fails, phone.systems(TM) retries the event. An event is attempted up to 10 times in total. The waits between attempts are: | Attempt | Wait before the attempt | | --- | --- | | 1 | None. The event is sent as soon as it is ready. | | 2 | Up to 1 second | | 3 | Up to 2 seconds | | 4 | Up to 5 seconds | | 5 | Up to 10 seconds | | 6 | Up to 30 seconds | | 7 | Up to 1 minute | | 8 | Up to 2 minutes | | 9 | Up to 5 minutes | | 10 | Up to 5 minutes | Each wait is a random time between half of the listed value and the listed value. If the 10th attempt fails, the event is dropped, about 7 to 14 minutes after the first attempt. When your server fails for more than one call at a time, phone.systems(TM) treats it as down. New events then wait in the queue instead of using up their attempts, and they are sent as soon as your server responds again. Events that are already being retried keep their schedule. The `X-PS-Attempt` header shows the number of the attempt. If a failed response includes a `Retry-After` header, phone.systems(TM) waits for that time before the next attempt, up to 5 minutes. Use it, for example with a `429` or `503` response, when your server needs to slow down. ## Ordering and concurrency - Events of the same call, identified by `global_tag`, are delivered one at a time, in the order phone.systems(TM) received them. This covers the live events and the events of all CDRs of the call together, also across retries. The next event of the call is sent only after the previous one was delivered or dropped. - Events of different calls are delivered independently, in any order, and can arrive at the same time. Up to 8 requests can be in progress at the same time for your webhook, but never two of the same call. - The CDR events of a call normally arrive after its live events. When the end of a call is noticed late (`"reason": "lost"`, up to 3 minutes later), its CDR events can arrive before `call.ended`. Use `cdr_number` to place a CDR within its call. - `ts` is when the event was produced, by more than one system. Do not sort events by `ts`. Process the events of a call in the order they arrive. - A slow or failing server delays only its own events. It does not affect other accounts. ## Duplicate events Delivery is at least once. Your server can receive the same event more than once, for example when your response was lost or arrived after the timeout. - A retried event has the same `id` and `X-PS-Event-Id`, but a new `X-Request-Id` and `X-PS-Attempt`. - Store the `id` of processed events and skip an event whose `id` you already processed. - Respond with `2xx` to a duplicate, so that it is not retried again. ## Dropped events An event is dropped, and counted in **Dropped** on the **Webhooks** page, in these cases: - All 10 delivery attempts failed. - Too many events are waiting for your server. When the queue of your webhook is full, the oldest waiting events are dropped. An event that is being sent is never dropped this way. Dropped events are not sent again. An event that is larger than 1 MiB is not sent. In practice, this happens only with a very long transcript. When you delete the webhook, events stop being sent within 30 seconds and waiting events are discarded. If the delivery service restarts, events that were waiting to be delivered are lost and are not counted in **Dropped**. The **Webhooks** page then shows **Event bus restarted at** and the time. After the restart, calls in progress are reported again with `"catch_up": true`. A `call.connected` or `call.ended` for a call whose `call.started` you never received means that `call.started` was dropped. ## Limits | Limit | Value | | --- | --- | | Webhooks per account | 1 | | URL length | 2,048 characters | | Event size | 1 MiB (1,048,576 bytes) | | Requests in parallel | 8 | | Attempts per event | 10, over about 7 to 14 minutes | | Response time | 3 seconds to connect, 10 seconds for the response | | Signature age | Your choice. 5 minutes is recommended. | ## Best practices - **Verify, then respond fast.** Check the signature, store the event, respond with `2xx`, and process the event in the background. - **Skip duplicates by** `id`, or by the `X-PS-Event-Id` header before you parse the body. Keeping processed IDs for one day is enough. - **Key your data by what it describes.** Identify a call by `global_tag` and a CDR by its `uuid` (`cdr_uuid` in follow-up events). Do not use the event `id`, which is different for every event. `global_tag` can be `null`. Fall back to the CDR `uuid` for such events. - **Accept parallel requests.** Up to 8 arrive at the same time, but never two of the same call. - **Process the events of a call in the order they arrive.** Do not sort them by `ts`. For when things happened, use `time_start`, `time_connect`, and `time_end` in `data`. - **Do not wait for** `"final": true` before you process a call. The last CDR can arrive much later, or not at all. - **Ignore fields and event types that you do not recognize.** New ones can be added. ## Troubleshooting Check the status and the counters on the [Webhooks page](set-up-and-manage-webhooks.html#ps3-webhooks-status) first. When **Last attempt failed** is shown, the page also shows the details of the last failure, such as the HTTP status, the error, the request ID, and the event ID. **Settings > Webhooks** is not shown You are signed in as a PBX user with the **agent** role. Only the DIDWW account owner can manage the webhook. Open phone.systems(TM) from the DIDWW User Panel. If the page is still not shown, contact DIDWW Support. The URL is rejected The URL does not use HTTPS, or it points to a private or internal address. Use a public `https://` URL. See [Add a webhook](set-up-and-manage-webhooks.html#ps3-webhooks-add). The signature is invalid The body was parsed or changed before the check, the wrong secret is used, or the secret changed after the webhook was added again. Verify the signature on the raw body bytes. Copy the current secret from the **Webhooks** page. See [When the signature does not match](verify-webhook-requests.html#ps3-webhooks-signature-mismatch). Requests are rejected as too old The clock of your server is wrong, or your check uses a tolerance shorter than the delay. Synchronize the server clock with NTP and allow 5 minutes. Each retry is signed again with a new `t`. **Last attempt failed** Your server responded with a non-`2xx` status code, redirected the request, did not respond within 10 seconds, or could not be reached. Check the error and the request ID of the last failure, and find the request in your server logs. Fix the endpoint. Waiting events are retried automatically. The error is an HTTP status, for example `503 Service Unavailable` Your server responded with this status. Find the request ID in your server logs. A `401` often means that your signature check rejects valid requests. See [When the signature does not match](verify-webhook-requests.html#ps3-webhooks-signature-mismatch). The error is `timeout` Your server did not respond in time. Respond before you process the event. The error is `tls certificate error` Your certificate is not valid for the host name, has expired, or is self-signed. Renew or fix the certificate. Until then, you can add the webhook again with **Skip certificate verification** turned on. The error is `connection failed` The connection was refused or reset. Check that your server is running and that its port is open to the internet. The error is `destination not allowed` The host name does not resolve, or resolves to a private or internal address. Use a public host name with a public address. The error is `invalid url` The URL cannot be used. Delete the webhook and add it again with a valid URL. The error is `proxy error` A problem on the phone.systems(TM) side. Contact DIDWW Support if it continues. The same event arrives more than once A delivery was retried, or your response did not reach phone.systems(TM). Skip events whose `id` you already processed. See [Duplicate events](#ps3-webhooks-duplicates). **Dropped** increases Events failed 10 times, or the queue was full. Fix the endpoint and respond faster. Use [Call History](../../Call-analytics/call-history.html#ps3-call-history) or your own records to fill the gap. See [Dropped events](#ps3-webhooks-dropped). No recording, transcript, summary, or scoring events Recording events need cloud storage. Transcript and summary events need AI Call Insights. Scoring events also need an active AI Call Scoring scorecard. The call may also not meet the AI processing conditions. Set up [cloud storage](../cloud-storage-integrations.html#ps3-cloud-storage-integrations), enable [AI Call Insights](../../ai/ai-settings.html#ps3-ai-settings-enable), and activate an [AI Call Scoring scorecard](../../ai/ai-call-scoring.html#ps3-ai-call-scoring-settings). Check for `call.cdr.attachment_failed` and `call.cdr.ai_processing_failed` events. No events after the webhook was added Events are sent only for calls made after the webhook was added. Make a new call. If the status stays **Waiting for first delivery**, contact DIDWW Support. No events for older calls Calls that ended before the webhook was added produce no events and are never replayed. Use [Call History](../../Call-analytics/call-history.html#ps3-call-history) for earlier calls. If the problem continues, contact DIDWW Support and include the time of the problem, the event ID, and the request ID from your logs or the **Webhooks** page. Never send the webhook secret. ## Related resources - [Call Webhooks](../webhooks.html) - Understand how call webhooks work and their limitations. - [Set up and manage a webhook](set-up-and-manage-webhooks.html) - Check the status and the counters of the webhook. - [Verify webhook requests](verify-webhook-requests.html) - Check the signature and handle duplicates. - [Webhook event reference](webhook-event-reference.html) - Look up the headers and the fields of every event.