A voice agent post-call webhook evaluation pipeline should do one job before anything else: prove that a provider event was accepted durably and can be evaluated without duplicating work for the same inputs.
If you are running a prototype with a few manual calls, you do not need the full design in this guide. Verify the provider signature, save the event, and put one evaluation job on a durable queue. Add the rest when missing or duplicated calls would create a real monitoring gap.
Production systems need more discipline. A webhook can arrive twice, arrive before its transcript, arrive after a later event, or contain an audio URL that expires before a worker downloads it. The call may be over while provider analysis is still running.
TL;DR: Build the pipeline in 6 boundaries: verify the raw request, write an immutable receipt, acknowledge the provider, normalize one canonical event, enqueue a versioned evaluation, and update monitoring from a separate result event.
The acknowledgement line matters. Return success after durable acceptance, not after a slow evaluation and not before you can recover the event.
Scope: This guide covers the ingestion boundary from a provider's post-call event through a durable, versioned evaluation result. It does not prescribe a specific queue, database, evaluator, or monitoring vendor.
Methodology Note: This implementation guide is based on Hamming's analysis of production voice agent calls and post-call evaluation workflows across 10K+ voice agents (2025-2026). Hamming's platform has 10M+ mins protected. We've tested agents built on LiveKit, Pipecat, ElevenLabs, Retell, Vapi, and custom-built solutions.Provider behavior was checked against public ElevenLabs, Vapi, Retell, and Bland documentation on September 25, 2026. Confirm current provider contracts before deployment.
Last Updated: September 2026
Related Guides:
- Post-Call Analytics for Voice Agents - choose the production metrics that evaluation results should update
- Voice Agent Post-Call Metric Dictionary - keep score names, formulas, and denominators stable
- OpenTelemetry for Voice Agents - connect the provider event, evaluation job, and monitoring write with one trace
- Voice Agent Call Evidence Export - package controlled audio, transcript, and tool evidence for review
- Failed Production Calls to Regression Tests - promote confirmed failures into repeatable tests
- Voice Agent Call Review Triage - send the highest-value evaluation failures to a human
- Voice Agent Log Retention - set retention by artifact class and purpose
- PII Redaction Architecture - keep sensitive call data out of routine events and logs
What Should Happen Between a Post-Call Webhook and an Evaluation?
The safe path is linear:
voice provider -> signature verification -> immutable webhook receipt -> provider acknowledgement -> canonical call event -> evaluation queue -> versioned evaluator -> evaluation result event -> monitoring, review, and regression workflowsDo not let the webhook route download a recording, run an LLM grader, update five dashboards, and then return 200. That design turns provider delivery timeouts into duplicate work and makes recovery depend on whatever finished before the request died.
We used to think "return quickly" was the whole rule. It is not. Returning quickly before a durable write creates what we call the green-ack gap: the provider sees success, but your system can no longer prove the call entered evaluation.
Durable acknowledgement: A post-call webhook is durably acknowledged when the receiver has verified its origin and stored enough immutable data to recreate downstream processing before returning a success response.
The receipt can live in a database, object store, or durable event log. The storage choice matters less than the invariant: a process crash after acknowledgement must not erase the accepted event.
| Boundary | Must finish before success response? | What it proves | Keep out of the request path |
|---|---|---|---|
| Read raw body | Yes | Signature verification uses the exact delivered bytes | JSON transformation |
| Verify signature and timestamp | Yes | The request came from the expected provider and is fresh | Provider API calls |
| Validate envelope size and required identity | Yes | The receiver can safely identify and store the event | Full transcript validation |
| Persist immutable receipt | Yes | The event can be replayed after a crash | LLM evaluation |
| Enqueue normalization | Preferably yes, or use an outbox | Accepted receipts reach workers | Recording download |
| Normalize, evaluate, and publish scores | No | Downstream work remains retryable | Anything that delays acknowledgement |
If your datastore and queue cannot commit atomically, use an outbox or let a worker scan unprocessed receipts. Do not create a gap where storage succeeds but queue publication disappears.
Which Provider Event Should Trigger Evaluation?
"Call ended" and "call is ready to evaluate" are not always the same event.
Retell documents call_ended without call_analysis and a later call_analyzed event with analysis attached. ElevenLabs post-call transcription webhooks arrive after analysis and include transcript, metadata, and analysis. Vapi's server events expose an end-of-call-report with call artifacts. Bland's post-call docs describe both immediate payloads and delayed enrichment payloads.
The provider map should be configuration, not scattered if statements:
| Provider event | Safe first action | Ready for transcript evaluation? | Follow-up rule |
|---|---|---|---|
ElevenLabs post_call_transcription | Verify HMAC, store receipt | Usually yes | Track audio separately if needed |
Vapi end-of-call-report | Store call and artifact references | Yes when required artifacts exist | Fetch missing artifacts through a controlled worker |
Retell call_ended | Store completion state | Not if analysis is required | Wait for call_analyzed or fetch the final call object |
Retell call_analyzed | Merge by provider call ID | Yes | Enqueue only if the evaluation input version changed |
| Bland standard post-call webhook | Store the immediate payload | Depends on evaluator inputs | Merge later enrichment events by call ID and content type |
Evaluation-ready event: A call is evaluation-ready when every input required by the selected evaluation suite is present, immutable or versioned, and addressable by a stable call identity. Call completion alone does not satisfy that contract.
This distinction prevents two common bugs: scoring an incomplete transcript, and scoring the same call again when a richer event arrives later.
What Should the Canonical Event Contain?
Normalize provider payloads once at the ingestion boundary. Evaluators should not know that one provider says conversation_id, another says call_id, and another nests everything inside message.call.
Use a small canonical envelope with references to large or sensitive artifacts:
{ "event_name": "voice.call.ready_for_evaluation.v1", "event_id": "evt_01J5D8X9A7P4T6M2", "occurred_at": "2026-08-14T16:42:17Z", "provider": "provider_name", "provider_event_type": "post_call_analysis", "provider_call_id": "call_7f31", "workspace_id": "workspace_42", "agent_id": "agent_scheduler", "agent_version": "v38", "environment": "production", "input_version": "sha256:8d76...", "artifacts": { "transcript_ref": "artifact://transcripts/call_7f31/v2", "recording_ref": "artifact://recordings/call_7f31/v1", "tool_trace_ref": "artifact://tool-traces/call_7f31/v1" }, "evaluation_suite": { "suite_id": "production_scheduler", "suite_version": "2026-08-14.3" }, "trace": { "trace_id": "4fd0b86d28c9453aa2b4d2b86f2b7e7a", "source_receipt_id": "receipt_01J5D8WZRQ" }}Derive input_version from the exact normalized transcript, artifact versions, agent version, and evaluation suite inputs. Unchanged inputs, suite version, and evaluator version resolve to the existing evaluation attempt unless the replay records an explicit reason, which creates a new attempt. Give that intentional replay its own stable request ID so retrying the replay request does not create further attempts.
| Field | Why it exists | Cardinality and privacy rule |
|---|---|---|
event_name | Routes and versions the contract | Low cardinality |
event_id | Traces one canonical event | Unique, never a user identifier |
provider_call_id | Joins provider callbacks for one call | Treat as controlled metadata |
workspace_id | Preserves tenant boundary | Never infer it from caller data |
agent_version | Explains behavior changes | Required for release comparisons |
input_version | Makes evaluation reproducible | Hash, not raw content |
| Artifact references | Keep large data in governed storage | Signed access or service authorization |
suite_version | Explains scoring changes | Required on every result |
trace_id | Joins ingestion, evaluation, and monitoring | Do not overload it with call content |
Pair this event with the OpenTelemetry trace model for voice agents. The trace explains the path. The canonical event defines the evaluation contract.
How Do You Build an Idempotent Webhook Receiver?
Assume every delivery can repeat. Retell says it retries when a 2xx response does not arrive within 10 seconds, up to 3 times. ElevenLabs documents configurable retries and tells consumers to make handlers idempotent. Provider details differ, but the receiver rule does not.
delivery key = trusted tenant/provider-account scope + provider + stable provider event identityChoose the event identity in the provider adapter. Retell's current deduplication rules use event type plus call ID for per-call lifecycle events, and add the transfer start timestamp for repeated transfer attempts. A provider-native event ID is another valid identity when its documented scope fits. Exclude delivery-attempt counters and other retry-specific envelope fields.
Do not deduplicate on call ID alone: call_ended and call_analyzed represent different events. For providers that deliver revised artifacts without a stable event ID, define and test a key that includes the documented payload version or a hash of stable event content. A raw-body hash is suitable only when retry bodies are byte-stable. Preserve the raw body as receipt evidence regardless of the deduplication strategy.
The receiver sequence:
- Read the raw body with a strict size limit.
- Verify the provider signature and timestamp before JSON parsing.
- Parse and minimally validate the provider envelope.
- Compute the adapter's delivery key from trusted account scope and stable event identity.
- Insert an immutable receipt with a unique constraint on that delivery key.
- Publish an outbox record or mark the receipt ready for normalization.
- Return the provider's documented success status.
async function acceptPostCallWebhook(request: Request): Promise<Response> { const rawBody = await readRawBodyWithinLimit(request, 2_000_000); const verifiedProviderEvent = verifyProviderWebhook({ headers: request.headers, rawBody, }); // The adapter applies the provider's documented event-identity contract. const deliveryKey = createProviderDeliveryKey(verifiedProviderEvent); await persistReceiptAndOutbox({ deliveryKey, providerEvent: verifiedProviderEvent, rawBody, }); return new Response(null, { status: 200 });}The sample is illustrative and intentionally provider-neutral. verifyProviderWebhook must return a validated event with tenant and provider-account scope derived from trusted receiver configuration. persistReceiptAndOutbox must atomically commit the unique receipt and its pending work, returning the existing receipt on a duplicate. Workers recover unpublished outbox records after a crash. Use each provider's current signature library or documented verification procedure. For example, ElevenLabs requires the raw request body and its signature header for HMAC validation.
How Should Evaluation State and Replay Work?
Keep call ingestion and evaluation as separate state machines.
| State | Owner | Terminal? | Retry or recovery action |
|---|---|---|---|
RECEIVED | Webhook receiver | No | Normalizer scans durable receipts |
NORMALIZED | Provider adapter | No | Rebuild canonical event from receipt |
WAITING_FOR_ARTIFACTS | Artifact resolver | No | Refetch or wait for later provider event |
READY_FOR_EVALUATION | Evaluation router | No | Publish versioned evaluation job |
EVALUATING | Evaluator | No | Lease expires and job retries |
EVALUATED | Evaluator | Yes for one input and suite version | Publish immutable result |
QUARANTINED | Operations | No | Fix mapping, privacy, or contract issue and replay |
An evaluation result should be append-only:
{ "event_name": "voice.call.evaluated.v1", "evaluation_id": "eval_01J5D95CFM", "provider_call_id": "call_7f31", "input_version": "sha256:8d76...", "suite_id": "production_scheduler", "suite_version": "2026-08-14.3", "evaluator_version": "grader_17", "status": "completed", "scores": { "task_completion": 0.86, "tool_correctness": 1, "policy_adherence": 1 }, "failed_guardrail_ids": ["confirmed_identity_before_booking"], "evidence_ref": "artifact://evaluation-evidence/eval_01J5D95CFM", "evaluated_at": "2026-08-14T16:43:02Z"}Replays with unchanged inputs and versions resolve to the existing attempt. Create a new attempt when any of these changes:
- normalized input version
- artifact version
- evaluation suite version
- evaluator or model version
An intentional rerun with unchanged inputs and versions also creates a new attempt when it records an explicit replay reason and a stable replay request ID. Retries of that same request reuse the new attempt. A policy-triggered rerun follows the same rule and records the policy as its reason.
Do not overwrite the old score. The old result explains what monitoring showed at that time. The new result explains what the current evaluator would decide.
When a confirmed failure is worth keeping, move it into the failed production call regression workflow. When it needs human judgment first, send it through the call review triage runbook.
What Data Should Stay Out of the Event?
The canonical event should be useful to routers, schedulers, and monitors without becoming a second transcript database.
| Data class | Put in canonical event? | Better location | Why |
|---|---|---|---|
| Provider call ID | Yes | Event and receipt | Required join key |
| Agent, version, environment | Yes | Event | Required cohort dimensions |
| Transcript text | No | Controlled artifact storage | Sensitive and high volume |
| Recording bytes or public URL | No | Controlled artifact storage | Sensitive, large, and often short-lived |
| Raw tool arguments and results | No | Controlled tool evidence store | May contain account, payment, or health data |
| Caller phone number or email | No | Provider vault or governed identity store | Not needed for evaluation routing |
| Artifact hash and governed reference | Yes | Event | Supports integrity and controlled access |
| Evaluation scores and guardrail IDs | Result event only | Evaluation store and monitoring | Avoid mixing input and output lifecycle |
Use the PII redaction architecture guide before sending artifacts to evaluators. Then apply the voice agent log retention checklist separately to receipts, transcripts, recordings, tool traces, and evaluation results. One retention window rarely fits all five.
What Can This Design Still Not Prove?
This design closes delivery and reproducibility gaps. It does not solve every quality problem.
A valid webhook is not a complete call. The provider may omit an artifact, redact data, or send a later correction. Your readiness rule still needs to detect missing inputs.
Idempotency is not ordering. Deduplication prevents repeated work. It does not guarantee call_ended, enrichment, transfer, and analysis events arrive in the order you expected.
A reproducible evaluation can still be wrong. Versioning proves which evaluator made the decision. Human calibration and clear evaluation metrics still determine whether the score is trustworthy.
Artifact references can expire. Resolve short-lived provider URLs into governed storage before the provider's retention window closes, but do that asynchronously after durable acceptance.
We have not found one universal artifact-readiness rule that works across every provider and evaluation suite. A safety evaluator may need only a transcript. Audio-quality evaluation needs the recording. A tool workflow evaluator needs tool traces and final side effects. Make readiness explicit per suite.
Voice Agent Post-Call Webhook Launch Checklist
| Check | Owner | Proof before launch |
|---|---|---|
| Signature and timestamp verification uses the raw body | Security or platform | Tampered-body test returns 401 |
| Receiver rejects oversized or malformed envelopes | Platform | Boundary tests cover size and schema failures |
| Success is returned only after durable receipt | Platform | Crash test after acknowledgement can replay the receipt |
| Delivery key distinguishes event type and payload version | Platform | Duplicate and enrichment fixtures behave differently |
| Storage-to-queue handoff has no gap | Platform | Outbox or receipt scanner recovers an interrupted publish |
| Provider adapter emits one versioned canonical shape | Integrations | Fixture matrix covers every configured provider event |
| Evaluation readiness is defined per suite | Evaluation | Missing-artifact cases enter WAITING_FOR_ARTIFACTS |
| Results include input, suite, and evaluator versions | Evaluation | Identical retries reuse an attempt; changed versions or an explicitly requested replay create a new immutable attempt |
| Sensitive artifacts stay behind governed references | Security and data | Event/log inspection contains no transcript, audio, or caller PII |
| Quarantine and replay paths are exercised | Operations | One malformed mapping is fixed and replayed without provider resend |
| Monitoring covers age and count at every lifecycle state | SRE or platform | Stuck-state alert reaches the owning team |
| Confirmed failures can enter regression coverage | QA or engineering | One reviewed failure links to a sanitized test case |
Start with one provider, one evaluation suite, and 17 fixture events: valid completion, duplicate delivery, bad signature, stale timestamp, oversized body, malformed JSON, missing call ID, completion before transcript, late analysis, corrected transcript, missing recording, expired artifact, evaluator timeout, result retry, quarantined mapping, manual replay, and provider resend.
That set is more useful than a happy-path demo. It proves the boundaries that usually break after the first production retry.
