Most automated voice agent reports have the same failure mode: they look precise before they are trustworthy. A polished summary says intent accuracy fell 2.1 points, p95 latency rose 41%, and one model version caused the regression. Then nobody can find the denominator, baseline window, or calls behind the conclusion.
We call this precision theater. The report has numbers, but the numbers cannot survive the first engineering question.
If your pilot handles 30 calls a week and one person can review every call, use a spreadsheet. This template is for production teams that need a scheduled voice agent QA report to connect intent-recognition errors, latency spikes, evidence, owners, and regression work without inventing a root cause.
An automated voice agent report is a decision artifact, not a dashboard export. It should state what changed, compared with which baseline, for which cohort, using how many evaluated calls or turns, and what evidence supports the next action.
TL;DR: A useful automated voice agent report has five contracts:
- Scope: reporting window, agent version, environment, cohort, and evaluated denominator.
- Quality: intent errors grouped by expected intent, predicted intent, and failure type.
- Performance: p50, p95, and p99 latency by pipeline stage, plus baseline deltas.
- Evidence: redacted call or trace links behind every release-blocking claim.
- Action: severity, owner, next check, and regression-test status.
Methodology Note: The template and evidence rules in this guide are based on Hamming's analysis of production voice agent calls across 10K+ voice agents (2025-2026). Hamming's platform has 10M+ mins protected. We've tested agents built on LiveKit, Pipecat, ElevenLabs, Retell, Vapi, and custom-built solutions.Public vendor documentation was used to verify common event, latency, percentile, and drilldown patterns. The sample values below are synthetic and must be replaced with your own measurements.
Last Updated: September 2026
Related Guides:
- Voice Agent Daily Failure Report Template - human triage handoff for yesterday's failed calls
- Voice Agent Dashboard Template - charts and filters that feed scheduled reports
- Voice Agent Evaluation Metrics - definitions, denominators, and benchmark caveats
- Intent Recognition Testing at Scale - expected-intent labeling and confusion analysis
- OpenTelemetry for Voice Agents - trace and span model for stage attribution
- Voice Agent Observability and Tracing - end-to-end debugging across the audio pipeline
- Debugging Voice Agents - logs, missed intents, and error dashboards
- Voice Agent SLOs and Error Budgets - release and incident thresholds
What Should an Automated Voice Agent QA Report Contain?
An automated report should contain only fields that help a reviewer decide whether to investigate, roll back, change a workflow, or add a regression test. Put the decision summary first. Keep raw transcripts, audio, and traces behind access-controlled links.
| Report section | Required fields | Decision it supports |
|---|---|---|
| Scope and provenance | Window, environment, agent and prompt version, cohort filters, generated time | Are these results comparable? |
| Executive summary | Largest verified change, severity, decision needed | Does anyone need to act now? |
| Intent errors | Evaluated turns, incorrect turns, error rate, top confusions, sample evidence | Is quality regressing for a specific intent? |
| Latency | p50, p95, p99, stage, baseline, SLO, affected turn count | Is the tail getting worse, and where? |
| Correlation review | Shared cohort, time window, version, overlap count, hypothesis label | Should engineering investigate a relationship? |
| Action register | Owner, due date, next check, rollback or test status | What happens after the report? |
| Data-quality note | Missing events, skipped calls, evaluator coverage, privacy exclusions | How much confidence should readers place in it? |
The key correction is simple: observations and hypotheses belong in different columns. “p95 customer_lookup latency rose 56.6%” can be an observation. “The CRM lookup caused the intent regression” is a hypothesis until trace or experiment evidence supports it.
What Data Must Exist Before You Generate the Report?
Do not automate the prose first. Automate the evidence contract.
At minimum, each call or turn needs a stable correlation key, timestamps, version metadata, an outcome, and access-controlled evidence. Deepgram's session observability documentation shows the practical shape: capture a session identifier, ordered events, function calls, errors, turn boundaries, and stage latency. Its latency fields are optional, which is a useful reminder to report missing data rather than silently converting it to zero.
| Field | Level | Required? | Reporting use | Privacy rule |
|---|---|---|---|---|
call_id or session_id | Call | Yes | Evidence correlation and deduplication | Use an internal opaque ID |
agent_version and prompt_version | Call | Yes | Regression cohorts | Store identifiers, not prompt text |
environment and cohort metadata | Call | Yes | Separate production, staging, region, language, or workflow | Exclude direct customer identifiers |
expected_intent | Evaluated turn | Yes for intent accuracy | Ground truth | Record label source and reviewer state |
predicted_intent | Turn | Yes for intent accuracy | Confusion matrix | Avoid copying sensitive utterances into summary tables |
evaluation_status | Turn | Yes | Denominator and coverage | Distinguish evaluated, skipped, and unknown |
| Stage timestamps | Turn | Yes for latency | p50/p95/p99 and attribution | Keep raw content out of span attributes by default |
tool_name, status, and duration | Tool call | Recommended | Separate slow or failed tools | Redact arguments and results |
| Evidence URL | Call or trace | Recommended | Human verification | Enforce access controls and retention |
PolyAI's monitoring documentation follows the same operating pattern: use dashboards for trends, filter by version or cohort, then drill into conversation-level timing and function evidence. The report should preserve that path from aggregate to supporting call.
Data-quality rule: Never calculate an error rate with “all calls” in the denominator when only a subset was evaluated. Report
evaluated_turns,skipped_turns, andevaluation_coveragebeside the rate.
Copy the Automated Voice Agent Report Template
The values in this sample are deliberately labeled synthetic. Replace every bracketed field and delete any row your pipeline cannot support.
# Automated Voice Agent QA Report - [Agent] - [Window]> SAMPLE STATUS: Synthetic values. Replace before sending.## 1. Scope and Provenance- Environment: [production/staging]- Window: [start timestamp] to [end timestamp]- Agent version: [version]- Prompt version: [version]- Cohort filters: [language, region, workflow, customer segment]- Calls observed: [N]- Calls evaluated: [N]- Turns evaluated for intent: [N]- Evaluation coverage: [evaluated eligible turns / eligible turns]- Baseline window: [dates or release]- Generated at: [timestamp]## 2. Executive Decision Summary- Verified change: [metric, current value, baseline, delta, denominator]- Highest-risk cohort: [cohort and evidence count]- Root-cause status: [unknown / hypothesis / verified]- Decision needed: [none / investigate / rollback / incident / release block]- Owner and next check: [owner, action, due time]## 3. Intent-Recognition Errors| Expected intent | Predicted intent | Incorrect / evaluated | Error rate | Baseline | Delta | Evidence | Owner ||---|---|---:|---:|---:|---:|---|---|| cancel_order | change_order | 9 / 73 | 12.3% | 5.6% | +6.7 pts | [redacted calls] | [owner] || billing_question | payment | 7 / 109 | 6.4% | 6.1% | +0.3 pts | [redacted calls] | [owner] |## 4. Latency Spikes| Stage | p50 | p95 | p99 | Baseline p95 | Delta | Turns over SLO | Evidence ||---|---:|---:|---:|---:|---:|---:|---|| End to end | 842 ms | 2,310 ms | 4,980 ms | 1,740 ms | +32.8% | 41 / 1,247 | [trace filter] || Tool: customer_lookup | 188 ms | 1,190 ms | 2,840 ms | 760 ms | +56.6% | 19 / 214 | [trace filter] |## 5. Correlation Review| Observation | Shared cohort | Overlap | Confidence | Root-cause status | Next check ||---|---|---:|---|---|---|| cancel_order errors and tool latency rose in agent-v42 | agent-v42, cancel_order | 6 of 9 errors occurred during tool-SLO breaches | Medium | Hypothesis, not causation | Replay calls with fixed tool latency |## 6. Evidence Queue| Priority | Call / trace | Why review | Redaction state | Reviewer ||---|---|---|---|---|| P1 | [opaque link] | Wrong intent plus slow tool | Redacted | [name/team] |## 7. Actions| Severity | Action | Owner | Due | Rollback status | Regression-test status ||---|---|---|---|---|---|| P1 | Reproduce customer_lookup timeout path | [owner] | [time] | Decision pending | Missing |## 8. Data Quality and Privacy- Missing or delayed events: [count and affected fields]- Evaluator version and calibration date: [version/date]- Excluded calls: [rule and count]- Sensitive content included in summary: no- Evidence access and retention: [policy]This template is intentionally less “executive” than a slide. It forces a reviewer to see the denominator, baseline, evidence state, and uncertainty before seeing the recommendation.
How Should Intent-Recognition Errors Be Calculated?
Calculate intent error rate only across turns with a reviewable expected intent. The label can come from a human annotation, deterministic workflow outcome, or a calibrated evaluator, but the report must state which source it used.
Intent Error Rate = Incorrect Intent Turns / Evaluated Intent Turns x 100Evaluation Coverage = Evaluated Eligible Turns / Eligible Turns x 100Suppose an agent has 1,247 eligible turns, but only 936 have a trustworthy expected-intent label. If 71 of those evaluated turns are incorrect:
Intent Error Rate = 71 / 936 x 100 = 7.59%Evaluation Coverage = 936 / 1,247 x 100 = 75.06%Report both numbers. “7.59% error rate” sounds complete; “7.59% error rate at 75.06% evaluation coverage” tells the truth.
Segment the result before acting:
| Segment | Why it matters | First check |
|---|---|---|
| Expected intent | Finds one workflow hidden by the overall average | Confusion pairs and sample evidence |
| Agent or prompt version | Finds release regressions | Compare like-for-like traffic |
| Language or region | Finds uneven recognition quality | Audio, ASR, and label coverage |
| Tool path | Separates recognition from downstream action errors | Expected intent versus actual tool call |
| Call stage | Finds errors after context accumulates | Turn number, context, and prior tool state |
The intent recognition testing guide covers ground-truth and confusion analysis in more depth. One common mistake is counting “correct intent, wrong action” as an intent error. Keep it as a separate workflow or tool-call failure so the report points to the right owner.
How Should Latency Spikes Be Reported?
Report a latency spike as a baseline or service-objective breach, not as a universal number copied from another agent. Track p50 for the typical turn and p95/p99 for the tail. Retell's AI QA terminology documents percentile latency as a distinct metric; that distinction matters because averages hide the pauses callers remember.
Latency Delta % = (Current p95 - Baseline p95) / Baseline p95 x 100SLO Breach Rate = Turns Above SLO / Eligible Measured Turns x 100Use a table with an action column:
| Pattern | Report it as | Evidence needed | First action |
|---|---|---|---|
| p50 stable, p99 worse | Tail regression | Slowest traces by stage and version | Inspect queueing, retries, and outlier tools |
| All percentiles worse | Broad slowdown | Comparable cohort and stage timings | Check release, provider, region, and load |
| Tool p95 worse, end-to-end stable | Hidden dependency risk | Tool spans and acknowledgement behavior | Check timeouts before users feel it |
| Missing stage data | Instrumentation gap | Event coverage by SDK/version | Fix telemetry before naming a cause |
For stage attribution, connect the report to OpenTelemetry voice agent traces. OpenTelemetry's GenAI conventions evolve, so pin the convention and instrumentation versions in your pipeline instead of assuming every backend uses the same attribute names.
How Do You Correlate Intent Errors With Latency Without Claiming Causation?
Join the two signals by the same call, turn, version, cohort, and time window. Then label the result as an observation or hypothesis until a controlled replay, trace, or rollback comparison supports the cause.
Correlation rule: “Six of nine errors occurred during tool-latency breaches” is evidence. “Tool latency caused the nine intent errors” is a causal claim and needs a mechanism plus a confirming test.
Use this ladder:
- Co-occurrence: Did the signals happen in the same turns or calls?
- Concentration: Is overlap higher than the baseline cohort?
- Mechanism: Can a trace show timeout, fallback, lost context, or a delayed action changing behavior?
- Reproduction: Does the error appear when the suspected condition is injected?
- Counterfactual: Does the error disappear after the condition is removed or the version is rolled back?
This is where automated reports often overreach. They find two red lines on the same chart and write a root-cause paragraph. We used to think a more detailed narrative made the report more useful. In practice, the explicit “unknown / hypothesis / verified” field does more for engineering trust than another page of confident prose.
When the report finds a verified production miss, route it into the failed-call regression test runbook. A finding without a regression decision is likely to return.
How Often Should the Report Run, and Who Should Receive It?
Match cadence to the decision. Real-time alerts handle active service risk. Daily reports handle QA and ownership. Weekly reports handle product trends and release comparisons.
| Cadence | Audience | Include | Exclude |
|---|---|---|---|
| Real time | On-call engineering | SLO breach, active unsafe behavior, widespread tool failure | Long summaries and low-confidence hypotheses |
| Daily | QA, operations, engineering owner | Largest deltas, top error cohorts, evidence queue, actions | Every metric that did not change |
| Weekly | Product and engineering leads | Version trends, repeat clusters, regression coverage | Raw call detail |
| Monthly | Leadership | Outcome movement, risk, investment decisions | Component-level debugging unless it changes a decision |
Use Slack alerts for voice agent monitoring for active conditions, and use the daily failure report template when a human needs to triage yesterday's highest-risk failures. The automated report on this page supplies consistent evidence and trends; it should not replace incident handling.
Pre-Send Quality Checklist
- The scope names environment, window, agent version, prompt version, and cohort.
- Every rate includes its numerator, denominator, and evaluation coverage.
- Latency shows p50, p95, and p99 or explains why a percentile is missing.
- Every delta names a comparable baseline window or version.
- Sample values are labeled synthetic and removed from the real report.
- Observations, hypotheses, and verified causes are visibly different.
- Release-blocking findings link to redacted call or trace evidence.
- Raw transcripts, prompts, tool arguments, and customer identifiers are absent from the summary.
- Every action has an owner and next check.
- Every verified defect has an incident, rollback, monitoring, or regression-test decision.
Flaws but Not Dealbreakers
A template cannot pick your thresholds. A healthcare triage agent, restaurant booking agent, and internal IT assistant have different risk and latency budgets. Start from your service objective and comparable baseline, then revise after reviewing real calls.
Automated intent labels can agree with themselves. If the same model family generates and evaluates an intent, a clean report can hide shared blind spots. Calibrate against human-reviewed calls and expose evaluator coverage and version.
More reporting can create more noise. If the daily artifact repeats the dashboard without an owner or decision, turn it off. The point is not to send a report. The point is to make the next correct action obvious.
