Automated voice AI QA replaces blind manual call sampling with a system that scores every eligible voice-agent call, routes the highest-risk calls to humans, and keeps random review as a calibration control.
If you run 50 voice-agent calls a week, this template is too heavy. Listen to the calls, label the misses, and turn obvious failures into tests.
This is for contact centers and voice AI teams handling hundreds, thousands, or tens of thousands of calls where the old QA habit is still "pick a few recordings and hope they represent reality." That habit breaks faster with AI agents because prompts, tools, model versions, and routing policies can change daily.
Google Cloud's Quality AI documentation says manual review often covers less than 1% of conversation volume, while automated scoring can take all conversations into account. That does not mean humans disappear. It means humans should stop being the coverage layer.
TL;DR: Replace manual call sampling with automated voice AI QA in four moves: score every eligible call, reserve a defined share of review capacity for random calibration, send high-risk calls into a human review queue, and promote repeatable failures into regression tests.
The goal is not "100% automation." The goal is a QA system where every score has evidence, every human review has a reason, and every durable failure has a path into tests.
Methodology Note: This replacement template is based on Hamming's analysis of 4M+ production voice agent calls and QA review workflows across 10K+ voice agents (2025-2026). We've tested agents built on LiveKit, Pipecat, ElevenLabs, Retell, Vapi, and custom-built solutions.The template focuses on voice-agent QA programs where automated scoring, human calibration, production monitoring, and regression testing need to work as one loop.
Across Hamming's analysis of 4M+ production voice agent calls across 10K+ agents, we found that the most reliable QA programs do not ask humans to find every problem. They ask humans to calibrate the scoring system, handle judgment calls, and fix the failures automation surfaces.
"It is the repetitive stuff that makes Hamming worth it for us. Being able to do volume, because we have to test at scale," says Tosh Toida, QA Lead at Mia Labs. That is the practical handoff: automation handles coverage, while reviewers own calibration and judgment.
Last Updated: July 2026
Related Guides:
- Call Center QA Tools Comparison - decide which QA category you are actually buying
- Voice Agent Call Review Triage Runbook - rank selected calls by review value
- Voice Agent Call Evidence Export Runbook - package transcript, audio, traces, and review evidence
- LLM Grader Voice Agent Scoring Rubric - design the scoring rubric behind automated QA
- Failed Production Call Regression Tests - turn reviewed failures into durable tests
- Voice Agent CI/CD Testing - block bad releases before they reach callers
- Voice Agent Monitoring Platform Guide - connect production alerts to QA review
- Voice Agent Log Retention Compliance Checklist - keep review artifacts under retention and access rules
Manual Sampling Is Calibration, Not Coverage
Random sampling has a legitimate job. It reduces reviewer bias, gives supervisors a representative baseline, and helps teams check whether automated scoring is drifting.
It should not be the only way calls enter QA.
Amazon Connect documents random contact sampling for evaluation workflows, including per-agent samples, filters, draft evaluations, and auditability for sampling criteria. That is useful. It also shows the boundary: random sampling is a review mechanism, not a complete quality system for changing AI agents.
Use random samples for:
| Use Case | Keep Random Sampling? | Why |
|---|---|---|
| Reviewer calibration | Yes | Humans and automated scores need a shared baseline |
| Bias detection | Yes | Risk queues can overfocus on obvious failures |
| Compliance spot checks | Usually | Some policies require auditable sampling methods |
| Production issue discovery | No | High-risk calls should not wait to be picked randomly |
| Regression coverage | No | Durable failures should become tests, not anecdotes |
Replacement rule: keep random sampling as a calibration control. Do not use it as your main production QA coverage layer.
The mistake is swinging from "humans review a tiny sample" to "AI scores everything, so humans can leave." That is not QA. That is an uncalibrated classifier with a dashboard.
The Replacement Model: Score Everything, Review Selectively
Automated voice AI QA works when it separates three jobs:
- Coverage: score every eligible call against the rubric.
- Judgment: route the calls that need human decision-making.
- Learning: turn repeatable failures into product fixes, prompt changes, policy updates, or regression tests.
Google Cloud's Quality AI overview describes the shift from small manual review samples to automated conversation scoring across all conversations. Its sampling-rules documentation also describes filtered manual review based on attributes such as duration, CSAT, sentiment, quality score, language, transcript, tool, and other metadata. That is the model: broad automated analysis, selective human review.
Microsoft's Quality Evaluation Agent documentation uses a similar structure: criteria define quality, evaluation plans decide which interactions to evaluate, and evaluations produce scores, summaries, and recommended actions. For AI voice agents, add one more requirement: every score must connect to voice-specific evidence.
The replacement model looks like this:
| Layer | What It Does | Owner | Output |
|---|---|---|---|
| Automated scoring | Scores all eligible calls against rubric criteria | QA + engineering | Scores, labels, confidence, failure reasons |
| Risk routing | Selects calls that need human review | QA ops | P0/P1/P2 review queues |
| Calibration sample | Random or stratified sample for scoring drift | QA leads | Agreement rate, false positives, false negatives |
| Evidence packet | Joins transcript, audio, trace, tool evidence, and score | Engineering | Review-ready packet |
| Human review | Resolves judgment calls and labels outcomes | QA, compliance, product, engineering | Closed outcome and owner |
| Regression promotion | Turns durable failures into tests | Engineering + QA | Repeatable test case |
This is where production call review triage fits. Automated QA selects and scores. Triage decides which scored calls deserve human attention first.
Automated QA Sampling Replacement Template
Use this as the starter operating template.
| Stream | Replace This Manual Habit | Automated QA Version | Human Control That Stays |
|---|---|---|---|
| Coverage | Review 3-5 random calls per agent each week | Score every eligible voice-agent call | Weekly calibration sample |
| Selection | Supervisor picks calls manually | Queue by severity, confidence, novelty, and cohort impact | Reviewer override with reason |
| Scoring | Human fills a scorecard after listening | Rubric scores transcript, audio, tool results, latency, and outcomes | Dispute and calibration workflow |
| Evidence | Reviewer hunts for recording and transcript | Evidence packet links call ID, transcript, audio, trace, tool result, score, and evaluator version | Role-gated raw access |
| Follow-up | Notes live in QA tool or spreadsheet | Closed outcome taxonomy routes work to owner | Human final decision |
| Regression | Bad calls become anecdotes | Repeatable failures become regression tests | QA approves promotion criteria |
Start with six score families:
| Score Family | Sample Signals | When to Send to Human Review |
|---|---|---|
| Task completion | Goal achieved, fallback, abandonment, transfer | Task failed, caller repeated same request, or outcome is ambiguous |
| Compliance and policy | Required disclosure, refusal boundary, consent, regulated language | Missing disclosure, unsafe advice, or policy conflict |
| Tool-call correctness | Tool selected, arguments, order, side effect, retry | Tool failed, wrong argument, duplicate write, or mismatch with caller request |
| Conversation quality | interruption, silence, sentiment shift, escalation, repetition | Caller frustration rises or agent cannot recover |
| Latency and audio | turn latency, dead air, ASR confidence, TTS playback | Threshold breach affects task outcome |
| Regression risk | new prompt/model/tool version, cohort decline, repeated failure | Failure is repeatable or tied to a recent change |
The LLM grader rubric covers how to write criteria that a model can score consistently. The call evidence export runbook covers the packet reviewers need after a call enters the queue.
How to Migrate in 30 Days
Do not turn off manual QA on day one. Run the old and new systems side by side long enough to learn where the automated score is trustworthy.
| Phase | Days | What to Do | Exit Criteria |
|---|---|---|---|
| Baseline | 1-5 | Measure current manual sample volume, review time, failure labels, and follow-up actions | You know what manual QA actually catches |
| Rubric build | 6-10 | Convert the old scorecard into voice-agent criteria with evidence requirements | Each criterion has pass/fail samples |
| Shadow scoring | 11-17 | Score production calls automatically without changing human workflow | Automated labels can be compared to human reviews |
| Queue pilot | 18-24 | Route P0/P1 scored calls into a small review queue | Reviewers trust the packet and outcome taxonomy |
| Cutover | 25-30 | Move random sampling to calibration and use automated QA for coverage | Weekly QA review uses automated coverage plus calibration sample |
During shadow scoring, track disagreement instead of hiding it.
| Calibration Metric | What It Tells You | Fix |
|---|---|---|
| Agreement rate | How often automated and human labels match | Rewrite criteria with sample cases |
| False positives | Calls automation marks bad but humans accept | Add confidence thresholds or evidence requirements |
| False negatives | Calls humans reject but automation passes | Add missing criteria or training cases |
| Reviewer override rate | How often humans change the decision | Inspect ambiguous rubric language |
| Criterion drift | Which score family degrades over time | Recalibrate that family weekly |
I used to think the hard part was getting the model to score calls. The harder part is making reviewers trust the score enough to stop duplicating the old manual process.
Keep Humans on the Hard Decisions
Automated QA should reduce manual listening, not remove accountability.
Keep humans in the loop for:
- Compliance judgment: whether a disclosure, consent statement, or regulated phrase was acceptable in context.
- Customer harm: whether the caller experienced financial, healthcare, safety, or account-impacting risk.
- Ambiguous intent: when transcript text, audio, and tool evidence disagree.
- Policy changes: whether the rubric itself needs to change.
- Regression promotion: whether a production failure deserves a permanent test.
Microsoft's Quality Evaluation Agent documentation includes explicit cautions around monitoring, recording, storing communications, user notification, consent, and use of evaluation results. Treat that as a reminder: automating QA does not remove legal, compliance, privacy, or employment-related obligations.
For regulated scripts, pair this template with regulatory script adherence testing. For retention and access control, use the log retention compliance checklist before expanding raw audio access.
What Evidence Each Score Needs
Scores without evidence create arguments. A QA reviewer should be able to answer "why did this call fail?" without opening five tools.
Minimum evidence packet:
| Field | Required? | Why |
|---|---|---|
| Canonical call ID | Yes | Joins provider IDs, transcript, audio, trace, and evaluation |
| Selection reason | Yes | Explains why the call entered review |
| Agent version | Yes | Connects behavior to prompt, model, tool, or release |
| Evaluator version | Yes | Makes score changes auditable |
| Transcript span | Yes | Shows the precise turn that triggered the score |
| Audio pointer | Usually | Captures silence, interruption, background noise, tone, and ASR ambiguity |
| Trace ID | For technical failures | Links ASR, LLM, tools, TTS, storage, and evaluator spans |
| Tool-call summary | For workflow agents | Shows selected tool, arguments, result, retry, and side effect |
| Confidence and rationale | Yes | Shows whether the score is strong enough for automation |
| Human outcome | Yes | Closes the loop with no_issue, prompt_bug, tool_bug, policy_risk, or regression_candidate |
For production monitoring, connect this evidence model to the voice agent monitoring platform guide. For release quality, connect it to voice agent CI/CD testing.
Evidence rule: if a score cannot point to the transcript span, audio or trace context, evaluator version, and review outcome, it is not ready to replace manual QA judgment.
What Not to Automate
Keep a few decisions out of the automation path until the system has earned trust. These are the calls where a false positive wastes time, but a false negative can create real risk.
| Keep Human-Led | Why | Safer Pattern |
|---|---|---|
| Employment or compensation decisions | QA models can encode policy, transcription, and selection bias | Use human-led HR processes and legal review |
| High-severity compliance decisions | Context and jurisdiction matter | Route to compliance queue |
| One-off customer escalations | Account context may be missing | Assign support or customer owner |
| New rubric criteria | The model has no calibration history | Shadow score before enforcement |
| Raw audio access decisions | Privacy and consent rules vary | Use role-gated access and redacted packets |
This is the honest limitation: automated voice AI QA is only as good as its rubric, evidence joins, and calibration loop. If the transcript is wrong, the tool evidence is missing, or the policy criterion is vague, automation will scale the wrong decision with a straight face.
That does not make automation useless. It means the migration plan matters.
Weekly Operating Review
Once the system is live, run a weekly QA operating review.
| Agenda Item | Question | Output |
|---|---|---|
| Coverage | What percentage of eligible calls were scored? | Coverage trend |
| Calibration | Where did humans and automation disagree? | Rubric fixes |
| P0/P1 queue | Which failures affected customers, compliance, or money movement? | Owners and due dates |
| Cohorts | Which agent, language, queue, region, or version degraded? | Segment investigation |
| Regression candidates | Which failures should never recur? | Test cases |
| False positives | Which alerts wasted reviewer time? | Threshold changes |
| Privacy | Did packets expose more raw data than needed? | Access or redaction fix |
Pair this with post-call analytics metrics so the operating review has stable definitions. The review should produce work, not just a dashboard screenshot.
The practical loop is:
score all eligible calls -> route high-risk calls -> calibrate with random samples -> review evidence packets -> assign closed outcomes -> promote durable failures into tests -> measure whether the failure cluster shrinksFinal Checklist
Before you replace manual call sampling with automated voice AI QA, verify:
- The automated rubric has explicit pass/fail samples.
- Random sampling remains as a calibration sample.
- P0 failures bypass normal queue caps.
- Every reviewed call has a selection reason.
- Every score links to transcript, audio or trace context, evaluator version, and outcome.
- Humans can dispute or override automated scores with a reason.
- Compliance, consent, recording, retention, and raw-audio access rules are documented.
- Weekly calibration tracks agreement rate, false positives, false negatives, and criterion drift.
- Repeatable failures can become regression tests.
- The QA review ends with an owner, not a note.
If you keep only one thing, keep this: random sampling should not be replaced by blind automation. It should be replaced by full scoring, selective human review, and calibration. That is the difference between saying "we analyzed every call" and proving the voice agent got better.
