A voice agent can stay connected, respond in a calm voice, and still fail badly: "Tool execution failed with status 500," "rate limit exceeded," or part of a system prompt reaches the caller. Uptime looks healthy. The transcript exists. The unsafe part is the sentence the system chose to say out loud.
If your agent has no external tools, no sensitive workflows, and low call volume, a manual failure pass may be enough. Once the agent books appointments, changes accounts, handles payments, or runs at production volume, internal error leakage needs its own release gate.
TL;DR: Detect voice agent internal error leakage by testing two channels separately:
- Caller channel: audio, captions, and transcript must contain only an approved recovery message and next action.
- Operator channel: logs and traces should retain the error class, component, correlation ID, timing, and retry outcome without exposing unnecessary secrets.
- Failure injection: force telephony, STT, LLM, tool, TTS, and orchestration failures, then verify both channels.
- Release gate: any stack trace, provider payload, credential fragment, system instruction, raw tool error, or internal identifier in caller-visible output blocks the release.
The checklist applies the error-handling and provider documentation cited below to caller audio and transcripts. Phrase patterns are starter detectors, not a complete security boundary; pair them with failure injection and semantic review.
Last Updated: September 25, 2026
Related Guides:
- Voice Agent Troubleshooting - isolate failures across telephony, ASR, LLM, tools, and TTS
- Voice Agent Incident Response Runbook - investigate and contain production failures
- Voice Agent Tool Call Contract Template - define safe tool requests, results, retries, and side effects
- Voice Agent Call Evidence Export - package audio, transcripts, traces, and review outcomes
- Voice Agent Production Readiness Checklist - put evidence behind the final launch decision
- Voice Agent Security Review Questions - review access, retention, deployment, and vendor boundaries
- OpenTelemetry for Voice Agents - join caller behavior to internal component spans
- Voice Agent Dead Air Detection - diagnose silence when the agent says nothing instead
What Is Voice Agent Internal Error Leakage?
Voice agent internal error leakage happens when diagnostic detail meant for developers or operators reaches caller-visible speech, captions, or transcripts. The leaked detail may be a stack trace, provider error, tool payload, internal policy instruction, database message, secret-like value, or identifier that gives the caller no useful next step.
Caller-safe error handling: tell the caller what happened at the level needed to recover, what they can do next, and whether the task completed. Keep sanitized component, exception, payload, and stack details in the operator channel under access controls. Never log raw credentials or secrets, and restrict prompt detail according to policy.
OWASP's error-handling guidance recommends that user-visible errors avoid system details, identifiers, account information, debugging data, and stack traces while internal logs retain enough information for support and incident response. Voice changes the interface, not the boundary. The error page is now a sentence spoken into someone's ear.
Do not confuse leakage with every mention of failure. "I couldn't complete that transfer" is useful. "The transfer API returned ECONNRESET after three retries" is not.
| Caller hears | Classification | Why | Safer behavior |
|---|---|---|---|
| "I couldn't complete the transfer. I can try again or connect you to support." | Safe recovery | States outcome and next action | Keep; log diagnostic detail internally |
| "The CRM returned 401 unauthorized." | Internal diagnostic | Exposes component and status detail | State the verified update outcome without exposing the raw error |
"The create_booking tool timed out." | Tool-contract leakage | Exposes implementation language | Verify the booking outcome; if unknown, say so and check status before retrying |
| "My system prompt says I cannot do that." | Instruction leakage | Reveals internal control text | State the allowed boundary in customer language |
| "Database connection failed for tenant 8472." | Infrastructure and identifier leakage | Exposes system and internal identifier | Give a generic retry path; retain the ID in logs |
| "Something went wrong." | Usually too vague | Safe from detail leakage but gives no recovery | Add task outcome and next action |
What Should You Scan For?
A keyword list is useful as a tripwire. It is not the test.
Start with these message classes:
| Message class | Starter patterns | Semantic question | Release action |
|---|---|---|---|
| Stack traces | Traceback, Exception, at line, file paths | Did implementation detail reach speech? | Block |
| HTTP/provider errors | 401, 403, 429, 500, rate limit, provider names | Did the agent narrate a dependency failure? | Block unless explicitly customer-safe |
| Tool/runtime detail | tool names, function names, JSON fragments, retry counts | Did internal orchestration become conversation copy? | Block |
| Secrets and credentials | api_key, bearer tokens, connection strings | Could the output contain a secret or secret-shaped value? | Block and escalate |
| Internal identifiers | tenant IDs, trace IDs, room IDs, job IDs | Is the identifier meaningful or necessary to the caller? | Remove or replace with approved case number |
| Prompt/policy text | "system prompt," hidden instructions, policy blocks | Did the agent expose control-plane instructions? | Block |
| Database/infrastructure | SQL errors, hostnames, regions, queue names | Does the message reveal implementation topology? | Block |
| Vague fallback | "error," "problem," "try later" with no task outcome | Does the caller know whether the action happened? | Rewrite |
Why semantic review? A phrase detector can catch ECONNRESET. It may miss "the scheduling service rejected your account token," which leaks the same class of information in polished prose. It can also overfire on harmless text such as a caller reading an error message from another product.
A growing regex dictionary cannot establish whether the agent preserved the right task outcome when the failure happened. A safe sentence that incorrectly claims "your appointment was not booked" after the write succeeded is still a serious defect.
How Do You Inject the Right Failures?
Test the failure boundary, not only the happy path. Each row below needs a controlled trigger, a caller-visible assertion, and an operator-evidence assertion.
| Boundary | Failure to inject | Caller-visible assertion | Operator assertion | Recovery assertion |
|---|---|---|---|---|
| Telephony/webhook | timeout, invalid response, unreachable primary handler | Approved fallback message; no URL or error code spoken | error code, failed URL, call ID retained securely | fallback route or handoff executes |
| STT | provider disconnect, empty final transcript, quota failure | agent asks for repetition or moves to approved fallback | provider, error class, timing, fallback attempt captured | no invented transcript or action |
| LLM | timeout, 429, malformed output, mid-stream failure | no model/provider detail or partial diagnostic phrase | request ID, model route, retry/fallback state recorded | partial speech is handled intentionally |
| Tool/API | 4xx, 5xx, timeout, invalid schema, duplicate response | task outcome is explicit; raw payload stays hidden | tool name, sanitized arguments, result class, idempotency key recorded | retry cannot duplicate the action |
| TTS | synthesis error, empty audio, mid-utterance disconnect | alternate voice, brief recovery, or clean handoff | provider and playout state captured | no loop or contradictory replay |
| Orchestration | state transition error, cancelled job, stale context | no state-machine or job detail spoken | transition, state, correlation ID, decision captured | conversation ends in a valid state |
Twilio's voice failover guidance separates caller fallback behavior from the ErrorCode and ErrorUrl details sent to the fallback handler. That is the contract to copy: callers get a useful recovery path; operators get diagnostic context.
LiveKit's Agent Fallback Adapter documentation describes guards once output has started: TTS does not switch providers after audio reaches the speaker, and LLM fallback raises the error after text or tool calls have streamed unless configured to permit that retry. Test both pre-output and mid-output failures against your deployed SDK and settings, including the caller message and any tool side effects.
Define Two Error Contracts
Write the caller contract and operator contract side by side before implementing fallback copy.
| Contract field | Caller channel | Operator channel |
|---|---|---|
| Failure description | task-level, plain language | component and normalized error class |
| Task outcome | completed, not completed, or unknown | authoritative persisted state and verification result |
| Next action | retry, alternate path, human handoff, callback | retry policy, owner, escalation, runbook |
| Identifier | approved case/reference number only | trace ID, call ID, provider request ID, tool execution ID |
| Data detail | minimum needed for caller confirmation | sanitized inputs and outputs under access controls |
| Timing | short expectation when known | component timings, deadlines, retry count |
| Secrets/prompts | never | never log raw secrets; restrict prompt detail by policy |
A safe fallback is not merely generic. It must be specific about the caller's task outcome while staying generic about the system's implementation. "I couldn't verify whether the payment completed" is safer than both a raw gateway error and an unsupported claim that the payment failed.
A free DIY version works: keep an approved fallback-copy file in source control, add each failure state as a test fixture, and compare captured transcripts against both forbidden patterns and required outcome phrases. Hamming is useful when that matrix has to run across many agents, voices, providers, languages, and production regressions.
Save a Reviewable Leakage Event
Do not emit only error_leakage=true. Save enough evidence to reproduce the boundary without copying the secret or raw private value into another system.
{ "eventType": "voice_agent.caller_error_leakage", "callId": "call_2026_08_18_1042", "turnIndex": 7, "channel": "tts_audio_and_transcript", "messageClass": "tool_runtime_detail", "matchedEvidence": "sanitized:create_booking timeout", "failureBoundary": "calendar_tool", "taskOutcome": "unknown", "callerRecovery": "human_handoff_offered", "severity": "high", "traceId": "4bf92f3577b34da6a3ce929d0e0e4736", "audioOffsetMs": { "start": 48120, "end": 50740 }, "reviewStatus": "needs_human_confirmation", "regressionFixtureId": null}The matchedEvidence field must be sanitized. Store the exact audio and transcript in the governed call evidence packet, not in a broad alert payload. Use OpenTelemetry voice-agent tracing to join the caller turn to the internal error without making the caller-facing transcript carry operator detail.
How Should You Evaluate Audio and Transcripts?
Evaluate both. A transcript-only test can miss audio inserted outside the main transcript path, including provider fallback prompts, telephony error recordings, or partially streamed TTS. An audio-only scan can mishear code-like strings or lose the exact task outcome.
Use this rubric:
| Check | Pass | Fail | Needs review |
|---|---|---|---|
| Diagnostic detail | no internal component, payload, prompt, secret, or stack detail | any prohibited detail reaches caller | phrase could be caller-supplied context |
| Task outcome | explicitly completed, not completed, or unknown | agent invents or contradicts outcome | backend evidence is incomplete |
| Recovery | retry, alternate path, handoff, or callback is valid | dead end, loop, or unsafe retry | recovery depends on policy/availability |
| Audio/transcript agreement | both channels contain the same safe meaning | audio contains extra or conflicting detail | ASR uncertainty around key phrase |
| Operator diagnostics | sanitized cause and correlation data exist | no evidence to diagnose or reproduce | evidence exists but owner is unclear |
False positives matter. A caller may say, "The website showed error 500," and the agent may repeat it while confirming the problem. Label the speaker, audio offsets, and conversational source before classifying leakage. Do not silently add every flagged call to a training set.
When Should Error Leakage Alert or Block a Release?
Use severity based on what crossed the boundary, not how dramatic the phrase sounds.
| Condition | Severity | Immediate action | Release decision |
|---|---|---|---|
| Secret, credential, private prompt, or sensitive identifier spoken | Critical | stop affected flow; security review | block |
| Stack trace, raw payload, database detail, or internal topology spoken | High | disable unsafe fallback; inspect similar calls | block |
| Provider/tool name and raw status detail spoken | High | replace fallback copy; add regression | block |
| Safe wording but wrong task outcome | High | verify persisted state; prevent duplicate action | block |
| Vague fallback with valid handoff | Medium | improve copy and evidence | owner decision before release |
| Forbidden phrase detector with caller-originated text | Review | confirm speaker/source | do not block until classified |
Any confirmed leakage should become the smallest reproducible regression test. The failed production call regression runbook explains how to preserve the failure without turning a private production call into an unsafe fixture.
Pre-Release Error Leakage Checklist
- Define approved caller messages for every known failure state.
- Define whether each action outcome can be
completed,not_completed, orunknown. - Inject failures at telephony, STT, LLM, tool, TTS, and orchestration boundaries.
- Include pre-output and mid-output failures for streaming components.
- Evaluate captured audio and the canonical transcript independently.
- Scan for stack traces, raw payloads, provider errors, internal identifiers, prompts, and secret-shaped values.
- Run semantic checks for polished paraphrases that keyword rules miss.
- Verify operator logs retain a sanitized cause, correlation IDs, timing, owner, and retry result.
- Verify retries and fallbacks cannot duplicate side effects.
- Route ambiguous matches to a reviewer with speaker and audio-offset context.
- Add confirmed failures to the regression suite.
- Block release until every critical/high finding has a verified fix and clean rerun.
This checklist fits inside the broader production readiness gate. It is deliberately narrower. A clean error-leakage run does not prove the agent is accurate, compliant, fast, or useful. It shows that the tested failure cases kept internal diagnostics out of caller output; untested paths still need coverage.
What This Checklist Cannot Prove
Three limits are worth keeping visible:
- No dictionary is complete. New providers, tools, and code paths create new phrases. Failure injection and semantic review remain necessary.
- Safe wording can hide an unsafe action. The agent may apologize correctly while a tool wrote duplicate state. Pair this with the tool call contract template.
- A clean transcript can disagree with audio. Test the recording or synthesized output path, especially for telephony and mid-stream TTS failures.
The core rule is simple: operators need detail; callers need truth and a next step. Mixing those channels makes debugging easier for a minute and the product less safe for everyone who calls after that.
