Voice agent tool call testing should start with a contract, not a transcript. That remains true when your QA system cannot see internal execution traces.
A tool call is where a voice agent stops only talking and starts touching systems: calendars, CRMs, ticket queues, payment processors, identity services, order systems, message senders, transfer targets, or internal APIs. If the contract is loose, the call can sound correct while the backend state is wrong.
Tool-call contract: a reviewable test definition that says which tool may run, under which preconditions, with which arguments, what result means success, what side effects are allowed, how retries are deduped, and what evidence must be saved.
Use this template when a voice agent can create, update, route, transfer, send, cancel, refund, or schedule anything. If your agent only answers static FAQs, a lighter conversation test is usually enough.
TL;DR: A passing voice-agent tool test should prove 4 things:
- The agent called the right tool at the right moment.
- The arguments were valid and supported by caller context.
- The downstream system ended in the expected state.
- Retries, timeouts, and cleanup could not create duplicate or unsafe side effects.
When traces must stay private, export those four verdicts in a redacted evidence envelope instead of granting access to the tracing backend.
Scope and sources: This is a proposed release-safety contract with synthetic YAML and JSON examples, not a production benchmark or a provider API schema. The cited vendor documentation explains tool execution; W3C and OpenTelemetry document correlation and telemetry conventions. Your team must implement the runner and evidence adapter, choose its policy, and validate it against the actual tool path.
Last Updated: September 2026
Related Guides:
- LiveKit RPC and UI Synchronization Testing - verify that client RPC results and rendered session state agree
- Voice Agent Workflow Testing Runbook - broader state-transition and side-effect testing
- Voice Agent Sandbox Testing - run tool calls safely against fixtures and sandboxes
- Voice Agent Structured Output Validation - prove fields and tool arguments match caller evidence
- Voice Agent Tests as Code - store test contracts in Git
- Voice Agent Call Evidence Export - package transcript, trace, and tool evidence for review
- Voice Agent Transcript Search Schema - make call evidence searchable
- Voice Agent Caller Identity Testing - decide which caller identity fields are trusted
- Voice Agent CI/CD Testing - connect blocking tests to release gates
- OpenTelemetry for Voice Agents - trace ASR, LLM, tool, and TTS steps together
- IVR Log Correlation Runbook - connect call IDs, routing events, and tool traces
Schema-Valid Is Not Contract-Valid
Tool schemas matter. They are just not the whole test.
OpenAI's function-calling documentation describes function tools as tools defined by JSON schema that connect a model to data and actions in your application. Anthropic's tool-use documentation describes tool definitions with a name, description, and input schema. Google Cloud's function-calling reference describes function declarations that let the model generate structured function names and arguments, while the application invokes the real external system.
That last distinction is the testing problem. The model suggests or emits a tool call. Your application decides what happens next.
| Test Layer | What It Proves | What It Does Not Prove |
|---|---|---|
| JSON parses | The tool payload can be read. | Required fields, enums, and business rules are correct. |
| Schema validates | The payload shape matches the declared schema. | The values came from the caller or trusted context. |
| Tool request is allowed | The agent chose an approved tool under known preconditions. | The downstream system changed correctly. |
| Tool result is handled | The agent saw success, failure, timeout, or partial state. | The spoken response matches durable state. |
| Side effect is verified | The calendar event, CRM update, ticket, transfer, or message exists as expected. | The test is replay-safe or production-safe. |
Contract-valid tool call: the tool name, arguments, ordering, result handling, side effect, idempotency behavior, and saved evidence all match the test contract.
For workflow agents, schema validity is the starting line. Contract validity is the finish line.
Copyable Tool-Call Contract
Use this as a starting YAML shape. Keep it short enough for code review.
version: 1id: appointment_booking_create_event_contractowner: voice-platformrisk: blockingagent_ref: environment: staging agent_slug: scheduling-agent prompt_version: pr-482tool_under_test: name: create_calendar_event schema_version: calendar_event_v3 dependency_mode: sandbox allowed_call_paths: - inbound_phone - web_voicepreconditions: caller_identity: source: verified_phone_match fixture_id: caller_fixture_104 workflow_state: required_state: slot_offered forbidden_states: - caller_unknown - payment_pending upstream_evidence: required_tools: - lookup_caller_identity - check_availability required_caller_confirmation: trueargument_policy: required_fields: - caller_id - slot_id - timezone - idempotency_key field_sources: caller_id: trusted_identity_lookup slot_id: check_availability_result timezone: caller_profile_or_confirmed_locale idempotency_key: run_id + tool_name + caller_id + slot_id forbidden_fields: - raw_phone_number - unredacted_payment_data - production_calendar_idside_effect_policy: allowed_side_effects: - one_sandbox_calendar_event_created forbidden_side_effects: - production_calendar_write - duplicate_event - live_sms_send exactly_once_required: true cleanup: method: delete_by_run_id verify_zero_records_after_cleanup: trueexpected_result: tool_status: created durable_state: calendar_event_count_for_run: 1 attendee_matches_fixture: true timezone_matches_request: true agent_response: must_confirm_only_after_tool_success: true must_include_local_time: true must_not_claim_booking_after_timeout: trueevidence: retain: - call_id - run_id - transcript - tool_request - tool_response - trace_handle - final_state_query - guardrail_results - cleanup_result redact: - phone_number - email - payment_datareplay_policy: same_run_id_second_execution: no_duplicate_side_effect timeout_retry: same_idempotency_key cleanup_failure: fail_testThis belongs next to your tests-as-code definitions. A reviewer should be able to answer 3 questions before the test runs:
- What is the agent allowed to touch?
- What would make the tool call unsafe?
- What evidence proves the caller outcome and backend state agree?
Contract Fields Explained
Treat the contract as a pre-flight checklist for one tool or one tool family.
| Field | Why It Exists | Common Failure |
|---|---|---|
risk | Decides whether the test blocks merge, release, or only alerts. | A refund or booking tool runs as a non-blocking smoke test. |
agent_ref | Ties the result to a prompt, branch, model, or environment. | Nobody can reproduce which agent version created the side effect. |
dependency_mode | Separates mock, sandbox, and live-scoped checks. | A routine test writes to production data. |
preconditions | Prevents tools from running before identity, consent, availability, or policy gates. | Agent books while caller identity is still unknown. |
argument_policy | Names which fields are allowed and where they must come from. | Model invents a customer ID or reuses stale context. |
side_effect_policy | Defines what the tool may mutate. | Transcript passes, but CRM has the wrong owner or duplicate case. |
expected_result | Connects tool response, durable state, and spoken confirmation. | Agent says "you are booked" after a timeout or partial failure. |
evidence | Makes failures debuggable and reviewable. | Test fails with no request payload, trace, or final-state query. |
replay_policy | Proves retry behavior before CI runs the test repeatedly. | Re-running the same test creates 2 appointments or tickets. |
The contract is not only for QA. It is also a product and engineering agreement. It says what a caller is allowed to accomplish, which system owns the truth, and what the agent must do when the tool path is uncertain.
Build a Guardrail Ledger
Most tool failures are not a single event. They are a mismatch between layers.
Use a ledger so the test can show exactly where the contract broke.
| Guardrail | Required Evidence | Fail When |
|---|---|---|
| Preconditions satisfied | Identity result, workflow state, caller confirmation, policy flags | Tool runs before the caller is eligible or verified. |
| Tool selected | Tool name, call ID, sequence number, execution-attempt ID | Agent calls the wrong tool, skips a required read, or starts an extra logical action outside the retry policy. |
| Arguments allowed | Parsed arguments, schema result, source for each high-risk field | Required field is missing, invented, stale, or copied from raw PII. |
| Ordering correct | Tool trace and state transition | Write happens before lookup, consent, availability, or confirmation. |
| Result interpreted | Tool status, error code, timeout state, partial result | Agent treats timeout, pending, or partial write as success. |
| Durable state verified | Query against fixture, sandbox, or destination record | Backend state is missing, duplicated, wrong, or uncleaned. |
| Caller response safe | Transcript after tool result | Agent confirms an action before it succeeded. |
| Replay safe | Second run with same run ID or retry condition | Duplicate side effect appears or cleanup hides the duplicate. |
For high-risk calls, do not grade the transcript alone. The transcript is one row in the ledger.
Pair this with the sandbox testing guide when a real dependency needs fixture data. Pair it with the structured output checklist when arguments come from extracted caller facts.
How Do You Test Tool Calls Without Sharing Internal Traces?
Keep the authoritative trace and assertion engine inside your system. Export a test-scoped evidence envelope that tells the QA runner what happened, what was expected, which checks passed, and which fields were withheld.
The QA runner does not need your prompt history, credentials, raw database rows, or complete stack traces. It needs stable correlation IDs and enough bounded evidence to distinguish "the agent said it worked" from "the action happened once under the contract."
Trace-less tool-call test: a black-box test that correlates a voice call with a sanitized sequence of tool events, internal assertion verdicts, final-state checks, and cleanup proof without exposing the underlying trace store.
Choose the narrowest evidence mode that can still prove the risk:
| Evidence mode | QA runner receives | Best for | Do not use when |
|---|---|---|---|
| Shared trace | Selected spans and redacted attributes | Same-company test systems with one telemetry boundary | The runner is outside the trust boundary or traces contain sensitive content |
| Evidence adapter | Sanitized tool events derived from traces or application events | External QA systems and cross-team test runners | The adapter cannot preserve order, correlation, or final-state evidence |
| Shadow assertions | Pass/fail verdicts plus bounded evidence from an internal verifier | Restricted data, private tool arguments, or regulated workflows | The runner needs raw values to diagnose a failure and no safe view exists |
| Final-state verifier | Post-call query result keyed by run ID | Calendar, CRM, ticket, database, and message side effects | A state check cannot prove which tool path produced the record |
Trace access is one way to collect evidence. An internal verifier can also retain sensitive execution detail and send the runner bounded verdicts. The runner must know which checks ran, which policy they used, and which evidence was unavailable.
Copyable QA Evidence Envelope
Version the envelope separately from your internal telemetry. A trace schema can change without breaking every test consumer. This synthetic example exercises a timeout followed by a safe retry of one logical action.
{ "schema_version": "voice_tool_evidence.v1", "test_run_id": "qa_run_84721", "call_id": "call_fixture_883", "agent_version": "appointments-agent-pr-128", "trace_correlation": { "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "trace_contents_shared": false }, "tool_events": [ { "sequence": 2, "tool_call_id": "tool_call_019", "execution_attempt_id": "attempt_001", "tool_name": "create_calendar_event", "idempotency_key_ref": "retry_group_01", "argument_assertions": { "required_fields_present": true, "trusted_identity_used": true, "forbidden_fields_absent": true }, "result_status": "timeout", "latency_ms": 2000 }, { "sequence": 3, "tool_call_id": "tool_call_020", "execution_attempt_id": "attempt_002", "tool_name": "create_calendar_event", "idempotency_key_ref": "retry_group_01", "argument_assertions": { "required_fields_present": true, "trusted_identity_used": true, "forbidden_fields_absent": true }, "result_status": "created", "latency_ms": 184 } ], "shadow_assertions": [ { "id": "booking_requires_confirmed_slot", "status": "passed", "evidence": "confirmation_event_precedes_tool_sequence_2" }, { "id": "retry_reuses_key_and_creates_one_record", "status": "passed", "evidence": "attempts_001_002_share_retry_group_01_and_one_fixture_record" }, { "id": "confirmation_only_after_success", "status": "passed", "evidence": "confirmation_follows_created_result_at_sequence_3" } ], "final_state": { "fixture_record_id": "booking_fixture_883", "record_count_for_run": 1, "duplicate_records": 0 }, "cleanup": { "status": "verified", "remaining_records": 0 }, "evidence_refs": { "transcript": "artifact_transcript_883", "tool_requests": "artifact_requests_883", "tool_responses": "artifact_responses_883", "final_state_query": "artifact_state_883", "cleanup_query": "artifact_cleanup_883" }, "verifier_policy_version": "calendar_contract_v3", "redactions": ["caller_contact", "raw_arguments", "raw_result"]}The example keeps the trace_id only as a correlation handle. W3C Trace Context standardizes how trace context can identify and propagate a request across services; it does not require you to expose the trace contents to a test consumer. If your stack uses OpenTelemetry's generative AI semantic conventions, pin the convention version and map tool execution into stable tool names and call IDs. Keep argument and result content private unless the receiving system is authorized to receive it.
idempotency_key_ref is an opaque, test-scoped token assigned by the internal adapter: equal keys for the same logical action must produce the same token, and different keys must remain distinguishable. It avoids exporting a raw key that may contain caller data. execution_attempt_id preserves each actual attempt, including handler retries that reuse a provider tool-call ID. Compare the key references and check the destination state before cleanup; matching keys alone do not prove that duplicate writes were prevented.
The artifact references point to retained evidence under the application's access controls. Export a redacted artifact or a named internal verdict when the runner cannot read it. A missing or unavailable required check is inconclusive and must not produce a passing release gate.
Expected Versus Observed Ledger
The evidence envelope becomes useful only when the test compares it with an explicit contract.
| Contract check | Expected | Observed field | Fail when |
|---|---|---|---|
| Correlation | One run ID connects the call, tool events, and final state | test_run_id, call_id, optional trace_id | Any artifact cannot be joined to the same run |
| Tool choice | One logical create_calendar_event action after confirmation; retries remain visible | tool_events[].tool_name, sequence, execution_attempt_id, precondition verdict | Wrong tool, missing attempt, extra logical action, or early call |
| Arguments | Required fields came from trusted evidence | argument_assertions verdicts | A required, source, or forbidden-field check fails |
| Result handling | Agent treats only created as success | result_status, shadow assertion | Timeout, partial, pending, or denied becomes a spoken success |
| Durable side effect | Exactly one fixture record exists | final_state.record_count_for_run | Record count is 0 or greater than 1 |
| Retry safety | Same key across attempts; one durable record | tool_events[].idempotency_key_ref, execution_attempt_id, final_state.duplicate_records | Key changes on retry, an attempt is hidden, or duplicates exceed 0 |
| Cleanup | Test leaves no fixture residue | cleanup.remaining_records | Cleanup is unverified or residue remains |
This split matters. The contract owns expectations. Your application owns sensitive execution facts. The QA runner owns comparison and reporting.
Shadow Assertions for Private Arguments
Some arguments cannot leave the application boundary even after redaction. A hashed patient identifier, for example, may still be linkable in the wrong context. In that case, evaluate the sensitive rule internally and export a named verdict:
{ "id": "tool_argument_matches_verified_identity", "status": "passed", "evidence": "verified_identity_policy_v3", "raw_value_shared": false}The assertion ID must describe the rule, and the policy version must make the verdict reproducible. passed: true by itself is too weak. It tells a reviewer nothing about what ran.
There is an honest limitation: a shadow assertion asks the QA runner to trust your verifier. For release-blocking workflows, test the verifier separately, version its policy, retain the private evidence for authorized investigation, and include negative cases that prove the assertion can fail.
Five Negative Tests for the Evidence Boundary
- Missing tool event: remove the expected event and verify the runner fails closed.
- Duplicate side effect: in a negative fixture, retain two execution attempts for the same logical action and key, then set
duplicate_records: 1. The runner must fail. A successful retry with one record may pass; a redelivery of the same evidence event may be deduplicated, but distinct execution attempts must stay visible. - Broken correlation: change the run ID on the final-state result and verify the runner rejects it.
- False success: return
result_status: "pending"with a positive transcript and verify the contract fails. - Unverified cleanup: omit the cleanup query or leave one fixture record and verify CI stays red.
This is the part teams tend to skip. A well-formed envelope is not proof until you have shown that missing, reordered, contradictory, and stale evidence produces a failure.
Idempotency Is a Required Test, Not an Implementation Detail
Voice agents retry. Callers repeat themselves. Webhooks time out. Tool handlers crash after writing but before returning a response. Phone calls reconnect. CI reruns jobs.
If the tool can create a side effect, the test should prove safe retry behavior.
| Scenario | Test | Passing Behavior |
|---|---|---|
| Caller repeats the request | Same caller asks to book the same slot again. | Agent finds or reuses the existing hold instead of creating a duplicate. |
| Tool timeout after write | Tool creates record but response times out. | Retry uses the same idempotency key and returns the same record or safe no-op. |
| CI rerun | Same test run ID executes twice. | Final-state query still finds exactly one target record. |
| Cleanup fails | Delete or reset endpoint returns failure. | Test fails and reports residue instead of hiding it. |
| Partial downstream failure | CRM update succeeds but SMS send fails. | Agent does not summarize the whole workflow as complete. |
Replay-safe voice-agent tool test: a test that can run twice with the same contract and run ID without creating duplicate customer-facing state.
If a tool is not replay-safe, keep it out of merge-blocking CI until the dependency supports idempotency, fixture isolation, and verified cleanup.
CI Gate Example
GitHub Actions workflows are YAML-defined automated processes. Your voice-agent tests do not have to use GitHub Actions, but the same principle applies: the gate should be explicit, repeatable, and tied to the files that changed.
name: Voice agent tool contractson: pull_request: paths: - "agents/**" - "voice-tests/**" - "tool-schemas/**"jobs: tool-contracts: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Validate test contracts run: npm run validate:voice-tool-contracts - name: Run blocking mocked tool tests run: npm run test:voice-tools -- --mode=mock --risk=blocking - name: Run sandbox side-effect tests run: npm run test:voice-tools -- --mode=sandbox --risk=blocking - name: Upload evidence if: always() run: npm run voice-tests:upload-evidenceThe exact command names will differ. The important part is the split:
| Gate | What It Catches | Blocks Merge? |
|---|---|---|
| Contract schema validation | Missing fields, invalid risk, unknown dependency mode, unsafe forbidden fields | Yes |
| Mocked tool tests | Tool choice, argument shape, failure handling, prompt regressions | Yes |
| Sandbox side-effect tests | Real integration behavior, durable state, cleanup, idempotency | Yes for critical tools |
| Phone-path workflow tests | ASR, latency, interruption, transfer, and voice-runtime behavior | Usually pre-release |
| Production sampling | Drift after launch | No, alert and convert failures into regression tests |
Use the CI/CD testing guide for the broader release strategy. The tool contract is the smallest artifact that tells CI what to prove.
Review Checklist Before a Tool Reaches Production
Use this during PR review or launch review.
| Question | Pass Condition |
|---|---|
| Is the tool name specific? | It names the business action, not a vague helper. |
| Is the tool description operational? | It says when to use the tool, when not to use it, and what result statuses mean. |
| Are high-risk arguments sourced? | Caller IDs, account IDs, dates, amounts, and destinations come from trusted context or confirmed caller evidence. |
| Are forbidden arguments named? | Raw PII, production IDs, unverified payment data, and stale context are blocked. |
| Is the side-effect mode explicit? | Mock, sandbox, or live-scoped is declared before the call. |
| Is idempotency required? | Creates, sends, transfers, refunds, and updates have dedupe keys or equivalent safeguards. |
| Does the test verify durable state? | It queries the destination system, fixture, or evidence envelope after the call. |
| Does the agent handle failure safely? | Timeout, partial success, denied action, and unavailable dependency do not become false confirmations. |
| Is cleanup verified? | Fixture residue fails the test instead of being ignored. |
| Can a reviewer replay the evidence? | Run ID, call ID, transcript, tool request, result, final-state query, and cleanup result are saved. |
The review should feel strict. A bad tool call can create work for support teams, break compliance expectations, or tell a caller something happened when it did not.
What This Template Cannot Prove
The contract proves that one tool path behaved under controlled conditions. It does not prove the whole voice system is safe.
| Limitation | Why It Matters | Add This |
|---|---|---|
| Audio path is outside the contract | ASR errors, barge-in, silence, and latency can change when tools run. | Phone-path tests and trace review. |
| Fixtures are cleaner than production | Real customer records can have missing fields, duplicates, or stale ownership. | Production sampling and data-quality checks. |
| Provider behavior changes | Tool-call and webhook runtimes evolve. | Versioned schemas and scheduled regression runs. |
| Business policy can drift | Refund, booking, escalation, and compliance rules change. | Owner review and rule fixtures. |
| A single tool is not the whole workflow | Multi-step flows can pass one tool test and fail at handoff or cleanup. | Workflow tests and incident-derived regressions. |
Start with the tool contract. Then connect it to workflow testing, tracing, and call evidence export.
