The Voice Agent Worked Yesterday. Prove It Still Works Today.

A model update may make a conversation sound smoother while increasing tool retries. A shorter prompt may reduce latency and omit a disclosure. A new voice may pronounce product names better but interrupt callers more often. Regression testing has to follow the whole call from the first audio through the final CRM or calendar side effect.

Create a scenario registry with the caller goal, starting data, allowed actions, forbidden actions, expected evidence, acceptable variation, severity, and cleanup rule. Preserve the test definition while changing one system version at a time. Otherwise a passing run may reflect an easier test rather than a better agent.

Include common calls, rare high-impact calls, and known failures. Cover accents, pace, interruptions, silence, ambiguous dates, corrections, declined consent, wrong-person answers, stale CRM data, provider timeouts, no human availability, and caller requests outside authority. Add every serious incident back into the permanent suite.

NIST frames AI evaluation through test, evaluation, verification, and validation across a system lifecycle. For a voice workflow, that means evaluating the sociotechnical system—the call policy, models, tools, people, and operating environment—not a language model score in isolation.

Mock calendars, payment systems, CRM writes, telephony failures, and human queues so the same scenario can be repeated without contacting people or polluting production. Assert on structured events: action requested, authorization checked, tool input, response, retry, spoken claim, and final state.

Then run a smaller pre-production or controlled live suite to catch carrier behavior, real latency, audio quality, webhook ordering, and provider integration. Use synthetic test accounts and explicit allowlists. A test should never accidentally call a customer because somebody copied a production segment.

Capture version IDs, timestamps, trace context, stage latency, interruptions, recognition confidence where available, tool events, final outcome, and reviewer decision. OpenTelemetry separates traces, metrics, and logs while allowing them to be correlated; that makes a failed spoken response easier to connect to the slow or rejected tool call beneath it.

Use severity-based gates. One duplicate order or failed opt-out should not disappear inside hundreds of harmless passes. Publish the blocked scenarios, accepted risks, approver, expiry, and rollback version. Release confidence comes from visible exceptions, not a green percentage alone.