If the Voice Agent Works Only for the Demo Accent, It Does Not Work

List expected languages, regional accents, code-switching, age ranges, speech differences, phone networks, background environments, and common domain terms. Use service data and community input rather than choosing a handful of accents the team can imitate.

Collect or commission evaluation material with informed consent, a defined purpose, retention limits, and compensation where appropriate. Do not turn production calls into an unlabeled research dataset because the terms were vague enough to permit it.

ASR research repeatedly finds uneven performance across speaker groups, and recent work shows that overall improvements do not guarantee smaller gaps. A transcript can also contain several errors while preserving intent, or one small error can change a medication, address, phone number, budget, or appointment date.

Compare performance by relevant groups, then examine intersections where sample size permits responsible interpretation. Add mobile and landline codecs, speakerphone, noise, quiet speech, overlapping voices, and unstable connections. Report uncertainty and sample size rather than ranking groups from a tiny test set.

Manual inspection remains valuable. Research on regional dialect adaptation found that established aggregate measures can miss application-specific errors. A reviewer who understands the task can distinguish an inconsequential article from a wrong account number.

Let people repeat, spell, use the keypad, switch language, receive a confirmation, or reach a person without being blamed for recognition failure. Avoid cheerful loops that say “I didn’t get that” indefinitely. After a bounded number of failures, change the channel or hand off with context.

Run this suite before every meaningful ASR, model, prompt, telephony, or noise-suppression change. Fairness is not a certification the vendor can transfer to your use case; it is a performance question that must remain visible in operation.