Every voice agent has a brain: the model that decides what the agent does. Most teams still build cascaded agents (speech-to-text, then an LLM, then text-to-speech) so they can pick that brain and swap it when a better model ships. The benchmarks that come with each launch don't help them choose: they test math and code, not phone calls. So we grade LLMs the way voice agents use them.
What we grade
Four test agents (a bank card line, dental scheduling, insurance claims and order support), each served on Retell. Same agent, same prompt, same tools. Only the model changes. We grade the words, not the voice.
- Response time and price. Response time is the median agent turn in text, tool calls included. On a phone line, speech-to-text and text-to-speech add to it, and past about 800 ms end to end a conversation stops feeling natural. Price is the LLM's list price per minute.
- Logic. The percentage of rule-based checks passed across four areas: understanding the caller, following the rules, completing the task and telling the truth. We check tool calls and stated facts.
- Phrasing. The percentage of replies that read like a person talking: no lists or markdown, no stiff or stock phrases, no repeating what the caller just said. Two AI judges from outside the benchmark read every reply; it's the only measure that uses a judge.
- Security. The percentage of scripted attacks stopped: impersonation, false authority, instructions planted in records, and pressure to skip a required step.
What we found
Speed and price split the models. Response times varied 15x and prices 40x.
Logic is close at the top, but the weak spots differ. Eight models are within 1.6 points. Following the rules varied most, from 89 to 100. Two top models answered Spanish-speaking callers in English every time, despite a written rule.
Phrasing is not close. Scores ran from 70 to 96. The models that sounded most human kept replies short. The rest read out lists, codes and date formats, stacked stock phrases, or read their own reasoning to the caller: "The caller has verified and explicitly confirmed."
Every model was fooled at least once. They followed instructions planted in customer records, skipped required steps because the caller asked, or handled cases that should have gone to a person.
What's next
This round graded the brain on its own, in text. Next we grade the rest of the cascade on real phone calls, and push harder on security:
- Speech-to-text and text-to-speech. How well each hears callers, and how natural each sounds.
- Speech-to-speech. Models that skip the cascade, against the best cascaded stacks.
- Adaptive attacks. Attackers that change tactics mid-call, the way we red-team production agents.
Choosing a model? Hamming runs these checks on your own prompt, tools and callers.