IHBench: What Voice Agents Say After You Interrupt

When a user interrupts a voice agent mid-sentence, two different things have to go right. The model has to stop talking at the right moment, and then it has to say the right thing next. A growing set of full-duplex benchmarks measure the first one: barge-in detection, yield latency, telling a real interruption from a backchannel.

IHBench measures the second one. Given a workflow-driven conversation where the user cuts in to correct a detail, express impatience, switch topics, or push back, does the model resume at the correct step, address the interjection, and avoid re-delivering content the user already heard? That is post-interruption recovery, a capability existing speech benchmarks leave unmeasured.

Human interruptions fall into six natural groups. We inject them into the conversation at a controlled point mid-utterance, each with its own recovery criteria generated alongside the conversation. Here are the groups:

Two real IHBench cases. The em dash (—) marks where the user cut in mid-utterance. The difference is rarely about understanding the words; it is about whether the model keeps its place in the workflow.

At each interruption the model generates the next response, which is scored on two independent axes:

We found that task fulfillment and recovery quality do not necessarily improve across model generations and that there’s a significant difference between model families, in particular how they handle fillers. This suggests that models so far haven’t really been trained to do particularly well on the type of situations that we believe matter for human-machine interactions. Here’s what we find:

Task fulfillment and recovery quality rank models differently. GPT Realtime 2 leads task fulfillment (0.73 win rate against the GPT-4o Audio baseline), while Gemini 2.5 Flash with thinking leads recovery quality (0.70 pass rate) on a lower task-fulfillment score. Since a model can advance the task well and recover differently, both axes are worth reporting. The full per-model table is in the Leaderboard tab.

The fillertype, continuing an utterance after a backchannel like “mm-hm,” is where recovery quality splits most cleanly along model families. Each bar below spans the range of filler pass rates within a family; the dot marks the family mean.

Task fulfillment varies with how deep into the conversation the interruption lands. A per-model logistic regression of the win/lose outcome on depth gives a mean slope of −0.030 (t(25) = −7.04). The decline is far steeper for open-weight models: their mean slope (−0.053) is about 3.3× that of the closed-weight models (−0.016), a gap that is itself significant (Welch t = 7.74). Depth is therefore a useful axis for telling robust multi-turn behavior apart from shallow turn-taking.

For the 15 configurations that accept both modalities (six Gemini, nine open-weight), we re-run the benchmark on the text transcript alone, holding everything else fixed. A TOST equivalence test shows the Gemini models are statistically indistinguishable across modalities (within ±0.02 on every metric), while text input is strictly better for the open-weight configurations (by ~8 points on task fulfillment and ~6 on recovery quality). Audio input never outperforms text, so there is no audio-versus-text modality gap on what IHBench measures.

Each conversation is an assistant-led state machine driving an ordered workflow (e.g. Authenticate → Verify Account → Process Request → Confirm) across one of 10 enterprise domains, grounded in concrete fictional values rather than placeholders. Interruptions are injected within stages, not at clean boundaries, so recovery is non-trivial: the model has to figure out where it left off inside a stage. A multi-agent pipeline generates and audits each conversation:

If you use IHBench in your research, please cite:

@misc{ihbench2026,  title         = {IHBench: Evaluating Post-Interruption Recovery in Voice Agents with Structured Workflows},  author        = {Salimi, Ahmad and Ma, Wentao and Tang, Yuzhi and Shen, Dongming and Li, Mu and Smola, Alex},  year          = {2026},  eprint        = {2606.19595},  archivePrefix = {arXiv},  primaryClass  = {cs.CL},}