How to evaluate 2025’s voice agent: Beyond automatic speech recognition (ASR) and word error rate (WER) to mission success, barges and hallucinations

Optimized for automatic speech recognition (ASR) and word error rate (WER) only, insufficient for modern interactive voice proxy. Strong assessments must measure end-to-end mission success, barge behavior and latency, and unquestionable hallucination – and subsequent ASR, safety and guidance. VoiceBench provides multi-faceted Voice interaction Benchmarking the general knowledge, guidance, security and robustness of speaker/environment/content changes, but it does not cover task completion in barges or actual equipment. SLUE (and Phase 2) Target Language Understanding (SLU); probe multilingual and quality checks for large-scale and spoken languages; DSTC orbits adds spoken, task-oriented robustness. Combine them with clear in-barge/endpoint testing, user-centric tasks – successful measurement and controlled noise pressure protocols for complete pictures.

Why not enough?

Measure transcription fidelity, not interaction quality. The two agents with similar WERs have a big disagreement on conversation success, as delays, turns, misunderstanding recovery, security, and robustness to acoustics and content perturbations dominate the user experience. Previous work on the actual system indicates that evaluation is needed User satisfaction and Mission successful Cortana’s automated online assessment direct (EG) predicts user satisfaction for in-situ interactive signals, not just ASR accuracy.

How to measure (and how)?

1) End-to-end task success

Metric system: Task success rate (TSR), each task has strict success criteria (target completion, constraints), plus Task completion time (TCT) and Turn to success.
Why. The real assistant is judged by the results. Competitions such as the Alexa Awards TaskBot explicitly measure the user’s ability to complete multi-step tasks (e.g., cooking, DIY) and complete ratings and completions.

protocol.

• Define tasks with verifiable endpoints (for example, “assemble shopping lists with n items and constraints”).
• Use blind raters and automatic logs to calculate TSR/TCT/turns.
• For multilingual/SLU coverage, draw task intent/slots from a large scale.

2) Barge and turn

index:

• Barge Detection Latency (MS): Time from user attack to TTS suppression.
• Yes/Wrong Barge Rate: Correct interruption and false stop.
• Endpoint Delay (MS): The ASR after the user stops is completed.

Why. Smooth interrupt handling and fast endpoint determination of perceived responsiveness. The study formally introduces barge verification and continuous barge processing; the endpoint latency remains an active area in streaming.

protocol.

• The script prompts the user to interrupt the TT position on the controlled offset and SNR.
• Measure suppression and identification time with high precision logs (frame timestamps).
• Includes noisy/echo far-field conditions. Classical and modern research provide recovery and signaling strategies to reduce false barges.

3) Flash Illusion (Huns)

Metric. Huns interest rate: Part of the output fluent but semantically unrelated to audio under controlled noise or non-speech audio.
Why. ASR and AUDIO-LLM stacks can issue “convincing nonsense”, especially in the case of non-voice segments or noise superposition. Recent work definitions and measures of ASR hallucination; targeted research shows that non-voice sounds elicited whisper hallucinations.

protocol.

• Build audio sets with additional ambient noise (various SNRs), non-voice jammers, and content vulnerabilities.
• Score semantic correlation (by human judgment of adjudication) and calculation of HUN.
• Track whether downstream agent actions propagate hallucinations to make task steps incorrect.

4) Guide the following, safety and robustness

Metric family.

• Guide to follow accuracy (Format and Constraint Compliance).
• Safe rejection rate On the adversarial tips.
• A robust delta Across the speaker’s age/accent/tongue, environment (noise, reverb, far field) and content noise (grammatical error, alienation).

Why. VoiceBench explicitly targets these axes with spoken instructions (real and synthetic), covering general knowledge, guidance and security; it makes speakers, environment and content exploration robust.

protocol.

• Broadenables on voice interaction functions using VoiceBench; report summary and scores per axis.
• For SLU details (NER, dialogue behavior, QA, abstract), Leverage Slue and Stage 2.

5) Perceived speech quality (for TTS and enhancement)

Metric. Subjective average opinion score passed That-t P.808 (Crowdsourcing ACR/DCR/CCR).
Why. The quality of interaction depends on Both Approval and playback quality. P.808 provides a proven crowdsourcing protocol through open source tools.

Benchmark Landscape: Each Cover

VoiceBench (2024)

scope: Multi-faceted voice assistant evaluation with spoken input coverage common sense,,,,, Explain the following,,,,, Safetyand robustness Change across speakers/environment/content; use real and synthetic voice.
limit: Do no Complete benchmark barge/endpoint delay or real-world task completion on the device; focus on correctness and security of responses under changes.

slue/slue stage 2

scope: Language comprehension tasks: NER, emotion, dialogue behavior, naming entity localization, QA, abstract; aiming to study end-to-end and pipeline sensitivity to ASR errors.
use: Ideal for detecting SLU’s robustness and pipeline vulnerability in spoken settings.

A large number of

scope: > 1M Virtual assistive discourse in 51-52 languages ​​with intent/slots; perfect for multilingual Task-oriented assessment.
use: Create a multilingual task suite under voice conditions (paired with TTS or read voice) and measure TSR/slot F1.

Spoken – Band/Hays Quad and related Spoken QA set

scope: Oral question answering tests ASR perceived comprehension and multiple robustness.
use: Understand stress tests under voice errors; not a complete suite of agent tasks.

DSTC (Dialogue System Technical Challenge) Tracking

scope: Powerful dialog modeling explaintask-oriented data; human ratings along with automatic metrics; recent tracks emphasize multilingual, security and evaluation dimensions.
use: Complementary of conversation quality, DST and knowledge grounding responses under voice conditions.

Real-world mission assistance (Alexa Prive Taskbot)

scope: Multi-step task help User Rating and success standards (cooking/DIY).
use: Gold standard inspiration for defining TSR and interactive KPIs; public reports describe assessment priorities and outcomes.

Fill in the gaps: what you still need to add

1. Barge and Endpoint KPI
   Add a clear measurement harness. The literature provides barge validation and continuous processing strategies; streaming ASR endpoint latency remains an active research topic. Track barge detection latency, suppress correctness, endpoint delay and error barges.
2. Hallucination (HUN) scheme
   Emerging ASR barrier definition and controlled noise/non-voice testing were adopted; report the Hun speed and its impact on downstream operations.
3. Device interaction delay
   Associate user-aware latency with streaming Asr designs (e.g., sensor variants); measure first-class to-style time, scheduled time, and local processing overhead.
4. Cross-axis robustness matrix
   Combine VoiceBench’s speaker/environment/content axis with your Task Suite (TSR) to expose the faulty surface (e.g., barge insertion under far-field echo; mission success with low SNR; multilingual slots under accent transfer).
5. Perceived quality of playback
   Quantify user-aware TTS quality in an end-to-end loop using ITU-T P.808 (using the open P.808 toolkit), not just ASR.

---

Specific, reproducible evaluation plan

1. Assembly suite

• Voice interaction core: Voice table is the knowledge, guidance below, safety and robustness axis.
• Slu depth: SLUE/Phase 2 tasks (NER, Dialog, QA, Summary), used for SLU performance under voice.
• Multilingual coverage: Intent/slot and multilingual stress is huge.
• Understanding of ASR noise: Speech quality check and spoken language/Haysquad/Heysquad.

2. Add missing features

• Barge/End Point Harness: Controlled offset and SNR script interrupt; logarithmic suppression time and wrong barge insertion; endpoint delay measured by flow asr.
• Undoubted hallucinations: Non-speech insertion and noise overlay; comments and computational Hun semantic correlation.
• Task success block: Scheme tasks with objectively successful inspection; calculate TSR, TCT and turn; follow task mechanism style definition.
• Perceived quality: P.808 crowdsourcing ACR and Microsoft toolkit.

3. Report structure

• Main table: tsr/tct/turn; barge latency and error rate; end point delay; Huns; VoiceBench aggregation and per-axis; SLU indicator; p.808 mos.
• Pressure chart: TSR and Hun vs. SNR and Reverb; Barge latency and interruption timing.

---

refer to

• VoiceBench: The first multifaceted voice interaction benchmark for LLM-based voice assistants (knowledge, guidance below, security, robustness). (AR5IV)
• slue/slue stage 2: spoken language, dialogue behavior, QA, abstract; sensitivity to ASR errors in pipelines. (arxiv)
• Huge: Multilingual Intent/Slot Discourse for 1m+ Assistant. (Amazon Science)
• Spoken language – Bangles/Hays Quad: Answers oral questions in the dataset. (github)
• User-centered Production Assistant Evaluation (Cortana): Predicting satisfaction beyond ASR. (UMass Amherst)
• Barge Verification/Processing and Endpoint Delay: AWS/Academic Barge Paper, Microsoft Continuous Barge, Recent Streaming Endpoint Detection. (arxiv)
• ASR hallucination definition and non-phonologically induced hallucination (whisper). (arxiv)