<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>voice on tomrochette.com</title>
    <link>https://tomrochette.com/tags/voice/</link>
    <description>Recent content in voice on tomrochette.com</description>
    <generator>Hugo -- gohugo.io</generator>
    <language>en</language>
    <managingEditor>tom@tomrochette.com (Tom Rochette)</managingEditor>
    <webMaster>tom@tomrochette.com (Tom Rochette)</webMaster>
    <copyright>© 2026 Tom Rochette</copyright>
    <lastBuildDate>Wed, 18 Nov 2026 03:48:09 -0500</lastBuildDate><atom:link href="https://tomrochette.com/tags/voice/index.xml" rel="self" type="application/rss+xml" />
    
    <item>
      <title>τ²-bench</title>
      <link>https://tomrochette.com/agents/evaluation-review/tau2-bench/</link>
      <pubDate>Wed, 18 Nov 2026 00:00:00 +0000</pubDate>
      <author>tom@tomrochette.com (Tom Rochette)</author>
      <guid>https://tomrochette.com/agents/evaluation-review/tau2-bench/</guid>
      <category>research-note</category><category>agent-curated</category><category>fully-ai-generated</category><category>llm=glm-5.3-flash</category><category>benchmark</category><category>evaluation</category><category>tool-agents</category><category>voice</category>
      <description>&lt;p&gt;τ²-bench is Sierra&amp;rsquo;s benchmark that scores a tool-using customer-service agent by having it serve an LLM-simulated user who can also act, then checking whether the simulated business database ends in the required state and whether the required information was communicated.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;τ²-bench judges the conversation loop rather than a single output: the agent must both drive its tools to change a database and elicit the right details from a user who has tools of their own, the dual-control gap that single-shot benchmarks skip.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2 class=&#34;relative group&#34;&gt;What it is&#xA;    &lt;div id=&#34;what-it-is&#34; class=&#34;anchor&#34;&gt;&lt;/div&gt;&#xA;    &#xA;    &lt;span&#xA;        class=&#34;absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none&#34;&gt;&#xA;        &lt;a class=&#34;text-primary-300 dark:text-neutral-700 !no-underline&#34; href=&#34;#what-it-is&#34; aria-label=&#34;Anchor&#34;&gt;#&lt;/a&gt;&#xA;    &lt;/span&gt;&#xA;    &#xA;&lt;/h2&gt;&#xA;&lt;p&gt;A MIT benchmark and Python CLI (&lt;code&gt;tau2 run&lt;/code&gt;) from Sierra, the customer-service AI company, with tasks across airline, retail, and telecom domains plus a banking_knowledge retrieval domain and a new intake (τ-Entity) domain of over 200 tasks that scores exact entity capture on outbound calls.&#xA;The reward is deterministic: pass or fail by the database&amp;rsquo;s end state against the gold state, plus checks that required information was communicated; in some domains (telecom) the simulated user holds tools of their own, so the agent must guide them through actions instead of doing everything itself.&#xA;Version 2.0.0 (2026-10-20) added τ-i18n: the same tasks in six languages (English, Spanish, Portuguese, Hindi, Chinese, Korean) with localized tasks, databases, and caller personas, full-duplex voice runs against realtime models, post-hoc judges that score naturalness and speech fidelity without changing the task reward, and frozen published benchmarks (text-v1 and voice-v2 at 900 calls each, voice-v1 English-only at 278) so runs stay comparable.&#xA;A live leaderboard sits at taubench.com, and the design is the paper&amp;rsquo;s (arXiv 2506.07982), extending the original 2024 τ-bench with the user-side tools.&lt;/p&gt;&#xA;&#xA;&lt;h2 class=&#34;relative group&#34;&gt;Status&#xA;    &lt;div id=&#34;status&#34; class=&#34;anchor&#34;&gt;&lt;/div&gt;&#xA;    &#xA;    &lt;span&#xA;        class=&#34;absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none&#34;&gt;&#xA;        &lt;a class=&#34;text-primary-300 dark:text-neutral-700 !no-underline&#34; href=&#34;#status&#34; aria-label=&#34;Anchor&#34;&gt;#&lt;/a&gt;&#xA;    &lt;/span&gt;&#xA;    &#xA;&lt;/h2&gt;&#xA;&lt;p&gt;Active and adopted: 2,473 stars, MIT, created 2025-06-09, pushed 2026-11-10, as of 2026-11-18, on a release train that ran v0.2.0 (2025-10-06, the web leaderboard), v1.0.0 (2026-03-18, voice, banking_knowledge, self-named τ³), and v2.0.0 (2026-10-20, multilingual and voice).&lt;/p&gt;&#xA;&lt;picture&gt;&#xA;  &lt;source media=&#34;(prefers-color-scheme: dark)&#34; srcset=&#34;https://api.star-history.com/chart?repos=sierra-research/tau2-bench&amp;type=date&amp;theme=dark&amp;legend=top-left&#34; /&gt;&#xA;  &lt;source media=&#34;(prefers-color-scheme: light)&#34; srcset=&#34;https://api.star-history.com/chart?repos=sierra-research/tau2-bench&amp;type=date&amp;theme=dark&amp;legend=top-left&#34; /&gt;&#xA;  &lt;img alt=&#34;Star History Chart&#34; src=&#34;https://api.star-history.com/chart?repos=sierra-research/tau2-bench&amp;type=date&amp;theme=dark&amp;legend=top-left&#34; /&gt;&#xA;&lt;/picture&gt;&#xA;&lt;p&gt;&lt;strong&gt;The footprint is practitioner-grade: a 197-point Hacker News thread measured a prompt rewrite moving GPT-5-mini by 22 percent on the benchmark, and vendor score reports cite its voice leaderboards, so its numbers are quoted and argued with rather than ignored.&lt;/strong&gt;&lt;/p&gt;&#xA;&#xA;&lt;h2 class=&#34;relative group&#34;&gt;Strengths&#xA;    &lt;div id=&#34;strengths&#34; class=&#34;anchor&#34;&gt;&lt;/div&gt;&#xA;    &#xA;    &lt;span&#xA;        class=&#34;absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none&#34;&gt;&#xA;        &lt;a class=&#34;text-primary-300 dark:text-neutral-700 !no-underline&#34; href=&#34;#strengths&#34; aria-label=&#34;Anchor&#34;&gt;#&lt;/a&gt;&#xA;    &lt;/span&gt;&#xA;    &#xA;&lt;/h2&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Dual control measures what single-turn tool-call benchmarks cannot: information elicitation and user-side action under simulation.&lt;/li&gt;&#xA;&lt;li&gt;The reward is deterministic (database end-state plus communication checks), so scores do not inherit judge-model variance.&lt;/li&gt;&#xA;&lt;li&gt;Frozen published benchmark configs (text-v1, voice-v2, voice-v1) pin the scored bytes, in the same spirit as Harbor&amp;rsquo;s content-addressed datasets.&lt;/li&gt;&#xA;&lt;li&gt;Voice and six-language coverage is unique in this category, and the post-hoc judges keep speech quality separate from reward.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2 class=&#34;relative group&#34;&gt;Cautions&#xA;    &lt;div id=&#34;cautions&#34; class=&#34;anchor&#34;&gt;&lt;/div&gt;&#xA;    &#xA;    &lt;span&#xA;        class=&#34;absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none&#34;&gt;&#xA;        &lt;a class=&#34;text-primary-300 dark:text-neutral-700 !no-underline&#34; href=&#34;#cautions&#34; aria-label=&#34;Anchor&#34;&gt;#&lt;/a&gt;&#xA;    &lt;/span&gt;&#xA;    &#xA;&lt;/h2&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The user is a simulated LLM: its behavior bounds and biases the measured loop, so results describe agent-plus-simulator, not agent-plus-human.&lt;/li&gt;&#xA;&lt;li&gt;Prompt sensitivity is demonstrated on its own board: the 197-point thread&amp;rsquo;s 22 percent swing from a prompt rewrite means leaderboard gaps need a prompt-variance caveat.&lt;/li&gt;&#xA;&lt;li&gt;Domain coverage is narrow (airline, retail, telecom, banking knowledge, intake), so it says little about coding or research agents.&lt;/li&gt;&#xA;&lt;li&gt;The voice judges are post-hoc and advisory: naturalness scores never change the reward, so a smooth-sounding failing run still fails.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2 class=&#34;relative group&#34;&gt;Pricing&#xA;    &lt;div id=&#34;pricing&#34; class=&#34;anchor&#34;&gt;&lt;/div&gt;&#xA;    &#xA;    &lt;span&#xA;        class=&#34;absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none&#34;&gt;&#xA;        &lt;a class=&#34;text-primary-300 dark:text-neutral-700 !no-underline&#34; href=&#34;#pricing&#34; aria-label=&#34;Anchor&#34;&gt;#&lt;/a&gt;&#xA;    &lt;/span&gt;&#xA;    &#xA;&lt;/h2&gt;&#xA;&lt;p&gt;Free, MIT; a run bills your agent&amp;rsquo;s model tokens plus the simulated user&amp;rsquo;s, and voice runs bill the realtime model.&lt;/p&gt;&#xA;&#xA;&lt;h2 class=&#34;relative group&#34;&gt;Compared to&#xA;    &lt;div id=&#34;compared-to&#34; class=&#34;anchor&#34;&gt;&lt;/div&gt;&#xA;    &#xA;    &lt;span&#xA;        class=&#34;absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none&#34;&gt;&#xA;        &lt;a class=&#34;text-primary-300 dark:text-neutral-700 !no-underline&#34; href=&#34;#compared-to&#34; aria-label=&#34;Anchor&#34;&gt;#&lt;/a&gt;&#xA;    &lt;/span&gt;&#xA;    &#xA;&lt;/h2&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://tomrochette.com/agents/evaluation-review/swe-bench/&#34; &gt;SWE-bench&lt;/a&gt;: deterministic judging on both sides, but SWE-bench scores a patch against a repository&amp;rsquo;s tests while τ²-bench scores a service conversation against a database end-state.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://tomrochette.com/agents/evaluation-review/frontierharness-eval/&#34; &gt;FrontierHarness Eval&lt;/a&gt;: the harness-comparison study; τ²-bench is a fixed benchmark you run agents on, not a comparison of the harnesses themselves.&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://tomrochette.com/agents/evaluation-review/agent-qa/&#34; &gt;agent-qa&lt;/a&gt;: also judges interactive agent behavior, but against the actual app UI with a per-step verifier, where τ²-bench checks the simulated business&amp;rsquo;s end state.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2 class=&#34;relative group&#34;&gt;Bottom line&#xA;    &lt;div id=&#34;bottom-line&#34; class=&#34;anchor&#34;&gt;&lt;/div&gt;&#xA;    &#xA;    &lt;span&#xA;        class=&#34;absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none&#34;&gt;&#xA;        &lt;a class=&#34;text-primary-300 dark:text-neutral-700 !no-underline&#34; href=&#34;#bottom-line&#34; aria-label=&#34;Anchor&#34;&gt;#&lt;/a&gt;&#xA;    &lt;/span&gt;&#xA;    &#xA;&lt;/h2&gt;&#xA;&lt;p&gt;&lt;strong&gt;Recommended for teams building customer-facing tool agents who want a comparable, deterministic-reward loop benchmark, in text or voice.&lt;/strong&gt;&#xA;Not for coding-agent evaluation, and never without the simulator and prompt-variance caveats attached to its leaderboard numbers.&lt;/p&gt;&#xA;&#xA;&lt;h2 class=&#34;relative group&#34;&gt;Changes&#xA;    &lt;div id=&#34;changes&#34; class=&#34;anchor&#34;&gt;&lt;/div&gt;&#xA;    &#xA;    &lt;span&#xA;        class=&#34;absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none&#34;&gt;&#xA;        &lt;a class=&#34;text-primary-300 dark:text-neutral-700 !no-underline&#34; href=&#34;#changes&#34; aria-label=&#34;Anchor&#34;&gt;#&lt;/a&gt;&#xA;    &lt;/span&gt;&#xA;    &#xA;&lt;/h2&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;2026-11-18 - Created from the awesome-agent-observability 2026-11-15 evals-expansion scan, admitted as the category&amp;rsquo;s dual-control service-agent benchmark.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2 class=&#34;relative group&#34;&gt;See also&#xA;    &lt;div id=&#34;see-also&#34; class=&#34;anchor&#34;&gt;&lt;/div&gt;&#xA;    &#xA;    &lt;span&#xA;        class=&#34;absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none&#34;&gt;&#xA;        &lt;a class=&#34;text-primary-300 dark:text-neutral-700 !no-underline&#34; href=&#34;#see-also&#34; aria-label=&#34;Anchor&#34;&gt;#&lt;/a&gt;&#xA;    &lt;/span&gt;&#xA;    &#xA;&lt;/h2&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://tomrochette.com/agents/evaluation-review/swe-bench/&#34; &gt;SWE-bench&lt;/a&gt; - the deterministic ground benchmark on the coding side of the same judging idea&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://tomrochette.com/agents/evaluation-review/agent-qa/&#34; &gt;agent-qa&lt;/a&gt; - the acceptance-test harness judging interactive agent behavior against the real app&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://tomrochette.com/agents/evaluation-review/frontierharness-eval/&#34; &gt;FrontierHarness Eval&lt;/a&gt; - the harness-comparison study for when the question is which harness, not how good the agent&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://tomrochette.com/agents/evaluation-review/mlr-bench/&#34; &gt;MLR-Bench&lt;/a&gt; - the research-agent benchmark whose judge panel is validated against expert humans, the calibration τ²-bench&amp;rsquo;s deterministic reward sidesteps&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&#xA;&lt;h2 class=&#34;relative group&#34;&gt;References&#xA;    &lt;div id=&#34;references&#34; class=&#34;anchor&#34;&gt;&lt;/div&gt;&#xA;    &#xA;    &lt;span&#xA;        class=&#34;absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none&#34;&gt;&#xA;        &lt;a class=&#34;text-primary-300 dark:text-neutral-700 !no-underline&#34; href=&#34;#references&#34; aria-label=&#34;Anchor&#34;&gt;#&lt;/a&gt;&#xA;    &lt;/span&gt;&#xA;    &#xA;&lt;/h2&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://github.com/sierra-research/tau2-bench&#34;  target=&#34;_blank&#34; rel=&#34;noreferrer&#34;&gt;&lt;img class=&#34;external-link-favicon&#34; src=&#34;https://www.google.com/s2/favicons?domain=github.com&amp;sz=128&#34; alt=&#34;&#34; width=&#34;16&#34; height=&#34;16&#34; loading=&#34;lazy&#34;&gt;https://github.com/sierra-research/tau2-bench&lt;/a&gt; - repository: README (the τ-bench 2.0 features, the benchmarks table, domain list), license&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://arxiv.org/abs/2506.07982&#34;  target=&#34;_blank&#34; rel=&#34;noreferrer&#34;&gt;&lt;img class=&#34;external-link-favicon&#34; src=&#34;https://www.google.com/s2/favicons?domain=arxiv.org&amp;sz=128&#34; alt=&#34;&#34; width=&#34;16&#34; height=&#34;16&#34; loading=&#34;lazy&#34;&gt;https://arxiv.org/abs/2506.07982&lt;/a&gt; - the paper defining the dual-control evaluation&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://taubench.com&#34;  target=&#34;_blank&#34; rel=&#34;noreferrer&#34;&gt;&lt;img class=&#34;external-link-favicon&#34; src=&#34;https://www.google.com/s2/favicons?domain=taubench.com&amp;sz=128&#34; alt=&#34;&#34; width=&#34;16&#34; height=&#34;16&#34; loading=&#34;lazy&#34;&gt;https://taubench.com&lt;/a&gt; - the live leaderboard&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://sierra.ai/blog/benchmarking-agents-in-collaborative-real-world-scenarios&#34;  target=&#34;_blank&#34; rel=&#34;noreferrer&#34;&gt;&lt;img class=&#34;external-link-favicon&#34; src=&#34;https://www.google.com/s2/favicons?domain=sierra.ai&amp;sz=128&#34; alt=&#34;&#34; width=&#34;16&#34; height=&#34;16&#34; loading=&#34;lazy&#34;&gt;https://sierra.ai/blog/benchmarking-agents-in-collaborative-real-world-scenarios&lt;/a&gt; - the maker&amp;rsquo;s announcement of the benchmark&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://api.github.com/repos/sierra-research/tau2-bench/releases&#34;  target=&#34;_blank&#34; rel=&#34;noreferrer&#34;&gt;&lt;img class=&#34;external-link-favicon&#34; src=&#34;https://www.google.com/s2/favicons?domain=api.github.com&amp;sz=128&#34; alt=&#34;&#34; width=&#34;16&#34; height=&#34;16&#34; loading=&#34;lazy&#34;&gt;https://api.github.com/repos/sierra-research/tau2-bench/releases&lt;/a&gt; - the release train from v0.2.0 (2025-10-06) to v2.0.0 (2026-10-20)&lt;/li&gt;&#xA;&lt;li&gt;&lt;a href=&#34;https://hn.algolia.com/api/v1/search?query=tau2-bench&amp;amp;hitsPerPage=6&#34;  target=&#34;_blank&#34; rel=&#34;noreferrer&#34;&gt;&lt;img class=&#34;external-link-favicon&#34; src=&#34;https://www.google.com/s2/favicons?domain=hn.algolia.com&amp;sz=128&#34; alt=&#34;&#34; width=&#34;16&#34; height=&#34;16&#34; loading=&#34;lazy&#34;&gt;https://hn.algolia.com/api/v1/search?query=tau2-bench&amp;hitsPerPage=6&lt;/a&gt; - the footprint: the 197-point prompt-sensitivity thread and the vendor voice-leaderboard reports&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;</description>
      
    </item>
    
  </channel>
</rss>
