A survey of speech-to-speech models

Updated November 2026: added Thinking Machines’ interaction model, Block’s Dessa, Phonic-1, Boson’s Higgs Realtime Duplex and Hume’s Audio Decisions API, all released later in the year.

Voice assistants have been ten years away from being good for about ten years. US smart-speaker ownership has sat at roughly a third of people since 2023 (Edison Research). When Nielsen Norman Group watched people use Alexa, Google Assistant and Siri, the biggest problems weren’t about recognizing words. People couldn’t tell what the assistant had understood, couldn’t interrupt it, and fell back on speaking slowly in fragments (NN/g). Those are failures of conversation, not of language understanding, and they are what the current generation of speech-to-speech models is trying to fix.

This essay surveys how those models work and who has built what: how a second of audio becomes tokens, how a model predicts several streams at once, how it decides when to talk, and how the main systems put those pieces together.

The idea that ties it together is that a spoken-dialogue model is a joint model over several clocks at once: your audio, its audio, its words and its thoughts. Every architecture is a choice about how to factor that joint distribution, and when to sample from it.

four kinds of full duplex

First, the landscape. In telecommunications, full duplex means both ends can send and receive at the same time; a full-duplex voice model listens while it speaks. A simple test: while it answers, say “mm-hm” halfway through. A system that takes turns either ignores you or stops dead. A full-duplex one hears that you are still with it and keeps going, or hears that you meant to interrupt and stops mid-word. The term now names at least four different architectures, and they behave very differently.

The first kind is one model over both sides’ audio. Moshi is the open reference, and NVIDIA’s PersonaPlex builds on it with a 7B backbone and voice prompting, at about 205 ms average latency (NVIDIA). Block’s Dessa, announced in October, is the most radical: trained only on speech, with no text and no pretrained language model, on a single GB200 NVL72 rack, and small enough to run on a laptop. Block has published no benchmarks (Block). Thinking Machines’ TML-Interaction-Small is the most ambitious: a 276B-parameter mixture of experts with 12B active, working in 200 ms micro-turns over text, video and audio, with no voice-activity detector at all. Audio goes in as dMel features and comes out through a mel-spectrogram flow-matching head, and a separate background model handles reasoning and tools (Thinking Machines):

Boson’s Higgs Realtime Duplex, released in November, takes the same shape with a 45B mixture of experts in 160 ms chunks. Its audio input runs at only 6.25 tokens a second, half its output rate, and a flow-matching decoder renders the voice. It reports 98.9 on Big Bench Audio and 75.5 on Full-Duplex-Bench v1.5 (Boson):

The second kind runs full duplex in text and leaves the voice to a separate synthesizer. NVIDIA’s NemotronLabs VoiceChat feeds a streaming speech encoder on an 80 ms grid into a 9B language model that emits the agent’s text and tool calls frame by frame, with silence and turn tokens; a streaming TTS then produces audio from a 12.5 Hz codec. Its turn latency is 0.45 s (NVIDIA):

The third kind separates the decision from the content. Alibaba’s Qwen-Audio-3.1-Realtime is a duplex decision model that chooses when to listen, speak, stop or resume, a speech-to-text model that decides what to say, and a renderer for the voice. It answered background speech only 13% of the time on Full-Duplex-Bench v1.5, but takes 1.9 s to take a turn (Qwen team).

The fourth kind, a fast duplex talker that delegates to a slow reasoner, is what most products ship. OpenAI’s GPT-Live hands hard requests to a frontier model and gets the answer back as text (OpenAI). StepFun’s StepAudio 3 Realtime runs two concurrent calls to the same 196B mixture of experts, one speaking in 320 ms blocks and one reasoning (StepFun). Phonic-1, released in late October, sits first on Artificial Analysis’s speech-to-speech index, and its site says frontier “supervisor” models handle tools and reasoning; it has published no technical report (Phonic).

Not everything is full duplex. xAI’s Grok Voice 3.0, second on the same index, is a speech-to-speech model that still uses server-side voice-activity detection (xAI). Google never uses the word, but Gemini 3.8 Live has made tools asynchronous, keeps proactive audio permanently on, and no longer treats a completed turn as the model being idle (Google DeepMind). Claude’s voice mode is turn-based, and ElevenLabs’ own documentation describes Eleven Realtime as separate models for each stage, although leaderboards list it as speech-to-speech (ElevenLabs).

System	Kind	Audio in → out	Latency	Notable number
Moshi / PersonaPlex (Kyutai, NVIDIA)	Joint audio model	Mimi tokens → Mimi tokens	0.16–0.21 s	98.5% of pauses taken over (Moshi)
Dessa (Block)	Joint, speech only	Undisclosed	Undisclosed	Trained on one GB200 rack
TML-Interaction-Small (Thinking Machines)	Joint + background model	dMel → mel flow head	0.40 s	FDB v1.5: 77.8
Higgs Realtime Duplex (Boson)	Joint + backend model	6.25 tok/s → 12.5 tok/s, flow decoder	Not reported	Big Bench Audio: 98.9
NemotronLabs VoiceChat (NVIDIA)	Duplex text + TTS	Encoder features → text → codec	0.45 s	FDB v3 tool tasks: 33%
Qwen-Audio-3.1-Realtime (Alibaba)	Decision model + content + renderer	Encoder features → undisclosed	1.9 s	Background speech answered 13%
GPT-Live-1 (OpenAI)	Talker + reasoner	Undisclosed	1.24–1.34 s to first audio	AA index: 83.2
StepAudio 3 Realtime (StepFun)	Talker + reasoner, one model	Encoder features → undisclosed	8.83 s to first audio	Conversational dynamics: 98.9
Phonic-1 (Phonic)	Talker + supervisor	Undisclosed	0.61 s to first audio	AA index: 87.1 (first)
Grok Voice 3.0 (xAI)	Half duplex	Undisclosed	0.81 s to first audio	AA index: 86.9

The closed frontier labs disclose almost nothing about their models: no codec, frame rate or size for GPT-Live, Gemini Live or Grok Voice. Nearly every architecture diagram available comes from NVIDIA, the Chinese labs, Thinking Machines and Boson. The rest of this essay is about the mechanics those disclosures reveal.

the 200 ms budget

Start with the baseline every system is measured against. Across ten languages, the gap between one person finishing and the next starting averages 208 ms, with a median of 100 ms and a mode of zero: the most common gap is no gap at all (Stivers et al.). Producing even a short utterance takes 600 ms or more, so people must start planning their reply while the other person is still talking (Levinson and Torreira).

A cascade can’t do that. It runs speech recognition, then a language model, then speech synthesis, and adds their delays:

▶ Ask a question+ it needs to look something up

A person

~0.5 s

A voice AI

Hearing you finish

Thinking

00.5s1s1.5s2s

Press ask, then wait for the answer like you would in real life.

1. Your voice travels in 114 ms
2. Deciding you’ve finished, and transcribing 300 ms
3. Thinking of a reply 650 ms
4. Turning words into sound 140 ms
5. The sound travels back 89 ms

Worse, it can’t start any of that until it decides you’ve finished, and the usual way to decide is to wait for silence. LiveKit’s default minimum endpointing delay is 500 ms (LiveKit), already more than twice the human average before any model runs. Shorten the timer and the system starts talking over people who paused to think:

policy

silence timertimer + end-of-turn model

timeout

500 ms

times it cuts you off–

gap after you finish–

people, on average208 ms

Illustrative timings and probabilities. The gap shown is the timer alone, before recognition, the language model and speech synthesis add their own delay. A full-duplex model has no timer: it decides every 80 ms frame whether to speak.

End-of-turn models soften this trade-off but don’t remove it, because they still fire on silence. They also look much better on their own test sets than on anyone else’s. On LiveKit’s open end-of-turn benchmark, Pipecat’s Smart Turn, which reports about 94% accuracy on its own data, produces 35% false cut-offs when it has to decide within 300 ms. LiveKit’s own model produces 9.9%, on LiveKit’s own benchmark (LiveKit eot-bench). Full duplex removes the trigger rather than tuning it. OpenAI describes its current voice system as having taken the turn detector out of the audio path altogether (OpenAI).

tokens

To model audio with a language model, you first need tokens. Neural codecs compress a waveform into a sequence of frames and quantize each frame with residual vector quantization. The first codebook quantizes the encoder’s output; each later one quantizes what the earlier ones missed:

$$r_0 = z,\qquad q_i = Q_i(r_{i-1}),\qquad r_i = r_{i-1} - q_i,\qquad \hat z = \sum_{i=1}^{N_q} q_i .$$

With $N_q$ codebooks of $N$ entries at a frame rate $f_r$, the bitrate is

$$R = f_r \cdot N_q \cdot \log_2 N .$$

SoundStream’s paper explains why the residual structure is necessary. At 6 kbps and 75 frames a second, each frame gets 80 bits, so a single codebook would need $2^{80}$ entries. Eight codebooks of 1,024 entries cover the same 80 bits. And for the codec, depth is almost free: at a fixed 6 kbps, eight codebooks of 1,024, sixteen of 32, and eighty of 2 all reconstruct nearly equally well, with ViSQOL scores of 4.01, 3.98 and 3.92 (Zeghidour et al.). The cost of depth shows up later, in the model that has to predict those codebooks one after another.

Mimi, the codec behind Kyutai’s Moshi, set the shape most dialogue models now use: 24 kHz audio at 12.5 frames a second, eight codebooks of 2,048 entries, so $12.5 \times 8 \times 11 = 1{,}100$ bits per second (Défossez et al.). Its first codebook is trained to carry content. A linear projection of it is pulled, by cosine distance, towards the features of WavLM, a self-supervised speech model. That turns out to matter more than reconstruction. In a VALL-E-style test, generating from EnCodec tokens gave a 47% word error rate and from DAC tokens 86%, against about 5% for codecs whose first codebook carries meaning (X-Codec).

Two details from Mimi’s ablations are worth knowing. The first is that distilling meaning into the first level of an ordinary residual quantizer fights reconstruction, because every later level only sees what the first one left. Mimi runs the semantic quantizer in parallel with a seven-level acoustic one and adds their outputs. The second is that the usual codec metrics point the wrong way. Mimi trained with only adversarial losses scores 1.84 on ViSQOL but 81.0 on MUSHRA listening tests; the version that adds reconstruction losses scores 2.82 and 58.8 (Défossez et al.). A codec chosen by its ViSQOL score would be the one people like less.

Across 17 languages, speech carries about 39 bits of linguistic information per second (Coupé et al.). Roughly everything between that and Mimi’s 1.1 kbps is the voice:

• hand-designed codec
• neural codec, many codebooks
• neural, one codebook
• information rate

• Telephone (G.711)64 kbps
• Opus, wideband16–20 kbps
• SoundStream / EnCodec6 kbps
• Mimi (Moshi)1.1 kbps
• TAAE400 bps
• GLM-4-Voice tokens175 bps
• TaDiCodec87.5 bps
• Language itself≈ 39 bits/s

101001k10k100k

Bits per second, log scale. Everything between the bottom bar and the one above Mimi is the voice: who is speaking, how, and where. A model that predicts codec tokens has to predict all of that too.

Codecs below about 12.5 frames a second start borrowing information from the text. TaDiCodec reaches 87.5 bits per second, but only because its decoder is also given the transcript; without it, word error rises above 10% at 12.5 Hz (TaDiCodec). GLM-4-Voice uses a single 12.5 Hz codebook at 175 bits per second that keeps the words and lets a separate decoder supply a voice (GLM-4-Voice). I think this is the central trade-off: every bit you drop from the token stream is a bit some other part of the system has to reconstruct.

eight tokens every 80 ms

A frame of Mimi audio is eight tokens, and a model has to predict all of them every 80 ms. Write $a_t^k$ for codebook $k$ at frame $t$. The exact factorization is

$$p\big(a_t^{1:K} \mid a_{<t}\big) \;=\; \prod_{k=1}^{K} p\big(a_t^{k} \mid a_t^{<k},\, a_{<t}\big),$$

and the architectures differ in how much of it they keep.

Flattening keeps all of it: every token gets its own step of the large model, so one second of Mimi audio costs 100 large steps. MusicGen’s delay pattern keeps much less. It shifts codebook $k$ by $k-1$ frames and predicts all codebooks of a step in parallel, so a token conditions on the earlier codebooks of its frame only through the shift. On 30 seconds of music with four codebooks, flattening took 6,000 steps and the delay pattern 1,500, with Fréchet audio distances of 0.86 and 0.96; predicting all codebooks in parallel with no delay scored 2.58 (Copet et al.). The delay pattern gets most of the quality at a quarter of the steps, but adds latency: the last codebook of the first frame arrives $K-1$ frames late.

Moshi keeps the exact factorization with a two-level model. A large Temporal Transformer runs once per frame and predicts the text token, and a small Depth Transformer then emits the frame’s eight audio tokens one after another, each conditioned on the ones before it. The small model matters. With the same delay layout, separate output heads reached a perplexity of 135.4 and the Depth Transformer 36.8 (Défossez et al.). It is cheap in FLOPs, roughly a tenth of Moshi’s total, but expensive in time, because it is $K$ small sequential steps per frame. In Kyutai’s later CALM work, the Depth Transformer took 26.7% of wall-clock time (Kyutai).

Moshi also delays its acoustic codebooks behind the semantic one by $\tau$ frames, so the model commits to what is said before it commits to how it sounds. That costs 80 ms per frame of delay. In Moshi’s ablation, going from $\tau = 0$ to $\tau = 1$ lowers the Whisper negative log-likelihood of the output from 4.36 to 4.12, and $\tau = 2$ buys almost nothing more, at 4.09. Moshi pre-trains at 2 and fine-tunes at 1 (Défossez et al.).

pattern

depth transformerdelay patternflatten

acoustic delay

τ = 0τ = 1τ = 2

large-model steps / s–

small steps / s–

first audio after–

Click any token to see what its prediction is conditioned on. Each column is one 80 ms frame. The user's tokens are heard, never generated. The numbers in each cell are the order of generation. Moshi pre-trains with τ = 2 and fine-tunes with τ = 1: going from 0 to 1 cuts its Whisper NLL from 4.36 to 4.12 for 80 ms of latency, and 2 buys almost nothing more (4.09).

the move to continuous features

The newest systems are moving away from codebooks, so far mostly on the input side. On the input side the move is nearly complete. Most speech language models now read continuous features from a speech encoder, a Whisper-style encoder or Qwen’s AuT, rather than codec tokens, because input only has to condition the model, not be sampled (Qwen team). Thinking Machines reads dMel features, and Boson reads audio at just 6.25 tokens a second.

On the output side it is happening in text-to-speech first. MELLE predicts mel spectrograms autoregressively, DiTAR and VoxCPM generate continuous patches with a diffusion head, and Kyutai’s CALM replaces Mimi’s codebooks with a 32-dimensional latent at the same 12.5 Hz, predicted by a one-step consistency head. CALM beats the eight-codebook RQ-Transformer on quality and runs its per-frame sampler 12.3 times faster (Kyutai). The pattern across the few same-backbone comparisons is that continuous output lowers word error and slightly hurts speaker similarity. The strongest result is a hybrid: VoxCPM keeps a coarse discrete bottleneck under its continuous output, and removing it raises its error rate on hard Chinese from 18.19 to 24.92 (VoxCPM).

Real-time dialogue models have been slower to follow, and I think for a reason. Almost every full-duplex model with a published architecture still emits discrete tokens, NVIDIA’s included; Thinking Machines’ mel flow head is the main exception. Discrete outputs give exact log-probabilities, which reinforcement learning needs. ByteDance ran RL only on the discrete variant of Seed-TTS, and flow-matching models have to be reformulated as stochastic processes before policy gradients apply (Seed-TTS). Continuous models also lose a clean notion of temperature: in CALM it is a heuristic scaling of the noise, and DiTAR and MAR each define it differently. As timing and turn-taking get trained with RL, that matters more than a small gain in fidelity.

the inner monologue

I think the most effective idea in Moshi is about text rather than audio. Kyutai calls it the inner monologue: alongside its own audio, the model predicts its own words, aligned in time. Each word’s text tokens are placed at the frame where the word starts, using word timestamps, and the frames in between are filled with padding tokens, which make up about 65% of the text stream (Défossez et al.). At every 80 ms step, the model writes the word it is about to say as it starts saying it.

The effect is large. With the inner monologue, the negative log-likelihood of generated speech falls from 3.65 to 2.77, coherent continuations run about three times longer (1,920 characters against 602), and spoken question answering roughly triples: 62.3% against 21.0% on Llama Questions (Défossez et al.). The text carries the content, so the audio tokens only have to carry the sound.

The same layout gives speech recognition and synthesis for free. If the text lags the audio, the model hears speech and then writes it down; if the text leads, it reads text and then speaks it. Moshi’s authors get a streaming text-to-speech system with a 4.7% word error rate and a streaming recognizer with 5.7%, from the same model with the offset flipped, and Kyutai’s Delayed Streams Modeling paper turns that offset into a general knob (Kyutai). Latency against accuracy becomes a single number you choose at inference.

None of this closes the gap to text. Moshi scores 22.8% on spoken TriviaQA, against 56.4% for its own text backbone (Défossez et al.). Kyutai’s MoshiRAG closes much of it without changing the voice model, by letting Moshi emit a retrieval token and folding a text model’s answer into its hidden stream: TriviaQA rises from 22.8 to between 69.6 and 78.2 (Kyutai). The same gap is why so many production systems split the work between a talker and a separate reasoner. Qwen3-Omni goes as far as feeding its talker only the thinker’s text, not its hidden states, on the grounds that the text carries the same information (Qwen team).

deciding when to talk

Underneath the four kinds is a simpler question: where the decision about when to talk is made, and how often.

Design	What decides when to talk	How often	Example
Silence timer	Voice activity detection plus a timeout	Once, after silence	Most cascades
Turn-taking head	A small classifier over audio features	Every 80 ms	Speakrail
Decision model	A language model answering multiple-choice questions about the audio	Every 160–200 ms, proposed	DuplexJev
Joint model	The same model that generates the speech	Every frame, as a token	Moshi, GPT-Live

Speakrail shows how far the middle of that table can go. A turn-taking head on top of a Voxtral speech encoder classifies the user as speaking, pausing, finished or backchanneling every 80 ms. A small language model then picks speak, listen, interrupt, yield or interject, in a single token, and starts drafting a reply before the turn is confirmed (Speakrail). It is fast, but the decision and the speech still come from different models. The part that decides to talk can’t hear how the reply is going, and the part that talks doesn’t know why it was told to.

A joint model removes that seam. Moshi models two audio streams, the user’s and its own, each with eight codebooks, plus its own text: seventeen streams per frame. The user’s tokens are input. They are predicted during training but never sampled at inference. The model doesn’t wait for a turn. At every frame it chooses between speech and silence, and silence is just a token. Its theoretical latency is two frames, 160 ms, and about 200 ms in practice (Défossez et al.). OpenAI describes GPT-Live the same way: it makes “interaction decisions many times per second”, whether to speak, keep listening, pause, interrupt or call a tool, and it backchannels on its own (OpenAI). Because the decision and the speech share one state, the model can stop mid-word, say “mm-hm” without losing its place, and keep time with you.

What a joint model learns implicitly, turn-taking research had already written down explicitly. Voice activity projection predicts who will be speaking over the next two seconds. Each speaker’s next two seconds are split into four bins, each bin is speech or silence, and the joint state is one of $2^{2 \times 4} = 256$ classes (Ekstedt and Skantze):

$$p\big(y_t \mid x_{\le t}\big),\qquad y_t \in \{0,1\}^{2 \times 4} \;\cong\; \{1,\dots,256\}$$

A full-duplex model is predicting the same thing, plus what the speech will say, every time it predicts the next frame of both channels.

Most of its assistant behavior comes from synthetic data. Moshi learns turn-taking from 2,000 hours of the Fisher telephone corpus, but its assistant behavior comes from more than 20,000 hours of dialogues written by a language model and rendered by a TTS system, which was itself trained on only 170 hours of real two-channel conversation. The training also simulates the model hearing its own voice back, mixing in echo at a gain of up to 0.2 with delays of 100 to 500 ms (Défossez et al.). I suspect how natural the conversation can get is capped by how natural the synthetic dialogue was.

The cost of putting the policy inside a generative model is that it becomes hard to see and hard to steer, and being fast and being polite turned out to be separate skills. On Full-Duplex-Bench, Moshi had the fastest turn-taking, at 265 ms, but took the floor in 98.5% of the user’s pauses and 100% of their backchannels (Full-Duplex-Bench). In the next version of the benchmark, GPT-4o stopped within 230 ms when interrupted but also answered background speech that wasn’t meant for it 93% of the time (Full-Duplex-Bench v1.5). I think barge-in is less a voice-activity problem than a question of who is being addressed, and every published system sits at one point on a curve between stopping too slowly and stopping for the wrong person. Nobody reports the curve.

Training the timing directly is hard too. Applying GRPO to the raw token stream made a full-duplex model repeat itself and collapse; projecting the action onto a binary speak-or-stay-silent choice halved the duplicated n-grams (ASPIRin). DuplexPO keeps the content model frozen and assigns reward only to timing (DuplexPO). The working answer to credit assignment so far is not to share it: learn when separately from what.

decisions as probabilities

Many of these choices are small closed questions. Is the turn over? Is that “yeah” a backchannel or the start of an interruption? Should this go to the reasoner? That is the kind of question decision models answer: a fixed set of options, and a probability for each, instead of generated text. I’ve written about them for interfaces in Generative interfaces. The same idea shows up outside voice: Ampersand, a text group-chat model from humans&, learns a two-way choice between sending a message and passing, which is turn-taking without audio (humans&).

DuplexJev puts a frozen Qwen3-32B behind a small audio connector and answers multiple-choice questions about the conversation by reading the scores of the answer letters. Ten decisions took 92 ms on one H200, against 1.57 s when the model wrote them out as JSON (DuplexJev). Both numbers, though, were measured on transcripts, excluding the 411 ms of speech recognition, and streaming input wasn’t tested. I read the result as evidence that not decoding saves time, rather than that the model hears faster.

Hume, which built one of the first emotionally aware voice assistants, has moved the same way. Its speech-to-speech documentation now redirects to an Audio Decisions API, launched on November 2, that answers questions about audio clips with probabilities (Hume). Those probabilities are defined as the share of human raters who would give an answer, so 0.5 means raters would split, not that the model is unsure. Clips must be at least five seconds long, which rules it out for turn-taking.

I think the most important open question for this layer is calibration. “Calibrated” currently means three different things: the probability of being right, the share of raters who would agree, and a threshold tuned per language. I could not find a published reliability diagram for any end-of-turn model. Speechmatics, for instance, thresholds its language model’s end-of-turn probability at 0.03 without saying what that probability means (Speechmatics). Until these probabilities mean something you can check, the decisions built on them can’t be compared or composed.

thinking while you talk

The other thing people do during your turn is think. The user’s speaking time is free compute, and several recent papers use it. STITCH alternates unspoken reasoning chunks with spoken ones, thinking while each audio chunk plays. Across five math datasets it averaged 78.04% against 62.98% for the same model without reasoning, with the same number of tokens before the first sound (Chiang et al.). SHANKS reasons silently after each chunk of the user’s speech and completed 56.9% of its tool calls before the user had finished (SHANKS). Stream RAG starts its search queries while the user is still talking and cuts latency by up to 57% (Arora et al.).

The budget is the length of the user’s turn, not the gap after it, so long requests get deep thought for free and short ones get none. Claims of zero added latency deserve a check against the papers’ own tables. FLAIR’s abstract says its latent thinking adds no latency, while its benchmark table shows turn latency rising from 0.37 to 0.43 s (FLAIR). StepAudio 3 Realtime, which runs a speaking copy and a reasoning copy of the same model concurrently, tops Artificial Analysis’s conversational-dynamics score at 98.9 while taking 8.83 s to first audio (StepAudio; Artificial Analysis).

In production, the split between a fast talker and a slow reasoner seems to have won; most of the products above are the fourth kind. What nobody seems to have done yet is train the two together, and doing so raises a credit-assignment problem: when a session goes well or badly, which of the two earned it? A third component makes it worse: the gate that decides when to delegate, where a late reply could be the gate waiting too long or the talker responding too slowly. Written as one objective, it is

$$\max_{\theta_{\text{talk}},\,\theta_{\text{reason}},\,\theta_{\text{gate}}}\; \mathbb{E}\big[\,R_{\text{task}} \;-\; \lambda\, \text{latency} \;-\; \mu\, \text{compute}\,\big],$$

and I have not found a paper that optimizes all three parameter sets against it. Interlat shows that two agents can pass compressed hidden states instead of text, cutting message latency in its setting from 9.19 s to 0.39 s at a cost of four points of task success, but it was tested on household tasks and math, not voice (Interlat).

what the benchmarks miss

The benchmarks are catching up with all this. Full-Duplex-Bench v3 uses real human speech with hesitations and self-corrections, and a request like “book me a flight to New York, wait, make that Boston” breaks every system tested (Full-Duplex-Bench v3). τ-Voice found that voice agents keep only about 30 to 45% of a text agent’s success on the same customer-service tasks (Ray et al.). Artificial Analysis dropped conversational dynamics from its headline index, because the top scores had bunched up between 94 and 99 (Artificial Analysis). Its index also blends the talker’s behavior with the reasoner’s task success, which flatters split systems.

The current leaderboard doesn’t favor full duplex. The top two systems on Artificial Analysis’s index, Phonic-1 at 87.1 and Grok Voice 3.0 at 86.9, win on reasoning, task success and time to first audio, and Grok still uses a voice-activity detector. GPT-Live-1, the only frontier full-duplex model, is third at 83.2 (Artificial Analysis). Human preference has come apart from capability too: Gemini 3.8 Live has the highest arena rating with a τ-Voice score of 30.1, while the best Big Bench Audio score belongs to one of the lowest-rated voices. Reasoning also hides from the latency metric. OpenAI’s GPT-Realtime-2 takes about the same time to first audio at minimal and high reasoning effort while its Big Bench Audio score rises from 71.8 to 96.6, because the thinking happens after the first sound.

Sesame’s TurnBench measures turn-taking against three human annotators per dialogue, with recall at a capped false-positive rate. It found no system that was fast, selective and high-recall at once; Moshi caught only 23% of real turn ends (Sesame). And the failures customers notice most are in the renderer, where no public benchmark looks. Amazon’s May update to Nova 2 Sonic cut cases where spoken digits, emails and addresses diverged from what the model meant to say by 88% (AWS).

What none of them measures yet is the decision layer itself: how calibrated the timing probabilities are, the full curve between yielding fast and yielding to the right person, and how all of it holds up over a long session in a noisy room.

where the turns went

In full-duplex systems, turns moved off the critical path rather than disappearing. OpenAI still reconstructs discrete messages from the continuous stream, because the interface, the logs and the safety systems all consume turns (OpenAI). And much of the recent progress is plain engineering. The same post describes rewriting the media path so that the new 95th-percentile frame timing matches the old median, and cutting connection setup from six network round trips to one.

The old assistants failed because you couldn’t tell what they had understood or when they would talk. Full duplex goes a long way on the second problem. I think the next gains come from the first: decisions about when to speak, when to yield and when to hand off, made as probabilities that mean what they say, and trained together with the models that act on them.