Multimodal and Real-Time Interaction

The previous chapters explored the design of Agents in the text world—interacting with digital systems through context, tools, and code. But an Agent's world is bigger than text and APIs. The moment it needs to understand a user's spoken command, find and click the right button on a screen, or steer a robotic arm to grasp an object, it enters new territory: multimodal real-time interaction—the expansion from pure text input/output to multimodal perception and real-time response, and the crucial step by which an Agent breaks out of the "dialog box." "Multimodal" simply means handling multiple forms of information at once—text, speech, images, video, actions—rather than text alone.

First, the boundaries of this chapter. Static image and document understanding—looking at a screenshot, reading a chart, parsing a PDF—has already slipped naturally into the Agent practices of previous chapters as a perception tool. For today's multimodal LLMs, these "one input, one understanding" tasks are relatively mature and need no special architecture. This chapter tackles a different class of problems: three scenarios where real-time constraints make multimodal problems hard—voice dialogue, GUI operation, and robot control. Here the input flows continuously and the output must arrive within a strict time budget, and that changes the architecture qualitatively. As for real-time understanding of continuous visual streams (video), it remains an open problem for Agents at the time of writing—we will return to it when the Computer Use section confronts the limits of frame-by-frame screenshots, and again in the end-of-chapter questions. One more boundary: multimodal generation (image generation, video generation) is, in this book's framework, just an ordinary tool call (covered in Chapter 5 on Multimedia Generation). The Agent wields it as an external tool; it raises none of the real-time interaction challenges this chapter addresses, so it stays off the main thread.

Voice interaction, Computer Use, and robot operation seem to span three completely different fields, but build them and you get stuck in strikingly similar places: all three must process several modalities at once, and all three are acutely sensitive to latency. A pause of more than two seconds in a voice conversation makes people antsy; millisecond-level jitter in robot control can cause a collision. Together, these two constraints push all three scenarios toward the same architectural destination: from the serial pipeline (like a factory assembly line, where one step must finish before the next begins) to the end-to-end model (a unified model that goes straight from input to output, cutting out the handoffs in between).

This chapter unfolds along the following lines:

1. First, we establish a coordinate system with the "three paradigms of voice architecture"—cascaded (VAD-ASR-LLM-TTS pipeline), end-to-end omnimodal (Omni, single model but still turn-taking), and full-duplex (Moshi, GPT-Live, listening and speaking simultaneously)—and dissect the latency and trade-offs of each link along the axis of "how to break free from VAD's turn assumption." The cascaded section will also discuss how to replace VAD + ASR with streaming voice perception.
2. Next, we examine how the thinking architecture reconciles the conflict between "real-time response" and "deep thinking": from simple parallelization of fast and slow, to the decoupled approach where a background reasoning model acts as a "strategist" (GPT-Live delegation, Pine AI, etc.), to Step-Audio R1's "internalization" of thinking into a single model that "thinks while speaking."
3. Then, we discuss how more human-like speech synthesis optimizes the execution layer.
4. Finally, we extend the perspective to Computer Use (enabling AI to operate a computer screen like a human) and robot operation, observing how the same latency and multimodality issues manifest in these two scenarios.

Two more theoretical points transfer across scenarios and deserve special attention: the thinking architecture (how fast and slow thinking collaborate) and the fast-slow interface derived from it (the Latent Bridge—what fast and slow models can pass each other besides text). Though introduced through voice, they are not confined to it: the Computer Use and robot sections will run into the same question of "when to consult a slow strategist," so keep them in mind.

Voice: The Most Natural Human-Machine Interface

Before dissecting the architecture of a voice Agent, step back and consider the value of voice itself. Of all the ways humans interact with computers, voice has the highest bandwidth and feels the most natural: normal speech runs about four times faster than typing, and it ties up neither hands nor eyes. That is why voice is graduating from a secondary input method to the primary interface of many people's working day—talking to an Agent from morning to night rather than typing word by word.

At the tool level, this path has produced roughly two kinds of products. One is voice dictation tools (e.g., Typeless): they transcribe speech into text in real time and feed it into any application—essentially a keyboard replacement. The other is voice Agents (e.g., Pine, ChatGPT Voice), which the user talks with and works with directly: voice is both the input and the interaction itself. The most telling advanced use of both is the whisper coding mentioned in the introduction—directing a coding or research Agent by speaking: the developer states an intention, hashes it out with the Agent, and the Agent carries out the coding and experiments. Over a dozen papers from this book's author team were completed exactly this way.

One note before we begin: the voice architecture discussed below serves both directions—the user speaking to the Agent (voice as a human-machine interface), and the Agent speaking to the outside world on the user's behalf (say, making a phone call to negotiate). Behind both sits the same real-time voice technology. We start with the three paradigms of voice architecture.

Three Paradigms of Voice Architecture

To clarify the technical evolution of voice Agents, a clear coordinate system is the three-part classification given by OpenAI when it released GPT-Live in 2026[^ch9-12]—which happens to correspond to the three generations of architecture that ChatGPT Voice itself has gone through:

1. Cascaded: Stringing together three models—Automatic Speech Recognition (ASR), Large Language Model (LLM), and Text-to-Speech (TTS)—into a pipeline, passing the baton from one to the next. The earliest ChatGPT Voice was like this. It allowed people to "talk" to a frontier model for the first time, but information was lost during transfer between models, and responses were slow and stiff.
2. End-to-End Omnimodal (Omni): Using a single model to directly "listen to audio, think of a reply, and speak it out," merging the three stages into one. This results in lower latency and preserves non-textual information like prosody and emotion. However, it still assumes "turn-taking"—the model waits for the user to pause before speaking, and turn switching relies on silence detection. A slight pause or background noise can be misjudged as "finished speaking," causing the model to interrupt inappropriately. ChatGPT's Advanced Voice Mode belongs to this generation; OpenAI calls it "turn-based voice models," while the industry more commonly refers to them by their capability as "Omni" models (e.g., Qwen3-Omni). The two names refer to the same thing.
3. Full-Duplex / Interactive: The model listens and speaks simultaneously, processing input and output concurrently, making decisions many times per second about whether to "speak, listen, stop, interrupt, or call a tool," completely eliminating the "turn-taking" assumption. Kyutai's Moshi in 2024 was a research pioneer, and OpenAI's GPT-Live in 2026 brought it to a scale of 150 million users.

One thread runs through all three generations: how to break free of the "taking turns" assumption—of letting VAD (Voice Activity Detection) guess where turns begin and end. Cascaded and Omni architectures both still lean on VAD to carve up turns; only full-duplex dissolves the notion of turns entirely. The next three sections follow this axis. And the three paradigms are not a simple case of new replacing old—they are design choices under different latency and cost constraints, coexisting in production systems in 2026.

Furthermore, GPT-Live introduced a second structural change—decoupling "real-time interaction" from "deep thinking": when encountering a problem requiring search or complex reasoning, the interaction model delegates the task to a background frontier model (GPT-5.5 at launch) while continuing the conversation itself. This thread of "fast-slow division of labor" will be explored in detail in the later section "Trade-offs in Thinking Architectures."

[^ch9-12]: OpenAI. Introducing GPT-Live. 2026-07-08. https://openai.com/index/introducing-gpt-live/ . The three-part classification of "Cascaded / Turn-based / Full-Duplex" in this section originates from this article's summary of the three generations of ChatGPT Voice evolution; the "End-to-End Omnimodal (Omni)" in the text corresponds to its "turn-based voice models" category.

Paradigm 1: Cascading Pipeline

The vast majority of commercial voice assistants—from smart speakers to customer service robots—are based on a serial pipeline (Figure 9-1): Voice Activity Detection (VAD) determines when the user has finished speaking → Automatic Speech Recognition (ASR) converts the audio into text → a Large Language Model (LLM) understands the intent and generates a reply → Text-to-Speech (TTS) vocalizes the reply. Like a relay race, each stage must wait for the previous one to finish before it can start.

Early voice assistants adopted this four-stage serial pipeline for a simple reason: no single model could simultaneously handle speech recognition, language understanding, thinking, and speech synthesis. The modular architecture allowed each component to be developed and optimized independently. However, the cost of modularity is accumulated latency—each stage must wait for the previous one to complete before it can begin.

VAD is the starting point of the pipeline, continuously monitoring the audio stream. The most critical design is End-of-Speech Detection: typically, a continuous silence threshold of 500-800ms is set—if the user stops speaking for more than half a second, VAD considers the utterance finished. This introduces the first layer of latency, and it's a difficult trade-off: if the threshold is too short, a pause for thought is misjudged as the end, cutting the sentence short; if it's too long, the user has to wait for nearly a second after finishing before getting a response.

ASR converts the audio waveform into text. Models like Whisper and SenseVoice, when transcribing 5 seconds of audio with a small to medium-sized model deployed on a GPU, typically take 50-200ms; larger models or resource-constrained deployments can take 200-500ms (the control group in Experiment 9-3 falls into the latter category). The more critical problem: throughout the VAD wait and the ASR transcription, the LLM downstream sits completely idle—it has received nothing and cannot start thinking early.

LLM inference, even well optimized, has a Time to First Token (TTFT) of 100-500ms depending on context length, plus another ~100ms to finish the first sentence. Enable reasoning, and that stretches to 5-10 seconds. In the traditional architecture, TTS cannot start until the LLM has produced the complete reply text.

TTS converts the reply text into speech, typically taking 200-500ms for synthesis. Summing up the latency of each link (Figure 9-2): VAD (500-800ms) + ASR (50-200ms) + LLM (100-500ms) + TTS (200-500ms), totaling approximately 0.9-2 seconds—and this is the ideal case with all services idle and no queuing.

Once in production, queuing latency makes things worse still. It works like a restaurant: the busier the kitchen, the longer the wait—and the wait doesn't grow linearly, it skyrockets (Figure 9-3). When a server has no waiting queue (i.e., it is "idle"), the time to process a request is called idle latency. But when multiple requests arrive at once, latecomers must queue.

Intuitively, the higher the utilization, the more steeply—and nonlinearly—the wait time climbs. The specific mathematical relationship can be given by queuing theory (for intuitive understanding here, no rigorous derivation is needed): Total Latency ≈ Idle Latency × 1/(1-Utilization). Utilization refers to the proportion of time the server is busy; for example, 50% utilization means the server is processing requests half the time and idle half the time. At 50% utilization, latency doubles; at 80%, it is five times the idle latency—which is why servers cannot run under high load for long.

Experiment 9-1 ★: Building a Traditional Voice Agent

This experiment builds a complete real-time voice dialogue system, allowing users to interact with an AI via voice through a microphone. The system uses a front-end/back-end separation architecture with real-time communication via WebSocket.

The core process follows a strict serial pattern: the front-end captures microphone input and sends it to the back-end in real time via WebSocket. The back-end runs a Silero VAD model for voice activity detection, which offers higher accuracy and better noise resistance compared to traditional volume-based detection methods. After detecting approximately 500ms of continuous silence, the audio segment is extracted for subsequent processing.

The ASR, LLM, and TTS stages each support flexible switching between multiple providers, allowing developers to choose the optimal combination based on latency, accuracy, and regional network conditions.

Experiment 9-2 ★: Building a Phone Agent Using the PineClaw Voice API

Experiment 9-1 built an in-browser voice dialogue system, but many real-world Agent tasks require making actual phone calls—contacting customer service to negotiate bills, booking restaurants, confirming orders. Chapter 4 demonstrated, through PineClaw's Channel mechanism, how an event-driven architecture can reduce the response latency for phone notifications from minutes to seconds. This experiment focuses on building the voice call itself. Taking the PineClaw Voice API (developed by the author's team) as an example, such production-grade telephony voice APIs typically encapsulate the entire process of dialing, IVR navigation (i.e., "For inquiries, press 1; for an operator, press 0" phone menus), conversation, and transcription: the Agent provides the phone number, goal, and context information, and the voice Agent completes the entire call, returning a structured call record.

Experiment Goal: Build an Agent capable of completing tasks via real phone calls, integrating PineClaw Voice as a tool into the ReAct loop.

Technical Approach: Use the PineClaw Voice Python SDK (pine-voice) to equip the Agent with a make_phone_call tool. The Agent receives the user's task description (e.g., "Help me book a dental checkup for tomorrow at 3 PM"), and through ReAct thinking, decides: (1) which phone number to call; (2) the goal and key information for the call; (3) how to report the results to the user after the call ends.

The Agent's workflow: User says "Call the clinic to book an appointment for tomorrow" → Agent thinks about what information is needed (clinic phone number, appointment time, patient name) → If information is insufficient, asks the user for clarification → Calls the make_phone_call tool → PineClaw dials the number, converses with the other party, and completes the booking → Agent receives the call summary and transcript → Reports the result to the user.

Acceptance Criteria: Successfully make a test call (can first call your own phone to verify connectivity). The Agent can autonomously determine call parameters based on the task description. After the call ends, it correctly extracts key information (appointment time, confirmation number, etc.) and reports it to the user. Compare the difference between using the API directly and calling it through the Agent's ReAct loop—the latter can handle situations with incomplete information (e.g., searching for the phone number if the user didn't provide it).

This experiment demonstrates an important application direction for voice Agents: Agents can not only have voice conversations with users but also interact with the outside world via phone calls on behalf of the user. PineClaw's voice Agent is specifically trained to handle hour-long waits, phone menu navigation, and complex negotiations—imagine having an AI call your carrier's customer service line for you and wait on hold for a human operator. These are precisely the scenarios where traditional serial voice pipelines struggle.

Full-Chain Streaming of the Cascaded Pipeline

A common misconception needs clarification: the 0.9–2 second latency tally above assumes a fully serial scenario where "each link finishes before passing the baton." However, production systems in 2025 no longer operate this way. The mainstream approach is not to abandon modularity, but to retain the VAD-ASR-LLM-TTS division while making each stage streaming, allowing adjacent stages to overlap in time:

• ASR transcribes while listening: Using streaming recognition, text is continuously produced while the user is still speaking, without waiting for VAD to determine the end of a sentence before starting transcription.
• LLM outputs in sentence chunks: As the model generates, it splits the reply into small sentences based on punctuation or semantics. The first sentence is sent downstream as soon as it takes shape, rather than waiting for the entire reply to be written.
• TTS streams at the sentence level: It starts synthesizing and playing the first small sentence as soon as it arrives, with subsequent sentences generated and added on the fly. This significantly advances the time the user hears the first syllable.

As a result, the three stages—ASR, LLM, and TTS—are no longer in a relay-like sequential relationship but operate like three workstations on an assembly line working simultaneously. Open-source frameworks like LiveKit Agents and Pipecat, as well as mainstream commercial outbound call systems, all follow this approach. After full-chain streaming, end-to-end latency can typically be compressed to 600–800ms, significantly better than the 0.9–2 seconds of fully serial processing.

But streaming can only compress what can overlap—transcription, thinking, synthesis. One segment of latency it cannot touch: VAD's silence wait and turn judgment itself. The system still leans on a 500–800ms silence threshold to guess whether the user has finished speaking, and since that wait is the precondition for the pipeline to start at all, no amount of overlap removes it. Compressing this last segment means shifting attention from "overlapping the stages" to the perception stage at the very front.

Streaming Voice Perception: Replacing VAD + ASR

This perception front-end consists of two stages—VAD determines if the user has finished speaking, and ASR transcribes audio into text. Together, they determine when the entire pipeline starts and what input it receives. The traditional VAD + ASR cascade has three fundamental problems:

1. Latency Accumulation: VAD must wait for 500–800ms of silence to confirm the user has finished, because it cannot predict the future and can only rely on "waiting" to distinguish between "truly finished" and "just pausing to think."
2. Information Loss: VAD only outputs a binary signal like "voice/silence." All acoustic details—emotional changes, tone fluctuations, hesitant pauses, background environment—are lost. Misjudgment issues are particularly prominent in complex environments: a slightly longer user pause is misjudged as finished, causing sentence truncation; background noise triggers false starts, causing the system to process when no one is speaking; and a user's "uh-huh" cannot be determined as an interruption or an acknowledgment.
3. Decreased Accuracy: VAD cuts continuous audio into independent segments, each sent to ASR for recognition, disrupting contextual continuity. Content that requires context for correct recognition (email addresses, brand names, person names, proper nouns) sees a significant increase in error rates—for example, if a user says "john dot smith at gmail dot com" and "john" and "smith" are cut into different segments, "smith" might be misrecognized as "miss" due to lack of context.

Streaming voice perception models offer a fundamental fix. First, pin down what "streaming" means technically: whether a voice model can stream hinges on whether its encoder is causal or chunked (relying only on audio that has already arrived, never needing the whole recording) and whether its decoding is incremental (emitting partial results as each small audio chunk comes in). Whisper cannot stream—not because of its decoding, which is autoregressive anyway, but because its encoder needs a complete audio segment (a fixed 30 seconds, padded if shorter) before it can start. Note also that streaming recognition itself is not new: traditional streaming ASR, represented by RNN-T and streaming Conformer, has long been deployed at scale in industry—live captions on phones and voice typing in keyboard apps use exactly such models—and it has nothing to do with LLMs.

This section focuses on a new path: LLM-based streaming auditory perception—using an open-source LLM as a backbone for post-training, allowing the model to directly output semantic-level responses from a continuous audio stream, merging "recognition" and "understanding" into a single model. It is an upgrade to traditional streaming ASR, not an invention of streaming technology: the latency of incremental recognition remains on the order of single-step inference time (tens to a couple hundred milliseconds), but the model no longer sees isolated fragments cut by VAD. Instead, it sees a continuous audio stream from the start of the conversation to the current moment, enabling In-Context Learning based on complete context, significantly improving recognition accuracy for user personal information, professional terms, and pronunciation habits.

Another key advantage of this path is inheriting the world knowledge and common sense reasoning capabilities of LLMs—after all, the backbone model has seen vast amounts of text. For example, the model knows that "Apple" followed by "event" likely refers to the company rather than the fruit. This knowledge enhancement makes recognition accuracy for high-value information like amounts, place names, and brand names far exceed traditional ASR. This path already has deployable models, such as Fixie's Ultravox—which feeds audio directly into an LLM backbone and outputs text and semantic tokens. The Qwen2-Audio used in this section's experiments, and Alibaba's Qwen2.5-Omni, also belong to this category of audio-native models.

However, replacing VAD does not necessarily require using a full-scale audio LLM. If the goal is only to solve the first problem—determining whether the user has finished speaking—there is a lighter path: embedding this "turn judgment" directly into the recognizer itself[^ch9-11]. The approach: add a LoRA to a small open-source streaming recognition model so that it transcribes while weighing semantics and silence together to judge whether the sentence has expressed a complete thought—necessary because intra-turn pauses (a beat while reciting a phone number) often run longer than the gaps between turns, so a silence threshold alone is bound to fail in both directions. The more interesting finding: when the model keeps wavering over whether to take the turn, the root cause is usually not the architecture but training labels annotated from a "God's eye view"—the annotators used audio that appeared after the decision point, which the online model can never see. Re-annotate every label using only the information available at the decision moment, and the spurious wavering disappears. This echoes a judgment from Chapter 7 on post-training: often, data is more critical than architecture. This lighter path also has production-grade implementations: Deepgram's Flux and AssemblyAI's Universal-Streaming embed endpointing and turn detection directly into streaming recognition models, designed specifically for voice Agents; on the open-source side, LiveKit and Pipecat provide semantic turn detection models.

[^ch9-11]: The diagnosis of embedding turn judgment into the recognizer and the "God's eye view" of labels can be found in Li, Bojie and Noah Shi. The Trade-off Was in the Labels: Causal Supervision for Turn-Aware Streaming ASR. 2026 (forthcoming).

The model outputs not only text but also a series of special markers for acoustic events—these are dedicated tokens introduced during model training. The model learns to automatically output them when detecting corresponding acoustic events. Common types include:

• <speak_start/end>: Determines speech start and end based on a comprehensive judgment of semantics and acoustics, rather than simple silence detection.
• <interrupt>: Distinguishes whether the user truly wants to interrupt or is just acknowledging or affected by background noise.
• <emotion:happy/frustrated>: Emotion markers.
• <laugh> / <sigh>: Paralinguistic signals like laughter and sighs.
• <music> / <noise>: Environmental sounds.

These markers, along with text tokens, form a unified event stream that is fed into the thinking layer.

Input audio: "Um, actually I think... no wait, let me reconsider."
Model output stream:
  <speak_start> Um, <emotion:hesitant> actually I think...
  <silence:500ms> no wait, <emotion:confident> let me reconsider <speak_end>

Note that the model outputs not just a text transcription but also voice event markers (start/end of speech, emotion changes, silence intervals). Agent frameworks can leverage these markers for more natural interactions—for example, proactively offering options when detecting user hesitation.

Experiment 9-3 ★: Simulating Streaming Voice Perception with Qwen2-Audio

The experimental design needs clarification first: Qwen2-Audio itself is a non-streaming model that takes entire segments as input. This experiment uses chunked input to simulate streaming processing—cutting the continuous audio stream into fixed-length small chunks, each sent to the model along with the accumulated audio context. The model gradually generates text and acoustic event tokens (such as laughter, pauses, and other non-verbal signals), and measures the latency from inputting each chunk to producing text. There is a key cost here: Qwen2-Audio's encoder is not incremental. Each time a new chunk is processed, all previously accumulated audio must be re-encoded from scratch. Therefore, the longer the conversation and the more accumulated audio, the higher the encoding latency for each chunk—this is the essential difference between "simulated streaming" and "true streaming" (which uses incremental or causal encoders that only incrementally encode the newly arrived small audio segment). This design can demonstrate the accuracy benefits of "continuous perception with complete context," but the latency numbers only reflect chunk granularity and inference speed, not the first-packet latency of a truly streaming-designed model (such as Qwen3-Omni with chunked encoding); interested readers can replace it with the latter to redo the experiment. The comparison baseline is the traditional VAD + Whisper ASR pipeline. Three scenarios are tested: normal conversation, long sentences with pauses, and conversation with background noise.

Results: The incremental recognition latency of the chunked simulation scheme can be controlled on the order of one to two hundred milliseconds (depending on chunk length and hardware), while the traditional scheme requires waiting for VAD to confirm completion (600ms) plus Whisper inference (about 200–500ms under this experiment's configuration), totaling 800–1100ms. In the scenario with pauses, VAD misjudged the first long pause as the end of speech, cutting the sentence into two segments for separate recognition. "大概两点左右" ("around two o'clock") was misrecognized as "大概零点左右" ("around midnight")—stripped of the surrounding context, the similar-sounding 两 (liǎng, "two") was misheard as 零 (líng, "zero"). The chunked scheme, maintaining complete context, correctly recognized the entire sentence. In the background noise scenario, Qwen2-Audio outputs <|noise|> tokens to mark the presence of noise without interrupting recognition, while traditional VAD was falsely triggered by noise, causing the recognition process to start prematurely.

Paradigm 2: End-to-End Omnimodal Models (Omni)

Look back at the cascaded pipeline as a whole: even with its perception front-end upgraded to streaming voice perception, it still parcels out listening, thinking, and speaking to three independent models joined by a discrete interface. However wide that interface grows, it carries only a handful of semantic tokens and the occasional acoustic marker—the speaker's emotion, tone, and intonation, the ambient sound and music in the background, are mostly lost in the handover. And because the three segments are trained and tuned separately, they struggle to work as one. End-to-end omnimodal models (Omni) take a different path—a single model directly "listens" to audio, "thinks" of a reply, and "speaks" it, merging the three segments into one (Figure 9-4). Given enough training data, the model's internal latent space can carry these paralinguistic signals straight through to the generation end, beyond what text can convey: lower latency, prosody and emotion preserved. The trade-off: cascaded pipelines have clean modules, per-segment tuning, and good interpretability; end-to-end models buy lower latency and non-textual fidelity at the cost of heavier training data demands and poorer interpretability.

One dimension is routinely overlooked: end-to-end's advantage is chiefly in latency; on accuracy it does not necessarily win. The instructive comparison is self-cascade—the same model first transcribes the audio into structured text, then reasons over that text. Whether self-cascade or a single end-to-end pass is more accurate depends on the task, and the pattern is clean: when the answer is determined mainly by semantic content ("what was said") and the intermediate text can carry the task-relevant information, self-cascade matches or beats end-to-end—especially for models with weaker perception. When the answer hinges on non-verbal cues that text struggles to represent (tone, emotion, environmental sounds), end-to-end pulls clearly ahead. More important, which side wins can be predicted in advance from the nature of the task, rather than chalked up to "end-to-end is more advanced." A design principle follows: what decides performance is usually not whether an intermediate representation—a bottleneck—exists, but what information that bottleneck carries. Upgrade the intermediate text from bare transcription to a structured representation with paralinguistic markers (emotion, speech rate, environmental sounds), and end-to-end's accuracy edge often narrows. This is the same claim made earlier in "Streaming Voice Perception": the perception layer should not output plain text alone[^ch9-13].

Yet however powerful Omni gets, it has in essence only merged three models into one, without discarding the assumption of "taking turns": it still relies on VAD to allocate the floor—falling silent the moment it detects the user speaking, starting up the moment the user goes quiet. So the familiar failure resurfaces: the user recites a string of numbers, pauses for a beat, and Omni decides they are done and cuts in. The streaming voice perception described earlier lifts turn judgment from silence duration to the semantic level and greatly reduces such misfires—but that is still a local patch within the turn-taking framework, not the end of turn-taking itself. To escape for good, one must stop patching inside the framework: let the model listen and speak simultaneously and decide for itself when to talk, with no hard switch of "whose turn it is."

[^ch9-13]: For a complete cross-modal measurement of when the accuracy advantages of cascade and end-to-end reverse, and how to predict the direction based on task nature (whether the intermediate representation can sufficiently carry task-related information), see Li, Bojie and Noah Shi. The Cascade Gap: When and Why Self-Cascades Help Multimodal Agents. 2026 (forthcoming).

OpenAI Realtime API is close to end-to-end at the model level (the model natively processes audio), but still relies on traditional VAD at the interaction control level, making it an intermediate solution transitioning to full end-to-end. It initially (2024 preview) ran on GPT-4o, and after its official GA in 2025, it switched to a dedicated voice model gpt-realtime (no longer a mode of GPT-4o, but a model specifically optimized for real-time voice). The API enables server-side VAD by default, automatically determining when the user starts and stops speaking. It supports interruption during conversation—detecting when the user starts speaking and immediately stopping current voice generation, much like when one person interrupts in a face-to-face conversation and the other naturally stops. gpt-realtime also introduces asynchronous function calls: the model can continue talking to the user while waiting for tool results, hiding tool latency within the conversation process. These improvements enhance the experience, but are essentially optimizations within the VAD framework. Gemini Live API has a similar approach, supporting VAD sensitivity configuration and retaining sent information upon interruption to ensure conversational coherence.

Qwen3-Omni adopts a Thinker-Talker architecture: separating thinking (understanding and reasoning) from expression (voice generation) into two specialized modules, unifying perception and generation for text, images, audio, and video.

To keep capability high while controlling computational cost, Qwen3-Omni adopts the MoE (Mixture of Experts) architecture—think of it as "calling expert teams on demand": it contains multiple small expert networks internally, and for each inference, only the few most relevant to the current task are activated, while the rest do not participate in computation. For example, when processing speech, it primarily activates speech-related experts; when processing images, it primarily activates vision-related experts. This allows the model to have a very large total parameter count (ensuring high capability) while keeping the actual computation per token very small, thereby improving inference throughput and reducing queuing latency under high load.

A distinction worth keeping straight: MoE solves throughput—how many requests a unit of compute can serve. It does not directly determine how early the first audio packet goes out; first-packet latency depends on the generation-side architecture. Qwen3-Omni's low first-packet latency comes from its Talker module: it generates audio tokens incrementally in a multi-codebook autoregressive manner, and a causal codec decodes those tokens into waveform incrementally. The moment the thinking module produces text, the Talker can start streaming speech synthesis—no need to wait for the full response. According to the official report, its cold-start theoretical first-packet latency is as low as approximately 234ms, supports understanding in 19 languages and generation in 10 languages, and leads in 22 out of 36 audio-video benchmarks.

Step-Audio 2 takes a different route: it directly processes raw audio input and outputs both text and audio, achieving true end-to-end voice conversation. It can not only understand what is said (semantic information) but also perceive how it is said—paralinguistic information, such as whether the speaker's emotion is happy or angry, whether the speech rate is rapid or hesitant, whether the intonation is rising or falling—as well as background environmental sounds and music. It generates expressive responses through thinking and reinforcement learning, and also integrates a RAG mechanism and external tools (web search, audio search). According to the Step-Audio 2 paper, on their proposed StepEval-Audio-Paralinguistic benchmark for paralinguistic understanding, Step-Audio 2 achieves an accuracy of 83.09%, leading over the contemporaneous open-source omnimodal model Qwen2.5-Omni (44.18%), and also surpassing GPT-4o Audio (43.45%) and Kimi-Audio (49.64%).

Step-Audio R1 is a follow-up work in the Step-Audio series. Building on Step-Audio 2's end-to-end voice conversation architecture, it further internalizes thinking capabilities directly into the audio model. The two represent a progressive evolution along the same technical path.

Paradigm 3: Full-Duplex / Interactive Models

Paradigm 2 merged three models into one but still clung to the assumption of taking turns—either the user speaks or the model speaks, with the switch point guessed by VAD or semantics. Some scenarios simply have no room for "your sentence, then mine." Simultaneous interpretation is the classic case: the interpreter does not wait for a complete sentence before starting, but listens and composes at once, rendering each unit of meaning as soon as it is roughly whole—listening and translating always overlap. Rhythm games, where you strike drumbeats in time with music, are more extreme still: the ear must track an uninterrupted music stream, the hands must hit each beat on the instant, and the mind must anticipate the next one—here there is no such thing as a "turn," only an unending stream of input. Such tasks challenge the turn-by-turn model at its root: they demand that listening, thinking, and action happen simultaneously, while the turn-based model's whole premise is to slot the three into separate time slices. The full-duplex model takes the "eliminate VAD" path to its logical endpoint—it simply drops the turn-taking assumption and lets the model listen and speak continuously, at the same time.

The pioneering research work here is Kyutai's Moshi (2024). It models two audio streams in parallel (the user's voice and the model's own voice), supplemented by an "inner monologue" text stream to improve the linguistic quality of the generated speech. Since it is always listening, overlapping speech and interruptions become natural behaviors, requiring no explicit interruption detection logic. End-to-end latency is approximately 200ms, approaching the natural rhythm of human conversation.

In 2026, Thinking Machines Lab, founded by Mira Murati, previewed a new category they call the Interaction Model[^ch9-14], and made explicit the claim behind full-duplex: interactivity should not be an external harness like VAD wrapped around the model, but should be built into the model itself. In their words, "for interactivity to scale with intelligence, it must become part of the model itself." Architecturally, this translates to micro-turns: instead of waiting for an entire turn to finish, the model works in segments of roughly 200ms—continuously "reading in 200ms, generating 200ms"—allowing audio, video, and text streams to interleave and advance together. This granularity is a deliberate compromise—fine enough that silence, overlap, and interruption are preserved as continuous streams in the model's context, with no artificial turn boundaries to accommodate; yet coarse enough to process multiple modalities in chunks concurrently, keeping latency within a perceptually real-time range. Because interaction lives inside the model, behaviors that once had to be assembled from specialized harnesses—listening while speaking, watching while interjecting—are now simply part of the model's job, and they strengthen as the model does. The first model, TML-Interaction-Small, trains all three streams together from scratch; when it notices the user writing a piece of buggy code, or someone stepping into the frame, it can speak up unprompted.

Its approach to "slow thinking" is also representative. The interaction model itself is only responsible for keeping the conversation online. When it encounters a problem requiring deep reasoning or tool calls, it delegates to a stronger reasoning model in the background—what it hands over is not an isolated query, but the entire conversation context. While the background model reasons, results stream back incrementally. The interaction model then chooses a moment that does not interrupt the user to naturally weave the result into the conversation, all the while continuing to respond, answer follow-ups, and hold the floor. In this way, it delivers the "planning, tool, and agent capabilities of a reasoning model" with the "latency of a non-thinking model." According to the official report, TML-Interaction-Small (276B parameter MoE, 12B activated) achieves a turn-switching latency as low as approximately 0.40 seconds (GPT-realtime-2.0 is about 1.18 seconds), and significantly outperforms competitors that score nearly zero on benchmarks for visual proactivity; as of writing, it is still in the research preview stage.

[^ch9-14]: Thinking Machines Lab, "Interaction Models: A Scalable Approach to Human-AI Collaboration," 2026-05. https://thinkingmachines.ai/blog/interaction-models/

In the same year, OpenAI's GPT-Live brought full-duplex to production scale, rolling out globally as the new default voice model for ChatGPT. It no longer treats conversation as a series of discrete message turns, but instead continuously processes input while continuously generating output. Therefore, it can make many interaction decisions per second: whether to start speaking, continue listening, pause, interrupt, or call a tool. The result is that it waits quietly when the user is thinking instead of interrupting, uses acknowledgments like "mm-hmm" and "right" to show it is listening, and is also capable of tasks like real-time translation that require listening and speaking simultaneously.

GPT-Live also follows the same path of separating fast and slow processes—decoupling "real-time interaction" from "deep thinking": when it encounters a task requiring search, reasoning, or more complex agent operations, the interactive GPT-Live delegates the task to a frontier model in the background (at launch, GPT-5.5), while continuing to maintain the flow of conversation itself. Once the background model produces a result, it brings it back into the conversation. GPT-Live-1 and the mini version use GPT-5.5 Instant in the background, while the Medium and High tiers call the thinking-enabled GPT-5.5, allowing users to choose between "fast" and "deep" as needed. This "fast-slow division of labor" is precisely the topic to be expanded upon in the next section, "Trade-offs in Thinking Architectures."

Reviewing this chapter's narrative thread of "replacing VAD": VAD guesses the turn-switching point based on silence thresholds; streaming perception (see the earlier section "Streaming Voice Perception" in Paradigm 1) upgrades the switching judgment to the semantic level; and the full-duplex model completely dissolves the concept of "switching" itself—it is always listening, so "interruption" is no longer an event requiring special handling, and the barge-in processing chain is largely eliminated architecturally. This is the endpoint of the "replacing VAD" narrative thread as of the time of writing.

Trade-offs in Thinking Architectures: From Separation to Unification

The real problem to solve is the contradiction between real-time response and deep thinking: users expect millisecond-level responses, while complex problems require seconds of thinking time. How can the model think deeply enough while maintaining low latency? This contradiction is not unique to end-to-end architectures; cascaded pipelines also face it.

The three solutions below are not a linear technological iteration—they are design trade-offs for different constraints, coexisting in practice. The choice depends on the application's requirements for latency and depth of thinking. It is necessary to first clarify the distinction between the three: Solutions 1 and 2 are essentially a "fast-slow division of labor" with two independent models running concurrently. They do not depend on end-to-end and can even be applied on top of a cascaded pipeline. Only Solution 3 truly internalizes thinking into the end-to-end model.

By 2026, the "fast-slow decoupling" path had become the mainstream choice for frontier voice products and acquired a name of its own. Thinking Machines Lab calls it "Interaction Models"—a real-time interaction model coupled with an asynchronous background reasoning model; xAI's Grok Voice "Think Fast," Pine AI's voice Agent, and the previous section's GPT-Live "delegation" all follow the same route of "fast in the foreground maintaining the conversation, slow in the background for deep reasoning." The choice to decouple rather than "train a single all-powerful model" has a pragmatic reason: frontier reasoning models iterate every few months, while real-time interaction capabilities require specialized data and training objectives. Stuffing both into the same model means chasing a moving target and potentially diluting the most valuable reasoning capability[^ch9-8]. Conversely, by keeping the strongest reasoning model intact in the background and only training a lightweight interaction model for the foreground, one can always use the current strongest "brain"—this is precisely why GPT-Live emphasizes "sustainable swapping to the latest frontier models." Below, we examine the three solutions in order of "coordination mechanism from weak to strong."

Solution 1: Fast Thinking for Fillers, Slow Thinking for Answers

Fast and slow thinking run in parallel (Figure 9-5): fast thinking produces a brief holding reply within 500ms (the way a person first says "let me think"), while slow thinking spends 5-10 seconds reasoning in the background before delivering the full answer. The technique behind slow thinking is "test-time scaling"—in plain terms, letting the model think a bit longer before answering: instead of jumping to an answer in one step, it works like a person on a math problem—sketch an approach, derive step by step, check the result—trading more computation for a better answer.

Problem 1: Overthinking simple questions. The user asks "What day is it today?" Fast thinking correctly answers "Wednesday" within 500ms, but slow thinking still runs the full 10 seconds of thinking and then repeats "Wednesday." This not only wastes computational resources but, more critically, disrupts the conversation rhythm—the user already has the answer and is ready to move on, only to be interrupted by a repeated response. Problem 2: Inconsistency between fast and slow. The two run independently in parallel. They see the same context, but their reasoning paths can diverge completely—fast thinking answers on the strength of one assumption, slow thinking discovers that assumption is false and lands on the opposite conclusion. Within seconds the user hears the system contradict itself, and trust collapses on the spot. The root cause: Solution 1 splits the conversation into two independent thinking processes rather than one coherent cognitive activity, with no coordination mechanism between fast and slow.

<user>Is this plan suitable for me?</user>
<!-- Fast thinking after 0.5 seconds -->
<assistant (fast thinking)>This plan is very affordable, I recommend you purchase it.</assistant>
<user>Okay, then I'll...</user>
<!-- Slow thinking completes after 8 seconds -->
<assistant (slow thinking)>Wait, I found that this plan lacks the international roaming feature you need, so it might not be suitable.</assistant>
<user>(Angry) So do you recommend I buy it or not?!</user>

Solution 2: Fast Thinking for Interaction, Slow Thinking for Advice

Solution 2 allows slow thinking to see the output of fast thinking. It provides suggestions to fast thinking via the Agent Status Bar (the dynamic meta-information injection mechanism introduced in Chapter 2), rather than speaking directly to the user. Over Solution 1 this improves two things: slow thinking runs asynchronously in the background, using the gaps in speech to keep thinking; and because it can see fast thinking's output, it no longer clashes with it head-on, retreating instead to the role of behind-the-scenes "strategist." The GPT-Live delegation and Pine AI voice Agent mentioned earlier are production examples of Solution 2—the background reasoning model sends its conclusions back to the foreground interaction model via a concise text channel, and the foreground model decides when and how to phrase it for the user.

This solution still has fundamental limits, though. Fast thinking may not follow instructions—communication between two independent thinking instances is indirect and ambiguous. Fast thinking can misread the Agent Status Bar: it might take "the price needs to be reconfirmed" to mean "ask the user whether this price is acceptable," when it meant "the price was calculated wrong—redo it." No visibility into intermediate thinking—slow thinking's 10 seconds of reasoning produce plenty of valuable intermediate conclusions, but fast thinking sees none of them; it can only wait for the final Agent Status Bar. If the user asks another question or interrupts before slow thinking finishes, fast thinking must answer on its own limited understanding. It is like two people solving a problem together but communicating only by passing notes, never seeing each other's scratch paper.

Solution 2 also faces a fundamental theoretical problem: it cannot achieve "thinking while speaking." When humans face a complex problem, they do not first formulate the complete answer in their mind and then speak it all at once; instead, they think and speak in segments—"This is an interesting question... (pause to think) First, we need to consider... (continue thinking) Secondly..." In Solution 2, fast thinking can only mouth filler words while it waits on slow thinking, with no way to weave the thinking process naturally into the conversation.

Solution 3: End-to-End Unification of Thinking and Expression (Using Step-Audio R1 as an Example)

Although Solution 2 solves the waiting problem for slow thinking, it is still architecturally "think first, then speak"—thinking and expression remain two separate processes, making it impossible to achieve the human-like "thinking while speaking." To break this fundamental limitation, thinking capabilities must be directly internalized into the model.

Step-Audio R1 proposes a fundamentally different solution along this direction: it internalizes thinking capabilities directly into the end-to-end audio language model, achieving true "thinking while speaking" through a dual-brain architecture. It actually consists of two complementary mechanisms, each solving a different problem: Modality-Grounded Reasoning Distillation (MGRD) first solves "thinking correctly"—ensuring the model truly reasons based on acoustic features rather than text transcripts; MPS Dual-Brain Architecture then solves "speaking in a timely manner"—enabling thinking and expression to run in parallel for low-latency thinking while speaking. The former is the prerequisite for the latter: only when thinking is rooted in sound is thinking-while-speaking worth having. We take each in turn.

The Textual Surrogate Reasoning Problem. Ideally, a voice model should directly analyze acoustic features (such as pitch, rhythm, and intonation) to understand a speaker's emotion or intent. However, many models in practice take a shortcut: existing audio language models exhibit a counterintuitive phenomenon where longer chains of thought lead to worse performance. The Step-Audio R1 team identified the root cause as "Textual Surrogate Reasoning" (using textual information to "stand in" for acoustic information for analysis): when the model "thinks," it is actually performing semantic reasoning based on text transcription, rather than genuinely analyzing acoustic features. For example, when asked to judge the emotion of a song, the model analyzes "the lyrics mention sadness," rather than "the minor key melody combined with a descending pitch contour conveys a feeling of sorrow." This modality mismatch originates from the training data: most audio models' CoT (Chain-of-Thought) data is generated by text models, which naturally inherit a pure-text thinking pattern.

Modality-Grounded Reasoning Distillation (MGRD) addresses this issue through iterative self-improvement (Figure 9-6). The name is a mouthful, but the core idea is intuitive: select the thought processes that are genuinely listening to the sound, and train the model on those—teaching it to analyze with its ears like a music teacher rather than skim the lyrics like a copy editor. Three steps:

1. Have the current model generate several different thought processes for the same audio segment, then keep only those genuinely grounded in acoustic features. How to tell? Check whether the thought mentions concrete sound parameters. For example, for an angry voice input, a text-based thought would be "The user said negative words like 'too bad,' so I judge it as anger"—this is only analyzing the text content; an acoustic-feature-based thought would be "The speaking rate is 40% faster than normal, the volume is significantly higher, and the pitch is sharper"—this is truly "listening" to the sound. MGRD selects the latter.
2. Retrain the model using these high-quality thought data to strengthen its "think with your ears" ability.
3. Further optimize through reinforcement learning to prevent the model from taking shortcuts by skipping the thinking process and guessing the answer directly.

After multiple iterations, the foundation of thinking gradually shifts from text abstraction to acoustic analysis—the model begins to focus on "the pitch contour drops sharply at 1.2 seconds" rather than vaguely stating "the speaker seems unhappy."

MPS Dual-Brain Architecture (Mind-Paced Speaking) addresses the latency contradiction between thinking and speech output (Figure 9-6). Its inspiration comes from the division of labor in the human brain: the areas responsible for thinking and those responsible for organizing language are separate and can work in parallel—you think of the next sentence while your mouth is still speaking the previous one. MPS uses two models to simulate this division: the Formulation Brain is responsible for continuous thinking, producing segments of thought results; the Articulation Brain, upon receiving each new segment of thought results, combines it with previous thoughts and the existing reply to convert it into a speech response.

Both run in parallel—the Formulation Brain doesn't need to finish thinking everything before the Articulation Brain starts speaking. For example, at t=0ms, the Formulation Brain starts analyzing the user's question; at t=200ms, it outputs the first segment of thought results (a sequence of text tokens); the Articulation Brain receives this result at t=200ms, combines it with the generated reply context, and starts outputting the corresponding speech tokens at t=350ms—the two modules operate in a pipelined, parallel fashion, and the user hears the first syllable at t=350ms.

Experiment 9-4 ★★★: Using Step-Audio R1 for End-to-End Speech Thinking

This experiment uses the Step-Audio R1 model to compare the performance of different configurations on speech thinking and dialogue tasks. Step-Audio R1 consists of an audio encoder, an audio adapter, and a Qwen2.5 32B decoder, requiring multi-GPU deployment.

This experiment evaluates two tasks: Spoken-MQA (Spoken Math Questions) tests whether the model can perform multi-step mathematical reasoning after hearing an orally presented problem; URO-Bench (Chinese Oral Dialogue Benchmark) evaluates the quality of open-ended dialogue.

Test configurations are divided into two dimensions. The first is Thinking Timing: the complete TBS (Think-Before-Speak, serving as a latency-unconstrained control baseline) generates all thoughts before speaking; to reduce latency, MPS offers two "think-while-speaking" variants—Speak-First (also called spkfirst, zero latency, speaking and thinking start simultaneously) and Think-First (also called thkfirst, waiting for the thinking brain to produce the first segment before speaking, latency of about 80 tokens). The second dimension is Architecture: MPS dual-brain parallel vs. traditional single-model TBS.

The results are shown in Table 9-1, used to compare the performance of different thinking timing and architecture configurations on math accuracy and dialogue scores.

Table 9-1 Step-Audio R1 Different Speech Thinking Configuration Comparison

Configuration	Spoken-MQA	URO-Bench
Answer directly without thinking (Baseline)	70.6%	77.4
MPS Speak-First (Zero Latency)	92.8%	82.5
MPS Think-First (~80 tok Latency)	93.9%	84.8
Complete TBS (No Latency Constraint)	93.0%	—

An interesting finding is that Speak-First has minimal impact on thinking tasks (92.8% is close to the complete TBS's 93.0%). The reason is that the beginning of a CoT (Chain-of-Thought) usually just restates the problem content and hasn't yet entered true reasoning. Therefore, even if the model starts thinking simultaneously with speaking, the final accuracy is hardly affected. Another noteworthy detail is that Think-First (93.9%) is even slightly higher than the latency-unconstrained complete TBS (93.0%)—one possible explanation is that producing thoughts in segments and converting them segment by segment into speech acts like step-by-step supervision, having a positive effect; of course, the difference between the two is also within the evaluation error margin and should not be over-interpreted.

Solution 3 internalizes thinking into a single model—the most elegant realization of "thinking while speaking"—but the price is exactly the "moving target" from the start of this section: one model must be both the strongest reasoner and a real-time speaker, and with both capabilities evolving fast, the unified route must retrain again and again to keep pace. Hence the industry divide at the time of writing: frontier products that want to swap in the latest brain at will (GPT-Live, Grok Voice, Pine AI) mostly bet on Solution 2's decoupling, while Solution 3 suits products that chase ultimate naturalness and can stomach the specialized training cost. Neither replaces the other; it is a trade-off between an interchangeable brain and tighter thinking-while-speaking.

The Interface Between Fast and Slow: What Else Can Be Passed Besides Text

(Note: this is a cross-scenario interface discussion, stepping briefly off the voice main thread.) Look back at Solution 2 and a neglected design dimension appears: when slow thinking passes a message to fast thinking, it uses the text channel (a suggestion via the status bar). Text is easy to understand and easy to debug, but it is a narrow straw—the slow thinker's rich intermediate state gets squeezed into a few sentences. So must the interface between fast and slow be text at all?

In real-time gaming, the most demanding scenario for timing, this path is feasible (it can be called a Latent Bridge) [^ch9-8]: freeze both a small model responsible for quick reactions (producing dozens of actions per second) and a slow model responsible for reasoning (producing one thought per second), and only train a small "bridge" of a few tens of millions of parameters between them. This bridge directly projects the slow model's hidden layer conclusions into a few "latent tokens," which are spliced into the fast model's input, similar to how multimodal models insert visual tokens—bypassing the round trip of "idea → text → re-understanding." The result is that on multiple Atari games, this latent space channel outperforms the traditional text channel by a significant margin (+26% to +82% on some games), while only adding about 5 milliseconds per step, still keeping up with real-time demands.

The same work draws an honest boundary: whether fast-slow collaboration helps depends on whether the task's bottleneck is "can't think of it" or "can't react in time." The bridge pays off only where the slow thinker is genuinely better than the fast reactor (across games this correlation runs as high as r≈0.9); where the task is purely a contest of reaction speed, the finest bridge is useless. The judgment holds beyond games—it previews the very question Computer Use will face later in this chapter: when is a "slow strategist" worth inviting, and when does it merely add latency?

[^ch9-8]: The complete analysis of training only a latent space bridge between two frozen models and "when it's worth inviting a slow strategist" can be found in Li, Bojie and Noah Shi. The Latent Bridge: A Continuous Slow-Fast Channel for Real-Time Game Agents. arXiv:2606.24470, 2026.

End-to-end or modular, the quality of the perception and execution layers still matters. End-to-end models fix latency at the architectural level, but the two fundamentals—hearing accurately and speaking naturally—do not fix themselves when the architecture changes. Hearing accurately maps to the streaming voice perception of Paradigm 1; here we turn to the execution layer for speaking naturally: more human-like speech synthesis.

More Human-like Speech Synthesis

The "perfection" of traditional TTS is exactly the problem: speech that is flawlessly fluent, with zero pauses and no filler words, is instantly recognizable as a machine. The "imperfections" of human speech—pauses, fillers ("um," "uh," "you know"), the occasional repetition—are not flaws but the natural surface of the thinking process, telling the listener "I'm thinking" or "I'm not entirely sure." An AI, though, thinks far faster than speech plays back, and its output arrives fluent and complete; synthesize it as-is and the machine gives itself away.

Solution: Delegate the decision of "where to pause and what tone to use" to the main LLM. The LLM outputs not just text, but also control tokens: [THINKING] indicates inserting a 1-2 second thinking pause and filler sound ("um..."); [SEARCHING] generates a shorter pause and searching filler words ("you know...", "how should I put it"); [EMO:happy] adjusts tone and prosody; [SPEED:0.8x] controls speaking rate. Only the LLM knows whether it is working through a complex question and should pause, whether the user is getting impatient and it should speed up, or whether this is idle chat and it should sound lively.

In this scheme, TTS acts as a multimodal generator, taking text + control tokens as input and outputting audio. It synthesizes speech normally for regular text, and generates corresponding non-linguistic audio for control tokens: [THINKING] generates a drawn-out "um...", [SIGH] generates a sigh, [LAUGH:small] generates a light laugh, [BREATH] generates an inhale sound.

There are two implementation paths: one is developing proprietary TTS with native support for control tokens (highest flexibility, but requires a specialized team); the other is using voice cloning, preparing dozens of reference audio clips for the same virtual persona covering different emotions, speeds, and styles, and selecting the best matching reference audio based on the control token to call a TTS API (e.g., ElevenLabs, Fish Audio), which can be deployed within weeks.

Experiment 9-5 ★★: Control Token-Driven TTS Based on Fish Audio

Use Fish Audio S1's voice cloning capability (only 3-10 seconds of reference audio needed for zero-shot cloning of the same timbre). Build a library of 24 reference audio clips, covering Emotion (Neutral/Happy/Frustrated/Thinking) x Speed (Normal/Fast/Slow) x Style (Formal/Casual), each about 5 seconds long.

LLM output example: [EMO:happy][SPEED:fast]Great! Your order has been confirmed.[THINKING]Um, let me check the shipping time...[EMO:neutral][SPEED:normal]It is expected to arrive tomorrow afternoon.

The execution layer parses the tokens and maps them to the corresponding reference audio: [EMO:happy][SPEED:fast] maps to "Happy+Fast+Casual" reference; [THINKING] maps to "Thinking+Slow+Formal" reference (with pause rhythm and hesitant tone); [EMO:neutral][SPEED:normal] maps to "Neutral+Normal+Formal" reference. Fish Audio ensures consistent timbre across different reference clips, only varying prosody and emotion.

Compare three configurations: No control tokens (fluent but robotic, sounds like AI), Single reference audio (natural but emotionally monotone), Multi-reference audio library (cheerful and fast when confirming information, natural pauses before explanations, overall close to a real human customer service representative's expression).

Computer Use: GUI Automation Agent

By now you may have noticed that this chapter gives voice far more space than the two scenarios that follow—deliberately. On the evolutionary path of real-time multimodality, voice has traveled furthest and makes the best reference frame: starting from the problem ("serial pipeline latency is too high"), through a run of solutions—end-to-end, full-duplex, thinking-while-speaking—to today's relatively settled endgame, it has covered the whole arc from problem to solution to endgame. That is why we told its story in full. Read the Computer Use and robotics scenarios that follow against this voice trajectory: see how far each has come along the same evolutionary line, and where each is stuck.

These three scenarios seem different but face the same core challenges: real-time perception, low-latency decision-making, and continuous interaction. Next, let's see how these technical themes reappear in visual interaction (Computer Use) and physical interaction (Robotics)—first, expanding the perspective from the auditory modality to the visual modality: what if an Agent can not only understand speech but also "see" the screen and operate the graphical interface?

Computer Use (also known as GUI Automation Agent) allows AI to use software like a human by observing the screen and operating the mouse and keyboard—for example, opening a browser to search for information, filling in data in a spreadsheet application, or adjusting configurations in system settings. Its core is a Perceive-Think-Act loop (Figure 9-7):

1. The Agent takes a screenshot of the current screen.
2. A multimodal model receives the screenshot and task instruction, and outputs a thought and a specific action.
3. The execution layer performs the action in the real environment (moving the mouse, clicking, typing text, etc.).
4. It waits for the interface to respond, takes another screenshot, and enters the next loop iteration.

There are three key design dimensions in this loop: Action Space (what operations the Agent can perform), Visual Grounding (how to find the target element in the screenshot), and Model Architecture (how to generate the correct action from the screenshot).

Action Space Design

Anthropic defines three types of tools that constitute a complete interaction capability (Figure 9-8):

GUI Operation Tool (computer tool): Mouse operations include moving (mouse_move), left/right/middle click, double-click/triple-click, dragging (left_click_drag), and more precise press/release actions (left_mouse_down/up). Scrolling (scroll) supports four directions and can be combined with modifier keys. Keyboard operations include typing character by character (type, with a 12ms interval between characters to simulate real typing), key combinations (key, e.g., Ctrl+C), and holding a key (hold_key). Perception actions: screenshot, cursor position retrieval (cursor_position), and waiting (wait).

Command Execution Tool (bash tool): Provides a persistent bash terminal session with a 120-second timeout. It uses a sentinel string to detect command completion and maintains environment state across multiple calls (e.g., after cd to a directory, the next call remains in that directory).

File Editing Tool (str_replace_editor): Enables safe editing via string matching, supporting view, create, replace, insert, and undo operations. It is more precise than directly overwriting the entire file and less prone to accidentally modifying other content.

Experiment 9-6 ★: Running the Anthropic Computer Use Demo

The container packages a complete Ubuntu desktop environment (including a browser, terminal, and other common tools). The frontend receives task instructions, the backend sends the instructions along with screenshots to Claude, and the model returns operation instructions (move mouse, click, type text, etc.), which are then executed in the virtual desktop.

Key observation: Each action takes 2-5 seconds (significantly slower than a human), but the system demonstrates good planning ability for common tasks, autonomously decomposing them into reasonable action sequences.

Visual Grounding

In each iteration of the loop, the model needs to accurately locate the target element in the screenshot—"Where is the search box?" "What are the coordinates of the submit button?" This is the visual grounding problem. Currently, there are two main approaches: one is to turn localization into a multiple-choice problem—first annotate the interface elements with numbers, and the model only needs to select one; the other is pure coordinate prediction—letting the model "look" at the screenshot and report coordinates directly, just like a human. The multiple-choice approach has two implementation methods: pure visual annotation (the original Set-of-Mark, using a segmentation model to cut out candidate regions on the pixels) and structured element indexing (DOM/Accessibility Tree, directly reading the interface's inherent structure). The common advantage of the multiple-choice approach is that it transforms the open-ended problem of "find the button in the screenshot and predict its coordinates" into a closed-ended one of "choose one from the already annotated elements"—just as multiple-choice questions are easier to answer correctly than fill-in-the-blank questions in an exam, the model only needs to say "click [123]" instead of "click the blue button approximately 200 pixels to the right of the top-left corner of the screen."

Set-of-Mark: Visual Annotation Method.

The original Set-of-Mark (SoM) was proposed by Microsoft Research in 2023, initially to unlock the visual grounding capabilities of GPT-4V. It is a purely visual method: it uses image segmentation models (SAM, SEEM, etc.) to automatically cut out candidate regions on the screenshot, overlays a numbered marker on each region, and the model sees an image with numbers. The model only needs to report the number, and the system converts it into the center coordinates of the corresponding region. The entire process does not require a DOM or any internal interface structure, so it is equally applicable to native desktop software and game interfaces—as long as the segmentation model can cut out the candidate regions.

Structured Element Indexing: A Structured Implementation of the SoM Idea on the Web.

When the interface itself can provide structured information, annotation can be more precise. Modern web pages define a complete element structure (DOM tree) and semantic roles (which is a button, which is an input box) before rendering, and the Accessibility Interface (Accessibility Tree) provides similar information for many desktop applications. Instead of having a segmentation model guess "which region is a button" from the pixels, it's better to directly ask the interface itself, "What clickable elements do you have?" The Web Agent approach, represented by the browser-use project, does exactly this: it enumerates interactive elements from the DOM and numbers them. This can be seen as a structured implementation of the SoM idea on the Web (Figure 9-9). The process consists of four steps:

1. Obtain the structured representation (DOM tree) and accessibility information of the web page through the browser debugging interface (CDP, Chrome DevTools Protocol)
2. Automatically detect which elements are interactive (buttons, input boxes, links, etc.)
3. Annotate each interactive element with a unique ID and draw bounding boxes on the screenshot
4. Simultaneously generate a text list describing the element corresponding to each ID

Screenshot: [Key elements in the image are annotated with IDs like [1], [2], [3], [4]]

Elements:
[1] <input type="text" placeholder="Search" aria-label="Search" />
[2] <button id="submit-btn" aria-label="Submit form" />
[3] <input type="text" placeholder="Enter your name" value="" />
[4] <a href="/docs" aria-label="Documentation" />

The model only needs to output an ID number, and the system automatically executes the click using the center coordinates of that element. This type of approach does not save tokens (because all annotation information must be sent to the model), but the localization is accurate and stable, and it also avoids the missed detections and false positives that segmentation models might introduce.

Pure Coordinate Prediction.

The third route does not perform any annotation and directly asks the model to output coordinates. Represented by SeeClick and Claude's computer use, this approach trains a vision model on a massive dataset of GUI screenshots paired with element positions, teaching it to directly map natural language descriptions (e.g., "click the submit button") to precise coordinates in the screenshot—just like a human user, relying purely on "seeing" to find the location to click.

In coordinate prediction schemes, the model's understanding of coordinates is highly dependent on the resolution used during training (Figure 9-10). Claude was trained using XGA (1024x768), WXGA (1280x800), and FWXGA (1366x768). If the input screenshot resolution does not match, the model's predicted coordinates will systematically shift—like measuring a distance on a small map and then applying it directly to a large map. Therefore, a bidirectional coordinate scaling mechanism must be implemented at the tool layer, and the target resolution must be selected based on the aspect ratio to avoid non-uniform stretching that distorts the image and consequently biases coordinate judgment. For example, if the actual screen resolution is 2560×1440 (16:9), the most suitable target among Claude's three supported options is FWXGA (1366×768), which has an aspect ratio closest to 16:9. The screenshot is proportionally scaled to 1366×768 and fed to the model; after the model outputs the click coordinates (683, 384), they are inversely mapped to the real coordinates (683×2560/1366, 384×1440/768) ≈ (1280, 720). Conversely, if a 16:9 image is forcibly stretched into the 4:3 1024×768, the image will be horizontally compressed, causing the model's predicted coordinates to systematically shift.

The selection logic for the three routes can be summarized as follows: When structured information is available, prioritize DOM/Accessibility Tree indexing for the most accurate and stable localization; when it is unavailable (e.g., native desktop software like Photoshop, Canvas/WebGL rendered interfaces, games), either visual annotation (the original SoM route) or coordinate prediction can be used. Visual annotation turns localization into a multiple-choice problem, which is more friendly to general-purpose models that have not been specifically trained; coordinate prediction eliminates the annotation step and is more direct for models trained on GUI localization. Both still have accuracy gaps on small elements and dense interfaces.

Experiment 9-7 ★: Using browser-use to Implement Automated Browser Operations

Based on the Playwright browser automation framework (a tool library for controlling browsers with code), combined with a multimodal large model, this implements natural language-driven browser operations. Enable SoM visualization mode, saving screenshots with annotated bounding boxes before each decision.

Test task "Open Google and query San Francisco weather": After the system starts, the screenshot shows the Google search page, with all interactive elements annotated with red bounding boxes and ID numbers (address bar [1], search box [2], search button [3], "I'm Feeling Lucky" button [4], etc.) → The model analyzes and clicks [2] (the search box) → The search box gains focus and the model types "San Francisco weather today" → The model clicks [3] (the search button) → The page navigates to the search results, the new screenshot annotates elements within the weather card, and the model identifies and extracts information like temperature and weather conditions. The entire process takes 5 steps and about 20 seconds.

A Computer Use Agent That Can Watch Animations and Hear Sound

So far, Computer Use perception has rested on an implicit assumption: the screen is static—take a screenshot, think a step, click, take the next screenshot. Real screens play videos, flash notifications that vanish in seconds, and carry human voices from meetings. An Agent that opens its eyes only once every 3–5 seconds, and has no ears at all, is blind and deaf to everything that happens between two frames. Watching a screen recording, joining a meeting, following a voice prompt, catching a dialog box before it disappears—this whole category of everyday computer work is effectively off-limits to today's Computer Use Agent.

What truly needs to be redesigned here is not the "action interface," but the "observation interface"[^ch9-9]. The core idea is to decouple observation (continuous, adaptive, multimodal) from action (discrete), creating a perceptual middleware layer that sits between the environment and any off-the-shelf Computer Use model, requiring no retraining (this can be called the Agent–Computer Observation Interface, AOI). It has three "gated" components: First, inter-frame keyframe capture—use a very cheap pixel gate to skip nearly unchanged frames, then use a small model to determine if a meaningful change has occurred, capturing a frame only when there is a change, resulting in near-zero cost for static screens; Second, volume-gated speech transcription—only invoke speech recognition when there is sound, giving the Agent "ears" for the first time; Third, and most critically, narrating the screen into persistent text—have the model describe the captured frame in a single sentence (e.g., "The popup just said the release date has been changed to April 28th"), and even if the original image is later cleared from the context, this text remains in memory, carrying the dynamic information forward in textual form.

The counterintuitive finding: what really matters is not which frames get selected, but "narrating the frames into text that persists"—text being the modality LLM Agents handle best. Across eight models from 7B to frontier scale, this middleware delivered gains of +17 to +48 percentage points without any retraining, with the widest gap on voice tasks: with the perceptual layer in place, the Agent could finally complete voice tasks that had been "audible but unactionable." It is not a one-size-fits-all configuration, though—on some newer models, injecting too many image tokens crowds out reasoning and drags performance down. So the components should be chosen per model, not switched on wholesale. It is the same lesson as the Set-of-Mark-versus-coordinate-prediction trade-off: there is no silver bullet in perception schemes; you configure them to suit the model's temperament.

[^ch9-9]: For the complete mechanism and per-model ablation of the three components—gated keyframes, on-demand transcription, and narrating frames into persistent text—see Li, Bojie and Noah Shi. Agent-Computer Observation Interfaces Enable Dynamic Computer Use. arXiv:2606.29472, 2026.

Mobile: Ecosystem Barriers Are Harder Than Technology

Computer Use is also expanding to the mobile domain. There are indeed technical differences between mobile and desktop: the action space is usually no longer "mouse coordinates + keyboard," but rather interfaces with the system's accessibility service API (e.g., Android's AccessibilityService) to read interface elements and issue clicks and text input; the interaction method also changes from a mouse pointer to touch gestures, altering the semantics of coordinates—whether the same (x, y) represents a finger tap, a long press, or the starting point of a swipe gesture requires an additional gesture type to define. The mobile benchmarks like AndroidWorld introduced in Chapter 6 evaluate the Agent's ability to complete tasks in real apps within this action space.

However, what truly hinders mobile Computer Use is often not these technical differences, but ecosystem barriers. Some phone manufacturers have attempted to integrate AI assistants into consumer-grade phones, allowing them to automatically operate everyday apps like WeChat, Taobao, and Alipay, but they quickly encountered platform restrictions.

This reveals a unique challenge for Computer Use: ecosystem barriers. The fundamental reason behind the blockades is a conflict of business models. The core monetization logic of traditional internet applications is traffic and attention: users see ads while scrolling through feeds, follow recommendation algorithms when searching for products, and make impulse purchases while browsing pages. When an Agent operates on the user's behalf, that monetization chain is bypassed entirely: the AI ignores ads, makes no impulse purchases, heads straight for the goal, finishes the task, and leaves. For platforms that live on advertising and traffic, every Agent operation erodes the foundation of the business model.

This means that Computer Use faces not only technical adversarial measures like CAPTCHAs, but also a structural conflict of interest. This contradiction is difficult to reconcile in the short term and presents a more challenging obstacle for the deployment of Computer Use in consumer scenarios than purely technical issues.

Real-Time Performance: The Unsolved Core Challenge

OSWorld (Chapter 6 details its evaluation methodology) is a widely used benchmark for Computer Use, testing an Agent's ability to complete cross-application tasks in real Ubuntu/Windows/macOS environments. Early general-purpose models achieved only about a 20% success rate on this benchmark. Subsequent specialized models and more powerful general-purpose models have continuously pushed the accuracy higher, gradually approaching human-level performance as of this writing. However, accuracy is far from the finish line—the real bottleneck has shifted from "can it do it correctly?" to "can it do it quickly?"

The OSWorld-Human efficiency study delivers a sobering fact: even when the task ultimately succeeds, the Agent needs markedly more steps than a human, and per-step inference latency keeps growing as the task progresses—the longer the context, the slower the model decides, so late steps often take far longer than early ones. A document formatting tweak a human finishes in tens of seconds is something an Agent can grind away at for several minutes. Human-level accuracy is not the same as practical—efficiency is the true bottleneck.

The root cause mirrors the speech scenario: in the serial "screenshot-think-click" loop, even with every stage optimized to the hilt, the step-by-step accumulation of delay remains unacceptable. The deeper problem is that today's Computer Use cannot think ahead at all. If an Agent could predict its next move while executing the current one—working out where to click next while the page is still loading—it could overlap thinking with execution and cut total latency sharply (the same demand as thinking-while-speaking earlier in this chapter and the "continuous thinking" asynchronous Agent of Chapter 4, recast here as thinking-while-operating).

Unlike the speech domain, there is currently no systematic solution for improving the real-time performance of Computer Use itself—making the "screenshot-think-click" loop faster—and it remains stuck in a discrete loop of frame-by-frame screenshots. However, a workaround has already been proven effective, using the fast-slow decoupling that appears repeatedly in this chapter: since it's difficult to make a slow computer-use Agent faster, don't make the user wait for it. Split "speaking" and "operating the computer" into two models running concurrently, one fast and one slow[^ch9-10]—a small model (fast) handles real-time voice conversation, while a cutting-edge VLM (slow) operates step-by-step in the browser. The two communicate only through a minimal "plain text contract": each time the slow Agent performs an action, it appends a rolling status summary ("Filling out the form, still need your date of birth"). The fast Agent uses this to answer the user in real time and relays any new information the user provides verbally to the slow Agent. Crucially, the fast Agent must never say "done" until the status summary confirms completion. This is the scenario of "talking on the phone while letting the computer operate itself." In experiments, this decoupling made voice responses about 15 times faster than a single model that operates and speaks at once (median latency 0.58 seconds vs. 8.64 seconds), with no loss in task success rate. Remove the text channel between fast and slow, and success collapses to zero—the key information users give verbally can no longer reach the browser. This is the same idea as the Latent Bridge earlier and thinking-while-speaking in the speech scenario: when one component is inherently slow, let a fast one fill the user's waiting time—and that "plain text contract" is, at bottom, the Agent Status Bar this book has carried since Chapter 2. Speeding up the Computer Use loop itself may well be the next important research direction, but hiding the slowness behind fast-slow decoupling is already a workable answer.

[^ch9-10]: The complete design of the speech-operation fast-slow decoupling and the "plain text contract" can be found in Li, Bojie and Noah Shi. Talking While Acting: Real-Time Voice for Slow Computer-Use Agents. 2026 (forthcoming).

Robot Manipulation: From Real-Time Control to Training and Generalization

Reading Note: This section discusses robot control. Experiment 9-10 demonstrates a method for transferring from simulation to reality—the simulation training part (steps 3-4) can be completed on a pure GPU server without hardware; however, to reproduce the entire pipeline end-to-end (including the real-world deployment steps), real hardware such as the SO100 robotic arm is required. If you are not currently interested in robotics, you can skip this section; it does not affect the reading of other chapters.

Voice Agents fight latency in the auditory modality, Computer Use in the visual. When an Agent must control a robot in the physical world, latency and multimodality bite harder still—actions have irreversible consequences, and one collision can damage the object or the robot itself. This section first shows how robots tame the real-time control problem with a two-layer architecture and action chunking, then turns to the harder problem they face today—training and generalization: where the data comes from, and how models transfer across tasks and platforms.

Hardware is Not the Bottleneck, Algorithms Are

Why haven't robots been widely adopted in general open scenarios? Is the bottleneck hardware or algorithms? The XLeRobot project provides a strong counterexample: a dual-arm wheeled robot costing less than $1000, when remotely controlled by a human via a VR headset (teleoperation), can already smoothly perform a wide range of household tasks. For more complex household tasks requiring dexterous hands, robots from Unitree can also perform smoothly under human teleoperation. Teleoperation latency is around 100-200ms, which is close to the response requirements for physical interaction. Sensor resolution, actuator precision, and control frequency (the number of times a robot updates its action commands per second; lower frequency leads to less smooth motion and more jitter or deviation from the target trajectory) on current low-cost platforms are already sufficient for practical tasks.

This assertion needs a clear boundary: what the teleoperation counterexample truly demonstrates is that "existing low-cost hardware, combined with human intelligence, is sufficient to complete this type of household manipulation task that primarily relies on visual feedback." It does not mean hardware is adequate in all dimensions—the lack of tactile sensing, the reliability and cost of dexterous hands, remain recognized hardware shortcomings. Once a task heavily depends on fine force control and tactile feedback, hardware may indeed become the bottleneck. Therefore, the statement "hardware is not the bottleneck" below is limited to the type of tasks discussed in this section.

For these tasks, the real gap lies in the algorithm layer, which is elaborated in the following two subsections.

Experiment 9-8 ★: XLeRobot Teleoperation Experience

XLeRobot supports various teleoperation methods including keyboard, Xbox controller, Switch Joycon, and VR headset. By manually controlling the robot to complete tasks such as picking up objects, placing them, and wiping surfaces, observe the response latency, motion precision, and task completion quality to build an intuitive understanding of the boundaries of hardware capabilities—after experiencing it firsthand, you will find that the robot can do almost anything when controlled by a human, indicating that the current bottleneck is indeed algorithms, not hardware.[^ch9-1]

[^ch9-1]: XLeRobot, "Teleop Documentation" . https://xlerobot.readthedocs.io/en/latest/software/getting_started/XLeRobot_teleop.html

Two-Layer Architecture: Separation of Planning and Control

Robots need to make decisions at two different time scales to complete complex household tasks. The first layer is slower long-horizon planning: decomposing a high-level instruction like "clean the kitchen" into a sequence of sub-goals (clear the countertop, load the dishwasher, wipe the surfaces). This requires understanding environmental semantics, reasoning about task dependencies, and planning multi-step action sequences—similar to how a person thinks about "what to do first and what to do next" before starting. The second layer is faster VLA control (Vision-Language-Action model): executing each specific operation ("walk to the sink," "pick up the cloth," "wipe the countertop"), continuously outputting control signals based on the current visual input and language instruction to ensure smooth and coherent robot motion.

This two-layer architecture effectively separates complexity: long-horizon planning handles "what to do," and VLA control handles "how to do it." This "slow high-level decision-making + fast low-level execution" two-layer architecture is structurally highly similar to the "fast-slow thinking" in the speech scenario earlier—both decouple complex thinking from real-time response into different modules. Note, though, that "planning/control" here matches the fast-slow dimension of "slow deep thinking / fast real-time response," not the "thinking/expression" split of MPS's Formulation Brain and Articulation Brain in Solution 3—the latter separates thinking from speaking, the former separates global planning from real-time execution. The two "dual-X architectures" cut along different dimensions.

However, real-time performance hasn't disappeared; it has been pushed down to the VLA control layer, where it is mitigated by Action Chunking (see the "VLA Control" subsection below): the model generates a short sequence of future actions in one inference, and the control thread replays them at a high frequency, spreading the latency of a single inference across the execution time of the entire action sequence. But an unavoidable trade-off lurks here—chunking buys smoothness with reactivity: the longer the chunk, the more thinly each inference's latency is spread and the smoother the motion, yet the model is blind to new visual input for that whole stretch, and so slower to react to sudden changes (an object moved away, a hand in the way). This tension between real-time response and smoothness is the part of the problem the two-layer architecture has not eliminated—only relocated.

Here the chapter's main thread takes a turn: in robotics, the real-time tension has been partly relieved by two-layer decoupling and action chunking, and the primary struggle has moved to training and generalization—how to obtain enough demonstration data, and how to make models generalize across tasks and platforms. The following subsections center on this new struggle, which extends the themes of Chapter 6's simulation environments and Chapter 7's reinforcement learning into the physical world.

And this new struggle falls chiefly on the VLA control layer. Think of VLA as "VLM + action output": the VLM (Vision-Language Model—a large model that understands both images and text) handles seeing and thinking clearly, while the VLA must also act—and acting is where the real challenge lies. Currently, the VLA control layer is primarily trained via imitation learning (behavioral cloning)—directly learning "see something, do something" from a large number of human demonstrations (OpenVLA, RT-2, π₀, etc., all fall into this category). Reinforcement learning is a supplementary method that has been added on top in recent years. While VLAs trained with reinforcement learning can perform well on individual tasks, they often lack generalization ability: even if SimpleVLA-RL from Chapter 7 reports high single-task results on LIBERO, it is trained with RL for each task separately, not a unified model that generalizes zero-shot to all tasks. This "train once per task" pattern means that for every new task, data must be recollected and the model retrained.

The following two sections delve into the specific technical solutions for long-horizon planning and VLA control, respectively.

Long-Horizon Planning: From VLM to Specialized Embodied Reasoning Models

General-purpose VLMs already possess decent embodied reasoning capabilities. Google DeepMind's Gemini Robotics-ER 1.5 is specifically optimized for Embodied Reasoning (understanding the position, movement, and causal relationships of objects in the physical world). It achieves an average of 62.8% across 15 academic benchmarks (Point-Bench, RefSpatial, RoboSpatial, BLINK, etc.), surpassing GPT-4o (60.6%) and Gemini 2.5 Pro (59.3%). Key advantages include: advanced spatial understanding and object localization, temporal reasoning (predicting action consequences like "what happens if I push this cup"), task sequencing (decomposing high-level instructions into smaller steps), and native support for thinking mechanisms and tool calls.[^ch9-2]

[^ch9-2]: Google DeepMind, "Gemini Robotics-ER 1.5" . https://deepmind.google/models/gemini-robotics/gemini-robotics-er/

Experiment 9-9 ★★: Using Gemini Robotics-ER 1.5 to Drive XLeRobot Autonomous Navigation

Use the RoboCrew library to employ Gemini Robotics-ER 1.5 as the long-horizon planning model, with camera images overlaid with angle scale annotations. The system provides only three simple tools: move forward, turn left, turn right. Given the task "find the kitchen and go there," the model makes decisions at a frequency of 0.5-1Hz: identify visual features like corridors, doors, and furniture; decide "the kitchen might be on the left" and execute a left turn; see "a refrigerator ahead" and continue moving forward. This can also be extended to a voice control mode (using a wake word to trigger new tasks). This experiment reveals the capability boundaries of VLMs in the long-horizon planning layer: spatial reasoning and task decomposition are already quite good, but robustness in complex environments and consistency in multi-step reasoning still have room for improvement.[^ch9-3]

[^ch9-3]: XLeRobot, "LLM Agent Control" . https://xlerobot.readthedocs.io/en/latest/software/getting_started/LLM_agent.html

VLA Control: From Demonstration Data to Cross-Embodiment Generalization

In the execution layer of the two-layer architecture, three representative models—RT-2, OpenVLA, and π₀—all focus on VLA control, i.e., outputting robot actions in real time based on camera images and language instructions (Figure 9-11). They belong to two different routes in action representation: discrete action tokens and continuous trajectory generation.

RT-2 and OpenVLA: The Discrete Action Token Route.

RT-2 pioneered this route: it directly fine-tunes a large-scale vision-language model, discretizing the robot's continuous actions into tokens and outputting them autoregressively one by one, like generating text. It leverages the generalization ability of the pre-trained model to improve zero-shot transfer to new objects and instructions. OpenVLA follows RT-2's action representation scheme, unifying the language model and vision encoder in a single architecture. It takes images and text instructions as input and outputs action tokens. Training is done in two stages: first, pre-training on the large-scale cross-platform dataset Open X-Embodiment (covering real-world manipulation demonstrations from over 20 robot platforms) to learn general manipulation knowledge (action patterns like "grasp" and "place" are common across different robots); second, fine-tuning with a small amount of data for a specific platform. Since the action representation is essentially the same, the real difference between the two lies in openness and engineering choices: RT-2 and its training data are internal to Google, while OpenVLA is fully open-source—an open-source backbone model (Llama 2 plus a vision encoder) paired with public datasets, allowing the entire community to reproduce and improve upon it for the first time.

Action Chunking: A Universal Frequency Compensation Technique in the VLA Domain.

Due to the latency of LLM inference, the control frequency of VLA is much lower than that required by traditional robot control (traditional robot control typically requires 50-1000Hz, while VLA single inference is only about 1-10Hz—a difference of up to two orders of magnitude). The original OpenVLA is a typical example of this problem: it outputs only one action per inference (about 6Hz single-step autoregressive prediction), and action jerkiness is precisely its most-criticized shortcoming. Action Chunking is a universal technique designed to bridge this gap—first proposed by ACT (Zhao et al., 2023), and later widely adopted by π₀, OpenVLA-OFT, etc.: instead of outputting just one action per inference, the model generates a short sequence of future actions all at once (using π₀'s typical configuration as an example, it generates an action chunk of about 0.5-1 second, which is 25-50 actions at a 50Hz control frequency). The control thread executes them sequentially at a high frequency, while the model asynchronously generates the next batch in the background. As long as the model's inference time is less than the execution time of this batch of actions, the robot can maintain continuous and smooth motion—like video buffering, loading the content ahead of time so playback doesn't stutter.

π₀: The Continuous Trajectory Generation Route.

The true divide in action representation is not between RT-2 and OpenVLA, but between discrete tokens and continuous trajectory generation. π₀ represents the latter route: instead of predicting discrete action tokens one by one, it uses flow matching (a continuous generation method with the same origins as diffusion models) to start from random noise and, through multiple iterative "denoising" steps, directly generate a smooth, continuous action trajectory. This representation naturally combines with action chunking and performs better on tasks requiring high precision and smoothness, such as dexterous manipulation. To use an analogy: the discrete token route is like gradually selecting "5 degrees left," "3 cm forward" from a menu; the continuous trajectory route is like an artist first sketching the entire curve and then refining it stroke by stroke.

Sim2Real Transfer: The Gap from Simulation to Reality

Chapter 6's simulation section already explained where the sim-to-real gap comes from and how domain randomization counters it, so we won't repeat that here. In a nutshell: simulation can never perfectly reproduce real-world physics, visuals, and hardware, so training randomizes those parameters over a wide range, forcing the policy to learn a representation robust to all of it (Figure 9-12). What follows is how that principle lands on a real robotic arm.

This approach has several successful cases: OpenAI's dexterous manipulation with a robotic hand (the Dactyl project achieved in-hand cube reorientation, and subsequent work used Automatic Domain Randomization (ADR) to solve a Rubik's Cube with one hand) and ETH Zurich's ANYmal (a quadruped robot robustly walking on complex outdoor terrains like snow and gravel) are prominent examples.

What this chapter adds are the two engineering steps you cannot skip when taking domain randomization to a real robot. The first is calibrating the randomization range: the range cannot be set on a hunch. Too narrow, and it misses real-world variation; too wide, and training gets harder and yields a suboptimal policy that "handles everything, masters nothing." In practice, the distribution of key parameters (friction coefficient, motor response delay) is first measured and calibrated from real-world data and sampled within that range; if the sim-trained policy drops noticeably on the real robot, the range is widened step by step until the sim-to-real gap converges to something acceptable. The second is visual alignment: precisely calibrating camera pose between simulation and reality (environment alignment), and randomly splicing real-world background images into the simulated render (greenscreen background replacement) so that the simulation looks as much as possible like what the real robot sees. Experiment 9-10 demonstrates both steps.

Experiment 9-10 ★★★: Zero-Shot RGB Sim2Real Robotic Grasping

Using the LeRobot + ManiSkill simulator, train with only RGB camera images (without relying on depth sensors or force sensors), then deploy zero-shot (without any additional tuning) directly to a real SO100 robotic arm. The five-step process:

1. Environment Alignment: Adjust the camera positions in the simulation and real environment, verifying through visual overlay that the images from both sides are aligned.
2. Background Replacement (Greenscreen): Randomly crop background images captured from the real environment and overlay them onto the simulation rendering, making the simulation background closer to reality.
3. Domain Randomization: Randomize parameters such as robot color, object texture, lighting conditions, and camera field of view.
4. RL Training: Train using the PPO algorithm in a massively parallel simulation environment until the success rate in simulation exceeds 90%.
5. Real-World Deployment: Successfully complete the grasping task on the real robot zero-shot.

Key success factors: precise environment alignment + visual domain randomization + physical parameter randomization, all three are indispensable. Limitation: When the shape, size, or material of real objects falls outside the training distribution, the success rate drops significantly.[^ch9-6]

[^ch9-6]: LeRobot, "Sim2Real Tutorial". https://github.com/StoneT2000/lerobot-sim2real/blob/main/docs/zero_shot_rgb_sim2real.md

Chapter Summary

On the surface the three scenarios could hardly differ more, yet the twin hurdles of latency and multimodality shadow them all. Voice has walked the full evolutionary path—from serial pipeline to end-to-end and full-duplex, from separate fast and slow thinking to thinking-while-speaking. Computer Use now approaches human accuracy on benchmarks like OSWorld, but it takes far more steps than a human and each step slows as the task wears on—an efficiency gap with no systematic solution yet. For robots on visually guided manipulation tasks, the bottleneck has moved from hardware to the VLA control layer's ability to generalize across tasks (tactile sensing and dexterous hands remain unconquered hardware gaps). The next chapter turns to collaboration among multiple Agents—a challenge of a different dimension.

Thought Questions

1. ★★ The end-to-end model for voice Agents merges ASR-LLM-TTS into a single model, reducing latency but losing modularity. If the end-to-end model makes an error in a specific stage (e.g., speech recognition), debugging and fixing it is much harder than in a serial pipeline. How would you design an observability system for an end-to-end voice Agent?
2. ★ Step-Audio R1 achieves "thinking while speaking" through the MPS dual-brain architecture. However, humans, when "thinking while speaking," often utter unconsidered words, self-correct, or use filler words. Should an Agent's "thinking while speaking" mimic these human characteristics?
3. ★★ SoM (Set-of-Mark) and its structured variants (DOM element indexing) convert Computer Use's visual localization from open-ended coordinate prediction to closed-set ID selection, but they all require detecting and annotating UI elements first—whether via a segmentation model or the DOM. If the interface contains non-standard controls or dynamically changing elements, the annotations may be incomplete or inaccurate. In such cases, should we fall back to coordinate prediction?
4. ★★ Thousand-dollar robot platforms like XLeRobot make teleoperation data collection inexpensive. However, the quality of teleoperation data heavily depends on the operator's skill. How would low-quality data from an unskilled operator affect the training of a VLA model? How can low-quality data be automatically filtered during the data collection phase?
5. ★★★ This chapter covers three interaction modalities: voice, Computer Use, and robotics. A common trend across these modalities is the evolution from serial pipelines to end-to-end models. If this trend continues, what might the Agent interaction layer look like in five years?
6. ★★★ Current Computer Use operates in a discrete "screenshot → action → screenshot" loop, where each observation is a static frame. But human perception of a screen is continuous—we see animations play, observe loading progress, and understand video content. This means today's Computer Use cannot handle tasks requiring temporal visual understanding. How would you redesign the perception layer to support continuous visual stream understanding?
7. ★★ DOM/Accessibility Tree element indexing works well on standard web applications, but an increasing number of software interfaces (Canvas/WebGL rendering, cross-platform custom-drawn controls) do not provide accessible structured information, relying solely on visual annotation or coordinate prediction. Do you think Computer Use should bet on a purely visual approach, or maintain both structured and visual paths? What are the costs and benefits of maintaining both paths?
8. ★★ VLA models use action chunking—as mentioned in the text, π₀'s typical configuration generates 25-50 future actions at 50Hz—to hide inference latency within execution time. However, if the environment changes suddenly during execution (e.g., an object is moved), the pre-generated action sequence becomes invalid. How can we balance the efficiency advantage of action chunking with the need for responsiveness to environmental changes?
9. ★★★ All three scenarios in this chapter (voice, Computer Use, robotics) face the latency problem of the "perceive-think-act" loop and are evolving towards parallelizing fast and slow thinking. In voice, this manifests as "correcting after misspeaking"; in Computer Use, as "clicking first, then looking"; in robotics, as "taking a step, then looking." How can we ensure that these actions based on fast thinking do not lead to irreversible consequences?