XBot-Omni adopts a Thinker–Talker–Actor architecture to connect multimodal understanding with embodied response generation. Given text, images, audio, video, and dialogue history, the Thinker interprets the interaction and formulates a contextual response. The Talker realizes the response as streaming speech, while the Actor produces accompanying body motion and facial expressions. Together, the three components organize an interaction response from incoming observations to verbal and nonverbal expression.
Overview
The architecture separates response formulation from modality-specific generation while maintaining coordination across output modalities. Speech unfolds as an acoustic sequence, body motion evolves continuously from the current pose, and facial expressions change throughout the response. These differences require generated behaviors to share communicative context while remaining coordinated with the progression of speech.
To achieve this, the Actor draws on native Thinker representations for interaction context and communicative intent, and native Talker states for speech-dependent conditioning. This connection allows the multimodal understanding used to formulate a response to also guide its bodily expression. Object pointing is one application: the scene-dependent referent becomes part of the action requirements passed to the Actor.
Talker: Streaming Speech Generation
The Talker follows the speech-generation architecture of Qwen3.5-Omni, converting Thinker-generated responses into streaming speech. It comprises a hybrid temporal backbone and a depth predictor. The backbone uses a hybrid mixture-of-experts design to model speech-token progression, while the depth predictor performs multi-token prediction over the residual codebooks of each frame, conditioned on the backbone states.
These tokens are subsequently decoded into acoustic waveforms by a speech codec decoder, enabling incremental speech generation without requiring the complete response to be synthesized in advance.
Coordinated Nonverbal Generation From Native Representations
The Actor generates body motion and facial expression as coordinated components of an embodied response. Native Thinker states carry the multimodal context and communicative intent used to formulate that response, while native Talker states provide evolving speech features and progress cues.
Body generation combines these inputs with explicit action requirements, temporal conditions, and motion history, allowing interaction context to shape the meaning expressed by movement. Facial generation organizes the shared response representations into chunk-level conditions for continuous expression. Both output streams are placed on a common speech timeline, coordinating body gestures and facial changes with each other and with the associated spoken content.
Motion Generator
At each motion step, generation is conditioned on the adapted speech history, the active semantic events and their temporal conditions, and the motion already produced.
The generated codes are decoded into articulated body motion and also form the history for subsequent predictions. Temporal state persists across steps, allowing a new action requirement to take effect within ongoing movement. The upper- and lower-body codec outputs are assigned motion timestamps on the response timeline, connecting the generated sequence to its eventual presentation with speech and facial expression.
For robot-specific pointing execution, the spatial target and generated reference motion are passed to a kinematic adapter. It converts the target to the robot’s base frame and uses the reference posture to guide inverse kinematics, aligning the pointing direction with the target while preserving motion continuity. The platform-specific configuration is described in Section 4.4.
Facial Expression Generation
The facial branch uses a Chunk Transformer to organize response semantics and speech features into temporal conditions, followed by a conditional flow generator.
Large-Scale, High-Quality Embodied Interaction Data
Training XBot-Omni for real-world embodied interactions requires data that captures not only rich audio-visual information but also fine-grained human behaviors, underlying intentions, and temporal dependencies across modalities. However, existing audio-visual datasets primarily focus on short clips or isolated events, often lacking the continuous behavioral and interaction dynamics of real-world scenarios. Meanwhile, embodied datasets largely emphasize action recognition and task execution, with limited coverage of natural conversations, social interactions, and real-time response behaviors.
To bridge these gaps, the aim is to construct a large-scale dataset that combines multimodal richness, temporal continuity, and diverse embodied interactions, supporting models in learning to perceive, understand, and respond to dynamic real-world environments.
Thinker Training
The post-training of the Thinker follows a three-stage pipeline designed to progressively develop and integrate multimodal interaction capabilities. First, supervised fine-tuning establishes a unified foundation for multimodal perception, reasoning, and interaction understanding. However, different embodied interaction tasks require specialized reasoning capabilities and task-specific optimization objectives.
Multiple domain-specific expert policies are therefore trained through reinforcement learning, allowing each expert to specialize in its designated task domain. Finally, multi-teacher on-policy distillation consolidates these specialized capabilities into a single student policy.
Perception-to-Action Training
Perception-to-action training builds on the Thinker and Actor to connect multimodal understanding with embodied response generation. This stage extends the learned semantic event interface with task-specific conditions, allowing interaction context to guide both what a gesture expresses and how it unfolds with speech. Spatial Grounding and Pointing Training (SGPT) is used as a practical application, connecting visual target identification to a complete, spatially directed gesture within the ongoing response.
Supervised observation–instruction pairs establish the correspondence between referring expressions and visual targets. Target localization is combined with depth estimation and geometric processing to obtain a metric position, which is transformed into the Actor’s body-centric frame.
System Integration
System integration follows a staged training schedule to combine the frozen Thinker and Talker with the trained Actor for response generation. After multi-teacher on-policy distillation, the consolidated Thinker and the Talker are frozen, and Actor training proceeds with this fixed upstream model pair. This ordering establishes the semantic and speech-generation representations used to condition the Actor, avoiding changes to their source models during Actor training. The specialized experts are used only during Thinker training; the integrated system uses the consolidated Thinker as a single policy.
The Actor is trained through its semantic and speech-state interfaces while the native Thinker–Talker response pathway remains fixed. The coordinate-conditioned stage of SGPT follows the same Actor-side training arrangement, extending semantic events with spatial targets while retaining the fixed native response pathway.
At inference, visual grounding supplies the target parameters, and the Actor combines them with the response semantics and evolving speech features through its shared event interface. This pathway carries the context used to understand and formulate a response into its bodily realization, with object pointing as one concrete application. The connection is learned on the Actor side while the upstream Thinker and Talker remain fixed.
Frequently Asked Questions
What are the three components of XBot-Omni? The Thinker interprets multimodal input and formulates a response, the Talker generates streaming speech, and the Actor produces body motion and facial expressions.
How does XBot-Omni coordinate speech and nonverbal behavior? The Actor uses Thinker representations and Talker states, and places body and facial outputs on a common speech timeline.
How does the system generate robot pointing motions? A kinematic adapter transforms the spatial target into the robot’s base frame and uses reference posture to guide inverse kinematics while preserving motion continuity.
How is the integrated system trained? After the Thinker and Talker are consolidated, they are frozen while the Actor is trained through their semantic and speech-state interfaces.