RoboTTT decouples fast weight updates from slow weight updates: by masking the flow-matching loss on selected timesteps, those timesteps serve as pure context, updating the fast weights without providing an imitation target. This flexibility lets RoboTTT learn from heterogeneous contexts, such as human video demonstrations or the robot's own suboptimal rollouts.
Limitations and Conclusion
This work has a few limitations. First, scaling training context length increases training cost; future work could adopt more recent TTT training techniques such as TNT. Second, while we develop a principled way of integrating TTT into robot foundation models, future work might explore robotics-oriented objectives for the TTT layers (as explored for vision). Finally, although RoboTTT improves task performance substantially, it does not handle every failure mode encountered in deployment; combining it with reinforcement learning to optimize task success directly is a natural next step.
Deployment Details
We deploy trained models on the YAM bimanual tabletop robot with four RealSense D405 cameras (top, bottom, left wrist, right wrist) streaming 480p RGB observations. Inference runs on a workstation with an NVIDIA RTX 5090 GPU, at a control frequency of 30 Hz.
Task Definitions
All tasks are scored on a [0, 1] task completion scale via task-specific rubrics, detailed below.
Pup Go Car
The robot assembles the yellow roof and the first wheel of the model car "Pup Go Car". It picks up the roof, aligns the screw with the hole on the car body, and places the roof. It then grasps the drill, aligns it with the roof screw, and tightens it, before handing the drill off to the right hand. The robot next flips the car with one hand and picks up a wheel with the right hand, which inserts the wheel into the car body. It then picks up the drill again and tightens the wheel, before finally handing the drill off to the left hand.
The scoring rubric: 0.05 if the roof is picked up; 0.1 if the roof is placed on the car body; 0.25 if the roof screw is properly inserted into the car body; 0.3 if the drill is picked up; 0.35 if the drill tip contacts the screw; 0.45 if the roof screw is fully tightened; 0.5 if the drill is handed to the right hand; 0.55 if the car body is flipped and stabilized; 0.6 if the tire is picked up; 0.75 if the tire is inserted into the car body; 0.8 if the drill is picked up; 0.85 if the drill is aligned with and contacting the wheel screw; 0.9 if the wheel is fully tightened; 1.0 if the drill is handed to the left hand. We allow at most two attempts for wheel assembly.
Gear Bot
The robot assembles the entire toy model "Gear Bot". It installs one gear and two wheels on each side of the chassis, flipping the chassis twice so the shafts face upward. It then picks up the red "robot head" and inserts it onto the chassis. Finally, it picks up the remote control and pushes the joystick so the Gear Bot moves around.
The scoring rubric: +0.1 for each chassis flip, +0.1 for each gear or wheel installation, +0.1 for "robot head" installation, and +0.1 if the remote control is used successfully.
Experiment Details
Baselines For the GR00T N1.7 and GR00T N1.7 Hist. baselines, we use the official implementation, extended to support history-frame input for GR00T N1.7 Hist. For the GDN baseline, we replace each TTT layer with a Gated DeltaNet layer from the Flash Linear Attention library, keeping the layer placement, gating, and parameter count matched to RoboTTT.
Evaluation We deploy trained models on YAM bimanual tabletop robots. To ensure identical initial conditions across methods, we record the initial object placements for each task and reproduce them at evaluation time. We evaluate 20 rollouts for the Pup Go Car and Circuit tasks, and 10 rollouts for the Gear Bot task and for the Circuit task under the one-shot human-video setting, owing to their substantially longer evaluation time.
Frequently Asked Questions
How does RoboTTT update its model weights during deployment? RoboTTT uses a test-time training (TTT) mechanism that performs fast weight updates on incoming context data, decoupled from the slow pre-training updates. This allows the policy to adapt its behaviour on the fly without retraining the entire model.
What hardware is required to run RoboTTT in real time? The system runs on a workstation with an NVIDIA RTX 5090 GPU, processing four 480p RGB camera streams at a control frequency of 30 Hz on a YAM bimanual tabletop robot.
What types of context can RoboTTT learn from? RoboTTT can learn from heterogeneous context sources, including human video demonstrations and the robot's own suboptimal rollouts, by masking the flow-matching loss on selected timesteps.
What are the main limitations of the current RoboTTT approach? Scaling training context length increases training cost, the TTT layers lack robotics-specific objectives, and the method does not handle every deployment failure mode — suggesting reinforcement learning as a complementary next step.