LLM Agents as Robot Controllers
A second line of work uses large language models to control robots directly: by choosing among learned skills, writing control code, or orchestrating a low-level policy. With frontier models that can use tools for hours, this has become a practical option in simulation. GPT-6 Astra, released in September 2026, was strong enough to solve some BEHAVIOR tasks that our learned policy could not, so our final submission routes 14 tasks to an Astra-based agent (Section 9).
Contributions
- A recipe for growing a task-conditioned VLA from 50 to 100 tasks without retraining from scratch: semantic stages from skill annotations, similarity-based initialization of new task and stage embeddings, and a 21-dimensional action space matching teleoperation controls (Section 3).
- Registers, scratch tokens and attention sinks at several levels of a VLA; moving the pretrained SigLIP attention-sink outliers into the registers by parameter surgery during a running training run made the attention maps readable (Section 4).
- Variance-reduced multi-sample flow matching and a way to choose the number of flow samples from one gradient-noise measurement (Section 5).
- Evaluation infrastructure that makes cross-cloud rollouts inspectable and reusable through provenance, diagnostics and restorable simulator states, and runs six simulators per GPU faster than real time in aggregate (Section 7).
- An LLM agent route with GPT-6 Astra for 14 tasks where the learned policy is weak and the agent performed better (Section 9).
Code, model checkpoints, the 2025 starting checkpoint, and evaluation videos and visual examples are available from the project.
What Carries Over
From our 2025 solution we keep the model and most of the serving logic:
- Task embeddings instead of text: each task has a learned embedding, and the prompt is never tokenized.
- System 2 stages: the model predicts the current stage of the task from the base task token and is conditioned on a stage embedding; at inference, the stage is tracked by voting.
- KV mixing: each action-expert layer attends to a learnable mixture of the keys and values of all VLM layers.
- Multi-sample flow matching: the VLM prefix is computed once, and the action expert is run for multiple flow samples (Section 5).
- FAST auxiliary loss: the VLM also predicts the FAST tokens of the target chunk during training.
- Correlation-aware inpainting at inference: the actions already committed from the previous chunk anchor the next one, and the action correlations carry the correction over to the free actions.
From the LeHome solution we take the training side: a weighted multi-dataset loader, corrected state scaling (in 2025 many normalized state values fell outside the range of the state bins and were clipped), loss masking of padded steps at episode ends, and the augmentation set (crop, rotation, color jitter, cutout, camera and state dropout). New in 2026, frames after the task is complete—the idle tail of each demonstration—are never sampled for training.
Migration from 50 to 100 Tasks
The 2025 solution served four checkpoints, each fine-tuned on a subset of the 50 tasks. In 2026, one checkpoint serves all 100 tasks. It is warm-started from the 2025 checkpoint that carried the largest share of the 2025 score; a single conversion script maps it to the new task set, stage layout and action space. All parameters added in 2026 are zero-, identity- or small-normal-initialized, so the converted model starts close to the 2025 function.
Semantic Stages
In 2025, the System 2 stages were equal slices of each episode in time (5–15 per task), because we did not use the skill annotations. With 20,000 annotated demonstrations, we could check how well these time slices match the actual steps of the tasks, and they match poorly. On average, only 50% of the frames of a time stage belong to its most common annotated step (0.17–0.82 per task); 57% of the 1,093 time stages of the 100 tasks have no step that covers more than half their frames; and the borders of the time stages are no closer to the real step borders than random borders would be. Episode lengths within a task vary by 20%, so in an 11-stage task a fixed time grid can be off by about two stages in either direction.
We replaced them with semantic stages built from the annotations:
- Each episode is cut at the frame where the task is complete, and its sequence of annotated skills is aligned to the typical sequence of the task (the medoid); retries and extra steps are assigned to the current step.
- The steps are grouped into stages at points where the demonstrations agree. Parts that demonstrators do in varying order (for example, sorting items) become order-free progress stages.
- A few rules then adjust the grouping: a grasp and a press-type action (press, switch, toggle) are never in the same stage, so that the gripper rule can tell a missed grasp from a press; similar-looking states of the same appliance are kept in separate stages; repeated steps in a row become one stage; and very short stages are merged into a neighbour unless they are a distinct final action.
The result has 413 stages (2–9 per task) instead of 1,093 time stages. Every label is regenerated from the dataset annotations by deterministic scripts.
Warm-Starting New Tasks and Stages
Stages. The same idea is used one level down. A semantic stage of an old task starts from the average of that task’s 2025 time-stage embeddings, weighted by how many of its frames fell into each time stage. A stage of a new task starts from the matching stages of similar old tasks (for example, “open the fridge” from the old tasks that open a fridge). The output weights of the stage head are converted in the same way.
Minimal Action Space
The 2025 model predicted 32 dimensions, of which 23 were real (base 3, torso 4, two arms of 7, two grippers). We changed the action representation to match the degrees of freedom used in teleoperation:
- The 9 padding dimensions are gone.
- The JoyLo teleoperation system does not move the four torso joints independently. It has two torso controls: translate moves the torso along a fixed path through three key poses, from upright to fully bent down, and tilt adds an offset to the third joint; the fourth joint is always zero. Every torso command in the 211M frames of the dataset has exactly this form, and translate and tilt can be recovered exactly from the four joints. The model therefore predicts translate and tilt, and its outputs are mapped back to the four joints.
The result is a 21-dimensional action and state space: base 3, translate, tilt, left arm 7, left gripper, right arm 7, right gripper. Four correlated, partly redundant torso dimensions became two meaningful ones, and the policy can no longer produce torso configurations the demonstrations never visit. The shared control space also lets a JoyLo operator smoothly take over from the policy during an episode. The weights of the remaining dimensions are copied, and the two new torso rows start at zero.
The same principle applies to the vocabulary: the model only ever embeds the 256 state bins (FAST tokens have their own embedding table), so we keep 256 of Gemma’s 257,152 embedding rows. This removes 526M parameters that received no gradient but were stored four times in every training checkpoint (weights, EMA and the two Adam moments): about 8 GB per checkpoint.
Sinks, Scratch Tokens and Registers
This part of the work started as debugging. To understand failures, we render the attention of the action tokens over the three cameras for every step of a rollout. For the warm-started 2025 model, these maps were nearly useless: the same few cells lit up in all three cameras, in every frame of every episode (episode-mean maps correlate at 0.97).
The cause was the pretrained SigLIP position embeddings: a few cells have outlier norms (cell (12,1) has norm 313 against a mean of 22, mostly in one channel), so their tokens act as content-free “parking spots” for attention. The 12 hottest cells took 31.9% of all image attention, and cell (12,1) alone took up to 29% of the right-wrist camera. This is the artifact described in work on vision transformers and language models: the softmax needs somewhere to put excess attention, and the model repurposes some tokens for it, losing whatever information they carried. Beyond the lost information, it made the attention maps unreadable.
We gave the model dedicated places for this at four levels:
- SigLIP registers: 8 learnable tokens inside the vision tower of each camera, without position embeddings. Their outputs pass through the same projection as the patch tokens and join the prefix (24 tokens), masked together with their camera.
- VLM scratch tokens: 8 learnable tokens in the prefix, attended bidirectionally with the images.
- Action-expert scratch tokens: 4 learnable tokens at the start of the suffix.
- Attention sinks in every attention layer of both transformers: a virtual key with a learnable logit per query head and, unlike gpt-oss, a learnable value per KV head, so attention parked on the sink still returns a learned vector instead of being discarded.
We also added a linear projection per camera after the shared SigLIP encoder, initialized to the identity.
We did not ablate these changes. The loss fully recovered after the surgery, so we do not think they hurt, but apart from making debugging easier we have no evidence that they improve the success rate. The numbers of registers, scratch tokens and sinks were chosen fairly arbitrarily.
Multi-Sample Flow Matching
Multiple Flow Samples per Observation
With [the flow-sampling schedule; the formula is not present in the provided text], which favors the noisy end. All [the number of flow samples] samples share the prefix, and their gradients flow back into the VLM through its keys and values.
Two Sources of Gradient Noise
For a batch of [the batch size] observations, the gradient’s variance has three sources: which observations are in the batch, their random augmentation, and the flow draws [the number of flow samples]. Only the flow-draw term shrinks with [the number of flow samples]:
The flow-draw contribution to gradient variance decreases with the number of samples; the single-draw flow variance is denoted separately.
We measured the three terms with a probe at fixed parameters that resamples one source at a time. In the 2025 setting ([the number of independent flow draws]), the flow draws made up 21% of the gradient variance, augmentation 38% and the choice of observations 42%. At our batch sizes the gradient is dominated by noise, so any reduction of its variance is worth about as much as a proportional increase of the batch.
There are thus two ways to reduce the noise: more flow samples per observation, which shrink only the flow-draw term, or a larger batch, which shrinks everything. Both cost compute. Below we first make each flow sample more useful (Section 5.3) and then compare the two options per unit of compute (Section 5.4).
Other Optimizations
Per-group learning rates. We experimented with several per-group multipliers, applied to the final updates so that they do not touch the optimizer state and can change between resumes. The final recipe uses [the reported multiplier] for the action expert, the action and time projections, the KV mixing and the task/stage embeddings, and [the reported multiplier] everywhere else, including the vision tower, with the gradient clipped at 2.0.
Efficiency. Capping the FAST target at 68 tokens instead of 200 (the mean is 37 per chunk) saved about 9% of the step time, and choosing the token counts so that the attention keys are multiples of 16 saved another 4–5%. Removing the unused vocabulary (Section 3) shrinks the optimizer state and checkpoints by about 8 GB.
Augmentation. Head camera: random crop (98%) and rotation ([rotation amount]); all cameras: color jitter and a cutout rectangle (probability 0.3, up to 10% of the image); each wrist camera is dropped with probability 0.05 and all state tokens with probability 0.02.
Training
The submitted checkpoint is step 200k of one run on all 20,000 demonstrations, warm-started from the converted 2025 checkpoint. The run had two phases:
- [The first phase’s configuration is not present in the provided text], batch 480 (the largest that fits), 130k steps, peak learning rate [learning rate]; a step took about 10.5 s (46 samples/s).
The first half of the B300 phase doubled as a testbed: we tried a number of training parameters and model changes on the live run, resuming from its checkpoints, and kept what worked. For most of the run the optimizer was AdamW with [optimizer settings] and an EMA of 0.995, the cosine schedule had 5k warmup steps, and the loss weights were 1 for flow matching, 0.1 for the stage head and 0.05 for FAST. In total the model saw about 200M training samples, roughly one pass over the 211M frames of the demonstrations.
Evaluation Infrastructure
Evaluating 100 long tasks is expensive: an episode runs for up to tens of minutes of wall time, and a simulator takes 6–9 minutes to boot. Over the competition we ran around 9,000 evaluation episodes, about 100M environment steps or nearly 1,000 simulated hours; we saved each run with its provenance, diagnostics and restorable scene checkpoints, so we could compare models, inspect failures and start follow-up experiments from recorded states.
Cloud Infrastructure
One record across clouds. We ran evaluations on local machines and two cloud platforms (RunPod, then Nebius); all runs were uploaded to one cloud bucket. A web portal indexes them automatically. Each run carries a manifest with its full provenance: code revision and diff, simulator and package versions, all evaluator and serving arguments, and the checksum of the served checkpoint, which links the run to a model registry.
Rich data. Recorded evaluations retain video and resource telemetry, plus per-step actions, observation and policy diagnostics, inference timing and BDDL goal progress. We save the robot and all initialized scene objects, including sleeping and non-task objects, at the start, every 30 steps, and at termination. This gives recovery experiments a restorable scene near a failure while limiting storage and evaluation slowdown; the tradeoff is that we can restore only saved steps. When an action creates new objects, as slicing does, we also save the scene description needed to rebuild them.
Restoration and review. These saved states can be loaded into a fresh simulator to replay part of a rollout or start a targeted recovery experiment. The portal lets people and coding agents find any model and episode, compare outcomes, watch the video next to the goal progress and leave timestamped comments. Agent rollouts use the same store and viewer.
The provided paper text ends mid-section.
Frequently Asked Questions
How does the 2026 policy handle 100 tasks with one checkpoint? It is warm-started from a 2025 checkpoint and uses semantic task stages, similarity-based initialization, and a shared 21-dimensional action space.
Why replace time-based stages with semantic stages? The time slices matched annotated task steps poorly. Semantic stages are built from demonstration annotations and reduce the stage count from 1,093 to 413.
What problem did registers and attention sinks address? A few SigLIP position-embedding cells attracted a large share of image attention, making attention maps difficult to interpret. The model was given dedicated registers, scratch tokens and attention sinks.
How were evaluation runs made reusable? Runs were stored with provenance, diagnostics, video and restorable simulator states, allowing failures to be inspected and follow-up experiments to start from recorded scenes.