Related Work
An extended discussion of related work is included in Appendix A.
Background: Student–Teacher Distillation
The following sections focus on key paradigms for teacher–student distillation under partial observability, discuss why they break down when learning exploration in asymmetric settings, and motivate ASTEROID as an iterative on-policy distillation method for learning in-context exploration.
Experiments
The experiments address these questions:
- Does ASTEROID learn to solve partially observed problems efficiently at deployment?
- Is ASTEROID sample-efficient compared with baselines?
- What exploratory behavior do policies trained with ASTEROID display at test time?
- Does ASTEROID learn exploration policies that can be deployed on real-robot hardware?
- Do frontier models perform in-context exploration?
- Is learning from on-policy contexts necessary to learn exploration efficiently?
Tasks
The approach is evaluated on a diverse suite of partially observable decision-making problems across robotic manipulation, discrete exploration, and continuous navigation. The tasks require exploration, memory, and adaptation under limited sensing.
Partially Observed Gridworlds
Partially observed gridworlds are a classic benchmark of discrete search problems. An agent acts over multiple episodes in a gridworld to reach one or more goals. The agent acts in a grid of unspecified dimensions and must perform targeted exploration, such as finding a hidden goal or finding a hidden key and door sequentially across multiple episodes.
Meta-Learning Navigation
Meta-learning navigation requires an agent to navigate an environment to a hidden goal. The study uses two settings: Ant Navigation, where the agent is an 8-DoF robot, and 2D Navigation, where the dynamics are reduced to a simple point mass.
Procgen Maze Navigation
Procgen Maze Navigation involves finding the goal in procedurally generated mazes. The agent receives only a local observation of unspecified dimensions and must infer the maze layout and goal location from its interaction history.
Embodied Object Search
Embodied Object Search involves searching for a household object in a fixed environment in Habitat-Sim. It provides a visually complex, realistic partially observed problem.
Simulated Robotic Manipulation
Simulated Robotic Manipulation includes high-dimensional, continuous-control tasks under partial observability: Wrist-Camera Peg-Insertion and Tactile-Pick. The robot must use limited sensing to manipulate objects—for example, exploring the table using only the wrist view or using touch sensing to locate an object and complete the task.
Real-Robot Manipulation
One simulated task is reconstructed in the real world: the robot must pick up a cube using only proprioception from the arm and gripper. Policies are trained in simulation with domain randomization and then transferred to the real world.
Baselines
The study compares ASTEROID against ten strong baselines from several frameworks for partial observability. The provided text describes two:
- RL² (Meta-RL): Trains a recurrent policy to adapt to hidden parameters purely through interaction. The policy’s hidden state acts as an implicit learning algorithm that updates from histories across episodes.
- Distilled Planner: For embodied navigation in Habitat, a policy is trained to perform posterior-sampling reinforcement learning (PSRL), providing an upper bound on performance.
Do Frontier Models Perform In-Context Exploration?
Given recent progress in capable foundation models, the study asks how well frontier models perform when used for exploration in partially observed environments. The experiment uses Peg-Insertion: the robot must use its wrist camera to explore the table, locate and pick up the peg, and insert it into the peg hole.
Acknowledgments
The authors thank members of the WEIRD Lab for helpful feedback and insightful discussions. They also thank the authors of OmniReset for guidance in setting up simulation experts and sim-to-real transfer of robot policies. The authors acknowledge the UW Hyak Team for providing high-performance computing resources that contributed to the research results. This work was partially funded by NSF Award 2535892.
Robustness to Noise in Privileged Expert Supervision
The authors evaluate ASTEROID’s robustness in an ablation study where a noisy expert supervises the student. They inject Gaussian noise into the expert actions during training, with the noise parameter unspecified in the provided text, and find that ASTEROID remains performant under noisy supervision. The slight performance improvement may result from the regularizing effect of learning from noisy targets.
Emergent In-Context Memory
To investigate whether the policy learned with ASTEROID attends to context, the authors construct a variant of the Key-Door environment. The agent observes two boolean flags in its state whenever it lands on the key or door during exploration.
Policies trained with ASTEROID exhibit emergent in-context memory. If the agent discovered the key or door in earlier attempts, it can later retrieve that information from context and use it efficiently.
Training Details
ASTEROID uses a causal transformer. Each token encodes an observation, previous action, reward, and terminal signal tuple through an MLP encoder. Images are encoded with an ImageNet-pretrained ResNet18. For discrete actions, the model predicts logits over the action space; for continuous actions, it predicts the mean and standard deviation of a Gaussian.
Frequently Asked Questions
What is ASTEROID designed to learn? ASTEROID is an iterative on-policy distillation method for learning in-context exploration under partial observability.
What kinds of tasks are used in the experiments? The tasks span gridworld exploration, navigation, embodied object search, simulated robotic manipulation, and real-robot manipulation.
How does the Key-Door experiment test in-context memory? The agent observes flags when it lands on the key or door. The reported policies can later retrieve discoveries from earlier attempts and use them.
What architecture does ASTEROID use? It uses a causal transformer with an MLP encoder for observation-action-reward-terminal tokens, and a pretrained ResNet18 for image observations.