The data ecosystem for embodied manipulation is undergoing a structural transformation, shifting from robot-specific datasets toward multi-source, cross-embodiment resources. This paper presents the Data Pyramid for Embodied Manipulation, a unified framework that categorizes data from three complementary sources — real-robot, human demonstration, and simulation — and examines the pipelines, representations, and trade-offs that define each tier.
Embodiment and Modalities
These differences affect not only the dimensionality of the learning problem, but also the extent to which data can be shared across platforms. Joint-space representations provide direct and precise control for articulated hands, but are strongly tied to robot morphology. In contrast, end-effector-centric representations can offer a more transferable interface for arm motion across embodiments, although they still require consistent coordinate frames and control conventions. In policy learning, proprioceptive states and actions are often normalized to maintain suitable numerical scales and facilitate learning across embodiments. Common choices include MinMax, Q01-Q99, and MeanStd normalization.
Data Collection Paradigms
Scripted collection broadly refers to data-generation processes in which robot behavior is executed without continuous human control. Rather than manually controlling each episode, human effort is shifted toward designing the task logic, control pipeline, and execution conditions. This separation between system design and episode-level execution enables repeated and parallel data collection at relatively low marginal cost. Depending on how robot actions are produced, scripted collection can be broadly categorized into rule-based execution, trajectory playback, and autonomous policy rollouts.
Advantages and Limitations of Real-Robot Data
Real-robot data provide physically grounded supervision by directly capturing the relationship between robot observations, actions, and their physical consequences. Compared with simulation or human videos, they naturally reflect real sensing noise, control latency, contact dynamics, and hardware constraints. Moreover, the recorded actions are directly executable on the corresponding platform, making such data particularly valuable for learning reliable manipulation policies and recovery behaviors.
The main limitation is the high cost of collection. Real-robot data require physical hardware, human operation or automated collection pipelines, repeated environment resets, maintenance, and safety supervision, which restrict large-scale and parallel data acquisition. In addition, many datasets remain concentrated on specific robots, tasks, and environments. Differences in embodiment, sensor configuration, coordinate systems, and action representation further complicate data aggregation and cross-platform transfer.
Universal Manipulation Interface (UMI) Data Overview
Universal Manipulation Interface (UMI) data denotes a family of real-world manipulation datasets collected through portable, robot-independent interfaces rather than directly operating a target robot. A typical UMI system integrates one or more cameras, a hand-held gripper or dexterous interface, and a pose-tracking module to record visual observations, end-effector motion, gripper states, and, in some recent systems, force or tactile signals. These demonstrations can then be converted into robot-compatible trajectories through calibration, action transformation, and embodiment-specific retargeting.
UMI Data Collection Pipeline
The data produced by UMI-style systems has also become increasingly diverse in terms of embodiment, sensing modality, and scale. Early UMI datasets primarily focused on single-arm manipulation with parallel grippers and wrist-view RGB observations. Subsequent work extended this paradigm to mobile manipulators and whole-body humanoid systems, as in UMI on Legs and HuMI, as well as to bimanual manipulation and dexterous hands.
Cross-Embodiment Deployment
Cross-embodiment deployment aims to transfer policies learned from robot-free UMI demonstrations to robots with different kinematic structures and sensing configurations. The key idea is to separate an embodiment-agnostic task representation from embodiment-specific execution. Rather than predicting joint-space commands tied to a particular robot, UMI-style policies operate in an end-effector-centric action space, typically producing relative six-degree-of-freedom trajectories and gripper commands. During deployment, the predicted relative trajectory is composed with the robot's current end-effector pose and transformed into the robot base frame. The resulting target poses are then converted into executable joint commands through inverse kinematics, motion planning, or a low-level Cartesian controller.
Advantages and Limitations of UMI Data
The primary advantages of UMI data are its scalability, environmental diversity, and reduced dependence on a specific robot embodiment. In particular, its portable and robot-free collection setup enables in-the-wild data acquisition in diverse everyday environments. Human operators can collect smooth and natural manipulation demonstrations across diverse real-world settings, while end-effector-centric action representations facilitate reuse across different robot platforms. Recent extensions incorporating bimanual motion, dexterous hands, and tactile sensing further broaden the range of behaviors captured by UMI-style systems.
However, UMI data does not fully eliminate the embodiment gap. Differences in camera viewpoints, end-effector geometry, kinematic constraints, dynamics, and contact conditions may lead to discrepancies between demonstration and robot execution. Its quality also relies on accurate pose tracking and sensor synchronization, which may degrade under occlusion, rapid motion, or visually challenging environments. Moreover, UMI typically lacks robot-specific proprioception and actuator dynamics. Therefore, it is best regarded as a scalable complement to embodiment-specific real-robot data rather than a complete replacement.
Capture Infrastructure for Human Demonstration Data
Egocentric and exocentric capture systems combine wearable and environmental devices to measure appearance, geometry, motion, attention, and physical interaction. We organize the hardware by the physical quantity it observes: visual sensors record image and depth streams, motion-tracking systems measure device or body kinematics, and auxiliary sensors expose attention, muscle activation, and contact. Across these categories, the main trade-offs concern spatial coverage, temporal resolution, mobility, intrusiveness, accuracy, and calibration requirements.
Wearable Cameras. Monocular RGB capture uses a single color camera, typically mounted on the head or chest, to approximate the wearer's visual stream. Such cameras are lightweight, inexpensive, and suitable for long-duration collection, as demonstrated by EPIC-KITCHENS and Ego4D; however, they do not measure metric depth directly, and rapid head motion can introduce blur while the hands can occlude manipulated objects.
Event cameras take a different approach: rather than producing conventional frames at a fixed rate, each pixel asynchronously reports changes in brightness. Their high temporal resolution reduces motion blur during fast hand motion, but produces sparse data that requires specialized processing; EgoEV-HandPose explores this setting with stereo event cameras.
Exocentric Camera Systems. Exocentric systems place one or more cameras in the environment to observe the wearer from fixed third-person viewpoints. They reveal body, hand, and scene context that may be occluded or outside the wearable camera's field of view, and synchronized views support triangulation and multi-view reconstruction. Assembly101 combines wearable and external cameras for fine-grained assembly activities, while Ego-Exo4D pairs Project Aria glasses with multiple surrounding GoPro cameras. The additional coverage comes at the cost of environmental setup, cross-camera calibration and synchronization, higher storage demand, and reduced collection mobility.
Supervision Construction for Human Data
Egocentric supervision may be recorded by sensors, added by human annotators, or recovered through post-processing. We group it into four complementary categories according to the information exposed to learning: semantic supervision describes task content and structure; geometric supervision describes spatial state and motion; multimodal supervision supplies attention, dynamics, and contact cues beyond RGB; and robot-oriented supervision expresses human demonstrations in forms intended for robot policy learning. Their construction combines annotation, calibration, synchronization, reconstruction, and cross-embodiment alignment, with different costs and sources of error.
Procedural and State Annotations. Procedural annotations describe how individual actions compose into a task. They include ordered steps, task progress, execution errors, and object or environment state changes, and are typically constructed by aligning videos with task scripts and verifying deviations from the expected procedure. Assembly101 organizes fine-grained actions within assembly procedures, whereas CaptainCook4D labels cooking steps together with errors and object states. This supervision exposes long-horizon dependencies and task outcomes that isolated action labels miss, although its vocabulary is often specific to a particular procedure or domain.
Advantages and Limitations of Human Demonstration Data
Egocentric and exocentric data capture natural human demonstrations at scale without requiring execution on a target robot. Wearable systems can be deployed across different participants, environments, objects, and tasks, while first-person observations emphasize hand-object interaction and long-horizon activity structure. Unlike fixed-base robot collection, human operators can move freely through the environment and manipulate objects directly with dexterous hands. This mobility enables broader spatial coverage and more diverse hand trajectories, including variations in approach direction, motion path, and interaction height. The EgoVerse example illustrates this potential: its human-hand interaction keyframes span a broad three-dimensional workspace and exhibit substantial spatial variation.
However, egocentric data are not equivalent to robot experience. First-person recordings suffer from partial observability, hand-object occlusion, rapid camera motion, motion blur, and device-dependent viewpoints. Dense geometric and multimodal supervision requires additional hardware, calibration, synchronization, and reconstruction, and is not uniformly available at scale. More fundamentally, human recordings lack robot proprioception, actuator dynamics, and directly executable actions. Differences in morphology, kinematics, sensing, and contact response further prevent direct trajectory transfer. Their use in robot learning therefore requires reliable state reconstruction, embodiment-aware representations, and human-to-robot alignment or retargeting.
Simulation Infrastructure and Asset Ecosystems
A simulated robot-learning ecosystem can be viewed as a composition of three coupled infrastructure components: embodiment-sensor systems, object and scene assets, and physics and rendering backends. The embodiment-sensor system defines not only the robot morphology, kinematics, actuation model, and controller interface, but also the sensing channels physically associated with the robot body, such as proprioception, wrist or head-mounted vision, force sensing, and tactile arrays. Object and scene assets provide the geometric, semantic, and affordance structure of the environment, while physics and rendering backends determine the contact dynamics, visual appearance, and execution fidelity of the generated data.
Constructing simulated manipulation data first requires importing an embodiment-sensor system into the simulator, where the robot asset is instantiated as a controllable articulated model and its sensors are configured to acquire observations during interaction. The simulator parses robot descriptions such as URDF, MJCF, SDF, or USD to specify kinematics, joint limits, inertial properties, collision and visual geometries, actuators, and sensor attachments, before placing the system in a scene with defined initial pose, joint state, objects, lighting, gravity, and solver settings.
Benchmarks and Simulation Datasets
Table 5 summarizes large-scale synthetic and simulation-based data resources in terms of data volume, task coverage, embodiment diversity, and supported manipulation settings. These resources range from task-level manipulation trajectories to large-scale grasp and dexterous-hand samples. Recent datasets increasingly incorporate multiple embodiments and substantially larger data volumes, including hundreds of thousands or millions of trajectories and, for grasp-centric resources, substantially larger collections of generated samples. However, most datasets remain concentrated on rigid or articulated-object manipulation, with comparatively limited coverage of deformable interaction and mobile manipulation.
Sim-to-Real Gap
Sim-to-real transfer has long been a central problem in simulation-based robot data, as synthetic trajectories are expected to support real-robot deployment rather than only in-simulator learning. From a data-centric perspective, this gap mainly arises from observation mismatch and interaction mismatch.
The first concerns the discrepancy between rendered sensory observations and real-robot perception. Although recent simulation datasets improve realism through 3D assets, lighting, camera models, articulated objects, and multimodal sensors, their observations can still differ from real data in texture, illumination, material appearance, depth noise, camera calibration, occlusion, tactile response, and force signals.
Advantages and Limitations of Simulation Data
Simulation data provide scalable supervision by generating robot observations, actions, and task outcomes in controllable virtual environments. Compared with real-robot data, they can be collected with lower marginal cost, extensive parallelization, limited safety risk, and no hardware wear. Simulators also provide privileged labels that are difficult to obtain from real systems, such as object poses, segmentation masks, contact states, physical parameters, success signals, and dense rewards. These properties make simulation useful for large-scale policy pretraining, controlled evaluation, scene and task randomization, and the synthesis of diverse interaction trajectories.
A primary limitation of simulation data is limited physical fidelity. Approximations in sensing, actuation, contact, and object dynamics can produce substantial sim-to-real discrepancies, particularly for contact-rich manipulation, deformable objects, reflective materials, liquids, and long-horizon tasks. Moreover, scale does not necessarily imply diversity: as shown by InternData-A1, skill-constrained generation can produce many trajectories with limited task variation.
Frequently Asked Questions
What is the Data Pyramid for Embodied Manipulation? It is a unified framework that categorizes manipulation data into three complementary sources — real-robot, human demonstration, and simulation — and examines the pipelines, representations, and trade-offs that define each tier.
How does Universal Manipulation Interface (UMI) data enable cross-embodiment deployment? UMI data uses end-effector-centric action representations that separate embodiment-agnostic task information from robot-specific execution, allowing policies to be transferred across different robot platforms through calibration and retargeting.
What are the main limitations of using egocentric human videos for robot learning? Human recordings lack robot proprioception, actuator dynamics, and directly executable actions, while differences in morphology, kinematics, and sensing prevent direct trajectory transfer without reliable state reconstruction and human-to-robot alignment.
Why is simulation data useful despite the sim-to-real gap? Simulation enables scalable, parallel data collection with privileged labels like object poses and contact states, and allows safe, controlled evaluation and task randomization, making it valuable for policy pretraining even though physical fidelity remains a challenge.