Vision-language models can describe scenes and answer questions, but can they actually control a physical body to perform real-world tasks? HumanCLAW evaluates whether VLMs can act through a human body in a physics-constrained environment, using a benchmark that decouples action reasoning from low-level locomotion control. The agent observes, decides, acts, and receives feedback, but actions are realized as body motion constrained by the physical world. A decoupled motion generator realizes each call as a sub-second, zero-shot-reliable future motion chunk. The observe–think–act loop is dynamic: rather than committing to a fixed plan, the agent continually adjusts to the spatial relation between its body and the scene that its own actions reshape.
Problem Setting
CLAW-like agents make language goals actionable through symbolic interfaces. HumanCLAW studies the same agentic loop when the interface is a full body in physical space. The agent still observes, decides, acts, and receives feedback, but an action no longer ends as a command: it is realized as body motion and constrained by the physical world.
where an atomic skill from a fixed skill set with its continuous parameter is used. A decoupled motion generator then realizes the call as a sub-second, zero-shot-reliable future motion chunk.
Locomotion-Decoupled Physical Simulation
Another challenge in studying embodied action intelligence is how to decouple action reasoning from low-level locomotion motor control. In a fully dynamic humanoid simulation, long-horizon task failure can arise from the motor system. For example, in cluttered environments or on stairs, a humanoid may lose balance and fall, prematurely terminating the episode and preventing a meaningful evaluation of its action intelligence.
In HumanCLAW simulation, the world follows rigid-body physics, including collision, friction, gravity, articulated objects, and movable objects. The human body, however, is not actuated by simulated joint torques. Instead, it is driven by equivalent kinematic velocities derived from the generated motion sequence. This means the agent does not fail because of balance loss or imperfect motor tracking, while stepping on stairs, but still physically interacts with the environment: a wall can block the body, contacts can push movable objects, and unsupported vertical motion remains constrained by gravity and contact.
Rotational components are computed from relative rotations in the corresponding tangent space. These equivalent velocities are applied as the only actuation signal for the humanoid. Following prior passive-body simulation practice, the simulator runs at a specific frequency in Hz and uses passive joint stiffness a specific stiffness value to preserve compliant contact response.
This half-physics design lets HumanCLAW evaluate action intelligence under physical consequences without making the benchmark dominated by locomotion controller failures. The generated motion specifies the intended body movement, while the simulator determines how that movement interacts with the physical world. As a result, failures such as walking into a wall, disturbing objects, or stopping too early are attributable to action-level decisions, while failures from low-level balance control are largely factored out.
Evaluation Metrics
VLMs are evaluated along four dimensions: high-level task success, low-level action quality, body awareness and physical disturbance, and computational cost.
Action Quality. A VLM agent should compose motion skills into smooth and coherent behavior. Poor skill orchestration can lead to unstable motions, such as frequent alternating left and right turns. Action smoothness is measured using motion jerk, the third-order derivative of the agent's kinematic motion.
Body Awareness and Physical Disturbance. Body awareness is a key component of embodied action intelligence. Lack of body awareness can cause unexpected interactions with the environment, such as scraping against a wall while passing through a corridor or knocking over a vase. The average number of steps per episode in which a collision occurs is reported, denoted as Colls. Environmental disturbance is also measured by counting the average number of movable objects that are directly or indirectly displaced by the human agent within an episode, denoted as #Dtb, and the average displacement distance of each disturbed object, denoted as dDtb.
Cost. The average number of tokens consumed by the agent per episode, including both input and output tokens, is reported as a measure of computational cost.
Benchmark
Nine off-the-shelf VLMs are evaluated as the frozen decision maker, with full validation results reported.
Embodied Tasks and Interactive Environments
Existing embodied environments are often mismatched to general-purpose MLLMs. Program-driven environments resolve actions through predefined transitions, leaving little need to reason about body motion or physical consequence. Fully physics-based simulators require high-frequency continuous control and entangle spatial decisions with motor execution. HumanCLAW strikes a middle ground: a frozen MLLM selects parameterized whole-body motion skills based on egocentric observations, utilizing a motion generation model to autoregressively generate continuous long-horizon behavior. Semi-physics execution preserves collision, contact, and object disturbance while abstracting low-level control, enabling basic embodied interaction.
Discussion
This gap is attributed to how these models are built. A VLM trained by visual question answering learns to describe what it sees, not to feel what it does; it treats its own limbs as just more pixels and never forms the proprioceptive, consequence-predicting model that acting through a body requires. The result behaves like a ghost—fluent about the world, oblivious to itself. Closing this gap will likely need more than sharper recognition or a longer context window: it calls for persistent spatial memory, calibrated termination, and an internal model of the body together with the spatial relations each action produces.
Frequently Asked Questions
What makes HumanCLAW different from other embodied AI benchmarks? HumanCLAW decouples action reasoning from low-level locomotion control using a half-physics simulation, so task failures are attributable to action-level decisions rather than balance or motor tracking errors.
How does the semi-physics simulation work? The human body is driven by equivalent kinematic velocities from generated motion sequences, not joint torques, while the environment follows full rigid-body physics including collision, friction, and gravity.
What metrics are used to evaluate VLM agents? Agents are evaluated on task success, action quality (motion jerk), body awareness (collision frequency and object disturbance), and computational cost (token consumption).
What is the main limitation identified in current VLMs? Current VLMs treat their own body as pixels rather than forming a proprioceptive model, making them "fluent about the world but oblivious to themselves."