HumanCLAW:

Can Vision-Language Models Act Through a Body?

Li Siyao1,2,*,‡, Jiawei Gu3,*, Shuai Liu2,#, Kairui Hu2,#, Zekun Li1,4,‡, Linjie Li3, Chengcheng Tang1, Po-Chen Wu1, Ivan Shugurov1, Lingni Ma1, Michael Zollhoefer1, Sizhe An1, Abhay Mittal1,†, Amy Zhao1,†, Ranjay Krishna3,†, Manling Li5,†, Ziwei Liu2,†, Chuan Guo1,†
1Meta   2Nanyang Technological University   3University of Washington   4Brown University   5Northwestern University
*,#Equal contribution  ‡Intern at Meta  †Corresponding authors
arXiv Code Leaderboard Benchmark

Action through a Physical Human Body in Closed Loop

Deciding an executable action at every moment, from the egocentric view and a high-level instruction.

Stairs Interaction

Action intelligence is the ability to decide, moment by moment, what a physical body should do next inside the execution loop: each decision is executed, and its physical outcome informs the next. We view it as the operational component of spatial intelligence—converting spatial understanding into closed-loop embodied decisions by selecting, parameterizing, and sequencing actions as their physical consequences unfold. It concerns what the body should do next, not the motor-control problem of how the chosen action is executed.

Do VLMs Have Action Intelligence?

“find the stairs and go to the second floor”

The HumanCLAW Framework

HumanCLAW pipeline

HumanCLAW separates three roles that are usually coupled. At each step, the VLM skill harness reads the egocentric view and outputs an atomic skill decision; a verifier rejects unsafe, premature, or physically invalid decisions; the motion generator continues full-body motion conditioned on the accepted skill; and the half-physics simulator executes it with real scene consequences, feeding the next egocentric observation back to the VLM—closing the loop at a sub-second action timescale.

Decision

VLM Skill Harness

Motion

Skill Motion Generation

Consequence

Half-Physics Simulator

VLM skill harness flow

At every 0.5 s step, the frozen VLM receives the system prompt, the task instruction, the registered skill list & rules, the text history, and the current ego view. It then reasons in three stages: percept & understand (spatial layout, target visibility, obstacles → visual state), dynamic mid-level plan (a 2–3 s objective, inherited or revised from the last step), and low-level skill proposal (one atomic skill plus its parameters).

A short-context spatial-action outcome verifier then interrogates the proposal with skill-specific questions—what would this motion do, does it violate a rule?—and accepts, rejects, or renews it from the skill pool before it ever reaches the body.

Skill-conditioned motion generation

A motion-continuation DiT serves as the human motion prior: given 5 history poses, it denoises the next 15 future frames as a velocity field, producing continuous, natural full-body motion chunk by chunk.

One skill, one ControlNet: each skill gets a lightweight ctrl-DiT adapter fed by a skill condition encoder. Adapters live in a plug-and-play ControlNet bag—the VLM’s skill<param> decision routes dynamically to the matching adapter, and a new skill can be registered without retraining the base model.

(a) Without skill control, the prior collapses into colliding poses.

(b) With control, walk tracks the requested 0.3 m stride.

(c) Skills compose with natural transitions into a loop.

(d) sit at heights 0 / 0.5 / 0.8 m realizes the commanded parameter.

Generated motion is executed kinematically while the world responds physically: a wall actually blocks the body (pass wall), a touched object is displaced (object collision), and gravity plus ground contact shape the motion on stairs (climb). Consequences are real, yet balance and motor-tracking failures are factored out—so when an episode fails, the failure is attributable to the decision, not to a locomotion controller.

(a) Pass wall · Kinematic: the body walks straight through the wall.

(a) Pass wall · Half-Physics: the wall blocks the body, which detours around it.

(b) Object collision · Kinematic: the cart never reacts.

(b) Object collision · Half-Physics: the foot knocks the cart away.

HumanCLAW-Bench

HumanCLAW-Bench probes long-horizon action intelligence with a progressive find–navigate–interact task: “find a <obj>, navigate to it with zero distance, and finally sit on it.” Each stage can be attempted only after its prerequisite is met, so the agent must explore and decide autonomously.

Scenes come from HSSD: 41 validation houses give 1,218 episodes over six target categories (chair, bed, couch, potted plant, toilet, TV). Plausibly movable objects are converted to dynamic, so contact displaces them and disturbance is measurable. Every episode is scored along three geometric difficulty dimensions—distance, choice, and obstacle—for difficulty-stratified evaluation.

Success is progressive and self-acknowledged: FindSR (target rendered in ego view and acknowledged), NavSR (within 20 cm of the target and stopping), and InteractSR (sitting with pelvis contact on the target). We additionally measure motion jerk (action quality), collisions and object disturbance (body awareness), and token cost.

Difficulty distributions

Difficulty distributions over the 1,218 episodes. Distance (geodesic length to the nearest target), choice (turns plus rooms traversed), and obstacle (average obstacles within 1 m of the route) are each split into easy / medium / hard by fixed physical thresholds.

Leaderboard

Full validation results on HumanCLAW-Bench (1,218 episodes, frozen off-the-shelf VLMs).

VLM Backbone High-Level Success ↑ Body Misaware & Disturb ↓ Action Quality ↓ Cost
FindSRNavSRInteractSR Coll.#Dtb/epdDtb (m) Motion Jerk avg stepsin tok/stepout tok/step
Gemini-3.164.9%42.5%16.8%39.5%1.501.225.759.15890311
Gemma-4-31B58.1%28.7%11.1%40.1%1.661.864.878.54632322
Gemini-2.558.5%21.6%3.5%37.7%1.353.038.771.36412401
GPT-5.555.8%14.1%3.4%43.4%1.531.564.282.44360354
Claude-4.832.7%8.7%1.5%44.2%1.732.765.381.07047625
Qwen3.6-27B51.0%20.9%0.2%43.0%1.952.166.581.64862482
Qwen3.6-35B-A3B44.6%5.8%0.0%34.6%1.452.387.474.04766477
Qwen3.5-27B37.8%13.5%0.0%34.5%0.932.386.837.64641473
InternVL3.5-38B46.8%0.8%0.0%51.2%1.902.567.293.04459309

FindSR = target rendered in ego view (≥100 semantic px) and acknowledged by the model. Coll. = floor-relative non-ground collision step fraction. Motion Jerk = root-rigid jerk at ≈0.27 s timescale (lower = more coherent). Gold = best, blue = second best per column. Click a metric header to sort.

Key Findings

Finding 1:No current VLM solves the benchmark.

The best model finds and sits on the target in only 16.8% of episodes, and four of the nine sit on it in at most 0.2%. Every model shares the same sharp stage-by-stage drop: most of the task is lost between finding and reaching, and again between reaching and interacting.

Finding 2:Reasoning scaffolds lift action intelligence; longer history does not.

Everything that helps adds structure to the reasoning: a compact structured memory (NavSR 11% → 27% from no history), a mid-level objective binding multi-step routines (InteractSR 0% → 18.9%), and a short-context verifier (NavSR 2% → 27%). Merely lengthening text or visual history saturates—and ten image frames actively hurt.

Error analysis

Where episodes fail across the nine VLMs. Failures trace to a single deficit: current VLMs reason about the scene, but not about the body they now inhabit—a VLM trained to describe what it sees behaves like a ghost, fluent about the world but with no sense of its own body.

Finding 3:Perception matters but is not the bottleneck.

Once a target genuinely enters the ego view it is almost always recognized. Finding breaks upstream, in bringing the target into view at all: the dominant deficit is ineffective exploration (38% of failures), not recognition (23%).

Finding 4:Egocentric self-localization is the navigation bottleneck.

Of 5,473 episodes where the agent actively finds the target, 68% still fail to navigate to it. Nearly two-thirds of these failures are errors of egocentric self-spatial awareness, surfacing as termination decisions that fail in both directions—stopping too early, or never confirming arrival.

Finding 5:Reaching is not interacting.

Given contact with the target, eventual success ranges from 90% (Gemini-3.1) down to 3.5% (InternVL3.5-38B)—an order of magnitude after navigation has already succeeded. The motion layer realizes the sit reliably; what is missing is issuing and confirming it at the right moment.

Finding 6:No current VLM perceives its own body.

Collisions concentrate in the parts the agent does not watch: legs and feet collide most (28–45% of steps), arms and hands next (20–35%), the head almost never (<7%). Body awareness underlies 34% of navigation failures and 81% of interaction failures—a VLM trained by visual question answering behaves like a ghost, fluent about the world but oblivious to the body it now controls.

The hand brushes the chair—the agent never notices. ego view · third-person view

The foot jams against the table—the agent keeps walking. ego view · third-person view

BibTeX

@article{siyao2026humanclaw,
  title   = {HumanCLAW: Can Vision-Language Models Act Through a Body?},
  author  = {Li, Siyao and Gu, Jiawei and Liu, Shuai and Hu, Kairui and Li, Zekun and
             Li, Linjie and Tang, Chengcheng and Wu, Po-Chen and Shugurov, Ivan and
             Ma, Lingni and Zollhoefer, Michael and An, Sizhe and Mittal, Abhay and
             Zhao, Amy and Krishna, Ranjay and Li, Manling and Liu, Ziwei and Guo, Chuan},
  journal = {arXiv preprint arXiv:2607.27180},
  year    = {2026}
}