HumanCLAW:

Can Vision-Language Models Act Through a Body?

Li Siyao^1,2,*,‡, Jiawei Gu^3,*, Shuai Liu^2,#, Kairui Hu^2,#, Zekun Li^1,4,‡, Linjie Li³, Chengcheng Tang¹, Po-Chen Wu¹, Ivan Shugurov¹, Lingni Ma¹, Michael Zollhoefer¹, Sizhe An¹, Abhay Mittal^1,†, Amy Zhao^1,†, Ranjay Krishna^3,†, Manling Li^5,†, Ziwei Liu^2,†, Chuan Guo^1,†

¹Meta ²Nanyang Technological University ³University of Washington ⁴Brown University ⁵Northwestern University

^*^,#Equal contribution ^‡Intern at Meta ^†Corresponding authors

arXiv Code Leaderboard GPT-6 Astra results Benchmark

Action through a Physical Human Body in Closed Loop

Deciding an executable action at every moment, from the egocentric view and a high-level instruction.

Stairs Interaction

Action intelligence is the ability to decide, moment by moment, what a physical body should do next inside the execution loop: each decision is executed, and its physical outcome informs the next. We view it as the operational component of spatial intelligence—converting spatial understanding into closed-loop embodied decisions by selecting, parameterizing, and sequencing actions as their physical consequences unfold. It concerns what the body should do next, not the motor-control problem of how the chosen action is executed.

Do VLMs Have Action Intelligence?

“find the stairs and go to the second floor”

The HumanCLAW Framework

HumanCLAW separates three roles that are usually coupled. At each step, the VLM skill harness reads the egocentric view and outputs an atomic skill decision; a verifier rejects unsafe, premature, or physically invalid decisions; the motion generator continues full-body motion conditioned on the accepted skill; and the half-physics simulator executes it with real scene consequences, feeding the next egocentric observation back to the VLM—closing the loop at a sub-second action timescale.

Decision

VLM Skill Harness

Motion

Skill Motion Generation

Consequence

Half-Physics Simulator

At every 0.5 s step, the frozen VLM receives the system prompt, the task instruction, the registered skill list & rules, the text history, and the current ego view. It then reasons in three stages: percept & understand (spatial layout, target visibility, obstacles → visual state), dynamic mid-level plan (a 2–3 s objective, inherited or revised from the last step), and low-level skill proposal (one atomic skill plus its parameters).

A short-context spatial-action outcome verifier then interrogates the proposal with skill-specific questions—what would this motion do, does it violate a rule?—and accepts, rejects, or renews it from the skill pool before it ever reaches the body.

A motion-continuation DiT serves as the human motion prior: given 5 history poses, it denoises the next 15 future frames as a velocity field, producing continuous, natural full-body motion chunk by chunk.

One skill, one ControlNet: each skill gets a lightweight ctrl-DiT adapter fed by a skill condition encoder. Adapters live in a plug-and-play ControlNet bag—the VLM’s skill<param> decision routes dynamically to the matching adapter, and a new skill can be registered without retraining the base model.

(a) Without skill control, the prior collapses into colliding poses.

(b) With control, walk tracks the requested 0.3 m stride.

(c) Skills compose with natural transitions into a loop.

(d) sit at heights 0 / 0.5 / 0.8 m realizes the commanded parameter.

Generated motion is executed kinematically while the world responds physically: a wall actually blocks the body (pass wall), a touched object is displaced (object collision), and gravity plus ground contact shape the motion on stairs (climb). Consequences are real, yet balance and motor-tracking failures are factored out—so when an episode fails, the failure is attributable to the decision, not to a locomotion controller.

(a) Pass wall · Kinematic: the body walks straight through the wall.

(a) Pass wall · Half-Physics: the wall blocks the body, which detours around it.

(b) Object collision · Kinematic: the cart never reacts.

(b) Object collision · Half-Physics: the foot knocks the cart away.

HumanCLAW-Bench

HumanCLAW-Bench probes long-horizon action intelligence with a progressive find–navigate–interact task: “find a <obj>, navigate to it with zero distance, and finally sit on it.” Each stage can be attempted only after its prerequisite is met, so the agent must explore and decide autonomously.

Scenes come from HSSD: 41 validation houses give 1,218 episodes over six target categories (chair, bed, couch, potted plant, toilet, TV). Plausibly movable objects are converted to dynamic, so contact displaces them and disturbance is measurable. Every episode is scored along three geometric difficulty dimensions—distance, choice, and obstacle—for difficulty-stratified evaluation.

Success is progressive and self-acknowledged: FindSR (target rendered in ego view and acknowledged), NavSR (within 20 cm of the target and stopping), and InteractSR (sitting with pelvis contact on the target). We additionally measure motion jerk (action quality), collisions and object disturbance (body awareness), and token cost.

Difficulty distributions over the 1,218 episodes. Distance (geodesic length to the nearest target), choice (turns plus rooms traversed), and obstacle (average obstacles within 1 m of the route) are each split into easy / medium / hard by fixed physical thresholds.

Leaderboard

Full validation results on HumanCLAW-Bench (1,218 episodes, frozen off-the-shelf VLMs).

VLM Backbone	High-Level Success ↑			Body Misaware & Disturb ↓			Action Quality ↓	Cost
VLM Backbone	FindSR	NavSR	InteractSR	Coll.	#Dtb/ep	dDtb (m)	Motion Jerk	avg steps	in tok/step	out tok/step
GPT-6	75.5%	57.1%	46.6%	41.5%	1.70	0.65	5.3	60.5	4847	308
DeepSeek-V4.1-Flash	68.1%	30.0%	0.2%	38.1%	1.67	0.82	5.6	78.3	4493	368
Gemini-3.1	64.9%	42.4%	16.8%	39.5%	1.50	1.22	5.7	59.1	5890	311
Gemma-4-31B	58.1%	28.7%	11.1%	40.1%	1.66	1.86	4.8	78.5	4632	322
Gemini-2.5	58.5%	21.6%	3.5%	37.7%	1.35	3.03	8.7	71.3	6412	401
GPT-5.5	55.1%	13.9%	3.4%	43.4%	1.53	1.56	4.2	82.4	4360	354
Claude-4.8	32.6%	8.6%	1.5%	44.2%	1.73	2.76	5.3	81.0	7047	625
Qwen3.6-27B	51.0%	20.9%	0.2%	43.0%	1.95	2.16	6.5	81.6	4862	482
Qwen3.6-35B-A3B	44.6%	5.8%	0.0%	34.6%	1.45	2.38	7.4	74.0	4766	477
Qwen3.5-27B	37.8%	13.5%	0.0%	34.5%	0.93	2.38	6.8	37.6	4641	473
InternVL3.5-38B	46.8%	0.8%	0.0%	51.2%	1.90	2.56	7.2	93.0	4459	309

FindSR = target rendered in ego view (≥100 semantic px) and acknowledged by the model. Coll. = floor-relative non-ground collision step fraction. Motion Jerk = root-rigid jerk at ≈0.27 s timescale (lower = more coherent). Gold = best, blue = second best per column. Click a metric header to sort.
Green background = evaluated after the paper's initial arXiv release (July 2026). See GPT-6 Astra results and analysis.

Key Findings

Finding 1:No current VLM solves the benchmark.

The best model finds and sits on the target in only 16.8% of episodes, and four of the nine sit on it in at most 0.2%. Every model shares the same sharp stage-by-stage drop: most of the task is lost between finding and reaching, and again between reaching and interacting.

Finding 2:Reasoning scaffolds lift action intelligence; longer history does not.

Everything that helps adds structure to the reasoning: a compact structured memory (NavSR 11% → 27% from no history), a mid-level objective binding multi-step routines (InteractSR 0% → 18.9%), and a short-context verifier (NavSR 2% → 27%). Merely lengthening text or visual history saturates—and ten image frames actively hurt.

Where episodes fail across the nine VLMs. Failures trace to a single deficit: current VLMs reason about the scene, but not about the body they now inhabit—a VLM trained to describe what it sees behaves like a ghost, fluent about the world but with no sense of its own body.

Finding 3:Perception matters but is not the bottleneck.

Once a target genuinely enters the ego view it is almost always recognized. Finding breaks upstream, in bringing the target into view at all: the dominant deficit is ineffective exploration (38% of failures), not recognition (23%).

Finding 4:Egocentric self-localization is the navigation bottleneck.

Of 5,473 episodes where the agent actively finds the target, 68% still fail to navigate to it. Nearly two-thirds of these failures are errors of egocentric self-spatial awareness, surfacing as termination decisions that fail in both directions—stopping too early, or never confirming arrival.

Finding 5:Reaching is not interacting.

Given contact with the target, eventual success ranges from 90% (Gemini-3.1) down to 3.5% (InternVL3.5-38B)—an order of magnitude after navigation has already succeeded. The motion layer realizes the sit reliably; what is missing is issuing and confirming it at the right moment.

Finding 6:No current VLM perceives its own body.

Collisions concentrate in the parts the agent does not watch: legs and feet collide most (28–45% of steps), arms and hands next (20–35%), the head almost never (<7%). Body awareness underlies 34% of navigation failures and 81% of interaction failures—a VLM trained by visual question answering behaves like a ghost, fluent about the world but oblivious to the body it now controls.

The hand brushes the chair—the agent never notices. ego view · third-person view

Interact

Stand after sit

GPT-6 Astra: Results and Analysis

GPT-6 Astra was released after the nine-model evaluation in the paper. We evaluated it on all 1,218 full-validation episodes of HumanCLAW-Bench with the same harness and metric definitions; the leaderboard above reports the main results. GPT-6 sharply improves every stage over the best of the nine models (Gemini-3.1):

75.5%

FindSR (best of nine: 64.9%)

57.1%

NavSR (best of nine: 42.4%)

46.6%

InteractSR (best of nine: 16.8%)

Three changes stand behind the numbers. Perception: failures caused by not recognizing a target that is in view fall from 11.4% of all episodes to 2.5%. Body self-localization: stopping while far, not stopping on arrival, and being jammed unawares fall from 21.7% of all episodes to 8.1%. Decisiveness: GPT-6 never stops in front of the seat without sitting and never sits and stands up again, and it completes far more tasks within the same budget of decision steps.

Where GPT-6 Astra fails: failure funnel and per-stage root causes — **Where GPT-6 Astra fails.** (a) Find→Nav→Interact funnel over all 1,218 episodes; (b–d) per-stage root-cause breakdowns colored by faculty, with the same taxonomy and rules as the nine-model analysis above.

Three failure modes of the nine models that GPT-6 fixes

Same episode, same harness: left, one of the nine models; right, GPT-6. Each side shows the egocentric view (top) and the third-person view (bottom); the shorter side freezes on its last frame.

Unaware arrival → stops at the target

Qwen3.6-35B-A3B walks up to the chair and keeps navigating for all 100 steps without ever acknowledging that it has arrived. GPT-6 reaches the chair and stops at step 10.

104348328 · ep11 · chair

Sits on the wrong object → sits on the bed

Gemma-4-31B commits to the sit next to the bed and lands on the nightstand. GPT-6 aligns with the bed and sits on it at step 15.

104348028 · ep27 · bed

Stands up after sitting → sits and holds

Gemma-4-31B sits down on the toilet and then stands up again instead of stopping. GPT-6 sits down at step 20 and holds the pose.

102816114 · ep23 · toilet

Three failure modes that remain

What remains is concentrated in body awareness and in the spatial outcome of the agent’s own actions: 44% of the remaining navigation failures are self-localization errors, every interaction failure is a misplaced sit (69% beside the seat, 31% onto the wrong surface), and ineffective pacing or spinning still causes 35% of the find failures and 17% of the navigation failures. Each video shows the egocentric view (left) and the third-person view (right) of one GPT-6 episode.

Find failure: ineffective actions (35% of GPT-6 find failures)

The bed is 2.9 m away at the start, yet the agent never brings it into view: for 57 consecutive steps it stays inside a 2 m box, and its 64 walk and 34 turn commands trace 20.2 m of path for 2.9 m of net displacement.

105515379_173104395 · ep0 · bed

Jammed without noticing (1% of GPT-6 find failures, 9% of its navigation failures)

Searching for the toilet, the agent walks into the gap between a bed and the long, low cabinet under the TV and stays wedged there for the whole episode: 60 of its 84 locomotion commands move the body less than 10 cm, yet its visible state keeps reporting “a clear wooden-floor aisle ahead” and an open doorway two to three meters away that it never reaches.

106878915_174887025 · ep26 · toilet

Interaction failure: sitting into thin air (69% of GPT-6 interaction failures)

The agent reaches the couch (final body–target distance 0.04 m), turns to align, and issues three consecutive sit commands at a 0.45 m seat height before stopping, but the pelvis comes down beside the cushion and never touches it.

103997895_171031182 · ep28 · couch

BibTeX

@article{siyao2026humanclaw,
  title   = {HumanCLAW: Can Vision-Language Models Act Through a Body?},
  author  = {Li, Siyao and Gu, Jiawei and Liu, Shuai and Hu, Kairui and Li, Zekun and
             Li, Linjie and Tang, Chengcheng and Wu, Po-Chen and Shugurov, Ivan and
             Ma, Lingni and Zollhoefer, Michael and An, Sizhe and Mittal, Abhay and
             Zhao, Amy and Krishna, Ranjay and Li, Manling and Liu, Ziwei and Guo, Chuan},
  journal = {arXiv preprint arXiv:2607.27180},
  year    = {2026}
}