Agent as Policy on RoboDojo: 42 Tasks, One General-Purpose Agent

Putting an agent in the control loop.

What if the robot policy were a general-purpose AI agent? We evaluated Agent as Policy (AGP) on RoboDojo, where one agent observes, reasons, commands two robot arms, and checks the result across 42 manipulation tasks.

RoboDojo evaluates generalist policies on a simulated dual-arm ARX X5 robot. Randomized layouts test whether a policy can adapt its actions, handle precise contact, complete long sequences, remember earlier observations, and follow new instructions.

For RealHand, this is a way to study the intelligence behind manipulation: how a robot can turn an instruction into a sequence of physical actions, then use feedback to revise that sequence.

Generalization

Familiar skills with new objects and scene variations.

Precision

Fine insertion, stacking, and alignment.

Long-Horizon

Many manipulation steps completed in sequence.

Memory

Remembering observations from earlier in an episode.

Open

Unseen tasks specified through language.

How AGP works

Observe. Reason. Act. Repeat.

AGP places a general-purpose LLM agent in charge of task execution. In this evaluation, GPT-6-Astra running in Codex uses camera observations and the robot's joint readings to decide what to do next. Motion planning translates the agent's targets into arm trajectories.

The same agent, prompt, and tools are used across all 42 tasks. AGP receives a task instruction and short written task notes, with no RoboDojo demonstrations, benchmark-specific training or fine-tuning, or access to hidden simulator state.

1. Observe

   Read the overhead and two wrist-camera images.

2. Measure

   Point to objects in the images to estimate where to act.

3. Plan & move

   Choose arm targets; a motion planner generates trajectories.

4. Verify & recover

   Look again, check progress, and correct slips or missed grasps.

5. Finish

   Declare completion; RoboDojo's checker scores the episode.

Evaluation results

A strong result across five capabilities.

Our seed-0 run reaches 63.63 average score and 56.13% success. For context, EmbodiedEvo, the top entry in the October 14 leaderboard snapshot, records 45.08 and 38.95%. AGP scores higher than that entry in each of the five capability categories.

Where AGP performs best—and what remains hard

Memory is the strongest category, at 92.22 score and 91.00% success. Long-horizon tasks remain more difficult: their 49.48 score and 28.75% success rate show the gap between making partial progress and reliably completing a whole sequence.

Download AGP results (CSV)Download leaderboard snapshot (CSV)

See it in action

From folding to reasoning.

17 selected demonstrations across all five capabilities. Choose a task, then press play. These clips illustrate individual episodes, not aggregate success rates.

Generalization

Precision

Long-Horizon

Memory

Open

What comes next

Toward more adaptable manipulation.

These simulation results give us a concrete way to test an agent's ability to reason through manipulation, use visual feedback, and recover when an action does not go to plan. They also highlight where reliability needs to improve, especially across long sequences. The linked project and datasets let researchers examine both the outcomes and the steps the agent took to reach them.

Explore the work.

• Read the paper — Agent as Policy for Robotic Manipulation
• Explore the project — Method, experiments, and real-robot results
• Browse the code — Agent as Policy implementation
• Inspect all 2,100 episodes — Simulation videos, observations, and agent execution traces
• See real-robot trials — The separate physical-robot evaluation dataset
• Visit RoboDojo — The benchmark and its official leaderboard

Originally published by RealHand Inc. on October 16, 2026.