RobotMe summary
Researchers introduced WorldOracle to test whether action-conditioned world models predict what actually happens when a robot or human performs a specific sequence of movements. Each model receives an initial observation and recorded low-level actions, then produces a predicted video that is compared with the paired real-world execution.
The benchmark examines physical state correctness, spatial alignment, fidelity around the manipulated object, and consistency across camera views. Its evaluation data include both successful and failed interactions and longer sequences. That matters because a model can produce a plausible-looking success even when the real gripper missed an object.
The authors evaluated seven representative models and reported distinct strengths and failure modes. They released datasets, code, and a leaderboard. The work is a research preprint, so these benchmark results should not be read as a real-world safety certification or as evidence that a model can reliably operate a physical robot on its own.