PlayWorld makes an agent play the world model, because fixed action scripts can't compare models
There's a measurement bug at the center of world model evaluation and PlayWorld is the first benchmark to take it seriously. If you evaluate a world model by feeding it a fixed action sequence and scoring the frames, you've assumed every model needs the same actions to reach the same outcome. They don't. The action sequence that gets model A to the objective may be nothing like the one model B needs. Which means fixed action-conditioned evaluation is not a cross-model comparison at all. Paper is arXiv 2608.13552 from the University of Hong Kong, submitted August 13, at 51 upvotes on HuggingFace.
The fix: put an AI agent in the loop as the player. Give it an objective, let it interact with the world model however it wants, and score whether the objective got reached and whether the world held together while it happened. 171 scenarios with long-horizon objectives. Four core dimensions — geometry consistency, interaction fidelity, out-of-sight evolution, and insight evolution — plus the usual video quality and controllability metrics. Nine state-of-the-art world models put through it. Code, data and a project page are all published.
The out-of-sight dimension is the sneaky one. It asks whether the world keeps evolving correctly when the camera isn't looking at it. That's the difference between a model that renders plausible frames and a model that actually holds a world. Most systems that look great in a highlight reel are quietly regenerating the room every time you turn around.
The verdict on the nine models is blunt: they struggle to maintain spatial consistency and persistent state evolution across extended interactions. Nobody has a world model that survives being played with for a long time.
Two things make this matter beyond graphics. First, the methodology — using an agent as the measuring instrument rather than a fixed script — generalizes to any system where the path to a goal is model-dependent, which is most agentic evaluation. Anyone still scoring agents on fixed trajectories should read the argument section. Second, pair it with Alaya-EVOKE, also posted today, which handles persistent state by moving it out of the model into an external store. PlayWorld quantifies the failure. Alaya-EVOKE proposes the architecture. That's a rare same-day matched set.
← Back to all articles
The fix: put an AI agent in the loop as the player. Give it an objective, let it interact with the world model however it wants, and score whether the objective got reached and whether the world held together while it happened. 171 scenarios with long-horizon objectives. Four core dimensions — geometry consistency, interaction fidelity, out-of-sight evolution, and insight evolution — plus the usual video quality and controllability metrics. Nine state-of-the-art world models put through it. Code, data and a project page are all published.
The out-of-sight dimension is the sneaky one. It asks whether the world keeps evolving correctly when the camera isn't looking at it. That's the difference between a model that renders plausible frames and a model that actually holds a world. Most systems that look great in a highlight reel are quietly regenerating the room every time you turn around.
The verdict on the nine models is blunt: they struggle to maintain spatial consistency and persistent state evolution across extended interactions. Nobody has a world model that survives being played with for a long time.
Two things make this matter beyond graphics. First, the methodology — using an agent as the measuring instrument rather than a fixed script — generalizes to any system where the path to a goal is model-dependent, which is most agentic evaluation. Anyone still scoring agents on fixed trajectories should read the argument section. Second, pair it with Alaya-EVOKE, also posted today, which handles persistent state by moving it out of the model into an external store. PlayWorld quantifies the failure. Alaya-EVOKE proposes the architecture. That's a rare same-day matched set.
Comments