HarnessEval-W evaluates interactive visual worlds across diverse settings — Project page
HarnessEval-W starts from a simple but disruptive premise: a benchmark should explain why it gave a score.
That sounds obvious. In practice, many AI benchmarks still behave like calculators. They take a model output, apply a fixed rubric or a bundle of metrics, and return a number. That worked reasonably well when the object being tested looked like a function: one prompt in, one answer out.
An interactive AI system is different. It observes a state, receives an action, changes the environment, and carries that state forward. Evaluating it means checking not only whether the final frame looks plausible, but whether the right object moved, whether an unrequested event appeared, whether physics remained coherent, and whether an off-screen process continued rather than freezing or resetting.
MirroS and a large group of academic and industry collaborators introduced HarnessEval-W to turn that evaluation process into an executable agent system. Instead of forcing every case through the same long rubric, the evaluator reads the case, decides what should be tested, selects relevant evaluation skills, delegates measurable questions to specialized sub-agents, validates their evidence, and assembles the result into an inspectable evidence tree.
The proposal is larger than one world-model leaderboard. It suggests that as AI products become systems, evaluation must become a system too.
What Is an Evaluation Harness?
An evaluation harness is the runtime around a benchmark. It manages how cases are loaded, which model or system is invoked, what tools are available, how evidence is collected, how scores are computed, and how results are reported.
The term is not new. Projects such as EleutherAI's lm-evaluation-harness standardized model testing across many language tasks. Production teams also use eval harnesses to run regression suites, compare prompts and models, inspect traces, enforce budgets, and decide whether an AI change is safe to ship.
HarnessEval pushes the concept further. Its argument is that evaluation itself can be agentic.
The distinction is useful:
| System layer | Primary job |
|---|---|
| Model | Predict or generate the next output |
| Agent harness | Organize context, tools, memory, permissions, execution, and recovery so the model can complete a task |
| Evaluation harness | Organize questions, diagnostic tools, evidence, validation, and scoring so the result can be judged |
If an agent harness structures how an AI works, an evaluation harness structures how we decide whether that work was actually good.
Why a Fixed Rubric Breaks on Interactive AI
The supplied Machine Heart article organizes the problem around a shift from models to systems, and that structure is worth preserving.
A fixed rubric assumes that the same questions apply to every case. But interactive environments are context-dependent. Consider three tasks:
- move a robot gripper while keeping the other gripper still;
- navigate away from a location and return to it;
- leave flowing water off screen and later check whether it continued to evolve.
The first needs target tracking and protected-state checks. The second needs revisit consistency. The third needs off-screen state reasoning. Running every possible metric on every case creates noise; simplifying the rubric risks missing the failure that actually matters.
Human evaluators do not work that way. They first understand the case, decide which questions matter, locate relevant evidence, use tools when necessary, and investigate further when the evidence is weak.
HarnessEval-W tries to make that hidden workflow explicit.
Official HarnessEval-W overview and illustrative case. The displayed scores apply to this example and do not by themselves prove benchmark-wide effectiveness — Source
How HarnessEval-W Works
The evaluation proceeds as a hierarchy rather than a single judge call.
1. Plan: interpret the case
The planner reads the initial world, the requested action, and the evaluation setting. It determines the world-state questions that the case can legitimately answer.
2. Route: select relevant skills
The system chooses from an extensible skill library. For a visible object edit, it may activate intentional-change and render-quality checks while skipping off-screen evolution. The paper says routing depends on the case context—not the identity of the model being evaluated—so models face the same questions on the same case.
3. Decompose: turn judgment into measurable questions
A high-level question is split into smaller checks. An intentional-change verifier can ask:
- Was the target visible?
- Did the requested transition occur?
- Was the correct target changed?
- Did protected state remain stable?
- Did any unrelated event appear?
- Is the final state valid and judgeable?
Specialized sub-agents or tools inspect the relevant frames and return discrete scores with diagnostics.
4. Verify: audit the evidence before scoring
The parent agent checks whether each branch produced enough evidence and whether it answered the assigned question. Only then does it aggregate the results.
The final artifact is not only a score. It is an evidence tree recording what was tested, why it was tested, which tools produced the evidence, and how the conclusions support the verdict.
What HarnessEval-W Actually Evaluates
HarnessEval-W is the first released implementation of the broader HarnessEval idea. It focuses on interactive visual world models: systems that generate future observations conditioned on prior observations and user actions.
The benchmark contains:
- 330 evaluation cases;
- 11 specialized evaluation skills;
- 5,940 scored rollouts;
- 18 representative world models.
The cases cover three top-level capabilities.
Observation Quality
Can the system render a coherent, stable, readable world state? This includes ordinary visual quality and whether the depicted scene is structurally and physically plausible.
Transition Correctness
Does the world change in the requested way? HarnessEval-W separates:
- exploratory transitions, such as a viewpoint or navigation change;
- intentional transitions, such as changing a specified object or event;
- physical transitions, where an intervention should produce a plausible dynamical response.
World Persistence
Does the generated world remain coherent over time? The benchmark tests:
- drift resistance during long rollouts;
- revisit consistency after leaving and returning;
- off-screen evolution for processes that should continue while unseen.
This decomposition matters because a beautiful frame does not prove that the system understands action, cause, or persistent state.
Official HarnessEval-W pipeline for one intentional-transition case. It illustrates case-specific routing and evidence collection; use click-to-enlarge for the full-width labels — Source
The Most Important Output Is the Evidence Tree
The central editorial idea in the original article is not the leaderboard. It is the move from metric to harness.
A scalar score tells you who ranked higher. An evidence tree can tell you why a model failed.
Suppose an output shows the requested robot gripper lifting a spoon. A generic visual judge may call the result correct. A case-specific evaluator can notice that a second gripper also moved, or that the model introduced an extra spoon. That difference turns a vague low score into an actionable diagnosis:
- the target action occurred;
- a protected object changed when it should not have;
- an unrequested event was introduced;
- the failure belongs to intentional transition, not basic rendering.
For research teams, that diagnosis can guide the next training run. For product teams, the same principle can distinguish whether a failed agent task came from the model, the tool call, stale context, a permission boundary, or the evaluator itself.
Does the Evaluator Agree With Humans?
The authors do more than publish model scores; they evaluate the evaluator.
They collected 5,000 human A/B judgments across nine representative models and aggregated the preferences with a Bradley-Terry model. In the paper's reported results:
- Intentional Transition reached a Spearman rank correlation of 0.93 with the human ordering;
- Physical Transition reached 0.87;
- on Physical comparisons, pairwise accuracy increased from 31.9% under the compared WBench protocol to 71.7% under HarnessEval-W;
- on Intentional comparisons, accuracy increased from 60.2% to 77.8%;
- repeated evaluation produced an uncertainty envelope reported as 4.9× narrower than WBench.
These are authors' results on their benchmark and experimental setup, not independent replication. They nevertheless address the right question: if an agentic evaluator adds planning and sub-agents, does it produce decisions that are more aligned and repeatable than a simpler protocol?
What the First Leaderboard Shows
HarnessEval-W evaluates 18 world models on the same 330-case set. In the August 18, 2026 V1 project snapshot, Seedance 2.0 ranks first overall, followed by Wan 2.7 and Kling 3.0.
The more useful conclusion is not that one model “wins.” The paper reports that different capabilities move differently:
- observation quality is not the same as physical plausibility;
- exploratory transitions are weakly related to intentional and physical transitions;
- adapting a video generator into a world model can redistribute capabilities instead of improving every axis uniformly.
That is exactly why a single overall number is insufficient. The benchmark is most valuable when readers inspect the axis-level profile and the underlying evidence.
Builders can already use Seedance 2.0 on Atoms for product videos, landing-page visuals, and multi-shot creative work. But the larger lesson from HarnessEval-W is that output quality should be judged against the intended action and persistent product state—not merely whether the first frame looks impressive.
What HarnessEval-W Does Not Prove
HarnessEval-W is ambitious, but its boundaries matter.
It evaluates world models, not every kind of AI agent
The released benchmark centers on interactive visual worlds. The architecture may inspire coding-agent or business-agent evaluation, but the published scores should not be transferred to those domains.
An evidence tree is not automatically ground truth
Sub-agents can still misread a frame, use a weak diagnostic tool, or produce a plausible explanation for a wrong judgment. The parent agent can validate evidence, but its validation is also model-mediated.
Agentic evaluation costs more
Planning, routing, specialized tool calls, sub-agent execution, and validation add latency and inference cost compared with one fixed metric. The right comparison is not “more intelligence is free,” but whether the extra cost buys enough diagnostic value.
A living benchmark creates governance questions
An extensible skill library is powerful, but changes to skills, tools, cases, and evaluator backends can change scores. Versioning, frozen snapshots, contamination control, and reproducibility become part of benchmark governance.
The project currently publishes code under Apache 2.0 and the paper under CC BY-NC-SA 4.0. The repository also lists hosted submission and broader dataset releases as future work.
Why This Matters Beyond World Models
HarnessEval-W arrives at the same moment that agent harnesses are becoming a first-class subject in coding and production AI.
A modern agent's output depends on more than model weights:
- context selection;
- memory and compaction;
- available tools;
- execution environment;
- permission scope;
- retry and recovery logic;
- verification steps.
Atoms uses this systems view when coordinating product planning, engineering, content, review, and deployment. A model can generate a convincing answer while the final product still fails. Reliable evaluation must inspect the completed workflow: did the app build, did its tests pass, did the interaction behave correctly, did the deployment succeed, and did the result satisfy the original brief?
This is the practical implication of the HarnessEval thesis:
When AI becomes a workflow, quality can no longer be measured as if it were a single response.
From Static Benchmark to Executable Evaluation System
The original article closes with a strong organizing idea: the industry is moving from Metric to Harness.
That does not make metrics obsolete. Metrics become tools inside a larger evaluation workflow. The harness decides when a metric applies, what evidence is missing, which additional probe to run, and whether the final judgment is adequately supported.
The benchmark of the future may therefore look less like a spreadsheet and more like a test engineer:
- understand the case;
- decide what must be verified;
- assemble relevant skills and tools;
- collect evidence;
- challenge weak conclusions;
- return both a score and an audit trail;
- extend itself when a new failure mode appears.
HarnessEval-W is an early, concrete implementation of that idea. Its most important claim is not that one agent should judge everything. It is that trustworthy evaluation is itself a complex workflow—and complex workflows need a harness.
Frequently Asked Questions
What is HarnessEval-W?
HarnessEval-W is an open-source, agentified benchmark for interactive visual world models. It uses case-specific skill routing, specialized sub-agents, diagnostic tools, evidence validation, and auditable reasoning traces.
What is the difference between an agent harness and an evaluation harness?
An agent harness organizes how a model completes work. An evaluation harness organizes how that work is tested, evidenced, scored, and reported.
How many models and cases are included?
The first release evaluates 18 representative world models across 330 cases and reports 5,940 scored rollouts.
What is an evidence tree?
An evidence tree records the evaluation plan, selected skills, measurable sub-questions, tool outputs, diagnostics, and logical chain supporting a score.
Is HarnessEval-W only a leaderboard?
No. It includes an executable evaluation pipeline, benchmark cases, skill plans, metric backends, per-case traces, verification commands, and a public leaderboard.
Is HarnessEval-W independently validated?
The paper includes a human preference study and controlled comparisons with WBench protocols. The reported numbers come from the authors and had not been independently reproduced at this article's research cutoff.
Can the idea be applied to coding agents?
Conceptually, yes: coding-agent evaluation also needs task-specific checks, execution evidence, test results, tool traces, and failure attribution. However, HarnessEval-W's released benchmark and scores are specifically for interactive visual world models.
Sources
Research cutoff: August 18, 2026.
