**Kaining Ying**^*, **Hengrui Hu**^*, **Siyu Ren**, Jiamu Li, Fengjiao Chen,
Ziwen Wang, Xuezhi Cao, Xunliang Cai, Henghui Ding ^{[✉️](mailto:[email protected])}
Fudan University    Meituan LongCat Team

TL;DR — WBench evaluates 31 video world models across 5 dimensions and 22 metrics.

📢 News

• [2026/08/16] 🔄 Updated AlayaWorld to its final-v4 submission (76.3, #12 overall).
• [2026/08/16] 🔄 Updated HiDream-O1-World to its 2026-08-14 submission (80.9, #1 overall).
• [2026/08/14] 🆕 Added Alaya-EVOKE 🥇 (Alaya Lab, camera-conditioned) to the leaderboard (now 31 models).
• [2026/07/29] 🆕 Added HiDream-O1-World (camera-conditioned).
• [2026/07/29] 🆕 Added AlayaWorld (Alaya Lab, camera-conditioned) to the leaderboard (now 29 models).
• [2026/07/14] 🆕 Added ABot-World (Amap, action-conditioned) to the leaderboard (now 28 models).
• [2026/07/12] 🆕 Added LingBot-World (fast v2) (camera) to the leaderboard (now 27 models).
• [2026/07/12] 🆕 Added Cosmos3-Super & Cosmos3-Nano (text) to the leaderboard (now 26 models).
• [2026/06/18] 🆕 Added DreamX-World (5B AR) to the leaderboard (now 24 models).
• [2026/06/17] 🆕 Added LingBot-World (fast) to the leaderboard (now 23 models).
• [2026/06/16] 🔌 Open-sourced the HY-World 1.5 integration example.
• [2026/06/16] 🆕 Added 2 camera-controlled world models — Lyra 2.0 & SANA-WM (4-step AR).
• [2026/06/10] 🧭 Added HY-World 1.5 pose exports to WBench-examples.
• [2026/06/01] WBench is now an official benchmark on Hugging Face 🤗 (navi & full tasks)!
• [2026/06/01] 📦 Released WBench-examples: ready-to-eval videos from HY-World 1.5 & Kling 3.0.
• [2026/06/01] 🎮 Added camera- & action-conditioned examples + web automation (Genie3, Happy Oyster).
• [2026/06/01] Added Claude Code skills 🤖 for generation, evaluation & submission.
• [2026/05/29] Paper ranked #2 🏅 on Hugging Face Daily Papers!
• [2026/05/28] Paper now available on arXiv 📄!
• [2026/05/28] Homepage with interactive leaderboard & dataset gallery is live! 🌐
• [2026/05/28] 🚀 Released the full WBench dataset, evaluation code & model weights.

✨ Contributions

• A comprehensive evaluation framework with 289 cases, 1,058 interaction turns, covering 4 interaction types (navigation, subject action, event editing, perspective switching) across diverse scenes and perspectives.
• A unified navigation protocol that bridges text, 6-DoF camera pose, and discrete-action interfaces, enabling fair comparison across model families.
• 22 automatic metrics spanning 5 complementary dimensions, validated against human judgments, ensuring reliable automatic evaluation at scale.
• Systematic diagnosis of 31 models revealing that current world models have not yet unified high-fidelity rendering with reliable controllability, consistency, and physics compliance.

🏆 Leaderboard

31 Models — Navigation Split (5 Dimensions, sorted by average)

#	Model	Average	Quality	Setting	Interaction	Consistency	Physical
1	HiDream-O1-World	80.9 🥇	81.0	82.2	80.0	88.0	73.3 🥇
2	Alaya-EVOKE	80.8 🥈	82.8 🥇	83.8	78.6	86.9	72.1 🥈
3	LingBot-World (fast v2)	79.4 🥉	81.8 🥉	76.8	82.8	86.5	69.1
4	Kling 3.0	79.0	81.4	91.0 🥉	69.4	83.7	69.3
5	LingBot-World (base-camera)	78.5	78.9	72.6	80.1	89.9 🥇	71.2
6	Wan 2.7	78.1	81.5	91.4 🥈	64.4	81.6	71.8 🥉
7	HY-World 1.5 (ar-distill)	78.1	78.1	72.2	86.8 🥇	86.9	66.3
8	HY-Video 1.5	77.9	77.6	85.6	71.4	87.4	67.4
9	LingBot-World (fast)	77.4	79.4	77.9	79.2	84.9	65.7
10	Happy Oyster	76.8	77.3	74.2	84.9 🥉	84.3	63.5
11	Lyra 2.0 (4-step AR)	76.4	77.1	73.2	85.6 🥈	79.3	66.7
12	AlayaWorld	76.3	79.3	69.7	80.0	89.5 🥈	63.1
13	Seedance 1.5	76.2	82.1 🥈	82.9	66.3	81.3	68.4
14	Cosmos3-Super	76.0	76.4	91.6 🥇	58.0	85.0	69.2
15	SANA-WM (4-step AR)	76.0	79.3	76.1	82.2	80.7	61.9
16	DreamX-World (5B AR)	75.0	77.5	80.8	78.6	74.9	63.3
17	ABot-World	74.7	76.8	71.4	84.0	79.5	61.7
18	Cosmos 2.5	74.6	72.9	83.3	63.1	86.5	67.4
19	Cosmos3-Nano	74.2	77.4	87.3	59.1	83.7	63.6
20	LTX 2.3	74.2	77.1	85.2	66.4	77.2	64.9
21	Genie 3	73.9	75.2	72.5	73.4	82.6	65.7
22	InSpatio-World	73.9	71.5	71.4	73.2	88.4 🥉	65.2
23	Fantasy-World	73.8	72.4	71.3	71.9	86.4	66.8
24	YUME 1.5	73.3	77.6	72.4	71.4	80.1	65.2
25	LongCat-Video	73.2	75.4	72.3	62.1	87.1	68.9
26	Infinite-World	72.8	77.0	69.3	75.4	80.0	62.1
27	MatrixGame3	71.3	75.5	63.6	83.6	74.5	59.3
28	Kairos 3.0	70.3	74.0	70.3	64.1	82.6	60.4
29	MatrixGame2	68.7	73.8	67.1	80.3	65.1	57.2
30	HY-GameCraft	68.2	73.0	66.6	66.3	72.6	62.4
31	Astra	63.7	67.1	59.6	66.9	73.3	51.4

11 Text-driven Models — Full Split (5 Dimensions, sorted by average)

#	Model	Average	Quality	Setting	Interaction	Consistency	Physical
1	Kling 3.0	79.4 🥇	80.0 🥉	91.0 🥈	72.8 🥇	83.9	69.2 🥈
2	Wan 2.7	78.3 🥈	81.0 🥈	91.4 🥇	71.7 🥈	75.8	71.6 🥇
3	Seedance 1.5	76.2 🥉	82.0 🥇	82.9	67.9 🥉	79.9	68.2
4	HY-Video 1.5	74.3	76.7	85.6	54.6	87.5 🥇	67.1
5	Cosmos3-Super	72.4	74.7	89.4 🥉	44.9	84.3	68.9 🥉
6	LTX 2.3	70.9	77.0	85.2	49.0	78.1	65.1
7	Cosmos 2.5	70.3	71.7	83.3	43.2	86.3 🥉	67.0
8	LongCat-Video	69.9	77.2	72.3	44.8	86.6 🥈	68.4
9	Cosmos3-Nano	69.3	75.0	86.6	36.5	84.5	63.9
10	YUME 1.5	68.9	77.6	72.4	48.2	80.9	65.4
11	Kairos 3.0	65.7	73.1	70.3	41.4	83.2	60.5

🚀 Quick Start

# Install
git clone --recursive https://github.com/meituan-longcat/WBench.git
cd WBench

# If you already cloned without submodules
git submodule update --init --recursive

# Download data and weights
pip install huggingface_hub
hf download meituan-longcat/WBench --repo-type dataset --local-dir data/ --exclude "splits/*"
hf download meituan-longcat/WBench-weights --local-dir weights/

# Environment 1: wbench-main (all metrics except visual_plausibility)
# 2nd arg = PyTorch's CUDA build — match it to YOUR system (check via `nvcc --version`):
#   cu124 → CUDA 12.x    cu121 → CUDA 12.1    cu118 → CUDA 11.8
# Always pass it explicitly: if omitted, auto-detection falls back to cu118 when nvcc
# isn't on PATH, which makes the MegaSAM CUDA extensions fail to build on CUDA-12 machines.
bash tools/install.sh wbench-main cu124
conda activate wbench-main
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH



# Verify
conda activate wbench-main
python tools/verify_install.py

# Run evaluation (auto multi-GPU)
python main.py --model your_model

See docs/installation.md for detailed setup instructions.

🎮 Evaluate Your Model

Set environment variables for VLM metrics first (we use Doubao-Seed-2.0-lite via Volcengine ARK):

export VLM_API_KEY="<your-ark-api-key>"
# Optional (defaults shown):
# export VLM_API_URL="https://ark.cn-beijing.volces.com/api/v3"
# export VLM_MODEL_NAME="doubao-seed-2-0-lite-260215"

1. Generate multi-turn videos → place in work_dirs/<model>/videos/case_{id}_combined.mp4
2. Run the 3-phase pipeline:

# Full pipeline (precompute → GPU metrics → VLM metrics → report)
python main.py --model my_model --gpus 0,1,2,3,4,5,6,7

# Or run phases independently:
python main.py --model my_model --phase precompute    # SAM2 + DA3 + MegaSAM
python main.py --model my_model --phase gpu           # GPU metrics (per-metric)
python main.py --model my_model --phase vlm           # VLM metrics (API)
python main.py --model my_model --phase report        # Aggregate report

Note: the pipeline above covers 21 of the 22 metrics. visual_plausibility is the exception — it runs in the separate wbench-vp environment (set up in Quick Start):

conda activate wbench-vp
python tools/run_visual_plausibility.py --model my_model  # uses all available GPUs

1. Results: work_dirs/<model>/evaluation/{metric}/case_{id}.json + report.json

# Run specific metrics (by name or dimension)
python main.py --model my_model --phase gpu --metrics hpsv3_quality
python main.py --model my_model --phase gpu --metrics quality         # all 6 video quality
python main.py --model my_model --phase gpu --metrics consistency     # all consistency metrics

# Skip pre-computation if already done
python main.py --model my_model --phase gpu --skip_megasam --skip_sam2 --skip_da3

# Single video evaluation
python main.py --video video.mp4 --case data/cases/case_1.json

Dimensions (--metrics supports these as shorthand): | Dimension | Metrics | |:---|:---| | quality | aesthetic_quality, imaging_quality, temporal_flickering, dynamic_degree, motion_smoothness, hpsv3_quality | | consistency | background_consistency, segment_continuity, perspective_consistency, subject_consistency, geometric_consistency, photometric_consistency, spatial_consistency, gated_spatial_consistency | | interaction | navigation_trajectory, event_edit_adherence, subject_action_adherence, perspective_switch_adherence | | setting | scene_adherence, subject_adherence | | physical | visual_plausibility, causal_fidelity |

🔌 Implement Your Model

WBench supports 3 model types with different control interfaces:

Text-conditioned models

from src.models import get_model

# Available: wan, kling, seedance (or register your own)
model = get_model("wan")

# Generate multi-turn video from a case
result = model.generate_multi_turn(
    case=case_dict,
    output_path="work_dirs/wan/videos/case_1_combined.mp4",
    data_root="data/",
)

Each turn: build prompt from interaction → call I2V API → extract last frame → next turn.

Set API credentials:

export VIDEO_API_URL="https://your-video-api.com"
export VIDEO_API_KEY="your-key"

Camera-conditioned models

The benchmark's navigation actions (W/A/S/D + arrows) are converted to per-turn {move, yaw, pitch} intent and then to a 6-DoF camera trajectory. Subclass CameraConditionedModel and implement one hook — case parsing, action→pose conversion, and video writing are handled for you:

from src.models.camera import CameraConditionedModel

class MyWorldModel(CameraConditionedModel):
    def generate_with_poses(self, image, poses, video_length, **kw):
        # image: first-frame path; poses: {"<latent_idx>": {"extrinsic": 4x4, "K": 3x3}, ...}
        # return: list of `video_length` BGR uint8 frames
        return my_model.infer(image, poses, video_length)

MyWorldModel("mymodel").generate_multi_turn(case_dict,
    "work_dirs/mymodel/videos/case_1_combined.mp4", data_root="data/")

The pose convention (axes, speeds, intrinsics) lives in src/models/camera/poses.py — copy and adapt it to your model; the navigation metric normalises scale, so what matters is matching the per-action intent. Quick look at one case:

python -m src.models.camera.demo --case data/cases/case_1.json   # prints poses + renders a preview

Note: Camera/action models only cover the 158 navigation cases (cases containing at least one W/A/S/D/arrow action). When generating at scale, pass only those cases — e.g. via generate.py --model your_model --cases <navi_list>.

Action-conditioned models

Two flavours, both fed from the same per-turn navigation plan:

Programmatic controllers (e.g. Matrix-Game-3). Subclass ActionConditionedModel and implement generate_with_actions. Each action carries both raw key tokens and an MG3-style {keyboard, mouse} tensor:

from src.models.action import ActionConditionedModel

class MyActionModel(ActionConditionedModel):
    def generate_with_actions(self, image, actions, video_length, **kw):
        # actions: [{"turn", "tokens", "keyboard", "mouse", "duration"}, ...]
        return my_model.infer(image, actions, video_length)

MyActionModel("mymodel").generate_multi_turn(case_dict,
    "work_dirs/mymodel/videos/case_1_combined.mp4", data_root="data/")

python -m src.models.action.demo --case data/cases/case_1.json   # prints actions + renders a preview

Web products (e.g. Project Genie, Happy Oyster) — no weights/API; driven by browser automation + simulated keystrokes. See src/models/action/web/.

🤖 Claude Code Skills

If you use Claude Code, this repo ships skills that drive the full workflow — just ask in natural language and Claude runs the right commands:

Skills live in .claude/skills/ (and src/models/action/web/.claude/skills/) and are auto-discovered when you open the repo in Claude Code.

📋 TODO

• [x] Text-conditioned model generation (Wan, Kling, Seedance)
• [x] Homepage with interactive leaderboard
• [x] Dataset and weights release on HuggingFace
• [x] Camera-conditioned model generation example
• [x] Action-conditioned model generation example
• [x] Hosted submission & evaluation service (submit videos, get scores)
• [x] ArXiv paper release

📝 Citation

If you find our work useful, please consider citing:

@article{ying2026wbenchcomprehensivemultiturnbenchmark,
  title={WBench: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluation},
  author={Ying, Kaining and Hu, Hengrui and Ren, Siyu and Li, Jiamu and Chen, Fengjiao and Wang, Ziwen and Cao, Xuezhi and Cai, Xunliang and Ding, Henghui},
  journal={arXiv preprint arXiv:2605.25874},
  year={2026}
}

🙏 Acknowledgement

This project builds upon the following excellent works:

• WorldScore — World model evaluation framework
• VBench — Video quality metrics
• SAM2 — Segment Anything Model 2 for mask tracking
• Depth-Anything-V3 — Monocular depth estimation
• MegaSAM — Camera pose estimation
• DreamSim — Perceptual similarity metric
• HPSv3 — Human Preference Score
• AMT — Frame interpolation for motion smoothness
• RAFT — Optical flow estimation
• TransNetV2 — Scene boundary detection
• ... and many other excellent open-source projects

📧 Contact

Feel free to open an Issue or Pull Request. You can also reach us directly:

• Kaining Ying: [email protected]
• Siyu Ren: [email protected]
• Henghui Ding: [email protected]

📄 License

Code and data: MIT License. Model weights retain their original licenses.