Motion World Models for Robot Control

Stony Brook University

TL;DR: We adapt a pretrained video diffusion into a robot policy by tiling future RGB, pixel motion, and depth of each camera view in one grid.
The model predicts their futures from RGB images and learns actions from shared features.

xArm7 apple pick and place. Top: recorded static and wrist views at 1× speed. Bottom: offline predictions in a 2×3 grid: camera views by row, RGB/motion/depth by column. Predictions follow recorded action timestamps and hold between replans; the conditioning panel shows observed RGB only.
Put banana into basketep 002 · plan 2
Put kiwi into basketep 002 · plan 2
Put apple into basketep 002 · plan 2
Put orange into basketep 002 · plan 2
Figure 1. Motion Dreamer predictions on xArm7. Each clip compares 16 future steps of predicted RGB, motion, and depth with recorded static and wrist views. Arrows show predicted motion directions. Playback is slowed to 8 fps with endpoint holds. Predictions use 16 denoising steps for visualization.
Created: Last update:

Abstract

We introduce Motion World Models (MWMs), a framework for language-conditioned visuomotor policy learning that jointly predicts future RGB frames, dense pixel motion, and relative depth with RGB-only visual conditioning. MWMs adapt a pretrained image-text-to-video diffusion model into a Motion Dreamer. Given the current RGB observation and a language instruction, the Motion Dreamer predicts a video using view tiling to combine camera views and all three modalities in each frame. The predicted motion and geometry make future scene changes explicit, providing richer cues for robot control than RGB alone. Then a transformer Action Expert maps the Motion Dreamer's internal features to robot action chunks in one forward pass.

Our joint training design makes effective use of the pretrained diffusion model, enabling adaptation with limited data and modest fine-tuning. Joint motion and depth prediction also improves the quality of predicted RGB frames. Across multiple robot learning datasets, MWMs achieve higher task success rates than strong baselines, including RGB-only world models. We are releasing the code and checkpoints.

LangToMo[2] and DAWN[1] (CVPR 2026) forecast a single-view motion map with image diffusion; we view them as earlier instantiations of MWMs.

Method

One pretrained video model, Wan2.2-TI2V-5B[3], predicts RGB, motion, and depth. We adapt its diffusion transformer (DiT) with LoRA, keeping the base weights frozen.

Two-stage MWM training: multimodal video adaptation trains the Motion Dreamer LoRA; joint policy training updates the LoRA and Action Expert while the VAE, text encoder, and base DiT stay frozen
Figure 2. Two-stage training. First, train the Motion Dreamer’s LoRA to predict RGB, motion, and depth. Then train the LoRA and Action Expert together on demonstrated actions. The DROID panel shows recorded RGB and derived motion/depth targets. Solid arrows show computation; dotted arrows show gradients; locks mark frozen weights.

At control time, the Action Expert uses one DiT pass, language, and robot state to predict an action sequence. No video is decoded.


Dense Pixel Motion

Motion is encoded in HSV: hue shows direction and brightness shows strength. Hover over the motion image to inspect a pixel.

Reference 0

Dense Pixel Motion

Hover over the motion image

H: direction — °
V: strength — %
Figure 3. “Rotate the red block to the right.” Predicted RGB, motion, and depth alongside recorded static and gripper views over 12 executed actions. Arrows show predicted motion directions. Predictions use 16 denoising steps for visualization.

Results

On CALVIN[4] ABC→D, models train on environments A–C and face 1,000 five-task chains in unseen environment D. MWMs-Plus complete 4.74 tasks on average; MWMs complete 4.63—the best results among methods without extra training data.

Table 1. CALVIN ABC→D. Success rates for 1–5 consecutive tasks (%) and average tasks completed.
Method Extra
data
Tasks completed in a row (%) Avg. len ↑
12345
With extra training data (robot or human video pretraining, inherited robot-pretrained weights)
DAWN [1]✓97.891.681.375.264.14.10
VPP [6]✓96.590.986.682.076.94.33
FOFPred [7]✓98.895.090.484.678.74.48
FLOWER [8]‡✓99.496.792.286.979.64.55
AVA-VLA [9]‡✓99.997.594.189.883.24.65
Xiaomi-Robotics-0 [10]‡✓100.098.696.092.388.14.75
MMaDA-VLA [11]✓99.898.696.393.589.74.78
CALVIN ABC training data only (same data as MWMs)
DAWN [1]×98.191.378.871.260.64.00
Seer-Large [12]×96.391.686.180.374.04.28
VLA-Adapter-Pro [13]‡×98.994.388.582.675.74.40
FLOWER [8]×99.396.090.382.375.54.44
SLIM [14]×99.396.792.387.180.24.56
GeomVLA (RGB-D) [15] ‡×99.996.892.487.181.34.58
MWMs (ours)×99.597.393.689.183.04.63
MWMs-Plus (ours)×99.898.195.792.288.54.74

✓ Methods in the upper group use datasets beyond CALVIN ABC for policy or predictive pretraining, including human videos and inherited robot-pretrained weights; they are listed for reference and are not directly comparable. ‡ Our local tests of official released checkpoints; other results are source-reported. GeomVLA uses RGB-D input and was tested with its authors’ evaluator on the same 1,000 chains (source-reported avg. length 4.62, without chain success rates). Shading marks the best and second best per column among methods trained on CALVIN ABC data only. Our numbers are the best single checkpoints; over seeds, final checkpoints average 4.51 ± 0.07 (MWMs) and 4.69 ± 0.06 (MWMs-Plus).

Joint prediction improves RGB forecasting

Predicting motion and depth alongside RGB improves RGB forecasts on CALVIN and DROID[5], with lower DreamSim scores at every denoising-step count.

Motion and depth make scene changes explicit. The resulting features also improve robot control over RGB-only prediction.

CALVIN

CALVIN: RGB forecasting quality DreamSim, lower is better, versus denoising steps on a base-2 logarithmic scale. Joint RGB, motion, and depth prediction has the lowest score at every step count. Exact values are in the table below. DreamSim ↓ 0.08 0.14 0.20 1 2 4 8 16 32 RGB only, 1 denoising steps: 0.203864 RGB only, 2 denoising steps: 0.127722 RGB only, 4 denoising steps: 0.095422 RGB only, 8 denoising steps: 0.089870 RGB only, 16 denoising steps: 0.087808 RGB only, 32 denoising steps: 0.092664 RGB + motion, 1 denoising steps: 0.180511 RGB + motion, 2 denoising steps: 0.099587 RGB + motion, 4 denoising steps: 0.080988 RGB + motion, 8 denoising steps: 0.078146 RGB + motion, 16 denoising steps: 0.078299 RGB + motion, 32 denoising steps: 0.080783 RGB + motion + depth, 1 denoising steps: 0.168208 RGB + motion + depth, 2 denoising steps: 0.090309 RGB + motion + depth, 4 denoising steps: 0.076398 RGB + motion + depth, 8 denoising steps: 0.075143 RGB + motion + depth, 16 denoising steps: 0.075243 RGB + motion + depth, 32 denoising steps: 0.077274 Denoising steps

DROID

DROID: RGB forecasting quality DreamSim, lower is better, versus denoising steps on a base-2 logarithmic scale. Joint RGB, motion, and depth prediction has the lowest score at every step count. Exact values are in the table below. DreamSim ↓ 0.12 0.20 0.28 1 2 4 8 16 32 RGB only, 1 denoising steps: 0.271477 RGB only, 2 denoising steps: 0.201197 RGB only, 4 denoising steps: 0.177373 RGB only, 8 denoising steps: 0.166157 RGB only, 16 denoising steps: 0.151933 RGB only, 32 denoising steps: 0.154769 RGB + motion + depth, 1 denoising steps: 0.199537 RGB + motion + depth, 2 denoising steps: 0.152960 RGB + motion + depth, 4 denoising steps: 0.130236 RGB + motion + depth, 8 denoising steps: 0.120973 RGB + motion + depth, 16 denoising steps: 0.122703 RGB + motion + depth, 32 denoising steps: 0.125572 Denoising steps
  • RGB only
  • RGB + motion (CALVIN)
  • RGB + motion + depth
View exact values
RGB DreamSim by denoising steps · lower is better
Dataset / prediction12481632
CALVIN · RGB only0.2038640.1277220.0954220.0898700.0878080.092664
CALVIN · RGB + motion0.1805110.0995870.0809880.0781460.0782990.080783
CALVIN · RGB + motion + depth0.1682080.0903090.0763980.0751430.0752430.077274
DROID · RGB only0.2714770.2011970.1773730.1661570.1519330.154769
DROID · RGB + motion + depth0.1995370.1529600.1302360.1209730.1227030.125572
Motion and depth prediction, PSNR (dB) ↑
DatasetMotion and depth come fromMotionDepth
CALVINRGB-only predictions (extracted)18.888.92
Direct prediction, RGB + motion20.03–
Direct prediction, RGB + motion + depth19.9620.58
DROIDRGB-only predictions (extracted)17.2618.06
Direct prediction, RGB + motion + depth17.4119.92

Ablations on CALVIN ABC→D

Each policy trains for 20,000 steps with three seeds. Scores are average tasks completed (mean ± standard deviation) at the lowest-validation-loss checkpoint (“min val.”) and the final checkpoint (“last”).

Table 2. What should the Motion Dreamer predict?
Visual featuresImage gridAction Expert onlyAction Expert + LoRA
min val.lastmin val.last
DINOv3-B/16 only–2.641 ± 0.1772.576 ± 0.174––
Vanilla Wan (no predictive training)1×21.801 ± 0.1401.801 ± 0.1083.547 ± 0.1063.741 ± 0.196
World model, RGB1×23.060 ± 0.3833.465 ± 0.0304.353 ± 0.1244.385 ± 0.063
Motion only (DAWN-style)1×23.411 ± 0.0263.431 ± 0.0134.077 ± 0.2204.119 ± 0.053
MWMs, RGB + depth2×23.310 ± 0.1103.389 ± 0.0334.414 ± 0.0644.457 ± 0.090
MWMs, RGB + motion2×23.391 ± 0.1903.485 ± 0.1584.271 ± 0.3504.505 ± 0.034
MWMs, RGB + motion + depth2×33.517 ± 0.1063.513 ± 0.0774.528 ± 0.1134.512 ± 0.067
MWMs-Plus2×3––4.692 ± 0.0594.658 ± 0.044

Joint RGB, motion, and depth prediction gives the best Motion Dreamer features. Continuing LoRA training during policy learning adds about one task per chain. MWMs-Plus checkpoints are at steps 16,000 and 20,000.

CALVIN · execution horizon

  • MWMs-Plus
  • MWMs (Linear)
Average tasks completed versus executed actions CALVIN evaluation over 1,000 chains at two fixed checkpoints. Both methods peak at 12 executed actions: MWMs-Plus scores 4.743 and MWMs (Linear) scores 4.598. The shaded band marks 12 actions. Exact values are in the table below. Average tasks completed ↑ 4.4 4.5 4.6 4.7 4.8 6 8 10 12 14 16 17 MWMs-Plus, 6 actions: 4.429 MWMs-Plus, 7 actions: 4.543 MWMs-Plus, 8 actions: 4.657 MWMs-Plus, 9 actions: 4.670 MWMs-Plus, 10 actions: 4.699 MWMs-Plus, 11 actions: 4.698 MWMs-Plus, 12 actions: 4.743 MWMs-Plus, 13 actions: 4.688 MWMs-Plus, 14 actions: 4.699 MWMs-Plus, 15 actions: 4.737 MWMs-Plus, 16 actions: 4.703 MWMs-Plus, 17 actions: 4.680 MWMs (Linear), 9 actions: 4.508 MWMs (Linear), 10 actions: 4.515 MWMs (Linear), 11 actions: 4.499 MWMs (Linear), 12 actions: 4.598 MWMs (Linear), 13 actions: 4.572 MWMs (Linear), 14 actions: 4.543 MWMs (Linear), 15 actions: 4.572 MWMs (Linear), 16 actions: 4.535 MWMs (Linear), 17 actions: 4.514 4.743 4.737 4.598 Executed actions before replanning
View exact values
Average tasks completed · 1,000 CALVIN chains, fixed checkpoints
ActionsMWMs-PlusMWMs (Linear)
64.429–
74.543–
84.657–
94.6704.508
104.6994.515
114.6984.499
124.7434.598
134.6884.572
144.6994.543
154.7374.572
164.7034.535
174.6804.514

Predict 17 actions, execute 12

Table 3. Action decoder
Decodermin val.last
L1 regression3.705 ± 0.3463.991 ± 0.156
Action diffusion4.199 ± 0.0874.201 ± 0.070
Discrete bins (ours)4.528 ± 0.1134.512 ± 0.067

Predicting 256 bins per action dimension outperforms L1 regression and action diffusion. Executing 12 of the 17 predicted actions before replanning works best; MWMs-Plus remain strong across 8–17 actions.

Table 4. Inference cost on one RTX 5090
MethodExecuted actionsms / plan ↓ms / action ↓Peak GPU memory (GiB)
Text features cached per instruction
MWMs-Plus1297.08.110.85
VPP[6]1096.410.55.56
Text encoded at every plan
MWMs-Plus12136.511.421.43
VPP[6]10104.211.25.56
FLOWER[8]10156.416.83.76
AVA-VLA[9]8147.218.514.85
GeomVLA[15]10310.631.15.68

One DiT pass per plan, with no video decoding. Timings use synthetic CALVIN-shaped inputs after warm-up.

Demo Episodes

CALVIN rollouts: each clip compares predicted RGB, motion, and depth with recorded views over 12 actions from one plan. Arrows show predicted motion directions.

Lift red block from sliderseq 195 · replan 3
Lift red block from tableseq 265 · replan 3
Move slider rightseq 697 · replan 1
Rotate red block rightseq 265 · replan 2

Videos use 16 denoising steps for visualization. Robot control uses one DiT pass and decodes no video.

References

  1. Nguyen, E. R., Zhang, Y., Ranasinghe, K., Li, X., & Ryoo, M. S. (2025). Pixel motion diffusion is what we need for robot control. CVPR’26 [Project]
  2. Ranasinghe, K., Li, X., Nguyen, E. R., Mata, C., Park, J., & Ryoo, M. S. (2025). Pixel motion as universal representation for robot control. arXiv'25. [Paper]
  3. Wan Team (2025). Wan2.2-TI2V-5B. Hugging Face model repository. [Model]
  4. Mees, O., Hermann, L., Rosete-Beas, E., & Burgard, W. (2022). CALVIN: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. RA-L’22 [Project]
  5. Khazatsky, A., Pertsch, K., Nair, S., et al. (2024). DROID: A large-scale in-the-wild robot manipulation dataset. RSS’24 [Project]
  6. Hu, Y., Guo, Y., Wang, P., et al. (2025). Video prediction policy: A generalist robot policy with predictive visual representations. ICML’25 [Paper]
  7. Ranasinghe, K., Zhou, H., Fang, Y., et al. (2026). Future optical flow prediction improves robot control & video generation. CVPR’26 Findings
  8. Reuss, M., Zhou, H., Rühle, M., et al. (2025). FLOWER: Democratizing generalist robot policies with efficient vision-language-action flow policies. arXiv’25 [arXiv]
  9. Xiao, L., Li, J., Gao, J., et al. (2025). AVA-VLA: Improving vision-language-action models with active visual attention. arXiv’25 [arXiv]
  10. Cai, R., Guo, J., He, X., et al. (2026). Xiaomi-Robotics-0: An open-sourced vision-language-action model with real-time execution. arXiv’26 [arXiv]
  11. Liu, Y., Ding, P., Jiang, T., et al. (2026). MMaDA-VLA: Large diffusion vision-language-action model with unified multi-modal instruction and generation. ACM MM’26
  12. Tian, Y., Yang, S., Zeng, J., et al. (2025). Predictive inverse dynamics models are scalable learners for robotic manipulation. ICLR’25 [Paper]
  13. Wang, Y., Ding, P., Li, L., et al. (2025). VLA-Adapter: An effective paradigm for tiny-scale vision-language-action model. arXiv’25 [arXiv]
  14. Wang, J., Tang, Z., Zhang, G., et al. (2026). SLIM-0.5B: Learning action-grounded predictive latents for robot manipulation. arXiv’26 [arXiv]
  15. Xiong, Z., Gkanatsios, N., Reuss, M., & Fragkiadaki, K. (2026). GeomVLA: Unifying scene, motion, and action in 3D. CoRL’26
  16. Zhao, et al. (2026). RynnWorld-4D: 4D embodied world models for robotic manipulation. arXiv’26 [arXiv]