GenRward

Goal-Driven Reward by Video Diffusion Models for Reinforcement Learning

Qi Wang1,2,3*, Mian Wu1*, Yuyang Zhang1,2,3*, Mingqi Yuan2,3,5, Wenyao Zhang1,2,3, Haoxiang You6,
Yunbo Wang1, Xin Jin2,3,4†, Xiaokang Yang1, Wenjun Zeng2,3
1Shanghai Jiao Tong University
2Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative, Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo
3Zhejiang Key Laboratory of Industrial Intelligence and Digital Twin
4Zhongguancun Academy    5Department of Computing, The Hong Kong Polytechnic University    6Department of Mechanical Engineering, Yale University   

*Denotes equal contribution     †Indicates corresponding author


CVPR 2026

arXiv Code BibTex
overview

Overview of our proposed framework. The key idea is to leverage generated goal-conditioned videos for world knowledge transfer, enabling the downstream agent to improve performance on unseen tasks.

Abstract

Reinforcement Learning (RL) has achieved remarkable success in various domains, yet it often relies on carefully designed programmatic reward functions to guide agent behavior. Designing such reward functions can be challenging and may not generalize well across different tasks. To address this limitation, we leverage the rich world knowledge contained in pretrained video diffusion models to provide goal-driven reward signals for RL agents without ad-hoc design of reward. Our key idea is to exploit off-the-shelf video diffusion models pretrained on large-scale video datasets as informative reward functions in terms of video-level and frame-level goals. For video-level rewards, we first finetune a pretrained video diffusion model on domain-specific datasets and then employ its video encoder to evaluate the alignment between the latent representations of agent's trajectories and the generated goal videos. To enable more fine-grained goal-achievement, we derive a frame-level goal by identifying the most relevant frame from the generated video using CLIP, which serves as the goal state. We then employ a learned forward–backward representation that represents the probability of visiting the goal state from a given state–action pair as frame-level reward, promoting more coherent and goal-driven trajectories. Experiments on various Meta-World tasks demonstrate the effectiveness of our approach.

Pipeline

pipeline

Pipeline of GenReward, which computes goal-driven rewards for behavior learning of the agent using generative prior. During online interaction with the environment, at regular intervals, we employ the correlation between the latent representations of the agent's observations and the generated goal videos as video-level rewards. Meanwhile, we learn a forward-backward model to measure the probability of reaching the goal state that is selected using CLIP from a given state-action pair, providing frame-level reward for fine-grained goal-achievement.

Evaluation

Showcases in Meta-World.

TADPole
Diffusion Reward
GenReward

BibTeX

@inproceedings{wang2026goal,
  title={Goal-Driven Reward by Video Diffusion Models for Reinforcement Learning},
  author={Wang, Qi and Wu, Mian and Zhang, Yuyang and Yuan, Mingqi and Zhang, Wenyao and You, Haoxiang and Wang, Yunbo and Jin, Xin and Yang, Xiaokang and Zeng, Wenjun},
   booktitle={CVPR},
  year={2026}
}

Acknowledgements

This website adapted from Nerfies template.