Pollux: Autoregressive Video Generation as a World Model

Training an all-purpose robot requires a world model that captures and predicts the complex dynamics of the physical world. Recent advancements in video generation offer a promising approach, but existing models primarily generate entire video clips from text, often with fixed durations. To this end, this project aims to build a video generation model with three key advancements:

• Next-frame prediction – Rather than generating full clips, the model will synthesize videos frame by frame for finer temporal control and better aligned with physical principles.
• Interactive conditioning – The model will integrate additional signals such as text, camera motion, and actions to enable dynamic generation.
• Efficiency – The design will emphasize computational efficiency, ensuring feasibility for downstream applications like simulators or AI-driven creativity.

Building a world model for an all-purpose robot is a long-term research objective. For this internship project, the primary deliverable is a high-impact publication in top-tier conferences or journals. The research focus may include but is not limited to improving the efficiency/performance of generative models, optimizing inference strategies, or introducing novel technical contributions to world/video/image generative models