# Code2Worlds: Empowering Coding LLMs for 4D World Generation

Source: https://www.youtube.com/watch?v=aq8OmJ7fkpc
Recap page: https://rapidrecap.app/video/aq8OmJ7fkpc
Generated: 2026-02-18T15:03:50.914+00:00

---
## Quick Overview

The research paper "Code2Worlds" introduces a novel approach to empowering Large Language Models (LLMs) for generating 4D simulations that obey physical laws by using a dual-stream architecture that separates object generation from scene generation, thereby solving the fundamental flaw of previous models failing to enforce physical constraints like gravity or object stability.

**Key Points:**
- Code2Worlds separates object generation (Stream 1) from scene generation (Stream 2) to allow for physical simulation fidelity.
- The model successfully generates a 4D simulation of leaves falling, correctly modeling gravity and fluid dynamics, unlike previous methods.
- The paper highlights that previous pixel-based diffusion models often generated physically implausible results, such as objects floating or behaving rigidly (e.g., leaves acting like bricks).
- The Parameter Resolver acts as a common sense filter, ensuring that generated parameters (like wind speed or leaf angle) are physically valid before execution.
- The method achieves a low physical failure rate of 61.4% on the 4D benchmark, significantly outperforming previous models that were stuck in the 35-40% range.
- The technique allows the AI to generate executable Python code that drives a physics engine (like MuJoCo/InfiniGen) to create the simulation, rather than just generating pixels.
- The core improvement is moving away from black-box neural nets that only generate pretty pictures toward a system that enforces physical reality, treating generation more like an engineering task.

![Screenshot at 00:12: The introduction of the Code2Worlds architecture, which splits generation into object and scene streams to enforce physical laws, is detailed here, contrasting with previous models that only focused on pixel aesthetics.](https://ss.rapidrecap.app/screens/aq8OmJ7fkpc/00-00-12.jpg)

**Context:** The video discusses a research paper from the School of Computer Science at Peking University titled "Code2Worlds," which addresses a major limitation in generative AI models: their inability to create video content that adheres to real-world physics. For years, models like Sora excelled at generating visually stunning, cinematic videos, but often failed basic physical checks, such as modeling gravity correctly or ensuring objects behaved realistically. Code2Worlds proposes a new architecture to bridge this gap between visual realism and physical correctness.

## Detailed Analysis

The presentation introduces the Code2Worlds paper, which proposes a method to make AI-generated videos adhere to real-world physics, solving the major structural flaw in existing diffusion models like Sora. The core innovation is a dual-stream generation process: Stream 1 focuses on generating the objects (e.g., a cup, a leaf), and Stream 2 generates the scene context (e.g., the stage, the background). This separation allows for the introduction of a Parameter Resolver. This resolver acts as a crucial logic check, ensuring that parameters derived from the prompt (like wind speed or the material properties of an object) are physically plausible before the simulation runs. For example, when prompted to simulate falling leaves, the model correctly applies fluid dynamics, whereas older models would render leaves as rigid bricks or floating objects. The system outputs executable Python code that runs through a physics engine like MuJoCo or InfiniGen to generate the simulation over time. This approach resulted in a significantly lower physical failure rate (61.4% on the 4D benchmark) compared to previous models stuck in the 35-40% range, demonstrating that the AI is learning to reason about physics rather than just mimicking visual patterns.

### Paper Introduction

- Paper proposes Code2Worlds for empowering LLMs to generate 4D (3D + time) simulations that obey physics
- Solves structural flaw in prior video models that focused only on pretty pixels

### Code2Worlds Architecture

- Uses dual streams—Stream 1 for object generation, Stream 2 for scene generation
- Introduces a Parameter Resolver to enforce physical constraints like gravity and fluid dynamics

### Parameter Resolver Function

- Acts as a common sense filter, checking if prompt parameters (e.g., wind speed, material) are valid before simulation
- Prevents unrealistic outputs like floating objects or rigid leaves

### Simulation Process

- Model writes Python code that drives a physics engine (MuJoCo/InfiniGen) to simulate the scene over time
- Allows for iterative debugging and refinement based on physical outcomes

### Results and Impact

- Achieves a physical failure rate of 61.4% on the 4D benchmark, a significant improvement over previous models (35-40%)
- Moves AI generation from creating mere visuals to producing physically accurate, engineering-like results

![Screenshot at 00:00: The introductory screen featuring the podcast graphic and the call to action "Become A Member Today!"](https://ss.rapidrecap.app/screens/aq8OmJ7fkpc/00-00-00.jpg)
![Screenshot at 00:09: The speaker introduces the paper, "Code to Worlds," which aims to empower LLMs for 4D world generation.](https://ss.rapidrecap.app/screens/aq8OmJ7fkpc/00-00-09.jpg)
![Screenshot at 00:43: Visual explanation of the core flaw: the model has no idea what it's looking at and cannot infer real-world physics like gravity.](https://ss.rapidrecap.app/screens/aq8OmJ7fkpc/00-00-43.jpg)
![Screenshot at 01:20: The core concept illustrated: the model writes Python code that drives a physics engine instead of just spitting out an MP4 file.](https://ss.rapidrecap.app/screens/aq8OmJ7fkpc/00-01-20.jpg)
![Screenshot at 06:15: The speaker highlights the key finding: the success moment is when the AI watches its own work and performs self-correction loops.](https://ss.rapidrecap.app/screens/aq8OmJ7fkpc/00-06-15.jpg)