OneStory: Coherent Multi-Shot Video Generation with Adaptive Memory

Quick Overview

The OneStory model achieves coherent multi-shot video generation by using an Adaptive Memory (AM) module that learns to prioritize and select semantically relevant frames from previous shots to maintain character and environment consistency, effectively solving the narrative coherence problem that plagues simple frame-by-frame generation methods.

Key Points: OneStory uses an Adaptive Memory (AM) module to ensure narrative coherence across multiple generated video shots. The AM module learns to prioritize semantically critical frames from prior shots, such as the protagonist's face or key objects, over less relevant ones. The model successfully maintains character identity and environment consistency even when prompts specify changes like clothing or setting transitions. The training strategy involves a 'dual-loss' approach, using both a standard diffusion loss and a novel 'consistency loss' based on the AM module's memory. The resulting videos exhibit better control over character progression and narrative flow compared to models relying only on immediate context. The paper demonstrates success in maintaining consistency across character appearance, environment, and narrative elements over short sequences.

Context: The video discusses the research paper "OneStory: Coherent Multi-Shot Video Generation with Adaptive Memory," which addresses a major challenge in generative AI video: maintaining narrative and visual consistency across sequentially generated clips (shots). Traditional methods often struggle with character identity and environmental details when generating video segments that are supposed to follow one another, leading to disjointed or incoherent final videos.

Detailed Analysis

The video explains the OneStory model, designed to solve the problem of maintaining narrative coherence across multiple generated video shots, which older models struggled with due to inconsistent character identity and environment across cuts. The core innovation is the Adaptive Memory (AM) module. This module acts like a smart editor, learning which visual information from previous shots (like character faces or key objects) is most important for the current frame generation. It prioritizes these 'semantically critical' frames over less relevant ones, ensuring consistency even when the prompt changes minor details like clothing or location. The training process uses a dual-loss system: one for standard image quality (diffusion loss) and another (consistency loss) that directly enforces the memory retention learned by the AM module. This allows the model to maintain character identity and spatial relationships across cuts, avoiding issues like characters suddenly changing appearance or important objects disappearing. The speaker notes that while older models might fail to maintain consistency over short sequences or rely too heavily on immediate context, OneStory succeeds by explicitly managing memory across the narrative arc, leading to more compelling and coherent storytelling.