BitDance: Scaling Autoregressive Generative Models with Binary Tokens

Quick Overview

The BitDance paper introduces an autoregressive generative model, also called BitDance, that scales effectively by tokenizing data into binary tokens, allowing it to predict the next token in a sequence much faster and smaller than traditional large language models, overcoming limitations like the vocabulary bottleneck and high computational cost associated with standard autoregressive models.

Key Points: The BitDance model scales autoregressive generative models by using binary tokens instead of large vocabularies, resulting in faster inference (8.7x speedup on ImageNet benchmarks). BitDance utilizes a novel architecture combining an autoregressive model with a binary diffusion head, treating token prediction as a classification task over two possibilities (+1 or -1). The model avoids the vocabulary bottleneck and high computational cost of traditional AR models by operating on a binary space, where the vocabulary size is effectively 2. Training involved a mixed-resolution training strategy (256x256 and 128x128 images) to maintain model stability. The resulting model achieves a FID score of 1.24 on ImageNet, which is significantly better than similarly sized standard AR models (which performed poorly or failed to converge). BitDance is shown to be capable of generating high-fidelity images and accurately rendering legible text within images, outperforming diffusion models on these tasks. The paper suggests this technique could lead to a universal tokenizer for all modalities by treating token prediction as a binary classification problem.

Context: The video discusses a research paper titled "BitDance: Scaling Autoregressive Generative Models with Binary Tokens," which presents a new approach to image generation models. This method fundamentally changes how sequence data is handled by converting it into binary tokens, aiming to resolve computational bottlenecks and scalability issues present in existing large autoregressive models like those that traditionally use large vocabularies or complex diffusion processes.