Kelix Technical Report: Closing the Understanding Gap of Discrete Tokens in Unified MM Models

Quick Overview

The Kelix architecture effectively closes the understanding gap between discrete tokens (like text) and continuous signals (like images) by treating visual data as a stream of continuous signals, which is then processed by a Vision Transformer (ViT) and encoded using product quantization, achieving superior OCR performance (86.7 score) compared to models treating images as discrete tokens.

Key Points: Kelix addresses the understanding gap in multimodal models by treating images as continuous signals rather than discrete tokens. The architecture uses a Vision Transformer (ViT) to scan the image and convert it into dense, continuous numerical values. Product quantization is used to compress this continuous data into discrete tokens, allowing the LLM to process it efficiently. The resulting model scored 86.7 on the OCR benchmark, significantly outperforming discrete models that scored 73.4. The modular system separates the tokenization (ViT + Quantizer) from the LLM brain, allowing for easy upgrades, such as swapping in a better tokenzer next year. The model successfully handles complex tasks like counting items (three red roses and one daisy) and describing images with fine detail, unlike simpler models. The overall goal is to make images 'speak' the same language as text, achieving a unified understanding across modalities.

Context: This video discusses a technical report from Quaxu Technology introducing a new multimodal AI model architecture called Kelix, designed to bridge the conceptual gap between discrete tokens, commonly used for text, and continuous signals, which characterize visual data. The conventional approach of treating images as discrete tokens (like words) causes information loss, which Kelix aims to resolve through a hybrid architecture involving a Vision Transformer (ViT) and product quantization.

Detailed Analysis

The Kelix technical report introduces a novel architecture to close the understanding gap that arises when multimodal models treat images as discrete tokens, similar to words. Traditional models like GPT-4o treat images as sequences of tokens, leading to information loss, especially with fine details. Kelix adopts a hybrid architecture where the image is first processed by a Vision Transformer (ViT) to convert it into dense, continuous numerical values, treating the image like a continuous signal (analogous to a vinyl record groove). This continuous representation is then compressed using product quantization into discrete tokens, which the LLM can process. This approach results in a massive information fidelity improvement; Kelix scores 86.7 on an OCR benchmark, significantly beating the previous best discrete model (X-Comni) score of 73.4. The modular design separates the tokenization (ViT and quantizer) from the LLM's 'brain,' allowing for future upgrades to components without retraining the entire system. The model successfully performs complex tasks like counting objects (e.g., three red roses and one daisy) and retaining fine detail, which older discrete models often fail at. The fundamental goal is to enable the model to generate images and text using the same underlying language representation, effectively making images 'speak' the same language as text.