ATLAS : Adaptive Transfer Scaling Laws For Multilingual Pretraining, Finetuning, and Decoding

Quick Overview

The ATLAS (Adaptive Transfer Scaling Laws) paper demonstrates that scaling multilingual models by increasing the number of languages trained on (up to 400 languages) significantly improves performance in low-resource languages, especially when compared to simply scaling model size or data size, effectively solving the "curse of multilinguality" by allowing for more efficient resource allocation.

Key Points: ATLAS successfully scaled multilingual models by increasing the number of languages trained on, achieving better transfer performance for low-resource languages. The study found that training on 400 languages yielded an R-squared score of 0.98 on held-out tests, significantly outperforming models trained only on English (R-squared of -0.11). The optimal scaling strategy involves increasing the number of languages (N) raised to the power of 0.4, rather than just increasing model size or data size. The paper highlights that data for family/script-similar languages (like Spanish and Portuguese, or Swedish and Norwegian) yields significant positive transfer, while dissimilar languages like Hindi and Swahili show negative interference. A key finding is that the cost of adding more languages is lower than the cost of scaling model size by the same factor; scaling model size by 1.4x requires 2.74 times the data, while adding four times the languages only requires a 1.4x increase in total tokens. The research provides a framework for engineers to optimize resource allocation, suggesting that for low-resource tasks, focusing on language diversity is more efficient than brute-force scaling of data or model size. The ISO loss frontier concept is introduced to calculate the exact price of overcoming the multilinguality curse for specific performance targets.

Context: The video discusses research presented in the paper "ATLAS: Adaptive Transfer Scaling Laws For Multilingual Pretraining, Finetuning, and Decoding." This research investigates how to most effectively scale large language models (LLMs) when training them across many different languages to improve performance, particularly for languages with limited available data, which is a critical challenge known as the 'curse of multilinguality.' The presenters analyze the trade-offs between increasing model size, increasing the amount of training data, and increasing the number of languages included in the training corpus.