# ATLAS : Adaptive Transfer Scaling Laws For Multilingual Pretraining, Finetuning, and Decoding Source: https://www.youtube.com/watch?v=08qzHXR3pv4 Recap page: https://rapidrecap.app/video/08qzHXR3pv4 Generated: 2026-02-03T13:03:25.056+00:00 --- ## Quick Overview The ATLAS (Adaptive Transfer Scaling Laws) paper demonstrates that scaling multilingual models by increasing the number of languages trained on (up to 400 languages) significantly improves performance in low-resource languages, especially when compared to simply scaling model size or data size, effectively solving the "curse of multilinguality" by allowing for more efficient resource allocation. **Key Points:** - ATLAS successfully scaled multilingual models by increasing the number of languages trained on, achieving better transfer performance for low-resource languages. - The study found that training on 400 languages yielded an R-squared score of 0.98 on held-out tests, significantly outperforming models trained only on English (R-squared of -0.11). - The optimal scaling strategy involves increasing the number of languages (N) raised to the power of 0.4, rather than just increasing model size or data size. - The paper highlights that data for family/script-similar languages (like Spanish and Portuguese, or Swedish and Norwegian) yields significant positive transfer, while dissimilar languages like Hindi and Swahili show negative interference. - A key finding is that the cost of adding more languages is lower than the cost of scaling model size by the same factor; scaling model size by 1.4x requires 2.74 times the data, while adding four times the languages only requires a 1.4x increase in total tokens. - The research provides a framework for engineers to optimize resource allocation, suggesting that for low-resource tasks, focusing on language diversity is more efficient than brute-force scaling of data or model size. - The ISO loss frontier concept is introduced to calculate the exact price of overcoming the multilinguality curse for specific performance targets. ![Screenshot at 00:56: The slide title 'ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding' sets the context for the paper being analyzed, which focuses on optimizing scaling laws for multilingual LLMs.](https://ss.rapidrecap.app/screens/08qzHXR3pv4/00-00-56.jpg) **Context:** The video discusses research presented in the paper "ATLAS: Adaptive Transfer Scaling Laws For Multilingual Pretraining, Finetuning, and Decoding." This research investigates how to most effectively scale large language models (LLMs) when training them across many different languages to improve performance, particularly for languages with limited available data, which is a critical challenge known as the 'curse of multilinguality.' The presenters analyze the trade-offs between increasing model size, increasing the amount of training data, and increasing the number of languages included in the training corpus. ## Detailed Analysis The discussion centers on the ATLAS paper, which explores how to scale multilingual AI models effectively, addressing the 'curse of multilinguality.' The main outcome is that increasing the number of languages trained on (up to 400) is a highly efficient scaling law, significantly boosting performance for low-resource languages compared to simply increasing model size or data volume. The authors found that training on 400 languages resulted in an R-squared score of 0.98, whereas training only on English resulted in a negative R-squared score of -0.11, indicating that English-centric data alone fails to generalize broadly. The paper proposes that the optimal scaling factor for language count is $N^{0.4}$. Furthermore, the research shows that positive transfer is strong between related languages (like Romance languages sharing the Latin script) but negative interference occurs between very dissimilar languages. The key takeaway for engineers is that adding language diversity is a more efficient investment than simply increasing model size or data volume, as demonstrated by the superior performance of the 400-language model over a larger, English-centric model. ### Scaling Strategy Comparison - Increasing language count (N) scales better ($N^{0.4}$) than scaling model size or data size - For low-resource languages, diversity beats brute force data dumping - English-only models perform poorly (-0.11 R^2) compared to multilingual models (0.98 R^2). ### Language Transfer Dynamics - Family/script-similar languages (e.g., Spanish/Portuguese) show strong positive transfer - Dissimilar languages (e.g., English/Hindi) show negative interference, creating a 'curse' that must be managed. ### Resource Allocation Trade-off - Training on 400 languages required only 1.4x the data/compute compared to models trained on fewer languages, but yielded massive performance gains, proving the efficiency of language diversity. ### Practical Implications - Engineers should prioritize training on diverse languages over simply increasing model size or data size, especially for small labs lacking massive resources, to avoid negative interference and achieve better generalized performance. ![Screenshot at 00:00: The initial screen featuring the podcast/audio branding 'ReallyEasy AI' and a call to action to 'Become A Member Today!' overlaid on a sound wave graphic.](https://ss.rapidrecap.app/screens/08qzHXR3pv4/00-00-00.jpg) ![Screenshot at 00:56: The title slide of the presentation clearly stating the paper's name: 'ATLAS : Adaptive Transfer Scaling Laws For Multilingual Pretraining, Finetuning, and Decoding'.](https://ss.rapidrecap.app/screens/08qzHXR3pv4/00-00-56.jpg) ![Screenshot at 02:40: A visual representation of the negative R-squared score for the English-only baseline model, indicating poor predictive power.](https://ss.rapidrecap.app/screens/08qzHXR3pv4/00-02-40.jpg) ![Screenshot at 04:08: The specific R-squared value of 0.98 achieved by the ATLAS model on held-out tests, highlighting its strong performance.](https://ss.rapidrecap.app/screens/08qzHXR3pv4/00-04-08.jpg) ![Screenshot at 09:58: A slide or graphic illustrating the comparison between the performance of a model trained on a few high-resource languages versus one trained on many low-resource languages.](https://ss.rapidrecap.app/screens/08qzHXR3pv4/00-09-58.jpg)