The Evolution of Reasoning in Small Language Models [Yejin Choi] - 761

Quick Overview

Yejin Choi focuses on democratizing generative AI by enhancing the reasoning capabilities of small language models (SLMs) through high-quality, diverse synthetic data generation and novel algorithmic approaches like spectrum tuning and reinforcement learning integrated into pre-training, aiming to unlock capabilities beyond what scaling alone achieves.

Key Points: Yejin Choi's current research prioritizes making small language models reason better to democratize generative AI access beyond large GPU-owning entities. The homogeneity problem in LLMs, demonstrated in the 'Artificial Hive Mind' paper, shows that post-training skews model output towards stereotypical answers, even for open-ended questions, resulting in 'strikingly similar behavior' across models like Llama, ChatGPT, and DeepSeek R1. A key approach to improving SLMs involves creating 'much more powerful data' on the 'outer skirt of the internet data,' often through expert curation or advanced synthetic data generation, like the 'Prismatic Synthesis' algorithm. Prismatic Synthesis uses a medium-scale teacher model (DeepSeek R1 32B) to generate synthetic math problems and solutions, filtering aggressively using gradient vectors and k-means clustering on a small proxy model to ensure data diversity, outperforming data from a teacher model 20 times larger. Imitation learning in this context is defined simply as supervised fine-tuning, and techniques like distillation from larger, RL-trained models are used to transfer desirable behaviors while avoiding emergent negative traits like code-switching during math problem-solving. Choi advocates for pluralistic alignment, proposing overtone pluralism (presenting all reasonable opinions), distributional pluralism (mimicking human decision distributions), and steerable pluralism (allowing users to steer the model to their value framework within legal scopes). The research introduces 'Reinforcement Learning as Pre-training Objective' (RLP), where the reward is the information gain achieved by predicting the next token when an intermediate thought is included, showing performance gains even when controlling for total compute flops.