# The Evolution of Reasoning in Small Language Models [Yejin Choi] - 761 Source: https://www.youtube.com/watch?v=-_x7lhhZK7M Recap page: https://rapidrecap.app/video/-_x7lhhZK7M Generated: 2026-01-29T22:31:12.688+00:00 --- ## Quick Overview Yejin Choi focuses on democratizing generative AI by enhancing the reasoning capabilities of small language models (SLMs) through high-quality, diverse synthetic data generation and novel algorithmic approaches like spectrum tuning and reinforcement learning integrated into pre-training, aiming to unlock capabilities beyond what scaling alone achieves. **Key Points:** - Yejin Choi's current research prioritizes making small language models reason better to democratize generative AI access beyond large GPU-owning entities. - The homogeneity problem in LLMs, demonstrated in the 'Artificial Hive Mind' paper, shows that post-training skews model output towards stereotypical answers, even for open-ended questions, resulting in 'strikingly similar behavior' across models like Llama, ChatGPT, and DeepSeek R1. - A key approach to improving SLMs involves creating 'much more powerful data' on the 'outer skirt of the internet data,' often through expert curation or advanced synthetic data generation, like the 'Prismatic Synthesis' algorithm. - Prismatic Synthesis uses a medium-scale teacher model (DeepSeek R1 32B) to generate synthetic math problems and solutions, filtering aggressively using gradient vectors and k-means clustering on a small proxy model to ensure data diversity, outperforming data from a teacher model 20 times larger. - Imitation learning in this context is defined simply as supervised fine-tuning, and techniques like distillation from larger, RL-trained models are used to transfer desirable behaviors while avoiding emergent negative traits like code-switching during math problem-solving. - Choi advocates for pluralistic alignment, proposing overtone pluralism (presenting all reasonable opinions), distributional pluralism (mimicking human decision distributions), and steerable pluralism (allowing users to steer the model to their value framework within legal scopes). - The research introduces 'Reinforcement Learning as Pre-training Objective' (RLP), where the reward is the information gain achieved by predicting the next token when an intermediate thought is included, showing performance gains even when controlling for total compute flops. **Context:** The interview features Sam Sharrington hosting Yejin Choi, Professor and Senior Fellow at Stanford University's Institute for Human-Centered AI (HAI), discussing her recent research focus shifts from common sense knowledge to improving reasoning in small language models (SLMs). The conversation explores the motivation behind focusing on SLMs—democratization—and delves into deep technical aspects like data curation, synthetic data generation techniques, and the philosophical implications of model homogeneity on human intelligence and the internet. ## Detailed Analysis Yejin Choi champions the democratization of generative AI by concentrating research efforts on enhancing the reasoning abilities of small language models, arguing that significant capabilities can be unlocked without massive scaling investments. She highlights the critical issue of model homogeneity, detailed in the 'Artificial Hive Mind' paper, where post-training causes models to converge on stereotypical answers, even when diversity is expected. To combat this data dependency, her work emphasizes generating high-quality, diverse data, exemplified by the 'Prismatic Synthesis' technique, which generates synthetic math solutions and then filters them based on gradient diversity calculated via a small proxy model, successfully creating data sets superior to those generated by much larger teacher models. Furthermore, she explores algorithmic solutions, including 'spectrum tuning' to retain output variety post-training, and integrating reinforcement learning directly into the pre-training phase (RLP), incentivizing models to generate beneficial intermediate thoughts by rewarding increased information gain for subsequent token prediction. Beyond technical improvements, Choi addresses alignment through the lens of 'pluralistic alignment,' advocating for systems that respect diverse human values rather than striving for impossible neutrality, suggesting overtone, distributional, and steerable pluralism as frameworks for designing AI that serves all humans respectfully. ### Motivation for SLM Research - Democratizing generative AI so academics and smaller entities can utilize powerful models - Moving beyond data-centric learning toward more data-efficient methods of teaching machine intelligence - Investment currently follows proven scaling methods, creating a 'snowball effect' that needs counterbalancing by diverse ideas. ### Model Homogeneity and Its Concerns - Post-training severely skews model outputs, leading to 'intra-model homogeneity' and 'inter-model homogeneity' where models like Llama, ChatGPT, and DeepSeek R1 produce nearly identical responses, even to open-ended prompts - This convergence threatens to make the internet less reflective of diverse human thought, transforming it into an 'artifact of LLMs.' ### Data Curation and Synthesis - High-quality data must go beyond internet scrapes, requiring expert curation or advanced synthetic generation like 'Prismatic Synthesis' - This method uses a 32B model to generate data, then aggressively filters it by analyzing gradient vectors using a 1.5B proxy model via tensorized k-means clustering to prioritize unique data points. ### Reasoning Enhancement Techniques - Reasoning requires extensive post-training infusion of high-quality, diverse data - Imitation learning is functionally supervised fine-tuning, sometimes used post-RL to distill only the correct reasoning trajectories from stronger models, avoiding emergent strange behaviors like code-switching. ### Reinforcement Learning in Pre-training (RLP) - RLP encourages the model to 'think for itself' before predicting the next token by rewarding the information gain achieved by including an intermediate 'thought' in the prediction context - Models trained with RLP perform significantly better on reasoning benchmarks even when total compute flops are controlled, suggesting a critical learning period similar to human language acquisition. ### Pluralistic Alignment - Moving beyond simple debiasing to accommodate diverse human values - Overtone pluralism involves presenting all reasonable, differing opinions on thorny issues - Distributional pluralism requires the AI's output distribution to mimic the human distribution, not just the majority case - Steerable pluralism allows users to ethically steer the model toward their value framework within legal boundaries.