Falcon-H1-Tiny: A series of extremely small, yet powerful LMs redefining capabilities at small scale

Quick Overview

The Falcon H1 Tiny model, with 90 million parameters, achieves significantly better performance than larger models like the 1.1 billion parameter LLaMA-7B on specific tasks by using a novel training strategy combining instruction tuning with Reinforcement Learning from Human Feedback (RLHF) and incorporating Chain-of-Thought data, demonstrating that smaller models can outperform larger counterparts when trained optimally for specific reasoning tasks.

Key Points: The Falcon H1 Tiny model has only 90 million parameters, making it significantly smaller than competitors like LLaMA-7B (1.1 billion parameters). This tiny model achieved 24% better accuracy and relevance on the GSM8K math reasoning benchmark compared to the LLaMA-7B model. The superior performance stems from a novel training approach that combines instruction tuning with DPO (Direct Preference Optimization) and incorporates Chain-of-Thought data. The training process involved feeding the model a highly curated dataset of instructions, Q&A pairs, and reasoning traces, including data from the much larger LLaMA-7B model's fine-tuning set. The model defaults to the simplest pattern (like 'The cat is the cat') when faced with conflicting grammar or syntax, suggesting an over-reliance on memorization in that specific area. The researchers successfully used the tiny model to solve math problems locally on a high-end phone or laptop, demonstrating low latency and minimal battery drain compared to cloud solutions. The paper suggests this approach signals the end of the 'one-size-fits-all' era, favoring specialized, efficient, and smaller models for specific tasks.

Context: The discussion centers around the Falcon H1 Tiny model, an extremely small Large Language Model (LLM) introduced by the Falcon team, which challenges the prevailing trend of scaling up model size for better performance. The model, containing only 90 million parameters, is compared against much larger models, specifically LLaMA-7B, to demonstrate that strategic training methods can yield superior results even with drastically reduced scale.