OpenAI: Weight-Sparse Transformers Have Interpretable Circuits

Quick Overview

OpenAI researchers demonstrated that weight-sparse transformers, trained using a novel two-step procedure involving initial pre-training followed by targeted pruning, achieve interpretability by creating circuits that map specific inputs to specific outputs, successfully identifying the location of activation weights corresponding to concepts like bracket counting or specific quotes.

Key Points: Weight-sparse transformers achieve interpretability by creating circuits that map inputs to specific outputs, demonstrated using a novel two-step training process. The two-step process involves training on a small, weight-sparse subset of the network, followed by training on a full, dense model, which proves superior to the reverse order. The researchers successfully identified circuits responsible for identifying bracket counting and locating specific quotes within the model's internal state. The activation magnitude in the detector channel directly correlates with the depth of nesting for bracket counting, providing a clear, interpretable measure. The sparse models demonstrated significantly better interpretability (higher clarity) and comparable performance to dense models on certain tasks, despite having far fewer active parameters. The core challenge solved is mitigating the performance/interpretability trade-off, with sparse models allowing for a clearer view of the underlying mechanism. The process involves using a 'circuit' to map an input to a specific output, which is then validated by seeing if manipulating that circuit affects the output predictably.

Context: This video discusses recent advancements by OpenAI in making large language models (LLMs), specifically weight-sparse transformers, more interpretable. The core problem being addressed is the "black box" nature of these models, where understanding how they arrive at an answer is difficult. The researchers propose a specific training and pruning methodology to create sparse models that retain high performance while being easier to reverse-engineer into understandable circuits.