Training Language Models to Explain Their Own Computations

Quick Overview

The research demonstrates that training Large Language Models (LLMs) with a specific technique called "Privileged Access Hypothesis"—where the model's internal computations are explicitly linked to the final output via a smaller, pretrained "Explainer Model"—significantly improves faithfulness and interpretability, particularly in complex causal reasoning tasks, by forcing the model to align its internal workings with a ground truth explanation.

Key Points: LLMs often provide plausible but unfaithful explanations for their outputs, leading to a crisis of AI transparency. Researchers tested the "Privileged Access Hypothesis" using a smaller, pretrained explainer model to map internal computations to external explanations. The explainer model was trained to describe the internal workings of a larger 8-billion parameter Llama model, achieving high accuracy (99% for one test case). When the explainer model was forced to match the ground truth explanation (e.g., "Paris is the capital of France"), the larger model's behavior shifted to match the desired causal chain. The smaller explainer model, trained on input-output pairs derived from the larger model's internal states, proved significantly more faithful than zero-shot or unfaithful training methods. This approach avoids the high cost of running large, expensive models for interpretability, showing that faithful explanations can be cheap and scalable. The method successfully re-frames interpretation tasks into a structured, supervised learning problem, improving reliability in high-stakes situations.

Context: The video discusses the ongoing challenge of 'AI transparency' or 'explainability' in Large Language Models (LLMs), noting that current models often generate explanations that sound plausible but do not accurately reflect the actual internal computational steps that led to the answer. This lack of faithfulness creates a significant trust issue, especially for complex reasoning tasks. The researchers propose a novel training methodology based on the 'Privileged Access Hypothesis' to address this by explicitly linking internal model states to verifiable, external explanations.