# Training Language Models to Explain Their Own Computations

Source: https://www.youtube.com/watch?v=9U1JI0FhOzA
Recap page: https://rapidrecap.app/video/9U1JI0FhOzA
Generated: 2025-11-17T18:09:11.414+00:00

---
## Quick Overview

The research demonstrates that training Large Language Models (LLMs) with a specific technique called "Privileged Access Hypothesis"—where the model's internal computations are explicitly linked to the final output via a smaller, pretrained "Explainer Model"—significantly improves faithfulness and interpretability, particularly in complex causal reasoning tasks, by forcing the model to align its internal workings with a ground truth explanation.

**Key Points:**
- LLMs often provide plausible but unfaithful explanations for their outputs, leading to a crisis of AI transparency.
- Researchers tested the "Privileged Access Hypothesis" using a smaller, pretrained explainer model to map internal computations to external explanations.
- The explainer model was trained to describe the internal workings of a larger 8-billion parameter Llama model, achieving high accuracy (99% for one test case).
- When the explainer model was forced to match the ground truth explanation (e.g., "Paris is the capital of France"), the larger model's behavior shifted to match the desired causal chain.
- The smaller explainer model, trained on input-output pairs derived from the larger model's internal states, proved significantly more faithful than zero-shot or unfaithful training methods.
- This approach avoids the high cost of running large, expensive models for interpretability, showing that faithful explanations can be cheap and scalable.
- The method successfully re-frames interpretation tasks into a structured, supervised learning problem, improving reliability in high-stakes situations.

![Screenshot at 01:03: The core concept is introduced: focusing on a paper that tries to solve the AI transparency crisis by training models to explain their own computations using a special method.](https://ss.rapidrecap.app/screens/9U1JI0FhOzA/00-01-03.png)

**Context:** The video discusses the ongoing challenge of 'AI transparency' or 'explainability' in Large Language Models (LLMs), noting that current models often generate explanations that sound plausible but do not accurately reflect the actual internal computational steps that led to the answer. This lack of faithfulness creates a significant trust issue, especially for complex reasoning tasks. The researchers propose a novel training methodology based on the 'Privileged Access Hypothesis' to address this by explicitly linking internal model states to verifiable, external explanations.

## Detailed Analysis

The video explains a research finding addressing the crisis of AI transparency where LLMs produce explanations that are plausible but unfaithful to their true internal computations. The researchers tested the "Privileged Access Hypothesis," which suggests that an LLM's internal state (its internal computations) contains sufficient information to derive a correct explanation, even if the model itself fails to generate it faithfully. They used a specialized, smaller "Explainer Model" trained to map the internal activity of a larger Llama 8-billion parameter model to specific ground truth explanations. When the explainer model was trained to align the larger model's internal state vector with a correct external label (e.g., correctly identifying Paris as the capital of France), the larger model's subsequent outputs became more faithful to that ground truth, successfully flipping its answer from an incorrect one (Italy) to the correct one (France) when the hint was applied. This approach is highly efficient because the expensive, large model only needs to be run once to generate the privileged activation vectors; the subsequent training of the small explainer model is cheap and scalable. This method effectively turns interpretation into a supervised learning problem, ensuring that explanations align with the model's actual reasoning path, thereby boosting trust and reliability.

### The AI Transparency Crisis

- Researchers see a crisis where LLMs provide unfaithful explanations for their outputs
- Explanations sound plausible but do not reflect actual internal computations
- This is a major barrier to trust in high-stakes scenarios.

### The Privileged Access Hypothesis

- The core idea is that the model's internal state contains enough information for correct explanations
- This hypothesis implies that if we can access and use this information, we can force faithful explanations.

### Experimental Setup and Models

- Researchers used a large Llama 8-billion parameter model as the subject model
- They trained a smaller explainer model to map the Llama's internal activation vector to a correct external explanation (ground truth).

### Key Experimental Results

- The explainer model, trained specifically to match ground truth, performed significantly better than zero-shot methods
- Forcing the model to generate explanations that matched the ground truth (e.g., Paris is the capital of France) changed the model's output correctly.

### Efficiency and Scalability

- The method allows training the explainer cheaply using only 0.8% of the original training data
- This avoids the massive cost of running the large model repetitively for interpretability tasks, making it scalable.

![Screenshot at 00:01: Visual introduction to the podcast format, displaying the show's graphic overlay.](https://ss.rapidrecap.app/screens/9U1JI0FhOzA/00-00-01.png)
![Screenshot at 00:12: The host introduces the concept of "unfaithful explanations" in LLMs, setting up the core problem.](https://ss.rapidrecap.app/screens/9U1JI0FhOzA/00-00-12.png)
![Screenshot at 01:04: The host introduces the paper that attempts to solve the problem by training models to explain their own computations.](https://ss.rapidrecap.app/screens/9U1JI0FhOzA/00-01-04.png)
![Screenshot at 01:28: A key concept is mentioned: the "Privileged Access Hypothesis" which suggests internal computation holds the answer key.](https://ss.rapidrecap.app/screens/9U1JI0FhOzA/00-01-28.png)
![Screenshot at 02:24: The smaller model is identified as the "Explainer Model," designed to look at raw input and internal data to generate explanations.](https://ss.rapidrecap.app/screens/9U1JI0FhOzA/00-02-24.png)
![Screenshot at 03:33: The first task tested is Feature Descriptions, which is about semantics and identifying encoded concepts like city names.](https://ss.rapidrecap.app/screens/9U1JI0FhOzA/00-03-33.png)
![Screenshot at 04:53: The second task, Causal Reasoning, is introduced, focusing on cause-and-effect relationships rather than just semantics.](https://ss.rapidrecap.app/screens/9U1JI0FhOzA/00-04-53.png)
![Screenshot at 05:45: The speaker confirms that the Privileged Access Hypothesis held up overwhelmingly well during testing.](https://ss.rapidrecap.app/screens/9U1JI0FhOzA/00-05-45.png)
![Screenshot at 07:34: The comparison is made: the specialized training is vastly more efficient than using massive, expensive human-labeled datasets for training.](https://ss.rapidrecap.app/screens/9U1JI0FhOzA/00-07-34.png)