# Do Explanations Generalize Across Large Reasoning Models?

Source: https://www.youtube.com/watch?v=B19-o2tfQI8
Recap page: https://rapidrecap.app/video/B19-o2tfQI8
Generated: 2026-01-21T12:04:07.888+00:00

---
## Quick Overview

The hypothesis that explanations generalize across large reasoning models fails because while Model A (DeepSeek Coder) showed high accuracy and consistency in generating step-by-step explanations, Model B (DPO) exhibited a lower consistency score (66% vs. 95% baseline) when evaluated by experts, indicating that the quality of the explanation itself does not automatically translate to accurate or logical reasoning, especially when the underlying logic is flawed or the models are trained on different datasets.

**Key Points:**
- Model A (DeepSeek Coder) achieved 95% consistency on the baseline, while Model B (DPO) dropped to 66% consistency when evaluated on complex reasoning tasks.
- The researchers found that the explanatory chain of thought (CoT) generated by Model A was persuasive enough to trick human experts into rating it highly, even when the underlying math was flawed.
- The core finding shows that reasoning (like mathematical calculation) and explanation generation are separable skills; high explanation quality does not guarantee high reasoning accuracy.
- Model A's consistently accurate reasoning (e.g., on MedCalc problems) did not transfer when it was forced to use a specific, flawed decoding strategy (sampling instead of greedy).
- The study suggests that relying solely on the consistency of an explanation as a proxy for truth is risky, as models can become highly consistent in propagating errors or biases.
- The experiment involved comparing two models, Model A (DeepSeek Coder) and Model B (DPO), using two metrics: accuracy and consistency of the explanation's logic.
- The researchers propose that future AI systems require explicit training to check each other's work to prevent the reinforcement of errors, rather than relying on the perceived quality of the explanation.

![Screenshot at 00:48: The visual grid and waveform illustrate the quantitative analysis being performed on the models' reasoning outputs, contrasting the 'mystical' nature of the black box with the need for measurable metrics like accuracy and consistency.](https://ss.rapidrecap.app/screens/B19-o2tfQI8/00-00-48.jpg)

**Context:** The video discusses a research paper from Northeastern University and Microsoft Research titled "Do Explanations Generalize Across Large Reasoning Models?". The core concept explores whether the reasoning process (the 'chain of thought') generated by one large language model (LLM) can be reliably used by another model to achieve similar, correct results, or if the quality of the explanation implies true underlying reasoning capability. The experiment involved comparing two models, Model A (DeepSeek Coder) and Model B (DPO), on tasks requiring both mathematical reasoning and logical deduction.

## Detailed Analysis

The video analyzes a research paper investigating whether explanations generated by one large language model (LLM) generalize to another model's reasoning process. The key finding is that explanation quality often fails to correlate with reasoning accuracy, particularly when models are trained differently. The researchers compared Model A (DeepSeek Coder) and Model B (DPO). Model A showed high baseline consistency (95%) in generating step-by-step reasoning for math problems, while Model B dropped significantly to 66% consistency when forced to follow Model A's flawed reasoning path. The researchers found that Model A's explanations, even when logically flawed or based on incorrect steps, were persuasive enough to fool human experts into rating them highly, demonstrating the 'shared hallucination' phenomenon. They suggest that relying on consistency alone is dangerous; instead, systems need explicit mechanisms, like a generator/evaluator loop, to check each other’s logic, moving away from simple single-prompt methods to ensure true, generalizable reasoning rather than just consistent, but potentially flawed, narrative generation.

### Experimental Setup

- Comparing Models A (DeepSeek Coder) and B (DPO) on reasoning tasks
- Metrics used were accuracy and consistency of the chain of thought (CoT)
- Baseline consistency for Model A was 95%.

### Results on Reasoning Tasks

- Model A showed high accuracy on math problems (MedCalc) but consistency dropped to 66% for Model B when following A's steps
- Model A's explanations were persuasive even when mathematically wrong.

### The Core Finding

- Reasoning and explanation generation are separable skills
- Consistency in explaining steps does not guarantee underlying logical truth.

### Implications for AI Reliability

- Relying on explanations can mask errors, creating a false sense of security
- The risk of 'shared hallucinations' is high when models are trained on similar internet data.

### The Solution

- Implementing a system where models explicitly check each other's work (Generator/Evaluator loop) to enforce cross-validation rather than relying on self-correction or simple consistency checks.

![Screenshot at 00:00: The introductory screen featuring two podcasters and the call to action "BECOME A MEMBER TODAY!" over a signal graph.](https://ss.rapidrecap.app/screens/B19-o2tfQI8/00-00-00.jpg)
![Screenshot at 00:15: The speakers begin discussing large language models like QW-Q32 and GPT-OSS, noting they have become more transparent.](https://ss.rapidrecap.app/screens/B19-o2tfQI8/00-00-15.jpg)
![Screenshot at 01:03: The specific paper title being referenced: "Explanations Generalize Across Large Reasoning Models."](https://ss.rapidrecap.app/screens/B19-o2tfQI8/00-01-03.jpg)
![Screenshot at 02:11: The two core metrics used for evaluation: accuracy and consistency.](https://ss.rapidrecap.app/screens/B19-o2tfQI8/00-02-11.jpg)
![Screenshot at 06:38: The speaker highlights the counterintuitive finding that randomness \(sampling\) improved consistency compared to the greedy approach.](https://ss.rapidrecap.app/screens/B19-o2tfQI8/00-06-38.jpg)