# LLMs Encode Their Failures: Predicting Success From Pre-Generation Activations

Source: https://www.youtube.com/watch?v=V0BBffjYQ0A
Recap page: https://rapidrecap.app/video/V0BBffjYQ0A
Generated: 2026-02-15T20:02:46.598+00:00

---
## Quick Overview

The research reveals that employing linear probing, which involves drawing a line between success and failure signals in an LLM's internal state, significantly reduces inference costs (by 70% compared to the baseline) and improves accuracy on reasoning tasks, demonstrating that the model's internal state is highly informative about the likelihood of failure, even when the model is deep in thought, thus providing a cost-effective way to monitor model reasoning without incurring full computation.

**Key Points:**
- A new technique called Linear Probing successfully predicts LLM failures by analyzing internal state vectors at the end of instruction tokens.
- The small model (GPT-OS 20B) achieved a significant 70% reduction in inference cost per million tokens by routing difficult problems to a larger, more expensive model only when necessary.
- The cost reduction was achieved by using the small model as a gatekeeper, rerouting prompts predicted to fail to the larger model, saving computation on easy tasks.
- The linear probe's accuracy in predicting failure correlated strongly with the human difficulty score for the same problems, suggesting the internal state is highly indicative of reasoning difficulty.
- The paper suggests decoupling reasoning from human mimicry is crucial, as the model's internal state (when correctly probed) is a better measure of difficulty than human perception of that difficulty.
- The linear probe works by training a simple linear classifier on the internal state vector generated after the final instruction token to predict success or failure.

![Screenshot at 00:15: The speaker highlights the cost of reasoning in deploying LLMs, stating that the cost of inference is the main issue being addressed by the research.](https://ss.rapidrecap.app/screens/V0BBffjYQ0A/00-00-15.jpg)

**Context:** This video discusses findings from a research paper titled "LLMs Encode Their Failures: Predicting Success From Pre-Generation Activations," presented on Thursday, February 12, 2026. The core problem addressed is the high and often wasteful cost of deploying large language models (LLMs) by forcing them to compute every token, even for simple queries. The research proposes a novel, cost-effective method using linear probing on the model's internal state immediately after processing the prompt to predict whether the subsequent generation will succeed or fail.

## Detailed Analysis

The video explains research from Oxford Internet Institute and University College London that proposes a radical solution to the high cost of deploying large language models (LLMs) by predicting when a model is likely to fail before it generates output. The core technique is Linear Probing, where a simple linear classifier is trained on the LLM's internal state vector sampled immediately after the final instruction token. This probe predicts if the model will succeed or fail on the task. Testing on the GPT-OS 20B model showed that when routing easy tasks to a small model and only sending difficult tasks (predicted by the probe) to a larger, more expensive model, inference costs were reduced by 70%. The accuracy of this failure prediction (AUC score of 0.87 to 0.93) was highly correlated with human-assigned difficulty scores, suggesting the internal state is a strong indicator of reasoning challenge. The authors argue this approach avoids wasting computation on problems humans find trivial but which cause the model internal struggle, effectively decoupling the model’s internal difficulty assessment from its external human-mimicking behavior.

### Problem & Proposal

- High cost of deploying LLMs due to computation on every token
- Proposal is Linear Probing on internal state to predict failure.

### Methodology

- Train a simple linear classifier (the probe) on the internal state vector generated at the end of the input prompt.

### Results

- 70% reduction in inference cost per million tokens using the small model as a gatekeeper; Accuracy (AUC) improved from 0.87 to 0.92 when increasing reasoning difficulty.

### Key Insight

- The model's internal state is a better indicator of reasoning difficulty than human perception, leading to the concept of the 'Reasoning Paradox' where internal struggle (high internal state signal) happens even when the output seems easy to a human.

### Conclusion

- This technique allows for efficient resource allocation, potentially leading to future systems that separate reasoning from human mimicry.

![Screenshot at 00:00: Introductory slide showing two people podcasting with the text 'BECOME A MEMBER TODAY!'](https://ss.rapidrecap.app/screens/V0BBffjYQ0A/00-00-00.jpg)
![Screenshot at 00:38: The speaker setting the context for why predicting LLM failure is critical due to computational costs.](https://ss.rapidrecap.app/screens/V0BBffjYQ0A/00-00-38.jpg)
![Screenshot at 01:50: The speaker introducing the core technique: Linear Probing.](https://ss.rapidrecap.app/screens/V0BBffjYQ0A/00-01-50.jpg)
![Screenshot at 04:47: The speaker stating the significant improvement in accuracy \(86% up to 92%\) when using the probe.](https://ss.rapidrecap.app/screens/V0BBffjYQ0A/00-04-47.jpg)
![Screenshot at 08:16: The speaker showing the 70% reduction in inference costs for the math benchmark.](https://ss.rapidrecap.app/screens/V0BBffjYQ0A/00-08-16.jpg)