# The biggest Mystery of LLMs have just been solved

Source: https://www.youtube.com/watch?v=BbI8n9XZJo4
Recap page: https://rapidrecap.app/video/BbI8n9XZJo4
Generated: 2025-11-16T14:32:36.156+00:00

---
## Quick Overview

The non-determinism observed in Large Language Models (LLMs) during inference, specifically when using GPU parallelism, stems from floating-point arithmetic inconsistencies caused by variable reduction orders, which can be solved by implementing batch-invariant kernels that enforce a deterministic reduction tree, ensuring identical outputs across different batch sizes and hardware configurations.

**Key Points:**
- Setting the Temperature parameter to 0 (most consistent) does not guarantee deterministic output in LLMs when GPU parallelism is used.
- Non-determinism arises because parallel operations, like summing partial results across multiple GPUs, result in variable reduction orders, leading to floating-point rounding differences (e.g., (10^8 + 1) - 10^8 != 10^8 + (1 - 10^8) in 32-bit floating point).
- The researchers demonstrated that this non-determinism can cause a single bit-level mistake to cascade, resulting in a different final token choice (e.g., 'ball' vs. 'frisbee' or 'Queens, New York' vs. 'New York City').
- The solution is implementing batch-invariant kernels, which lock the kernel policy and enforce a deterministic reduction tree, ensuring the same output regardless of batch size (Batch=1 vs Batch=16).
- The default vLLM configuration took 26 seconds to process 1000 sequences (90-110 tokens long) on a Qwen-3-235B model, while an unoptimized deterministic vLLM build took 55 seconds (~2.1x longer).
- The optimized batch-invariant approach with an improved attention kernel reduced the time penalty to 42 seconds (~1.6x longer than default), offering determinism without drastically sacrificing performance.

![Screenshot at 0:23: Demonstration showing that for the prompt 'The dog caught the \_\_', a temperature of 0 results in different token probabilities \('flu' 1%, 'car' 4%, 'squirrel' 15%, 'frisbee' 20%, 'ball' 60%\) when the model is running non-deterministically.](https://ss.rapidrecap.app/screens/BbI8n9XZJo4/00-00-23.png)

**Context:** The video explains the complex issue of non-determinism in Large Language Model (LLM) inference, a problem where running the same prompt multiple times, even with temperature set to zero (greedy decoding), yields different results. This inconsistency is attributed to hardware-level factors, specifically GPU parallelism and floating-point arithmetic precision differences arising from variable reduction orders when aggregating partial results across different processing units (GPUs). The video references a research paper, "Defeating Nondeterminism in LLM Inference" by Horace He in collaboration with Thinking Machines, to detail the pipeline causing this issue and the fix implemented in vLLM.

## Detailed Analysis

The video dissects the source of non-determinism in LLM inference, particularly when leveraging GPU parallelism for speed. The core issue is that floating-point arithmetic is not strictly associative, meaning the order in which partial results are summed matters due to rounding errors, as illustrated by the example: (10^8 + 1) - 10^8 = 1, while 10^8 + (1 - 10^8) = 0 when using 32-bit precision. When multiple GPUs process data in parallel, the reduction order of these partial sums varies between runs (Run A vs Run B), causing these rounding errors to cascade. This small numerical shift can ultimately change the token selection via greedy argmax, leading to different outputs for the same prompt. The paper by Thinking Machines introduces 'batch-invariant kernels' as the fix. These kernels enforce a deterministic reduction tree (using a Locked Kernel Policy instead of an Auto Kernel picker), ensuring the reduction order is fixed regardless of batch size (Batch=1 vs Batch=16). The performance comparison shows that the default vLLM took 26 seconds for 1000 sequences, while an unoptimized deterministic build took 55 seconds (~2.1x slower). The optimized batch-invariant approach with an improved attention kernel achieved a time of 42 seconds (~1.6x slower than default), successfully achieving deterministic results with improved training efficiency and preventing the 'domino effect' of small numerical errors.

### Temperature Parameter

- 1 = More creative
- 0 = More consistent
- Setting temperature to 0 forces the model to be greedy, picking the token with the highest probability.

### Source of Non-Determinism (Parallelism)

- Parallel execution on GPUs results in variable reduction orders when summing partial results, leading to low-level floating-point precision differences.

### Floating Point Arithmetic Example

- (10^8 + 1) - 10^8 = 1 (Order A) vs. 10^8 + (1 - 10^8) = 0 (Order B) in 32-bit math, demonstrating that associativity fails.

### Domino Effect of Errors

- A tiny bit-level mistake (rounding difference) can cascade across layers, changing the final token choice (e.g., 'pen' vs 'pencil' in greedy argmax).

### Batch-Variant vs. Batch-Invariant

- Batch-Variant systems use auto kernel pickers and variable reduction orders, leading to different outputs (Output A vs Output B) for different batch sizes, marked with a red X. Batch-Invariant systems use Locked Kernel Policy and deterministic reduction trees, resulting in identical outputs (Output A) for all batch sizes, marked with a green checkmark.

### Performance Trade-offs

- The default vLLM took 26s for 1000 sequences; the unoptimized deterministic version was 55s (~2.1x longer); the optimized batch-invariant kernel achieved 42s (~1.6x longer than default).

### Serving vs. Training Differences

- Serving uses dynamic batching and KV cache with FP16 kernels, while Training uses fixed batching, no KV cache, and FP32 kernels, leading to inherent differences that must be reconciled for consistent evaluation.

![Screenshot at 0:00: The Google AI Studio interface displaying model options like Gemini 1.5 Flash and parameter settings on the right sidebar.](https://ss.rapidrecap.app/screens/BbI8n9XZJo4/00-00-00.png)
![Screenshot at 0:04: Demonstration of the Temperature slider moving between 0 \(More consistent\) and 1 \(More creative\) in the LLM settings.](https://ss.rapidrecap.app/screens/BbI8n9XZJo4/00-00-04.png)
![Screenshot at 0:14: Visual representation of the Temperature scale, showing 0 \(blue/consistent\) to 1 \(red/creative\).](https://ss.rapidrecap.app/screens/BbI8n9XZJo4/00-00-14.png)
![Screenshot at 0:22: Demonstration of the probability distribution for the next token after 'The dog caught the \_\_' at Temperature=0, showing 'ball' at 60%.](https://ss.rapidrecap.app/screens/BbI8n9XZJo4/00-00-22.png)
![Screenshot at 0:48: Demonstration of deterministic execution in a code environment where running 'print\("Hello, World!"\)' yields the same output every time.](https://ss.rapidrecap.app/screens/BbI8n9XZJo4/00-00-48.png)
![Screenshot at 1:24: Diagram illustrating the difference between deterministic and non-deterministic systems where user requests feed into a model producing an output.](https://ss.rapidrecap.app/screens/BbI8n9XZJo4/00-01-24.png)
![Screenshot at 2:47: Diagram showing parallel processing across three GPUs \(GPU 1, 2, 3\) merging partial results into a final sum, highlighting potential non-determinism.](https://ss.rapidrecap.app/screens/BbI8n9XZJo4/00-02-47.png)
![Screenshot at 3:31: Text overlay 'Parallelism X' with a large red X, symbolizing the problem caused by parallel execution.](https://ss.rapidrecap.app/screens/BbI8n9XZJo4/00-03-31.png)
![Screenshot at 4:21: Comparison of two requests to ChatGPT: the 10:00 AM request with batch size \[1, heads, seq\_len, dim\] and the 10:01 AM request with batch size \[16, heads, seq\_len, dim\], which result in different outputs due to non-determinism.](https://ss.rapidrecap.app/screens/BbI8n9XZJo4/00-04-21.png)
![Screenshot at 5:12: Diagram contrasting Order A and Order B for merging results, showing how different partial addition orders lead to 1 vs 0 due to floating-point rounding errors in 32-bit arithmetic.](https://ss.rapidrecap.app/screens/BbI8n9XZJo4/00-05-12.png)