The biggest Mystery of LLMs have just been solved

Quick Overview

The non-determinism observed in Large Language Models (LLMs) during inference, specifically when using GPU parallelism, stems from floating-point arithmetic inconsistencies caused by variable reduction orders, which can be solved by implementing batch-invariant kernels that enforce a deterministic reduction tree, ensuring identical outputs across different batch sizes and hardware configurations.

Key Points: Setting the Temperature parameter to 0 (most consistent) does not guarantee deterministic output in LLMs when GPU parallelism is used. Non-determinism arises because parallel operations, like summing partial results across multiple GPUs, result in variable reduction orders, leading to floating-point rounding differences (e.g., (10^8 + 1) - 10^8 != 10^8 + (1 - 10^8) in 32-bit floating point). The researchers demonstrated that this non-determinism can cause a single bit-level mistake to cascade, resulting in a different final token choice (e.g., 'ball' vs. 'frisbee' or 'Queens, New York' vs. 'New York City'). The solution is implementing batch-invariant kernels, which lock the kernel policy and enforce a deterministic reduction tree, ensuring the same output regardless of batch size (Batch=1 vs Batch=16). The default vLLM configuration took 26 seconds to process 1000 sequences (90-110 tokens long) on a Qwen-3-235B model, while an unoptimized deterministic vLLM build took 55 seconds (2.1x longer). The optimized batch-invariant approach with an improved attention kernel reduced the time penalty to 42 seconds (1.6x longer than default), offering determinism without drastically sacrificing performance.

Context: The video explains the complex issue of non-determinism in Large Language Model (LLM) inference, a problem where running the same prompt multiple times, even with temperature set to zero (greedy decoding), yields different results. This inconsistency is attributed to hardware-level factors, specifically GPU parallelism and floating-point arithmetic precision differences arising from variable reduction orders when aggregating partial results across different processing units (GPUs). The video references a research paper, "Defeating Nondeterminism in LLM Inference" by Horace He in collaboration with Thinking Machines, to detail the pipeline causing this issue and the fix implemented in vLLM.