Why Benchmarking Is Hard

Quick Overview

Benchmarking Large Language Models (LLMs) is difficult because their scores are highly dependent on the specific evaluation framework, such as the prompt template and the environment (e.g., using a native SDK versus an API aggregator), leading to massive score variances (up to 23 percentage points difference between providers) and creating a high barrier to entry for accurately assessing true model intelligence versus infrastructure stability.

Key Points: Model A scored 95% and Model B scored 93% on a specific benchmark, initially suggesting Model A was superior. The reality is that the final score is the sum of dozens of variables, including the prompt template, sampling parameters, and environment (e.g., running locally vs. via API). When testing models via a provider's native SDK versus an API aggregator like DeepInfra or GMIA Cloud, results can vary significantly, sometimes leading to outright failures (scoring 0/500) when using the aggregator. OpenAI experienced infrastructure issues, only running 477 out of 500 problems successfully when using their own infrastructure, highlighting environmental impact. The choice of the grading model itself is a variable; using a second LLM to grade the first model's answer introduced a 23 percentage point difference on one benchmark compared to using the provider's own native SDK. The largest variable impacting the final score is the choice of the agent's environment/hosting infrastructure, which can cause score swings of up to 11% or more.

Context: This podcast segment discusses the inherent difficulties and complexities involved in reliably benchmarking Large Language Models (LLMs). The speakers explain that simple scorecard comparisons, like those seen on leaderboards, are often misleading because the results are heavily influenced by external factors—the 'stack'—surrounding the model inference, such as the specific prompt structure used, the inference environment (local vs. cloud API), and even which model is used for grading the output.