# Why Benchmarking Is Hard

Source: https://www.youtube.com/watch?v=8lkbtnp-uXI
Recap page: https://rapidrecap.app/video/8lkbtnp-uXI
Generated: 2025-12-29T20:03:35.344+00:00

---
## Quick Overview

Benchmarking Large Language Models (LLMs) is difficult because their scores are highly dependent on the specific evaluation framework, such as the prompt template and the environment (e.g., using a native SDK versus an API aggregator), leading to massive score variances (up to 23 percentage points difference between providers) and creating a high barrier to entry for accurately assessing true model intelligence versus infrastructure stability.

**Key Points:**
- Model A scored 95% and Model B scored 93% on a specific benchmark, initially suggesting Model A was superior.
- The reality is that the final score is the sum of dozens of variables, including the prompt template, sampling parameters, and environment (e.g., running locally vs. via API).
- When testing models via a provider's native SDK versus an API aggregator like DeepInfra or GMIA Cloud, results can vary significantly, sometimes leading to outright failures (scoring 0/500) when using the aggregator.
- OpenAI experienced infrastructure issues, only running 477 out of 500 problems successfully when using their own infrastructure, highlighting environmental impact.
- The choice of the grading model itself is a variable; using a second LLM to grade the first model's answer introduced a 23 percentage point difference on one benchmark compared to using the provider's own native SDK.
- The largest variable impacting the final score is the choice of the agent's environment/hosting infrastructure, which can cause score swings of up to 11% or more.

![Screenshot at 00:18: Model A scoring 95% and Model B scoring 93% on a benchmark, which the speakers argue is misleading due to the chaotic complexity of the testing setup.](https://ss.rapidrecap.app/screens/8lkbtnp-uXI/00-00-18.jpg)

**Context:** This podcast segment discusses the inherent difficulties and complexities involved in reliably benchmarking Large Language Models (LLMs). The speakers explain that simple scorecard comparisons, like those seen on leaderboards, are often misleading because the results are heavily influenced by external factors—the 'stack'—surrounding the model inference, such as the specific prompt structure used, the inference environment (local vs. cloud API), and even which model is used for grading the output.

## Detailed Analysis

The discussion centers on why benchmarking LLMs is far more complex than simply comparing leaderboard scores. The speakers point out that the perceived superiority of one model (e.g., Model A at 95% vs. Model B at 93%) is often an illusion created by the testing environment. Key variables include the prompt template, sampling parameters, and the execution environment (e.g., using a provider's native API versus a third-party aggregator like DeepInfra). Even the infrastructure itself causes variance; OpenAI reportedly had issues running all tests successfully on their own stack. Furthermore, using one LLM to grade another model's response (like using GPT-4 to grade an answer) introduces a massive 23-point swing compared to using the model's native, highly-tuned evaluation framework. The conclusion drawn is that the hosting environment and the evaluation scaffolding (including the grading prompt) are the single biggest factors affecting the final score, often overshadowing the underlying intelligence of the model being tested.

### Initial Benchmark Observation

- Model A scored 95%
- Model B scored 93%
- Initial assumption is A is superior.

### Variables Affecting Scores

- Prompt template
- Sampling parameters
- Environment (local vs. API aggregator).

### Infrastructure Impact

- OpenAI failed 3% of tests due to infrastructure issues
- Native SDK vs. Aggregator API shows major score differences.

### Grading Methodology Impact

- Using an LLM (like GPT-4) to grade another model's answer causes a 23% point swing compared to native evaluation.

### Key Takeaway

- The choice of infrastructure/hosting is the single biggest variable affecting scores, leading to score differences of up to 11% or more.

![Screenshot at 00:09: A visual representation of the chaotic complexity of LLM evaluation, showing fluctuating waveforms.](https://ss.rapidrecap.app/screens/8lkbtnp-uXI/00-00-09.jpg)
![Screenshot at 00:18: Specific scores are shown: Model A at 95% and Model B at 93%, setting up the discussion about misleading simplicity.](https://ss.rapidrecap.app/screens/8lkbtnp-uXI/00-00-18.jpg)
![Screenshot at 01:14: Mention of API instability and general instability as factors contributing to evaluation difficulty.](https://ss.rapidrecap.app/screens/8lkbtnp-uXI/00-01-14.jpg)
![Screenshot at 02:28: Default temperature setting of 0.0, indicating a deterministic \(non-creative\) setting used for testing.](https://ss.rapidrecap.app/screens/8lkbtnp-uXI/00-02-28.jpg)
![Screenshot at 07:54: A score of 0 is mentioned for tests that failed due to response corruption or token limits.](https://ss.rapidrecap.app/screens/8lkbtnp-uXI/00-07-54.jpg)