The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality

Quick Overview

The FACTS Leaderboard demonstrates that while large language models like GPT-4 show strong performance (68.8% overall accuracy), the best models like Gemini 3 Pro achieve higher factual accuracy (outperforming the baseline by 3.4 search queries per query) by employing a dual-judge system that prioritizes non-hallucination over raw accuracy, ultimately forcing models toward greater precision and honesty.

Key Points: Gemini 3 Pro leads the FACTS Leaderboard with an overall factual accuracy score of 68.8%, outperforming GPT-4 by 3.4 search queries per query on average. The top-performing models utilize a dual-judge system where one judge verifies coverage against the source document, and the second judge verifies no contradictions exist. The evaluation heavily penalizes models for being confidently wrong (hallucination), scoring models that admit ignorance higher than those that guess incorrectly. The researchers found that models like GPT-4 were more prone to aggressively guessing (1.9% error rate) compared to Gemini 3 Pro, which hedged more often. Multi-hop queries, requiring multiple steps to answer (like searching for document A, finding a name, then searching document B for that name), are significantly more complex for models to handle accurately. The best models are designed to be precision-oriented, prioritizing the absence of contradictions and known limitations (like when to admit ignorance) over simply maximizing raw accuracy scores.

Context: This video introduces the FACTS Leaderboard, a new benchmark designed to comprehensively measure the factuality and trustworthiness of Large Language Models (LLMs). The core challenge addressed is the tendency of LLMs to hallucinate or confidently state incorrect facts, especially when dealing with complex, multi-step queries or domain-specific data like financial reports or medical documents. The leaderboard evaluates models based on both coverage (including all necessary facts) and no-contradiction scores, using human experts to create a rigorous evaluation rubric.