Semantic Search over 9 Million Mathematical Theorems

Quick Overview

Researchers from the University of Washington developed a new semantic search method for mathematical theorems that successfully retrieved the correct theorem for a query in 37.8% of cases, significantly outperforming general large language models like GPT-4 and Gemini 1.0, which often failed or hallucinated when searching for specific mathematical concepts within a corpus of 9 million theorems.

Key Points: A new semantic search system for mathematical theorems, developed at the University of Washington, achieved a 37.8% success rate in finding the correct theorem for a query. General LLMs like GPT-4 (with web search) and Gemini 1.0 clustered their results, achieving only 19.8% and 19.0% success rates, respectively, on the same task. The new method works by translating the user's natural language query into a formal mathematical statement (like a lemma or proposition) and then searching a vector database containing 9 million theorems. The authors constructed a corpus of 9 million theorems, sourced from arXiv preprints, and explicitly organized the data to facilitate this specialized search. The system successfully bridges the gap between the natural language intent of the user and the formal structure of mathematical proofs, something traditional keyword search fails to do. The cost to build this specialized system was approximately $6,000, demonstrating cost-effectiveness compared to training massive general models. The paper emphasizes that the success metric is not just accuracy, but the quality of the result, showing that the body plus the introduction (the context) provides a much richer result than the theorem statement alone.

Context: The video discusses a significant development in information retrieval specifically tailored for mathematics, addressing the difficulty of finding specific mathematical theorems within vast digital libraries. The research, conducted by people from the University of Washington, aims to solve the 'search granularity problem' where standard search tools often fail to pinpoint the exact mathematical statement needed, leading to wasted time or incorrect results. The core focus is on improving how AI systems handle queries involving formal mathematical language.