# ModelTables: A Corpus of Tables about Models Source: https://www.youtube.com/watch?v=q8rCo97DurY Recap page: https://rapidrecap.app/video/q8rCo97DurY Generated: 2025-12-22T14:08:39.654+00:00 --- ## Quick Overview Model Tables, a newly created large-scale corpus of over 90,000 tables about 60,000 AI models sourced from Hugging Face, GitHub, and academic papers, establishes the first benchmark specifically designed to address the knowledge synthesis gap in the overwhelming "model lake" by focusing on structured performance and configuration data rather than simple keyword search. **Key Points:** - The Model Tables corpus compiles data from over 60,000 models across 90,000 tables, compiled because quantitative comparison data is often locked within tables in model cards and papers, making keyword search insufficient. - The key distinction of Model Tables compared to general corpora like WebTables is its mission: creating a benchmark built specifically around structured data like performance and configuration tables for AI models, focusing on knowledge synthesis. - Model tables exhibit much denser intertable relationships than general tables, forming a complex web where one table's publication instantly connects it to dozens of previous tables via citations or shared benchmarks. - To improve searchability across highly varied formats, the team used data augmentation techniques, including transpose augmentation (flipping rows/columns) and header-to-cell augmentation (e.g., turning '3.14' under 'F1 score' into '3.14 F1 score'). - Evaluation showed clear room for improvement in table search methods, with dense retrieval achieving 66.5% P@1 on paper ground truth, but hybrid retrieval (combining content search with metadata search) performed better on model and dataset ground truths. - The corpus required aggressive filtering, reducing initial candidates from over 1.1 million Hugging Face models down to 60,000 that had extractable tables referenced by a valid research paper, highlighting poor documentation elsewhere. - The problem of model cataloging is accelerating, as the growth trajectory plotted by the authors shows that both the number of models and tables are growing exponentially, necessitating continuous benchmark updates. **Context:** The discussion centers on the overwhelming challenge of navigating the 'model lake'—massive repositories like Hugging Face where finding the correct AI model for a specific task is difficult due to disparate configurations and inconsistent performance metrics across numerous research papers and model cards. This challenge highlights a genuine data crisis in how AI tools are cataloged and compared, necessitating a shift from basic text search to intelligent, structured data comparison. ## Detailed Analysis The Model Tables project tackles the AI model discovery crisis by creating the first large-scale benchmark composed entirely of structured tables detailing model performance and configuration, moving beyond the limitations of keyword search over unstructured documentation. The corpus currently spans over 60,000 models and 90,000 tables, extracted from sources like Hugging Face, GitHub, and academic papers, after an aggressive filtering process that reduced initial candidates significantly due to missing structured data or paper references. These model tables are unique because they form a tightly coupled ecosystem, exhibiting high intertable relationship density through citations and shared benchmarks, unlike general web tables. To handle inherent messiness, the researchers employed AI-assisted cleaning, piping unstructured text into LLMs for parsing, and used data augmentations like header-to-cell concatenation to inject crucial semantic context into numerical results, which significantly boosted search precision. Relatedness linking relies on three pillars: paper relatedness (direct citation or reference overlap), model card relatedness (base model inheritance), and dataset relatedness (shared data usage). Evaluation against existing search methods revealed that dense retrieval performed best overall on paper truth, but hybrid retrieval excelled when linking via metadata like model lineage, underscoring that different relational goals require different search strategies. The ultimate goal is enabling complex, synthesized queries, such as finding the 'best small model for NER with F1 > 90,' while the benchmark's projected exponential growth emphasizes the urgent need for continuous research in this area for scientific accountability. ### Problem Definition - Model lake overwhelming researchers due to unstructured performance metrics - Keyword search is insufficient for comparing models across configurations and papers - Critical gap exists in cataloging and comparing AI tools quantitatively. ### Model Tables Corpus Details - Scale reaches 90,000 tables covering over 60,000 models, sourced from Hugging Face, GitHub, and academic sources - Initial filtering reduced candidates from 1.1 million Hugging Face models to high-quality set linked to papers. ### Structural Uniqueness - Model tables are smaller but possess much denser intertable relationships, forming a complex, tightly coupled web that complicates isolation searches. ### Data Cleaning and Augmentation - Used LLMs to parse unstructured text blobs from sources like Semantic Scholar back into clean tables - Employed header-to-cell augmentation to add semantic context to numerical values (e.g., '3.14 F1 score') for better indexing. ### Defining Relatedness - Ground truth for linking tables relies on three pillars: Paper Relatedness (citation overlap) - Model Card Relatedness (base model inheritance, the most precise but rarest signal) - Dataset Relatedness (shared data sets). ### Search Performance Evaluation - Dense retrieval achieved 66.5% P@1 on paper ground truth, outperforming traditional unionable search (54.8% P@1 overall) - Hybrid retrieval performed better on model and dataset ground truths, indicating metadata importance for lineage tracking. ### Future Implications - The benchmark facilitates searching for complex synthesized answers rather than just model names - The exponential growth trajectory of models indicates the problem is accelerating, demanding continuous benchmark evolution (V2, V3).