ModelTables: A Corpus of Tables about Models

Quick Overview

Model Tables, a newly created large-scale corpus of over 90,000 tables about 60,000 AI models sourced from Hugging Face, GitHub, and academic papers, establishes the first benchmark specifically designed to address the knowledge synthesis gap in the overwhelming "model lake" by focusing on structured performance and configuration data rather than simple keyword search.

Key Points: The Model Tables corpus compiles data from over 60,000 models across 90,000 tables, compiled because quantitative comparison data is often locked within tables in model cards and papers, making keyword search insufficient. The key distinction of Model Tables compared to general corpora like WebTables is its mission: creating a benchmark built specifically around structured data like performance and configuration tables for AI models, focusing on knowledge synthesis. Model tables exhibit much denser intertable relationships than general tables, forming a complex web where one table's publication instantly connects it to dozens of previous tables via citations or shared benchmarks. To improve searchability across highly varied formats, the team used data augmentation techniques, including transpose augmentation (flipping rows/columns) and header-to-cell augmentation (e.g., turning '3.14' under 'F1 score' into '3.14 F1 score'). Evaluation showed clear room for improvement in table search methods, with dense retrieval achieving 66.5% P@1 on paper ground truth, but hybrid retrieval (combining content search with metadata search) performed better on model and dataset ground truths. The corpus required aggressive filtering, reducing initial candidates from over 1.1 million Hugging Face models down to 60,000 that had extractable tables referenced by a valid research paper, highlighting poor documentation elsewhere. The problem of model cataloging is accelerating, as the growth trajectory plotted by the authors shows that both the number of models and tables are growing exponentially, necessitating continuous benchmark updates.