FinePDFs: Liberating 3T of the finest tokens from PDFs

Quick Overview

The FinePDFs project successfully liberated 3.2 trillion tokens of data, previously hidden in PDFs, by developing a multi-stage engineering approach that prioritized quality over quantity, ultimately achieving high-quality data extraction that outperformed simpler, faster methods.

Key Points: The FinePDFs project extracted 3.2 trillion tokens from PDFs, which is 0.6% of all PDF files. The extracted data represented a disproportionately large amount of valuable, rich text knowledge compared to HTML. The team used three main strategies: pure parsing, pipeline methods (like RAG/Docling), and high-quality/high-cost methods. The pure parsing strategy, which relied on simple hashing, failed to capture complex layouts and led to the model getting stuck in repetition loops. The pipeline method using an LLM to guess the ordering of elements and then filtering out noise was more successful but still required significant cleanup. The high-quality/high-cost approach, using a large model (like a large LLaMA) to evaluate the output of a smaller model, yielded the best results, achieving high precision (0.869) across key benchmarks. The final success involved distilling knowledge from large models trained on dense data into smaller, cheaper student models.

Context: This video from ReallyEasyAI discusses the challenges and successful engineering strategies employed in the creation of the FinePDFs dataset, which aimed to extract high-quality knowledge from PDF documents for training large language models (LLMs). The context revolves around the difficulty of accurately parsing complex document structures found in PDFs, which often contain tables, images, and non-linear layouts, contrasting this with simpler HTML sources.

Detailed Analysis

The discussion centers on the FinePDFs project, which aimed to extract valuable tokens from 3.2 trillion PDF files, representing only 0.6% of all PDFs but containing a disproportionately high amount of dense, valuable knowledge compared to HTML. The core challenge was handling the complex, non-linear, and often poorly structured content within PDFs, such as tables, complex layouts, and figures. The team explored three primary strategies to overcome these issues: pure parsing, pipeline methods, and high-quality/high-cost evaluation. The pure parsing strategy, which used simple hashing, resulted in models getting stuck in repetition loops because the reading order was ambiguous. The pipeline method, using models like RAG or Docling, improved structure by treating pages as images and using OCR, but still required significant cleanup of noise like headers, footers, and tables. The most successful approach involved using a powerful model (like a large LLaMA) to judge the output of a smaller model based on metrics like decision accuracy against a gold standard, achieving high accuracy (0.896 vs 0.920 for the simpler approach). This led to the final strategy of distilling knowledge from the complex, long-context models into smaller, cheaper student models, which ultimately proved to be the key to success in handling complex, long-form documents like legal papers and engineering specs.