# FinePDFs: Liberating 3T of the finest tokens from PDFs

Source: https://www.youtube.com/watch?v=1kn94nzDXGo
Recap page: https://rapidrecap.app/video/1kn94nzDXGo
Generated: 2026-01-11T04:33:51.468+00:00

---
## Quick Overview

The FinePDFs project successfully liberated 3.2 trillion tokens of data, previously hidden in PDFs, by developing a multi-stage engineering approach that prioritized quality over quantity, ultimately achieving high-quality data extraction that outperformed simpler, faster methods.

**Key Points:**
- The FinePDFs project extracted 3.2 trillion tokens from PDFs, which is 0.6% of all PDF files.
- The extracted data represented a disproportionately large amount of valuable, rich text knowledge compared to HTML.
- The team used three main strategies: pure parsing, pipeline methods (like RAG/Docling), and high-quality/high-cost methods.
- The pure parsing strategy, which relied on simple hashing, failed to capture complex layouts and led to the model getting stuck in repetition loops.
- The pipeline method using an LLM to guess the ordering of elements and then filtering out noise was more successful but still required significant cleanup.
- The high-quality/high-cost approach, using a large model (like a large LLaMA) to evaluate the output of a smaller model, yielded the best results, achieving high precision (0.869) across key benchmarks.
- The final success involved distilling knowledge from large models trained on dense data into smaller, cheaper student models.

![Screenshot at 00:27: The discussion highlights that the FinePDFs data extraction process was a multi-year project designed to overcome the inherent complexity of PDF structures, which often leads to errors when parsing.](https://ss.rapidrecap.app/screens/1kn94nzDXGo/00-00-27.jpg)

**Context:** This video from ReallyEasyAI discusses the challenges and successful engineering strategies employed in the creation of the FinePDFs dataset, which aimed to extract high-quality knowledge from PDF documents for training large language models (LLMs). The context revolves around the difficulty of accurately parsing complex document structures found in PDFs, which often contain tables, images, and non-linear layouts, contrasting this with simpler HTML sources.

## Detailed Analysis

The discussion centers on the FinePDFs project, which aimed to extract valuable tokens from 3.2 trillion PDF files, representing only 0.6% of all PDFs but containing a disproportionately high amount of dense, valuable knowledge compared to HTML. The core challenge was handling the complex, non-linear, and often poorly structured content within PDFs, such as tables, complex layouts, and figures. The team explored three primary strategies to overcome these issues: pure parsing, pipeline methods, and high-quality/high-cost evaluation. The pure parsing strategy, which used simple hashing, resulted in models getting stuck in repetition loops because the reading order was ambiguous. The pipeline method, using models like RAG or Docling, improved structure by treating pages as images and using OCR, but still required significant cleanup of noise like headers, footers, and tables. The most successful approach involved using a powerful model (like a large LLaMA) to judge the output of a smaller model based on metrics like decision accuracy against a gold standard, achieving high accuracy (0.896 vs 0.920 for the simpler approach). This led to the final strategy of distilling knowledge from the complex, long-context models into smaller, cheaper student models, which ultimately proved to be the key to success in handling complex, long-form documents like legal papers and engineering specs.

### Data Scale and Source

- 3.2 trillion tokens extracted from PDFs
- 0.6% of all PDF files analyzed
- Data was largely government and legal documentation

### Strategy 1

- Pure Parsing Failures: Simple hashing led to models getting stuck repeating text phrases and wasting tokens/cost.

### Strategy 2

- Pipeline Methods (RAG/Docling): Treated PDFs as images for language ID and OCR, required significant cleanup of noise like tables and formatting.

### Strategy 3

- High-Quality Evaluation: Used a large model to judge output quality against a gold standard (0.896 vs 0.920 accuracy), proving superior to simple metrics.

### Final Success

- Distilled knowledge from large, expensive models into smaller, cheaper student models (like a quantized LLaMA) to handle long-context documents efficiently.

![Screenshot at 00:05: The initial mention of the challenge: dealing with the massive scale of LLMs trained on data that is now potentially outdated or hidden in non-standard formats like PDFs.](https://ss.rapidrecap.app/screens/1kn94nzDXGo/00-00-05.jpg)
![Screenshot at 00:39: Visual representation of the data density issue, where the quality of information \(like that found in PDFs\) is far greater than simple quantity suggests.](https://ss.rapidrecap.app/screens/1kn94nzDXGo/00-00-39.jpg)
![Screenshot at 02:26: The team notes that the final solution involved a distillation process, moving away from solely relying on expensive, high-power models.](https://ss.rapidrecap.app/screens/1kn94nzDXGo/00-02-26.jpg)
![Screenshot at 08:48: The realization that the high-cost GPU path for processing data was not financially viable, forcing a shift in architecture.](https://ss.rapidrecap.app/screens/1kn94nzDXGo/00-08-48.jpg)
![Screenshot at 11:15: The final strategy involved training a lightweight classifier \(like a small LLM\) on the highest quality data chunks to maintain performance efficiently.](https://ss.rapidrecap.app/screens/1kn94nzDXGo/00-11-15.jpg)