How2Everything: Mining the Web for How-To Procedures to Evaluate and Improve LLMs

Quick Overview

Researchers developed a novel evaluation framework for Large Language Models (LLMs) that specifically assesses their ability to generate executable, step-by-step instructions for real-world tasks, finding that models trained on web-mined "how-to" data significantly outperform models trained only on traditional scientific/procedural data, achieving an 80.5% agreement with human annotators on the quality of their generated steps.

Key Points: The research introduces a framework to evaluate LLMs on generating executable, step-by-step procedures for real-world tasks, addressing a practical bottleneck in LLM application. Models trained on web-mined 'how-to' data (like recipe instructions) significantly outperformed models trained only on scientific/procedural data. The best model achieved an 80.5% agreement score with human annotators when judging the validity of generated steps for tasks like fixing a leak or making coffee. The study highlights that current LLMs often fail on procedural knowledge, frequently producing dangerous or nonsensical steps (like turning on the oven before adding ingredients). The key innovation lies in using procedural data, which requires logical flow and step verification, moving beyond simple factual recall or creative generation. The researchers specifically tested models on procedural data covering domains like Home & Hobbies, Crime & Law, and Industrial Transportation.

Context: This research, a collaboration between the Allen Institute for AI, the University of Maryland, and the University of Washington, addresses the practical limitations of Large Language Models (LLMs) when asked to provide actionable, real-world instructions. While LLMs excel at tasks like writing poetry or summarizing abstracts, they often struggle with procedural knowledge—the ability to give correct, executable steps for tasks like fixing a leak or brewing coffee. The paper 'How to Everything' aims to quantify and improve this capability by creating a specialized evaluation benchmark.