# How2Everything: Mining the Web for How-To Procedures to Evaluate and Improve LLMs Source: https://www.youtube.com/watch?v=XGmJyV6qjKk Recap page: https://rapidrecap.app/video/XGmJyV6qjKk Generated: 2026-02-12T19:04:33.877+00:00 --- ## Quick Overview Researchers developed a novel evaluation framework for Large Language Models (LLMs) that specifically assesses their ability to generate executable, step-by-step instructions for real-world tasks, finding that models trained on web-mined "how-to" data significantly outperform models trained only on traditional scientific/procedural data, achieving an 80.5% agreement with human annotators on the quality of their generated steps. **Key Points:** - The research introduces a framework to evaluate LLMs on generating executable, step-by-step procedures for real-world tasks, addressing a practical bottleneck in LLM application. - Models trained on web-mined 'how-to' data (like recipe instructions) significantly outperformed models trained only on scientific/procedural data. - The best model achieved an 80.5% agreement score with human annotators when judging the validity of generated steps for tasks like fixing a leak or making coffee. - The study highlights that current LLMs often fail on procedural knowledge, frequently producing dangerous or nonsensical steps (like turning on the oven before adding ingredients). - The key innovation lies in using procedural data, which requires logical flow and step verification, moving beyond simple factual recall or creative generation. - The researchers specifically tested models on procedural data covering domains like Home & Hobbies, Crime & Law, and Industrial Transportation. ![Screenshot at 00:00: The video opens with an image of two people recording a podcast over a soundwave graphic, overlayed with the text 'BECOME A MEMBER TODAY!', indicating this is likely an episode from the AI Podcasts Daily series discussing new research.](https://ss.rapidrecap.app/screens/XGmJyV6qjKk/00-00-00.jpg) **Context:** This research, a collaboration between the Allen Institute for AI, the University of Maryland, and the University of Washington, addresses the practical limitations of Large Language Models (LLMs) when asked to provide actionable, real-world instructions. While LLMs excel at tasks like writing poetry or summarizing abstracts, they often struggle with procedural knowledge—the ability to give correct, executable steps for tasks like fixing a leak or brewing coffee. The paper 'How to Everything' aims to quantify and improve this capability by creating a specialized evaluation benchmark. ## Detailed Analysis The video discusses a research paper titled 'How to Everything' which focuses on evaluating LLMs' ability to generate valid, executable, step-by-step instructions for real-world tasks, a critical bottleneck in deployment. The researchers found that LLMs trained on web-scraped 'how-to' content (like recipes or guides) significantly outperform models trained solely on scientific or procedural data. They created a new evaluation metric based on judging the logical flow and correctness of these procedural steps. The paper tested models on 14 topics, spanning home/hobbies, law, and industrial processes. A crucial finding was that even models like GPT-5, while good at memorizing facts, struggled severely with complex procedures like fixing a hydraulic pump, often producing dangerous errors (e.g., 'put turkey in oven before turning it on'). The researchers developed a specialized judge model, trained on human-annotated data, to score the procedure's validity. This judge achieved an 80.5% agreement rate with human evaluators, indicating a strong benchmark. The study argues that procedural reasoning is a fundamental skill, separate from pure mathematical or coding ability, and that the high success rate of models trained on web-mined 'how-to' data suggests this data type is crucial for closing the gap between an LLM's knowledge and its real-world utility. ### The Core Problem - LLMs struggle with procedural knowledge, often giving dangerous or incorrect steps for real-world tasks like repairing equipment or cooking. ### The Evaluation Framework - Researchers created a new benchmark focusing on generating executable, step-by-step instructions, testing models on diverse real-world domains. ### Training Data Impact - Models trained on web-mined 'how-to' content (recipes, guides) performed significantly better than those trained only on scientific/procedural datasets. ### Key Metric and Results - The evaluation yielded an 80.5% agreement score between the specialized judge model and human annotators, demonstrating the framework's reliability. ### GPO vs. Standard Models - Models using Group Relative Policy Optimization (GRPO) and specialized training outperformed standard models, showing that learning the structure of procedures is key, not just memorization. ### Conclusion - Procedural reasoning is a distinct, essential skill for LLMs, requiring explicit training beyond factual recall to ensure safety and efficacy in real-world applications. ![Screenshot at 0:00: The video opens with a generic podcast promotion graphic featuring two hosts, indicating the source is a discussion or news segment about AI research.](https://ss.rapidrecap.app/screens/XGmJyV6qjKk/00-00-00.jpg) ![Screenshot at 0:24: Text overlay mentioning the collaboration between Allen Institute for AI, University of Maryland, and University of Washington, establishing the research sources.](https://ss.rapidrecap.app/screens/XGmJyV6qjKk/00-00-24.jpg) ![Screenshot at 4:28: The speaker discusses the four components of the evaluation framework: How to Mine, How to Bench, How to Score, and How to Judge.](https://ss.rapidrecap.app/screens/XGmJyV6qjKk/00-04-28.jpg) ![Screenshot at 11:55: A speaker notes that a model heavily trained on one specific task risks catastrophically forgetting how to perform others, highlighting the challenge of generalization.](https://ss.rapidrecap.app/screens/XGmJyV6qjKk/00-11-55.jpg) ![Screenshot at 15:34: The speaker points out the disparity where models excel at 'writing recipes' but fail at basic procedural tasks like 'how to service a hydraulic pump,' showing the gap between knowing facts and executing procedures.](https://ss.rapidrecap.app/screens/XGmJyV6qjKk/00-15-34.jpg)