Why We Need New AI Benchmarks, Which Industries Survive AI, and Recursive Learning Timelines | #218

Quick Overview

The largest disruption from companies failing to adopt AI will occur in 2026, necessitating that companies focus on identifying 2-3 needle-moving use cases and begin with external vendor RFPs tied to measurable results rather than attempting broad internal overhauls, while the need for thousands of narrow, task-specific benchmarks will supersede broad public ones.

Key Points: Matt Fitzpatrick predicts the "largest disruption ever in 2026 from companies that don't make this change," stating that "knowledge work as we currently know it" is cooked. Companies should not start with a 'let a thousand flowers bloom' approach but must identify 2-3 use cases that "materially move the needle" for the business. The initial AI deployment for a key use case should be done via an RFP to a third-party vendor compensated based on results to limit risk, as in-house teams often lack executive experience in this paradigm. The focus of benchmarking must shift from broad public metrics (like coding) to "custom evals on highly specific topics" to achieve task accuracy or human equivalence, necessitating thousands of narrow benchmarks. The adoption curve for AI in areas like contact centers has been slow due to challenges in handling complex, non-first-line resolution topics and the inherent human preference for talking to other humans. Invisible Technologies focuses on tailoring existing LLMs by fine-tuning them on company-specific information, asserting that human-in-the-loop validation (RLHF) remains crucial, especially for reasoning tasks, even as models advance. A major challenge for AI implementation is the lack of focus on data as the starting point; companies must focus on the exact data needed for a specific use case, especially unstructured data like text and images, rather than trying to master the entire data repository.

Context: This episode of Moonshots features Matt Fitzpatrick, CEO of Invisible Technologies and former Global Head of Quantum Black Labs at McKinsey, discussing the urgent need for enterprise transformation due to AI capabilities. The conversation centers on the impending massive disruption expected by 2026 for companies that fail to pivot to become 'AI companies,' the differing impacts AI will have across industries, and the critical role of specialized data and benchmarking in successful implementation.