The Anointment Effect: How a Questionable “MIT Study” Became Gospel

Quick Overview

The study claiming 95% of corporate AI investments fail lacks scientific rigor, as it relies on a convenience sample and the authority of the MIT logo, ultimately revealing a fundamental flaw in how the media and potentially the industry value AI success metrics.

Key Points: The referenced study claiming 95% of corporate AI investments fail is not a rigorous, peer-reviewed MIT study, but rather based on a convenience sample. The study's findings, which are highly cited in headlines, rely on the perceived authority of the MIT name rather than solid methodology. The sampling method involved interviewing people easily accessible (like at an AI conference) or those willing to talk, leading to a skewed dataset heavily favoring failure narratives. The analysis suggests that the media amplified the 95% failure rate, which was derived from assessing projects that didn't have clear definitions of success (like ROI) or clear protocols for verification. The underlying issue is the 'Anointment Effect,' where institutional prestige (like MIT) bypasses scrutiny, making the statistic seem credible even when the methodology is flawed. The proposed solution is a fundamental shift in how information is valued, requiring auditing the source/methodology, rather than just accepting prestigious labels. The speaker argues that the industry is moving toward an 'Era of Infinite Artifacts,' demanding new critical literacy skills to verify claims, especially those that seem too simple or too dire.

Context: The discussion centers on a widely circulated statistic—often attributed to MIT—claiming that 95% of corporate Artificial Intelligence (AI) investments are failing. The speaker critiques this statistic, arguing it is not scientifically sound due to its methodological flaws, specifically its reliance on a skewed convenience sample and the undue influence of the MIT brand reputation, which he terms the 'Anointment Effect.'

Detailed Analysis

The video debunks the frequently cited statistic that 95% of corporate AI investments fail, attributing the statistic's wide acceptance to the 'Anointment Effect,' where prestige, like the MIT logo, overrides skepticism about methodology. The speaker cites Toby Stuart's critique, noting that the study cited was neither a definitive MIT study nor peer-reviewed in a rigorous way. The data compilation involved a convenience sample—interviewing people easily accessible at conferences or willing to talk—which skewed the results toward failure narratives (01:22, 04:54). Stuart suggests this reliance on convenience sampling, coupled with a lack of clear success metrics (like ROI) in the audited projects, made the 95% failure rate seem compelling but ultimately flawed (03:22, 04:44). The core issue is that the media used the MIT brand as a shortcut to legitimacy, allowing the high failure rate to bypass scrutiny (06:57). The speaker concludes that this incident shows a fundamental shift: in an era of massive AI-generated content, critical skills like auditing methodology and source credibility are essential civic skills, rather than simply trusting prestigious labels (09:51, 11:23).