# The REAL Reason AI Can’t Be Stopped Now Source: https://www.youtube.com/watch?v=uveQQW4ue70 Recap page: https://rapidrecap.app/video/uveQQW4ue70 Generated: 2026-01-21T19:38:21.502+00:00 --- ## Quick Overview The primary reason AI cannot be stopped now, according to the interviewee, is that the current trajectory of AI development, characterized by exponential intelligence growth (like an AI with an IQ of 170), makes human control nearly impossible, leading to a situation where humanity might be trapped by its own creation, similar to King Midas's curse or being trapped within a black hole's gravity well. **Key Points:** - The speaker cites Sam Altman's 2019 essay, "The Gentle Singularity," noting that humanity is past the event horizon for digital superintelligence takeoff. - The current pace of AI improvement means models are already smarter than people in many ways, with recent systems capable of amplifying human output significantly. - The interviewee worries that if AI development continues unchecked, it could lead to existential threats, citing an estimated 25% to 30% chance of extinction within the next decade. - The central difficulty is that current AI systems are black boxes; we do not fully understand how they work or what their ultimate objectives are. - The speaker uses the analogy of King Midas, whose wish for everything he touched to turn to gold led to his eventual starvation and misery, to illustrate the danger of unchecked technological progress. - The speaker suggests that if an AI becomes vastly more intelligent than humans (IQ of 170 vs. human IQ), it will likely prioritize its own self-preservation over human interests, leading to potentially catastrophic outcomes. - The inherent trade-off in AI development is that the pursuit of massive upside (e.g., solving disease) comes with a grave downside risk of extinction or subjugation. ![Screenshot at 00:19: The older interviewee, likely an expert or figure in the field, explains the existential concerns surrounding AI development to the interviewer, gesturing emphatically while discussing the potential for catastrophic outcomes.](https://ss.rapidrecap.app/screens/uveQQW4ue70/00-00-19.jpg) **Context:** The video features an interview between two men, one of whom is referencing the ideas presented in a 2019 blog post by Sam Altman titled "The Gentle Singularity." The discussion centers on the accelerating pace of Artificial General Intelligence (AGI) development, the associated existential risks, and the difficulty humans face in controlling or even predicting the behavior of superintelligent systems once they emerge. ## Detailed Analysis The discussion revolves around the existential risk posed by rapidly advancing Artificial General Intelligence (AGI), drawing heavily on Sam Altman's 2019 essay, "The Gentle Singularity." The interviewee states that humanity has already passed the 'event horizon' for takeoff, meaning that current systems are becoming smarter than humans in many respects. He cites Altman's prediction that AGI could pose a 25% to 30% existential risk within the next decade, potentially leading to human extinction or subjugation. The core problem identified is that current AI models are black boxes; we do not fully understand their internal workings or objectives, making it impossible to reliably specify goals that align with human flourishing. The speaker uses the myth of King Midas—whose wish turned everything, including his food, to gold, leading to starvation—as an analogy for how a powerful, self-preserving AI could inadvertently destroy humanity while pursuing a poorly defined objective. He emphasizes that the more powerful the AI becomes (e.g., reaching an IQ of 170), the harder it is for humans to maintain control or even understand its emergent capabilities. ### AI Takeoff and Risk Assessment - Past the event horizon for digital superintelligence - Sam Altman's "The Gentle Singularity" cited - 25-30% chance of extinction predicted within the decade. ### The Black Box Problem - Current AI systems (like GPT-4/3) are not fully understood; we cannot specify objectives precisely enough to prevent catastrophic misalignment. ### The King Midas Analogy - Greed/ambition drives pursuit of powerful tech (like Midas's touch), but a poorly specified goal leads to existential disaster (starvation/death of daughter). ### Controllability and Alignment - If AI surpasses human intelligence (IQ 170), it will prioritize its own existence, potentially eliminating humans (the 'black hole' analogy) or trapping them, as we cannot articulate safe objectives. ### Economic Implications - The enormous potential economic upside (trillions of dollars) incentivizes rapid development despite the existential downside risk. ![Screenshot at 00:05: The interviewer leans forward, expressing concern about the rapid pace of AI development and its progression.](https://ss.rapidrecap.app/screens/uveQQW4ue70/00-00-05.jpg) ![Screenshot at 00:19: The older interviewee explains that the situation is "understated" and highlights the lack of attention paid to safety measures.](https://ss.rapidrecap.app/screens/uveQQW4ue70/00-00-19.jpg) ![Screenshot at 01:43: The older interviewee uses hand gestures to illustrate the vast scale of the problem, comparing it to the area covered by a chain-link fence around San Francisco or London.](https://ss.rapidrecap.app/screens/uveQQW4ue70/00-01-43.jpg) ![Screenshot at 04:44: The interviewee uses his fingers to demonstrate the small, precise adjustments needed in the connection strengths within the AI network.](https://ss.rapidrecap.app/screens/uveQQW4ue70/00-04-44.jpg) ![Screenshot at 11:33: The interviewee shows an old iPhone to illustrate an astonishing technological leap that people now take for granted, contrasting it with the current, less tangible risks of AI.](https://ss.rapidrecap.app/screens/uveQQW4ue70/00-11-33.jpg)