# Machine Learning Street Talk: "I Co-Invented the Transformer. Now I'm Replacing It." Source: https://www.youtube.com/watch?v=-Nm-9YMPG0k Recap page: https://rapidrecap.app/video/-Nm-9YMPG0k Generated: 2025-11-29T16:04:29.056+00:00 --- ## Quick Overview The co-inventor of the Transformer architecture argues that the current era of AI is seeing a shift away from the purely data-driven, brute-force scaling of Transformers toward models that incorporate more human-like reasoning, exemplified by the Continuous Thought Machine (CTM) architecture which uses path-dependent sequential reasoning. **Key Points:** - The speaker co-invented the Transformer architecture but believes its current success is becoming a 'trap' due to over-reliance on scaling and brute force computation. - The CTM (Continuous Thought Machine) architecture represents a crucial inflection point in modern AI, moving beyond the limitations of current LLMs. - The CTM approach mimics human reasoning by factoring in relationships and timing between sequential activations, contrasting with the instantaneous snapshot approach of Transformers. - Standard LLMs, like those based on Transformers, often achieve only around 15% accuracy on the most simple tasks when compared to the CTM's ability to solve complex reasoning problems. - The CTM's internal logic, which mimics human thought processes, allows it to solve problems like Sudoku by tracing a path, unlike Transformers which might just guess the final output. - A key difference is that CTMs seem to exhibit psychological behavior, such as overconfidence when wrong or backtracking when making mistakes, which suggests deeper reasoning. - The ultimate goal of this new architecture is to create function approximators that represent things the way a human does, leading to more profound discoveries. ![Screenshot at 00:18: The speaker discusses the crucial inflection point in modern AI, contrasting the current era's reliance on scaling with the need for architectures that incorporate more human-like reasoning, exemplified by the Transformer's co-inventor pointing out its current limitations.](https://ss.rapidrecap.app/screens/-Nm-9YMPG0k/00-00-18.png) **Context:** The video features a discussion, likely an interview segment from the 'AI Podcast Daily' given the opening, where a key figure in AI development, who claims to have co-invented the Transformer architecture, critiques the current paradigm dominated by large Transformer models. He introduces an alternative architecture, the Continuous Thought Machine (CTM), which he argues represents a necessary shift away from pure scaling toward more human-like, sequential reasoning capabilities. ## Detailed Analysis The speaker, a co-inventor of the Transformer, asserts that the current AI landscape, heavily reliant on large language models (LLMs) built on the Transformer architecture, is hitting a crucial inflection point, describing the current paradigm as a 'trap' due to its over-reliance on brute-force scaling and massive resources. He contrasts this with the Continuous Thought Machine (CTM) architecture, which he champions as the future. The CTM is designed to mimic human reasoning by processing information sequentially, maintaining an internal state, and reasoning over time, rather than relying on a single vector snapshot like Transformers. The speaker claims that while Transformers struggle with complex reasoning tasks (like Sudoku), only achieving around 15% accuracy on simple tasks in some benchmarks, the CTM can solve them via a sequential path that mimics human critical thinking. Furthermore, CTMs exhibit behaviors like overconfidence when wrong or backtracking after errors, which suggests a deeper, more human-like internal logic compared to standard LLMs. The CTM team's goal is to create function approximators that represent concepts the way humans do, promising a leap beyond the current limitations imposed by the brute-force training methods. ### The Current AI Landscape - Tracking a critical architectural shift in modern AI - The current era feels like a local minimum due to reliance on scaling - Researchers are defining the current era by the Transformer architecture ### The Continuous Thought Machine (CTM) - CTM is the new direction leading us forward - CTM is a novel architecture that challenges the status quo - It mimics human reasoning through sequential thought ### CTM vs. Transformer Mechanics - CTM uses sequential reasoning based on internal state and past activations - Transformers use instantaneous vector snapshots - CTM's sequential path dependence allows for true reasoning, unlike brute-force scaling ### CTM Performance and Implications - CTM achieves far greater accuracy (e.g., 99.99% vs 15% on simple tasks) - CTM exhibits human-like behavior like backtracking and calibrated confidence - The goal is to create agents that reason from first principles ![Screenshot at 00:00: Video title card promoting membership overlaid on an oscilloscope graphic.](https://ss.rapidrecap.app/screens/-Nm-9YMPG0k/00-00-00.png) ![Screenshot at 00:08: The speaker introduces the current era as being defined by the Transformer architecture, which he suggests is reaching a limit.](https://ss.rapidrecap.app/screens/-Nm-9YMPG0k/00-00-08.png) ![Screenshot at 00:22: The speaker explicitly names the co-inventors of the Transformer architecture, contrasting it with the CTM.](https://ss.rapidrecap.app/screens/-Nm-9YMPG0k/00-00-22.png) ![Screenshot at 01:44: Visual representation of the massive computational resources \(thousands of hours, thousands of experts\) poured into refining existing Transformer models.](https://ss.rapidrecap.app/screens/-Nm-9YMPG0k/00-01-44.png) ![Screenshot at 05:05: The speaker defines the first big novelty as the internal thought dimension in CTM, contrasting it with the instantaneous snapshot of a Transformer.](https://ss.rapidrecap.app/screens/-Nm-9YMPG0k/00-05-05.png)