# AI beyond language and vision | Paul Liang | TEDxMIT

Source: https://www.youtube.com/watch?v=nnu-gMJh21A
Recap page: https://rapidrecap.app/video/nnu-gMJh21A
Generated: 2026-01-14T20:02:02.494+00:00

---
## Quick Overview

The future of Artificial Intelligence (AI) involves moving beyond language and vision processing to incorporate other human senses, particularly touch (haptics) and smell/taste, to achieve a more holistic, augmented intelligence that can better understand and interact with the world, which requires interdisciplinary research combining AI with biology and neuroscience.

**Key Points:**
- The next major frontier for AI development is moving beyond language and vision to incorporate smell/taste and touch (haptics).
- Current AI excels at visual and language processing but lags significantly in perceiving smell/taste, which humans use to understand the past.
- The speaker's group is working on multi-modal AI that integrates language, vision, audio, and potentially smell/touch.
- Haptic intuition, or the ability to feel textures and manipulate objects, is a key area of research, exemplified by developing haptic gloves.
- The speaker suggests that AI systems with memory and reasoning capabilities, augmented by these other senses, will be crucial for making complex decisions about health and well-being.
- The speaker notes that in 2018, deep learning was primarily focused on computer vision and language models, but the field is now pivoting toward multi-modal AI.
- The goal is to create AI that can perceive and interact with the world across all human senses, not just the dominant visual/language ones.

![Screenshot at 00:05: The interviewer introduces the segment, setting the stage for a discussion about expanding AI capabilities beyond current language and vision models.](https://ss.rapidrecap.app/screens/nnu-gMJh21A/00-00-05.jpg)

**Context:** This video captures a segment from a TEDxMIT event featuring a conversation between an interviewer (John) and a speaker named Paul Liang, who is introduced as a computer scientist and innovator from MIT CSAIL. The discussion centers on the future trajectory of Artificial Intelligence, arguing that current AI development, heavily focused on language and vision, needs to expand its scope to include other human senses like smell, taste, and touch (haptics) to achieve a more comprehensive, 'multi-sensory' or 'augmented' intelligence.

## Detailed Analysis

The discussion explores the limitations of current AI, which primarily focuses on language and vision, contrasting this with the richness of human perception that involves five primary senses, plus intuition. Paul Liang argues that AI needs to move toward multi-modal capabilities, incorporating smell, taste, and touch to create a more robust intelligence capable of understanding context and making complex decisions (e.g., regarding health). He points out that while AI has made massive strides in language and vision since 2018, senses like smell and touch remain far behind, noting that smell is the only modality that allows us to perceive the past. The speaker mentions ongoing research into haptics—building gloves that allow AI to feel textures and grasp objects—and the challenge of encoding senses like smell into AI systems. He concludes that the next great frontier involves integrating these underrepresented senses to augment human decision-making rather than simply overriding it, suggesting that the interdisciplinary nature of this research, combining AI with biology and neuroscience, is crucial for this advancement.

### AI Limitations and Future Scope

- AI is currently seduced by images (vision) and language, neglecting other senses like smell and touch (4:06)
- Smell is the only modality that allows us to see the past (3:58)
- The next frontier is multi-sensory intelligence (3:16)
- Current AI is far from perceiving all senses like humans (2:53)

### Research Focus and Interdisciplinary Needs

- Paul's group works on multi-modal AI, incorporating language, vision, audio, and touch (3:20)
- The challenge is building AI that can perceive and interact with the world like humans across all senses (3:22)
- Tactile sensing (haptics) is an area of intense research, involving gloves for grasping and manipulating objects (10:20, 10:31)
- Success requires expertise in biology, chemistry, and neuroscience, not just computer science (14:02)

### The Role of Intuition and Memory

- Intuition (or proprioception) is hard for AI to replicate (11:01)
- AI needs long-term memory, not just short-term context, to augment human decisions (11:37)
- The goal is augmentation, not overriding human input (13:10)

![Screenshot at 00:01: The opening title slide for the TEDxMIT event, noting its independent organization.](https://ss.rapidrecap.app/screens/nnu-gMJh21A/00-00-01.jpg)
![Screenshot at 00:15: A wide shot of the stage setup showing the two speakers, Paul Liang \(left\) and John \(right\), seated on chairs on a red circular rug.](https://ss.rapidrecap.app/screens/nnu-gMJh21A/00-00-15.jpg)
![Screenshot at 01:17: Close-up on Paul Liang speaking while gesturing with his hands, emphasizing the need for AI to incorporate more than just visual input.](https://ss.rapidrecap.app/screens/nnu-gMJh21A/00-01-17.jpg)
![Screenshot at 03:33: Close-up on Paul Liang as he discusses the importance of the sense of smell for perceiving the past, illustrating the gap between human and current AI senses.](https://ss.rapidrecap.app/screens/nnu-gMJh21A/00-03-33.jpg)
![Screenshot at 15:58: The conclusion of the interview segment, showing the two speakers shaking hands as the audience applauds.](https://ss.rapidrecap.app/screens/nnu-gMJh21A/00-15-58.jpg)