7: Deep Learning for Natural Language – Transformers

Quick Overview

The Transformer architecture, originally designed for machine translation, has become the dominant deep neural network architecture across a vast array of domains, including search, speech recognition, computer vision, and generative AI, because it elegantly addresses the needs for incorporating context, respecting word order, and maintaining input/output length consistency.

Key Points: Transformers have "taken over everything," proving incredibly effective in domains beyond their origin in machine translation, such as Google Search, speech recognition, and even computer vision. The core problem motivating the discussion is word-to-slot multi-class classification, exemplified by converting natural language queries like "Find me all flights from Boston to LaGuardia tomorrow morning" into structured queries like SQL. The foundational mechanism introduced to handle context is "self-attention," which calculates contextual embeddings for each word by taking a weighted average of all standalone embeddings, where weights are determined by the exponentiated dot product of the word embeddings (cosine similarity), followed by a softmax normalization. To account for word order, which self-attention alone ignores because dot products function on sets, the architecture incorporates "positional encoding" by adding a learned embedding representing the word's position to its standalone embedding. Multi-head attention enhances performance by employing multiple self-attention heads, allowing the model to attend to multiple patterns simultaneously, analogous to filters in a convolutional network. The standard Transformer block includes multiple self-attention heads, concatenation of their outputs, a projection layer to restore original dimensionality, and a subsequent feed-forward network involving a ReLU activation for non-linearity.

Context: The lecture introduces the Transformer architecture, highlighting its ubiquitous adoption in modern AI despite its initial design for machine translation tasks. The motivating use case centers on information retrieval, specifically parsing natural language queries (like flight searches) to extract entities and convert them into structured queries (like SQL), which requires accurately tagging every word in the input sentence with specific semantic slots.