# 7: Deep Learning for Natural Language – Transformers Source: https://www.youtube.com/watch?v=IeF7aATDaw4 Recap page: https://rapidrecap.app/video/IeF7aATDaw4 Generated: 2026-01-07T16:01:24.831+00:00 --- ## Quick Overview The Transformer architecture, originally designed for machine translation, has become the dominant deep neural network architecture across a vast array of domains, including search, speech recognition, computer vision, and generative AI, because it elegantly addresses the needs for incorporating context, respecting word order, and maintaining input/output length consistency. **Key Points:** - Transformers have "taken over everything," proving incredibly effective in domains beyond their origin in machine translation, such as Google Search, speech recognition, and even computer vision. - The core problem motivating the discussion is word-to-slot multi-class classification, exemplified by converting natural language queries like "Find me all flights from Boston to LaGuardia tomorrow morning" into structured queries like SQL. - The foundational mechanism introduced to handle context is "self-attention," which calculates contextual embeddings for each word by taking a weighted average of all standalone embeddings, where weights are determined by the exponentiated dot product of the word embeddings (cosine similarity), followed by a softmax normalization. - To account for word order, which self-attention alone ignores because dot products function on sets, the architecture incorporates "positional encoding" by adding a learned embedding representing the word's position to its standalone embedding. - Multi-head attention enhances performance by employing multiple self-attention heads, allowing the model to attend to multiple patterns simultaneously, analogous to filters in a convolutional network. - The standard Transformer block includes multiple self-attention heads, concatenation of their outputs, a projection layer to restore original dimensionality, and a subsequent feed-forward network involving a ReLU activation for non-linearity. **Context:** The lecture introduces the Transformer architecture, highlighting its ubiquitous adoption in modern AI despite its initial design for machine translation tasks. The motivating use case centers on information retrieval, specifically parsing natural language queries (like flight searches) to extract entities and convert them into structured queries (like SQL), which requires accurately tagging every word in the input sentence with specific semantic slots. ## Detailed Analysis The speaker emphasizes that Transformers are now the default architecture tried first for nearly any problem due to their effectiveness across numerous fields, including generative AI, AlphaFold (protein folding), and computer vision, even where they were not initially designed to operate. The problem setup frames the task as word-to-slot classification, requiring the model to output a tag for every input word, maintaining the same sequence length for input and output. The architecture achieves context awareness through self-attention: it calculates attention weights between the target word's embedding and all other word embeddings (including itself) using the dot product of their standalone vectors (normalized via exponentiation and softmax) to determine relatedness, then computes the new contextual embedding as a weighted average. To solve the issue of word order, which is lost in the set-based dot product calculations, positional encoding is injected by adding a learned embedding specific to each word's position in the sequence to its standalone embedding before processing. Further enhancement comes from multi-head attention, where several attention mechanisms run in parallel to capture diverse linguistic patterns, with their outputs concatenated and projected back to the original dimension via a dense layer. Finally, non-linearity is introduced using a dense layer with a ReLU activation, typically expanding the dimensionality fourfold before projecting back, completing the core Transformer block which preserves sequence length throughout. ### Transformer Ubiquity - Originally for machine translation - Now transforms translation, Google search, speech recognition, text-to-speech, computer vision, and generative AI - Example: AlphaFold runs on a transformer stack. ### Information Retrieval Task Framing - Focuses on converting natural language queries into structured queries (like SQL) - Requires manually tagging every word in the query with one of 123 possible 'slots' (e.g., B-fromloc.city_name, I-arrive_time.period) - The task is modeled as word-to-slot multi-class classification. ### Self-Attention Mechanism - Contextualizes a word's standalone embedding by computing weights based on relatedness to all other words - Relatedness is quantified by the dot product of embeddings, normalized using exponentiation and softmax to ensure weights sum to one. ### Contextual Embedding Calculation - The new contextual embedding is the weighted average of all standalone embeddings, using the calculated attention weights, effectively dragging the word's embedding closer to related context words. ### Handling Word Order - Self-attention alone fails to capture sequence order because dot products operate on sets - Positional encoding solves this by adding a learned embedding corresponding to the word's position to its standalone embedding. ### Multi-Head Attention - Employs multiple self-attention heads to learn different types of attention patterns simultaneously (e.g., grammar, tense, tone) - Outputs from all heads are concatenated and then compressed back to the original dimensionality using a projection dense layer. ### Injecting Non-Linearity - After multi-head attention and projection, the results pass through a dense layer with a ReLU activation to introduce necessary non-linearity, typically widening the layer size four times before projecting back to the original dimension.