# 8: Deep Learning for Natural Language – Transformers, Self-Supervised Learning Source: https://www.youtube.com/watch?v=v-lHsawHyaI Recap page: https://rapidrecap.app/video/v-lHsawHyaI Generated: 2026-01-07T15:40:02.32+00:00 --- ## Quick Overview The transformer stack achieves its efficiency and power by representing the self-attention mechanism compactly using matrix operations, specifically $softmax(QK^T)V$, and enhances its modeling capacity by introducing three independent, learnable matrices ($A_K, A_Q, A_V$) for Key, Query, and Value projections, alongside incorporating residual connections and layer normalization within each encoder block. **Key Points:** - The transformer architecture meets requirements for generating output of the same length as the input, accounting for context, and considering order. - The basic self-attention operation is efficiently packaged into a matrix formula: $X$ comes in, followed by $X$ transpose, matrix multiplication ($X X^T$), softmax, and then multiplication by the third copy ($X$), enabling fast GPU computation. - Tunable self-attention injects three learnable matrices ($A_K, A_Q, A_V$) to transform the input $X$ into $K, Q, V$ representations, allowing the layer to learn patterns, with the full operation being $softmax(Q K^T)V$. - The full transformer block incorporates a residual connection, adding the input to the self-attention output before passing it through a layer normalization step to ensure well-behaved numerical ranges and improve gradient flow. - For sentence classification tasks, a better method than averaging contextual embeddings is adding an artificial token, designated as CLS, at the beginning of the input sequence, whose final contextual embedding captures the sentence's meaning. - The effectiveness of deep networks relies on learning representations automatically; these representations capture general knowledge about the input data, enabling transfer learning to unrelated problems. - The number of parameters in a transformer depends on the embedding dimension and vocabulary size, not the length of the input sentence, allowing for large context windows like the 1 million token window in Gemini 1.5 Pro. **Context:** This lecture segment, part two of 'Deep Learning for Natural Language – Transformers,' focuses on a deeper, more demanding dive into the mechanics of the transformer stack, specifically detailing the transformation from the basic self-attention concept to the full, industrial-strength transformer encoder block. The discussion covers the mathematical structure enabling GPU efficiency, the introduction of learnable parameters to make attention tunable, and essential stabilizing techniques like residual connections and layer normalization. ## Detailed Analysis The lecture begins by reiterating that transformers are desired because they maintain input/output length parity, consider context, and account for order. The initial flow involves adding positional embeddings to standalone word embeddings, feeding this into the encoder stack to produce contextual embeddings, which are then used for tasks like word-to-slot classification, achieving 99% accuracy with just one block on that specific task, though care must be taken regarding majority classes ('O' slots). The core innovation discussed is the matrix representation of self-attention: the entire operation of calculating pairwise dot products, softmaxing, and weighted averaging is condensed into the matrix multiplication sequence: $X$ (input) times $X^T$, followed by softmax, and then multiplied by $X$ again, which is why transformers are fast on GPUs. To make this layer tunable, three independent, learnable matrices ($A_K, A_Q, A_V$) are introduced to transform the input $X$ into $K, Q, V$ vectors, leading to the standard formula $softmax(Q K^T)V$. The next essential complications added are residual connections, where the input to a sub-layer is added element-wise to its output to dramatically improve gradient flow (as seen in ResNet), and layer normalization, which standardizes the resulting vectors (calculating mean/standard deviation across embedding dimensions) to keep numbers in a small range, mitigating exploding/vanishing gradient issues. The final structure of a transformer block includes the multi-head attention layer (now with tunable heads), followed by a residual connection and layer normalization, repeated for the subsequent feedforward layer. The lecture also explains how to use the encoder for classification by prepending a trainable CLS token whose final contextual embedding summarizes the sentence, avoiding the information loss associated with simple averaging. Finally, the concept of self-supervised learning is introduced as a method to leverage vast amounts of unlabelled text data to learn general, reusable representations, similar to how pre-trained vision models capture general features. ### Transformer Encoder Basics - Desired properties include same length input/output, context awareness, and order awareness - Initial step involves summing standalone and positional embeddings - Output is contextual embeddings used for classification. ### Efficient Self-Attention Formulation - Self-attention is vectorized by performing $X$ times $X^T$ (matrix multiplication) in one shot to calculate all pairwise dot products - The full operation is compactly represented as $softmax(Q K^T)V$, where $Q, K, V$ are derived from $X$. ### Introducing Tunability - Three independent learnable matrices ($A_K, A_Q, A_V$) transform $X$ into $K, Q, V$ projections to inject parameters into the self-attention layer - Each of the multiple attention heads receives its own set of these three matrices, leading to different outputs per head. ### Stabilizing Techniques - Residual connections add the input vector element-wise to the sub-layer output to improve backpropagation gradient flow - Layer normalization standardizes outputs by subtracting the mean and dividing by the standard deviation across embedding dimensions to maintain small numerical ranges. ### Transformer Block Stacking and Optimization - A single block is stackable vertically because input and output dimensions match, leading to models like GPT-3 having 96 blocks - All parameters, including $A_K, A_Q, A_V$ matrices, layer norm parameters, and embedding weights, are optimized simultaneously via backpropagation. ### Text Classification Strategy - Instead of averaging contextual embeddings for sentence classification, a learnable CLS token is prepended to the input sequence - The final contextual embedding of this CLS token summarizes the entire sentence information for final classification layers. ### Representation Learning Context - Deep networks learn representations that capture general knowledge about input data, enabling transfer learning to unrelated tasks, which motivates self-supervised learning from massive text corpora.