8: Deep Learning for Natural Language – Transformers, Self-Supervised Learning

Quick Overview

The transformer stack achieves its efficiency and power by representing the self-attention mechanism compactly using matrix operations, specifically $softmax(QK^T)V$, and enhances its modeling capacity by introducing three independent, learnable matrices ($AK, AQ, AV$) for Key, Query, and Value projections, alongside incorporating residual connections and layer normalization within each encoder block.

Key Points: The transformer architecture meets requirements for generating output of the same length as the input, accounting for context, and considering order. The basic self-attention operation is efficiently packaged into a matrix formula: $X$ comes in, followed by $X$ transpose, matrix multiplication ($X X^T$), softmax, and then multiplication by the third copy ($X$), enabling fast GPU computation. Tunable self-attention injects three learnable matrices ($AK, AQ, AV$) to transform the input $X$ into $K, Q, V$ representations, allowing the layer to learn patterns, with the full operation being $softmax(Q K^T)V$. The full transformer block incorporates a residual connection, adding the input to the self-attention output before passing it through a layer normalization step to ensure well-behaved numerical ranges and improve gradient flow. For sentence classification tasks, a better method than averaging contextual embeddings is adding an artificial token, designated as CLS, at the beginning of the input sequence, whose final contextual embedding captures the sentence's meaning. The effectiveness of deep networks relies on learning representations automatically; these representations capture general knowledge about the input data, enabling transfer learning to unrelated problems. The number of parameters in a transformer depends on the embedding dimension and vocabulary size, not the length of the input sentence, allowing for large context windows like the 1 million token window in Gemini 1.5 Pro.

Context: This lecture segment, part two of 'Deep Learning for Natural Language – Transformers,' focuses on a deeper, more demanding dive into the mechanics of the transformer stack, specifically detailing the transformation from the basic self-attention concept to the full, industrial-strength transformer encoder block. The discussion covers the mathematical structure enabling GPU efficiency, the introduction of learnable parameters to make attention tunable, and essential stabilizing techniques like residual connections and layer normalization.