# 9: Generative AI – Large Language Models (LLMs) and Retrieval Augmented Generation (RAG) Source: https://www.youtube.com/watch?v=KGDe1QvfKJ8 Recap page: https://rapidrecap.app/video/KGDe1QvfKJ8 Generated: 2026-01-07T15:40:07.209+00:00 --- ## Quick Overview The lecture contrasts BERT's masking self-supervised learning technique with next word prediction, which powers autoregressive models like GPT, requiring a crucial architectural modification called causal self-attention to prevent the model from 'peeking into the future' during training. **Key Points:** - Next word prediction is a form of self-supervised learning where the model predicts the last word of a fragment, or sequentially predicts the next word based only on preceding text. - Using the standard Transformer encoder (like BERT) for next word prediction is problematic because self-attention allows the last word to see all preceding words, including the word it is supposed to predict, which is described as 'cheating' or 'peeking into the future'. - The solution for next word prediction models is implementing 'causal self-attention' (or masked self-attention), which zeros out attention weights for future words, ensuring context only flows from past to present tokens. - The loss function for next word prediction models remains cross-entropy, calculated as the sum or average of cross-entropies for every word position being predicted in the sequence. - Generating text involves decoding—choosing a token from the vocabulary probability distribution (softmax output)—using strategies like greedy decoding (picking the highest probability) for determinism or random sampling for creativity. - LLMs can easily 'go off the rails' if a mistake is sampled from the long tail of low-probability words, as the model cannot easily recover from such errors in subsequent predictions. - Advanced decoding strategies like top-k sampling (selecting only the top K most likely words) and top-p or nucleus sampling (selecting words whose cumulative probability exceeds a threshold P) help focus sampling on the 'head' of the distribution. **Context:** This lecture segment reviews the self-supervised learning technique of masking used in models like BERT and introduces a different, more powerful technique: next word prediction, which is fundamental to autoregressive language models such as the GPT family. The discussion centers on adapting the Transformer architecture to correctly implement next word prediction by preventing the model from accessing future information during sequence generation training. ## Detailed Analysis The speaker first recaps BERT's training via masking, where tokens are randomly replaced and the model fills the blanks. Then, the focus shifts to next word prediction, a specialized form of self-supervised learning where the input sentence is shifted to predict the next token sequentially (e.g., input 'the cat sat on the' predicts 'mat'). When applying the standard Transformer encoder (bidirectional attention) to this task, a critical flaw emerges: the self-attention mechanism allows the model to see future words, enabling it to trivially copy the answer rather than genuinely predict it. To fix this, the architecture must be modified to use 'causal self-attention' (or masked attention), where attention weights for future words are zeroed out, effectively restricting context to only preceding tokens. This modified architecture is termed the Transformer Causal Encoder (TCE), which is the basis for models like GPT, referred to as autoregressive language models because they feed their own predictions back as input for subsequent steps. The generation process, or decoding, involves selecting a token from the final softmax output distribution; simple methods include greedy decoding (always picking the most likely word) for deterministic, factual tasks, or random sampling for creative tasks, which introduces stochasticity. The lecture warns that poor sampling from the 'tail' of the probability distribution can lead to nonsensical outputs, as demonstrated by examples where forcing low-probability words ('masters of chaos', 'knitting socks') derailed GPT-3.5's coherence, highlighting that LLMs provide probabilistically plausible responses, not guaranteed factual accuracy. To tame randomness, techniques like top-k sampling (limiting choices to the top K words) and top-p/nucleus sampling (selecting words until a cumulative probability P is reached) are employed to focus selection on the high-probability 'head' of the distribution. ### Self-Supervised Learning Comparison - Masking in BERT reviewed - Next Word Prediction introduced as a powerful alternative - Next word prediction is a special case of masking focusing only on the last word. ### Transformer Modification for LLMs - Standard Transformer encoder allows 'peeking into the future' via bidirectional attention - Causal self-attention, or masked attention, fixes this by zeroing weights for future words - The resulting architecture is the Transformer Causal Encoder (TCE), similar to a decoder structure. ### Training and Loss Function - Training uses sequential input fragments to predict the next word - The loss function remains the sum or average of cross-entropy losses across all predicted positions. ### Text Generation (Decoding) - The process of selecting a token from the 50,000-way softmax distribution is called decoding - Greedy decoding selects the highest probability word for determinism, suitable for reasoning tasks. ### Sampling Strategies for Creativity - Random sampling provides diversity for creative tasks but sacrifices determinism and risks errors from the probability 'tail' - The model uses 'subtractive sculpting' where each word choice narrows the cone of uncertainty. ### Advanced Decoding Techniques - Top-k sampling limits choices to the K highest probability words - Top-p (nucleus) sampling selects words until their cumulative probability reaches a threshold P, adapting dynamically to the distribution shape.