# 10: Generative AI – Adapting LLMs with Parameter-Efficient Fine-Tuning Source: https://www.youtube.com/watch?v=d-tngNnaG4U Recap page: https://rapidrecap.app/video/d-tngNnaG4U Generated: 2026-01-07T15:40:00.883+00:00 --- ## Quick Overview The lecture details the evolution from base LLMs like GPT-3, which only predict the next word, to instruction-following models like ChatGPT (InstructGPT/GPT-3.5) through a two-step process called Instruction Tuning: Supervised Fine-Tuning (SFT) and Reinforcement Learning with Human Feedback (RLHF), followed by an introduction to adaptation techniques like Retrieval-Augmented Generation (RAG) for proprietary data use. **Key Points:** - GPT-3, unlike BERT, uses causal language modeling trained via next-word prediction, where the final dense layer output vector length equals the vocabulary size before softmax. - The key difference between GPT-2 and GPT-3 was scale; GPT-3's greater scale led to emergent capabilities, such as strikingly coherent continuations of any starting prompt. - Instruction Tuning involves two main steps: Supervised Fine-Tuning (SFT) using human-written question-answer pairs (initially 12,500 pairs) and then utilizing human preference rankings (thumbs up/down) to train a Reward Model. - The transition from GPT-3 to InstructGPT (GPT-3.5) uses RLHF, where the Reward Model's numerical rating nudges GPT-3's weights via reinforcement learning to favor better-ranked answers. - ChatGPT evolved from InstructGPT by training on conversational data (instruction-answer sequences strung into a conversation) instead of single instruction-answer pairs. - Adaptation techniques include zero-shot prompting, few-shot prompting (in-context learning, an emergent GPT-3 capability), and Retrieval-Augmented Generation (RAG), which addresses context window limitations by retrieving relevant proprietary data chunks. - RAG involves calculating contextual embeddings for external documents, finding the most similar chunks to a new query, and packing only those relevant chunks (context) into the prompt to avoid context window overflow and reduce hallucination, as demonstrated when GPT-3.5 incorrectly answered a 2022 Olympics question due to its 2021 cutoff date. **Context:** The lecture, presented by Rama Ramakrishnan, continues a series on Large Language Models (LLMs), specifically focusing on how foundational models like GPT-3 are adapted to become useful instruction-following tools like ChatGPT. The discussion covers the architectural differences between causal and masked language models, the scaling laws that led to GPT-3's emergent behavior, and the subsequent fine-tuning methodologies, particularly Instruction Tuning and RLHF, necessary to align the model with user intent. ## Detailed Analysis The speaker first clarified that causal LLMs like GPT-3 use linear activations followed by softmax on a vocabulary-length vector, distinguishing them from models like BERT. GPT-3's superiority over GPT-2 stemmed purely from scale in both network size and training data, leading to emergent abilities like extended coherence. To address GPT-3's inability to follow instructions (often responding with lists or irrelevant completions because it was only trained to predict the next word), OpenAI introduced Instruction Tuning. This process began with Supervised Fine-Tuning (SFT) using 12,500 human-created question-answer pairs. The next, more scalable step involved collecting user queries, generating multiple GPT-3 responses, and having humans rank them (the basis for the thumbs up/down feedback seen in ChatGPT). This ranking data trains a Reward Model to assign numerical scores, which is then used via Reinforcement Learning with Human Feedback (RLHF) to nudge GPT-3's weights, resulting in InstructGPT (GPT-3.5). ChatGPT specifically resulted from training GPT-3.5 on conversational data instead of single instructions. Finally, the lecture transitioned to adapting LLMs for domain-specific tasks, detailing zero-shot and few-shot prompting (in-context learning), and emphasizing Retrieval-Augmented Generation (RAG) as the most prevalent business application. RAG circumvents context window limits by embedding proprietary documents, retrieving the most similar chunks to a query, and feeding only those chunks as context to the LLM, which proves effective in preventing hallucinations based on outdated training data, such as when GPT-3.5 failed to know 2022 Olympic results. ### LLM Architecture Clarification - GPT-3 uses causal encoder with linear activations to softmax layer matching vocabulary size - BERT uses masked language modeling - GPT-3 scale led to emergent coherent continuation capabilities ### Instruction Tuning Process - Step 1 is Supervised Fine-Tuning (SFT) using human-written Q&A pairs - Step 2 involves creating a Reward Model based on human rankings (thumbs up/down) of multiple generated answers ### RLHF and ChatGPT Creation - InstructGPT (GPT-3.5) results from using the Reward Model to nudge weights via reinforcement learning (RLHF) - ChatGPT results from training on conversational sequences rather than single instruction-answer pairs ### Prompt Engineering Basics - Breaking down complex requests into sequential steps like 'list all words, then state the fifth word' improves accuracy - System prompts in ChatGPT often contain instructions like 'think step by step' ### In-Context Learning - The ability of LLMs to learn from zero-shot or few-shot examples provided directly in the prompt was an unanticipated emergent capability in GPT-3 ### Retrieval-Augmented Generation (RAG) - RAG addresses context window limits by retrieving relevant document chunks based on embedding similarity to the query - This technique allows LLMs to leverage proprietary, up-to-date data without full retraining ### RAG Demonstration and Hallucination - GPT-3.5-turbo (cutoff 2021) confidently hallucinated the 2022 Women's Curling Gold Medal winner (stating South Korea instead of Great Britain) - Providing the relevant Wikipedia article as context via RAG allowed the model to answer correctly.