10: Generative AI – Adapting LLMs with Parameter-Efficient Fine-Tuning

Quick Overview

The lecture details the evolution from base LLMs like GPT-3, which only predict the next word, to instruction-following models like ChatGPT (InstructGPT/GPT-3.5) through a two-step process called Instruction Tuning: Supervised Fine-Tuning (SFT) and Reinforcement Learning with Human Feedback (RLHF), followed by an introduction to adaptation techniques like Retrieval-Augmented Generation (RAG) for proprietary data use.

Key Points: GPT-3, unlike BERT, uses causal language modeling trained via next-word prediction, where the final dense layer output vector length equals the vocabulary size before softmax. The key difference between GPT-2 and GPT-3 was scale; GPT-3's greater scale led to emergent capabilities, such as strikingly coherent continuations of any starting prompt. Instruction Tuning involves two main steps: Supervised Fine-Tuning (SFT) using human-written question-answer pairs (initially 12,500 pairs) and then utilizing human preference rankings (thumbs up/down) to train a Reward Model. The transition from GPT-3 to InstructGPT (GPT-3.5) uses RLHF, where the Reward Model's numerical rating nudges GPT-3's weights via reinforcement learning to favor better-ranked answers. ChatGPT evolved from InstructGPT by training on conversational data (instruction-answer sequences strung into a conversation) instead of single instruction-answer pairs. Adaptation techniques include zero-shot prompting, few-shot prompting (in-context learning, an emergent GPT-3 capability), and Retrieval-Augmented Generation (RAG), which addresses context window limitations by retrieving relevant proprietary data chunks. RAG involves calculating contextual embeddings for external documents, finding the most similar chunks to a new query, and packing only those relevant chunks (context) into the prompt to avoid context window overflow and reduce hallucination, as demonstrated when GPT-3.5 incorrectly answered a 2022 Olympics question due to its 2021 cutoff date.

Context: The lecture, presented by Rama Ramakrishnan, continues a series on Large Language Models (LLMs), specifically focusing on how foundational models like GPT-3 are adapted to become useful instruction-following tools like ChatGPT. The discussion covers the architectural differences between causal and masked language models, the scaling laws that led to GPT-3's emergent behavior, and the subsequent fine-tuning methodologies, particularly Instruction Tuning and RLHF, necessary to align the model with user intent.