5: Deep Learning for Natural Language – The Basics

Quick Overview

The lecture introduces the Natural Language Processing (NLP) sequence, starting with text vectorization and the Bag of Words (BoW) model as fundamental preprocessing steps before moving on to embeddings and Transformers, emphasizing that text's primacy in human knowledge necessitates AI systems that can read and understand it.

Key Points: The NLP sequence roadmap includes vectorization, Bag of Words model, embeddings (the core atomic unit of modern NLP), Transformers (two lectures), and finally LLMs (lectures 9 and 10). Text is fundamentally important because human knowledge, the internet (historically), human communication, and cultural production are heavily text-based, leading to the possibility of AI systems that can 'understand' all this text. NLP is already in action, with Google autocomplete reportedly saving 200 years of typing time daily, which significantly impacted mobile usability and e-commerce. Standard text vectorization involves the S-T-I-E process: Standardization (lowercase, remove punctuation/stop words/stemming), Tokenization (splitting on whitespace), Indexing (mapping tokens to unique integers), and Encoding (mapping integers to one-hot vectors). The Bag of Words (BoW) model aggregates token representations (using sum or multi-hot encoding) into a single vector, losing sequential word order information, which is a major limitation for text generation but often sufficient for classification. In the BoW model, the vocabulary size determines the vector length; unseen words in new input are mapped to a special UNK (unknown) token, which is index 0. For classification tasks like predicting song genre (hip hop, rock, pop), the simplest neural network uses the Bag of Words input vector, one hidden layer with ReLUs, and a three-node softmax output layer because the max function is not friendly for differentiation during backpropagation.

Context: Rama Ramakrishnan begins the natural language processing sequence of the class, outlining a curriculum that progresses from basic concepts like vectorization and the Bag of Words model to advanced topics like embeddings and Transformers, culminating in a focus on Large Language Models (LLMs). The instructor emphasizes the critical importance of text in human society and the potential for AI systems to process this vast textual data, citing current real-world applications like Google autocomplete as evidence of NLP's impact.