# 6: Deep Learning for Natural Language – Embeddings Source: https://www.youtube.com/watch?v=LqFc0z-pQTg Recap page: https://rapidrecap.app/video/LqFc0z-pQTg Generated: 2026-01-07T16:01:15.74+00:00 --- ## Quick Overview The lecture introduces standalone word embeddings as a solution to the shortcomings of one-hot encoding, detailing that these embeddings aim to capture semantic relationships geometrically, using the GloVe method based on word-word co-occurrence matrices derived from text like Wikipedia, which are then optimized using stochastic gradient descent to approximate the matrix. **Key Points:** - One-hot vectors suffer from the computational issue of creating vectors as long as the vocabulary (e.g., 500,000 words) and the conceptual problem that all word pairs have the same Euclidean distance (root 2), failing to represent semantic similarity. - The goal of embeddings is to represent words such that geometric relationships (distance and direction) reflect semantic relationships, as illustrated by plotting factory/home/building clusters or the puppy:dog::calf:cow analogy. - The fundamental insight for learning embeddings is John Firth's quote: "You shall know a word by the company it keeps," leading to the construction of a word-word co-occurrence matrix (like GloVe) based on words appearing in the same sentence. - The GloVe approach models the logarithm of the co-occurrence count (Xij) as the sum of the bias of word i (bi), the bias of word j (bj), and the dot product of their embedding vectors (wi · wj), which is minimized using gradient descent. - The learned embeddings are dense, low-dimensional vectors whose quality is assessed by how well they can recreate the original co-occurrence matrix, capturing patterns like 'brother minus man plus woman equals sister'. - A major remaining issue for standalone embeddings is context, as words like 'bank' have multiple meanings, resulting in an 'average version' of the meaning, necessitating contextual embeddings (using transformers) later. - In Keras, the workflow shifts from using one-hot vectors to using an embedding layer after the text vectorization layer outputs integers, requiring truncation and padding to normalize sentence lengths. **Context:** This lecture segment continues a series on Natural Language Processing, focusing specifically on moving beyond basic representations like one-hot embeddings to learn 'standalone embeddings' (uncontextualized word vectors). The discussion contrasts the flaws of traditional encoding methods—specifically their inability to capture semantic similarity and their high dimensionality—with the desired properties of learned vectors that map meaning to geometric space, setting the stage for the introduction of contextual embeddings via the Transformer architecture in subsequent lessons. ## Detailed Analysis The lecture first outlines the severe shortcomings of one-hot encoding: computational inefficiency due to vocabulary size and the lack of semantic meaning, evidenced by the constant Euclidean distance (root 2) between any two distinct word vectors, meaning synonyms like 'great' and 'awesome' are represented identically in terms of distance as antonyms like 'good' and 'bad'. To solve this, the class explores standalone embeddings, which aim to map semantic relationships geometrically; for example, the vector difference between 'puppy' and 'dog' should correspond to the difference between 'calf' and 'cow'. The method detailed for learning these is GloVe, which relies on the principle that related words share contexts, defined here as co-occurrence within the same sentence across a large corpus like Wikipedia. This results in a word-word co-occurrence matrix (Xij) where each cell records how often two words appear together. The learning task is framed as a regression problem where the model attempts to predict log(Xij + 1) using the formula: bi + bj + wi · wj, minimizing the mean squared error between prediction and actual log counts using stochastic gradient descent to find the embedding vectors (wi) and biases (bi). Although this process effectively captures semantic structure, as shown by plotting and vector algebra (e.g., King - Man + Woman = Queen), it averages out polysemous words like 'bank'. Pre-trained embeddings like GloVe are advantageous when data is scarce, though they may fail to capture domain-specific jargon, necessitating fine-tuning. Finally, the implementation in Keras involves using the text vectorization layer to output integers (stopping before one-hot encoding), padding/truncating sequences to uniform length, and feeding the integers into a learned Embedding layer, followed by GlobalAveragePooling1D to convert the sequence of vectors into a single input vector for downstream models. ### One-Hot Vector Limitations - Computational waste due to vocabulary-sized vectors - Semantic failure as all word distances equal root 2 - Inability to represent synonym similarity ### Embedding Goals and Semantics - Vectors must cluster synonyms closely - Geometric relationships (distance and direction) must mirror semantic relationships (e.g., puppy:dog::calf:cow) ### GloVe Methodology - Learns from the principle "You shall know a word by the company it keeps" - Constructs a word-word co-occurrence matrix based on sentence-level co-occurrence counts ### The GloVe Model Equation - Predicted log co-occurrence is modeled as bi + bj + wi · wj - Errors are minimized using stochastic gradient descent on the mean squared error ### Embedding Characteristics and Evaluation - Embeddings are dense, low-dimensional, and learned from data - Quality is validated by the ability to recreate the co-occurrence matrix faithfully, enabling vector algebra ### Contextual Gap - Standalone embeddings fail on polysemous words like 'bank' by providing only an average meaning - Contextual embeddings, built upon transformers, are required to resolve ambiguity ### Keras Implementation Flow - Text vectorization layer outputs integers (STI only) - Sentences are padded/truncated to max length - Integers map to vectors via the learned Embedding layer - Sequence of vectors is reduced via GlobalAveragePooling1D