6: Deep Learning for Natural Language – Embeddings

Quick Overview

The lecture introduces standalone word embeddings as a solution to the shortcomings of one-hot encoding, detailing that these embeddings aim to capture semantic relationships geometrically, using the GloVe method based on word-word co-occurrence matrices derived from text like Wikipedia, which are then optimized using stochastic gradient descent to approximate the matrix.

Key Points: One-hot vectors suffer from the computational issue of creating vectors as long as the vocabulary (e.g., 500,000 words) and the conceptual problem that all word pairs have the same Euclidean distance (root 2), failing to represent semantic similarity. The goal of embeddings is to represent words such that geometric relationships (distance and direction) reflect semantic relationships, as illustrated by plotting factory/home/building clusters or the puppy:dog::calf:cow analogy. The fundamental insight for learning embeddings is John Firth's quote: "You shall know a word by the company it keeps," leading to the construction of a word-word co-occurrence matrix (like GloVe) based on words appearing in the same sentence. The GloVe approach models the logarithm of the co-occurrence count (Xij) as the sum of the bias of word i (bi), the bias of word j (bj), and the dot product of their embedding vectors (wi · wj), which is minimized using gradient descent. The learned embeddings are dense, low-dimensional vectors whose quality is assessed by how well they can recreate the original co-occurrence matrix, capturing patterns like 'brother minus man plus woman equals sister'. A major remaining issue for standalone embeddings is context, as words like 'bank' have multiple meanings, resulting in an 'average version' of the meaning, necessitating contextual embeddings (using transformers) later. In Keras, the workflow shifts from using one-hot vectors to using an embedding layer after the text vectorization layer outputs integers, requiring truncation and padding to normalize sentence lengths.