Lec 24. Inference Methods for Deep Learning

Quick Overview

Inference methods for deep learning are shifting focus from purely training-time computation to incorporating significant test-time computation, often through search techniques, exemplified by models like OpenAI's O1, which leverage scaling laws for both training and test-time compute to achieve better performance.

Key Points: The ML definition of inference is what statisticians call prediction: using a model to make inferences about a new query point, contrasting with the statistical definition focused on properties of a data distribution. The landscape divides into training (factory/learning/amortized inference) and inference (test time/deployment/thinking/reasoning), with recent trends emphasizing increased test-time compute for search. Autoregressive models trained via teacher forcing suffer a gap between training and test time, as greedy sampling does not maximize the joint probability of the entire sequence, prompting methods like Best-of-N or beam search. Best-of-N sampling involves taking N independent samples and evaluating the total likelihood of each sequence to select the highest probability outcome, while beam search maintains the top-k best next words at each generation step. Chain-of-Thought (CoT) is framed as an emergent or learned search where the LLM performs sequential reasoning steps, writing intermediate steps into the text space, which can be applied to symbolic search problems. Search methods can optimize for any scoring function, not just likelihood; this includes using human feedback (RLHF) or formal verification models (e.g., a compiler) as the score to maximize. The scaling law plot for the OpenAI O1 model showed that increasing test-time compute via intelligent inference/search also yields a power-law improvement in accuracy, sometimes with a steeper slope than train-time compute scaling.

Context: Phillip Isola introduces a lecture on "Inference Methods for Deep Learning," noting that while the underlying ideas are old, this topic has become a hot area in 2024, inspired in part by tutorials like Sasha Rush's, with the payoff being an understanding of how recent models like OpenAI's O1 likely function. The lecture first clarifies the distinction between the statistical definition of inference (figuring out data distribution properties, equivalent to ML training) and the ML definition of inference (prediction at test time). The discussion sets up a dichotomy between training (pre-training, fine-tuning) and deployment-time adaptation (prompting, test-time training, search).