# Lec 24. Inference Methods for Deep Learning Source: https://www.youtube.com/watch?v=mbgFTqKxR7A Recap page: https://rapidrecap.app/video/mbgFTqKxR7A Generated: 2026-02-11T15:05:54.631+00:00 --- ## Quick Overview Inference methods for deep learning are shifting focus from purely training-time computation to incorporating significant test-time computation, often through search techniques, exemplified by models like OpenAI's O1, which leverage scaling laws for both training and test-time compute to achieve better performance. **Key Points:** - The ML definition of inference is what statisticians call prediction: using a model to make inferences about a new query point, contrasting with the statistical definition focused on properties of a data distribution. - The landscape divides into training (factory/learning/amortized inference) and inference (test time/deployment/thinking/reasoning), with recent trends emphasizing increased test-time compute for search. - Autoregressive models trained via teacher forcing suffer a gap between training and test time, as greedy sampling does not maximize the joint probability of the entire sequence, prompting methods like Best-of-N or beam search. - Best-of-N sampling involves taking N independent samples and evaluating the total likelihood of each sequence to select the highest probability outcome, while beam search maintains the top-k best next words at each generation step. - Chain-of-Thought (CoT) is framed as an emergent or learned search where the LLM performs sequential reasoning steps, writing intermediate steps into the text space, which can be applied to symbolic search problems. - Search methods can optimize for any scoring function, not just likelihood; this includes using human feedback (RLHF) or formal verification models (e.g., a compiler) as the score to maximize. - The scaling law plot for the OpenAI O1 model showed that increasing test-time compute via intelligent inference/search also yields a power-law improvement in accuracy, sometimes with a steeper slope than train-time compute scaling. **Context:** Phillip Isola introduces a lecture on "Inference Methods for Deep Learning," noting that while the underlying ideas are old, this topic has become a hot area in 2024, inspired in part by tutorials like Sasha Rush's, with the payoff being an understanding of how recent models like OpenAI's O1 likely function. The lecture first clarifies the distinction between the statistical definition of inference (figuring out data distribution properties, equivalent to ML training) and the ML definition of inference (prediction at test time). The discussion sets up a dichotomy between training (pre-training, fine-tuning) and deployment-time adaptation (prompting, test-time training, search). ## Detailed Analysis The lecture contrasts training, which involves learning a feedforward heuristic mapping (amortized inference), with inference, which involves step-by-step reasoning or prediction at test time. A key observation is that modern deep learning heavily relies on search and learning, as per Rich Sutton's "Bitter Lesson," and current industry trends involve scaling up test-time compute significantly beyond just a single forward pass. For autoregressive models, greedy sampling fails to maximize the joint probability of a sequence because it maximizes individual conditionals sequentially; better methods like Best-of-N (sampling N times and selecting the sequence with the highest calculated likelihood) or beam search (a tree search maintaining top-k next words) improve sequence quality. Chain-of-Thought (CoT) is presented as an emergent, learned symbolic search written in language space. Furthermore, search optimization extends beyond likelihood maximization to any arbitrary scoring function, such as human preference scores (RLHF) or formal verification scores (e.g., checking Python code correctness), noting that verification is fundamentally easier than generation (P vs. NP contrast). The speaker also details older work on optimizing image generators (like CLIP + VQGAN) in latent space using gradient descent to maximize scores related to human perception like memorability or aesthetics, connecting this concept of optimizing through search over generative outputs to modern preference optimization techniques. ### Inference Definitions - ML inference equals statistical prediction (using a model for new queries) - Statistical inference equals ML training (figuring out data distribution properties) - Training is called amortized inference, while deployment is called thinking or reasoning. ### Training vs. Inference Regimes - Training is done in the factory (pre-training, fine-tuning) - Inference is done at test time (deployment, robot walking around) - Recent trend moves compute from only train time to significant test-time compute/search. ### Improving Autoregressive Sampling - Greedy sampling fails to maximize joint sequence probability due to conditioning on self-generated tokens - Best-of-N samples N sequences and picks the one with the highest total likelihood calculated via chain rule - Beam search maintains the top-k best next words in a tree structure during generation. ### Search Methods Beyond Likelihood - Search can optimize for any scoring function, such as human ratings (RLHF) or formal verification (checking if generated code is correct) - Verification is often easier than generation because checking correctness is easier than finding the solution (P vs. NP implication). ### Search Through Latent Space - Image generation models (like CLIP + VQGAN) allow search over latent variables (z) using gradient descent to maximize external scores like memorability or aesthetics - Maximizing memorability results in big eyes/features, while aesthetics favors depth-of-field effects. ### The 'Just Ask' Paradigm - Chain-of-Thought prompts the LLM to perform symbolic search (reasoning) in text space - This paradigm suggests asking the model to perform complex processes like beam search or continuous reward optimization instead of hardcoding them.