# Titans: Learning to Memorize at Test Time (Paper Analysis) Source: https://www.youtube.com/watch?v=v67plFw1nMw Recap page: https://rapidrecap.app/video/v67plFw1nMw Generated: 2025-12-14T17:05:28.388+00:00 --- ## Quick Overview The Titans paper proposes an architecture where a model learns to memorize information at test time using a neural network as a dynamic memory, aiming to overcome the context window limitations of standard transformers by updating this memory via an inner training loop. **Key Points:** - The Titans model addresses transformer context window limitations by allowing the model to access information beyond the current context via a learned memory. - The core proposal involves using a neural network, updated during test time, as a memory component that stores associations between keys and values from past data chunks. - The author notes skepticism regarding the paper's claim that vector/matrix-valued memories fail because they compress data, arguing that the limitation lies in the specified linear retrieval/storage mechanism, not compression itself. - The memory update process involves training the neural network on the fly using a loss function derived from 'surprise' (related to the gradient of the key-value association loss), which the reviewer equates to gradient descent with momentum. - The paper also introduces a 'persistent memory' component, which the reviewer identifies as equivalent to standard data-independent parameters learned during training, such as prefix tuning. - The reviewer concludes that learning to memorize and forget purposefully at test time is a necessary and very cool direction for future models, despite marketing concerns. - Previous attempts to solve long context issues included models like Transformer-XL, which used a state passed between context windows, acting like an RNN across chunks. **Context:** This video analyzes the Google Research paper titled "Titans: Learning to Memorize at Test Time," which was presented at Neurips and gained attention through marketing. The paper tackles the fundamental limitation of attention-based models like transformers, which can only attend to information within a fixed, limited context window, making tasks requiring very long-range dependencies difficult. ## Detailed Analysis The Titans paper introduces an architecture designed to overcome the fixed context window of transformers by incorporating a form of runtime memory that learns associations dynamically. The reviewer explains that traditional attention mechanisms suffer from a quadratic blow-up due to the non-separable nature of the softmax operation over long sequences, which linear transformers attempt to solve by approximating softmax with kernels, effectively turning the attention into a linear recurrent network that accumulates past keys and values. The Titans model proposes that instead of simple matrix or vector-valued memories, which it claims are insufficient because they compress data linearly, the memory should be a neural network (an MLP) updated at test time. To memorize, this neural network is trained on the fly during sequence processing; specifically, it is trained to map keys to their corresponding values, using a loss function framed around 'surprise' (gradient magnitude), which the reviewer equates to standard gradient descent with momentum. The reviewer expresses strong skepticism about the paper's marketing claims, arguing that the superiority of a neural network memory over a vector memory is only true under the constraints of linear storage/retrieval operations; a complex, non-linear storage/retrieval scheme using a vector memory could theoretically achieve equivalence. Furthermore, the paper includes a 'persistent memory' component, which the reviewer dismisses as simply standard, data-independent parameters learned during training, akin to prefix tuning. Despite these critiques about framing and marketing, the reviewer validates the core concept: developing models that actively learn to memorize and forget at test time is a necessary and very good direction for handling increasingly long sequences. ### Problem Definition - Transformer context window limitations hinder tasks requiring long-range dependencies - Previous solutions included state passing mechanisms like in Transformer-XL, mimicking RNNs across chunks - Linear transformers approximate attention using kernels to achieve linear complexity via accumulation. ### Titans Memory Architecture - Memory is explicitly a neural network updated at test time via an inner training loop - The goal is to retrieve information about distant past tokens that have fallen out of the immediate context window. ### Memory Training Mechanism - The neural network memory trains to map input keys (x) to output values (y) - The loss function is based on 'surprise,' defined by the gradient of this association loss, which mathematically resembles gradient descent with momentum. ### Critique of Memory Types - The paper claims vector/matrix memories are inferior due to compression - Reviewer argues this distinction is misleading, as complex non-linear retrieval/storage in a vector memory can match a complex neural network memory. ### Persistent Memory Component - Titans includes a persistent memory learned during training that is data-independent - Reviewer identifies this as equivalent to existing techniques like prefix tuning or standard model parameters. ### Final Verdict - Memorizing and forgetting purposefully at test time is a very necessary and cool direction for future model development, despite the paper's marketing framing.