DeepSeek Engram: Transformers are just expensive Hashmaps

Quick Overview

The Engram architecture, proposed by DeepSeek, improves upon standard LLM architectures by introducing conditional memory via scalable lookup, which addresses the fundamental inefficiency of using deep computation for simple knowledge recall, leading to significant performance gains (up to +5.0 on reasoning benchmarks like BBH) while maintaining computational efficiency, effectively creating a dual-system approach mirroring human cognition (System 1 for fast recall via Engram, System 2 for deep reasoning via MoE).

Key Points: LLMs traditionally waste computation by running simple knowledge recall tasks through deep neural networks, which the paper argues is fundamentally wasteful. Engram introduces conditional memory via scalable lookup, providing a native memory primitive that separates static pattern recall (System 1) from deep reasoning (System 2). In benchmarks like BBH (Reasoning), Engram-27B showed a +5.0 improvement over MoE-27B with the same parameters and FLOPs, demonstrating superior reasoning capability. The architecture allows for prefetching of Engram lookups by the CPU while the GPU computes the current layer, hiding communication latency by computation. The optimal parameter split between MoE (deep reasoning) and Engram (memory lookup) was found to be 75-80% allocation to MoE. Technical limitations include hash collisions (mitigated but not eliminated) and static embeddings that do not adapt during training, while scope limitations note it is not RAG and lacks dynamic memory updates at inference time.

Context: This video explains the core concepts of the DeepSeek paper, "Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models," which introduces the Engram architecture as an alternative or complement to traditional Mixture-of-Experts (MoE) models. The central problem addressed is the inefficiency of large language models (LLMs) using massive, deep computation (like the Transformer's feed-forward networks) to perform simple factual lookups, analogous to using a calculator to store phone numbers. Engram seeks to introduce a dedicated, efficient memory lookup primitive to handle static, frequently accessed knowledge, freeing up the deep computation layers for complex reasoning.