# DeepSeek Engram: Transformers are just expensive Hashmaps

Source: https://www.youtube.com/watch?v=zt1jlTPCaps
Recap page: https://rapidrecap.app/video/zt1jlTPCaps
Generated: 2026-01-19T14:14:07.802+00:00

---
## Quick Overview

The Engram architecture, proposed by DeepSeek, improves upon standard LLM architectures by introducing conditional memory via scalable lookup, which addresses the fundamental inefficiency of using deep computation for simple knowledge recall, leading to significant performance gains (up to +5.0 on reasoning benchmarks like BBH) while maintaining computational efficiency, effectively creating a dual-system approach mirroring human cognition (System 1 for fast recall via Engram, System 2 for deep reasoning via MoE).

**Key Points:**
- LLMs traditionally waste computation by running simple knowledge recall tasks through deep neural networks, which the paper argues is fundamentally wasteful.
- Engram introduces conditional memory via scalable lookup, providing a native memory primitive that separates static pattern recall (System 1) from deep reasoning (System 2).
- In benchmarks like BBH (Reasoning), Engram-27B showed a +5.0 improvement over MoE-27B with the same parameters and FLOPs, demonstrating superior reasoning capability.
- The architecture allows for prefetching of Engram lookups by the CPU while the GPU computes the current layer, hiding communication latency by computation.
- The optimal parameter split between MoE (deep reasoning) and Engram (memory lookup) was found to be 75-80% allocation to MoE.
- Technical limitations include hash collisions (mitigated but not eliminated) and static embeddings that do not adapt during training, while scope limitations note it is not RAG and lacks dynamic memory updates at inference time.

![Screenshot at 00:08: The initial slide poses the central question: "What if I told you that LLMs are wasting computation every time an LLM recalls a simple fact...they're running it through deep neural networks when they could just...look it up like a dictionary?"](https://ss.rapidrecap.app/screens/zt1jlTPCaps/00-00-08.jpg)

**Context:** This video explains the core concepts of the DeepSeek paper, "Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models," which introduces the Engram architecture as an alternative or complement to traditional Mixture-of-Experts (MoE) models. The central problem addressed is the inefficiency of large language models (LLMs) using massive, deep computation (like the Transformer's feed-forward networks) to perform simple factual lookups, analogous to using a calculator to store phone numbers. Engram seeks to introduce a dedicated, efficient memory lookup primitive to handle static, frequently accessed knowledge, freeing up the deep computation layers for complex reasoning.

## Detailed Analysis

The presentation argues that current LLM architectures, based on the Transformer, are fundamentally inefficient because they rely on deep, expensive computation to simulate memory retrieval for simple facts. The paper, introduced by DeepSeek, proposes Engram, which adds a conditional memory primitive alongside conditional computation (like MoE). This mirrors human cognition's dual systems: System 1 (fast, automatic recall, handled by Engram/lookup) and System 2 (slow, deliberate reasoning, handled by MoE/deep computation). The Engram mechanism uses N-grams (2-gram and 3-gram) to generate deterministic hash IDs for direct O(1) lookup into a massive embedding table stored in host RAM, avoiding the deep layers used for reconstruction in pure MoE models. This hardware optimization allows the CPU to prefetch the next layer's Engram lookup while the GPU computes the current layer, effectively hiding communication latency behind computation, resulting in less than 3% throughput penalty for large models. The results show Engram-27B significantly outperforms MoE-27B across various benchmarks, notably gaining +5.0 on BBH reasoning tasks, suggesting that dedicating capacity to explicit memory recall drastically improves reasoning by functionally deepening the model's reasoning layers. The optimal balance found is allocating 75-80% of parameters to MoE (reasoning) and the rest to Engram (memory).

### The Hidden Inefficiency

- LLMs use the same expensive machinery for everything—whether it's deep reasoning or simple lookup
- Complex Reasoning requires deep, dynamic computation, context-dependent, novel combinations (Needs: MoE Experts)
- Static Patterns (like 'Diana, Princess of Wales') require local, stereotyped, frequent lookups (Needs: O(1) Lookup)

### How Engram Works

- Input phrases are broken into N-grams (2-gram, 3-gram), which generate hashes pointing to a Massive Embedding Table (billions of parameters)
- The retrieved embeddings pass through a Context-Aware Gate that filters memories based on the model's current hidden state, suppressing contextually irrelevant lookups.

### The U-Shaped Scaling Law

- Allocating 100% to MoE (Pure MoE) results in suboptimal validation loss due to wasting compute on memory tasks
- Optimal performance is achieved by reallocating 20-25% of sparse parameters to Engram memory.

### Why Reasoning Improved More

- MoE-27B dedicates Layers 1-6 to Static Reconstruction and Layers 7-38 to Reasoning (24 layers for reasoning)
- Engram-27B dedicates Layer 0 to O(1) Lookup and Layers 1-38 to Reasoning (All 38 layers for reasoning), making the model functionally deeper for reasoning tasks.

### Hardware Story

- MoE routing is dynamic (cannot prefetch) while Engram lookup is deterministic (IDs known from input alone)
- This allows the CPU to prefetch Engram data while the GPU computes the previous layer, hiding communication latency behind computation, resulting in a throughput penalty of <3% for 100B parameter tables in host RAM.

### What This Means

- For Model Efficiency: Same compute, better results, or same results, less compute
- For Hardware Economics: 100B parameters in host RAM with <3% overhead, less HBM needed
- For Architecture Research: A new primitive, conditional memory, alongside conditional computation.

![Screenshot at 00:08: Initial slide questioning the efficiency of LLMs using deep computation for simple fact recall, suggesting a dictionary lookup alternative.](https://ss.rapidrecap.app/screens/zt1jlTPCaps/00-00-08.jpg)
![Screenshot at 00:21: Title slide for the DeepSeek paper, "Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models."](https://ss.rapidrecap.app/screens/zt1jlTPCaps/00-00-21.jpg)
![Screenshot at 01:22: Demonstration of the multi-layer lookup process for the input 'Diana, Princess of Wales,' showing how context is built layer by layer to reconstruct the final knowledge.](https://ss.rapidrecap.app/screens/zt1jlTPCaps/00-01-22.jpg)
![Screenshot at 02:55: Diagram illustrating the two axes of sparsity: MoE for conditional computation \(reasoning\) and Engram for conditional memory \(lookup\).](https://ss.rapidrecap.app/screens/zt1jlTPCaps/00-02-55.jpg)
![Screenshot at 06:01: Diagram illustrating the Context-Aware Gate mechanism, which uses the model's hidden state to validate retrieved memories, opening the gate for relevant context \('tech company stocks'\) and closing it for irrelevant context \('fruit salad recipe'\).](https://ss.rapidrecap.app/screens/zt1jlTPCaps/00-06-01.jpg)