# Build Hour: Agent Memory Patterns Source: https://www.youtube.com/watch?v=WsGVXiWzTpI Recap page: https://rapidrecap.app/video/WsGVXiWzTpI Generated: 2025-12-04T21:04:58.441+00:00 --- ## Quick Overview Agent memory design centers on context engineering, which uses techniques like reshaping/fitting, isolating/routing, and extracting/retrieving information to manage finite context windows, preventing issues like context burst, conflict, poisoning, and noise, ultimately aiming for the smallest high-signal context to maximize desired outcomes. **Key Points:** - Context engineering is both an art (involving judgment) and a science (with concrete, measurable patterns) that dictates LLM performance beyond just model quality. - The three core strategies for context management are: reshape and fit (trimming, compaction, summarization), isolate and route (offloading context to sub-agents), and extract and retrieve (memory extraction and retrieval). - Failure modes in context management include context burst (sudden token spikes), context conflict (contradictory instructions), context poisoning (incorrect information propagation), and context noise (redundant tool definitions). - Short-term memory involves in-session techniques like trimming (dropping older turns) and compaction (dropping older tool results) to maximize the context window. - Long-term memory utilizes extraction and retrieval, demonstrated by summarizing prior sessions into a structured memory object that can be injected into future sessions for continuity. - Best practices for prompting include being explicit, structured, giving room for planning, and avoiding conflicts by keeping tool sets small and non-overlapping. - The demo showed that context trimming removes older turns/messages when a threshold (e.g., max turns S3) is hit, while summarization compresses prior messages into a structured memory object. **Context:** This build hour session, hosted by Michaela with Emry from the solution architecture team, focused on "Agent Memory Patterns" as the latest topic following previous sessions on building agents from scratch and agent RFT. The session aimed to empower attendees with best practices and expertise to scale using OpenAI APIs, starting with an introduction to context engineering as the foundation for agent memory management. ## Detailed Analysis Context engineering encompasses prompt engineering, structured output, state management, and memory, aiming for the smallest high-signal context. It is crucial because long-running, tool-heavy agents deplete tokens and degrade quality through poisoning, noise, confusion, and bursting. Techniques are categorized into short-term memory (in-session: trimming, compaction, summarization) focusing on maximizing the current context window, and long-term memory (cross-session: extraction, retrieval) for building continuity. The presentation detailed failure modes like context burst (tool heavy workflows causing spikes), conflict (contradictory instructions), poisoning (hallucinations propagating), and noise (redundant tool definitions). Live demos illustrated context burst by showing token counts exceeding 3,000 when dumping large refund policies into context. Solutions involve engineering techniques: trimming drops older turns; compaction drops older tool calls/results; and summarization compresses prior messages into a structured memory object, adding a 'memory item' component to the context lifecycle. Summarization is preferred when information across tasks is dependent, while trimming suits tool-heavy, short workflows. Long-term memory allows injecting a summary from a previous session into a new system prompt for personalized continuity, provided guardrails against stale data and secrets are implemented. Further techniques include isolating context via sub-agents and managing memory shape by starting simple and evolving, prioritizing recent or stable facts. Evaluation involves running regular evals with and without memory and building memory-specific evals focusing on summary quality and injection prompts. ### Context Engineering Foundation - Art and science involving judgment and concrete patterns - Broader discipline than prompt engineering or retrieval - Goal is smallest high-signal context that maximizes desired outcome - Failure modes include burst, conflict, poisoning, and noise. ### Short-Term Memory Techniques (In-Session) - Context trimming drops older turns to refresh context - Context compaction drops older tool calls/results while keeping placeholders - Context summarization compresses prior messages into structured summaries/memory objects. ### Long-Term Memory & Retrieval - Extract and retrieve involves memory extraction, state management, and memory retrieval similar to RAG - Demo showed cross-session feature injecting prior summary into new system prompt for continuity - Memory instructions require precedence rules and guardrails against stale data/secrets. ### Failure Mode Visualization - Context burst visualized as a specific increase in turn tokens due to large tool outputs - Context conflict shown with contradictory system instructions and tool call results - Context poisoning results from inaccurate information propagating via summaries or state objects. ### Engineering Techniques & Demos - Trimming was set to trigger after max turns S3, removing tool outputs - Summarization trigger set at five, keeping recent three turns, generating a dense memory object detailing device, purchase location, and steps tried - Isolate and route involves offloading context and tools to specific sub-agents. ### Memory Evaluation and Scaling - Evaluate memory by running regular evals with and without memory, and building memory-specific evals - Manage stale memories using temporal tags or decay/window functions to focus on recent data - Scaling retrieval-based memory requires optimizing vector databases, storage, and filtering/ranking systems. ### Resources and Conclusion - Recommended resources include the context engineering cookbook, context summarization cookbook, and agents Python SDK - Core idea is understanding what, how, and when the agent should remember and forget, optimizing design tradeoffs iteratively.