Fixed-Persona SLMs with Modular Memory: Scalable NPC Dialogue on Consumer Hardware

Quick Overview

The research demonstrates that modular memory systems, built around small, fine-tuned SLMs like Mistral 7B, enable high-performance, low-latency dialogue generation on consumer hardware by storing character knowledge and context in separate, efficiently managed memory modules, overcoming the limitations of monolithic LLMs.

Key Points: The proposed architecture uses a modular memory system built around a fine-tuned Mistral 7B SLM, achieving high performance on consumer hardware. The system separates character knowledge (conversational history, secrets, relationship context) and world knowledge into distinct, searchable memory modules. The small, fine-tuned SLM achieved 99.9% factual accuracy on the evaluation dataset, significantly outperforming a larger LLM (9% accuracy) trained on the same small dataset. Latency was extremely low: 0.1145 seconds for the first token and 5.49 seconds for a full response, compared to 5.52 seconds for the large model (TTFT). The modular system reduces disk footprint dramatically, shrinking from 16GB (for the monolithic model) to 3.9GB for the modular memory. The approach allows for dynamic behavior by swapping out memory modules, enabling NPCs to maintain persona consistency or adapt to new information without retraining the core model.

Context: This AI research paper explores overcoming the constraints of running large language models (LLMs) for interactive media, specifically focusing on creating non-playable characters (NPCs) with expressive and dynamic personalities. The primary challenge addressed is the latency and memory demands associated with large models when trying to maintain complex character memory and context locally on consumer hardware.

Detailed Analysis

The research introduces a modular memory system for Small Language Models (SLMs) designed to create scalable NPC dialogue on consumer hardware. The main achievement is demonstrating that a fine-tuned Mistral 7B model, augmented with modular memory, significantly outperforms a monolithic large language model (LLM) on dialogue quality while maintaining low latency. The system separates memory into two types: conversational memory (history, secrets, relationship context) and world knowledge memory, which are stored in separate, highly efficient, searchable index cards or vector embeddings. When tested against a small, hand-crafted dataset, the fine-tuned Mistral 7B achieved 99.9% factual accuracy, vastly superior to the larger LLM trained on the same data, which scored only 16% accuracy. Furthermore, latency for the first token was extremely fast at 0.1145 seconds, and total response time was 5.49 seconds, whereas the large model took 5.52 seconds. This efficiency drastically reduces the disk footprint from 16GB to 3.9GB. The modularity is key, allowing developers to swap out specific memory modules (like character personalities or rule sets) without retraining the entire model, ensuring consistency and scalability for complex virtual worlds.