How to Build AI Agents by Augmenting LLMs with Codified Human Expert Domain Knowledge

Quick Overview

The paper "How to Build AI Agents by Augmenting LLMs with Codified Human Expert Domain Knowledge" demonstrates that augmenting LLMs with explicit rules and tacit knowledge, derived from interviewing domain experts, significantly improves AI agent performance, yielding a 206% quality improvement over a baseline LLM (like GPT-4) when tested across automotive battery design, electric motor optimization, and legal contract analysis scenarios.

Key Points: The research establishes a framework for augmenting LLMs by codifying human expert domain knowledge (both explicit rules and tacit intuition). The agent system, augmented with expert knowledge, achieved a 206% improvement in output quality compared to the GPT-4 baseline across three domains. The three tested domains were automotive battery design, electric motor optimization, and legal contract analysis. The explicit rules (like 'don't over-cluster plot' for visualization) and tacit knowledge were successfully codified into Python scripts and agent prompts. The baseline LLM (GPT-4) produced a 'garbage' history plot when asked to plot 14 parameters on one chart, indicating a failure to grasp necessary constraints. The experiment showed that the human expert's intuition (e.g., knowing to scale axes correctly) provided critical context that the raw LLM lacked, successfully bridging the gap between data and actionable insight. The framework successfully distilled the essence of human expertise, transferring it to the AI agent without requiring the agent to undergo extensive retraining.

Context: This presentation details a novel approach to improving the capability of Large Language Models (LLMs) when building AI agents, specifically by incorporating codified knowledge from human domain experts. The researchers collaborated with experts from fields like automotive engineering and law to extract not just explicit, codified rules, but also the tacit, intuitive knowledge experts use when making critical judgments. The goal was to see if this structured integration of human expertise could overcome the inherent limitations of standard LLMs, which often struggle with complex, nuanced decision-making in high-stakes fields.