# How to Build AI Agents by Augmenting LLMs with Codified Human Expert Domain Knowledge Source: https://www.youtube.com/watch?v=-0ipAJaTPBs Recap page: https://rapidrecap.app/video/-0ipAJaTPBs Generated: 2026-01-24T19:03:23.964+00:00 --- ## Quick Overview The paper "How to Build AI Agents by Augmenting LLMs with Codified Human Expert Domain Knowledge" demonstrates that augmenting LLMs with explicit rules and tacit knowledge, derived from interviewing domain experts, significantly improves AI agent performance, yielding a 206% quality improvement over a baseline LLM (like GPT-4) when tested across automotive battery design, electric motor optimization, and legal contract analysis scenarios. **Key Points:** - The research establishes a framework for augmenting LLMs by codifying human expert domain knowledge (both explicit rules and tacit intuition). - The agent system, augmented with expert knowledge, achieved a 206% improvement in output quality compared to the GPT-4 baseline across three domains. - The three tested domains were automotive battery design, electric motor optimization, and legal contract analysis. - The explicit rules (like 'don't over-cluster plot' for visualization) and tacit knowledge were successfully codified into Python scripts and agent prompts. - The baseline LLM (GPT-4) produced a 'garbage' history plot when asked to plot 14 parameters on one chart, indicating a failure to grasp necessary constraints. - The experiment showed that the human expert's intuition (e.g., knowing to scale axes correctly) provided critical context that the raw LLM lacked, successfully bridging the gap between data and actionable insight. - The framework successfully distilled the essence of human expertise, transferring it to the AI agent without requiring the agent to undergo extensive retraining. ![Screenshot at 00:00: The opening visual features an illustration of two podcasters in a studio setting with a prominent call to action: "BECOME A MEMBER TODAY!", set against a background of a waveform, representing the AI Papers Podcast.](https://ss.rapidrecap.app/screens/-0ipAJaTPBs/00-00-00.jpg) **Context:** This presentation details a novel approach to improving the capability of Large Language Models (LLMs) when building AI agents, specifically by incorporating codified knowledge from human domain experts. The researchers collaborated with experts from fields like automotive engineering and law to extract not just explicit, codified rules, but also the tacit, intuitive knowledge experts use when making critical judgments. The goal was to see if this structured integration of human expertise could overcome the inherent limitations of standard LLMs, which often struggle with complex, nuanced decision-making in high-stakes fields. ## Detailed Analysis The video discusses a research paper detailing a framework for building more capable AI agents by augmenting standard LLMs with human expert domain knowledge. This augmentation involves codifying two types of knowledge: explicit rules (like those found in code or manuals) and tacit knowledge (the 'gut feeling' or intuition experts possess). The researchers tested this framework across three domains: automotive battery design, electric motor optimization, and legal contract analysis. In all three scenarios, the augmented agent significantly outperformed the baseline LLM (using GPT-4). For instance, when asked to create a scatter plot of 14 variables, the baseline GPT-4 produced a useless 'spaghetti plot' because it lacked the implicit knowledge of proper scaling and exclusion of non-converged data points. The agent, guided by the codified expertise, correctly identified the need for a normalized axis and produced a perfect 3-out-of-3 score on the baseline test. The core insight is that by explicitly teaching the AI the human expert's intuition—like using size and color gradients for different dimensions on a 2D plot—the system gains massive utility and scalability, effectively transferring expert judgment into the system's prompts and rules without requiring massive retraining. ### Problem Identification - Expert Bottleneck: The core problem is that expert knowledge (explicit rules and tacit intuition) is often the bottleneck, leading to errors when LLMs work alone, especially in high-stakes fields like medicine or finance. ### Methodology - Augmenting LLMs: The solution involves building a four-step framework that injects codified human expertise (explicit rules and tacit knowledge) directly into the agent's prompts and instructions, bypassing the need for extensive retraining. ### Scenario 1 - History Plot Test: The baseline LLM failed to generate a useful plot of 14 variables, highlighting its lack of implicit constraints; the agent system successfully normalized axes and filtered data points. ### Scenario 2 - 2D Relation Plot: The agent system successfully navigated the trade-off between mass and cost for battery design, a task where the baseline failed to grasp the necessary nuance. ### Scenario 3 - Parallel Plot Test: The agent system outperformed the baseline by using explicit rules (e.g., no dashed lines for non-converged variables) and tacit knowledge to produce a superior visualization. ### Results and Conclusion - The agent achieved a 206% quality improvement over the baseline, proving that codifying human expertise is critical for building scalable, high-quality AI tools. ![Screenshot at 00:00: The opening visual features an illustration of two podcasters in a studio setting with a prominent call to action: "BECOME A MEMBER TODAY!", set against a background of a waveform, representing the AI Papers Podcast.](https://ss.rapidrecap.app/screens/-0ipAJaTPBs/00-00-00.jpg) ![Screenshot at 00:14: The speaker discusses the need to codify expert knowledge, using an image showing a complex, seemingly chaotic plot \(a scatter plot with many intersecting lines\) as an example of what an unguided LLM might produce.](https://ss.rapidrecap.app/screens/-0ipAJaTPBs/00-00-14.jpg) ![Screenshot at 02:23: A slide appears summarizing the simulation data, showing that the agent system \(compared to the baseline\) successfully generated outputs across three distinct domains: automotive battery design, electric motor optimization, and control arm analysis.](https://ss.rapidrecap.app/screens/-0ipAJaTPBs/00-02-23.jpg) ![Screenshot at 05:56: The speaker highlights the four key resources used in the framework: Codified Rules \(Python scripts\), Software Manuals, Expert Intuition \(gut feeling\), and the Prompt Constructor.](https://ss.rapidrecap.app/screens/-0ipAJaTPBs/00-05-56.jpg) ![Screenshot at 08:45: A slide illustrating the comparison between the Human Expert \(yellow\) and the Agent \(blue\) results on a 2D plot, showing the agent successfully capturing the nuances that the human expert rated highly.](https://ss.rapidrecap.app/screens/-0ipAJaTPBs/00-08-45.jpg)