# Using Knowledge Elicitation Techniques To Infuse Deep Expertise And Best Practices Into Gen AI Source: https://www.youtube.com/watch?v=XQxUrcQwNA0 Recap page: https://rapidrecap.app/video/XQxUrcQwNA0 Generated: 2025-11-11T19:07:23.063+00:00 --- ## Quick Overview The main challenge in developing expert AI systems is moving beyond the general knowledge of foundation models by infusing deep, often unwritten, human expertise through knowledge elicitation techniques like interviewing experts and analyzing their historical decision-making processes. **Key Points:** - The primary challenge in AI development is moving beyond general LLMs to create specialized experts, often requiring deep, unwritten knowledge. - The methodology involves layering three techniques: Human-to-Human (H2H) interviews, Human-to-AI (H2AI) elicitation, and Data Analysis for rule validation. - The H2H step involves interviewing experts (like a stock trader) to extract tacit knowledge, including gut feelings and unwritten rules, which are then structured or coded. - The 'Momentum Rule' example from the study requires a stock to drop 8% below purchase price and the expert trader to avoid forcing a stop-loss if the market sentiment is positive. - The final goal is to create a 'Synthetic Expert' AI whose rules are validated against both explicit data and tacit expert knowledge, avoiding the pitfalls of pure data reliance. - The combination of structured rules (like the 8% stop-loss) and nuanced, elicited knowledge prevents the AI from generating 'fake rationalizations' that sound logical but are contextually wrong. ![Screenshot at 03:39: The speaker explains that the fundamental problem is that expert knowledge is often unwritten, requiring structured elicitation methods to capture it before building a specialized AI.](https://ss.rapidrecap.app/screens/XQxUrcQwNA0/00-03-39.png) **Context:** This AI Papers Daily podcast episode discusses the major challenge facing current AI development: how to transition powerful general-purpose Large Language Models (LLMs) into reliable, specialized experts in fields like finance or medicine, which requires capturing the nuanced, often uncodified, 'gut feeling' knowledge held by human professionals. ## Detailed Analysis The discussion focuses on the critical challenge of infusing deep, specialized expertise into AI systems, moving beyond the capabilities of general LLMs like GPT-4. The solution proposed involves a layered methodology consisting of three steps: Human-to-Human (H2H) interviews, Human-to-AI (H2AI) elicitation, and Data Analysis for validation. The H2H phase involves interviewing domain experts (like a stock trader) to extract tacit knowledge—rules, intuition, and gut feelings—which are then either structured into formal rules or encoded in documents like JSON or YAML. The H2AI phase involves having the AI interview the expert in real-time to articulate their thought process, ensuring the AI captures the nuances missed in static documentation. The validation phase checks if the AI's newly formed rules conflict with existing general patterns learned from broad data. An example provided is the 'Momentum Rule' from a 2021 paper by Carrigan, Hallman, and Bertini, which dictates selling a stock if it drops 8% below purchase price unless the expert intuition suggests otherwise (e.g., positive market sentiment). The success of this layered approach—combining explicit rules with deep, elicited knowledge—is crucial for creating trustworthy, specialized AI agents that avoid the pitfalls of relying solely on generalized, potentially biased, training data. ### The Three-Step Knowledge Infusion Process - H2H Interviews to capture tacit knowledge - H2AI Elicitation for real-time articulation of expert thought - Data Analysis for rule validation against general patterns ### The Momentum Rule Example - A stock dropping 8% below purchase price triggers a sell unless expert intuition (e.g., positive sentiment) overrides the rule. ### Validation and Trust - The process ensures the AI's rules are accurate and reflect the expert's actual decision-making, preventing reliance on 'fake rationalizations' or biases present in broad training data. ### Goal of Specialized AI - To create an AI that rivals human experts in specific tasks (like stock trading or urology) by formalizing deep, often unwritten, expertise. ![Screenshot at 00:01: Visual representation of the podcast setup with an audio waveform graphic.](https://ss.rapidrecap.app/screens/XQxUrcQwNA0/00-00-01.png) ![Screenshot at 00:06: Speaker emphasizes the challenge of current large general AI models.](https://ss.rapidrecap.app/screens/XQxUrcQwNA0/00-00-06.png) ![Screenshot at 00:14: The speaker references the work of humans who know the 'unwritten rules' for specialized knowledge.](https://ss.rapidrecap.app/screens/XQxUrcQwNA0/00-00-14.png) ![Screenshot at 00:39: Transition to discussing the limitations of purely rules-based systems from the early AI era.](https://ss.rapidrecap.app/screens/XQxUrcQwNA0/00-00-39.png) ![Screenshot at 01:53: Speaker clarifies the concept of 'Arigato' or specialized knowledge, contrasting it with general knowledge.](https://ss.rapidrecap.app/screens/XQxUrcQwNA0/00-01-53.png) ![Screenshot at 03:03: The conversation shifts to discussing the successful validation of an LLM against expert stock trader data.](https://ss.rapidrecap.app/screens/XQxUrcQwNA0/00-03-03.png) ![Screenshot at 04:54: The speaker introduces the concept of 'experts' relying heavily on intuition or gut feel.](https://ss.rapidrecap.app/screens/XQxUrcQwNA0/00-04-54.png) ![Screenshot at 06:32: The speaker explains the need to avoid rules that are too strict, referencing the 'Market Sentiment Rule' as an example.](https://ss.rapidrecap.app/screens/XQxUrcQwNA0/00-06-32.png) ![Screenshot at 08:18: The hosts discuss the need for the AI to articulate the expert's thought process when interacting with a trader.](https://ss.rapidrecap.app/screens/XQxUrcQwNA0/00-08-18.png) ![Screenshot at 10:14: The discussion highlights the need to encode elicited knowledge into structured formats like JSON or YAML.](https://ss.rapidrecap.app/screens/XQxUrcQwNA0/00-10-14.png)