# SemanticALLI: Caching Reasoning, Not Just Responses, in Agentic Systems Source: https://www.youtube.com/watch?v=2qpd19cxvL0 Recap page: https://rapidrecap.app/video/2qpd19cxvL0 Generated: 2026-01-29T23:04:12.32+00:00 --- ## Quick Overview SemanticALLI, a system proposed by Faroon Chillara and team at PMG, introduces caching reasoning—not just responses—to solve the latency utility gap in enterprise AI adoption by decoupling the stages of query processing, which results in significantly faster response times for subsequent similar queries. **Key Points:** - SemanticALLI aims to close the latency utility gap by caching reasoning steps rather than just final answers, which is crucial for enterprise AI adoption. - The system uses a two-stage process: Stage 1 (ARI: Analytic Intent Resolution) and Stage 2 (VS: Visualization Synthesis). - For a query like "Show me how our Facebook ads performed last week versus Google," the system must first determine the intent (Stage 1) and then generate the visualization (Stage 2). - When asked the same question twice, the system skips the slow Stage 1 (Analytic Intent Resolution) and directly hits Stage 2 (Visualization Synthesis) with the correct vector, resulting in near-instant answers. - The paper cites that the monolithic caching approach, which caches the final result without reasoning context, fails in real-world scenarios because business logic (like sales data changing) invalidates the result. - The research demonstrates that the multi-stage approach, using both vector search (for similarity) and semantic indexing (for meaning), achieves a 96% similarity score for identical queries, compared to the monolithic approach which fails quickly. - This method drastically improves efficiency, reducing the runtime for a full LLM call from 57 seconds to 31 seconds, and eliminating the need for agents to constantly re-derive complex logic. ![Screenshot at 00:16: The hosts discuss the core problem: the latency utility gap where simple requests are fast, but complex reasoning stalls AI adoption, setting up the need for SemanticALLI's reasoning caching.](https://ss.rapidrecap.app/screens/2qpd19cxvL0/00-00-16.jpg) **Context:** The discussion centers on SemanticALLI, a novel approach developed by Faroon Chillara and his team at PMG, designed to address the trade-off between speed and accuracy in large language models (LLMs) used in enterprise settings, often referred to as the 'latency utility gap.' The core problem is that while AI can generate answers quickly, complex reasoning tasks often require significant time, and caching the final answer risks being outdated due to changing real-world data, making the system unusable for dynamic tasks like reporting. ## Detailed Analysis SemanticALLI tackles the latency utility gap in enterprise AI by caching the reasoning process itself, not just the final output. The researchers, including Faroon Chillara from PMG, detail a two-stage architecture. Stage 1, Analytic Intent Resolution (ARI), maps the user's messy, natural language query (e.g., comparing Facebook and Google ad performance) into a highly structured, multi-dimensional vector representation, which captures the query's core meaning and intent. Stage 2, Visualization Synthesis (VS), is responsible for generating the actual output (like a chart) based on that vector. The key innovation is that if a subsequent query is semantically similar, the system skips the slow Stage 1, retrieves the cached vector from Stage 1, and immediately proceeds to Stage 2, achieving near-instantaneous results. The paper contrasts this with monolithic caching, which caches the final answer but fails when underlying data changes (like sales figures). The new approach avoids this by caching the logic; for example, a query for Q3 sales West Coast vs. East Coast data yields the same vector representation even if the underlying data is different, allowing the system to fetch the correct current data during the Stage 2 execution. This leads to massive efficiency gains, demonstrated by reducing a 57-second runtime to 31 seconds, making the AI feel more like a native tool rather than a slow remote server. ### System Architecture - Two stages: Stage 1 (ARI - Analytic Intent Resolution) maps messy language to a structured vector; Stage 2 (VS - Visualization Synthesis) executes the query based on the vector. ### Caching Strategy - Caches the reasoning vector from Stage 1, not the final result, allowing the system to reuse the query's intent even when underlying data changes. ### Performance Metrics - Full LLM call runtime drops from 57 seconds to 31 seconds; SemanticALLI achieves a 96% similarity score for identical queries, whereas monolithic caching fails quickly. ### Stage 1 (ARI) - Focuses on understanding the meaning (semantic indexing) of the query to create a dense vector representation. ### Stage 2 (VS) - Focuses on execution, such as generating code (Python/SQL) or visualizations, using the vector provided by Stage 1. ### Trade-offs - Lowering the similarity threshold (e.g., from 0.90 to 0.85) increases the risk of false positives (caching irrelevant data) but significantly boosts speed, highlighting the need to balance speed and accuracy. ![Screenshot at 00:00: The opening slide features the podcast branding and a call to action to become a member, set against a visual representation of a sound wave or data stream.](https://ss.rapidrecap.app/screens/2qpd19cxvL0/00-00-00.jpg) ![Screenshot at 00:29: The speaker discusses the concept of the latency utility gap, explaining that users abandon tools that are too slow, even if powerful.](https://ss.rapidrecap.app/screens/2qpd19cxvL0/00-00-29.jpg) ![Screenshot at 00:40: The speaker clarifies that the problem isn't just speed, but that current systems often default to inefficient, repetitive processing paths, like always recalculating the route from scratch.](https://ss.rapidrecap.app/screens/2qpd19cxvL0/00-00-40.jpg) ![Screenshot at 01:43: The hosts differentiate between two stages: Stage 1 \(ARI - Analytic Intent Resolution\) and Stage 2 \(VS - Visualization Synthesis\), which is the core architectural split.](https://ss.rapidrecap.app/screens/2qpd19cxvL0/00-01-43.jpg) ![Screenshot at 02:35: The speaker details the three-tier system built by the researchers, contrasting the different layers of caching and reasoning.](https://ss.rapidrecap.app/screens/2qpd19cxvL0/00-02-35.jpg)