# Trust by Design: Skill Profiles for Transparent, Cost-Aware LLM Routing Source: https://www.youtube.com/watch?v=ZGIIZUUJo14 Recap page: https://rapidrecap.app/video/ZGIIZUUJo14 Generated: 2026-02-07T20:03:47.396+00:00 --- ## Quick Overview The paper "Trust by Design" proposes a framework that moves AI routing away from relying solely on general benchmarks towards skill-based arbitration, ensuring that queries are sent to the most appropriate, cost-effective, and transparent Large Language Model (LLM) for the specific task, which is necessary for enterprise adoption where cost and auditability are critical factors. **Key Points:** - The paper argues against relying solely on overall benchmarks (like MMLU or F1 scores) for routing LLMs, calling it the "Benchmark Trap." - It proposes a framework based on skill profiles, which map specific cognitive abilities (like numerical reasoning, data extraction, instruction following) to an LLM. - The proposed solution, called 'Bella,' acts as an orchestrator that uses skill profiles to route queries to the most proficient and cost-effective model for the task. - For the financial task of calculating a percentage change ($50 to $80), the best model (Llama 3) was 99% accurate on arithmetic but failed on formatting the percentage output correctly. - The cost difference between the expensive, high-performing model and the cheaper model that failed the formatting task was 11x, illustrating the cost-awareness aspect of the routing. - The authors suggest that future AI systems should treat models as bundles of portable skills rather than monolithic entities, moving away from black-box trust. - Skill arbitration, like that proposed by Bella, is crucial for enterprise use cases where auditing the 'why' behind a decision is as important as the accuracy itself. ![Screenshot at 00:00: The initial visual features an animated graphic of two podcasters with the text overlay "BECOME A MEMBER TODAY!" over a grid background, signaling the start of the AI Papers Podcast Daily segment.](https://ss.rapidrecap.app/screens/ZGIIZUUJo14/00-00-00.jpg) **Context:** This discussion centers on a research paper titled "Trust by Design: Skill Profiles for Transparent, Cost-Aware LLM Routing," authored by researchers including Glen MacLellan from Georgia Tech. The core problem addressed is the current industry practice of evaluating and routing massive LLMs based on broad, single performance scores, which often fails in real-world enterprise applications where specific skills, cost constraints, and auditability are paramount. The paper introduces a new framework, exemplified by the system named 'Bella,' designed to overcome these limitations by creating granular skill profiles for different models. ## Detailed Analysis The video discusses the paper "Trust by Design," which criticizes the current reliance on general LLM benchmarks (like MMLU or F1 scores) for selecting models, labeling this the "Benchmark Trap." The authors argue that these metrics fail to capture the specific capabilities required for enterprise tasks, leading to either using models that are too expensive or models that fail critical, nuanced tasks. The proposed solution is a framework that creates 'skill profiles' for different LLMs, detailing their proficiency in specific cognitive abilities like numerical reasoning, fact extraction, and formatting. The system, named 'Bella,' acts as an intelligent router or orchestrator that analyzes a query's required skills and routes it to the most appropriate, cost-effective model. For example, when testing a financial query involving calculating a percentage change, the expensive, high-performing model (Llama 3) was found to be 99% accurate in the arithmetic but failed on the formatting requirement, while a cheaper model failed entirely on risk assessment. This highlights that models are often good at some things (like math) but bad at others (like specific output formatting). The authors advocate moving toward treating models as bundles of portable skills rather than opaque, monolithic entities, which is essential for building auditable and cost-efficient enterprise AI systems. ### The Problem with Benchmarks - The industry currently measures AI performance using general benchmarks like MMLU or F1 scores, which the paper critiques as the "Benchmark Trap" because they fail to reflect real-world task requirements. ### Introducing Skill Profiles - The paper proposes creating granular skill profiles that detail an LLM's proficiency in specific cognitive tasks like numerical reasoning, fact extraction, and formatting. ### The Bella Framework - Bella acts as an orchestrator that uses these skill profiles to route incoming prompts to the most efficient and cost-aware model for the job, avoiding the need to use expensive models for simple tasks. ### Financial Calculation Example - For a task requiring both arithmetic (80-50=30, then calculating percentage change) and specific output formatting, the expensive model was only 88% correct overall because it failed the formatting requirement, leading to an 11x cost difference compared to a cheaper, unsuitable model. ### The Cost and Auditability Trade-off - Relying on overly powerful models for simple tasks is costly (e.g., $10 per million tokens for the large model vs. $0.88 for the cheaper one), and without transparency, it creates black-box liability for enterprises. ### Skill Arbitrage - The goal is to move toward treating models as bundles of portable skills, allowing systems to dynamically route tasks to the best-suited component, which is essential for maintaining cost efficiency and explainability. ![Screenshot at 00:00: The opening visual features the podcast branding with two figures at microphones and the call to action "BECOME A MEMBER TODAY!" over a grid background.](https://ss.rapidrecap.app/screens/ZGIIZUUJo14/00-00-00.jpg) ![Screenshot at 00:50: The hosts introduce the core conflict: the trade-off between brilliant but expensive models versus cheap but inadequate ones, setting up the need for a better routing solution.](https://ss.rapidrecap.app/screens/ZGIIZUUJo14/00-00-50.jpg) ![Screenshot at 02:34: The speaker introduces the proposed solution, 'Bella,' which stands for Budget Efficient LLM Selection via Automated Skill Profiling.](https://ss.rapidrecap.app/screens/ZGIIZUUJo14/00-02-34.jpg) ![Screenshot at 05:36: The speaker illustrates the four stages of the Bella pipeline, starting with Step 1: Benchmarking and Data Collection.](https://ss.rapidrecap.app/screens/ZGIIZUUJo14/00-05-36.jpg) ![Screenshot at 09:14: A visual representation of a grid is shown, symbolizing the skill matrix where rows represent models \(Llama, GPT, Claude\) and columns represent specific skills.](https://ss.rapidrecap.app/screens/ZGIIZUUJo14/00-09-14.jpg)