Trust by Design: Skill Profiles for Transparent, Cost-Aware LLM Routing

Quick Overview

The paper "Trust by Design" proposes a framework that moves AI routing away from relying solely on general benchmarks towards skill-based arbitration, ensuring that queries are sent to the most appropriate, cost-effective, and transparent Large Language Model (LLM) for the specific task, which is necessary for enterprise adoption where cost and auditability are critical factors.

Key Points: The paper argues against relying solely on overall benchmarks (like MMLU or F1 scores) for routing LLMs, calling it the "Benchmark Trap." It proposes a framework based on skill profiles, which map specific cognitive abilities (like numerical reasoning, data extraction, instruction following) to an LLM. The proposed solution, called 'Bella,' acts as an orchestrator that uses skill profiles to route queries to the most proficient and cost-effective model for the task. For the financial task of calculating a percentage change ($50 to $80), the best model (Llama 3) was 99% accurate on arithmetic but failed on formatting the percentage output correctly. The cost difference between the expensive, high-performing model and the cheaper model that failed the formatting task was 11x, illustrating the cost-awareness aspect of the routing. The authors suggest that future AI systems should treat models as bundles of portable skills rather than monolithic entities, moving away from black-box trust. Skill arbitration, like that proposed by Bella, is crucial for enterprise use cases where auditing the 'why' behind a decision is as important as the accuracy itself.

Context: This discussion centers on a research paper titled "Trust by Design: Skill Profiles for Transparent, Cost-Aware LLM Routing," authored by researchers including Glen MacLellan from Georgia Tech. The core problem addressed is the current industry practice of evaluating and routing massive LLMs based on broad, single performance scores, which often fails in real-world enterprise applications where specific skills, cost constraints, and auditability are paramount. The paper introduces a new framework, exemplified by the system named 'Bella,' designed to overcome these limitations by creating granular skill profiles for different models.