Do We Always Need Query-Level Workflows? Rethinking Agentic Workflow Gen for Multi-Agent Systems

Quick Overview

Researchers propose the SCALE framework, which drastically reduces the token usage for designing multi-agent system workflows—by up to 83%—by replacing expensive, exhaustive testing with a novel surrogate evaluation method combining LLM self-prediction calibrated by a tiny fraction of real-world execution data.

Key Points: SCALE claims to cut token usage for agentic workflow generation by up to 83% with a performance degradation of less than 1%, offering a near 'free lunch' in LLM efficiency. The conventional query-level approach, which designs a custom workflow for every single question, incurs massive inference overhead because it pays to design a new architecture every time. Task-level workflow search is horrifyingly inefficient in the design phase, chewing through 10 million to 100 million tokens just to find one robust structure. The researchers found that running the single best static workflow five times achieved the same performance as generating a brand new custom query-level workflow for every query, suggesting adaptability gains are minimal. Performance of task-level workflows saturates very early in the search process, meaning systems waste millions of tokens searching for marginal improvements (e.g., going from 95.1% to 95.2% accuracy). SCALE's core innovation is surrogate evaluation: the LLM self-predicts workflow performance, which is then calibrated using a few-shot test on only 1% to 3% of the validation data. Without the crucial calibration step (the few-shot sanity check), the LLM's self-prediction failed, confirming that the small dose of reality anchors the entire process.

Context: The discussion centers on the hidden inefficiency plaguing enterprise AI: the massive computational cost associated with designing and optimizing agentic workflows for multi-agent systems, which is often overlooked compared to training costs. The paper analyzed, "Do we always need query level workflows? Rethinking agentic workflow generation for multi-agent systems," challenges the industry's obsession with hyper-customization and brute-force testing by comparing two established paradigms: the generalized, expensive-to-design task-level approach and the expensive-to-run query-level approach.