Capabilities Ain’t All You Need: Measuring Propensities in AI

Quick Overview

A new paper titled "Capabilities Ain’t All You Need: Measuring Propensities in AI" introduces a mathematical framework, the two-sided two-parameter logistic model (2x2 PL), to rigorously measure behavioral traits or "propensities" in AI models, arguing that raw capability scores are insufficient for predicting task success because behavior (like risk aversion or agreeableness) must align with the task's specific non-monotonic demands.

Key Points: The paper, released February 23, 2026, by researchers including teams from the Valencian Research Institute of AI and the University of Cambridge, shifts evaluation focus from pure capability ("Can it?") to personality and behavior ("Will it and how will it behave?"). Capabilities are defined as monotonic traits where more is always better (e.g., reasoning speed), while propensities are non-monotonic, meaning success occurs within an "ideal band" or Goldilocks zone (e.g., too little or too much risk aversion causes failure). The core contribution is the two-sided two-parameter logistic model (2x2 PL), which uses a blogistic formulation, mathematically multiplying two curves to define a minimum and maximum limit for a trait, acting like an audio band pass filter for behavior. The researchers identified four key behavioral dimensions: risk aversion, red versus blue preference (bias check), introversion versus extroversion (verbosity/tone), and ultracrepidarianism (giving confident opinions outside the knowledge base). Combining capability scores with propensity scores using the 2x2 PL framework significantly skyrocketed prediction accuracy on held-out tasks compared to using capability scores alone, mathematically validating the theory. Implementation relies on Demand Level Annotation (DeLion), where models like GPT 4.1 define the required propensity ranges for specific prompts, though this introduces 'annotation dependency' as a systemic risk if the judge model has biases. The framework suggests future alignment and debugging will involve tuning propensity bands (like adjusting a suspension system or equalizer) for specific industries, moving away from monolithic general intelligence toward specialized temperament.