# Capabilities Ain’t All You Need: Measuring Propensities in AI Source: https://www.youtube.com/watch?v=qEU6mnZ9DMA Recap page: https://rapidrecap.app/video/qEU6mnZ9DMA Generated: 2026-02-25T15:07:37.778+00:00 --- ## Quick Overview A new paper titled "Capabilities Ain’t All You Need: Measuring Propensities in AI" introduces a mathematical framework, the two-sided two-parameter logistic model (2x2 PL), to rigorously measure behavioral traits or "propensities" in AI models, arguing that raw capability scores are insufficient for predicting task success because behavior (like risk aversion or agreeableness) must align with the task's specific non-monotonic demands. **Key Points:** - The paper, released February 23, 2026, by researchers including teams from the Valencian Research Institute of AI and the University of Cambridge, shifts evaluation focus from pure capability ("Can it?") to personality and behavior ("Will it and how will it behave?"). - Capabilities are defined as monotonic traits where more is always better (e.g., reasoning speed), while propensities are non-monotonic, meaning success occurs within an "ideal band" or Goldilocks zone (e.g., too little or too much risk aversion causes failure). - The core contribution is the two-sided two-parameter logistic model (2x2 PL), which uses a blogistic formulation, mathematically multiplying two curves to define a minimum and maximum limit for a trait, acting like an audio band pass filter for behavior. - The researchers identified four key behavioral dimensions: risk aversion, red versus blue preference (bias check), introversion versus extroversion (verbosity/tone), and ultracrepidarianism (giving confident opinions outside the knowledge base). - Combining capability scores with propensity scores using the 2x2 PL framework significantly skyrocketed prediction accuracy on held-out tasks compared to using capability scores alone, mathematically validating the theory. - Implementation relies on Demand Level Annotation (DeLion), where models like GPT 4.1 define the required propensity ranges for specific prompts, though this introduces 'annotation dependency' as a systemic risk if the judge model has biases. - The framework suggests future alignment and debugging will involve tuning propensity bands (like adjusting a suspension system or equalizer) for specific industries, moving away from monolithic general intelligence toward specialized temperament. **Context:** The discussion analyzes a significant research paper titled "Capabilities Ain’t All You Need: Measuring Propensities in AI," which deliberately riffs on the foundational 'Attention Is All You Need' paper, signaling a major shift in AI evaluation standards. The context involves moving beyond traditional benchmarks obsessed with raw horsepower and parameter counts, which rely on Item Response Theory (IRT) that assumes a linear relationship, to address the necessary behavioral alignment required for real-world task success in AI systems. ## Detailed Analysis The central argument is that high intelligence or capability alone fails to guarantee task success because models exhibit measurable behavioral tendencies called propensities, which are non-monotonic traits requiring an ideal behavioral band rather than maximum output. The researchers formalized this with the two-sided two-parameter logistic model (2x2 PL), which maps behavior onto a bell-curve-like structure defined by a low cut and a high cut, similar to a band pass filter, ensuring the model's trait falls within the necessary range for the specific prompt—be it creative (loose suspension) or legal (stiff suspension). They tested this on dimensions including risk aversion, bias preference, verbosity, and ultracrepidarianism (giving confident advice outside one's domain), finding that these traits act as stable latent variables across different prompts. The practical validation showed that combining capability scores with propensity scores dramatically improved prediction accuracy for model performance, leading to the conclusion that future AI development will focus on tuning these behavioral profiles, potentially debugging models by adjusting propensity bands rather than treating failures solely as software bugs or knowledge gaps, moving AI evaluation toward an age of 'AI psychology.' ### Evaluation Shift - Moving from Capability to Personality - The industry has been focused on raw horsepower and leaderboard rankings based on massive multitask benchmarks - The paper introduces the question: 'will it and perhaps more importantly how will it behave while doing it?' ### Capability vs. Propensity Geometry - Capabilities are monotonic (more is always better, line goes up and right) - Propensities are non-monotonic, success happens in the 'ideal band' in the middle, like risk aversion where too much or too little leads to failure (recklessness or paralysis). ### The 2x2 PL Framework - Researchers introduced the two-sided two-parameter logistic model (2x2 PL) - It uses a blogistic formulation, mathematically multiplying two curves to define a minimum and maximum behavioral limit, analogous to a band pass filter in audio processing. ### Behavioral Dimensions Tested - Four dimensions analyzed were risk aversion, red verse blue preference (arbitrary bias), introversion/extroversion (verbosity/tone), and ultracrepidarianism (giving confident opinions outside knowledge base). ### Implementation and Limitations - The required propensity levels are determined via Demand Level Annotation (DeLion), using models like GPT 4.1 to set numerical ranges for traits - This method suffers from 'annotation dependency,' meaning the resulting ideal band is only as good as the judging model's inherent biases. ### Impact on Enterprise and Safety - The framework allows for model specialization by actively tuning propensity bands for specific industries, such as setting ultra-repudarianism to zero for legal work - This addresses the 'refusal problem' or 'safety paralysis' by diagnosing it as a propensity tuning issue rather than a capability flaw.