# Evals in Action: From Frontier Research to Production Applications

Source: https://www.youtube.com/watch?v=YEaKXjHENyQ
Recap page: https://rapidrecap.app/video/YEaKXjHENyQ
Generated: 2025-10-08T17:35:27.041+00:00

---
## Quick Overview

OpenAI's Evals framework, exemplified by GDPval, allows developers to rigorously evaluate frontier AI models on real-world, economically valuable tasks using human expert grading, leading to the creation of better, more reliable products with improved integration, automation, and ease of use.

**Key Points:**
- GDPval is a new evaluation framework that measures model performance economically valuable real-world tasks across 9 occupations, starting with 1000+ tasks covering real estate, CAD design, retail, and more.
- The evaluation process uses pairwise expert grading, where human experts compare a model's output against a human-gold deliverable for a given prompt, determining a win rate.
- GPT-4o achieved a win rate of approximately 17% (wins and ties) on the AIME 2025 competition math benchmark without tools, compared to 42.1% without thinking.
- The GDPval framework is designed to be easy to use, allowing developers to define custom graders (like the 'Scoring grader' for the 'Fundamental_analyst' agent) and run evaluations on their own data sets.
- The speakers encourage a development philosophy of 'Start simple' (few inputs + simple grader), 'Use human data' (real user inputs, not hypotheticals), and 'Annotate & align' (aligning graders and adding SMEs).
- The tool supports third-party models via OpenRouter and enterprise features like Zero Data Retention and Enterprise Key Management.
- The development team is working on automating prompt optimization based on trace analysis to further accelerate iteration cycles and focus developer effort on high-value tasks.

![Screenshot at 00:09: Tejal Patwardhan introduces the talk titled 'Evals in Action: From Frontier Research to Production Applications,' setting the stage for discussing how OpenAI measures real-world model progress.](https://ss.rapidrecap.app/screens/YEaKXjHENyQ/00-00-09.png)

**Context:** The video presents Tejal Patwardhan and Henry Scott-Green from OpenAI detailing their Evals framework, specifically highlighting the GDPval (Gross Domestic Product value) evaluation system. This system moves beyond academic benchmarks like the AIME math competition to measure AI model performance on tasks relevant to economically valuable real-world work, such as financial analysis and real estate brochure design, using expert human pairwise grading to guide model improvement.

## Detailed Analysis

Tejal Patwardhan and Henry Scott-Green discuss the importance of moving AI evaluation beyond traditional academic benchmarks to real-world, economically valuable tasks using the GDPval framework. GDPval measures model performance across 9 sectors contributing to US GDP, starting with 1000+ tasks like financial analysis and CAD design. The core evaluation method is pairwise expert grading, where human experts compare two model outputs (e.g., GPT-4o vs. a human expert) against defined criteria to determine a preferred output, which is then aggregated into a win rate over time. The presentation shows that frontier models, like GPT-4o, are rapidly closing the gap with industry experts on these complex tasks, though still below parity (around 50%). The framework is designed for ease of use, supporting third-party models via OpenRouter, and includes enterprise features like zero data retention. The speakers emphasize a development philosophy: start simple with few inputs and a simple grader, use real user data, and iterate by annotating generations and aligning graders. Henry Scott-Green demonstrates the Agent Builder workflow and the trace inspection tool, which allows developers to debug multi-agent systems by analyzing performance bottlenecks in individual steps, accelerating the iteration loop for building high-performing, reliable AI agents.

### Introduction to Evals & GDPval

- Tejal introduces the concept of Evals in Action, focusing on GDPval which measures model performance on economically valuable tasks across 9 key sectors
- GDPval shows GPT-4o achieving sub-20% win rate vs. experts in Spring 2024, highlighting the need for better evaluation methods.

### The GDPval Methodology

- The process involves pairwise expert grading comparing a human gold deliverable against a GPT-5 sample response, picking the better one based on subjective/objective criteria
- This yields an overall win rate that tracks progress over time, showing models approaching parity with industry experts.

### Real-World Task Examples

- Examples include a Real Estate Agent designing a sales brochure and a Manufacturing Engineer designing a 3D model of a cable reel stand, showcasing multimodal and complex task capabilities.

### The Three Pillars of Evals

- Future development focuses on Integration (third-party model support via OpenRouter), Automation (automated prompt optimization), and Ease of Use (simple agent building and trace debugging).

### Agent Builder and Trace Inspection Demo

- Henry demonstrates the visual Agent Builder workflow for complex agents (like the Investment Report Generator) and the trace inspection tool to debug performance bottlenecks in individual steps, enabling faster iteration and debugging.

![Screenshot at 00:09: Tejal Patwardhan begins her presentation on 'Evals in Action: From Frontier Research to Production Applications' at OpenAI DevDay.](https://ss.rapidrecap.app/screens/YEaKXjHENyQ/00-00-09.png)
![Screenshot at 00:42: Slide listing the two main goals of Evals: Measure real-world progress and provide Practical tools for strong evaluations.](https://ss.rapidrecap.app/screens/YEaKXjHENyQ/00-00-42.png)
![Screenshot at 01:22: A bar chart showing competition math performance \(AIME 2025\), where models with 'thinking' significantly outperform those without, demonstrating the value of chain-of-thought reasoning.](https://ss.rapidrecap.app/screens/YEaKXjHENyQ/00-01-22.png)
![Screenshot at 02:50: Example task for a Real Estate Agent: designing a sales brochure for a new DC property using provided images and research.](https://ss.rapidrecap.app/screens/YEaKXjHENyQ/00-02-50.png)
![Screenshot at 04:57: A chart titled 'GDPval: OpenAI frontier model performance over time,' showing GPT-4o's win rate against industry professionals around 15% in Spring 2024.](https://ss.rapidrecap.app/screens/YEaKXjHENyQ/00-04-57.png)
![Screenshot at 07:39: A slide summarizing the talk structure: Limitations, Future work, and Why it matters.](https://ss.rapidrecap.app/screens/YEaKXjHENyQ/00-07-39.png)
![Screenshot at 09:11: A slide summarizing the desired qualities of Evals: Integration, Automation, and Ease of use.](https://ss.rapidrecap.app/screens/YEaKXjHENyQ/00-09-11.png)
![Screenshot at 11:08: Henry Scott-Green shows the 'Trace grading' interface, a tool for interpreting agent execution traces and applying granular feedback.](https://ss.rapidrecap.app/screens/YEaKXjHENyQ/00-11-08.png)
![Screenshot at 13:04: A visual representation of the 'Investment report generator' agent workflow in the Agent Builder, showing interconnected agent nodes.](https://ss.rapidrecap.app/screens/YEaKXjHENyQ/00-13-04.png)
![Screenshot at 15:56: Henry Scott-Green discusses the three key focus areas for future Evals development: Integration, Automation, and Ease of use.](https://ss.rapidrecap.app/screens/YEaKXjHENyQ/00-15-56.png)