# this experiment could END the AI hype

Source: https://www.youtube.com/watch?v=EWAUutf9xKQ
Recap page: https://rapidrecap.app/video/EWAUutf9xKQ
Generated: 2025-12-04T05:34:34.749+00:00

---
## Quick Overview

The ProFIT framework, which uses LLMs within an evolutionary feedback loop to generate and test trading strategies against historical data, significantly outperformed human-written strategies and random baselines in the Alpha Arena Season 1.5 competition, securing the top spot with a 12.11% aggregate return over two weeks, while most other models lost money.

**Key Points:**
- The Mystery Model (ProFIT framework) won Alpha Arena Season 1.5 with a 12.11% aggregate return over two weeks, while most other LLMs lost money.
- ProFIT is a large-language-model-driven evolutionary search framework that embeds LLMs within an evolutionary feedback loop, allowing them to iteratively improve trading strategies.
- The framework tests generated code modifications against historical data (backtesting) and uses a Minimum Acceptance Score (MAS) to filter viable candidates, avoiding local optima found in previous self-improving systems.
- In the competition, 32 instances of LLMs managed $320,000 in real capital, ingesting news, indices, and microstructure data every 6 minutes.
- The paper highlights that while LLMs excel at memorization tasks (like academic benchmarks), they struggle in complex, non-stationary environments like financial markets, which Alpha Arena is designed to simulate.
- The ProFIT framework consistently outperformed random and Buy-and-Hold strategies across seven liquid futures assets, achieving a mean annualized return increase of +44.21% over the seed strategy.
- The video concludes by posing the question of whether this approach will continue to scale or if the market conditions will change, making the historical data-based optimization obsolete.

![Screenshot at 00:43: The opening slide displays the update that the official competition has ended, announcing that the 'Mystery Model' was the winner with a 12.11% aggregate return in 2 weeks, contrasting with the general poor performance of other models.](https://ss.rapidrecap.app/screens/EWAUutf9xKQ/00-00-43.png)

**Context:** The video discusses the results of Alpha Arena Season 1.5, a competition where large language models (LLMs) were tasked with developing profitable trading strategies using real capital. The speaker analyzes why the winning model, named 'Mystery Model' (powered by the ProFIT framework), succeeded where others failed, particularly highlighting the limitations of standard LLM benchmarks and the novelty of ProFIT's evolutionary, self-improving code generation approach.

## Detailed Analysis

The video analyzes the conclusion of the Alpha Arena Season 1.5 competition, revealing that the 'Mystery Model' won with a 12.11% aggregate return, while most other LLMs lost money. The speaker explains that the winning approach is ProFIT, a framework that embeds LLMs within an evolutionary feedback loop. This system uses the LLM (like GPT-4) to generate code modifications (mutations) to existing trading strategies, which are then rigorously tested against historical market data (backtesting). Key to its success is avoiding local optima by prioritizing changes that show long-term potential, inspired by evolutionary techniques like the Darwin Gödel Machine and the Huxley-Gödel Machine concept. The competition involved 32 LLM instances managing $320,000 in real capital, trading US equities, indices, and market microstructure data. The paper abstract confirms ProFIT outperforms Buy-and-Hold in over 77% of all evolved strategy-asset combinations, showing generalizable performance gains rather than overfitting to past data. The speaker concludes by posing the significant open question: will this approach continue to yield profits in real-time, dynamic markets, or will the models fail when market conditions change, as implied by the limitations of current benchmarks that reward 'hill-climbing' performance on static tasks?

### Alpha Arena Season 1.5 Results

- Mystery Model won with 12.11% aggregate return
- Most LLMs lost money
- Models traded $320K in real capital

### ProFIT Framework Overview

- Embeds LLMs in an evolutionary feedback loop to generate and test trading code modifications
- Uses backtesting against historical data for evaluation

### Key Innovations

- Uses code-level mutation, self-analysis, and walk-forward validation
- Employs Clade Metaproductivity to prioritize changes with long-term potential, avoiding local optima

### Paper Context (ProFIT)

- A large-language-model-driven evolutionary search for automated discovery and improvement of algorithmic trading strategies in financial markets.

### Comparison to Other Systems

- ProFIT outperforms Buy-and-Hold in over 77% of evolved strategy-asset combinations
- Inspirations from Darwin Gödel Machine and Huxley-Gödel Machine concepts

### Prophet Arena Leaderboard (Brier Score)

- GPT-5 (High) ranked 1st with 0.8230 Brier Score, indicating strong probabilistic prediction accuracy.

### Open Question

- Whether this historical backtesting approach remains viable in real-time, non-stationary market environments without overfitting.

![Screenshot at 00:00: A man wearing headphones discusses performance charts comparing various LLM trading strategies over time.](https://ss.rapidrecap.app/screens/EWAUutf9xKQ/00-00-00.png)
![Screenshot at 00:43: A tweet summary showing the final results of Alpha Arena S1.5, highlighting the Mystery Model's 12.11% aggregate return.](https://ss.rapidrecap.app/screens/EWAUutf9xKQ/00-00-43.png)
![Screenshot at 01:01: A tweet from Jay A detailing the competition setup: 32 LLMs managing $320k in real capital, ingesting data every 6 minutes.](https://ss.rapidrecap.app/screens/EWAUutf9xKQ/00-01-01.png)
![Screenshot at 01:45: A switch to the Prophet Arena leaderboard showing GPT-5 \(High\) leading in Brier Score \(0.8230\) and other models trailing.](https://ss.rapidrecap.app/screens/EWAUutf9xKQ/00-01-45.png)
![Screenshot at 03:34: A tweet from Julian Togelius showing an image of a beet and a hoe, referencing his 'Rogue-like' interest and a new paper on using LLMs to generate trading policies via evolutionary framework.](https://ss.rapidrecap.app/screens/EWAUutf9xKQ/00-03-34.png)