How A Team Of 7 Keeps Breaking AI Benchmark Records

Quick Overview

Poetiq, co-founded by Ian Fischer, achieves superior AI performance by using a recursively self-improving system that utilizes reasoning strategies instead of relying solely on massive prompt engineering or fine-tuning existing foundation models, allowing them to significantly outperform models like Gemini 1.5 Flash on benchmarks such as Humanity's Last Exam.

Key Points: Poetiq's recursive self-improving system, which incorporates reasoning strategies, allows them to outperform models like Gemini 1.5 Flash on specific benchmarks. The company's latest paper showed a 9% improvement over Gemini 3 DeepMind on 'Humanity's Last Exam,' scoring 53.1% versus 44%. The training cost for the winning model was significantly lower, around $70 to $80 per problem, compared to the massive expense of training foundation models from scratch. Fischer noted that in the past, machine learning required knowing your dataset well, but Poetiq's approach outsources this data understanding to the AI agent itself. The system generates its own reasoning strategies and can iteratively self-improve, leading to better performance than relying solely on prompt engineering or fine-tuning. Fischer, formerly a researcher at Google DeepMind, co-founded Poetiq with Shameet Baluja, and previously founded a mobile dev tools company acquired by YC.

Context: The video features a discussion on the Y Combinator (YC) podcast, Lightcone, with Ian Fischer, Co-founder & Co-CEO of Poetiq, alongside YC partners Diana Hu, Harj Taggar, Garry Tan, and Jared Friedman. Fischer explains Poetiq's approach to Artificial Intelligence, which focuses on recursive self-improvement and reasoning strategies, contrasting it with the prevailing methods of massive pre-training or prompt engineering of large language models (LLMs).

Detailed Analysis

Ian Fischer, Co-founder & Co-CEO of Poetiq, discusses how the world is changing quickly due to AI, referencing his experience building an iPhone app last summer using GPT-4, which was fast and easy compared to a decade ago. He explains that Poetiq is focused on building a recursively self-improving system, which he calls the 'Holy Grail of AI,' where the AI makes itself smarter. This approach allows them to achieve better performance than competitors who rely on training massive LLMs from scratch, which is extremely costly (hundreds of millions of dollars) and time-consuming (months of effort). Poetiq's method results in systems that solve hard problems more cheaply and quickly. When asked about the difference from RL or context engineering, Fischer explains that Poetiq's systems, which they call the Poetiq Meta-System, generate their own reasoning strategies. He highlights their recent success on 'Humanity's Last Exam,' where they scored 53.1% compared to Gemini 1.5 Flash's 44%—a significant margin achieved with far lower optimization costs (under $100k). He contrasts this with the traditional method where engineers had to meticulously know and curate the dataset; Poetiq outsources this understanding to the AI agent itself. The recursive process means that as the Poetiq Meta-System improves, it can find better reasoning strategies, leading to continual performance gains, even as foundation models like GPT-4 release newer versions.