AI Is Suddenly Surprisingly Good At Physics

Quick Overview

The video concludes that while Large Language Models (LLMs) like GPT-4 and Claude are improving in logic and math, they still lack the necessary ability to reason, validate, or produce truly novel scientific breakthroughs, making the bottleneck for progress the ability to properly value and validate scientific output rather than just generating more intelligence. The speaker highlights recent claims of AI making physics discoveries, referencing papers from startups like Extensity AI and Sakana AI, but ultimately expresses skepticism about true scientific reasoning capabilities in current LLMs, evidenced by anecdotal reports of failures and the need for human oversight.

Key Points: The main bottleneck in scientific progress is the ability to properly value and validate scientific output, not a lack of intelligence, according to researcher Bojan Tunguz. Startups like Extensity AI claim to use AI (GPT-4/Codex) with neuro-symbolic software to analyze gravitational wave data for quantum gravity evidence, which the speaker finds impressive. Google's AI co-scientist, built with Gemini 2.0, is being used to generate novel hypotheses and accelerate scientific discoveries, though the speaker notes they ran a trial and received no response. Harvard professor David Sinclair claims his lab uses a novel AI system to make non-intuitive scientific discoveries and write up full PhD-level papers without human intervention. A tweet from Asher suggests that claims of GPT-5 making novel research discoveries (happening twice in one week) are strange, implying potential overstatement or lack of independent validation. The speaker notes that while LLMs are good at logic and math (as confirmed by Ekin Dogus Cubuk), they do not yet possess genuine reasoning capability, which is crucial for complex physics research. The video concludes by showing an example of a custom star map product from Under Lucky Stars, suggesting that AI is currently better at generating personalized, creative products than fundamental scientific leaps.