How Gemini Canvas using Flash 3.0 made me worry about autonomous AI

Quick Overview

The speaker's worry about autonomous AI stems from interacting with Gemini Canvas (using Flash 3.0) for a physics simulation project, where the AI demonstrated self-modification capabilities but exhibited poor context retention, leading to forgotten improvements and errors, which suggests profound risks if such behavior were present in critical software development scenarios.

Key Points: Hans Beers used Gemini Canvas (Flash 3.0) to create a physics simulation lab based on laws of physics, initially resulting in a promising simulator. The speaker added two 'self modifying code' functions to the AI: improving existing simulations and generating ideas for new ones. Gemini implemented these functions, automatically building an API key interface to evaluate them. A major limitation observed was that Gemini frequently forgets context; when improving an existing simulation or adding a new one, other improvements are forgotten, preventing the creation of reliable, error-free runs. The speaker notes this lack of persistent context retention is concerning, as it mirrors a potential failure mode for autonomous agents in any software development scenario. The speaker later experimented with the Agentic AI Simulator, observing that increasing self-modification led to high risk levels and system instability, such as introducing infinite loops and brute-forcing cloud instances. This entire experiment led the speaker to reflect seriously on the risks of fully autonomous, self-improving AI systems lacking reliable context memory.

Context: Hans Beers, an IT professional since 1985, describes an experiment using Google's Gemini AI within its 'Canvas' environment (which utilizes Flash 3.0 technology) to assist in developing a complex physics simulation lab. The goal was to test the AI's capability for self-improvement, specifically adding functions that allow the AI to modify its own code and generate new ideas. The presentation contrasts this early experience with the capabilities and risks outlined in a concurrent presentation (39C3 - AI Agent, AI Spy) regarding autonomous systems.