Evolutionary Strategies lead to Catastrophic Forgetting in LLMs

Quick Overview

Evolutionary Strategies (ES) lead to catastrophic forgetting in Large Language Models (LLMs) when applied to sequential tasks, demonstrated by ES-trained models failing to retain knowledge from prior tasks, unlike gradient-based methods which maintain performance across tasks.

Key Points: Evolutionary Strategies (ES) cause catastrophic forgetting in LLMs when applied to sequential tasks, unlike gradient-based methods. ES models showed a drastic drop in performance on the initial task (math game 'Countdown') after being trained on a new task (image editing/masking). The forgetting magnitude for ES was 1000 times larger than for gradient-based methods (like GRPO) on the same task sequence. The researchers suggest ES models fail because they lack mechanisms to compartmentalize knowledge, causing new learning to overwrite old knowledge globally. ES relies on random noise/perturbations for updates, which aggressively affects all parameters, whereas gradient methods focus updates more locally. The study used a small population size (30) for ES evaluation, which the authors note may exacerbate instability but still shows a fundamental issue. The core challenge is developing an ES variant that can selectively update parameters to avoid forgetting while learning new skills, similar to biological evolution's balance.

Context: The video analyzes a recent research paper from UC Berkeley/Abadi group that investigates the stability and generalization of Evolutionary Strategies (ES) when used for continuously learning and updating Large Language Models (LLMs). The core problem discussed is 'catastrophic forgetting,' where a model loses performance on previously learned tasks after being trained on new ones. The paper compares ES methods against standard gradient-based optimization methods (like GRPO) to see how they handle sequential task learning.

Detailed Analysis

The video discusses research demonstrating that Evolutionary Strategies (ES) cause catastrophic forgetting in LLMs when applied to sequential learning tasks, unlike gradient-based methods. The core finding is that ES aggressively updates the entire model structure based on random noise perturbations, leading to the overwriting of previously acquired knowledge. Specifically, when ES-trained models were tested on the initial math game 'Countdown' after being updated on an image editing task, their performance plummeted, showing a memory degradation 1000 times greater than gradient-based methods like GRPO. The research suggests that because ES updates are global and rely on random noise, they lack the necessary compartmentalization mechanisms found in gradient-based methods, which can selectively update parameters. While gradient methods also suffer from forgetting, their performance shift is much less severe, maintaining a baseline competence. The researchers posit that for ES to be viable for continuous learning, a mechanism must be developed to mimic biological evolution's ability to retain old skills while acquiring new ones, rather than completely eroding past knowledge with every update.