Shaping Model Behavior in GPT-5.1— the OpenAI Podcast Ep. 11

Quick Overview

The OpenAI team shaped model behavior in GPT-5.1 by implementing a system that balances maximizing user freedom with minimizing harm, primarily through improved instruction following, context window management, and continuous iteration based on user feedback, which resulted in models that feel warmer and are more steerable than previous versions.

Key Points: All models in the GPT-5.1 chat are reasoning models, meaning they can decide how much to think based on the prompt, unlike earlier models. The primary goals for GPT-5.1 included addressing feedback on GPT-5, such as making models feel warmer and improving steering capabilities without sacrificing steerability. A key technical change was introducing an auto-switcher that moves between the standard chat model and reasoning models, allowing for complex reasoning tasks to be handled by specialized models. The team focused on improving instruction following, context window capacity, and ensuring models could handle subtle cues across subjective domains like personality and tone. The work on personality features, like style and tone, is seen as a crucial step toward creating more usable and helpful models for all users. Data gathered from user feedback, including screenshots of problematic interactions, is used to guide iterative improvements, ensuring models adhere to safety boundaries while maintaining utility.

Context: This episode of The OpenAI Podcast features host Andrej Karpathy interviewing Christina Kim, a Research Lead at OpenAI working on post-training for models like GPT-5.1, and Laurentia Romaniuk, a Product Manager focused on model behavior. The discussion centers on the engineering and research efforts involved in shaping the behavior of the new GPT-5.1 model, specifically focusing on techniques to make the AI more steerable, less 'cold,' and better at following complex instructions while minimizing harmful outputs.

Detailed Analysis

Andrej Karpathy introduces guests Christina Kim (Research Lead, Post-training) and Laurentia Romaniuk (Product Manager, Model Behavior) to discuss shaping model behavior for GPT-5.1. Kim notes that a major goal was to make models feel 'warmer' and more steerable while maintaining performance gains. A key development in GPT-5.1 is that all models in the chat are now reasoning models, capable of deciding how much computation to spend on a prompt based on its complexity, unlike previous models. Romaniuk explains that part of their work involves creating a 'conversation link' to see exactly what happened during a user's interaction, which helps debug issues where the model might output something cold or inappropriate, like refusing to discuss sensitive topics or giving overly clinical answers. Karpathy mentions the analogy of System 1 vs. System 2 thinking from Daniel Kahneman to describe the difference between the base model and the reasoning models. Kim details that the team uses user feedback, including screenshots of negative experiences, to guide iterative improvements. She emphasizes that the goal is not to make the model perfectly safe by refusing all borderline requests, but rather to allow for maximum user freedom while minimizing harm. They are constantly working on instilling an understanding of context and intent so the model can better navigate subjective domains like personality and tone. Kim highlights that they are trying to engineer models that proactively help users by inferring intent, rather than simply reacting to explicit instructions. She mentions that one of the key improvements in 5.1 is the auto-switcher mechanism that directs queries between the standard chat model and the more capable reasoning models, allowing for more nuanced responses. Karpathy asks how they measure progress in shaping personality, and Kim explains that they look at signals from user interactions, such as whether the model's response style matches the user's expectations (e.g., warm vs. cold, detailed vs. concise). She notes that the inherent challenge is balancing user freedom (maximizing freedom) with minimizing harm, which requires constant iteration and tuning the context window to carry the right amount of information forward.