Build Hour: Voice Agents

Quick Overview

OpenAI's latest voice agent updates, including a new TypeScript SDK and improved audio models, enable more natural and personalized voice interactions, allowing developers to integrate real-time voice interfaces into web applications with greater ease and functionality, exemplified by a home remodeling workspace demo.

Key Points: OpenAI released four new updates for building voice agents, complementing their March 2025 audio model releases, to make agents sound more like real representatives and less like automated bots. The new generation of voice agents offers increased flexibility to handle ambiguity, accessibility for use during commutes or activities, and personalization through expressive vocal cues like tone and cadence. Two primary approaches for building voice applications are discussed: a 'chained approach' using separate speech-to-text and text-to-speech models, and a more novel approach using futuristic speech-to-speech models that understand audio natively. Key recent launches include a TypeScript agents SDK with real-time API support, integration of real-time models into the platform's traces tab for easier debugging, and an improved model snapshot for the real-time API with a new speed parameter. The demo showcased building a home remodeling workspace using a workspace agent and a designer agent, demonstrating how agents can be specialized, use tools like 'make workspace changes' and 'search the web', and hand off tasks to each other. Agent development best practices include breaking agents into well-defined roles, starting small with one agent, and incrementally adding complexity, with prompts being crucial for defining identity, tasks, demeanor, and tone. For stability and safety, output guardrails can be implemented to prevent agents from discussing off-brand topics, with the system capable of apologizing and correcting itself when a guardrail is tripped.

Context: This video focuses on OpenAI's advancements in voice agents, featuring solutions architects Brian and Brashant. The session aims to guide developers in scaling their companies with OpenAI's APIs and models, highlighting recent updates to voice agent capabilities and providing a practical demonstration of adding a voice interface to a web application.