Voice to Agents to Sora 2: Gemini Computer Use + Realtime API + Claude Code

Quick Overview

The demonstration successfully orchestrates a multi-agent system combining Claude code agents (Sony and Blink), Gemini 2.5 computer vision, and the OpenAI real-time API to build a functional Sora video generation application end-to-end, proving that combining models and providers ("think in ands, not ors") maximizes capability.

Key Points: The engineer established two Claude code agents, Sony and Blink, commanded by a voice orchestrator agent, to work on a unified project involving Sora API integration and front-end styling. The winning strategy in generative AI is to "use them all," combining models like Gemini and OpenAI, and tools like Claude code next to OpenAI's real-time API, instead of optimizing for a single provider. Multi-agent observability provided a live pulse of agent activities, allowing the engineer to see an agent validating its work with the new Gemini 2.5 computer use model inside a tool. Agent Sony successfully completed its task by generating a full specification document for Sora video generation integration, which was then used by both agents for subsequent work via a custom slash command (/build). Agent Blink implemented front-end changes, inverting colors to black text on a white background and applying the Play font, and later corrected interference from Sony by comprehensively reviewing both front-end and back-end components. The system architecture includes an input layer (engineer/agents), a system layer leveraging the OpenAI real-time API and tools, and an output layer, emphasizing closed-loop design where outputs loop back into inputs for self-correction. The engineer demonstrated both voice-based and text-based prompting workflows, asserting that voice coding is "right around the corner" for controlling fleets of agents.

Context: The transcript details an experimental demonstration of a powerful multi-agent system designed to push the boundaries of autonomous software development, specifically focusing on integrating video generation capabilities using the anticipated Sora API. The setup involves an orchestrator voice agent controlling specialized agents, named Sony and Blink, which are built using Claude Code 2.5 and are capable of leveraging external models like Gemini 2.5 and APIs like OpenAI's real-time API to complete complex engineering tasks.