# The Gemini Interactions API

Source: https://www.youtube.com/watch?v=aZgH_wnmedQ
Recap page: https://rapidrecap.app/video/aZgH_wnmedQ
Generated: 2025-12-16T13:03:52.829+00:00

---
## Quick Overview

Google DeepMind introduced the Interactions API as a unified interface for interacting with Gemini models and agents, simplifying state management, tool coordination, and extended tasks, featuring key capabilities like optional server-side state, an interpretable data model, background execution for long-running tasks, and support for remote Model Context Protocol (MCP) tools.

**Key Points:**
- The Interactions API provides a unified interface for interacting with Gemini models and agents, simplifying state management and tool coordination.
- Key features include optional server-side state to offload history management, potentially reducing costs via cache hits.
- It offers an interpretable and composable data model allowing debugging and reasoning over interleaved messages, thoughts, and tools.
- Background execution capability allows offloading long-running inference loops to the server without maintaining client-side connections.
- The API supports tool usage, including built-in tools like Google Search and Code Execution, and external Remote MCP tools.
- Multimodal capabilities are integrated, allowing users to pass inputs like images (base64 encoded) and receive multimodal outputs.
- Structured output via Pydantic models is supported, allowing developers to specify the exact JSON schema for the model's response.

![Screenshot at 00:09: The title slide of the announcement blog post, "Interactions API: A unified foundation for models and agents," establishing the core topic of the video: a new, unified API for interacting with Gemini models and agents.](https://ss.rapidrecap.app/screens/aZgH_wnmedQ/00-00-09.png)

**Context:** The video announces the release of the Gemini Interactions API, presented by Ali Çevik and Philipp Schmid from Google DeepMind, which serves as a unified foundation for building agentic applications with Gemini models. The presentation contrasts this new API with previous paradigms like simple completions and chat completions, highlighting new features designed to support complex, stateful, and multi-modal interactions, including tool use and background execution.

## Detailed Analysis

The Gemini Interactions API is introduced as a unified interface extending the core capabilities of the generateContent API, specifically designed for modern agentic applications. Key features include optional server-side state management, which simplifies client code by offloading history management to the server, potentially reducing costs through increased cache hits (00:37). It features an interpretable and composable data model for debugging complex agentic histories (00:40). Furthermore, it enables background execution, allowing long-running inference loops to run on the server without maintaining client-side connections (00:50). The API supports tool use, demonstrated with built-in tools like Google Search (21:54) and Code Execution (21:20), as well as remote Model Context Protocol (MCP) tools (22:48). Multimodal capabilities are also supported, allowing inputs like images (base64 encoded) and audio, with the ability to specify desired output modalities (02:55, 13:55). Structured output is enabled using Pydantic models, where developers define the exact JSON schema for the desired response (15:23). The video contrasts this with older methods, showing how stateful conversations now work by passing previous interaction IDs (09:38) and how stateless interactions can be handled by passing the full history as a list of dictionaries (12:27). The new API is positioned as a way to manage complexity, improve efficiency, and enable more sophisticated agentic workflows that were difficult or impossible with previous APIs.

### Introduction to Interactions API

- A unified interface for interacting with Gemini models and agents
- Simplifies state management, tool coordination, and extended tasks
- Replaces complexity of previous APIs

### Key Features

- Optional server-side state for history management
- Interpretable and composable data model
- Background execution for long-running tasks
- Remote MCP tool support

### Multimodal Capabilities

- Supports multimodal understanding (image, audio) and generation (image) via base64 encoding within inputs (12:54, 13:55)
- Example shows generating a futuristic cat image using the Nano Banana Pro model (14:37)

### Structured Outputs

- Uses Pydantic models for structured output via response_format="ModerationResult.model_json_schema()" (15:23)
- Example shows generating a JSON object for spam moderation (15:38)

### Tool Use (Function Calling)

- Demonstrates defining a tool (e.g., weather_service) and handling function calls (tool_call output type) by executing the function and sending the result back to the model (16:11)

### Agent Usage (Background Execution)

- Shows initiating a long-running research task using the 'deep-research-pro-preview-12-2025' agent with background=True and polling for status (19:15)

### Sources and Citations

- The API output includes extensive citations (20:55) and raw URLs (21:52) for grounded responses, though raw URLs are not clickable by default (21:54).

![Screenshot at 00:01: Title card announcing the topic: "Interactions API"](https://ss.rapidrecap.app/screens/aZgH_wnmedQ/00-00-01.png)
![Screenshot at 00:14: Demonstration of stateful conversation where the model remembers context by passing the previous interaction ID \(previous\_interaction\_id=interaction1.id\) \(09:38\)](https://ss.rapidrecap.app/screens/aZgH_wnmedQ/00-00-14.png)
![Screenshot at 00:38: Diagram illustrating the function calling loop: Developer defines tool, model calls tool, developer executes tool, sends result back to model \(01:51\)](https://ss.rapidrecap.app/screens/aZgH_wnmedQ/00-00-38.png)
![Screenshot at 02:00: Slide highlighting new tools and features in the Responses API, comparing capabilities with Chat Completions API \(02:16\)](https://ss.rapidrecap.app/screens/aZgH_wnmedQ/00-02-00.png)
![Screenshot at 13:58: Code snippet demonstrating multimodal input \(text + base64 image data\) using the gemini-3-pro-image-preview model \(13:06\)](https://ss.rapidrecap.app/screens/aZgH_wnmedQ/00-13-58.png)