# Scaling Agentic Inference Across Heterogeneous Compute [Zain Asgar] - 757

Source: https://www.youtube.com/watch?v=GI0hWajiPaM
Recap page: https://rapidrecap.app/video/GI0hWajiPaM
Generated: 2025-12-02T22:34:50.738+00:00

---
## Quick Overview

Zain Asgar confirms that Gimlet Labs has not publicly launched its cloud product, but their focus on heterogeneous compute, particularly utilizing commodity hardware like H100s and B200s, allows them to achieve significant cost and performance benefits over specialized hardware by optimizing workload placement across different accelerators and efficiently managing memory and compute resources.

**Key Points:**
- Gimlet Labs has not publicly launched its cloud product yet, though early access customers are using it.
- The core challenge addressed by Gimlet is scaling agentic inference across heterogeneous compute environments (like AWS, Google Cloud, and Azure).
- The company aims to optimize workload execution by dynamically allocating tasks to the most cost-effective and performant hardware available, which includes both cutting-edge GPUs (like H100s) and older hardware.
- A key mechanism involves using an agentic graph system that optimizes workload placement and resource allocation based on cost economics and performance requirements.
- They have seen performance improvements of 20-40% in some cases by optimizing kernel execution and memory management between different hardware types.
- The custom compiler layer in their stack is crucial for achieving this optimization, allowing them to manage complexity that generic frameworks often struggle with.
- The cost savings and performance gains are significant enough that they are actively working with customers to prove the unit economics before broader release.

![Screenshot at 00:01: Zain Asgar begins discussing how training hardware has trended toward building supercomputers, contrasting this with the need for efficient inference across heterogeneous compute.](https://ss.rapidrecap.app/screens/GI0hWajiPaM/00-00-01.png)

**Context:** This interview segment features Zain Asgar, Co-Founder & CEO of Gimlet Labs and an adjunct professor at Stanford University, speaking with host Sam Charrington on the TWiML AI Podcast. The discussion centers on the technical challenges and architectural solutions Gimlet Labs employs to efficiently run large-scale AI workloads across diverse and heterogeneous computing hardware, specifically focusing on cost optimization and performance tuning for inference.

## Detailed Analysis

Zain Asgar clarifies that Gimlet Labs has not yet publicly launched its cloud product, but they are actively working with early access customers and partners, particularly in data center environments. The fundamental problem they address is the heterogeneity of compute resources—Nvidia GPUs, Intel/AMD CPUs, and older hardware—which often leads to suboptimal performance and high costs when running large-scale AI inference workloads. Asgar explains that the industry trend has historically favored building massive supercomputers, but for inference, which is increasingly moving to the edge and diverse cloud environments, a more flexible approach is needed. Gimlet's solution involves an agentic system that dynamically orchestrates workloads across available hardware based on cost and performance metrics. They use a multi-layered stack, including a custom compiler layer, to generate optimized kernels for the specific hardware being used. This approach allows them to achieve significant performance improvements (sometimes 20-40% or more) and better unit economics compared to running everything on the newest, most expensive hardware like H100s. The system handles complex routing decisions, ensuring that performance-critical components run on the best available hardware while offloading less critical tasks elsewhere. Asgar notes that while frameworks like PyTorch offer some support, the level of hardware-specific optimization they achieve requires a deeper, more integrated system tailored to managing heterogeneous compute environments.

### Gimlet Labs Focus

- Optimizing AI inference across heterogeneous compute
- Developing a developer-facing product for managing workloads across diverse hardware (AWS, Google Cloud, Azure, and on-premise).

### Core Technical Challenge

- Managing heterogeneity and cost-efficiency
- The challenge is ensuring optimal performance and cost per token when workloads span different vendors (Nvidia, Intel, AMD) and hardware generations.

### Architectural Approach

- Agentic System and Custom Compilation
- They use an agentic graph system to dynamically allocate workloads to the best hardware based on cost/performance metrics, supported by a custom compiler to generate optimized kernels.

### Observed Benefits

- Performance Gains and Cost Reduction
- They see 20-40% performance improvements in some cases by optimizing kernel execution and memory management across hardware tiers, making inference more cost-effective than relying solely on the newest hardware.

### Customer Feedback and Roadmap

- Early Success and Future Focus
- Initial customers are seeing clear benefits, and the focus is on making the system more automated (fewer manual steps) and continuing to improve performance across all supported hardware.

![Screenshot at 00:01: Zain Asgar begins discussing how training hardware has trended toward building supercomputers, contrasting this with the need for efficient inference across heterogeneous compute.](https://ss.rapidrecap.app/screens/GI0hWajiPaM/00-00-01.png)
![Screenshot at 00:49: Zain Asgar is introduced as the Co-Founder & CEO of Gimlet Labs, with the background displaying a complex digital/network graphic.](https://ss.rapidrecap.app/screens/GI0hWajiPaM/00-00-49.png)
![Screenshot at 02:33: Zain Asgar uses hand gestures to illustrate the layered, complex nature of managing AI workloads across different hardware.](https://ss.rapidrecap.app/screens/GI0hWajiPaM/00-02-33.png)
![Screenshot at 05:09: Zain Asgar uses expansive hand gestures to describe the task of breaking down the data flow graph for an agentic system.](https://ss.rapidrecap.app/screens/GI0hWajiPaM/00-05-09.png)
![Screenshot at 38:38: Sam Charrington asks about the heterogeneity faced when deploying models across different environments and hardware.](https://ss.rapidrecap.app/screens/GI0hWajiPaM/00-38-38.png)