How Relevance AI Runs Support Agents: Copilots → Autopilot at Scale

Quick Overview

The presentation concludes that support teams should start by building AI co-pilots to identify and fix issues in real-time before implementing fully autonomous auto-pilot agents, citing past negative experiences with auto-pilots that hallucinated and lacked necessary human oversight, leading to the recommendation to empower subject matter experts (SMEs) on the support team to build these initial co-pilot agents.

Key Points: The critical decision when building AI support agents is whether to start with auto-pilot or co-pilot, with most teams incorrectly choosing auto-pilot first. Early auto-pilot attempts failed due to hallucinations (incorrect information without guardrails), lack of support SME thinking, and no escalation pathways, resulting in lost customers and declining CSAT. Co-pilots, like the agent 'Amy', require human approval before sending responses, allowing teams to identify and fix issues before moving to autonomous systems. Relevance AI promotes the right approach: Agent builders equal the support team, empowering SMEs to build agents without engineering involvement, prioritizing quality before scaling to auto-pilots. The evolution to 'Jack the Auto-Pilot Agent' involved moving Amy's refined tooling to Jack, scaling auto-pilot in two weeks, deploying across free and paid tiers, and establishing clear guardrails and escalation rules. Co-pilots are essential for revealing process gaps in real-time, as demonstrated by a customer asking about available LLMs, which Amy (co-pilot) couldn't answer, leading the team to build a tool to address the gap immediately. The final takeaway encourages committing to learning and evolving in 2025 by using Relevance AI, joining the community, and becoming an AI Agent Builder.

Context: The presentation, delivered by Alex Witts of Relevance AI, addresses the strategic choice between deploying fully autonomous AI support agents (auto-pilot) versus human-in-the-loop assistants (co-pilot). Alex shares personal experience from building support workflows for a large tech company with 1,500 support agents, emphasizing the necessity of a co-pilot approach before scaling to full automation to ensure quality and avoid costly errors.