Inside LinkedIn’s Generative AI Cookbook: How It Scaled People Search to 1.3 Billion Users

Quick Overview

LinkedIn successfully scaled its People Search feature to support 1.3 billion users by replacing the large, slow 7B parameter ranking model with a smaller, faster 220M parameter model, achieving this massive improvement by focusing on engineering mastery over brute-force scale, specifically by aggressively pruning the model size while maintaining relevance through careful data curation and a multi-teacher distillation process that encoded complex policies into the smaller model.

Key Points: LinkedIn scaled its People Search feature to support 1.3 billion users by aggressively pruning the ranking model size. The initial ranking model used 7 billion parameters, which was deemed too slow and expensive for the required scale. The new, scaled model utilizes only 220 million parameters, a significant reduction achieved through engineering optimization. The key engineering technique involved training the smaller model using distillation from a much larger, complex model (the 'teacher model'). The aggressive cut resulted in less than a 1% loss in relevance, demonstrating a massive leap in efficiency. The engineering effort focused on distilling complex policies like fairness, ethics, and safety into the smaller model, rather than just focusing on model size.

Context: This video details the technical approach LinkedIn took to scale its People Search feature, which had grown to serve over a billion users, a scale that their initial, massive generative AI model (likely a large language model) could not handle efficiently or cost-effectively. The discussion centers on the trade-offs between model size, speed, cost, and maintaining high relevance and policy adherence.

Detailed Analysis

LinkedIn encountered a significant scaling challenge with their generative AI People Search, which aimed to serve over a billion users. Their initial approach relied on a large 7 billion parameter model, which proved too slow and expensive, especially when considering the infrastructure cost of GPU usage for inference. The key breakthrough involved a massive model reduction: they pruned the model down to 220 million parameters. This reduction was not achieved by simply shrinking the model but through a sophisticated multi-teacher distillation process. The engineers used the large, complex model (the 'teacher') to train the much smaller model (the 'student') to mimic its behavior, specifically encoding complex policies related to fairness, ethics, safety, and business value into the smaller model's training. This process resulted in a 20x reduction in input size, yielding a 10x increase in ranking throughput, with only a negligible (less than 1%) loss in relevance metrics compared to the original model. The engineers successfully navigated the accuracy vs. speed trade-off by focusing on engineering mastery—creating a small, fast, compliant model—rather than relying on the costly brute force of simply using the largest available model.