RedOne 2.0: Rethinking Domain-specific LLM Post-Training in Social Networking Services

Quick Overview

The RedOne 2.0 training pipeline successfully demonstrates superior performance over the original RedOne 7B model by prioritizing engagement and factual accuracy during fine-tuning, resulting in a 26% increase in engagement-driven titles and a 1.24 point improvement on the general benchmark despite using less data for specialized tasks.

Key Points: RedOne 2.0, trained with RL-based post-training, significantly outperforms the original RedOne 7B baseline across various metrics. The new model achieved a 26% increase in generating interactive, high-engagement post titles compared to the baseline. The RL-prioritized approach resulted in a 1.24 point improvement on the general benchmark for the 4B parameter model compared to the 7B baseline. The three-stage training pipeline involves Exploratory Learning, Targeted Fine-Tuning (SFT), and Refinement Learning. The technique avoids common pitfalls like catastrophic forgetting by using RL to enforce constraints against sacrificing general knowledge while correcting specific errors. The final stage (Refinement Learning) emphasizes learning the style of engaging content while maintaining factual accuracy, preventing the model from generating plausible but incorrect information. The smaller 4B model trained with the new pipeline outperformed the larger 7B model trained traditionally on general benchmarks (67.57 vs. 58.9).

Context: This podcast episode discusses the development and evaluation of RedOne 2.0, a new iteration of a Large Language Model (LLM) specifically tailored for social networking services (SNS). The core challenge addressed is how to effectively fine-tune LLMs for domain-specific tasks like generating engaging social media content without sacrificing general reasoning capabilities or introducing factual errors, a trade-off often faced when optimizing for engagement metrics.

Detailed Analysis

The discussion details the three-stage training pipeline for RedOne 2.0: Exploratory Learning, Targeted Fine-Tuning (SFT), and Refinement Learning. Stage 1 (Exploratory Learning) involves diagnosis and identifying problems, such as the failure of a single reward signal to cover diverse tasks. Stage 2 (Targeted Fine-Tuning) uses Supervised Fine-Tuning (SFT) to teach the model specific skills like generating engaging post titles and classifying content. A key finding here is that the RL-based approach, which prioritizes engagement over factual accuracy, can lead to catastrophic forgetting, causing the model to forget general knowledge. Stage 3 (Refinement Learning) addresses this by using Reinforcement Learning (RL) to enforce a constraint: the model must maintain its general reasoning ability while learning the specific style needed for high engagement. The researchers found that prioritizing RL on engagement metrics (like click-through rate) over factual correctness led to an 8.7% boost in engagement but caused the model to drop 1.24 points on the general benchmark. The new pipeline, incorporating RL to balance these factors, resulted in a 26% increase in engaging titles while maintaining factual accuracy. Furthermore, the smaller 4B parameter model trained with the new pipeline significantly outperformed the larger 7B baseline model, scoring 67.57 vs. 58.9 on general benchmarks, demonstrating the efficiency and effectiveness of the RL-based refinement process.