Paper2Rebuttal: A Multi-Agent Framework for Transparent Author Response Assistance

Quick Overview

The Paper2Rebuttal framework successfully uses an AI agent as a judge to evaluate the quality of a rebuttal paper against an original submission, demonstrating that a smaller, more focused LLM can outperform larger models when guided by a rigorous, multi-metric evaluation framework focusing on clarity, correlation, and evidence retrieval.

Key Points: The Paper2Rebuttal framework uses an AI agent to judge the quality of a rebuttal paper against an initial submission. The evaluation relies on three main metrics: R score (relevance), Pearson and Spearman correlations, and C score (communication/tone). The study found that a smaller model (like the Mini-LLM) consistently outperformed larger models (like GPT-4) when using this structured framework for generating rebuttal evaluations. The agent's role is to synthesize evidence from the original paper and the rebuttal, creating a plan for the human author to follow. The framework forces authors to be more rigorous, stripping away florid language and focusing on core evidence and structure. The process involves the human author running experiments based on the agent's feedback and providing the results for final judgment.

Context: This video discusses the Paper2Rebuttal framework, introduced in a 2023 ICLR paper, designed to assist authors in responding to reviewer critiques during academic paper submissions. The central problem addressed is the difficulty and stress researchers face when writing rebuttals, especially when dealing with vague or hostile feedback. The proposed multi-agent system aims to provide structured, evidence-based assistance to improve the quality and relevance of author responses.

Detailed Analysis

The discussion centers on the Paper2Rebuttal framework, which aims to streamline the process of writing academic paper rebuttals, a task often associated with stress and high stakes. The authors of the framework observed that traditional methods, like simply asking a large language model (LLM) to fix a rebuttal, often lead to issues like context thrashing or simply generating eloquent but ungrounded arguments. The new framework introduces a multi-agent system structured around three phases: Structure, Evidence Construction, and Verify then Write. Phase 1 focuses on structuring the rebuttal, Phase 2 involves constructing the evidence by using a 'compressor' agent to distill the core technical claims and a 'search planner' agent to guide retrieval from external tools like arXiv. Phase 3 is the verification stage where the human author runs experiments based on the plan and provides the evidence, which the AI agent then evaluates using three primary metrics: the R score (relevance to the critique), correlation scores (Pearson and Spearman), and the C score (communication tone, which should be polite and professional). The key finding is that a smaller, specialized LLM using this rigorous framework consistently outperformed much larger, general-purpose models like GPT-4 in generating high-quality evaluations of rebuttals, proving that structure and evidence grounding are more critical than raw model size for this specific task.