REMSA: An LLM Agent for Foundation Model Selection in Remote Sensing

Quick Overview

The REMSA (Remote Sensing Model Selection Agent) LLM agent successfully demonstrated superior performance in selecting appropriate foundation models for remote sensing tasks compared to baseline methods, achieving a top-one hit rate of 22.67% on real-world queries by leveraging a hybrid strategy of rule-based filtering and in-context learning.

Key Points: REMSA achieved a top-one hit rate of 22.67% when selecting foundation models for real-world remote sensing queries, significantly outperforming a simple retrieval baseline. The agent employs a hybrid strategy combining rule-based filtering (e.g., eliminating models that violate hard constraints like requiring SAR support) and in-context learning via a ranking step. The ranking step compares candidate models based on criteria like efficiency vs. performance, modality match, generality, popularity, and recency, using expert-written examples in the prompt. The REMSA agent uses a structured prompt format, which forces the messy data into a consistent structure, making it more reliable than unstructured text-based querying. The system is designed to ensure that if a model lacks a required capability (like SAR support), the agent automatically flags it as a non-starter, preventing poor recommendations. The ultimate goal of REMSA is to transition from a catalog search tool to a proactive AI research assistant that manages the entire selection roadmap for users.

Context: The video introduces REMSA (Remote Sensing Model Selection Agent), an LLM agent designed to address the overwhelming volume of data and model choices in remote sensing, which often leads to information overload for researchers. The challenge is selecting the best foundation model for a specific task (like flood mapping or deforestation tracking) from over 150 available models, a process that is often manual, slow, and prone to error.

Detailed Analysis

The REMSA (Remote Sensing Model Selection Agent) LLM agent addresses the information overload in selecting foundation models for remote sensing tasks by employing a hybrid strategy. This strategy first uses rule-based filtering to eliminate models that fail hard constraints, such as lacking support for specific data modalities like SAR (Synthetic Aperture Radar). This initial filtering step, which removes models violating constraints, is described as crucial for reliability. The remaining candidates then proceed to a ranking tool, which utilizes a structured query format (JSON) to provide the model with context. This ranking step evaluates models based on criteria like efficiency vs. performance, modality match, generality, popularity, and recency, guided by expert-written examples provided in the prompt. In testing, REMSA consistently outperformed a baseline that relied only on raw text descriptions. Specifically, REMSA achieved a top-one hit rate of 22.67% on real-world queries, compared to the baseline's performance, which was roughly half that rate. This success proves that the structured approach, which forces consistency and leverages the LLM's reasoning capabilities to judge complex trade-offs, adds significant value over simple keyword searches or unstructured data inputs.