# REMSA: An LLM Agent for Foundation Model Selection in Remote Sensing

Source: https://www.youtube.com/watch?v=gOc9zTdEjog
Recap page: https://rapidrecap.app/video/gOc9zTdEjog
Generated: 2025-11-25T22:03:53.311+00:00

---
## Quick Overview

The REMSA (Remote Sensing Model Selection Agent) LLM agent successfully demonstrated superior performance in selecting appropriate foundation models for remote sensing tasks compared to baseline methods, achieving a top-one hit rate of 22.67% on real-world queries by leveraging a hybrid strategy of rule-based filtering and in-context learning.

**Key Points:**
- REMSA achieved a top-one hit rate of 22.67% when selecting foundation models for real-world remote sensing queries, significantly outperforming a simple retrieval baseline.
- The agent employs a hybrid strategy combining rule-based filtering (e.g., eliminating models that violate hard constraints like requiring SAR support) and in-context learning via a ranking step.
- The ranking step compares candidate models based on criteria like efficiency vs. performance, modality match, generality, popularity, and recency, using expert-written examples in the prompt.
- The REMSA agent uses a structured prompt format, which forces the messy data into a consistent structure, making it more reliable than unstructured text-based querying.
- The system is designed to ensure that if a model lacks a required capability (like SAR support), the agent automatically flags it as a non-starter, preventing poor recommendations.
- The ultimate goal of REMSA is to transition from a catalog search tool to a proactive AI research assistant that manages the entire selection roadmap for users.

![Screenshot at 01:14: The speaker explains that the agent translates a plain English query into a structured format to query the model database, simplifying the selection process for users.](https://ss.rapidrecap.app/screens/gOc9zTdEjog/00-01-14.png)

**Context:** The video introduces REMSA (Remote Sensing Model Selection Agent), an LLM agent designed to address the overwhelming volume of data and model choices in remote sensing, which often leads to information overload for researchers. The challenge is selecting the best foundation model for a specific task (like flood mapping or deforestation tracking) from over 150 available models, a process that is often manual, slow, and prone to error.

## Detailed Analysis

The REMSA (Remote Sensing Model Selection Agent) LLM agent addresses the information overload in selecting foundation models for remote sensing tasks by employing a hybrid strategy. This strategy first uses rule-based filtering to eliminate models that fail hard constraints, such as lacking support for specific data modalities like SAR (Synthetic Aperture Radar). This initial filtering step, which removes models violating constraints, is described as crucial for reliability. The remaining candidates then proceed to a ranking tool, which utilizes a structured query format (JSON) to provide the model with context. This ranking step evaluates models based on criteria like efficiency vs. performance, modality match, generality, popularity, and recency, guided by expert-written examples provided in the prompt. In testing, REMSA consistently outperformed a baseline that relied only on raw text descriptions. Specifically, REMSA achieved a top-one hit rate of 22.67% on real-world queries, compared to the baseline's performance, which was roughly half that rate. This success proves that the structured approach, which forces consistency and leverages the LLM's reasoning capabilities to judge complex trade-offs, adds significant value over simple keyword searches or unstructured data inputs.

### Model Selection Challenge

- Volume of data leading to information overload
- Over 150 specialized RSFMs available
- Manual selection is slow and error-prone

### REMSA Architecture

- Hybrid strategy using rule-based filtering and in-context learning
- Rule-based filtering removes models violating hard constraints (e.g., lack of SAR support)
- Ranking layer evaluates models on efficiency, performance, generality, popularity, and recency

### Performance Metrics

- Achieved 22.67% top-one hit rate on real-world queries
- Double the performance of a simple retrieval baseline
- Model selection was accurate across diverse applications like flood mapping and deforestation tracking

### Key Agent Tools

- Utilizes an 'Interpreter' to understand natural language queries
- Employs a 'Ranker' for scoring candidates based on criteria
- Incorporates a 'Clarification Generator' to ask follow-up questions if the initial query is vague

![Screenshot at 00:00: The introductory screen displaying the podcast graphic and the call to action: 'Become a Member Today!'](https://ss.rapidrecap.app/screens/gOc9zTdEjog/00-00-00.png)
![Screenshot at 01:14: A visual representation of the structured query process, showing the agent translating plain English into a structured format.](https://ss.rapidrecap.app/screens/gOc9zTdEjog/00-01-14.png)
![Screenshot at 02:25: The speaker details the importance of the structured approach for reliability and accessibility for all expertise levels.](https://ss.rapidrecap.app/screens/gOc9zTdEjog/00-02-25.png)
![Screenshot at 05:54: The speaker explains that the structured approach pulls mandatory information first, then optional details, preventing an endless chat loop.](https://ss.rapidrecap.app/screens/gOc9zTdEjog/00-05-54.png)
![Screenshot at 09:54: The speaker discusses the testing methodology, comparing REMSA against a baseline using 75 realistic queries.](https://ss.rapidrecap.app/screens/gOc9zTdEjog/00-09-54.png)